A cockpit AR glasses rendering method and system based on spatial audio positioning

Abstract

This invention discloses a cockpit AR glasses presentation method and system based on spatial audio positioning, belonging to the field of intelligent cockpit and augmented reality interaction technology. The invention constructs a three-dimensional acoustic perception network by deploying an omnidirectional microphone array and establishes an offline calibrated cockpit acoustic transfer function database; it collects audio signals in real time and dynamically selects a subset of transfer functions for sound source localization based on the user's head position; it transforms the sound source coordinates to a head-mounted coordinate system by combining head posture data; it introduces a semantic parsing module to prioritize concurrent sound sources; and finally, it renders virtual objects in the AR glasses according to perspective rules, constrained by a dynamic gaze focus area. This invention solves the problems of sound-visual spatial mismatch, multi-source interference, and insufficient environmental robustness in existing technologies, achieving high consistency alignment between virtual content and physical sound source positions, significantly improving the spatial accuracy and interactive orderliness of AR information presentation within the cockpit.

Description

Technical Field

[0001] This invention belongs to the field of intelligent cockpit and augmented reality interaction technology, specifically relating to a cockpit AR glasses presentation method and system based on spatial audio positioning. Background Technology

[0002] Against the backdrop of the rapid evolution of smart cockpit technology, augmented reality (AR) human-machine interaction systems are gradually developing from auxiliary information display towards a deep integration of immersive spatial perception. As a key technological path to improve driving safety and the naturalness of interaction, in-vehicle AR systems aim to accurately anchor virtual prompts, navigation guidance, or occupant interaction elements into real physical space, thereby constructing a spatial cognitive framework that conforms to human intuition. In this process, how to achieve high-fidelity spatial alignment between virtual content and dynamic sound sources, user head posture, and fixed structures within the cockpit has become a core indicator for measuring the system's immersion and interaction efficiency.

[0003] Existing in-vehicle augmented reality solutions mostly adopt technical approaches based on visual recognition or preset coordinate mapping. Typical implementations include using a forward-facing camera to detect lane lines and overlay navigation arrows, or projecting fixed controls onto specific locations using a pre-calibrated 3D cockpit model. These methods are practical in static or well-lit environments, and their design logic is based on the premise that visual features are stable and extractable. However, in actual driving, cameras are susceptible to multiple interferences such as strong glare, rain and fog, and occupant movement, leading to feature point loss or mismatch. Simultaneously, preset coordinate mapping struggles to adapt to differences in occupant posture and changes in perspective caused by free head rotation, resulting in virtual objects drifting, misaligning, or even completely inaccurate in the augmented reality view. Especially in multi-occupant scenarios, if the system cannot accurately distinguish the target user's visual and auditory focus, it will further exacerbate the confusion in information presentation and weaken the efficiency of conveying interactive intentions.

[0004] The aforementioned problems stem from the fact that existing technologies fail to treat spatial audio positioning and augmented reality presentation as an inseparable perception-presentation closed loop at the system architecture level, instead separating them into two independent functional modules. This separation not only leads to inaccurate sound-visual spatial mapping but also prevents the system from performing end-to-end optimization for the unique acoustic and physical characteristics of the cockpit and human-computer interaction patterns.

[0005] Therefore, how to construct a unified framework that deeply integrates spatial audio perception, head posture tracking, and cockpit geometry modeling, so that augmented reality content can be dynamically presented with low latency, high consistency, and semantic priority judgment capabilities based on the high-precision sound source location after acoustic environment correction, combined with the user's current gaze direction and interaction intent, has become a key challenge and a technical problem that needs to be solved by those skilled in the art. Summary of the Invention

[0006] To address the problems of sound-visual spatial mismatch, multi-source interference, and insufficient environmental robustness in existing technologies, this invention provides a cockpit AR glasses presentation method, system, device, and storage medium based on spatial audio positioning. This method constructs a unified sound-visual spatial coordinate system and integrates cockpit geometric modeling, multi-channel audio signal processing, head posture tracking, and semantic priority discrimination mechanisms to achieve high consistency, low latency, and strong robust alignment between virtual content and physical sound source locations in the augmented reality field of view.

[0007] This invention is achieved through the following technical solution:

[0008] In a first aspect, the present invention provides a cockpit AR glasses presentation method based on spatial audio positioning, comprising the following steps:

[0009] S1. An omnidirectional microphone array is deployed at multiple fixed locations within the cockpit. The omnidirectional microphone array consists of at least four omnidirectional microphone units, forming a three-dimensional acoustic perception network covering the area of ​​the occupant's head movement.

[0010] S2. Establish a cockpit acoustic transfer function database, which contains direct sound path gain, phase delay, and early reflection sound energy distribution parameters from any spatial point in the cockpit to each microphone unit at different frequencies. This database is obtained through offline calibration.

[0011] S3. Real-time acquisition of multi-channel audio signals in the cockpit, conversion of time-domain signals into frequency-domain representation through short-time Fourier transform, dynamic selection of the corresponding acoustic transfer function subset based on the current user head position, and calculation of the three-dimensional position coordinates of the sound source in the local coordinate system of the cockpit using the maximum likelihood estimation method.

[0012] S4. The user's head posture data is continuously acquired through the inertial measurement unit and magnetometer module integrated in the AR glasses; the three-dimensional position coordinates of the sound source calculated in step S3 and the head posture data are input together into the central collaborative controller to perform coordinate system alignment operation, and the position of the sound source is transformed from the local coordinate system of the cockpit to the head-mounted coordinate system with the center of the user's eyes as the origin.

[0013] S5. Introduce a sound source semantic parsing module, which combines the text transcription results, voiceprint feature vectors and driving task status information output by the vehicle system to classify the interaction intent and prioritize concurrent sound sources.

[0014] S6. Based on the effective sound source location after semantic weighting, render the corresponding virtual object on the AR glasses display panel. The spatial projection of the virtual object follows the perspective transformation rule. Its depth value is determined by the distance from the sound source to the user's head. The horizontal offset is determined by the azimuth angle of the sound source relative to the line of sight in front. The vertical offset is adjusted by the elevation angle parameter. The visibility of the virtual object is constrained by the gaze focus area.

[0015] Furthermore, in step S1, the multiple fixed positions include the dashboard, A-pillar, roof, and seat headrest;

[0016] Each microphone unit in the omnidirectional microphone array is equipped with an independent analog-to-digital converter, with a sampling rate of no less than 48KHz, a quantization bit depth of no less than 24bit, and a time synchronization error between channels controlled within 5us. The synchronization mechanism is implemented by a hardware trigger signal distributed from the master clock to the frame synchronization pin of each analog-to-digital converter.

[0017] Furthermore, in step S2, the cockpit acoustic transfer function database adopts a block sparse storage structure, which is divided into three sub-databases according to frequency range: low frequency band, mid frequency band, and high frequency band. The low frequency band covers 80Hz to 500Hz, the mid frequency band covers 500Hz to 2KHz, and the high frequency band covers 2KHz to 8KHz. The response features of each sub-database are compressed and stored in the form of spherical harmonic function coefficients. During reconstruction, the complete directional response is recovered by expanding the spherical harmonic series.

[0018] Furthermore, in step S2, the calibration process uses a wideband pulse excitation signal to transmit at gridded sampling points in the cockpit, and simultaneously records the signals received by each microphone unit. After deconvolution processing, the spatial-frequency response features are extracted.

[0019] Furthermore, in step S3, the inertial measurement unit includes a three-axis accelerometer, a three-axis gyroscope, and a three-axis magnetometer, and its output is fused by a Kalman filter to generate a head orientation quaternion.

[0020] The specific implementation of the maximum likelihood estimation method includes: constructing a candidate grid for sound source location with a grid resolution of 10cm, covering the entire sound-generating area of ​​the cockpit; for each candidate point, using the subset of acoustic transfer functions corresponding to the current head position index, synthesizing the expected received signal spectrum of each microphone channel; calculating the normalized cross-correlation coefficient between the measured spectrum and the expected spectrum; selecting the candidate point corresponding to the maximum cross-correlation coefficient as the estimated sound source location, and improving the positioning accuracy to the sub-grid level through quadratic interpolation.

[0021] Furthermore, in step S4, the central collaborative controller has a built-in dual buffering mechanism. While the audio positioning result is written to the front buffer, the back buffer maintains the stable output of the previous frame. The switching operation is triggered by the vertical synchronization signal to ensure that the AR rendering engine always reads complete and consistent spatial position data, avoiding screen tearing or shaking due to processing delay.

[0022] Furthermore, in step S5, the driving task status information includes vehicle speed, turn signal status, brake pedal travel, and key node markers of the navigation path;

[0023] The sound source semantic parsing module adopts a two-level discrimination architecture. The first level is a sound source role classifier, which determines the sound source affiliation based on the voiceprint feature vector and distinguishes between the driver, front passenger, rear passenger, or synthesized speech from the vehicle system. The second level is a task relevance evaluator, which combines the matching degree of navigation command keywords, emergency braking event flags, and passenger wake-up word detection results to generate a priority weight value between 0 and 1. Weight values ​​below the threshold of 0.3 are marked as non-interactive background noise, and their corresponding virtual objects are not rendered.

[0024] Furthermore, in step S6, the AR glasses display panel uses optical waveguide technology to achieve virtual-real fusion display. Its optical engine includes a microdisplay, a collimating lens group, and a diffraction grating coupler. The pixel coordinates of the virtual object are mapped to the microdisplay driving signal after being transformed by the perspective projection matrix. The field of view parameters of the perspective projection matrix are dynamically matched with the measured value of the user's interpupillary distance. The matching process is achieved through periodic calibration by the infrared eye-tracking module.

[0025] Furthermore, in step S6, the focal point area is a variable-angle cone-shaped area centered on the current line of sight, and its angle is dynamically adjusted according to the current driving load state; the corresponding virtual object is only activated and displayed when the sound source falls into the dynamic cone-shaped area.

[0026] In a second aspect, the present invention also provides a cockpit AR glasses presentation system based on spatial audio positioning, for implementing the method described in the first aspect, the system comprising:

[0027] The cockpit audio acquisition module includes an omnidirectional microphone array deployed at preset locations within the cockpit for acquiring multi-channel audio signals;

[0028] The acoustic environment modeling module is used to store and call a pre-calibrated cockpit acoustic transfer function database;

[0029] The sound source localization engine is communicatively connected to the cockpit audio acquisition module, the acoustic environment modeling module, and the semantic priority discrimination module, respectively. It is used to receive multi-channel audio signals and call the corresponding subset of acoustic transfer functions from the acoustic environment modeling module. It calculates the three-dimensional position coordinates of the sound source in the local coordinate system of the cockpit in combination with the current user's head position, and outputs the three-dimensional position coordinates to the semantic priority discrimination module.

[0030] The semantic priority discrimination module is used to receive semantic information and driving task status information output by the vehicle system, classify and prioritize the concurrent sound sources received from the sound source localization engine, and output the effective sound source location and its semantic information after semantic weighting.

[0031] The head pose tracking module, integrated into the AR glasses, is used to acquire the user's head pose data;

[0032] The coordinate system alignment unit is communicatively connected to the semantic priority discrimination module and the head posture tracking module, respectively. It is used to receive the effective sound source location and its semantic information, and combine it with the head posture data to transform the sound source location from the cockpit local coordinate system to the head-mounted coordinate system, and output the aligned sound source location and its semantic information.

[0033] The AR rendering engine is connected in communication with the coordinate system alignment unit. It is used to render the corresponding virtual object on the AR glasses display panel according to the aligned sound source position and its semantic information, and to impose a gaze focus area constraint on the visibility of the virtual object.

[0034] Compared with the prior art, the advantages of the present invention are as follows:

[0035] 1. The present invention provides a cockpit AR glasses presentation method and system based on spatial audio positioning. By constructing a unified sound-visual spatial coordinate system, spatial audio positioning is deeply embedded into the augmented reality presentation process. A three-dimensional spatial anchoring mechanism is established based on the physical acoustic characteristics of the cockpit, with the user's head posture as a reference and interactive semantics as a filtering criterion. This fundamentally solves the problems of sound-visual spatial mismatch and insufficient environmental robustness caused by reliance on visual information in the prior art.

[0036] 2. By introducing a sound source semantic parsing module and dynamic gaze focus area constraints, this invention effectively addresses multi-sound source interference, avoids information overload, and improves the orderliness of interaction and driving safety. Ultimately, it achieves accurate, orderly, and ergonomically sound presentation of virtual information in the real cockpit space, significantly enhancing the user's immersive experience. Attached Figure Description

[0037] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0038] Figure 1 A schematic diagram illustrating the workflow of a cockpit AR glasses presentation method based on spatial audio positioning;

[0039] Figure 2 A schematic diagram of the overall structure of a cockpit AR glasses presentation system based on spatial audio positioning;

[0040] Figure 3 A schematic diagram showing the deployment location of the omnidirectional microphone array inside the cockpit and the construction of the three-dimensional acoustic sensing network;

[0041] Figure 4 This is a block diagram showing the module connection of the sound source localization and head posture data fusion processing flow described in this invention.

[0042] Figure 5 This is a schematic diagram illustrating the coordinate system alignment principle for the transformation from the cockpit local coordinate system to the head-mounted coordinate system in this invention;

[0043] Figure 6 This is a schematic diagram illustrating the virtual object rendering mechanism of the AR glasses described in this invention and its constraint relationship with the gaze focus area. Detailed Implementation

[0044] To clearly and completely describe the technical solution and its specific working process of the present invention, the specific embodiments of the present invention are as follows, in conjunction with the accompanying drawings:

[0045] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0046] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "over," and "on top" of the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0047] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0048] Example 1

[0049] like Figure 1 As shown, this embodiment provides a cockpit AR glasses presentation method based on spatial audio positioning, including the following steps:

[0050] S1. An omnidirectional microphone array is deployed at multiple fixed locations within the cockpit, including the dashboard, A-pillars, ceiling, and seat headrests; the omnidirectional microphone array consists of at least four omnidirectional microphone units, forming a three-dimensional acoustic perception network covering the area of ​​occupant head movement, such as... Figure 3 As shown;

[0051] In this embodiment, each omnidirectional microphone unit is equipped with an independent analog-to-digital converter, with a sampling rate of no less than 48kHz, a quantization bit depth of no less than 24bit, and the time synchronization error between each channel is controlled within 5µs. The synchronization mechanism is implemented by a hardware trigger signal distributed from the master clock to the frame synchronization pin of each analog-to-digital converter.

[0052] S2. Establish a cockpit acoustic transfer function database, which contains direct sound path gain, phase delay, and early reflection sound energy distribution parameters from any spatial point in the cockpit to each microphone unit at different frequencies. This database is obtained through offline calibration.

[0053] The cockpit acoustic transfer function database adopts a block sparse storage structure, which is divided into three sub-databases according to frequency range: low frequency, mid frequency and high frequency. The low frequency range covers 80Hz to 500Hz, the mid frequency range covers 500Hz to 2KHz, and the high frequency range covers 2KHz to 8KHz. The response features in each sub-database are compressed and stored in the form of spherical harmonic function coefficients. During reconstruction, the complete directional response is recovered by expanding the spherical harmonic series.

[0054] S3. Real-time acquisition of multi-channel audio signals in the cockpit, conversion of time-domain signals into frequency-domain representation through short-time Fourier transform, dynamic selection of the corresponding acoustic transfer function subset based on the current user head position, and calculation of the three-dimensional position coordinates of the sound source in the local coordinate system of the cockpit using the maximum likelihood estimation method.

[0055] The specific implementation of the maximum likelihood estimation method includes: constructing a candidate grid for sound source location with a grid resolution of 10cm, covering the entire sound-generating area of ​​the cockpit; for each candidate point, using the subset of acoustic transfer functions corresponding to the current head position index, synthesizing the expected received signal spectrum of each microphone channel; calculating the normalized cross-correlation coefficient between the measured spectrum and the expected spectrum; selecting the candidate point corresponding to the maximum cross-correlation coefficient as the estimated sound source location, and improving the positioning accuracy to the sub-grid level through quadratic interpolation.

[0056] S4. The inertial measurement unit and magnetometer module integrated into the AR glasses continuously acquire the user's head posture data; the three-dimensional position coordinates of the sound source calculated in step S3 are input together with the head posture data into the central collaborative controller to perform a coordinate system alignment operation, transforming the sound source position from the cockpit local coordinate system to the head-mounted coordinate system with the center of the user's eyes as the origin, such as... Figure 4 and Figure 5 As shown;

[0057] The central collaborative controller has a built-in dual buffering mechanism. While the audio positioning results are written to the front buffer, the back buffer maintains the stable output of the previous frame. The switching operation is triggered by the vertical synchronization signal, ensuring that the AR rendering engine always reads complete and consistent spatial position data, avoiding screen tearing or shaking due to processing delays.

[0058] S5. Introduce a sound source semantic parsing module, which combines the text transcription results, voiceprint feature vectors and driving task status information output by the vehicle system to classify the interaction intent and prioritize concurrent sound sources.

[0059] The sound source semantic parsing module adopts a two-level discrimination architecture. The first level is a sound source role classifier, which determines the sound source affiliation based on the voiceprint feature vector and distinguishes between the driver, front passenger, rear passenger, or synthesized speech from the vehicle system. The second level is a task relevance evaluator, which combines the matching degree of navigation command keywords, emergency braking event flags, and passenger wake-up word detection results to generate a priority weight value between 0 and 1. Weight values ​​below the threshold of 0.3 are marked as non-interactive background noise, and their corresponding virtual objects are not rendered.

[0060] S6. Based on the semantically weighted effective sound source location, render the corresponding virtual object on the AR glasses display panel. The spatial projection of the virtual object follows perspective transformation rules. Its depth value is determined by the distance from the sound source to the user's head. The horizontal offset is determined by the azimuth angle of the sound source relative to the line of sight directly in front. The vertical offset is adjusted by the elevation angle parameter. Furthermore, the visibility of the virtual object is constrained by the focal point area of ​​the gaze. Figure 6 As shown;

[0061] The AR glasses display panel uses optical waveguide technology to achieve virtual-real fusion display. Its optical engine includes a microdisplay, collimating lens group and diffraction grating coupler. The pixel coordinates of the virtual object are transformed by the perspective projection matrix and mapped to the microdisplay driving signal. The field of view parameters of the perspective projection matrix are dynamically matched with the measured value of the user's interpupillary distance. The matching process is achieved by periodic calibration through the infrared eye tracking module.

[0062] The cone-shaped boundary of the focus area is not a fixed geometry, but is dynamically adjusted according to the current driving load. When the vehicle is cruising on the highway and there is no indication of an approaching turn, the cone angle expands to 45 degrees. When an intersection or lane change is detected ahead, the angle shrinks to 20 degrees to concentrate the user's attention on key navigation elements.

[0063] Example 2

[0064] like Figure 2 As shown, this embodiment provides a cockpit AR glasses presentation system based on spatial audio positioning, used to implement the method described in Embodiment 1. The system includes:

[0065] The cockpit audio acquisition module includes an omnidirectional microphone array deployed at preset locations within the cockpit for acquiring multi-channel audio signals;

[0066] The acoustic environment modeling module is used to store and call a pre-calibrated cockpit acoustic transfer function database;

[0067] The sound source localization engine is communicatively connected to the cockpit audio acquisition module, the acoustic environment modeling module, and the semantic priority discrimination module, respectively. It is used to receive multi-channel audio signals and call the corresponding subset of acoustic transfer functions from the acoustic environment modeling module. It calculates the three-dimensional position coordinates of the sound source in the local coordinate system of the cockpit in combination with the current user's head position, and outputs the three-dimensional position coordinates to the semantic priority discrimination module.

[0068] The semantic priority discrimination module is used to receive semantic information and driving task status information output by the vehicle system, classify and prioritize the concurrent sound sources received from the sound source localization engine, and output the effective sound source location and its semantic information after semantic weighting.

[0069] The head pose tracking module, integrated into the AR glasses, is used to acquire the user's head pose data;

[0070] The coordinate system alignment unit is communicatively connected to the semantic priority discrimination module and the head posture tracking module, respectively. It is used to receive the effective sound source location and its semantic information, and combine it with the head posture data to transform the sound source location from the cockpit local coordinate system to the head-mounted coordinate system, and output the aligned sound source location and its semantic information.

[0071] The AR rendering engine is connected in communication with the coordinate system alignment unit. It is used to render the corresponding virtual object on the AR glasses display panel according to the aligned sound source position and its semantic information, and to impose a gaze focus area constraint on the visibility of the virtual object.

[0072] The modules mentioned above are interconnected via a high-speed serial link. The data stream uses a timestamp mechanism to ensure cross-module synchronization. The timestamp is generated uniformly by a global real-time clock chip and broadcast to all functional units.

[0073] In this embodiment, the system further includes an environmental interference suppression unit, which monitors the noise from the air conditioning vent, tire rolling noise, and engine harmonic components in real time. It uses an adaptive notch filter to eliminate known interference frequency bands in the frequency domain, retaining the effective frequency bands for voice and prompts, and ensuring that the sound source localization is not affected by steady-state mechanical noise pollution.

[0074] The system is deployed at the domain controller level in the vehicle's electronic and electrical architecture, sharing memory resources with the smart cockpit domain controller. Raw audio data is transmitted directly to a dedicated audio processing accelerator via the PCIe interface, avoiding latency jitter caused by the general operating system kernel. The end-to-end processing latency is controlled within 25ms.

[0075] The system deeply embeds spatial audio positioning into the augmented reality presentation process, establishing a three-dimensional spatial anchoring mechanism based on the physical acoustic characteristics of the cockpit, the user's head posture as a reference, and interactive semantics as a filtering criterion. This fundamentally solves the problems of sound-visual spatial mismatch, multi-source interference, and insufficient environmental robustness in existing technologies, and realizes the accurate, orderly, and human factors engineering-compliant presentation of virtual information in the real cockpit space.

[0076] The preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the specific details of the above embodiments. Within the scope of the technical concept of the present invention, various simple modifications can be made to the technical solution of the present invention, and these simple modifications all fall within the protection scope of the present invention.

[0077] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable way without contradiction. In order to avoid unnecessary repetition, the present invention will not describe the various possible combinations separately.

[0078] Furthermore, various different embodiments of the present invention can be combined in any way, as long as they do not violate the spirit of the present invention, they should also be regarded as the content disclosed by the present invention.

Claims

1. A cockpit AR glasses presentation method based on spatial audio positioning, characterized in that, Includes the following steps: S1. An omnidirectional microphone array is deployed at multiple fixed locations within the cockpit. The omnidirectional microphone array consists of at least four omnidirectional microphone units, forming a three-dimensional acoustic perception network covering the area of ​​the occupant's head movement. S2. Establish a cockpit acoustic transfer function database, which contains direct sound path gain, phase delay, and early reflection sound energy distribution parameters from any spatial point in the cockpit to each microphone unit at different frequencies. This database is obtained through offline calibration. S3. Real-time acquisition of multi-channel audio signals in the cockpit, conversion of time-domain signals into frequency-domain representation through short-time Fourier transform, dynamic selection of the corresponding acoustic transfer function subset based on the current user head position, and calculation of the three-dimensional position coordinates of the sound source in the local coordinate system of the cockpit using the maximum likelihood estimation method. S4. The user's head posture data is continuously acquired through the inertial measurement unit and magnetometer module integrated in the AR glasses; the three-dimensional position coordinates of the sound source calculated in step S3 and the head posture data are input together into the central collaborative controller to perform coordinate system alignment operation, and the position of the sound source is transformed from the local coordinate system of the cockpit to the head-mounted coordinate system with the center of the user's eyes as the origin. S5. Introduce a sound source semantic parsing module, which combines the text transcription results, voiceprint feature vectors and driving task status information output by the vehicle system to classify the interaction intent and prioritize concurrent sound sources. S6. Based on the effective sound source location after semantic weighting, render the corresponding virtual object on the AR glasses display panel. The spatial projection of the virtual object follows the perspective transformation rule. Its depth value is determined by the distance from the sound source to the user's head. The horizontal offset is determined by the azimuth angle of the sound source relative to the line of sight in front. The vertical offset is adjusted by the elevation angle parameter. The visibility of the virtual object is constrained by the gaze focus area.

2. The cockpit AR glasses presentation method based on spatial audio positioning as described in claim 1, characterized in that, In step S1, the multiple fixed positions include the dashboard, A-pillar, roof, and seat headrest; Each microphone unit in the omnidirectional microphone array is equipped with an independent analog-to-digital converter, with a sampling rate of no less than 48KHz, a quantization bit depth of no less than 24bit, and a time synchronization error between channels controlled within 5us. The synchronization mechanism is implemented by a hardware trigger signal distributed from the master clock to the frame synchronization pin of each analog-to-digital converter.

3. The cockpit AR glasses presentation method based on spatial audio positioning as described in claim 1, characterized in that, In step S2, the cockpit acoustic transfer function database adopts a block sparse storage structure, which is divided into three sub-databases according to frequency range: low frequency band, mid frequency band and high frequency band. The low frequency band covers 80Hz to 500Hz, the mid frequency band covers 500Hz to 2KHz, and the high frequency band covers 2KHz to 8KHz. The response features of each sub-database are compressed and stored in the form of spherical harmonic function coefficients. During reconstruction, the complete directional response is recovered by expanding the spherical harmonic series.

4. The cockpit AR glasses presentation method based on spatial audio positioning as described in claim 1, characterized in that, In step S2, the calibration process uses a wideband pulse excitation signal to transmit at gridded sampling points in the cockpit, and simultaneously records the signals received by each microphone unit. After deconvolution processing, the spatial-frequency response features are extracted.

5. The cockpit AR glasses presentation method based on spatial audio positioning as described in claim 1, characterized in that, In step S3, the inertial measurement unit includes a three-axis accelerometer, a three-axis gyroscope, and a three-axis magnetometer. Its output is fused by a Kalman filter to generate a head orientation quaternion. The specific implementation of the maximum likelihood estimation method includes: constructing a candidate grid for sound source location with a grid resolution of 10cm, covering the entire sound-generating area of ​​the cockpit; for each candidate point, using the subset of acoustic transfer functions corresponding to the current head position index, synthesizing the expected received signal spectrum of each microphone channel; calculating the normalized cross-correlation coefficient between the measured spectrum and the expected spectrum; selecting the candidate point corresponding to the maximum cross-correlation coefficient as the estimated sound source location, and improving the positioning accuracy to the sub-grid level through quadratic interpolation.

6. The cockpit AR glasses presentation method based on spatial audio positioning as described in claim 1, characterized in that, In step S4, the central collaborative controller has a built-in dual buffering mechanism. While the audio positioning result is written to the front buffer, the back buffer maintains the stable output of the previous frame. The switching operation is triggered by the vertical synchronization signal to ensure that the AR rendering engine always reads complete and consistent spatial position data, avoiding screen tearing or shaking due to processing delay.

7. The cockpit AR glasses presentation method based on spatial audio positioning as described in claim 1, characterized in that, In step S5, the driving task status information includes vehicle speed, turn signal status, brake pedal travel, and key node markers of the navigation path; The sound source semantic parsing module adopts a two-level discrimination architecture. The first level is a sound source role classifier, which determines the sound source affiliation based on the voiceprint feature vector and distinguishes between the driver, front passenger, rear passenger, or synthesized speech from the vehicle system. The second level is a task relevance evaluator, which combines the matching degree of navigation command keywords, emergency braking event flags, and passenger wake-up word detection results to generate a priority weight value between 0 and 1. Weight values ​​below the threshold of 0.3 are marked as non-interactive background noise, and their corresponding virtual objects are not rendered.

8. The cockpit AR glasses presentation method based on spatial audio positioning as described in claim 1, characterized in that, In step S6, the AR glasses display panel uses optical waveguide technology to achieve virtual-real fusion display. Its optical engine includes a microdisplay, a collimating lens group and a diffraction grating coupler. The pixel coordinates of the virtual object are mapped to the microdisplay driving signal after being transformed by the perspective projection matrix. The field of view parameters of the perspective projection matrix are dynamically matched with the measured value of the user's interpupillary distance. The matching process is achieved through periodic calibration by the infrared eye tracking module.

9. A cockpit AR glasses presentation method based on spatial audio positioning as described in claim 1, characterized in that, In step S6, the focal point area is a variable-angle cone-shaped area centered on the current line of sight, and its angle is dynamically adjusted according to the current driving load state; the corresponding virtual object is only activated and displayed when the sound source falls into the dynamic cone-shaped area.

10. A cockpit AR glasses presentation system based on spatial audio positioning, used to implement the method as described in any one of claims 1-9, characterized in that, The system includes: The cockpit audio acquisition module includes an omnidirectional microphone array deployed at preset locations within the cockpit for acquiring multi-channel audio signals; The acoustic environment modeling module is used to store and call a pre-calibrated cockpit acoustic transfer function database; The sound source localization engine is communicatively connected to the cockpit audio acquisition module, the acoustic environment modeling module, and the semantic priority discrimination module, respectively. It is used to receive multi-channel audio signals and call the corresponding subset of acoustic transfer functions from the acoustic environment modeling module. It calculates the three-dimensional position coordinates of the sound source in the local coordinate system of the cockpit in combination with the current user's head position, and outputs the three-dimensional position coordinates to the semantic priority discrimination module. The semantic priority discrimination module is used to receive semantic information and driving task status information output by the vehicle system, classify and prioritize the concurrent sound sources received from the sound source localization engine, and output the effective sound source location and its semantic information after semantic weighting. The head pose tracking module, integrated into the AR glasses, is used to acquire the user's head pose data; The coordinate system alignment unit is communicatively connected to the semantic priority discrimination module and the head posture tracking module, respectively. It is used to receive the effective sound source location and its semantic information, and combine it with the head posture data to transform the sound source location from the cockpit local coordinate system to the head-mounted coordinate system, and output the aligned sound source location and its semantic information. The AR rendering engine is connected in communication with the coordinate system alignment unit. It is used to render the corresponding virtual object on the AR glasses display panel according to the aligned sound source position and its semantic information, and to impose a gaze focus area constraint on the visibility of the virtual object.

Images