Audio processing method, head-mounted display device, and computer-readable storage medium

Abstract

The application discloses an audio processing method, a head-mounted display device and a computer readable storage medium. The audio processing method comprises the following steps: determining a first audio signal attenuation degree corresponding to a target audio picked up by each second head-mounted display device; selecting a reference head-mounted display device with a first audio signal attenuation degree less than a preset attenuation degree threshold from each second head-mounted display device, and obtaining reference audio information obtained by picking up the target audio by the reference head-mounted display device; obtaining first audio information by picking up the target audio, and performing compensation adjustment on the first audio information according to the reference audio information to obtain second audio information; and rendering the second audio information to an extended reality environment. The application can improve the accuracy of sound effect rendering in extended reality applications.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of head-mounted display devices, and particularly relates to an audio processing method, a head-mounted display device and a computer readable storage medium. BACKGROUND

[0002] Virtual Reality (VR) technology is a technology for simulating a three-dimensional virtual world by using a computer or other intelligent computing devices, providing users with visual, auditory and tactile simulations of the senses, so that people can feel as if they were there. At present, head-mounted display devices such as AR (Augmented Reality) / VR glasses or AR / VR headsets based on VR technology have appeared in the market, and users can experience an extended reality scene by wearing the head-mounted display devices.

[0003] The current head-mounted display devices cannot accurately render the audio signals in the real environment to the extended reality environment when presenting the extended reality scene, so that the sound heard by the user in the extended reality environment is not accurate enough, and the user's experience of the sound of the real world in the extended reality environment is reduced. That is, the accuracy of sound effect rendering in the prior art is low, which seriously affects the immersion of the user in the virtual reality environment. SUMMARY

[0004] The main purpose of the present application is to provide an audio processing method, a head-mounted display device and a computer readable storage medium, which aims to solve the technical problem of how to improve the accuracy of sound effect rendering in the extended reality application.

[0005] To achieve the above-mentioned purpose, the present application provides an audio processing method, which is applied to a first head-mounted device, the first head-mounted device and a plurality of second head-mounted devices construct a communication link, and the method comprises the following steps:

[0006] determining the first audio signal attenuation degree corresponding to the target audio picked up by each second head-mounted device;

[0007] selecting a reference head-mounted device with a first audio signal attenuation degree less than a preset attenuation degree threshold from each second head-mounted device, and obtaining reference audio information obtained by picking up target audio by the reference head-mounted device;

[0008] picking up the target audio to obtain first audio information, and performing compensation adjustment on the first audio information according to the reference audio information to obtain second audio information;

[0009] rendering the second audio information to an extended reality environment.

[0010] Optionally, after the step of determining the first audio signal attenuation degree of each of the second HMD device picking up the target audio, the method further comprises:

[0011] determining a second audio signal attenuation degree of the first HMD device picking up the target audio;

[0012] judging whether there is a first audio signal attenuation degree less than the second audio signal attenuation degree in each of the first audio signal attenuation degrees;

[0013] if yes, performing the step of selecting a reference HMD device with a first audio signal attenuation degree less than a preset attenuation threshold from each of the second HMD devices.

[0014] Optionally, the acoustic parameter includes a beam amplitude, and the step of compensating and adjusting the first audio information according to the reference audio information comprises:

[0015] identifying the reference audio information to obtain a reference audio intensity of the target audio identified by the reference HMD device, and identifying the first audio information to obtain a to-be-calibrated audio intensity of the target audio identified by the first HMD device;

[0016] determining an audio intensity loss value of the target audio identified by the first HMD device according to the reference audio intensity and the to-be-calibrated audio intensity, and determining a beam amplitude loss value of the first audio information according to the audio intensity loss value;

[0017] compensating and adjusting the beam amplitude of the first audio information according to the beam amplitude loss value.

[0018] Optionally, the acoustic parameter includes a beam phase, and the step of compensating and adjusting the first audio information according to the reference audio information comprises:

[0019] identifying the reference audio information to obtain a relative spatial position of a sound emitting object of the target audio relative to the reference HMD device;

[0020] acquiring a first device position of the reference HMD device at present through the communication link, and determining a reference sound emitting position of the target audio identified by the reference HMD device according to the first device position and the relative spatial position;

[0021] determining a first sound emitting position of the target audio identified by the first HMD device through sound source positioning of the target audio by the first audio information;

[0022] determine a sound emitting position deviation value of the target audio recognized by the first HMD device according to the reference sound emitting position and the first sound emitting position, and determine a beam phase offset value of the first audio information according to the sound emitting position deviation value;

[0023] perform compensation adjustment on the beam phase of the first audio information according to the beam phase offset value.

[0024] Optionally, before the step of selecting the reference HMD device with the first audio signal attenuation degree less than the preset attenuation degree threshold from the second HMD devices, the method further comprises:

[0025] if there is no first audio signal attenuation degree less than the preset attenuation degree threshold in the first audio signal attenuation degrees, selecting the reference HMD device with the lowest first audio signal attenuation degree from the second HMD devices, and performing the step of obtaining the reference audio information picked up by the reference HMD device from the target audio;

[0026] if there is a first audio signal attenuation degree less than the preset attenuation degree threshold in the first audio signal attenuation degrees, performing the step of selecting the reference HMD device with the first audio signal attenuation degree less than the preset attenuation degree threshold from the second HMD devices.

[0027] Optionally, the step of determining the audio signal attenuation degree corresponding to the target audio picked up by each of the second HMD devices comprises:

[0028] obtaining the current second device position of each of the second HMD devices through the communication link;

[0029] determining a second sound emitting position of the target audio recognized by each of the second HMD devices through sound source positioning of the target audio by each of the second HMD devices;

[0030] determining the audio propagation path of the target audio relative to each of the second HMD devices according to each of the second device positions and each of the second sound emitting positions;

[0031] determining the audio signal attenuation degree corresponding to the target audio picked up by each of the second HMD devices according to the audio propagation path of each of the second HMD devices.

[0032] Optionally, the step of determining the audio signal attenuation degree corresponding to the target audio picked up by each of the second HMD devices according to the audio propagation path of each of the second HMD devices comprises:

[0033] obtaining the blocking range formed by the blocking physical object in the direction perpendicular to the audio propagation path in each of the audio propagation paths through the communication link;

[0034] determine, according to each audio occlusion area and each audio diffusion area, an audio signal attenuation degree corresponding to the target audio picked up by each second head-mounted device, wherein each second head-mounted device corresponds to one audio occlusion area and one audio diffusion area.

[0035] obtain an audio diffusion area covered by the target audio in each audio propagation path, and determine, according to each audio occlusion area and each audio diffusion area, an audio signal attenuation degree corresponding to the target audio picked up by each second head-mounted device, wherein each second head-mounted device corresponds to one audio occlusion area and one audio diffusion area.

[0036] Optionally, the step of determining, according to each audio occlusion area and each audio diffusion area, an audio signal attenuation degree corresponding to the target audio picked up by each second head-mounted device comprises:

[0037] perform difference calculation on the audio diffusion area and the audio occlusion area corresponding to each second head-mounted device to obtain an audio propagation area corresponding to each second head-mounted device to which the target audio is propagated by the sound-emitting object;

[0038] determine, according to each audio propagation area, a sound propagation angle of the target audio propagated by the sound-emitting object to each second head-mounted device, and determine, according to each sound propagation angle, a sound-picking offset angle of each second head-mounted device picking up the target audio;

[0039] query, from a preset mapping data table, an audio signal attenuation degree mapped by each sound-picking offset angle;

[0040] determine, according to the audio signal attenuation degree mapped by each sound-picking offset angle, an audio signal attenuation degree corresponding to the target audio picked up by each second head-mounted device.

[0041] The application further provides a head-mounted display device, which is a physical device. The head-mounted display device comprises a memory, a processor, and a program of the audio processing method stored in the memory and executable on the processor. When the program of the audio processing method is executed by the processor, the steps of the audio processing method described above can be implemented.

[0042] The application further provides a computer readable storage medium having a program for implementing an audio processing method stored thereon. When the program for implementing the audio processing method is executed by a processor, the steps of the audio processing method described above can be implemented.

[0043] The application further provides a computer program product comprising a computer program. When the computer program is executed by a processor, the steps of the audio processing method described above can be implemented.

[0044] Due to different distances of the same sound source from different users, different head-mounted devices can recognize different accuracies of the target audio information, for example, the farther the head-mounted device is from the sound source or the greater the density / volume of the structure between the head-mounted device and the sound source, the more incomplete / inaccurate the target audio information picked up by the head-mounted device.

[0045] Based on this, the present application provides an audio processing method, which is applied to a first head-mounted device, and the first head-mounted device constructs a communication link with a plurality of second head-mounted devices. The technical solution of the present application is to determine the first audio signal attenuation degree of each second head-mounted device picking up the target audio, thereby analyzing the audio information completeness / accuracy of the target audio that each second head-mounted device can pick up, and selecting a reference head-mounted device from the second head-mounted devices, the first audio signal attenuation degree of which is less than a preset attenuation degree threshold, thereby determining the head-mounted device with higher audio information completeness / accuracy among the other head-mounted devices in collaborative interaction, to ensure the real accuracy of picking up the target audio information, and then obtaining reference audio information by picking up the target audio of the reference head-mounted device, and then picking up the target audio to obtain first audio information, and then compensating and adjusting the acoustic parameters of the first audio information according to the reference audio information to obtain second audio information, and then rendering the second audio information to the extended reality environment, thereby accurately rendering the audio signal in the real environment to the extended reality environment.

[0046] The present application makes it possible to pick up the target audio by the head-mounted devices worn by each user in some application scenarios of multi-person collaborative interaction (for example, in a performance, a concert, and a multi-person meeting scenario), and share the target audio with other head-mounted devices in collaborative interaction. After receiving the shared target audio, the other head-mounted devices can play back and display the target audio information (which can include audio semantic information and sound source position), and compensate and adjust the acoustic parameters of the first audio information picked up by the other head-mounted devices according to the target audio shared by the other head-mounted devices, so as to calibrate the spatial position deviation of the target audio or the loss of the intensity of the target audio caused by the head-mounted device worn by the user due to the distance from the sound source or the existence of obstacles between the head-mounted device and the sound source, thereby enabling the user to more clearly and accurately obtain the target audio information and distinguish the sound source position corresponding to the target audio, enabling the user to hear more accurate and real sounds in the extended reality environment, and thereby overcoming the technical problem of how to improve the accuracy of sound effect rendering in the extended reality application. BRIEF DESCRIPTION OF DRAWINGS

[0047] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present application and, together with the specification, serve to explain the principles of the present application.

[0048] In order to more clearly illustrate the technical solutions in the embodiments or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without any creative effort.

[0049] Figure 1 The flowchart of the first embodiment of the audio processing method of the present application;

[0050] Figure 2 The flowchart of the second embodiment of the audio processing method of the present application;

[0051] Figure 3 The scene diagram of multi-user multi-sound source audio recognition in the embodiment of the present application;

[0052] Figure 4 The module structure diagram of the audio processing device in the embodiment of the present application;

[0053] Figure 5 The device structure diagram of the hardware running environment related to the head-mounted display device in the embodiment of the present application.

[0054] The purposes, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0055] In order to make the above-mentioned purposes, features and advantages of the present application more apparent and easy to understand, the technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative effort fall within the scope of protection of the present application.

[0056] In the embodiment, the head-mounted display device includes but is not limited to a mixed reality (Mixed Reality) - MR device (such as MR glasses or MR helmet), an augmented reality (Augmented Reality) - AR device (such as AR glasses or AR helmet), a virtual reality (Virtual Reality) - VR device (such as VR glasses or VR helmet), an extended reality (Extended Reality) - XR device or some combination thereof, and the like.

[0057] Embodiment one

[0058] The current head-mounted display device cannot accurately render the audio signal in the real environment to the extended reality environment when presenting the extended reality scene, so that the sound heard by the user in the extended reality environment is not accurate enough, and the user's experience of the real world sound in the extended reality environment is reduced, that is, the accuracy of the sound effect rendering of the prior art is low, which seriously affects the immersion of the user in the virtual reality environment.

[0059] Based on this, please refer to Figure 1 The audio processing method is applied to a first head-mounted device, the first head-mounted device constructs a communication link with a plurality of second head-mounted devices, and the method comprises the following steps:

[0060] Step S10, determining the first audio signal attenuation degree corresponding to the target audio picked up by each second head-mounted device;

[0061] In the embodiment, the first head-mounted device refers to the execution subject of the audio processing method of the application, which is a head-mounted device worn by a first user, and the second head-mounted device is a head-mounted device worn by a second user different from the first user. The target audio refers to the audio required to be picked up by the second head-mounted device.

[0062] As known by those skilled in the art, the propagation of sound will produce attenuation of the sound signal with the increase of distance or the blocking of a blocking object (such as a building body, a glass body, a sponge body, etc.). That is, under the condition that other factors are the same (single variable), the farther the distance of sound, the greater the audio signal attenuation degree. Or under the condition that other factors are the same (single variable, such as equal distance, same material of blocking object, same material of blocking object, and equal blocking area of blocking object in the direction of audio propagation), the greater the blocking thickness of the blocking object that attenuates the sound, the greater the audio signal attenuation degree. Or under the condition that other factors are the same (single variable, such as equal distance, same material of blocking object, same material of blocking object, and equal thickness of blocking object), the greater the blocking area of the blocking object in the direction of audio propagation, the greater the audio signal attenuation degree.

[0063] In the embodiment, the first audio signal attenuation degree refers to the audio signal attenuation degree corresponding to the target audio picked up by the second head-mounted device.

[0064] Step S20, selecting a reference head-mounted device with a first audio signal attenuation degree less than a preset attenuation degree threshold from each second head-mounted device, and obtaining reference audio information obtained by picking up the target audio by the reference head-mounted device;

[0065] In the embodiment, the reference HMD device refers to a HMD device selected from the second HMD devices, and the attenuation degree of the first audio signal of the HMD device is less than a preset attenuation threshold. The reference audio information refers to audio information obtained by the reference HMD device picking up the target audio.

[0066] The preset attenuation threshold can be set by those skilled in the art according to actual conditions, and the embodiment is not limited specifically, so as to better select a HMD device with a lower audio signal attenuation degree from the second HMD devices without affecting the identification of the target audio information (avoiding distortion of the target audio).

[0067] In step S30, first audio information is obtained by picking up the target audio, and the first audio information is compensated and adjusted in terms of acoustic parameters based on the reference audio information, so as to obtain second audio information.

[0068] In the embodiment, the first audio information refers to audio information obtained by the first HMD device picking up the target audio. The acoustic parameters can include frequency (one information feature of beam phase), amplitude (beam amplitude) and period (one information feature of beam phase).

[0069] For example, the acoustic parameters include beam amplitude and / or beam phase.

[0070] For example, the beam amplitude of the first audio information can be compensated to increase the volume of the first audio information and reduce the audio signal attenuation degree of the first audio information, so as to better enhance the first audio information and avoid distortion of the target audio to affect the user's perception of the information of the target audio.

[0071] Of course, the beam phase displacement of the first audio information can also be compensated to more accurately present the spatial position of the sound source corresponding to the first audio information, so as to better help the user clearly and accurately distinguish the sound emitting position corresponding to the first audio information. Those skilled in the art can know that existing technologies such as beam forming, adaptive filtering and adaptive volume can be used to compensate and adjust the acoustic parameters of the first audio information, so as to better restore the target audio and avoid the audio signal attenuation of the target audio to affect the user's perception of the information of the target audio.

[0072] In step S40, the second audio information is rendered to an extended reality environment.

[0073] In the embodiment, because the same sound source is different relative to different user positions, different HMD devices can identify the target audio information with different accuracies. For example, the farther the HMD device is from the sound source, or the greater the structure density / volume between the HMD device and the sound source, the more incomplete / inaccurate the HMD device picks up the information of the target audio.

[0074] Based on this, the embodiment of the present application provides an audio processing method, which is applied to a first head-mounted device, and the first head-mounted device constructs a communication link with a plurality of second head-mounted devices. The technical solution of the embodiment of the present application is to determine the first audio signal attenuation degree of each second head-mounted device picking up the target audio, thereby analyzing the audio information completeness / accuracy of each second head-mounted device capable of picking up the target audio, and selecting a reference head-mounted device from each second head-mounted device, wherein the first audio signal attenuation degree of the reference head-mounted device is less than a preset attenuation degree threshold. Thus, the head-mounted device with higher audio information completeness / accuracy is determined among other head-mounted devices in collaborative interaction, so as to ensure the real accuracy of picking up the target audio information. Then, the reference audio information obtained by the reference head-mounted device picking up the target audio is acquired, and then (the first head-mounted device) picks up the target audio to obtain first audio information. According to the reference audio information, the first audio information is compensated and adjusted in terms of acoustic parameters to obtain second audio information. The second audio information is rendered to an extended reality environment, so as to accurately render the audio signal in the real environment to the extended reality environment.

[0075] The embodiment of the present application makes it possible to pick up the target audio by the head-mounted device worn by each user in some multi-user collaborative interaction application scenarios (for example, in a performance, a concert, and a multi-user conference scenario) in which there is target audio that needs to be paid attention to by the user. The target audio is shared to other head-mounted devices in collaborative interaction. After receiving the shared target audio, other head-mounted devices can play back and display the target audio information (which can include audio semantic information and sound source position), and compensate and adjust the first audio information picked up by the head-mounted device in terms of acoustic parameters according to the target audio shared by other head-mounted devices. In this way, the spatial position deviation of the target audio or the intensity loss of the target audio caused by the head-mounted device worn by the user due to the long distance from the sound source or the existence of obstacles between the sound source and the head-mounted device is calibrated, so that the user can more clearly and accurately obtain the target audio information and distinguish the sound emitting position corresponding to the target audio. The sound heard by the user in the extended reality environment is more accurate and real. Thus, the technical problem of how to improve the accuracy of sound effect rendering in the extended reality application is solved.

[0076] It is worth mentioning that, in a multi-user scenario, there can be multiple sound sources at the same time. The head-mounted device worn by the user can identify the difference in the position of the sound source due to the different positions of the same sound source relative to different users. For example, user 1 wears a device and identifies the position of the sound source through sound source positioning technology, and classifies the sound source as sound source 1 information. User 1 can share the sound source 1 information to other users, including the sound source position, sound source data, and sound source type information. User 2 receiving the sound source 1 information can play back the sound source 1 and display the position of the sound source 1. The application scenarios include performances, concerts, and multi-user conference scenarios, as shown in FIG. Figure 3 ​Figure 3 A scene diagram for multi-user multi-sound source audio recognition in an embodiment of the present application. In a multi-user multi-sound source scene, after a certain head-mounted device locates a sound source, the head-mounted device (sound source sender) can identify and classify the sound source, and share the type and location information of the sound source with other head-mounted devices. After receiving the sound source information, the other devices can play back and display the sound source location, and the other head-mounted devices can calibrate the sound source location and feed back to the sound source sender, thereby increasing the interaction between users, enhancing the interaction experience between head-mounted device users, and improving the accuracy of sound source positioning.

[0077] Exemplarily, before the step of selecting the reference head-mounted device with the first audio signal attenuation degree less than the preset attenuation degree threshold from the second head-mounted devices, the method further comprises:

[0078] Step A10, before the step of selecting the reference head-mounted device with the first audio signal attenuation degree less than the preset attenuation degree threshold from the second head-mounted devices, the method further comprises:

[0079] Step A20, if there is no first audio signal attenuation degree less than the preset attenuation degree threshold in the first audio signal attenuation degrees, the reference head-mounted device with the lowest first audio signal attenuation degree is selected from the second head-mounted devices, and the step of obtaining the reference audio information obtained by the reference head-mounted device picking up the target audio is performed.

[0080] Step A30, if there is a first audio signal attenuation degree less than the preset attenuation degree threshold in the first audio signal attenuation degrees, the step of selecting the reference head-mounted device with the first audio signal attenuation degree less than the preset attenuation degree threshold from the second head-mounted devices is performed.

[0081] In this embodiment, if there is no first audio signal attenuation degree less than the preset attenuation degree threshold in the first audio signal attenuation degrees, it indicates that each second head-mounted device is possibly far away from the sound source position where the target audio is generated, or there is a larger obstacle between the sound source and the second head-mounted device, resulting in that the audio signal attenuation degree of each second head-mounted device is high, so that the accuracy of identifying the target audio information cannot be very accurate.

[0082] Due to different positions of the same sound source relative to different users in multiple user scenarios, different devices can identify different positions of the sound source. That is, due to different positions of the same sound source relative to different users, the devices can identify different accuracies of the target audio information. For example, the farther the head-mounted device is from the sound source, or the greater the density / volume of the structure between the head-mounted device and the sound source, the more incomplete / inaccurate the head-mounted device is in picking up the target audio information. Therefore, the present embodiment first determines whether there is a first audio signal attenuation degree less than the preset attenuation degree threshold in each first audio signal attenuation degree. If there is, the reference head-mounted device with the first audio signal attenuation degree less than the preset attenuation degree threshold is selected from each second head-mounted device, so as to filter the reference head-mounted device with a lower audio signal attenuation degree from each second head-mounted device without affecting the identification of the target audio information, thereby avoiding distortion of the target audio. If there is not, the head-mounted device with the lowest first audio signal attenuation degree is selected as the reference head-mounted device from each second head-mounted device, so as to restore the target audio as much as possible and avoid the audio signal attenuation of the target audio affecting the user's perception of the target audio to the greatest extent.

[0083] In an implementable manner, after the step of determining the first audio signal attenuation degree of each second head-mounted device picking up the target audio, the method further comprises:

[0084] Step B10, determining the second audio signal attenuation degree of the first head-mounted device picking up the target audio;

[0085] In the present embodiment, the second audio signal attenuation degree refers to the audio signal attenuation degree of the first head-mounted device picking up the target audio.

[0086] Step B20, determining whether there is a first audio signal attenuation degree less than the second audio signal attenuation degree in each first audio signal attenuation degree.

[0087] Step B30, if yes, performing the step of selecting the reference head-mounted device with the first audio signal attenuation degree less than the preset attenuation degree threshold from each second head-mounted device.

[0088] In the present embodiment, if there is no first audio signal attenuation degree less than the second audio signal attenuation degree in each first audio signal attenuation degree, it indicates that the first head-mounted device is closer to the position of the sound source producing the target audio than each second head-mounted device, or there is no obstacle between the first head-mounted device and the sound source, so that the first head-mounted device has a higher accuracy in identifying the target audio information. At this time, no processing is needed, that is, no acoustic parameter compensation adjustment is needed for the first audio information, which is sufficient for the user to clearly obtain the information of the target audio.

[0089] However, if there is a first audio signal attenuation degree in each first audio signal attenuation degree that is less than the second audio signal attenuation degree, it indicates that at least one of the second head-mounted devices (i.e., a reference head-mounted device) is closer to the sound source position generating the target audio than the first head-mounted device, or there is no obstacle between the sound source and the reference head-mounted device that is as large as the structure density / volume of the first head-mounted device, so that the reference head-mounted device has higher accuracy in identifying the target audio information. At this time, the reference head-mounted device with a first audio signal attenuation degree less than a preset attenuation threshold can be selected from each second head-mounted device, and subsequent reference audio information obtained by picking up the target audio of the reference head-mounted device is executed. According to the reference audio information, the first audio information picked up by the first head-mounted device is compensated and adjusted in terms of acoustic parameters, so that the user can more clearly and accurately obtain the target audio information and distinguish the sound source position corresponding to the target audio, and further ensure that the sound heard by the user in the extended reality environment is more accurate and real.

[0090] In a possible implementation, the acoustic parameter includes a beam amplitude, and the step of compensating and adjusting the first audio information in terms of the acoustic parameter according to the reference audio information includes:

[0091] Step C10, identifying the reference audio information to obtain a reference audio intensity of the target audio identified by the reference head-mounted device, and identifying the first audio information to obtain a to-be-calibrated audio intensity of the target audio identified by the first head-mounted device;

[0092] Step C20, determining an audio intensity loss value of the target audio identified by the first head-mounted device according to the reference audio intensity and the to-be-calibrated audio intensity, and determining a beam amplitude loss value of the first audio information according to the audio intensity loss value;

[0093] In this embodiment, the reference audio intensity refers to the audio intensity of the target audio identified by the reference head-mounted device. The to-be-calibrated audio intensity refers to the audio intensity of the target audio identified by the first head-mounted device.

[0094] Step C30, compensating and adjusting the first audio information in terms of the beam amplitude according to the beam amplitude loss value.

[0095] In the embodiment, under the condition of other factors being the same (single variable), the farther the sound distance, the greater the audio intensity loss value of the first audio information, resulting in the greater the beam amplitude loss value of the identified target audio, and further resulting in the lower the accuracy of identifying the target audio information. Or under the condition of other factors being the same (single variable, for example, equal distance, same barrier material, same barrier material, and equal barrier area of the barrier in the audio propagation direction), the greater the barrier thickness, the greater the audio intensity loss value of the first audio information, resulting in the greater the beam amplitude loss value of the identified target audio, and further resulting in the lower the accuracy of identifying the target audio information. Or under the condition of other factors being the same (single variable, for example, equal distance, same barrier material, same barrier material, and equal barrier thickness), the greater the barrier area in the audio propagation direction, the greater the audio intensity loss value of the first audio information, resulting in the greater the beam amplitude loss value of the identified target audio, and further resulting in the lower the accuracy of identifying the target audio information.

[0096] Based on this, the embodiment identifies the reference audio information to obtain the reference audio intensity of the target audio identified by the reference HMD device, identifies the first audio information to obtain the to-be-calibrated audio intensity of the target audio identified by the first HMD device, determines the audio intensity loss value of the target audio identified by the first HMD device according to the reference audio intensity and the to-be-calibrated audio intensity, determines the beam amplitude loss value of the first audio information according to the audio intensity loss value, and then compensates and adjusts the beam amplitude of the first audio information according to the beam amplitude loss value, so that the user can more clearly and accurately obtain the target audio information, the sound heard by the user in the extended reality environment is more accurate and real, and the accuracy of sound effect rendering of the extended reality application is further improved.

[0097] In an implementable manner, the acoustic parameter includes a beam phase, and the step of compensating and adjusting the first audio information according to the reference audio information includes:

[0098] Step D10, identifying the reference audio information to obtain the relative spatial position of the sound object of the target audio relative to the reference HMD device;

[0099] Step D20, obtaining the current first device position of the reference HMD device through the communication link, and determining the reference sound position of the target audio identified by the reference HMD device according to the first device position and the relative spatial position;

[0100] In the embodiment, the first device position refers to the sound pickup position of the reference HMD device for picking up the target audio at present.

[0101] Step D30, determining a first sound source position of the target audio recognized by the first HMD device through sound source localization of the target audio by the first audio information;

[0102] Step D40, determining a sound source position deviation value of the target audio recognized by the first HMD device according to the reference sound source position and the first sound source position, and determining a beam phase offset value of the first audio information according to the sound source position deviation value;

[0103] In this embodiment, the reference sound source position refers to the sound source position of the target audio recognized by the reference HMD device. The first sound source position refers to the sound source position of the target audio recognized by the first HMD device.

[0104] Step D50, compensating and adjusting the beam phase of the first audio information according to the beam phase offset value.

[0105] Step D60, compensating and adjusting the beam phase of the first audio information according to the beam phase offset value.

[0106] It is easy to understand that, in general, in the extended reality environment, the sound source position of the target audio refers to the actual position of the sound object, and due to the propagation of sound, the sound signal will be attenuated with the increase of distance or the blocking of blocking objects (such as building bodies, glass bodies, sponge bodies, etc.), so that the positions of the sound objects recognized by the HMD devices at different positions through sound positioning are not the same. And those skilled in the art know that, under the same conditions of other factors (single variable), the farther the sound distance, the greater the beam phase offset value of the first audio information, resulting in a greater sound source position deviation value of the recognized sound object position. Or under the same conditions of other factors (single variable, such as equal distance, same blocking object material, same blocking object material, and equal blocking area of the blocking object in the direction of audio propagation), the greater the blocking thickness of the blocking object, the greater the beam phase offset value of the first audio information, resulting in a greater sound source position deviation value of the recognized sound object position. Or under the same conditions of other factors (single variable, such as equal distance, same blocking object material, same blocking object material, and equal blocking thickness of the blocking object), the greater the blocking area of the blocking object in the direction of audio propagation, the greater the beam phase offset value of the first audio information, resulting in a greater sound source position deviation value of the recognized sound object position.

[0107] Based on this, the embodiment obtains the relative spatial position of the sound emitting object of the target audio relative to the reference HMD device by identifying the reference audio information, and obtains the current first device position of the reference HMD device through the communication link, and determines the reference sound emitting position of the target audio identified by the reference HMD device according to the first device position and the relative spatial position. The sound source of the target audio is positioned through the first audio information to determine the first sound emitting position of the target audio identified by the first HMD device. Then, the sound emitting position deviation value of the target audio identified by the first HMD device is determined according to the reference sound emitting position and the first sound emitting position. Then, the beam phase offset value of the first audio information is determined according to the sound emitting position deviation value, and the beam phase of the first audio information is compensated and adjusted based on the beam phase offset value. Thus, the user can more accurately distinguish the sound emitting position corresponding to the target audio, and the sound heard by the user in the extended reality environment is more accurate and real, the accuracy of sound effect rendering of the extended reality application is improved, the immersion of the user in the virtual reality environment is strengthened, and a more real and more immersive spatial audio experience is provided for the user.

[0108] To help understand the technical concept or technical principle of the embodiments of the present application, a specific embodiment is cited, please refer to Figure 3 , Figure 3 The scene diagram for audio recognition of multiple users and multiple sound sources in the embodiments of the present application is as follows:

[0109] 1. In this scene, multiple users are in the same space, and the user's position relative to the space is fixed, that is, the user's position coordinates in the space are determined, and the position coordinates of each user are shared.

[0110] 2. After user 1 performs sound source positioning, the identifiable positioned sound sources are sorted and classified, for example, the sound sources can be sorted according to the distance of the sound source relative to the user's position, from near to far, as follows: sound source 1 position 1 type 1, sound source 2 position 2 type 2, sound source 3 position 3 type 3…

[0111] 3. User 1 (sound source sender) shares the sound source information in step 2 to other users.

[0112] 4. After user N (N is a positive integer greater than 1) receives the sound source data of user 1, the user can select the sound source position to be displayed in the received sound source list and play back the sound source data. For example, user N selects to play back sound source 1 and display the position information of sound source 1. At the same time, device N worn by user N retrieves the related information of sound source 1 in the device and compares it with the sound source 1 data received from user 1 to calibrate the position of sound source 1 and feed back the result to user 1.

[0113] 5. The above steps can be implemented between each user from user 1 to user N.

[0114] Through the method steps of the embodiment, in a multi-user and multi-sound source scene, after a device (i.e., a head-mounted device) locates a sound source, the device can identify and classify the sound source, and share the type and position information of the sound source with other devices. After receiving the sound source information, the device can play back and display the sound source position, and feed back the sound source position to the sound source sender for calibration, so as to increase the interaction mode between users, enhance the interaction experience between users of multiple devices, and improve the accuracy of sound source positioning.

[0115] It should be noted that the details described in the specific embodiments are only used to understand the technical concept of the present application and do not constitute a limitation of the present application. Based on the technical concept of the present application, more forms of simple transformation should be within the protection scope of the present application.

[0116] Embodiment Two

[0117] Based on the first embodiment of the present application, in another embodiment of the present application, the same or similar contents as the above-mentioned embodiment one can refer to the above introduction, and will not be described in detail. On this basis, please refer to Figure 3 The step of determining the audio signal attenuation degree corresponding to the target audio picked up by each second head-mounted device includes:

[0118] In step S50, the current second device position of each second head-mounted device is obtained through the communication link.

[0119] In the embodiment, the second device position refers to the sound pickup position of the second head-mounted device currently picking up the target audio.

[0120] In step S60, the sound source of the target audio is located by each second head-mounted device, and the second sound emitting position of the target audio recognized by each second head-mounted device is determined.

[0121] The second sound emitting position refers to the sound emitting position of the target audio recognized by the second head-mounted device.

[0122] In step S70, the audio propagation path of the sound emitting object of the target audio relative to each second head-mounted device is determined according to each second device position and each second sound emitting position.

[0123] In the embodiment, it is known to those skilled in the art that after the second device position and the second sound emitting position are determined, the audio propagation path of the sound propagation between the second device position and the second sound emitting position can be constructed by the second device position and the second sound emitting position. It should be noted that when constructing the audio propagation path, the influence of the sound propagation path caused by the shielding object (or sound propagation barrier) is not considered.

[0124] Step S80, according to the audio propagation path of each second head-mounted device, determine the audio signal attenuation degree corresponding to the target audio picked up by each second head-mounted device.

[0125] As can be appreciated by those skilled in the art, since the position of each second head-mounted device is different, there is an audio propagation path corresponding to the target audio picked up by each second head-mounted device, and thus there is an audio signal attenuation degree corresponding to each second head-mounted device.

[0126] For example, the step of determining the audio signal attenuation degree corresponding to the target audio picked up by each second head-mounted device according to the audio propagation path of each second head-mounted device includes:

[0127] Step E10, through the communication link, obtain the occlusion range formed by the occlusion object in the direction perpendicular to the audio propagation path in each audio propagation path;

[0128] In this embodiment, the occlusion object refers to a barrier or occlusion object that attenuates sound signals (i.e. audio signals) (for example, a building body, a glass body, a sponge body, etc.).

[0129] Step E20, according to each of the occlusion range, determine the audio occlusion area covered by the occlusion object in each of the audio propagation paths;

[0130] As can be appreciated by those skilled in the art, since the position of each second head-mounted device is different, the audio propagation path corresponding to the target audio picked up by each second head-mounted device is also different, so that the occlusion range formed by the occlusion object in the direction perpendicular to the audio propagation path corresponding to each second head-mounted device is also different, and thus the audio occlusion area covered by the occlusion object in each audio propagation path (i.e. the audio propagation path corresponding to each second head-mounted device) is different.

[0131] Step E30, obtain the audio diffusion area covered by the target audio in each of the audio propagation paths, and according to each of the audio occlusion area and each of the audio diffusion area, determine the audio signal attenuation degree corresponding to the target audio picked up by each second head-mounted device, wherein each second head-mounted device corresponds to an audio occlusion area and an audio diffusion area.

[0132] In one implementable manner, the step of determining the audio signal attenuation degree corresponding to the target audio picked up by each second head-mounted device according to each of the audio occlusion area and each of the audio diffusion area includes:

[0133] Step F10, calculate the ratio of the audio blocking area and the audio diffusion area to obtain an audio blocked area ratio; according to the audio blocked area ratio, query the audio signal attenuation degree associated with the audio blocked area ratio from a preset association data table, and take the associated audio signal attenuation degree as the audio signal attenuation degree corresponding to the target audio picked up by the second head-mounted device.

[0134] The embodiment calculates the ratio of the audio blocking area and the audio diffusion area to obtain the audio blocked area ratio, and then queries the audio signal attenuation degree associated with the audio blocked area ratio from a preset association data table according to the audio blocked area ratio, so as to accurately obtain the audio signal attenuation degree corresponding to the target audio picked up by the second head-mounted device, and facilitate subsequent compensation and adjustment of the acoustic parameters of the first audio information according to the audio signal attenuation degree, so as to more accurately and truly restore the audio information of the target audio, and further provide a more realistic spatial audio experience for the user.

[0135] In an implementable manner, the step of determining the audio signal attenuation degree corresponding to the target audio picked up by each second head-mounted device according to each audio blocking area and each audio diffusion area comprises:

[0136] Step G10, difference calculation is performed on the audio diffusion area and the audio blocking area corresponding to each second head-mounted device to obtain an audio propagation area of the target audio propagated by the sound emitting object to each second head-mounted device;

[0137] Step G20, according to each audio propagation area, determine a sound propagation angle of the target audio propagated by the sound emitting object to each second head-mounted device, and according to each sound propagation angle, determine a sound pickup offset angle of each second head-mounted device picking up the target audio;

[0138] Step G30, query the audio signal attenuation degree mapped by each sound pickup offset angle from a preset mapping data table;

[0139] Step G40, according to the audio signal attenuation degree mapped by each sound pickup offset angle, determine the audio signal attenuation degree corresponding to the target audio picked up by each second head-mounted device.

[0140] Compared with the method of directly determining the audio attenuation coefficient by calculating the proportion of the audio blocked area in the step F10, the embodiment also considers the influence of the pickup offset angle on the audio attenuation. It is easy to understand that the pickup orientation when the pickup position is directly opposite the sound source position is the target pickup orientation, and the angle by which the pickup orientation of the pickup position deviates from the target pickup orientation is the pickup offset angle. It is easy to understand that the smaller the pickup offset angle is, the louder and clearer the sound received by the pickup position is. The larger the pickup offset angle is, the quieter and blurrier the sound received by the pickup position is. Therefore, the embodiment obtains the target audio propagated from the sound object to the audio diffusion area corresponding to each second head-mounted device by difference calculation of the audio diffusion area and the audio blocked area corresponding to each second head-mounted device, determines the sound propagation angle of the target audio propagated from the sound object to each second head-mounted device according to each audio propagation area, determines the pickup offset angle of each second head-mounted device for picking up the target audio according to each sound propagation angle, queries the audio signal attenuation degree mapped by each pickup offset angle from the preset mapping data table, and determines the audio signal attenuation degree corresponding to each second head-mounted device for picking up the target audio according to the audio signal attenuation degree mapped by each pickup offset angle, so as to facilitate more accurate acoustic parameter adjustment of the first audio information in the subsequent process, and further improve the immersion and reality of the hearing in the extended reality application.

[0141] Embodiment three

[0142] The embodiment of the application also provides an audio processing device, please refer to Figure 4 , the audio processing device is applied to a head-mounted display device, the first head-mounted device and a plurality of second head-mounted devices construct a communication link, and the audio processing device comprises:

[0143] A determination module 10 is configured to determine a first audio signal attenuation degree corresponding to each second head-mounted device for picking up a target audio.

[0144] A screening module 20 is configured to select a reference head-mounted device with a first audio signal attenuation degree less than a preset attenuation degree threshold from the second head-mounted devices, and obtain reference audio information obtained by the reference head-mounted device for picking up the target audio.

[0145] A compensation module 30 is configured to pick up the first audio information of the target audio, perform acoustic parameter compensation adjustment on the first audio information according to the reference audio information, and obtain second audio information.

[0146] A rendering module 40 is configured to render the second audio information to an extended reality environment.

[0147] Optionally, the determination module 10 is further configured to:

[0148] determining a second audio signal attenuation degree corresponding to the target audio picked up by the first HMD device;

[0149] determining whether there is a first audio signal attenuation degree less than the second audio signal attenuation degree in each of the first audio signal attenuation degrees;

[0150] if yes, performing the step of selecting a reference HMD device with a first audio signal attenuation degree less than a preset attenuation threshold from each of the second HMD devices.

[0151] Optionally, the acoustic parameter includes a beam amplitude, and the compensation module 30 is further configured to:

[0152] identify the reference audio information to obtain a reference audio intensity of the target audio identified by the reference HMD device, and identify the first audio information to obtain a to-be-calibrated audio intensity of the target audio identified by the first HMD device;

[0153] determine an audio intensity loss value of the target audio identified by the first HMD device according to the reference audio intensity and the to-be-calibrated audio intensity, and determine a beam amplitude loss value of the first audio information according to the audio intensity loss value;

[0154] perform compensation adjustment of the beam amplitude on the first audio information according to the beam amplitude loss value.

[0155] Optionally, the acoustic parameter includes a beam phase, and the compensation module 30 is further configured to:

[0156] identify the reference audio information to obtain a relative spatial position of a sound emitting object of the target audio relative to the reference HMD device;

[0157] obtain a first device position of the reference HMD device currently through the communication link, and determine a reference sound emitting position of the target audio identified by the reference HMD device according to the first device position and the relative spatial position;

[0158] perform sound source positioning on the target audio through the first audio information to determine a first sound emitting position of the target audio identified by the first HMD device;

[0159] determine a sound emitting position deviation value of the target audio identified by the first HMD device according to the reference sound emitting position and the first sound emitting position, and determine a beam phase offset value of the first audio information according to the sound emitting position deviation value;

[0160] perform compensation adjustment of the beam phase on the first audio information according to the beam phase offset value.

[0161] Optionally, the screening module 20 is further configured to:

[0162] If there is no first audio signal attenuation degree less than the preset attenuation degree threshold in the first audio signal attenuation degrees, a reference HMD device with the lowest first audio signal attenuation degree is selected from the second HMD devices, and the step of obtaining the reference audio information obtained by the reference HMD device picking up the target audio is performed.

[0163] If there is a first audio signal attenuation degree less than the preset attenuation degree threshold in the first audio signal attenuation degrees, the step of selecting a reference HMD device with a first audio signal attenuation degree less than the preset attenuation degree threshold from the second HMD devices is performed.

[0164] Optionally, the screening module 20 is further configured to:

[0165] The current second device positions of the second HMD devices are obtained through the communication link.

[0166] The second sound source positions of the target audio recognized by the second HMD devices are determined by sound source positioning of the target audio through the second HMD devices.

[0167] The audio propagation paths of the target audio relative to the second HMD devices are determined according to the second device positions and the second sound source positions.

[0168] The audio signal attenuation degrees of the target audio picked up by the second HMD devices are determined according to the audio propagation paths of the second HMD devices.

[0169] Optionally, the screening module 20 is further configured to:

[0170] The occlusion ranges formed by the occlusion objects in the audio propagation paths in a direction perpendicular to the audio propagation paths are obtained through the communication link.

[0171] The audio occlusion areas covered by the occlusion objects in the audio propagation paths are determined according to the occlusion ranges.

[0172] The audio diffusion areas covered by the target audio in the audio propagation paths are obtained, and the audio signal attenuation degrees of the target audio picked up by the second HMD devices are determined according to the audio occlusion areas and the audio diffusion areas, wherein each second HMD device corresponds to an audio occlusion area and an audio diffusion area.

[0173] Optionally, the screening module 20 is further configured to:

[0174] differences between the audio diffusion area and the audio occlusion area corresponding to each second head-mounted device, to obtain an audio propagation area of the target audio propagated by the sound emitting object to each second head-mounted device;

[0175] According to the audio propagation area, a sound propagation angle of the target audio propagated by the sound emitting object to each second head-mounted device is determined, and a sound pickup offset angle of each second head-mounted device picking up the target audio is determined according to the sound propagation angle.

[0176] From a preset mapping data table, an audio signal attenuation degree mapped by each sound pickup offset angle is queried.

[0177] According to the audio signal attenuation degree mapped by each sound pickup offset angle, an audio signal attenuation degree corresponding to the target audio picked up by each second head-mounted device is determined.

[0178] The audio processing device provided by the embodiment can solve the technical problem of how to improve the accuracy of sound effect rendering in extended reality application by using the audio processing method in the above-mentioned embodiment one or embodiment two. Compared with the prior art, the audio processing device provided by the embodiment has the same beneficial effects as the audio processing method provided by the above-mentioned embodiment, and other technical features in the audio processing device are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0179] Embodiment four

[0180] The embodiment provides a head-mounted display device, which comprises at least one processor and a memory in communication connection with the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the audio processing method in the above-mentioned embodiment one.

[0181] Reference will be made to the following description Figure 5 which shows a structural schematic diagram of a head-mounted display device suitable for implementing the embodiments of the present disclosure. The head-mounted display device in the embodiments of the present disclosure includes but is not limited to a mixed reality (Mixed Reality) - MR device (such as MR glasses or MR helmet), an augmented reality (Augmented Reality) - AR device (such as AR glasses or AR helmet), a virtual reality (Virtual Reality) - VR device (such as VR glasses or VR helmet), an extended reality (Extended Reality) - XR device or some combination thereof, and the like. Figure 5 The head-mounted display device shown is only an example and should not impose any limitation on the functions and use range of the embodiments of the present disclosure.

[0182] As shown in Figure 5 The head-mounted display device can include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM 1002) or programs loaded from a storage device into a random access memory (RAM 1004). Various programs and data required for the operation of the head-mounted display device are also stored in the RAM 1004. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface is also connected to the bus 1005.

[0183] Generally, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication devices 1009 can allow the head-mounted display device to communicate wirelessly or wired with other devices to exchange data. Although the head-mounted display device with various systems is shown in the figure, it should be understood that all the shown systems are not required to be implemented or possessed. More or less systems can be alternatively implemented or possessed.

[0184] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by the communication devices, or installed from the storage devices 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are performed.

[0185] The head-mounted display device provided by the present application adopts the audio processing method in the above-mentioned embodiments, and can solve the technical problem of how to improve the accuracy of sound effect rendering in extended reality applications. Compared with the prior art, the head-mounted display device provided by the embodiments of the present application has the same beneficial effects as the audio processing method provided by the above-mentioned embodiments, and other technical features in the head-mounted display device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0186] It should be understood that various parts of this disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0187] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0188] Example 5

[0189] This invention provides a computer-readable storage medium having computer-readable program instructions stored thereon, which are used to execute the audio processing method described in the above embodiments.

[0190] The computer-readable storage medium provided in this embodiment of the invention may be, for example, a USB flash drive, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0191] The aforementioned computer-readable storage medium may be included in the head-mounted display device; or it may exist independently and not assembled into the head-mounted display device.

[0192] The computer readable storage medium described above carries one or more programs, when the one or more programs are executed by the head-mounted display device, the head-mounted display device is caused to: determine a first audio signal attenuation degree corresponding to a target audio picked up by each of the second head-mounted display devices; select a reference head-mounted display device from each of the second head-mounted display devices, the reference head-mounted display device has a first audio signal attenuation degree less than a preset attenuation degree threshold, and obtain reference audio information obtained by picking up the target audio by the reference head-mounted display device; pick up the target audio to obtain first audio information, perform compensation adjustment on the first audio information according to the reference audio information, and obtain second audio information; and render the second audio information to an extended reality environment.

[0193] Computer program code for carrying out operations of the present disclosure can be written in any of one or more programming languages or combinations of languages including object or visual programming languages such as Java, Smalltalk, C++ or conventional procedural programming languages such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0194] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations of blocks in the block diagrams and / or flow diagrams, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0195] The modules described in the embodiments of the present disclosure can be implemented in the form of software or in the form of hardware. In some cases, the names of the modules do not constitute a limitation on the modules themselves.

[0196] The computer readable storage medium provided by the present application stores computer readable program instructions for executing the audio processing method described above, and can solve the technical problem of how to improve the accuracy of sound effect rendering of extended reality application. Compared with the prior art, the beneficial effects of the computer readable storage medium provided by the embodiment of the present application are the same as those of the audio processing method provided by the first embodiment or the second embodiment, which will not be repeated here.

[0197] Embodiment six

[0198] The embodiment of the present application also provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to realize the steps of the audio processing method described above.

[0199] The computer program product provided by the present application can solve the technical problem of how to improve the accuracy of sound effect rendering of extended reality application. Compared with the prior art, the beneficial effects of the computer program product provided by the embodiment of the present application are the same as those of the audio processing method provided by the first embodiment or the second embodiment, which will not be repeated here.

[0200] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent process transformation using the content of the specification and drawings, or direct or indirect application in other related technical fields, are also included in the patent processing scope of the present application.

Claims

1. An audio processing method, characterized in that, The audio processing method is applied to a first head-mounted display device, which establishes a communication link with multiple second head-mounted display devices. The method includes: Determine the first audio signal attenuation degree corresponding to the target audio picked up by each of the second head-mounted displays; Select a reference head-mounted display device from each of the second head-mounted display devices whose first audio signal attenuation is less than a preset attenuation threshold, and obtain reference audio information obtained by the reference head-mounted display device picking up the target audio. The target audio is picked up to obtain first audio information. Based on the reference audio information, the acoustic parameters of the first audio information are compensated and adjusted to obtain second audio information. The second audio information is rendered into the extended reality environment.

2. The audio processing method as described in claim 1, characterized in that, After the step of determining the first audio signal attenuation corresponding to the target audio picked up by each of the second head-mounted displays, the method further includes: Determine the second audio signal attenuation degree corresponding to the target audio picked up by the first head-mounted display device; Determine whether any of the first audio signal attenuation levels is less than the second audio signal attenuation level; If so, then the following step is performed: selecting a reference head-mounted display device from each of the second head-mounted display devices whose first audio signal attenuation is less than a preset attenuation threshold.

3. The audio processing method as described in claim 2, characterized in that, The acoustic parameters include beam amplitude, and the step of compensating and adjusting the acoustic parameters of the first audio information based on the reference audio information includes: The reference audio information is identified to obtain the reference audio intensity of the target audio identified by the reference head-mounted display device, and the first audio information is identified to obtain the audio intensity to be calibrated of the target audio identified by the first head-mounted display device. Based on the reference audio intensity and the audio intensity to be calibrated, the audio intensity loss value of the target audio identified by the first head-mounted display device is determined, and the beam amplitude loss value of the first audio information is determined based on the audio intensity loss value. Based on the beam amplitude loss value, the first audio information is adjusted to compensate for the beam amplitude loss.

4. The audio processing method as described in claim 2, characterized in that, The acoustic parameters include beam phase, and the step of compensating and adjusting the acoustic parameters of the first audio information based on the reference audio information includes: Identify the reference audio information to obtain the relative spatial position of the sound-emitting object of the target audio relative to the reference head-mounted display device; The first device position of the reference head-mounted display device is obtained through the communication link, and the reference sound source position of the target audio recognized by the reference head-mounted display device is determined based on the first device position and the relative spatial position. The first audio information is used to locate the sound source of the target audio, and the first sound source position of the target audio identified by the first head-mounted display device is determined. Based on the reference sound emission position and the first sound emission position, the sound emission position deviation value of the target audio recognized by the first head-mounted display device is determined, and based on the sound emission position deviation value, the beam phase offset value of the first audio information is determined. Based on the beam phase offset value, the first audio information is adjusted to compensate for the beam phase.

5. The audio processing method according to any one of claims 1 to 4, characterized in that, Before the step of selecting a reference head-mounted display device from each of the second head-mounted display devices whose first audio signal attenuation is less than a preset attenuation threshold, the method further includes: If none of the first audio signal attenuation degrees is less than a preset attenuation threshold, then the reference head-mounted display device with the lowest first audio signal attenuation degree is selected from each of the second head-mounted display devices, and the step of obtaining the reference audio information obtained by the reference head-mounted display device picking up the target audio is executed. If any of the first audio signal attenuation values ​​is less than a preset attenuation threshold, then the following step is performed: select a reference head-mounted display device from each of the second head-mounted display devices whose first audio signal attenuation value is less than the preset attenuation threshold.

6. The audio processing method according to any one of claims 1 to 4, characterized in that, The step of determining the audio signal attenuation corresponding to the target audio picked up by each of the second head-mounted display devices includes: The current location of each second head-mounted display device is obtained through the communication link; The target audio is located by each of the second head-mounted display devices, and the second sound source position of the target audio identified by each of the second head-mounted display devices is determined. Based on the location of each of the second devices and the location of each of the second sound sources, the audio propagation path of the target audio source relative to each of the second head-mounted display devices is determined; Based on the audio propagation path of each second head-mounted display device, determine the audio signal attenuation corresponding to the target audio picked up by each second head-mounted display device.

7. The audio processing method as described in claim 6, characterized in that, The step of determining the audio signal attenuation corresponding to the target audio picked up by each of the second head-mounted displays based on the audio propagation path of each second head-mounted display includes: Through the communication link, the occlusion range formed by the obstructing object in the direction perpendicular to the audio propagation path is obtained in each of the audio propagation paths; Based on the obstruction range, determine the audio obstruction area covered by the obstructing object in each audio propagation path; The audio diffusion area covered by the target audio in each of the audio propagation paths is obtained. Based on each of the audio occlusion areas and each of the audio diffusion areas, the audio signal attenuation corresponding to the target audio picked up by each of the second head-mounted display devices is determined. Each of the second head-mounted display devices corresponds to an audio occlusion area and an audio diffusion area.

8. The audio processing method as described in claim 7, characterized in that, The step of determining the audio signal attenuation corresponding to the target audio picked up by each of the audio occlusion regions and the audio diffusion regions includes: The difference between the audio diffusion area and the audio blocking area corresponding to each second head-mounted display device is calculated to obtain the target audio propagated from the sound-emitting object to the audio propagation area corresponding to each second head-mounted display device. Based on each of the audio propagation areas, the sound propagation angle of the target audio from the sound-emitting object to each of the second head-mounted display devices is determined, and the sound pickup offset angle of each of the second head-mounted display devices for picking up the target audio is determined based on each of the sound propagation angles. The audio signal attenuation degree of each pickup offset angle mapping can be obtained from the preset mapping data table; Based on the audio signal attenuation mapped by each of the aforementioned pickup offset angles, the audio signal attenuation corresponding to the target audio picked up by each of the second head-mounted display devices is determined.

9. A head-mounted display device, characterized in that, The head-mounted display device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the steps of the audio processing method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program for implementing an audio processing method, which is executed by a processor to implement the steps of the audio processing method as claimed in any one of claims 1 to 8.

Images