An audio playing method and device for XR, an electronic device, and a storage medium

Abstract

The present disclosure relates to an audio playing method and device for XR, an electronic device and a storage medium. The method comprises: obtaining an audio capture plug-in available for a server and a target sound card device in a current platform, the server being provided with a plurality of audio capture plug-ins adapted to sound card devices of a plurality of platforms; determining a target audio capture plug-in matched with the target sound card device; obtaining audio data output by the target sound card device based on the target audio capture plug-in; processing the audio data to obtain target audio data, and sending the target audio data to a terminal device, the terminal device being configured to play the target audio data. The embodiments of the present disclosure can capture audio output by a sound card device across platforms in an XR music playing scenario.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more particularly to an audio playback method and apparatus, electronic device and storage medium for XR. Background Technology

[0002] In cloud-based Extended Reality (XR) technology, audio capture utilizes platform-specific Application Programming Interfaces (APIs), employing the underlying APIs of different audio capture systems on different platforms. Examples include DirectSound, WasAPI, and WaveX APIs for Windows systems, and ALSA and PulseAudio for Linux systems.

[0003] For cloud-based audio capture systems for XR, the cloud server relies on the platform's API to collect audio data output from the sound card device when running on different platforms. However, the APIs provided by different platforms are different, making cross-platform audio capture impossible. Summary of the Invention

[0004] This disclosure proposes an audio playback technology solution for XR.

[0005] According to one aspect of this disclosure, an audio playback method for XR is provided, applied on a server side, comprising:

[0006] Obtain the available audio capture plugins on the server and the target sound card device on the current platform. The server is equipped with a variety of audio capture plugins that are compatible with sound card devices on multiple platforms.

[0007] Identify the target audio capture plugin that matches the target sound card device;

[0008] Based on the target audio capture plugin, acquire the audio data output by the target sound card device;

[0009] The audio data is processed to obtain target audio data, and the target audio data is sent to a terminal device, which plays the target audio data.

[0010] In one possible implementation, acquiring the audio data output by the target sound card device based on the target audio capture plugin includes:

[0011] Based on the target audio capture plugin, the audio data output by the target sound card device is obtained through the interface corresponding to the target audio capture plugin.

[0012] In one possible implementation, acquiring the audio data output by the target sound card device based on the target audio capture plugin includes:

[0013] Based on the target audio capture plugin, and according to the identifier of the target process, the audio data of the target process output by the target sound card device is obtained through the interface corresponding to the target audio capture plugin.

[0014] In one possible implementation, processing the audio data to obtain target audio data and sending the target audio data to the terminal device includes:

[0015] Receive a timestamp sent by the client, the timestamp indicating the moment the client sent the timestamp;

[0016] The timestamp is added to the audio data, and the target audio data with the timestamp added is sent to the terminal device.

[0017] In one possible implementation, the server is an XR server, and the timestamp is a timestamp in the XR client used to identify the pose, so as to instruct the target sound card device to play the audio corresponding to the pose.

[0018] In one possible implementation, processing the audio data to obtain target audio data and sending the target audio data to the terminal device includes:

[0019] Based on the requirements of the terminal device for sound quality and latency in the application scenario, determine the encoding method that matches the application scenario;

[0020] The target audio data is encoded using the aforementioned encoding method to obtain target audio data, and then the target audio data is sent to the terminal device.

[0021] In one possible implementation, determining the encoding method that matches the application scenario based on the sound quality and latency requirements of the terminal device includes:

[0022] When the sound quality requirement is higher than the latency requirement in the application scenario, an encoding method with a compression ratio lower than the set compression threshold is adopted;

[0023] When the sound quality requirement is lower than the latency requirement in the application scenario, an encoding method with a compression ratio higher than the set compression threshold is adopted.

[0024] In one possible implementation, the step of acquiring audio data output by the target sound card device based on the target audio capture plugin, and the step of processing the audio data to obtain target audio data and sending the target audio data to the terminal device include:

[0025] Create and build an audio capture pipeline;

[0026] The target audio capture plugin is used as the input component of the audio capture pipeline, and the acquisition target of the input component is set to the target sound card device;

[0027] Add an audio processing component and an audio output component after the input component;

[0028] The task in the audio capture pipeline is executed to obtain audio data output by the target sound card device, process the audio data to obtain target audio data, and send the target audio data to the terminal device.

[0029] According to one aspect of this disclosure, an audio playback method for XR is provided, applied to a client, comprising:

[0030] Create an audio playback pipeline;

[0031] Receive target audio data sent by the server and extract the timestamp from the target audio data;

[0032] If the difference between the current time and the timestamp is within a set threshold range, the target audio data is sent to the audio playback pipeline via a callback function to execute the task in the audio playback pipeline and play the target audio data.

[0033] In one possible implementation, the client is an extended reality (XR) client, and the server is an XR server;

[0034] The timestamp is used in the XR client to identify the pose, instructing the target sound card device to play the audio corresponding to the pose.

[0035] According to one aspect of this disclosure, an audio playback device for XR is provided, applied on a server side, comprising:

[0036] The device acquisition unit is used to acquire the audio capture plugins available on the server and the target sound card device in the current platform. The server is equipped with a variety of audio capture plugins that are compatible with sound card devices on multiple platforms.

[0037] A plugin determination unit is used to determine a target audio capture plugin that matches the target sound card device.

[0038] An audio acquisition unit is used to acquire audio data output by the target sound card device based on the target audio capture plugin;

[0039] An audio processing unit is used to process the audio data to obtain target audio data and send the target audio data to a terminal device, wherein the terminal device is used to play the target audio data.

[0040] In one possible implementation, the audio acquisition unit is used to acquire audio data output by the target sound card device based on the target audio capture plugin and through an interface corresponding to the target audio capture plugin.

[0041] In one possible implementation, the audio acquisition unit is used to acquire the audio data of the target process output by the target sound card device based on the target audio capture plugin and according to the identifier of the target process, through an interface corresponding to the target audio capture plugin.

[0042] In one possible implementation, the audio processing unit is configured to:

[0043] Receive a timestamp sent by the client, the timestamp indicating the moment the client sent the timestamp;

[0044] The timestamp is added to the audio data, and the target audio data with the timestamp added is sent to the terminal device.

[0045] In one possible implementation, the server is an XR server, and the timestamp is a timestamp in the XR client used to identify the pose, so as to instruct the target sound card device to play the audio corresponding to the pose.

[0046] In one possible implementation, the audio processing unit is configured to:

[0047] Based on the requirements of the terminal device for sound quality and latency in the application scenario, determine the encoding method that matches the application scenario;

[0048] The target audio data is encoded using the aforementioned encoding method to obtain target audio data, and then the target audio data is sent to the terminal device.

[0049] In one possible implementation, the audio processing unit is configured to:

[0050] When the sound quality requirement is higher than the latency requirement in the application scenario, an encoding method with a compression ratio lower than the set compression threshold is adopted;

[0051] When the sound quality requirement is lower than the latency requirement in the application scenario, an encoding method with a compression ratio higher than the set compression threshold is adopted.

[0052] In one possible implementation, the audio acquisition unit is configured to: create and construct an audio capture pipeline; use the target audio capture plugin as an input component of the audio capture pipeline, and set the acquisition object of the input component to the target sound card device; and add an audio processing component and an audio output component after the input component.

[0053] The audio processing unit is used to execute tasks in the audio capture pipeline to obtain audio data output by the target sound card device, process the audio data to obtain target audio data, and send the target audio data to the terminal device.

[0054] According to one aspect of this disclosure, an audio playback device for XR is provided, applied to a client, comprising:

[0055] The pipe creation unit is used to create audio playback pipes;

[0056] An audio receiving unit is used to receive target audio data sent by the server and extract the timestamp from the target audio data;

[0057] An audio playback unit is configured to, when the difference between the current time and the timestamp is within a set threshold range, send the target audio data input callback function to the audio playback pipeline, execute the task in the audio playback pipeline, and play the target audio data.

[0058] In one possible implementation, the client is an extended reality (XR) client, and the server is an XR server;

[0059] The timestamp is used in the XR client to identify the pose, instructing the target sound card device to play the audio corresponding to the pose.

[0060] According to one aspect of this disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to perform the method described above.

[0061] According to one aspect of this disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the above-described method.

[0062] In this embodiment, by acquiring available audio capture plugins on the server and the target sound card device on the current platform, the server is equipped with various audio capture plugins adapted to sound card devices on multiple platforms. From these various audio capture plugins, a target audio capture plugin matching the target sound card device is determined. Based on the target audio capture plugin, audio data output by the target sound card device is acquired. The audio data is processed to obtain target audio data, which is then sent to a terminal device for playback. Thus, by determining a target audio capture plugin matching the target sound card device from a variety of audio capture plugins and acquiring the audio data output by the target sound card device based on the target audio capture plugin, the server can be adapted to multiple different platforms, enabling cross-platform capture of audio output from sound card devices in XR music playback scenarios.

[0063] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0064] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the specification, serve to illustrate the technical solutions of this disclosure.

[0065] Figure 1 A flowchart illustrating an audio playback method for XR according to an embodiment of the present disclosure is shown.

[0066] Figure 2 A flowchart illustrating another audio playback method for XR according to an embodiment of the present disclosure is shown.

[0067] Figure 3 This diagram illustrates an application scenario of an audio playback method for XR according to an embodiment of the present disclosure.

[0068] Figure 4 A block diagram of an audio playback apparatus for XR according to an embodiment of the present disclosure is shown.

[0069] Figure 5 A block diagram of another audio playback device for XR according to an embodiment of the present disclosure is shown.

[0070] Figure 6 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0071] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0072] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0073] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0074] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0075] Figure 1 A flowchart of an audio playback method for XR according to an embodiment of the present disclosure is shown, the method being applied to a server, such as... Figure 1 As shown, the method includes:

[0076] In step S11, obtain the audio capture plugins available on the server and the target sound card device in the current platform;

[0077] The server-side component here can be a cloud-based Extended Reality (XR) server, a cloud-based XR server that provides more flexible and efficient extended reality services by migrating the physical hardware used by the terminal to process audio to the cloud or edge. Cloud XR servers can provide high-definition, low-latency extended reality experiences while reducing the cost and complexity of terminal devices.

[0078] The server can be configured with various audio capture plugins compatible with sound card devices on multiple platforms. Each plugin can capture audio data from the corresponding platform's sound card. These plugins can then adapt to the audio APIs of multiple platforms. For example, Windows systems primarily use audio APIs such as DirectSound, WasAPI, and WaveXAPI, while Linux systems primarily use audio APIs such as ALSA and PulseAudio. Different plugins can capture audio from the sound card based on their respective audio APIs.

[0079] Sound card devices often differ across platforms, and cloud systems typically lack physical sound cards, instead using virtual sound cards for sound acquisition. Furthermore, different sound card devices correspond to different audio APIs. This method allows scanning all sound card devices on the platform to obtain the identifier of the target sound card device on the current platform, thus identifying the target sound card device on the current platform.

[0080] In step S12, a target audio capture plugin that matches the target sound card device is determined;

[0081] Once the target sound card device is identified, a matching audio capture plugin can be selected from a variety of audio capture plugins. In one example, a target audio capture plugin that corresponds to the target sound card device can be obtained based on a pre-established correspondence between sound card devices and audio capture plugins.

[0082] For example, if the target sound card device only supports WASDAPI, then select an audio capture plugin that can obtain audio data from the target sound card device through WASDAPI as the target audio capture plugin.

[0083] In step S13, based on the target audio capture plugin, the audio data output by the target sound card device is obtained;

[0084] In one possible implementation, acquiring the audio data output by the target sound card device based on the target audio capture plugin includes: acquiring the audio data output by the target sound card device through an interface corresponding to the target audio capture plugin.

[0085] The target audio capture plugin can obtain the audio data output by the target sound card device through the corresponding API. For example, if the target sound card device only supports WASI API, the matched target audio capture plugin will obtain the audio data of the target sound card device through the WASI API interface.

[0086] In step S14, the audio data is processed to obtain target audio data, and the target audio data is sent to a terminal device, which plays the target audio data.

[0087] After obtaining the audio data output by the target sound card device, the audio data can be further processed, such as conversion and encoding, to obtain the target audio data that the terminal can play. Then, the target audio data is sent to the terminal device. After receiving the target audio data, the terminal device can decode and play the target audio data.

[0088] In this embodiment, by acquiring available audio capture plugins on the server and the target sound card device on the current platform, the server is equipped with various audio capture plugins adapted to sound card devices on multiple platforms. From these various audio capture plugins, a target audio capture plugin matching the target sound card device is determined. Based on the target audio capture plugin, audio data output by the target sound card device is acquired. The audio data is processed to obtain target audio data, which is then sent to a terminal device for playback. Thus, by determining a target audio capture plugin matching the target sound card device from a variety of audio capture plugins and acquiring the audio data output by the target sound card device based on the target audio capture plugin, the server can be adapted to multiple different platforms, enabling cross-platform capture of audio output from sound card devices in XR music playback scenarios.

[0089] In one possible implementation, acquiring the audio data output by the target sound card device based on the target audio capture plugin includes: acquiring the audio data of the target process output by the target sound card device through an interface corresponding to the target audio capture plugin, based on the identifier of the target process.

[0090] Considering that when capturing audio from the server computer, not only the audio of the target process (such as an XR application) will be captured, but also the audio of other processes, the audio data can be obtained from the target sound card device based on the identifier of the target process to be captured. This audio data is obtained through the interface corresponding to the target audio capture plugin.

[0091] In this embodiment of the disclosure, by identifying the target process, only the audio data of the target process can be collected, avoiding audio interference from other applications, improving the accuracy of the captured audio, and thus improving the user experience of the client user. For example, in a cloud XR system, only the audio of the XR application can be collected, avoiding the mixing of other applications' sounds and reducing the user experience. The XR client only plays the audio of the XR application, improving the user experience of the XR client user.

[0092] In one possible implementation, the step of processing the audio data to obtain target audio data and sending the target audio data to the terminal device includes: receiving a timestamp sent by a client, the timestamp being used to indicate the time when the client sent the timestamp; adding the timestamp to the audio data; and sending the target audio data with the timestamp added to the terminal device.

[0093] In related technologies, the video frames received by the cloud XR client from the server are outdated images corresponding to the current pose. These images are then rendered using timewarping to match the current pose. Similarly, the audio data captured by the server and received by the client is always outdated relative to the client's current pose. Since XR is primarily video-based, emphasizing real-time performance and smoothness, traditional audio-video synchronization methods based on audio time references cannot achieve XR audio-video synchronization.

[0094] In this embodiment, the client can send timestamped data to the server. The timestamp indicates the moment the client sends the data to the server. When the target audio capture plugin receives the audio data output from the target sound card device, it adds a timestamp to the audio data and sends the timestamped target audio data to the terminal device. Upon receiving the timestamped target audio data, the client on the terminal device can determine whether the difference between the current time and the timestamp is within a set threshold range. Based on the determination result, it can discard or play the target audio data, achieving coarse audio-video synchronization and avoiding a poor user experience due to significant differences between sound and video.

[0095] If the difference between the current time and the timestamp is greater than the set threshold, it indicates that the time between the current time and the time of the sending timestamp is relatively long, meaning that the timeliness of the target audio data sent by the server is poor, so it can be discarded; if the difference between the current time and the timestamp is within the set threshold, it indicates that the time between the current time and the time of the sending timestamp is relatively short, meaning that the timeliness of the target audio data sent by the server is good, so the target audio data can be played.

[0096] In one possible implementation, processing the audio data to obtain target audio data and sending the target audio data to the terminal device includes: determining an encoding method that matches the application scenario based on the application scenario requirements of the terminal device for sound quality and latency; encoding the target audio data using the encoding method to obtain target audio data; and sending the target audio data to the terminal device.

[0097] In one possible implementation, determining the encoding method that matches the application scenario based on the application scenario's requirements for sound quality and latency includes: when the sound quality requirement of the application scenario is higher than the latency requirement, using an encoding method with a compression ratio lower than a set compression threshold; and when the sound quality requirement of the application scenario is lower than the latency requirement, using an encoding method with a compression ratio higher than a set compression threshold.

[0098] In different application scenarios, the requirements for sound quality and latency are often different. In the embodiments of this disclosure, an encoding method that matches the requirements for sound quality and latency in different application scenarios can be adopted to improve the user experience in the application scenario.

[0099] In certain applications, such as music playback and video conferencing, high sound quality is required, while latency requirements are relatively low. In these scenarios, the transmission and processing of audio data must ensure the highest possible sound quality, while latency is not a primary consideration.

[0100] To meet the audio quality requirements of these applications, high-quality audio codecs, audio processing algorithms, and audio transmission protocols can be used. These technologies ensure that distortion, noise, and other audio quality losses are minimized during the acquisition, transmission, and processing of audio data. Simultaneously, audio signal enhancement techniques, such as noise reduction, echo cancellation, and automatic gain control, can be employed to further improve audio quality.

[0101] In applications such as online games, latency requirements are high, while audio quality requirements are relatively low. In these scenarios, the transmission and processing of audio data need to ensure the lowest possible latency, while high audio quality is not the primary consideration.

[0102] For applications requiring high audio quality, high-quality encoding formats such as WAV and FLAC can be used. For others, lossy compression formats with high compression ratios such as Opus and AAC can be used.

[0103] To meet the audio quality requirements of these applications, high-compression-ratio audio codecs, audio processing algorithms, and audio transmission protocols can be used. These technologies ensure that audio data consumes as little bandwidth as possible, guaranteeing low-latency transmission.

[0104] In one possible implementation, the steps of acquiring audio data output by the target sound card device based on the target audio capture plugin, processing the audio data to obtain target audio data, and sending the target audio data to the terminal device include: creating an audio capture pipeline; using the target audio capture plugin as an input component of the audio capture pipeline and setting the acquisition object of the input component to the target sound card device; adding an audio processing component and an audio output component after the input component; executing the tasks in the audio capture pipeline to acquire the audio data output by the target sound card device, processing the audio data to obtain the target audio data, and sending the target audio data to the terminal device.

[0105] According to one aspect of this disclosure, an audio playback method for XR is also provided. Figure 2 This diagram illustrates a flowchart of yet another audio playback method for XR according to an embodiment of the present disclosure, the method being applied to a client, such as... Figure 2 As shown, the method includes:

[0106] In step S21, an audio playback pipeline is created;

[0107] In step S22, the target audio data sent by the server is received, and the timestamp in the target audio data is extracted;

[0108] In step S23, if the difference between the current time and the timestamp is within a set threshold range, the target audio data input callback function is sent to the audio playback pipeline to execute the task in the audio playback pipeline to play the target audio data.

[0109] When creating an audio playback pipeline, an input component is added to the pipeline, and its input callback function is set. Then, an audio processing component and an audio output component are added after the input component. In one example, the audio processing component could be a component for audio conversion and decoding, such as opusparse, opusdec, faad, mad, etc.

[0110] The server and client pipelines can remain running after startup. The audio data in the server's sound card is captured by the input component (source component) in the pipeline, and then passed to subsequent components in the pipeline for audio conversion and encoding into an audio format data stream. Finally, it is output and sent to the client on the terminal device via WebSocket.

[0111] After receiving audio data from the service, the client compares the current time with the timestamp of the audio data. If the difference is within a certain threshold, the input component in the client's pipeline can receive the target audio data through the input callback function. The data is then passed to subsequent components in the pipeline for audio decoding and final playback. If the difference between the current time and the timestamp of the audio data exceeds the threshold, the audio data is discarded.

[0112] The following describes an application scenario of this disclosure. In this scenario, the client is an extended reality (XR) client, and the server is an XR server; the timestamp is a timestamp used in the XR client to identify the pose. The implementation process of this application scenario can be found in [link to relevant documentation]. Figure 3 , Figure 3 This diagram illustrates an application scenario of an audio playback method for XR according to an embodiment of the present disclosure, such as... Figure 3 As shown, the specific process of the audio playback method for XR includes:

[0113] The XR service lists the currently available audio capture plugins, then iterates through the platform's sound card devices, matching audio components with sound card devices. If an audio capture component corresponding to the sound card device is matched, a capture pipeline is created and started to capture audio. When audio data is captured, a timestamp is added to the received audio data and sent to the XR headset client. The XR headset client starts the playback pipeline. When audio data is received, it compares the difference between the current time and the timestamp to whether it is within a set threshold range. If the difference is within the set threshold range, the audio data is decoded and played through the playback pipeline. If the difference exceeds the set threshold range, the audio data is discarded.

[0114] In addition, this disclosure can also be used for capturing audio data on cloud desktops and transmitting the captured audio data to the client. It should be noted that this disclosure can also be applied to more application scenarios, which will not be listed one by one.

[0115] The audio playback method for XR provided in this disclosure can be implemented based on GStreamer. GStreamer supports capturing audio data in various audio formats. For example, the wasapi2src element in GStreamer is used to obtain audio data from WASAPI (Windows Audio Session API). WASAPI is an API in the Windows operating system used for audio input / output, providing high-quality, low-latency audio transmission. The wasapi2src element allows the use of WASAPI as an audio source in GStreamer. Other audio capture plugins in GStreamer are not described in detail here.

[0116] You can use GStreamer's Device Monitor to iterate through all sound card devices, then filter for audio capture plugins in GStreamer that match the sound card devices, or filter for audio devices that match available audio capture plugins, and add the matching audio capture plugins to the pipeline. In one example, if the target sound card device supports the WASAPI API, you can add the audio capture plugin wasapi2src as the input source component in the pipeline. Then, set the target sound card device identifier and target process identifier for the input source component. The target sound card device identifier indicates the sound card from which audio data needs to be acquired; the target audio capture plugin can acquire audio data from the target sound card device via the API based on the target sound card device identifier. The target process identifier indicates the process from which audio data is to be captured.

[0117] Furthermore, other audio conversion and encoding components (such as opusenc, avenc_aac, lamemp3enc, etc.) can be added to the pipeline to form an audio stream (audio data); an output component appsink can be added to the pipeline, and a resampling callback function can be set for the output component to receive the captured audio data. The output component appsink in Gstreamer can enable the exchange of streaming media data in the pipeline with other applications and send audio data to the terminal device.

[0118] The client on the terminal device simultaneously starts an audio playback thread, creating a playback pipeline. The `appsrc` data source component from GStreamer is added to the pipeline as the sound source; `appsrc` in GStreamer is used to send application data to the pipeline. Then, the data input function callback of the source component is set to provide data to the data source. Audio conversion and decoding components (such as `opusparse`, `opusdec`, `faad`, `mad`, etc.) are added to the pipeline; the `autoaudiosink` output component from GStreamer is added to the pipeline, and playback begins. GStreamer's `autoaudiosink` automatically selects the appropriate audio output component for the current environment, automatically choosing the best audio output device based on the current environment, and rendering the audio data onto that device to achieve audio playback.

[0119] In one possible implementation, the audio playback method for XR can be executed by electronic devices such as terminal devices and servers. The terminal device can be a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. The method can be implemented by a processor calling computer-readable instructions stored in memory.

[0120] In addition, this disclosure also provides an audio playback device, electronic device, computer-readable storage medium, and program for XR. All of the above can be used to implement any of the audio playback methods for XR provided in this disclosure. The corresponding technical solutions and descriptions are described in the corresponding descriptions in the method section and will not be repeated here.

[0121] Figure 4 A block diagram of an audio playback apparatus for XR according to an embodiment of the present disclosure is shown, such as Figure 4 As shown, the device 40 is applied to the server and includes:

[0122] The device acquisition unit 41 is used to acquire the audio capture plugins available on the server and the target sound card device in the current platform. The server is equipped with a variety of audio capture plugins that are compatible with sound card devices on multiple platforms.

[0123] Plug-in determination unit 42 is used to determine the target audio capture plug-in that matches the target sound card device;

[0124] The audio acquisition unit 43 is used to acquire the audio data output by the target sound card device based on the target audio capture plugin;

[0125] The audio processing unit 44 is used to process the audio data to obtain target audio data and send the target audio data to the terminal device, which is used to play the target audio data.

[0126] In one possible implementation, the audio acquisition unit is used to acquire audio data output by the target sound card device based on the target audio capture plugin and through an interface corresponding to the target audio capture plugin.

[0127] In one possible implementation, the audio acquisition unit is used to acquire the audio data of the target process output by the target sound card device based on the target audio capture plugin and according to the identifier of the target process, through an interface corresponding to the target audio capture plugin.

[0128] In one possible implementation, the audio processing unit is configured to:

[0129] Receive a timestamp sent by the client, the timestamp indicating the moment the client sent the timestamp;

[0130] The timestamp is added to the audio data, and the target audio data with the timestamp added is sent to the terminal device.

[0131] In one possible implementation, the server is an XR server, and the timestamp is a timestamp in the XR client used to identify the pose, so as to instruct the target sound card device to play the audio corresponding to the pose.

[0132] In one possible implementation, the audio processing unit is configured to:

[0133] Based on the requirements of the terminal device for sound quality and latency in the application scenario, determine the encoding method that matches the application scenario;

[0134] The target audio data is encoded using the aforementioned encoding method to obtain target audio data, and then the target audio data is sent to the terminal device.

[0135] In one possible implementation, the audio processing unit is configured to:

[0136] When the sound quality requirement is higher than the latency requirement in the application scenario, an encoding method with a compression ratio lower than the set compression threshold is adopted;

[0137] When the sound quality requirement is lower than the latency requirement in the application scenario, an encoding method with a compression ratio higher than the set compression threshold is adopted.

[0138] In one possible implementation, the audio acquisition unit is configured to: create and construct an audio capture pipeline; use the target audio capture plugin as an input component of the audio capture pipeline, and set the acquisition object of the input component to the target sound card device; and add an audio processing component and an audio output component after the input component.

[0139] The audio processing unit is used to execute tasks in the audio capture pipeline to obtain audio data output by the target sound card device, process the audio data to obtain target audio data, and send the target audio data to the terminal device.

[0140] Figure 5 This diagram illustrates another audio playback device for XR according to an embodiment of the present disclosure, such as... Figure 5 As shown, the device 50 is applied to a client and includes:

[0141] Pipe creation unit 51 is used to create an audio playback pipe;

[0142] The audio receiving unit 52 is used to receive target audio data sent by the server and extract the timestamp from the target audio data;

[0143] The audio playback unit 53 is used to send the target audio data input callback function to the audio playback pipeline and execute the task in the audio playback pipeline to play the target audio data when the difference between the current time and the timestamp is within a set threshold range.

[0144] In one possible implementation, the client is an extended reality (XR) client, and the server is an XR server;

[0145] The timestamp is used in the XR client to identify the pose, instructing the target sound card device to play the audio corresponding to the pose.

[0146] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0147] This disclosure also proposes a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the above-described method. The computer-readable storage medium can be volatile or non-volatile.

[0148] This disclosure also proposes an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to execute the above-described method.

[0149] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0150] Electronic devices can be provided as terminals, servers, or other forms of devices.

[0151] Figure 6 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. For example, the electronic device 1900 may be provided as a server or a terminal device. (Refer to...) Figure 6 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0152] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output (I / O) interface 1958. Electronic device 1900 can operate on an operating system stored in memory 1932, such as Microsoft Server operating system (Windows Server). TM Apple's graphical user interface-based operating system (Mac OSX) TM ), a multi-user, multi-process computer operating system (Unix) TM Linux is a free and open-source Unix-like operating system. TM ), the open-source Unix-like operating system (FreeBSD) TM (or similar.)

[0153] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of an electronic device 1900 to perform the above-described method.

[0154] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0155] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, (but not limited to) electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0156] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0157] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0158] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0159] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0160] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0161] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0162] The computer program product can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0163] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0164] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0165] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0166] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. An audio playback method for XR, characterized in that, Applied to the server side, including: Obtain the available audio capture plugins on the server and the target sound card device on the current platform. The server is equipped with a variety of audio capture plugins that are compatible with sound card devices on multiple platforms. Identify the target audio capture plugin that matches the target sound card device; Based on the target audio capture plugin, acquire the audio data output by the target sound card device; The audio data is processed to obtain target audio data, and the target audio data is sent to a terminal device, which plays the target audio data. The step of processing the audio data to obtain target audio data and sending the target audio data to the terminal device includes: Receive a timestamp sent by the client, the timestamp indicating the moment the client sent the timestamp; The timestamp is added to the audio data, and the target audio data with the timestamp added is sent to the terminal device.

2. The method according to claim 1, characterized in that, The step of acquiring audio data output by the target sound card device based on the target audio capture plugin includes: Based on the target audio capture plugin, the audio data output by the target sound card device is obtained through the interface corresponding to the target audio capture plugin.

3. The method according to claim 1, characterized in that, The step of acquiring audio data output by the target sound card device based on the target audio capture plugin includes: Based on the target audio capture plugin, and according to the identifier of the target process, the audio data of the target process output by the target sound card device is obtained through the interface corresponding to the target audio capture plugin.

4. The method according to claim 1, characterized in that, The step of processing the audio data to obtain target audio data and sending the target audio data to the terminal device includes: Based on the requirements of the terminal device for sound quality and latency in the application scenario, determine the encoding method that matches the application scenario; The target audio data is encoded using the aforementioned encoding method to obtain target audio data, and then the target audio data is sent to the terminal device.

5. The method according to claim 4, characterized in that, The step of determining the encoding method that matches the application scenario based on the sound quality and latency requirements of the terminal device includes: When the sound quality requirement is higher than the latency requirement in the application scenario, an encoding method with a compression ratio lower than the set compression threshold is adopted; When the sound quality requirement is lower than the latency requirement in the application scenario, an encoding method with a compression ratio higher than the set compression threshold is adopted.

6. The method according to any one of claims 1-5, characterized in that, The step of acquiring audio data output by the target sound card device based on the target audio capture plugin, and processing the audio data to obtain target audio data, and sending the target audio data to the terminal device, includes: Create and build an audio capture pipeline; The target audio capture plugin is used as the input component of the audio capture pipeline, and the acquisition target of the input component is set to the target sound card device; Add an audio processing component and an audio output component after the input component; The task in the audio capture pipeline is executed to obtain audio data output by the target sound card device, process the audio data to obtain target audio data, and send the target audio data to the terminal device.

7. An audio playback method for XR, characterized in that, Applied to the client side, including: Create an audio playback pipeline; The server receives target audio data sent by the server and extracts the timestamp from the target audio data; wherein the server is used to implement the method described in any one of claims 1-6. If the difference between the current time and the timestamp is within a set threshold range, the target audio data is sent to the audio playback pipeline via a callback function to execute the task in the audio playback pipeline and play the target audio data.

8. The method according to claim 7, characterized in that, The client is an extended reality (XR) client, and the server is an XR server; The timestamp is used in the XR client to identify the pose, instructing the target sound card device to play the audio corresponding to the pose.

9. An audio playback device for XR, characterized in that, Applied to the server side, including: The device acquisition unit is used to acquire the audio capture plugins available on the server and the target sound card device in the current platform. The server is equipped with a variety of audio capture plugins that are compatible with sound card devices on multiple platforms. A plugin determination unit is used to determine a target audio capture plugin that matches the target sound card device. An audio acquisition unit is used to acquire audio data output by the target sound card device based on the target audio capture plugin; An audio processing unit is configured to process the audio data to obtain target audio data and send the target audio data to a terminal device, wherein the terminal device is configured to play the target audio data. The audio processing unit is used for: Receive a timestamp sent by the client, the timestamp indicating the moment the client sent the timestamp; The timestamp is added to the audio data, and the target audio data with the timestamp added is sent to the terminal device.

10. An audio playback device for XR, characterized in that, Applied to the client side, including: The pipe creation unit is used to create audio playback pipes; An audio receiving unit is configured to receive target audio data sent by a server and extract the timestamp from the target audio data; wherein the server includes the apparatus as described in claim 9; An audio playback unit is configured to, when the difference between the current time and the timestamp is within a set threshold range, send the target audio data input callback function to the audio playback pipeline, execute the task in the audio playback pipeline, and play the target audio data.

11. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to implement the method described in any one of claims 1 to 8.

12. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 8.

Images