Audio processing method and apparatus, electronic device, and storage medium

Abstract

Embodiments of the present disclosure provide an audio processing method and apparatus, an electronic device, and a storage medium. The audio processing method comprises: detecting a target sound source position of target audio data; on the basis of the target sound source position, determining target spatial audio coordinates corresponding to a target microphone seat; and using the target spatial audio coordinates to adjust and output the target audio data corresponding to the target microphone seat. In the embodiments of the present disclosure, the dynamic spatial audio effect is improved.

Description

Audio processing method, device, electronic device and storage medium

[0001] This application claims priority to Chinese Patent Application No. 202310955279.8 filed on July 31, 2023, and the contents of the above-mentioned Chinese patent application disclosure are hereby incorporated by reference in their entirety as a part of this application. Technical Field

[0002] The embodiments of the present disclosure relate to an audio processing method, apparatus, electronic device, and storage medium. Background Art

[0003] With the continuous development of internet technology, live streaming has become an indispensable part of daily life, such as interactive live streaming. Live streaming scenarios can generally be divided into single-person broadcasts and multi-person online interactions. In existing live streaming scenarios, the audio quality of the live streaming room is simple, resulting in a low listening quality and low immersion in the audio experience.

[0004] Summary of the Invention

[0005] The present disclosure provides an audio processing method, device, electronic device, and storage medium to improve the audio effect in a live broadcast and microphone connection scenario.

[0006] The present disclosure provides an audio processing method, the method comprising:

[0007] Detecting a target sound source position of target audio data, where the target audio data is audio data associated with a target microphone position in the live broadcast room. The target sound source position is used to describe the sound location of a target sound source object when emitting a sound corresponding to the target audio data. The target sound source object is the sound source object corresponding to the target microphone position.

[0008] Determining target spatial audio coordinates corresponding to the target microphone position based on the target sound source position, wherein the spatial audio coordinates are used to simulate a three-dimensional spatial sound effect;

[0009] The target audio data corresponding to the target microphone position is adjusted using the target spatial audio coordinates and output.

[0010] The present disclosure also provides an audio processing device, the device comprising:

[0011] a position detection module configured to detect a target sound source position of target audio data, wherein the target audio data is audio data associated with a target microphone position in the live broadcast room, wherein the target sound source position is used to describe the sound source position when the target sound source object emits the sound corresponding to the target audio data, and the target sound source object is the sound source object corresponding to the target microphone position;

[0012] an audio determination module configured to determine target spatial audio coordinates corresponding to the target microphone position according to the target sound source position;

[0013] The audio processing module is configured to adjust and output the target audio data corresponding to the target microphone position using the target spatial audio coordinates.

[0014] An embodiment of the present disclosure further provides an electronic device, the electronic device comprising:

[0015] one or more processors;

[0016] a storage device configured to store one or more programs,

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the audio processing method described in any embodiment of the present disclosure.

[0018] An embodiment of the present disclosure further provides a storage medium containing computer-executable instructions, wherein the computer-readable storage medium stores computer instructions, and when the computer-executable instructions are executed by a computer processor, they are used to execute the audio processing method described in any embodiment of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale.

[0020] FIG1 is a flow chart of an audio processing method provided by an embodiment of the present disclosure;

[0021] FIG2 is a flow chart of another audio processing method provided by an embodiment of the present disclosure;

[0022] FIG3 is a flow chart of another audio processing method provided by an embodiment of the present disclosure;

[0023] FIG4 is a schematic structural diagram of an audio processing device provided by an embodiment of the present disclosure; and

[0024] FIG5 is a schematic structural diagram of an electronic device for implementing an audio processing method provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0026] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0027] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Other terms are defined in the following description.

[0028] It should be noted that the concepts of "first", "second", "target", "reference", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0029] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0030] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0031] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0032] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

[0033] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0034] It is understandable that the above notification and user authorization process are merely illustrative and do not limit the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0035] Figure 1 is a flow chart of an audio processing method provided by an embodiment of the present disclosure. The embodiment of the present disclosure is applicable to the situation where the audio data of each microphone position is adjusted when a live broadcast is carried out. The method can be executed by an audio processing device, which can be implemented in the form of software and / or hardware. Optionally, it can be implemented by being configured on any electronic device with network communication function, which can be a mobile terminal, PC or server, etc.

[0036] As shown in FIG1 , the audio processing method in this embodiment may include but is not limited to the following processes:

[0037] S110. Detect the target sound source position of the target audio data. The target audio data is the audio data associated with the target microphone position in the live broadcast room. The target sound source position is used to describe the sound position when the target sound source object emits the sound corresponding to the target audio data. The target sound source object is the sound source object corresponding to the target microphone position.

[0038] Livestream hosts and guests typically use mono channels, ensuring balanced sound levels on both headphones. However, in livestream scenarios with multiple microphones, the audio data from the microphone in the livestream can lead to a suboptimal audio experience and lack of immersion. This can cause issues such as flattened audio and a lack of authenticity. In multi-player scenarios, the audio becomes chaotic after merging, and it's difficult to distinguish which voices belong to which. Therefore, adjustments to the audio data from the microphone in the livestream are necessary to improve both the audio quality and immersion.

[0039] The live broadcast room can include at least one microphone. Different users can interact remotely through multiple microphones in the same live broadcast room. Each user can input audio on the device corresponding to the microphone they use, and the input audio can be shared through the microphones in the live broadcast room during the interaction.

[0040] For the target microphone position in the live broadcast room, audio data associated with the target microphone position can be obtained by audio capture on the device corresponding to the target microphone position, which is recorded as the target audio data. At the same time, the target audio data can be the audio data generated when the sound source object corresponding to the target microphone position emits sound on the device corresponding to the target microphone position, and the sound source position when the sound corresponding to the target audio data is emitted by the sound source object corresponding to the target microphone position can be used as the target sound source position of the target audio data.

[0041] S120 . Determine target spatial audio coordinates corresponding to the target microphone position based on the target sound source position. The spatial audio coordinates are used to simulate a three-dimensional spatial sound effect.

[0042] To further optimize the audio immersion and allow the audio corresponding to the connected microphone in the live broadcast room to break free from the limitations of the microphone position, this solution no longer solidifies the live broadcast room audio and the microphone position. Instead, for audio data emitted from different positions corresponding to the target microphone position, the target audio source position corresponding to the target audio data is dynamically adapted to the target microphone position using the target audio source position. This allows the target microphone position to have different spatial audio coordinates when the audio source position of the audio data corresponding to the target microphone position is different, thereby ensuring that the same target microphone position uses different 3D spatial sound effect simulation results depending on the audio source position of the corresponding audio data.

[0043] S130: Use the target spatial audio coordinates to adjust and output the target audio data corresponding to the target microphone position.

[0044] For the target audio data of the target microphone in the live broadcast room, the target spatial audio coordinates can be used to simulate the three-dimensional spatial sound effect to adjust the audio effect of the target audio data of the target microphone in real time, so that the target audio data of the target microphone in the live broadcast room can be dynamically adjusted along with the change of the sound position of the target audio data of the target microphone, and then the adjusted audio data is output, so that the audio effect of the target audio data is more in line with the actual picture perception during audio output, and the audio listening and immersion are enhanced, and the problem of low listening and immersion of the audio experience in the live broadcast and microphone scene is solved, and as much as possible, it is ensured that the sound of each microphone is listened to from the audience's perspective with a clearer sense of direction, and a better dynamic spatial audio effect is achieved.

[0045] Optionally, different spatial audio coordinates are associated with different spatial audio parameters. Using these different spatial audio parameters to adjust the audio effects of the same target audio data can result in different spatial audio effects for the same target audio data, i.e., simulating different three-dimensional spatial sound effects. Optionally, the target audio data is adjusted using the target spatial audio coordinates, and the adjusted audio data is output through the target microphone position in the live broadcast room.

[0046] As an optional but non-limiting implementation, adjusting the target audio data corresponding to the target microphone position using the target spatial audio coordinates may include steps A1-A2:

[0047] Step A1: Based on the target spatial audio coordinates corresponding to the target microphone position, the original spatial audio coordinates used by the target microphone position are adjusted and updated.

[0048] Step A2: Adjust the target audio data using the spatial audio coordinates used by the adjusted and updated target microphone position.

[0049] For the microphone positions in the live broadcast room, each microphone position has a preset spatial audio coordinate adjustment interface. By calling the preset spatial audio coordinate adjustment interface, the original spatial audio coordinates used by the target microphone position are adjusted in real time and dynamically to ensure that the spatial audio coordinates to be used by the microphone position can be adjusted as the sound source position of the audio data corresponding to the microphone position changes, that is, it is adjusted as the corresponding sound position changes when the audio data is emitted.

[0050] After updating and adjusting the spatial audio coordinates to be used by the target microphone, the updated spatial audio coordinates can be used to adjust the audio effect of the target audio data of the target microphone in real time, so that the target audio data of the target microphone in the live broadcast room can be dynamically adjusted as the sound position of the target audio data of the target microphone changes.

[0051] Optionally, when the target sound source position of the target audio data is detected and the target spatial audio coordinates corresponding to the target microphone position are not obtained, the default spatial audio coordinates pre-associated with the target microphone position may be used as the target spatial audio coordinates corresponding to the target microphone position.

[0052] The technical solution of the disclosed embodiment detects the sound source position corresponding to the target audio data of the target microphone in the live broadcast room, and matches the target microphone with suitable target spatial audio coordinates based on the detected sound source position of the target audio data of the target microphone. The target spatial audio coordinates can simulate the three-dimensional spatial sound effect to adjust the target audio data of the target microphone in real time, so that the target audio data of the target microphone in the live broadcast room has a spatial audio effect, thereby solving the problem of low listening and immersive audio experience in the live broadcast scene, and ensuring as much as possible that the audience has a clearer sense of direction when listening to the sound of each microphone, thereby achieving a better dynamic spatial audio effect.

[0053] Figure 2 is a flow chart of another audio processing method according to an embodiment of the present disclosure. The technical solution of this embodiment further optimizes the process of detecting the target sound source position of the target audio data in the above embodiment. This embodiment can be combined with various optional solutions in one or more of the above embodiments. As shown in Figure 2, the audio processing method of this embodiment may include the following process:

[0054] S210. Determine the target part position of the target sound source object corresponding to the target audio data. The target part position is the position of the target part of the target sound source object in the target picture. The target picture is the image picture of the target sound source object corresponding to when the target sound source object emits the sound corresponding to the target audio data.

[0055] Among them, the target audio data is the audio data associated with the target microphone position in the live broadcast room, the target sound source position is used to describe the sound position when the target sound source object emits the sound corresponding to the target audio data, and the target sound source object is the sound source object corresponding to the target microphone position.

[0056] When the target sound source object emits a sound corresponding to the target audio data, an image screen including the target sound source object may be obtained as a target screen.

[0057] S220: Determine the target sound source position of the target audio data according to the target part position.

[0058] Optionally, for the target part of the target sound source object, the position distribution of the sound part of the target sound source object on the entire target part can be pre-stored (the sound part can be a part of the target part, and the target part includes the sound part). Combined with the position distribution of the sound part on the entire target part and the position of the target part, the position of the sound part of the target sound source object can be roughly estimated, and the estimated position of the sound part can be used as the target sound source position of the target audio data.

[0059] As an optional but non-limiting implementation, determining the target sound source position of the target audio data based on the target part position may include steps B1-B3:

[0060] Step B1: determining a target part image from a target image according to the position of the target part.

[0061] Step B2: determining a center position of a reference side boundary of the target part image, where the reference side boundary is a preset side boundary of the target part image.

[0062] Step B3: Determine the target sound source position of the target audio data based on the reference side boundary center position.

[0063] During live broadcasts, the sound source objects corresponding to the various microphone positions are not fixed. That is, the sound source objects may tilt their heads when speaking, resulting in the determined target part positions being the position coordinates corresponding to the head tilt, rather than the target part positions determined directly in front of the target part. This may lead to deviations in the position of the vocal part determined based on the target part positions, and in turn, incorrect spatial audio coordinates being obtained subsequently.

[0064] To this end, a target part image encompassing the entire target part of the target sound source object can be determined from the target image based on the position of the target part of the target sound source object in the target image. Furthermore, the lower boundary of the target part image can be used as a reference boundary, and the center position of the reference boundary can be used as the location of the sound source object's sound source. The estimated location of the sound source location can then be used as the target sound source position of the target audio data.

[0065] S230: Determine target spatial audio coordinates corresponding to the target microphone position based on the target sound source position, where the spatial audio coordinates are used to simulate a three-dimensional spatial sound effect.

[0066] As an optional but non-limiting implementation, determining the target spatial audio coordinates corresponding to the target microphone position based on the target sound source position may include steps C1-C2:

[0067] Step C1. Determine the target relative position information of the target sound source position relative to the center of the target picture. The target relative position information includes the distance and / or angle from the target sound source position to the center of the target picture. The target picture is an image picture obtained by capturing the target sound source object when the target sound source object emits a sound corresponding to the target audio data.

[0068] Step C2: determining the target spatial audio coordinates corresponding to the target microphone position according to the target relative position information, wherein the target spatial audio coordinates change as the target relative position is adjusted.

[0069] In actual multi-person communication scenarios, the audio produced by different locations has a certain correlation. If the target sound source location is directly converted into the target spatial audio coordinates after detection, the lack of reference between the target sound source locations will cause the audio data of each microphone position to differ significantly from the actual multi-person scene.

[0070] To this end, the target relative position information of the target sound source position relative to the center of the target picture can be calculated. The target relative position information includes the distance and / or angle from the target sound source position to the center of the target picture. The calculated target relative position information is then used to calculate the final spatial audio coordinates. In this way, different and relative spatial audio coordinates can be assigned based on the relative situation of the sound source position to the center of the picture, which is more in line with the relative position situation between each sound source object in an actual multi-person scene.

[0071] As an optional but non-limiting implementation, determining the target spatial audio coordinates corresponding to the target microphone position based on the target sound source position may include steps D1-D3:

[0072] Step D1: Determine a target position difference of the target audio data. The target position difference is determined based on a position difference between a target sound source position obtained by the current target sound source position detection operation and a target sound source position obtained by the previous target sound source position detection operation.

[0073] Step D2: If the target position difference does not meet the preset difference confidence condition, the target spatial audio coordinates corresponding to the target microphone position are determined according to the target sound source position.

[0074] Step D3: If the target position difference satisfies a preset difference confidence condition, the spatial audio coordinates originally used by the target audio data are determined as the target spatial audio coordinates corresponding to the target microphone position.

[0075] Each time the target spatial audio coordinates corresponding to the target microphone position are detected, not every detected spatial audio coordinates will be very different from the previous detection. If the difference is not large, if the new target spatial audio coordinates are used to adjust the target audio data, it will be found that the audio effect after adjusting the audio data using the previous and next target spatial audio parameters will not change significantly. In other words, it is acceptable to use the original target spatial audio coordinates.

[0076] However, when the difference is not significant, using the new target spatial audio coordinates will incur a significant setup overhead if you call the preset spatial audio adjustment API or use the preset spatial audio adjustment information stream to reset the spatial audio coordinates used by the target microphone position. This shows that frequently calling the API to update spatial audio coordinates will result in a significant performance overhead.

[0077] To this end, this case sets a preset difference confidence condition, first based on the target position difference between the target sound source position obtained by the current target sound source position detection operation and the target sound source position obtained by the previous adjacent target sound source position detection operation, so as to use the target position difference to detect whether the two spatial audio coordinates produce the same or similar sound effects after adjusting the same audio data.

[0078] If the target position difference is greater than or equal to the confidence λ corresponding to the preset difference confidence condition, it means that the current target spatial audio coordinates and the previous target spatial audio coordinates can produce obvious changes in the sound effects of the same audio data. At this time, the target spatial audio coordinates corresponding to the target microphone position can be determined based on the target sound source position, and the spatial audio parameters to be used by the target microphone position can be reconfigured.

[0079] If the target position difference is less than the confidence λ corresponding to the preset difference confidence condition, it means that the sound effect changes produced by the current target spatial audio coordinates and the previous target spatial audio coordinates for the same audio data are not very obvious. Even if the interface is called to change the spatial audio coordinates, the actual effect cannot be clearly distinguished from the audience's hearing sense, which is an invalid change. Therefore, this spatial audio change is discarded and the spatial audio coordinates originally used by the target audio data remain unchanged.

[0080] The above method can determine whether the spatial audio coordinates used by the target microphone position need to be reconfigured based on whether the audio effect changes significantly after the audio data is adjusted based on the two target spatial audio parameters obtained from the two previous and subsequent detections, thereby reducing performance overhead.

[0081] S240: Use the target spatial audio coordinates to adjust and output the target audio data corresponding to the target microphone position.

[0082] The technical solution of the disclosed embodiment detects the sound source position corresponding to the target audio data of the target microphone in the live broadcast room, and matches the target microphone with suitable target spatial audio coordinates based on the detected sound source position of the target audio data of the target microphone. The target spatial audio coordinates can simulate the three-dimensional spatial sound effect to adjust the target audio data of the target microphone in real time, so that the target audio data of the target microphone in the live broadcast room has a spatial audio effect, thereby solving the problem of low listening and immersive audio experience in the live broadcast scene, and ensuring as much as possible that the audience has a clearer sense of direction when listening to the sound of each microphone, thereby achieving a better dynamic spatial audio effect.

[0083] FIG3 is a flow chart of another audio processing method according to an embodiment of the present disclosure. The technical solution of this embodiment further optimizes the process of detecting the target sound source position of the target audio data in the above embodiment. This embodiment can be combined with various optional solutions in one or more of the above embodiments. As shown in FIG3, the audio processing method of this embodiment may include the following process:

[0084] S310: Determine a reference detection time interval used when performing a target sound source position detection operation.

[0085] S320: Detect the target sound source position of the target audio data according to a reference detection time interval, where the reference detection time interval is used to ensure that the time interval between two adjacent target sound source position detection operations is greater than a preset time length.

[0086] Among them, the target audio data is the audio data associated with the target microphone position in the live broadcast room, the target sound source position is used to describe the sound position when the target sound source object emits the sound corresponding to the target audio data, and the target sound source object is the sound source object corresponding to the target microphone position.

[0087] Although some of the aforementioned solutions mentioned that the detection efficiency of the sound source position is improved by optimizing the detection process of the sound source position of the audio data, the overhead of real-time sound source position detection is still relatively large compared to a real-time live broadcast. Frequent detection may cause device lag. To this end, a dynamically adjustable reference detection time interval t(s) can be set, and the target sound source position detection of the target audio data needs to be periodically performed according to the interval indicated by the reference detection time interval, so that the time interval between the current target sound source position detection operation and the previous target sound source position detection operation that is located before the current target sound source position detection operation in time is greater than the preset time length.

[0088] Optionally, it is detected whether the time difference between the start time of the current target sound source position detection operation and the start time of the previous target sound source position detection operation exceeds a reference detection time interval; if it is detected that the time difference exceeds the reference detection time interval, a sound source position detection operation is performed on the target audio data of the target microphone position in the live broadcast room; if it is detected that the time difference does not exceed the reference detection time interval, the sound source position detection operation is not performed on the target audio data of the target microphone position in the live broadcast room.

[0089] By adopting the above solution, the sound source location detection operation can be performed intermittently by setting an appropriate reference detection time interval, appropriately reducing the detection frequency of the sound source location, and avoiding device freezes caused by high detection frequency as much as possible. At the same time, it can also reduce the delay in the device detection process caused by high detection frequency.

[0090] As an optional but non-limiting implementation, determining a reference detection time interval for performing a sound source location detection operation may include steps E1-E3:

[0091] Step E1: Determine the target detection time consumed when performing a target sound source position detection operation on the target audio data.

[0092] Step E2: If the target detection time is greater than or equal to the preset time threshold, the reference detection time interval is adjusted from the first time interval to the second time interval, and the first time interval is smaller than the second time interval.

[0093] Step E3: If the target detection time is less than the preset time threshold, the reference detection time interval is adjusted from the third time interval to a fourth time interval, and the third time interval is greater than the fourth time interval.

[0094] The device performance of the device end performing the target sound source position detection is not static. The performance of the device end may decrease or increase during the detection process. Therefore, the reference detection time interval used for the interval target sound source position detection can be dynamically adjusted. The duration corresponding to the reference detection time interval can be dynamically adjusted as the device performance of the device end performing the target sound source position detection changes.

[0095] Optionally, when the device performance of the device end performing the target sound source position detection becomes higher, the duration corresponding to the reference detection time interval can be reduced, indicating that the device end can appropriately increase the detection frequency; when the device performance of the device end performing the target sound source position detection becomes lower, the duration corresponding to the reference detection time interval can be increased, indicating that the device end can appropriately reduce the detection frequency.

[0096] The performance of a device performing target sound source location detection can be described by the time it takes to perform the target sound source location detection operation on target audio data. The target detection time can be determined based on the performance of the device performing the target sound source location detection operation, and the target detection time is negatively correlated with the device performance.

[0097] If the target detection time is greater than or equal to the preset time threshold, the reference detection time interval will be increased from the first time interval to the second time interval, and the time interval between two adjacent target sound source position detections will be increased. The detection frequency will be reduced as much as possible when the performance of the device is weak, so as to avoid the cumulative increase in the device temperature due to maintaining the original detection frequency, which will cause the device to reduce its frequency, resulting in a higher delay in the detection process.

[0098] If the target detection time is less than the preset time threshold, the reference detection time interval will be lowered from the third time interval to the fourth time interval, reducing the time interval between two adjacent target sound source position detections, and increasing the detection frequency as much as possible when the performance of the device side is stronger, so as to avoid the device performance on the device side not being better utilized due to maintaining the original detection frequency, resulting in waste, thereby affecting the detection efficiency.

[0099] The above optional solution can ensure the best spatial audio experience while reducing the detection sampling rate. The idea of ​​game theory is used to balance the losses between the two. An initial preset time threshold and reference detection time interval are given. If the detection time is greater than the preset time threshold, the reference detection time interval will be appropriately increased, otherwise it will be reduced.

[0100] Optionally, the reference detection time interval is adjusted according to a preset step size, and the target detection time is determined based on the performance of the device performing the target sound source position detection operation, and the target detection time is negatively correlated with the device performance.

[0101] Given a preset time threshold, when the target detection time is less than this preset time threshold, it means that the performance status of the device side is good, and the reference detection time interval can be appropriately shortened by one step; when the target detection time is greater than this preset time threshold, it means that the performance status of the device side is poor. In order to ensure a stable live broadcast room connection experience, the reference detection time interval can be increased by one step.

[0102] As an optional but non-limiting implementation, determining a reference detection time interval for performing a sound source location detection operation may include steps F1-F3:

[0103] Step F1: Determine the temperature of the device when performing a target sound source position detection operation on the target audio data.

[0104] Step F2: If the temperature at the device end is lower than the preset temperature threshold, the reference detection time interval is adjusted from the first time interval to the second time interval, where the first time interval is lower than the second time interval.

[0105] Step F3: If the temperature of the device end is greater than or equal to the preset temperature threshold, the reference detection time interval is adjusted from the third time interval to a fourth time interval, and the third time interval is greater than the fourth time interval.

[0106] The performance of a device performing target sound source location detection can be described by the device temperature used when performing the target sound source location detection operation on target audio data. The device temperature can be determined based on the performance of the device performing the target sound source location detection operation, and the device temperature is positively correlated with the device performance. As device performance decreases, the device temperature gradually increases; as device performance improves, the device temperature decreases.

[0107] S330: Determine target spatial audio coordinates corresponding to the target microphone position based on the target sound source position, where the spatial audio coordinates are used to simulate a three-dimensional spatial sound effect.

[0108] S340: Use the target spatial audio coordinates to adjust and output the target audio data corresponding to the target microphone position.

[0109] As an optional but non-limiting implementation, when determining the target spatial audio coordinates corresponding to the target microphone position based on the target sound source position, the following steps F1-F2 are also included:

[0110] Step F1, receiving the reference spatial audio coordinates corresponding to the reference microphone position, the reference microphone position and the target microphone position belong to the same live broadcast room, the reference spatial audio coordinates are determined based on the reference sound source position, the reference sound source position is used to describe the sound position when the reference sound source object corresponding to the reference audio data emits the sound corresponding to the reference audio data, the reference audio data is the audio data associated with the reference microphone position in the live broadcast room, the reference sound source object is the sound source object corresponding to the reference microphone position, the target sound source position is detected at the device end corresponding to the target microphone position, the reference sound source position is detected at the device end corresponding to the reference microphone position, and the reference microphone position and the target microphone position correspond to different device ends.

[0111] Step F2: Adjust the reference audio data using the reference spatial audio coordinates, and output the audio through the reference microphone position in the live broadcast room according to the adjusted audio data.

[0112] The target mic position in a livestream studio and the reference mic position in a livestream studio are separate devices. If the audio data processing logic for both the target and reference mic positions is entirely handled on the target mic device, it can put significant pressure on the target mic device. For example, if the target mic position is the host's mic position and the reference mic position is the guest's mic position, if all audio processing logic is handled by the host's device, it can put significant pressure on the host, potentially leading to device overheating, freezing, and other issues.

[0113] Based on the above situation, the detection processing of the target audio data corresponding to the target microphone position and the detection processing of the reference audio data corresponding to the reference microphone position can be processed separately. In other words, the detection operation of the target sound source position of the target audio data can be placed on the device end corresponding to the target microphone position, and the detection operation of the reference sound source position of the reference audio data can be placed on the device end corresponding to the reference microphone position.

[0114] For the device corresponding to the target microphone position, it only needs to call the preset spatial audio adjustment interface to receive the reference spatial audio coordinates obtained by the device corresponding to the reference microphone position performing the reference sound source position detection operation of the reference audio data. The device corresponding to the target microphone position will not perform the reference sound source position detection operation of the reference audio data, but will be performed by the device corresponding to the reference microphone position. By separately performing the sound source position detection operations for the audio data corresponding to different microphone positions in the same live broadcast room on the device corresponding to each microphone position, the sudden increase in device overhead caused by concentrating on one device is reduced.

[0115] The technical solution of the disclosed embodiment detects the sound source position corresponding to the target audio data of the target microphone in the live broadcast room, and matches the target microphone with suitable target spatial audio coordinates based on the detected sound source position of the target audio data of the target microphone. The target spatial audio coordinates can simulate the three-dimensional spatial sound effect to adjust the target audio data of the target microphone in real time, so that the target audio data of the target microphone in the live broadcast room has a spatial audio effect, solves the problem of low listening and immersive audio experience in the live broadcast scene, and ensures as much as possible that the audience has a clearer sense of direction when listening to the sound of each microphone, and achieves better dynamic spatial audio effect. It can be seen that this solution can get rid of the limitation of the microphone position, so that the audio of the connected microphone can produce different sound effects from left to right and from top to bottom.

[0116] Figure 4 is a structural diagram of an audio processing device provided by an embodiment of the present disclosure. The embodiment of the present disclosure is applicable to the situation where the audio data of each microphone position is adjusted when a live broadcast is carried out. The device can be implemented in the form of software and / or hardware. Optionally, it can be implemented by being configured on any electronic device with network communication function, which can be a mobile terminal, PC or server, etc.

[0117] As shown in FIG. 4 , the audio processing device of this embodiment includes a position detection module 410 , an audio determination module 420 , and an audio processing module 430 .

[0118] Position detection module 410 is configured to detect a target sound source position of target audio data, where the target audio data is audio data associated with a target microphone position in the live broadcast room. The target sound source position is used to describe the sound source position of the target sound source object when emitting the sound corresponding to the target audio data. The target sound source object is the sound source object corresponding to the target microphone position.

[0119] The audio determination module 420 is configured to determine the target spatial audio coordinates corresponding to the target microphone position according to the target sound source position;

[0120] The audio processing module 430 is configured to adjust and output the target audio data corresponding to the target microphone position using the target spatial audio coordinates.

[0121] Based on the above embodiment, optionally, detecting the target sound source position of the target audio data includes:

[0122] Determine a target part position of a target sound source object corresponding to the target audio data, where the target part position is the position of the target part of the target sound source object in a target image, where the target image is an image of the target sound source object corresponding to the sound corresponding to the target audio data emitted by the target sound source object;

[0123] A target sound source position of the target audio data is determined according to the target part position.

[0124] Based on the above embodiment, optionally, determining the target sound source position of the target audio data according to the target part position includes:

[0125] determining a target part image from a target picture according to a position of the target part;

[0126] Determining a center position of a reference lateral boundary of the target portion image, where the reference lateral boundary is a preset lateral boundary of the target portion image;

[0127] The target sound source position of the target audio data is determined according to the reference side boundary center position.

[0128] Based on the above embodiment, optionally, determining the target spatial audio coordinates corresponding to the target microphone position according to the target sound source position includes:

[0129] Determining target relative position information of the target sound source position relative to the center of the target image, where the target relative position information includes the distance and / or angle from the target sound source position to the center of the target image, where the target image is an image image obtained by capturing the target sound source object when the target sound source object emits a sound corresponding to the target audio data;

[0130] The target spatial audio coordinates corresponding to the target microphone position are determined based on the target relative position information, and the target spatial audio coordinates change as the target relative position is adjusted.

[0131] Based on the above embodiment, optionally, determining the target spatial audio coordinates corresponding to the target microphone position according to the target sound source position includes:

[0132] Determining a target position difference value of the target audio data, where the target position difference value is determined based on a position difference between a target sound source position obtained by a current target sound source position detection operation and a target sound source position obtained by a previous target sound source position detection operation;

[0133] If the target position difference does not meet the preset difference confidence condition, the target spatial audio coordinates corresponding to the target microphone position are determined based on the target sound source position;

[0134] If the target position difference satisfies a preset difference confidence condition, the spatial audio coordinates originally used by the target audio data are determined as the target spatial audio coordinates corresponding to the target microphone position.

[0135] Based on the above embodiment, optionally, detecting the target sound source position of the target audio data includes:

[0136] Determining a reference detection time interval used when performing a target sound source location detection operation;

[0137] The target sound source position of the target audio data is detected according to a reference detection time interval, where the reference detection time interval is used to make the time interval between two adjacent target sound source position detection operations greater than a preset time length.

[0138] Based on the above embodiment, optionally, determining a reference detection time interval used when performing the sound source position detection operation includes:

[0139] Determining a target detection time when performing a target sound source position detection operation on target audio data;

[0140] If the target detection time is greater than or equal to the preset time threshold, the reference detection time interval is adjusted from the first time interval to the second time interval, and the first time interval is less than the second time interval;

[0141] If the target detection time is less than the preset time threshold, the reference detection time interval is adjusted from the third time interval to the fourth time interval, and the third time interval is greater than the fourth time interval.

[0142] Based on the above embodiment, optionally, the reference detection time interval is adjusted according to a preset step size when adjusting, and the target detection time is determined based on the performance of the device performing the target sound source position detection operation, and the target detection time is negatively correlated with the device performance.

[0143] Based on the above embodiment, optionally, when determining the target spatial audio coordinates corresponding to the target microphone position according to the target sound source position, the method further includes:

[0144] Receive reference spatial audio coordinates corresponding to a reference microphone position, where the reference microphone position and the target microphone position belong to the same live broadcast room. The reference spatial audio coordinates are determined based on a reference sound source position. The reference sound source position is used to describe the sound position when a reference sound source object corresponding to the reference audio data emits a sound corresponding to the reference audio data. The reference audio data is audio data associated with the reference microphone position in the live broadcast room. The reference sound source object is the sound source object corresponding to the reference microphone position. The target sound source position is detected on a device corresponding to the target microphone position. The reference sound source position is detected on a device corresponding to the reference microphone position. The reference microphone position and the target microphone position correspond to different device ends.

[0145] The reference audio data is adjusted using the reference space audio coordinates, and the audio is output through the reference microphone position in the live broadcast room according to the adjusted audio data.

[0146] Based on the above embodiment, optionally, adjusting the target audio data corresponding to the target microphone position by using the target spatial audio coordinates includes:

[0147] Based on the target spatial audio coordinates corresponding to the target microphone position, the original spatial audio coordinates used by the target microphone position are adjusted and updated;

[0148] The target audio data is adjusted using the spatial audio coordinates used by the updated target microphone position.

[0149] The technical solution provided by the embodiment of the present disclosure detects the sound source position corresponding to the target audio data of the target microphone in the live broadcast room, and matches the target microphone with suitable target spatial audio coordinates based on the detected sound source position of the target audio data of the target microphone. The target spatial audio coordinates can simulate the three-dimensional spatial sound effect to adjust the target audio data of the target microphone in real time, so that the target audio data of the target microphone in the live broadcast room has a spatial audio effect, solves the problem of low listening and immersive audio experience in the live broadcast scene, and ensures as much as possible that the audience has a clearer sense of direction when listening to the sound of each microphone, and achieves better dynamic spatial audio effect. It can be seen that this solution can get rid of the limitation of the microphone position, so that the audio of the connected microphone can produce different sound effects from left to right and from top to bottom.

[0150] The audio processing device provided by the embodiments of the present disclosure can execute the audio processing method provided by any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0151] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present disclosure.

[0152] FIG5 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Referring to FIG5 , a schematic diagram of the structure of an electronic device (such as a terminal device or server in FIG5 ) 500 suitable for implementing an embodiment of the present disclosure is shown below. The terminal device in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. The electronic device shown in FIG5 is merely an example and should not impose any limitations on the functions and scope of use of the embodiments of the present disclosure.

[0153] As shown in FIG5 , the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. Various programs and data required for the operation of the electronic device 500 are also stored in the RAM 503. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An edit / output (I / O) interface 505 is also connected to the bus 504.

[0154] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although FIG5 shows the electronic device 500 with various devices, it should be understood that not all of the devices shown are required to be implemented or present. More or fewer devices may alternatively be implemented or present.

[0155] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0156] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0157] The electronic device provided by the embodiment of the present disclosure and the audio processing method provided by the above embodiment belong to the same inventive concept. For technical details not fully described in this embodiment, please refer to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0158] An embodiment of the present disclosure provides a computer storage medium on which a computer program is stored. When the program is executed by a processor, the audio processing method provided by the above embodiment is implemented.

[0159] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0160] In some embodiments, the client and server can communicate using any currently known or later developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or later developed network.

[0161] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0162] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:

[0163] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: detects the target sound source position of the target audio data, the target audio data is the audio data associated with the target microphone position in the live broadcast room, the target sound source position is used to describe the sound position when the target sound source object emits the sound corresponding to the target audio data, and the target sound source object is the sound source object corresponding to the target microphone position; determines the target spatial audio coordinates corresponding to the target microphone position based on the target sound source position, and the spatial audio coordinates are used to simulate three-dimensional spatial sound effects; and uses the target spatial audio coordinates to adjust and output the target audio data corresponding to the target microphone position.

[0164] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0165] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0166] The units involved in the embodiments described in this disclosure may be implemented in software or hardware. In some cases, the name of a unit does not limit the unit itself. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses."

[0167] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0168] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0169] According to one or more embodiments of the present disclosure, Example 1 provides an audio processing method, including:

[0170] Detect the target sound source position of the target audio data. The target audio data is the audio data associated with the target microphone position in the live broadcast room. The target sound source position is used to describe the sound position when the target sound source object emits the sound corresponding to the target audio data. The target sound source object is the sound source object corresponding to the target microphone position.

[0171] Determine the target spatial audio coordinates corresponding to the target microphone position based on the target sound source position. The spatial audio coordinates are used to simulate three-dimensional spatial sound effects.

[0172] The target audio data corresponding to the target microphone position is adjusted and output using the target spatial audio coordinates.

[0173] According to one or more embodiments of the present disclosure, Example 2 detects a target sound source position of target audio data according to the method of Example 1, including:

[0174] Determine a target part position of a target sound source object corresponding to the target audio data, where the target part position is the position of the target part of the target sound source object in a target image, where the target image is an image of the target sound source object corresponding to the sound corresponding to the target audio data emitted by the target sound source object;

[0175] A target sound source position of the target audio data is determined according to the target part position.

[0176] According to one or more embodiments of the present disclosure, Example 3 determines the target sound source position of the target audio data based on the target part position according to the method of Example 2, including:

[0177] determining a target part image from a target picture according to a position of the target part;

[0178] Determining a center position of a reference lateral boundary of the target portion image, where the reference lateral boundary is a preset lateral boundary of the target portion image;

[0179] The target sound source position of the target audio data is determined according to the reference side boundary center position.

[0180] According to one or more embodiments of the present disclosure, Example 4, based on the method of Example 1, determines the target spatial audio coordinates corresponding to the target microphone position according to the target sound source position, including:

[0181] Determining target relative position information of the target sound source position relative to the center of the target image, where the target relative position information includes the distance and / or angle from the target sound source position to the center of the target image, where the target image is an image image obtained by capturing the target sound source object when the target sound source object emits a sound corresponding to the target audio data;

[0182] The target spatial audio coordinates corresponding to the target microphone position are determined based on the target relative position information, and the target spatial audio coordinates change as the target relative position is adjusted.

[0183] According to one or more embodiments of the present disclosure, Example 5, based on the method of Example 1, determines the target spatial audio coordinates corresponding to the target microphone position according to the target sound source position, including:

[0184] Determining a target position difference value of the target audio data, where the target position difference value is determined based on a position difference between a target sound source position obtained by a current target sound source position detection operation and a target sound source position obtained by a previous target sound source position detection operation;

[0185] If the target position difference does not meet the preset difference confidence condition, the target spatial audio coordinates corresponding to the target microphone position are determined based on the target sound source position;

[0186] If the target position difference satisfies a preset difference confidence condition, the spatial audio coordinates originally used by the target audio data are determined as the target spatial audio coordinates corresponding to the target microphone position.

[0187] According to one or more embodiments of the present disclosure, Example 6 detects a target sound source position of target audio data according to the method of any one of Examples 1 to 5, including:

[0188] Determining a reference detection time interval used when performing a target sound source location detection operation;

[0189] The target sound source position of the target audio data is detected according to a reference detection time interval, where the reference detection time interval is used to make the time interval between two adjacent target sound source position detection operations greater than a preset time length.

[0190] According to one or more embodiments of the present disclosure, Example 7, based on the method of Example 6, determines a reference detection time interval used when performing a sound source location detection operation, including:

[0191] Determining a target detection time when performing a target sound source position detection operation on target audio data;

[0192] If the target detection time is greater than or equal to the preset time threshold, the reference detection time interval is adjusted from the first time interval to the second time interval, and the first time interval is less than the second time interval;

[0193] If the target detection time is less than the preset time threshold, the reference detection time interval is adjusted from the third time interval to the fourth time interval, and the third time interval is greater than the fourth time interval.

[0194] According to one or more embodiments of the present disclosure, Example 8 is based on the method described in Example 7, and the reference detection time interval is adjusted according to a preset step size when adjusting, and the target detection time is determined based on the performance of the device that performs the target sound source position detection operation, and the target detection time is negatively correlated with the device performance.

[0195] According to one or more embodiments of the present disclosure, Example 9, according to the method of Example 1, further includes, when determining the target spatial audio coordinates corresponding to the target microphone position based on the target sound source position:

[0196] Receive reference spatial audio coordinates corresponding to a reference microphone position, where the reference microphone position and the target microphone position belong to the same live broadcast room. The reference spatial audio coordinates are determined based on a reference sound source position. The reference sound source position is used to describe the sound position when a reference sound source object corresponding to the reference audio data emits a sound corresponding to the reference audio data. The reference audio data is audio data associated with the reference microphone position in the live broadcast room. The reference sound source object is the sound source object corresponding to the reference microphone position. The target sound source position is detected on a device corresponding to the target microphone position. The reference sound source position is detected on a device corresponding to the reference microphone position. The reference microphone position and the target microphone position correspond to different device ends.

[0197] The reference audio data is adjusted using the reference space audio coordinates, and the audio is output through the reference microphone position in the live broadcast room according to the adjusted audio data.

[0198] According to one or more embodiments of the present disclosure, Example 10 uses the method of Example 1 to adjust the target audio data corresponding to the target microphone position using the target spatial audio coordinates, including:

[0199] Based on the target spatial audio coordinates corresponding to the target microphone position, the original spatial audio coordinates used by the target microphone position are adjusted and updated;

[0200] The target audio data is adjusted using the spatial audio coordinates used by the adjusted and updated target microphone position.

[0201] According to one or more embodiments of the present disclosure, Example 11 provides an audio processing device, including:

[0202] a position detection module configured to detect a target sound source position of target audio data, where the target audio data is audio data associated with a target microphone position in the live broadcast room, and the target sound source position is used to describe the sound position of the target sound source object when emitting the sound corresponding to the target audio data, and the target sound source object is the sound source object corresponding to the target microphone position;

[0203] an audio determination module configured to determine target spatial audio coordinates corresponding to a target microphone position based on a target sound source position;

[0204] The audio processing module is configured to adjust and output the target audio data corresponding to the target microphone position using the target spatial audio coordinates.

[0205] According to one or more embodiments of the present disclosure, Example 12 provides an electronic device, the electronic device including:

[0206] one or more processors;

[0207] a storage device configured to store one or more programs,

[0208] When one or more programs are executed by one or more processors, the one or more processors implement the audio processing method as described in any one of Examples 1-10.

[0209] According to one or more embodiments of the present disclosure, Example 13 provides a storage medium containing computer-executable instructions, which are used to perform the audio processing method of any one of Examples 1-10 when executed by a computer processor.

[0210] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0211] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0212] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. An audio processing method, comprising: Detecting a target sound source position of target audio data; Determining the target spatial audio coordinates corresponding to the target microphone position according to the target sound source position; The target audio data corresponding to the target microphone position is adjusted and outputted using the target spatial audio coordinates.

2. The method according to claim 1, wherein: The target audio data is the audio data associated with the target microphone position in the live broadcast room, the target sound source position is the position of the target sound source object corresponding to the target audio data, and the target sound source object is the sound source object corresponding to the target microphone position.

3. The method according to claim 1, wherein: The detecting the target sound source position of the target audio data comprises: Determine a target part position of a target sound source object corresponding to the target audio data; The target sound source position of the target audio data is determined according to the target part position.

4. The method according to claim 3, wherein: The step of determining the target sound source position of the target audio data according to the target part position includes: Determining a target part image from the target picture according to the target part position, wherein the target part position is the position of the target part of the target sound source object in the target picture, and the target picture is an image picture including the target sound source object; Determining a center position of a reference side boundary of the target part image, wherein the reference side boundary is a preset side boundary of the target part image; The target sound source position of the target audio data is determined according to the reference side boundary center position.

5. The method according to claim 1, wherein: The step of determining the target spatial audio coordinates corresponding to the target microphone position according to the target sound source position includes: Determine target relative position information of the target sound source position relative to the center of a target picture, wherein the target relative position information includes a distance and / or angle from the target sound source position to the center of the target picture, and the target picture is an image picture including the target sound source object; The target spatial audio coordinates corresponding to the target microphone position are determined according to the target relative position information, and the target spatial audio coordinates change with the adjustment of the target relative position.

6. The method according to claim 1, wherein: The step of determining the target spatial audio coordinates corresponding to the target microphone position according to the target sound source position includes: Determining a target position difference of the target audio data, wherein the target position difference is determined based on a position difference between a target sound source position obtained by a current target sound source position detection operation and a target sound source position obtained by a previous target sound source position detection operation; If the target position difference does not meet the preset difference confidence condition, determining the target spatial audio coordinates corresponding to the target microphone position according to the target sound source position; If the target position difference satisfies a preset difference confidence condition, the spatial audio coordinates originally used by the target audio data are determined as the target spatial audio coordinates corresponding to the target microphone position.

7. The method according to any one of claims 1 to 6, wherein: The detecting the target sound source position of the target audio data comprises: Determining a reference detection time interval used when performing a target sound source location detection operation; The target sound source position of the target audio data is detected according to the reference detection time interval, wherein the reference detection time interval is used to make the time interval between two adjacent target sound source position detection operations in time greater than a preset time length.

8. The method according to claim 7, wherein: The step of determining a reference detection time interval used when performing a sound source location detection operation comprises: Determining a target detection time consumption when performing the target sound source position detection operation on the target audio data; If the target detection time is greater than or equal to a preset time threshold, adjusting the reference detection time interval from a first time interval to a second time interval, wherein the first time interval is less than the second time interval; If the target detection time is less than a preset time threshold, the reference detection time interval is adjusted from a third time interval to a fourth time interval, wherein the third time interval is greater than the fourth time interval.

9. The method according to claim 8, wherein: The reference detection time interval is adjusted according to a preset step size when adjusting, the target detection time is determined based on the performance of the device that performs the target sound source position detection operation, and the target detection time is negatively correlated with the performance of the device.

10. The method according to claim 1, wherein: When determining the target spatial audio coordinates corresponding to the target microphone position according to the target sound source position, it also includes: Receiving reference spatial audio coordinates corresponding to a reference microphone position, wherein the reference microphone position and the target microphone position belong to the same live broadcast room, and the reference spatial audio coordinates are determined based on a reference sound source position; The reference audio data is adjusted using the reference spatial audio coordinates, and audio output is performed through a reference microphone position in a live broadcast room according to the adjusted audio data.

11. The method according to claim 10, wherein: The reference sound source position is the position of the reference sound source object corresponding to the reference audio data, the reference audio data is the audio data associated with the reference microphone position in the live broadcast room, and the reference sound source object is the sound source object corresponding to the reference microphone position.

12. An audio processing device, comprising: a position detection module, configured to detect a target sound source position of target audio data, wherein the target audio data is audio data associated with the target microphone position in the live broadcast room, the target sound source position is used to describe a sound position when a target sound source object emits a sound corresponding to the target audio data, and the target sound source object is a sound source object corresponding to the target microphone position; an audio determination module, configured to determine the target spatial audio coordinates corresponding to the target microphone position according to the target sound source position; and The audio processing module is configured to adjust and output the target audio data corresponding to the target microphone position using the target spatial audio coordinates.

13. An electronic device comprising: at least one processor; as well as a storage device configured to store at least one program, Wherein, when the at least one program is executed by the at least one processor, the at least one processor implements the audio processing method as described in any one of claims 1-11.

14. A storage medium comprising computer executable instructions, wherein: When the computer executable instructions are executed by a computer processor, they are used to perform the audio processing method according to any one of claims 1 to 11.

Images