Audio data processing method, device, system, electronic device and storage medium

By splitting the virtual sound source into mirrored sound sources on the boundary of the speaker array and distributing gain, the virtual sound source positioning inside the speaker array is realized, solving the problem of the inability to accurately locate the sound source inside the array in the prior art, and improving the The realism and accuracy of audio data processing.

CN118678286BActive Publication Date: 2024-11-26TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411071080.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-06
Publication Date
2024-11-26
Estimated Expiration
2044-08-06

AI Technical Summary

Technical Problem

The existing speaker array technology cannot locate virtual sound sources inside the speaker array. The VBAP algorithm can only locate sounds on the deployment boundary of the array, and cannot achieve accurate sound source locations inside the array.

Method used

By splitting the virtual sound source into two mirrored sound sources on the deployment boundary of the speaker array, and assigning the audio gain of the virtual sound source to each mirrored sound source, the audio gain of each mirrored sound source is allocated to the associated target On the speaker, the virtual sound source positioning inside the speaker array is realized.

Benefits of technology

It solves the problem that the VBAP algorithm cannot locate the virtual sound source inside the speaker array, realizes accurate reproduction of the audio image inside the speaker array, and improves the accuracy and reality of audio data processing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118678286B_ABST
    Figure CN118678286B_ABST
Patent Text Reader

Abstract

The embodiments of the present application provide an audio data processing method, device, system, electronic device and storage medium, which may involve the fields of audio processing. The method includes: determining two mirror sound source positions of the virtual sound source position on the deployment boundary of the speaker array according to the reference position of the speaker array deployment and the virtual sound source position in the audio file, determining the first audio gain weight corresponding to each mirror sound source position according to the virtual sound source position and the two mirror sound source positions, determining the target audio gain of each target speaker according to each mirror sound source position, the position of each associated target speaker and its corresponding first audio gain weight, and the audio gain in the audio file, and obtaining the target audio data to be played by each target speaker according to each target audio gain and the audio data in the audio file. Based on this method, accurate positioning of the virtual sound source inside the speaker array is achieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the field of computer technology and may involve fields such as audio processing. Specifically, the present application relates to an audio data processing method, device, system, electronic device and storage medium. Background Art

[0002] With the development of multimedia technology, people's demand for audio quality is getting higher and higher, and they are beginning to pursue a more realistic and immersive audio experience.

[0003] There are currently a variety of spatialization technologies that use speaker arrays to virtually position sound sources. These technologies usually assume that the listener's position is known and fixed, and the speakers surround the listener in a two-dimensional ring or a three-dimensional sphere. Among them, common speaker layouts include 5.1 surround sound, 7.1 surround sound, and 7.1.4 panoramic sound.

[0004] VectorBasedAmplitude is often used in related technologies.

[0005] The VBAP (Visual Basic Panning) algorithm is used to locate audio objects and use three adjacent speakers in the space to form a three-dimensional sound vector to realize the positioning of the virtual sound source in the three-dimensional space.

[0006] However, this algorithm can only locate the sound on the two-dimensional ring or three-dimensional sphere formed by the speaker array, and cannot locate the position of the virtual sound source inside the speaker array. Summary of the invention

[0007] The purpose of the embodiments of the present application is to provide an audio data processing method, device, system, electronic device and storage medium that can realize virtual sound source localization inside a speaker array. To achieve this purpose, the technical solutions provided by the embodiments of the present application are as follows:

[0008] On the one hand, an embodiment of the present application provides an audio data processing method, which is applied to an audio data processing system, wherein the system includes an audio processing device and a speaker array deployed around a reference position, wherein the speaker array includes a plurality of speakers; the method is performed by the audio processing device, and the method includes:

[0009] Acquire an audio file; the audio file includes audio data and metadata of the audio data, wherein the metadata includes a virtual sound source position and an audio gain corresponding to the audio data;

[0010] Determine, according to the reference position and the virtual sound source position, two mirror image sound source positions of the virtual sound source position on the deployment boundary of the speaker array;

[0011] Determine, according to the virtual sound source position and the two image sound source positions, a first audio gain weight corresponding to each image sound source position;

[0012] For each of the image sound source positions, determining at least one target speaker associated with the image sound source position from the plurality of speakers according to the image sound source position and the positions of the speakers;

[0013] For each of the image sound source positions, determining the target audio gain weights of the target speakers associated with the image sound source position according to the image sound source position, the positions of the target speakers associated with the image sound source position, and the first audio gain weight corresponding to the image sound source position;

[0014] For each target speaker, the target audio gain of the target speaker is determined according to the target audio gain weight of the target speaker and the audio gain; and the target audio data to be played by the target speaker is obtained according to the target audio gain of the target speaker and the audio data.

[0015] On the other hand, an embodiment of the present application further provides an audio data processing apparatus, wherein the apparatus is deployed in an audio processing device in an audio data processing system, wherein the audio data processing system comprises the audio processing device and a speaker array deployed around a reference position, wherein the speaker array comprises a plurality of speakers; the apparatus comprises:

[0016] An acquisition module, used to acquire an audio file; the audio file includes audio data and metadata of the audio data, and the metadata includes a virtual sound source position and an audio gain corresponding to the audio data;

[0017] An image sound source localization module, used to determine two image sound source positions of the virtual sound source position on the deployment boundary of the speaker array according to the reference position and the virtual sound source position;

[0018] A first weight determination module, used to determine a first audio gain weight corresponding to each image sound source position according to the virtual sound source position and the two image sound source positions;

[0019] a target speaker determination module, configured to determine, for each of the image sound source positions, at least one target speaker associated with the image sound source position from the plurality of speakers according to the image sound source position and the positions of the speakers;

[0020] A second weight determination module is used to determine, for each of the image sound source positions, a target audio gain weight of each target speaker associated with the image sound source position according to the image sound source position, the positions of each target speaker associated with the image sound source position, and the first audio gain weight corresponding to the image sound source position;

[0021] The target audio data determination module is used to determine the target audio gain of each target speaker according to the target audio gain weight of the target speaker and the audio gain; and obtain the target audio data to be played by the target speaker according to the target audio gain of the target speaker and the audio data.

[0022] Optionally, the first weight determination module may be used to:

[0023] Determine a first vector pointing from the reference position to the virtual sound source position;

[0024] For each image sound source position, determining a second vector pointing from the reference position to the image sound source position;

[0025] Determine a first audio gain weight corresponding to each of the image sound source positions according to the first vector, the second vector corresponding to each of the image sound source positions, and the first condition;

[0026] Among them, the first condition is a constraint condition satisfied by the first audio gain weights corresponding to the two mirror sound source positions; the adjusted vectors corresponding to the two mirror sound source positions and the first vector satisfy the second condition, and the adjusted vector corresponding to each of the mirror sound source positions is obtained by adjusting the second vector corresponding to the mirror sound source position based on the first audio gain weight corresponding to the mirror sound source position.

[0027] Optionally, the two mirror image sound source positions include a first mirror image sound source position and a second mirror image sound source position;

[0028] The first condition includes:

[0029] ;in, is a first audio gain weight corresponding to the first image sound source position; is the first audio gain weight corresponding to the second image sound source position;

[0030] The second condition includes:

[0031] ; Wherein, O represents the reference position, b represents the virtual sound source position, represents the first vector, represents the position of the first image sound source, Represents the second vector corresponding to the position of the first image sound source, represents the adjusted vector corresponding to the first image source position, Indicates the position of the second image sound source, Represents the second vector corresponding to the position of the second image sound source, Represents the adjusted vector corresponding to the position of the second image sound source.

[0032] Optionally, the second weight determination module may be used to:

[0033] Determine a second vector pointing from the reference position to the mirror image sound source position;

[0034] For each target speaker associated with the image sound source position, determining a third vector pointing from the reference position to the position of the target speaker;

[0035] Determine the contribution of each target speaker to the image sound source position according to the second vector, the third vector corresponding to each target speaker, and the third condition; wherein the third condition is a constraint condition satisfied by the contribution of each target speaker to the image sound source position; the adjusted vector corresponding to each target speaker and the second vector satisfy the fourth condition, and the adjusted vector corresponding to each target speaker is obtained by adjusting the third vector corresponding to the target speaker based on the contribution of the target speaker to the image sound source position;

[0036] For each target speaker associated with the image sound source position, the product of the contribution of the target speaker to the image sound source position and the first audio gain weight corresponding to the image sound source position is used as the target audio gain weight of the target speaker.

[0037] Optionally, the image source position is associated with n target speakers, n 1, and n is an integer;

[0038] The third condition includes:

[0039] ;

[0040] in, Indicates the contribution of the nth target loudspeaker associated with the image sound source position to the image sound source position;

[0041] The fourth condition includes:

[0042] ;

[0043] Wherein, O represents the reference position, Indicates the position of the image sound source, Represents the second vector corresponding to the position of the image sound source, Indicates the position of the nth target speaker associated with the image sound source position, represents the third vector corresponding to the associated n-th target speaker, Represents the adjusted vector corresponding to the associated n-th target speaker.

[0044] Optionally, the target speaker determination module may be used to:

[0045] Determine a second distance between the position of each loudspeaker and the position of the image sound source;

[0046] Selecting at least one speaker whose second distance from the image sound source position meets a preset condition from among the speakers as at least one target speaker associated with the image sound source position;

[0047] The preset condition includes at least one of the following:

[0048] The distance between the position of the loudspeaker and the position of the image sound source is less than or equal to a first threshold;

[0049] According to the order of the distance between the position of each speaker and the position of the image sound source from near to far, the speakers with the preset number of corresponding distances are sorted in the front.

[0050] Optionally, the image sound source localization module can be used for:

[0051] Determine a straight line between the reference position and the virtual sound source position;

[0052] The two intersection points of the straight line and the deployment boundary are respectively used as two mirror image sound source positions corresponding to the virtual sound source position.

[0053] Optionally, the virtual sound source position is located inside the speaker array or on a deployment boundary.

[0054] Optionally, the deployment boundary of the speaker array includes:

[0055] In the case where the plurality of speakers in the speaker array are arranged in a two-dimensional space, the deployment boundary of the speaker array is a circumscribed circle of the plurality of speakers;

[0056] When the plurality of speakers in the speaker array are arranged in a three-dimensional space, the deployment boundary of the speaker array is a spherical surface of a circumscribed sphere of the plurality of speakers.

[0057] The embodiment of the present application also provides an audio data processing system, the system comprising an audio processing device, and a speaker array deployed around a reference position, the speaker array comprising a plurality of speakers;

[0058] in:

[0059] The audio processing device is used to obtain an audio file; the audio file includes audio data and metadata of the audio data, and the metadata includes a virtual sound source position and an audio gain corresponding to the audio data; according to the reference position and the virtual sound source position, two mirror sound source positions of the virtual sound source position on the deployment boundary of the speaker array are determined; according to the virtual sound source position and the two mirror sound source positions, a first audio gain weight corresponding to each mirror sound source position is determined; for each mirror sound source position, according to the mirror sound source position and the positions of each of the speakers, at least one target speaker associated with the mirror sound source position is determined from the multiple speakers; according to the mirror sound source position, the positions of each target speaker associated with the mirror sound source position, and the first audio gain weight corresponding to the mirror sound source position, a target audio gain weight of each target speaker associated with the mirror sound source position is determined; for each of the target speakers, according to the target audio gain weight of the target speaker and the audio gain, a target audio gain of the target speaker is determined; according to the target audio gain of the target speaker and the audio data, the target audio data corresponding to the target speaker is obtained; and the corresponding target audio data is sent to each of the target speakers respectively;

[0060] The target speaker is used to play the received target audio data.

[0061] An embodiment of the present application further provides an electronic device, which includes a memory and a processor, wherein a computer program is stored in the memory, and the processor executes the computer program to implement a method provided in any optional embodiment of the present application.

[0062] On the other hand, an embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the method provided in any optional embodiment of the present application is implemented.

[0063] On the other hand, an embodiment of the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the method provided in any optional embodiment of the present application.

[0064] The beneficial effects of the technical solution provided by the embodiment of the present application are as follows:

[0065] The audio data processing method provided in the embodiment of the present application is applied to an audio processing system, which includes an audio processing device and a speaker array deployed around a reference position. The method maps the virtual sound source position to the deployment boundary of the speaker array to obtain two mirror sound source positions, and then determines each target speaker and its target audio gain associated with each mirror sound source position based on the two mirror sound source positions, the virtual sound source position, and the position of each speaker, and then obtains the target audio data to be played by each target speaker. Based on this method, by splitting the virtual sound source into two mirror sound sources on the deployment boundary of the speaker array, and allocating the audio gain of the virtual sound source to each mirror sound source, and then allocating the audio gain of each mirror sound source to the associated target speaker in the speaker array, the problem that the VBAP algorithm can only locate the virtual sound source position on the deployment boundary of the speaker array, but cannot locate the virtual sound source inside the speaker array, can be accurately reproduced The sound and image inside the speaker array. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in describing the embodiments of the present application are briefly introduced below.

[0067] Figure 1a A schematic diagram of the structure of an audio data system provided in an embodiment of the present application;

[0068] Figure 1b A flowchart of an audio data processing method provided in an embodiment of the present application;

[0069] Figure 2a A schematic diagram of positioning a virtual sound source position in a two-dimensional spatial layout provided in an embodiment of the present application;

[0070] Figure 2b A schematic diagram of positioning a virtual sound source position in a three-dimensional space layout provided in an embodiment of the present application;

[0071] Figure 3 A schematic diagram of locating the position of an image sound source provided in an embodiment of the present application;

[0072] Figure 4a A schematic diagram of image sound source positioning in a two-dimensional space layout provided in an embodiment of the present application;

[0073] Figure 4b A schematic diagram of image sound source positioning in a three-dimensional space layout provided in an embodiment of the present application;

[0074] Figure 5a A schematic diagram of positioning an image sound source based on a speaker position in a two-dimensional spatial layout provided by an embodiment of the present application;

[0075] Figure 5bA schematic diagram of positioning an image sound source based on a speaker position in a three-dimensional spatial layout provided by an embodiment of the present application;

[0076] Figure 6 A schematic diagram of multi-directional division of image sound sources based on the MDAP algorithm provided in an embodiment of the present application;

[0077] Figure 7 A schematic diagram of a process for performing audio rendering on other mobile terminals based on the MOAP algorithm provided in an embodiment of the present application;

[0078] Figure 8 A schematic diagram of a flow chart of audio playback provided in an embodiment of the present application;

[0079] Fig. 9 A schematic diagram of the structure of another audio data processing system provided in an embodiment of the present application;

[0080] Fig.10 A schematic diagram of the structure of an audio data processing device provided in an embodiment of the present application;

[0081] Fig.11 A schematic diagram of the structure of an audio data processing device provided in an embodiment of the present application;

[0082] Fig.12 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0083] The embodiments of the present application are described below in conjunction with the drawings in the present application. It should be understood that the implementation methods described below in conjunction with the drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions of the embodiments of the present application.

[0084] It will be understood by those skilled in the art that, unless specifically stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the terms "including" and "comprising" used in the embodiments of the present application refer to that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements and / or components, but do not exclude the implementation as other features, information, data, steps, operations, elements, components and / or combinations thereof supported by the technical field. It should be understood that when we say that an element is "connected" or "coupled" to another element, the one element may be directly connected or coupled to the other element, or it may refer to that the one element and the other element establish a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein indicates at least one of the items defined by the term, for example, "A and / or B" may be implemented as "A", or as "B", or as "A and B". When describing multiple (two or more) items, if the relationship between the multiple items is not clearly defined, the multiple items may refer to one, multiple or all of the multiple items. For example, the description of "parameter A includes A1, A2, A3" can be implemented as parameter A including A1 or A2 or A3, and can also be implemented as parameter A including at least two of the three items A1, A2, A3.

[0085] In the related art, the VBAP algorithm is used to locate the virtual sound source position of the audio object. The VBAP algorithm activates three adjacent speakers according to the virtual sound source position, and adjusts the gain of each speaker according to the direction vector of the virtual sound source to achieve the positioning of the virtual sound source in space. Since the VBAP algorithm activates three speakers adjacent to the virtual sound source position, the sound source synthesized by the three adjacent speakers is on the plane where the three speakers are located, close to the sphere formed by the speaker array, and the virtual sound source position is usually located on the sphere formed by the speaker array. Therefore, the VBAP algorithm cannot locate the virtual sound source inside the speaker array.

[0086] Based on the above problems, the embodiments of the present application provide an audio data processing method, device, system, electronic device and storage medium, which can be applied to scenarios such as three-dimensional sound production, audio playback of home theaters, and spatial audio playback in augmented reality (Extended Reality), and can realize the positioning of virtual sound sources inside the speaker array. This method splits the virtual sound source into two mirror sound sources on the deployment boundary of the speaker array, and distributes the audio gain of the virtual sound source to each mirror sound source, and then distributes the audio gain of each mirror sound source to the target speaker in the speaker array. This solves the problem that the VBAP algorithm can only locate the virtual sound source position on the deployment boundary of the speaker array, but cannot locate the virtual sound source inside the speaker array, and can accurately reproduce the sound and image inside the speaker array.

[0087] It should be noted that in the optional embodiments of the present application, the data related to the object information involved (such as the played audio data), when the embodiments in the present application are applied to specific products or technologies, need to obtain the permission or consent of the object, and the collection, use and processing of the relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions. In other words, if the embodiments of the present application involve data related to the object, it needs to be obtained with the authorization and consent of the object, the authorization and consent of the relevant departments, and in compliance with the relevant laws, regulations and standards of the country and region. If personal information is involved in the embodiments, the acquisition of all personal information needs to obtain the consent of the individual. If sensitive information is involved, the separate consent of the information subject needs to be obtained. The embodiments also need to be implemented with the authorization and consent of the object.

[0088] In order to better understand and illustrate the method provided in the embodiments of the present application, some technical terms involved in the embodiments of the present application are first explained and illustrated below.

[0089] Audio object: A sound perceived as a whole or as a sound emitted by a single source independent of the environment.

[0090] Metadata: Data that describes information about audio data.

[0091] Audio gain: A description of the degree to which the current, voltage or power of a component, circuit, device or system is increased. Usually expressed in decibels (dB).

[0092] Sound image: also known as virtual sound source or perceived sound source. When two or more speakers are used for stereo playback, the listener can perceive the spatial position of the sound parts. The sound picture formed based on the spatial position of each sound part shown in the listener's hearing is called sound image.

[0093] The audio data processing method provided in the embodiment of the present application can be applied to an audio data processing system, such as Figure 1a As shown, the system includes an audio processing device 10, and a speaker array deployed around a reference position, wherein the speaker array includes multiple speakers (seven are shown in the figure, namely 11 to 17, forming a 7.1 surround sound layout). The reference position is a preset listener position (also called a listening point), and the audio processing device can communicate directly or indirectly with each speaker 11 to 17 in the speaker array by wired or wireless means, and send target audio data to each speaker 11 to 17, so that each speaker plays audio. The audio processing device can be any electronic device, such as a user terminal or a server.

[0094] The server may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server (also called the cloud) that provides cloud computing services. User terminals (also called user devices) may be smartphones, tablet computers, laptop computers, desktop computers, intelligent voice interaction devices (such as smart speakers), wearable electronic devices (such as smart watches), vehicle-mounted terminals, smart home appliances (such as smart TVs), AR / VR devices, etc., but are not limited to these.

[0095] The following describes several embodiments to illustrate the technical solutions of the embodiments of the present application and the technical effects produced by the technical solutions of the present application. It should be noted that the following embodiments can refer to, draw on or combine with each other, and the same terms, similar features and similar implementation steps in different embodiments will not be described repeatedly.

[0096] Figure 1b A schematic diagram of a flow chart of an audio data processing method provided in an embodiment of the present application is shown. The method can be applied to an audio data processing system (which may include but is not limited to Figure 1a In the system shown in the figure), the method can be executed by an audio processing device in the audio data processing system, wherein the method adopts a mirror-object amplitude panning algorithm (Mirror-Object Amplitude Panning, MOAP) to realize the positioning of the virtual sound source, and specifically can include the following steps S110 to S160.

[0097] Step S110: Acquire an audio file.

[0098] The audio file is an audio file of an audio object to be processed, including audio data of the audio object and metadata of the audio data, wherein the metadata includes the virtual sound source position (of the audio object) and audio gain corresponding to the audio data. The virtual sound source position indicates the position of the virtual sound source in space, and the virtual sound source position contained in the metadata is a relative position relative to the reference position in the audio data processing system. The audio gain indicates the volume / intensity / loudness of the audio, and is positively correlated with the amplitude of the audio.

[0099] Optionally, the virtual sound source position in the metadata may be a two-dimensional / three-dimensional coordinate position with the reference position as the origin, or may be a polar coordinate position with the reference position as the origin. This embodiment of the application does not limit this.

[0100] For example, Figure 2a As shown in , when the layout of each speaker in the speaker array is a two-dimensional spatial layout, that is, each speaker is deployed on a two-dimensional plane, the virtual sound source position can be a two-dimensional coordinate in a two-dimensional coordinate system constructed with the reference position (the center point of the figure) as the origin and the direction facing the listener at the reference position as the positive direction of the y-axis. Figure 2b As shown, when the layout of each speaker in the speaker array is a three-dimensional space layout, the virtual sound source position can be a three-dimensional coordinate in a three-dimensional coordinate system constructed with a reference position (the center point of the figure) as the origin and the direction facing the listener at the reference position as the positive direction of the y-axis.

[0101] Step S120: determining two mirror image sound source positions of the virtual sound source position on the deployment boundary of the speaker array according to the reference position and the virtual sound source position.

[0102] In an embodiment of the present application, based on the reference position and the virtual sound source position, the virtual sound source position is mapped to the deployment boundary of the speaker array to obtain the mirror sound source positions of the two mirror objects corresponding to the virtual sound source position.

[0103] In an optional implementation, a straight line between the reference position and the virtual sound source position is determined based on the reference position and the virtual sound source position, and two intersection points of the straight line and the speaker array deployment boundary are respectively used as two mirror image sound source positions corresponding to the virtual sound source position, wherein the direction of one mirror image sound source is the same as the direction of the virtual sound source, and the direction of one mirror image sound source is opposite to the direction of the virtual sound source.

[0104] In an optional implementation, the distance between each speaker and the reference position in the audio data processing system can be the same. Then, when the multiple speakers are arranged in a two-dimensional space, the circumscribed circle (ring) of the multiple speakers is used as the deployment boundary of the speaker array; when the multiple speakers are arranged in a three-dimensional space, the spherical surface of the circumscribed sphere (body) of the multiple speakers is used as the deployment boundary of the speaker array.

[0105] For example, Figure 3 As shown, Figure 3 The 8 speakers - A speaker array is deployed around the listener position O, where each speaker in the speaker array is deployed on the same two-dimensional plane, and the distance between each speaker and the listener position is equal to R. The deployment boundary formed by the speaker array is a circle with O as the center and R as the radius. The virtual sound source position to be rendered is b, and the distance d between the virtual sound source position b and the listener position is less than R. According to the listener position O and the virtual sound source position b, the two intersection points of the straight line Ob and the circle O are used as the two mirror sound source positions corresponding to the virtual sound source position b.

[0106] Step S130: determining a first audio gain weight corresponding to each image sound source position according to the virtual sound source position and the two image sound source positions.

[0107] In the embodiment of the present application, the virtual sound source is split into two mirror sound sources on the deployment boundary of the speaker array, and the virtual sound source is synthesized by the two mirror sound sources. By adjusting the gain of the mirror sound source, the sound energy of the two mirror sound sources forms a vector / vector pointing from the reference position to the virtual sound source position in space. The contribution of each mirror sound source to the playback of the virtual sound source is the corresponding first audio gain weight.

[0108] In an optional implementation, based on the virtual sound source position and the reference position, a first vector pointing from the reference position to the virtual sound source position, that is, the direction vector of the virtual sound source, is determined; for each mirror sound source position, a second vector pointing from the reference position to the mirror sound source position, that is, the direction vector of the mirror sound source, is determined; based on the first vector, the second vector corresponding to each mirror sound source position, and the first condition, a first audio gain weight corresponding to each mirror sound source position is determined.

[0109] The first condition is a constraint condition satisfied by the first audio gain weights corresponding to the two mirror sound source positions. The adjusted vectors corresponding to the two mirror sound source positions and the first vector also need to satisfy the second condition, and the adjusted vector corresponding to the mirror sound source position is obtained by adjusting the second vector corresponding to the mirror sound source position based on the first audio gain weight corresponding to the mirror sound source position.

[0110] Optionally, the two mirror image sound source positions include a first mirror image sound source position and a second mirror image sound source position. In order to ensure that the sum of the sound energy of the mirror image sound source is equal to the sound energy of the virtual sound source, it is necessary to meet the following conditions:

[0111]

[0112] Where b represents the position of the virtual sound source, represents the position of the first image sound source, Indicates the position of the second image sound source, Represents the sound energy corresponding to the virtual sound source, Represents the sound energy corresponding to the first image sound source, Indicates the sound energy corresponding to the second image sound source.

[0113] Furthermore, since the sound energy is proportional to the square of the sound amplitude, the amplitude of the image sound source and the amplitude of the virtual sound source satisfy:

[0114]

[0115] Where A represents the amplitude of the virtual sound source, represents the amplitude of the first image sound source, Indicates the amplitude of the second image sound source.

[0116] by express ,by express ,but .in, Indicates the contribution / playback weight of the first image sound source to the virtual sound source; Indicates the contribution / playback weight of the second image sound source to the virtual sound source.

[0117] Therefore, the constraints satisfied by the first audio gain weights corresponding to the two image sound source positions include:

[0118]

[0119] in, is the first image source position a corresponding first audio gain weight; The second image source position The corresponding first audio gain weight.

[0120] In addition, in order for the sound energy of the two mirror sound sources to form a vector / vector pointing from the reference position to the virtual sound source position in space, the adjusted vectors corresponding to the positions of the two mirror sound sources and the first vector also need to meet the second condition:

[0121]

[0122] Where O represents the reference position, b represents the virtual sound source position, represents the position of the first image sound source, Indicates the position of the second image sound source.

[0123] For example, Figure 4a Schematic diagram of virtual sound source positioning in two-dimensional space. Figure 4b Schematic diagram of virtual sound source positioning in three-dimensional space. The first vector representing the reference position O pointing to the virtual sound source position b, i.e., the direction vector of the virtual sound source, Indicates that the reference position O points to the first image sound source position The second vector, that is, the direction vector of the first image sound source, represents the adjusted vector corresponding to the first image source position, Indicates that the reference position O points to the second image sound source position The second vector, that is, the direction vector of the second image sound source, Represents the adjusted vector corresponding to the position of the second image sound source. The positional relationship between the virtual sound source and the first image sound source and the second image sound source satisfies:

[0124]

[0125]

[0126] Step S140: for each image sound source position, at least one target speaker associated with the image sound source position is determined from a plurality of speakers according to the image sound source position and the positions of each speaker.

[0127] Step S150: For each mirror sound source position, determine the target audio gain weights of the target speakers associated with the mirror sound source position according to the mirror sound source position, the positions of the target speakers associated with the mirror sound source position, and the first audio gain weight corresponding to the mirror sound source position.

[0128] The mirror sound source is synthesized / created by each target loudspeaker associated with the mirror sound source position, so that the sound energy generated by each associated target loudspeaker forms a vector / vector in space pointing from the reference position to the mirror sound source position. The first audio gain weight corresponding to the mirror sound source position represents the contribution of the mirror sound source to the virtual sound source, and the target audio gain weight of the target loudspeaker represents the contribution of the target loudspeaker to the virtual sound source.

[0129] In an optional implementation, the second distance between the position of each speaker in the speaker array and the position of the image sound source is determined respectively. Then, at least one speaker whose second distance to the position of the image sound source meets a preset condition is selected from each speaker as at least one target speaker associated with the position of the image sound source. The preset condition includes at least one of the following:

[0130] The distance between the position of the loudspeaker and the position of the image sound source is less than or equal to a first threshold;

[0131] According to the order of the distance between the position of each speaker and the position of the image sound source from near to far, a preset number of speakers with the corresponding distance rankings are selected.

[0132] For example, assuming that the speaker array includes 5 speakers, and the distances between speakers 1-5 and the image sound source position are 5m, 4m, 2.5m, 2m, and 1m respectively. If the first threshold is 3, speakers 3, 4, and 5 are used as target speakers associated with the image sound source position. If the preset number is 2, speakers 4 and 5 are used as target speakers associated with the image sound source position.

[0133] In an optional implementation, for each mirror image sound source position, a second vector pointing from the reference position to the mirror image sound source position, i.e., the direction vector of the mirror image sound source, may be first determined; for each target speaker associated with the mirror image sound source position, a third vector pointing from the reference position to the position of the target speaker, i.e., the direction vector of the target speaker, may be determined; based on the second vector, the third vector corresponding to each target speaker, and the third condition, the contribution of each target speaker to the mirror image sound source position is determined. For each target speaker associated with the mirror image sound source position, the product of the contribution of the target speaker to the mirror image sound source position and the first audio gain weight corresponding to the mirror image sound source position is used as the target audio gain weight of the target speaker.

[0134] Among them, the third condition is the constraint condition satisfied by the contribution of each target speaker to the mirror sound source position; the adjusted vector corresponding to each target speaker and the second vector satisfy the fourth condition, and the adjusted vector corresponding to each target speaker is obtained by adjusting the third vector corresponding to the target speaker based on the contribution of the target speaker to the mirror sound source position.

[0135] Optionally, for each image sound source position, in order to ensure that the sum of the sound energies of the target loudspeakers associated with the image sound source position is equal to the sound energy of the image sound source (the sound source corresponding to the image sound source position), it is necessary to satisfy the following conditions:

[0136]

[0137] in, represents any of the two image source positions, represents the sound energy corresponding to the image sound source position, n represents the number of target speakers associated with the image sound source position, Indicates the sound energy corresponding to the nth target loudspeaker associated with the image sound source position.

[0138] Optionally, the number of target loudspeakers associated with different image sound source positions may be the same or different.

[0139] Based on the same principle as above, the contribution of the n target speakers associated with the image sound source position to the image sound source position satisfies the third condition:

[0140]

[0141] in, Indicates the contribution of the nth target loudspeaker associated with the image sound source position to the image sound source position.

[0142] In addition, in order to enable the sound energy of each target loudspeaker associated with the image sound source position to form a vector / vector pointing from the reference position to the image sound source position in space, the adjusted vector corresponding to each target loudspeaker and the second vector satisfy the fourth condition:

[0143]

[0144] Wherein, O represents the reference position, represents any of the two image source positions, , Represents the second vector corresponding to the position of the image sound source, Indicates the position of the nth target speaker associated with the image sound source position, represents the third vector corresponding to the associated n-th target speaker, Represents the adjusted vector corresponding to the associated n-th target speaker.

[0145] In addition, since the image sound source position is usually associated with surrounding target speakers, as an optional method, the number of target speakers associated with a mirror sound source position can be limited, such as limiting n 4.

[0146] Based on the above formula, the contribution of each target speaker associated with the image sound source position to the image sound source position is solved. For each target speaker associated with the image sound source position, the product of the contribution of the target speaker to the image sound source position and the first audio gain weight corresponding to the image sound source position is used as the target audio gain weight of the target speaker. is the first image source position When the first image source position is associated with the target speaker The target audio gain weight is , the target speaker associated with the first image source position The target audio gain weight is , and so on.

[0147] In an optional implementation of the present application, the VBAP algorithm can be used to locate each mirror sound source position. Specifically, for each mirror sound source position, with the minimum number of target speakers as the goal, at least one target speaker associated with the mirror sound source position is determined from the speaker array according to the mirror sound source position and the positions of each speaker. Based on the second vector pointing from the reference position to the mirror sound source position, the third vector corresponding to each target speaker, and the third condition, the contribution of each target speaker to the mirror sound source position is determined.

[0148] Optionally, when the VBAP algorithm is used to select at least one target speaker associated with each mirror sound source position, if the layout of the speakers in the speaker array is a two-dimensional spatial layout, the two speakers closest to the mirror sound source position are used as the target speakers associated with the mirror sound source; if the layout of the speakers in the speaker array is a three-dimensional spatial layout, the three speakers closest to the mirror sound source position are used as the target speakers associated with the mirror sound source.

[0149] Furthermore, when the layout of each speaker in the speaker array is a three-dimensional spatial layout, when selecting at least one target speaker associated with the image sound source position, any three adjacent and non-collinear speakers can be selected from the speaker array to obtain multiple speaker triangle groups, wherein the triangular areas formed by different speaker triangle groups do not overlap. Afterwards, according to the direction vector of the image sound source and the position of each speaker in each speaker triangle group, the target speaker triangle group associated with the image sound source is determined, and each speaker in the target speaker triangle group is used as the target speaker associated with the image sound source.

[0150] When constructing a speaker triangle group, the directional angles between the three speakers that make up the speaker triangle group must be smaller than a preset angle to avoid excessive spacing between speakers in the same group.

[0151] Optionally, after obtaining multiple speaker triangle groups, a greedy algorithm may be used to traverse all speaker triangle groups and select a speaker triangle group whose target audio gain is a non-negative value as the target speaker triangle group associated with the image sound source.

[0152] In one implementation, if the layout of the speakers in the speaker array is a two-dimensional spatial layout, such as Figure 5a As shown, for the first image source position , the first image source position Linking two target speakers and , i.e. the target speaker and The first image sound source can be synthesized, then , , satisfy , and the third condition is satisfied . So the target speaker can be solved For the first image source position The contribution of , target speaker The target audio gain weight is ; Target speaker For the first image source position The contribution of , target speaker The target audio gain weight is .

[0153] Similarly, for the second image source position , the second image sound source position Linking two target speakers and , i.e. the target speaker , The second image sound source can be synthesized, then , , satisfy , and the third condition is satisfied . So the target speaker can be solved For the second image source position The contribution of , target speaker The target audio gain weight is ; Target speaker For the second image source position The contribution of , target speaker The target audio gain weight is .

[0154] In one implementation, if the layout of the speakers in the speaker array is a three-dimensional spatial layout, such as Figure 5b As shown, for the first image source position , the first image source position Associate three target speakers , and , i.e. the target speaker , and The first image sound source can be synthesized, then , , , satisfy , and the third condition is satisfied . So the target speaker can be solved For the first image source position The contribution of , target speaker The target audio gain weight is ; Target speaker For the first image source position The contribution of , target speaker The target audio gain weight is , and the target speaker For the first image source position The contribution of , target speaker The target audio gain weight is .

[0155] Similarly, for the second image source position , the second image sound source position Associate three target speakers , and , i.e. the target speaker , and The second image sound source can be synthesized, then , , , satisfy , and the third condition is satisfied . So the target speaker can be solved For the second image source position The contribution of , target speaker The target audio gain weight is ; Target speaker For the second image source position The contribution of , target speaker The target audio gain weight is ; and the target speaker For the second image source position The contribution of , target speaker The target audio gain weight is .

[0156] In another optional implementation of the present application, the amplitude allocated to each target speaker is inversely proportional to the distance from the image sound source position to the target speaker, that is, the contribution of each target speaker to the image sound source position is inversely proportional to the distance between the target speaker and the image sound source position. This can be expressed by the following formula:

[0157]

[0158] in, represents the attenuation coefficient, Indicates the contribution of the nth target speaker associated with the image sound source position to the image sound source position, Indicates the distance between the image sound source position and the position of the associated nth target speaker.

[0159] Therefore, the distance between the image sound source position and each target speaker can be determined according to the image sound source position and the positions of the target speakers associated with the image sound source position. The contribution of each target speaker to the image sound source position is determined according to the distance between the image sound source position and each target speaker and the third condition. For each target speaker, the product of the contribution of the target speaker to the image sound source position and the first audio gain weight corresponding to the image sound source position is used as the target audio gain weight of the target speaker.

[0160] Step S160: For each target speaker, determine the target audio gain of the target speaker according to the target audio gain weight and the audio gain of the target speaker, and obtain the target audio data to be played by the target speaker according to the target audio gain and the audio data of the target speaker.

[0161] The target audio gain of the target speaker indicates the playback volume / loudness of the target speaker. For example, assuming that the audio gain in the metadata of the audio file is A, for the above Figure 5a Target speaker in , then the target speaker The target audio gain is .

[0162] In the embodiment of the present application, for each target speaker, the target audio gain of the target speaker is multiplied by the audio data in the metadata to obtain the target audio data of the target speaker, which can then be sent to the corresponding target speaker for playback.

[0163] based on Figure 1bThe audio data processing method shown in the invention maps the virtual sound source to two mirror sound sources on the deployment boundary of the speaker array through the MOAP algorithm, distributes the audio gain of the virtual sound source to each mirror sound source, and then distributes the audio gain of each mirror sound source to the target speaker in the speaker array. This solves the problem that the VBAP algorithm can only locate the position of the virtual sound source on the deployment boundary of the speaker array but cannot locate the virtual sound source inside the speaker array, and can accurately reproduce the sound and image inside the speaker array.

[0164] Optionally, in the above step S110, the acquired audio file may also contain audio data of multiple different audio objects. For each audio object, the audio data processing method shown in the above steps S120 to S160 may be used to obtain the target audio data played by each target speaker corresponding to the audio object. Finally, the target audio data of each audio object on the target speaker is superimposed and played back on each corresponding target speaker.

[0165] The audio data processing method provided in the embodiment of the present application can realize the positioning of a virtual sound source inside a speaker array or a virtual sound source at the deployment boundary of a speaker array.

[0166] In an optional implementation of the present application, after determining the two mirror sound source positions corresponding to the virtual sound source position, a multi-directional amplitude panning algorithm (MDAP) can be further used to realize the positioning of each mirror sound source position, that is, one mirror sound source position is divided into sub-mirror sound source positions in multiple directions, and then each sub-mirror sound source is respectively assigned to each target speaker.

[0167] Specifically, after obtaining two image sound source positions, for each image sound source position, the image sound source position is divided into a plurality of sub-image sound source positions based on a preset angle. Afterwards, each sub-image sound source position is regarded as an "image sound source position", and the target audio gain weight of each target speaker associated with each sub-image sound source position is determined. This specification will not be repeated here, and reference may be made to the contents of the above steps S130 to S150.

[0168] The MDAP algorithm achieves directional diffusion of the image sound source by panning the image sound source to multiple directions, keeping it roughly constant in different panning directions, thereby providing a more consistent auditory experience. In addition, the MDAP algorithm dynamically adjusts the number and gain of activated speakers as the virtual sound source moves, achieving a smoother and more natural sound source movement effect.

[0169] For example, Figure 6 As shown, assuming that the first image source position (The gray circle in the figure) is located at the speaker Position, then the first image source position The associated target speaker is the speaker , the first image source position Split into the first sub-image sound source position in the right direction with an angle of 20° And the second sub-image sound source position at a left angle of 20° After that, the first sub-image source position The associated target speaker is the speaker , , the second sub-image source position The associated target speaker is the speaker , , realizing the first image sound source position The directional diffusion of ,and .

[0170] The audio data processing method provided in the embodiment of the present application can be applied not only to audio playback scenarios with multiple speakers, such as home theaters, car audio and video, large movie theaters, etc., but also to scenarios with external speakers using headphones and mobile terminal devices.

[0171] like Figure 7 As shown, due to the limited number of speakers of devices such as headphones and mobile terminal devices, when using the MOAP algorithm for audio rendering, it is necessary to create a virtual speaker array as an intermediary, first use the MOAP algorithm to distribute the three-dimensional sound audio to be rendered to each virtual speaker in the virtual speaker array, and then distribute the gain of each virtual speaker to the real speaker of the headphones, mobile terminal devices, etc. Among them, the creation and distribution of the virtual speaker array can be referred to the existing solution, and this application does not improve this.

[0172] In order to facilitate a better understanding and explanation of the method provided in the embodiment of the present application, the optional implementation mode of the method provided in the present application is introduced below in conjunction with a specific embodiment of audio playback under the spatial layout of a two-dimensional speaker array.

[0173] Figure 8 The flowchart of the audio playback provided in the embodiment of the present application is executed by the audio processing device, and specifically includes steps S210-S250. Among them:

[0174] S210: Acquire audio data and metadata of the audio object to be played back.

[0175] The metadata includes a virtual sound source position b and an audio gain A of the audio object.

[0176] S220: Calculate the direction vector of the audio object and the direction vector of the mirror object.

[0177] The mirror object refers to the sound emitted by the mirror sound source position.

[0178] like Figure 4a As shown, the virtual sound source position of the audio object is b, the base position of the listener is O, and the two-dimensional speaker array is 8 speakers - The distance R between each speaker and the reference position O (radius of the speaker array) is 1. - The virtual sound source position b is located. The distance between the virtual sound source position b and the reference position O is less than the speaker array radius R, and the direction vector of the virtual sound source position b is .

[0179] According to the line connecting the virtual sound source position b and the reference position O, two mirror image sound source positions are constructed on the two-dimensional ring formed by the speaker array, namely the first mirror image sound source position and the second image source position , and determine the position of the first image sound source Direction vector , the second image source position Direction vector ,in, and The same direction, , and in the opposite direction, .

[0180] S230: Calculate speaker gain.

[0181] Based on the virtual sound source position b, the first image sound source position is and the second image source position Assign weights so that the first image source position and the second image source position The virtual sound source position b is obtained by synthesis.

[0182] Assume b (0.3, 0.4), then = (0.6, 0.8), = (-0.6, -0.8), the virtual sound source positions b are respectively at the first mirror image sound source position The playback weight is , at the second image source position The playback weight is , then it satisfies:

[0183]

[0184] In addition, in order to ensure that the volume of the image sound source is consistent with the virtual sound source, the weight of the image sound source needs to meet the following requirements:

[0185]

[0186] The solution is: , .

[0187] Afterwards, the VBAP algorithm is used to and Constructing the first image source position The corresponding sound source, through the speaker , Constructing the second image source position The corresponding sound source, such as Figure 5a shown.

[0188] Continue according to

[0189]

[0190]

[0191] Solve for the first image source position Assign the sound source to the speakers Play weight , assigned to the speaker Play weight .

[0192] according to

[0193]

[0194]

[0195] Solve for the second image source position Assign the sound source to the speakers Play weight , Assign to Speaker Play weight .

[0196] Then the sound source at the virtual sound source position b is at the speaker The playback weight is ,speaker The speaker gain is ;

[0197] The sound source at virtual sound source position b is located at the speaker The playback weight is ,speaker The speaker gain is ;

[0198] The sound source at virtual sound source position b is located at the speaker The playback weight is ,speaker The speaker gain is ;

[0199] The sound source at virtual sound source position b is located at the speaker The playback weight is ,speaker The speaker gain is .

[0200] S240: Determine target audio data of the speaker according to the audio data and the speaker gain.

[0201] For each activated speaker, the speaker gain of the speaker is multiplied by the audio data in the audio file to obtain the target audio data that the speaker needs to play.

[0202] S250: Speaker playback.

[0203] For each speaker that is activated, playback is performed according to the received target audio data.

[0204] There is also a method of audio rendering in the existing solution that uses a distance-based amplitude panning algorithm (DBAP). The DBAP algorithm needs to activate all speakers in the speaker array and determine the contribution of each speaker according to the distance between the virtual sound source position and each speaker. Since this algorithm needs to activate all speakers, it is impossible to accurately locate the virtual sound source, and the calculation complexity is high, and the perceived width of the sound and image is also increased. The audio data processing method provided in the embodiment of the present application only activates the target speaker associated with the mirror sound source position, reduces the calculation complexity, and avoids an excessive increase in the perceived sound source width.

[0205] Fig. 9 A schematic diagram of the structure of another audio data processing system provided in an embodiment of the present application, the audio data processing system comprises: an audio processing device 20 and a speaker array 21 deployed around a reference position (not marked in the figure), the speaker array 21 comprising a plurality of speakers. The audio processing device 20 can send target audio data to the corresponding speaker to control the playback of the speaker.

[0206] The audio processing device 20 can render the audio data to be played and send it to the corresponding speaker for playing. Specifically, an audio file to be played can be obtained, wherein the audio file includes audio data and metadata of the audio data, and the metadata includes a virtual sound source position and an audio gain corresponding to the audio data. Afterwards, two mirror sound source positions of the virtual sound source position on the deployment boundary of the speaker array are determined according to the reference position and the virtual sound source position; and the first audio gain weight corresponding to each mirror sound source position is determined according to the virtual sound source position and the two mirror sound source positions. For each mirror sound source position, at least one target speaker associated with the mirror sound source position is determined from a plurality of speakers according to the mirror sound source position and the positions of each speaker. For each mirror sound source position, the target audio gain weight of each target speaker associated with the mirror sound source position is determined according to the mirror sound source position, the positions of each target speaker associated with the mirror sound source position, and the first audio gain weight corresponding to the mirror sound source position. For each target speaker, the target audio gain of the target speaker is determined according to the target audio gain weight and audio gain of the target speaker; the target audio data to be played by the target speaker is obtained according to the target audio gain of the target speaker and the audio data, and sent to the corresponding target speaker.

[0207] The specific rendering method of the audio data may refer to the detailed contents of the above steps S110 - S160 , which will not be described in detail in this specification.

[0208] The target speaker in the speaker array 21 plays the target audio data when receiving the target audio data.

[0209] Based on the same principle as the audio data processing method provided in the embodiment of the present application, the embodiment of the present application provides an audio data processing device, which is deployed in an audio processing device in an audio data processing system. The audio data processing system includes an audio processing device and a speaker array deployed around a reference position, and the speaker array includes multiple speakers. Fig.10 As shown, the audio data processing device 300 may include: an acquisition module 310, an image sound source localization module 320, a first weight determination module 330, a target speaker determination module 340, a second weight determination module 350 and a target audio data determination module 360.

[0210] in:

[0211] An acquisition module 310 is used to acquire an audio file; the audio file includes audio data and metadata of the audio data, and the metadata includes a virtual sound source position and an audio gain corresponding to the audio data;

[0212] An image sound source localization module 320 is used to determine two image sound source positions of the virtual sound source position on the deployment boundary of the speaker array according to the reference position and the virtual sound source position;

[0213] A first weight determination module 330, configured to determine a first audio gain weight corresponding to each image sound source position according to the virtual sound source position and the two image sound source positions;

[0214] A target speaker determination module 340 is used to determine, for each of the image sound source positions, at least one target speaker associated with the image sound source position from the plurality of speakers according to the image sound source position and the positions of the speakers;

[0215] A second weight determination module 350 is used to determine, for each of the image sound source positions, a target audio gain weight of each target speaker associated with the image sound source position according to the image sound source position, the positions of each target speaker associated with the image sound source position, and the first audio gain weight corresponding to the image sound source position;

[0216] The target audio data determination module 360 ​​is used to determine the target audio gain of each target speaker according to the target audio gain weight of the target speaker and the audio gain; and obtain the target audio data to be played by the target speaker according to the target audio gain of the target speaker and the audio data.

[0217] Optionally, the first weight determination module 330 may be used to:

[0218] Determine a first vector pointing from the reference position to the virtual sound source position;

[0219] For each image sound source position, determining a second vector pointing from the reference position to the image sound source position;

[0220] Determine a first audio gain weight corresponding to each of the image sound source positions according to the first vector, the second vector corresponding to each of the image sound source positions, and the first condition;

[0221] Among them, the first condition is a constraint condition satisfied by the first audio gain weights corresponding to the two mirror sound source positions; the adjusted vectors corresponding to the two mirror sound source positions and the first vector satisfy the second condition, and the adjusted vector corresponding to each of the mirror sound source positions is obtained by adjusting the second vector corresponding to the mirror sound source position based on the first audio gain weight corresponding to the mirror sound source position.

[0222] Optionally, the two mirror image sound source positions include a first mirror image sound source position and a second mirror image sound source position;

[0223] The first condition includes:

[0224] ;in, is a first audio gain weight corresponding to the first image sound source position; is the first audio gain weight corresponding to the second image sound source position;

[0225] The second condition includes:

[0226] ; Wherein, O represents the reference position, b represents the virtual sound source position, represents the first vector, represents the position of the first image sound source, Represents the second vector corresponding to the position of the first image sound source, represents the adjusted vector corresponding to the first image source position, Indicates the position of the second image sound source, Represents the second vector corresponding to the position of the second image sound source, Represents the adjusted vector corresponding to the position of the second image sound source.

[0227] Optionally, the second weight determination module 350 may be used to:

[0228] Determine a second vector pointing from the reference position to the mirror image sound source position;

[0229] For each target speaker associated with the image sound source position, determining a third vector pointing from the reference position to the position of the target speaker;

[0230] Determine the contribution of each target speaker to the image sound source position according to the second vector, the third vector corresponding to each target speaker, and the third condition; wherein the third condition is a constraint condition satisfied by the contribution of each target speaker to the image sound source position; the adjusted vector corresponding to each target speaker and the second vector satisfy the fourth condition, and the adjusted vector corresponding to each target speaker is obtained by adjusting the third vector corresponding to the target speaker based on the contribution of the target speaker to the image sound source position;

[0231] For each target speaker associated with the image sound source position, the product of the contribution of the target speaker to the image sound source position and the first audio gain weight corresponding to the image sound source position is used as the target audio gain weight of the target speaker.

[0232] Optionally, the image source position is associated with n target speakers, n 1, and n is an integer;

[0233] The third condition includes:

[0234] ;

[0235] in, Indicates the contribution of the nth target loudspeaker associated with the image sound source position to the image sound source position;

[0236] The fourth condition includes:

[0237] ;

[0238] Wherein, O represents the reference position, Indicates the position of the image sound source, Represents the second vector corresponding to the position of the image sound source, Indicates the position of the nth target speaker associated with the image sound source position, represents the third vector corresponding to the associated n-th target speaker, Represents the adjusted vector corresponding to the associated n-th target speaker.

[0239] Optionally, the target speaker determination module 340 may be used to:

[0240] Determine a second distance between the position of each loudspeaker and the position of the image sound source;

[0241] Selecting at least one speaker whose second distance from the image sound source position meets a preset condition from among the speakers as at least one target speaker associated with the image sound source position;

[0242] The preset condition includes at least one of the following:

[0243] The distance between the position of the loudspeaker and the position of the image sound source is less than or equal to a first threshold;

[0244] According to the order of the distance between the position of each speaker and the position of the image sound source from near to far, a preset number of speakers with the corresponding distance rankings are selected.

[0245] Optionally, the image sound source localization module 320 may be used for:

[0246] Determine a straight line between the reference position and the virtual sound source position;

[0247] The two intersection points of the straight line and the deployment boundary are respectively used as two mirror image sound source positions corresponding to the virtual sound source position.

[0248] Optionally, the virtual sound source position is located inside the speaker array or on a deployment boundary.

[0249] Optionally, the deployment boundary of the speaker array includes:

[0250] In the case where the plurality of speakers in the speaker array are arranged in a two-dimensional space, the deployment boundary of the speaker array is a circumscribed circle of the plurality of speakers;

[0251] When the plurality of speakers in the speaker array are arranged in a three-dimensional space, the deployment boundary of the speaker array is a spherical surface of a circumscribed sphere of the plurality of speakers.

[0252] In another embodiment of the present application, the audio data processing device is deployed in an audio processing device in an audio data processing system, and the system includes an audio processing device and a speaker array deployed around a reference position. Fig.11 As shown, the audio data processing device 400 may include: a file reading unit 410, a metadata acquisition unit 420, an audio data acquisition unit 430, an audio image and its mirror image calculation unit 440, a (speaker's) target audio gain calculation unit 450, and a (speaker's) target audio data generation unit 460.

[0253] in:

[0254] The file reading unit 410 is used to read the audio file, parse the metadata and the audio data, and send the metadata to the metadata acquisition unit and the audio data to the audio data acquisition unit respectively;

[0255] The metadata acquisition unit 420 is used to parse the metadata, obtain the virtual sound source position and audio gain of the audio object, and send the parsed virtual sound source position and audio gain to the sound image and its mirror image calculation unit;

[0256] The audio data acquisition unit 430 is used to acquire audio data and send the audio data to the target audio data generation unit;

[0257] The sound image and its mirror image calculation unit 440 is used to determine the mirror sound source positions of the two mirror sound sources of the audio object on the deployment boundary of the speaker array according to the virtual sound source position of the audio object; and based on the reference position, respectively determine the first vector pointing from the reference position to the virtual sound source position, and the second vector pointing from the reference position to each mirror sound source position; and send the first vector, each second vector and the audio gain to the target audio gain calculation unit;

[0258] The target audio gain calculation unit 450 is used to select a target speaker grid according to each image sound source position and the position of each speaker, wherein the target speaker grid includes at least one target speaker (i.e., an activated speaker) associated with each image sound source position; determine a first audio gain weight corresponding to each image sound source position according to a first vector and a second vector corresponding to each image sound source position; determine a target audio gain weight of each target speaker according to the second vector corresponding to each image sound source position and a third vector pointing from a reference position to each target speaker; determine a speaker gain (i.e., a target audio gain) of each target speaker according to the target audio gain weight of each target speaker and the audio gain in the metadata;

[0259] The target audio data generating unit 460 is used to multiply the speaker gain of each target speaker by the audio data of the audio object to obtain the target audio data to be played by the target speaker, and send it to the corresponding target speaker for playing.

[0260] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0261] An embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory. When the processor executes the computer program stored in the memory, the method in any optional embodiment of the present application can be implemented.

[0262] Fig.12 FIG. 1 is a schematic diagram showing a structure of an electronic device to which an embodiment of the present invention is applicable. Fig.12 As shown, the electronic device may be a server or a user terminal, and the electronic device may be used to implement the method provided in any embodiment of the present invention.

[0263] like Fig.12 As shown in FIG. , the electronic device 2000 may mainly include at least one processor 2001 ( Fig.12 ), memory 2002, communication module 2003 and input / output interface 2004 and other components. Optionally, the components can be connected and communicated through bus 2005. It should be noted that Fig.12The structure of the electronic device 2000 shown in the figure is merely illustrative and does not constitute a limitation on the electronic device to which the method provided in the embodiment of the present application is applicable.

[0264] The memory 2002 may be used to store an operating system and an application program, etc. The application program may include a computer program that implements the method shown in the embodiment of the present invention when called by the processor 2001, and may also include a program for implementing other functions or services. The memory 2002 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and computer programs, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compact optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0265] The processor 2001 is connected to the memory 2002 via the bus 2005, and implements corresponding functions by calling the application program stored in the memory 2002. The processor 2001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof, which may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present invention. The processor 2001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0266] The electronic device 2000 can be connected to the network through the communication module 2003 (which may include but is not limited to components such as a network interface) to communicate with other devices (such as a user terminal or a server, etc.) through the network to achieve data interaction, such as sending data to other devices or receiving data from other devices. Among them, the communication module 2003 may include a wired network interface and / or a wireless network interface, etc., that is, the communication module may include at least one of a wired communication module or a wireless communication module.

[0267] The electronic device 2000 can be connected to the required input / output devices, such as a keyboard, a display device, etc., through the input / output interface 2004. The electronic device 2000 itself can have a display device, and can also be connected to other display devices through the interface 2004. Optionally, a storage device, such as a hard disk, can also be connected through the interface 2004, so that data in the electronic device 2000 can be stored in the storage device, or data in the storage device can be read, and data in the storage device can also be stored in the memory 2002. It can be understood that the input / output interface 2004 can be a wired interface or a wireless interface. Depending on the actual application scenario, the device connected to the input / output interface 2004 can be a component of the electronic device 2000, or it can be an external device connected to the electronic device 2000 when needed.

[0268] The bus 2005 for connecting the components may include a path for transmitting information between the above components. The bus 2005 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. According to different functions, the bus 2005 may be divided into an address bus, a data bus, a control bus, etc.

[0269] Optionally, for the solution provided by the embodiment of the present invention, the memory 2002 can be used to store a computer program for executing the solution of the present invention, and run by the processor 2001. When the processor 2001 runs the computer program, the action of the method or device provided by the embodiment of the present invention is implemented.

[0270] Based on the same principle as the method provided in the embodiment of the present application, the embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the corresponding content of the aforementioned method embodiment can be implemented.

[0271] An embodiment of the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the corresponding content of the aforementioned method embodiment can be implemented.

[0272] It should be noted that the terms "first", "second", "third", "fourth", "1", "2", etc. (if any) in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the application described herein can be implemented in an order other than that shown in the figure or described in the text.

[0273] It should be understood that, although each operation step is indicated by arrows in the flowchart of the embodiment of the present application, the implementation order of these steps is not limited to the order indicated by the arrows. Unless clearly stated herein, in some implementation scenarios of the embodiment of the present application, the implementation steps in each flowchart can be performed in other orders according to demand. In addition, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on actual implementation scenarios. Some or all of these sub-steps or stages may be executed at the same time, and each sub-step or stage in these sub-steps or stages may also be executed at different times respectively. In different scenarios of execution time, the execution order of these sub-steps or stages may be flexibly configured according to demand, and the embodiment of the present application does not limit this.

[0274] The above is only an optional implementation method for some implementation scenarios of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the solution of the present application, other similar implementation methods based on the technical ideas of the present application are also within the protection scope of the embodiments of the present application.

Claims

1. An audio data processing method, characterized in that: The method is applied to an audio data processing system, the system comprising an audio processing device and a speaker array deployed around a reference position, the speaker array comprising a plurality of speakers; the method is executed by the audio processing device, and comprises: Acquire an audio file; the audio file includes audio data and metadata of the audio data, wherein the metadata includes a virtual sound source position and an audio gain corresponding to the audio data; Determine, according to the reference position and the virtual sound source position, two mirror image sound source positions of the virtual sound source position on the deployment boundary of the speaker array; Determine, according to the virtual sound source position and the two image sound source positions, a first audio gain weight corresponding to each image sound source position; For each of the image sound source positions, determining at least one target speaker associated with the image sound source position from the plurality of speakers according to the image sound source position and the positions of the speakers; For each of the image sound source positions, determining the target audio gain weights of the target speakers associated with the image sound source position according to the image sound source position, the positions of the target speakers associated with the image sound source position, and the first audio gain weight corresponding to the image sound source position; For each target speaker, the target audio gain of the target speaker is determined according to the target audio gain weight of the target speaker and the audio gain; and the target audio data to be played by the target speaker is obtained according to the target audio gain of the target speaker and the audio data.

2. The method according to claim 1, characterized in that: The two image sound source positions include a first image sound source position and a second image sound source position; The step of determining a first audio gain weight corresponding to each image sound source position according to the virtual sound source position and the two image sound source positions includes: Determine a first vector pointing from the reference position to the virtual sound source position; For each image sound source position, determining a second vector pointing from the reference position to the image sound source position; Determine a first audio gain weight corresponding to each of the image sound source positions according to the first vector, the second vector corresponding to each of the image sound source positions, and the first condition; The first condition is a constraint condition satisfied by the first audio gain weights corresponding to the two image sound source positions; the adjusted vectors corresponding to the two image sound source positions and the first vector satisfy the second condition, and the adjusted vector corresponding to each of the image sound source positions is obtained by adjusting the second vector corresponding to the image sound source position based on the first audio gain weight corresponding to the image sound source position; the first condition includes: ;in, is a first audio gain weight corresponding to the first image sound source position; is the first audio gain weight corresponding to the second image sound source position; The second condition includes: ; Wherein, O represents the reference position, b represents the virtual sound source position, represents the first vector, represents the position of the first image sound source, Represents the second vector corresponding to the position of the first image sound source, represents the adjusted vector corresponding to the first image source position, Indicates the position of the second image sound source, Represents the second vector corresponding to the position of the second image sound source, Represents the adjusted vector corresponding to the position of the second image sound source.

3. The method according to claim 1, characterized in that For each of the image sound source positions, determining the target audio gain weights of the target speakers associated with the image sound source position according to the image sound source position, the positions of the target speakers associated with the image sound source position, and the first audio gain weight corresponding to the image sound source position, includes: Determine a second vector pointing from the reference position to the mirror image sound source position; For each target speaker associated with the image sound source position, determining a third vector pointing from the reference position to the position of the target speaker; Determine the contribution of each target speaker to the image sound source position according to the second vector, the third vector corresponding to each target speaker, and the third condition; wherein the third condition is a constraint condition satisfied by the contribution of each target speaker to the image sound source position; the adjusted vector corresponding to each target speaker and the second vector satisfy the fourth condition, and the adjusted vector corresponding to each target speaker is obtained by adjusting the third vector corresponding to the target speaker based on the contribution of the target speaker to the image sound source position; For each target speaker associated with the image sound source position, multiplying the contribution of the target speaker to the image sound source position by the first audio gain weight corresponding to the image sound source position as the target audio gain weight of the target speaker; For each of the image sound source positions, the image sound source position is associated with n target speakers, n 1, and n is an integer; The third condition includes: ; in, Indicates the contribution of the nth target loudspeaker associated with the image sound source position to the image sound source position; The fourth condition includes: ; Wherein, O represents the reference position, Indicates the position of the image sound source, Represents the second vector corresponding to the position of the image sound source, Indicates the position of the nth target speaker associated with the image sound source position, represents the third vector corresponding to the associated n-th target speaker, Represents the adjusted vector corresponding to the associated n-th target speaker.

4. The method according to claim 1, characterized in that: For each image sound source position, determining at least one target speaker associated with the image sound source position from the plurality of speakers according to the image sound source position and the positions of each of the speakers, comprises: Determine a second distance between the position of each loudspeaker and the position of the image sound source; Selecting at least one speaker whose second distance from the image sound source position meets a preset condition from among the speakers as at least one target speaker associated with the image sound source position; The preset condition includes at least one of the following: The distance between the position of the loudspeaker and the position of the image sound source is less than or equal to a first threshold; According to the order of the distance between the position of each speaker and the position of the image sound source from near to far, a preset number of speakers with the corresponding distance rankings are selected.

5. The method according to claim 1, characterized in that The step of determining two mirror image sound source positions of the virtual sound source position on a deployment boundary of the speaker array according to the reference position and the virtual sound source position includes: Determine a straight line between the reference position and the virtual sound source position; The two intersection points of the straight line and the deployment boundary are respectively used as two mirror image sound source positions corresponding to the virtual sound source position.

6. The method according to any one of claims 1 to 5, characterized in that: The virtual sound source position is located inside the speaker array or on a deployment boundary.

7. The method according to claim 1, characterized in that The deployment boundary of the speaker array includes: In the case where the plurality of speakers in the speaker array are arranged in a two-dimensional space, the deployment boundary of the speaker array is a circumscribed circle of the plurality of speakers; When the plurality of speakers in the speaker array are arranged in a three-dimensional space, the deployment boundary of the speaker array is a spherical surface of a circumscribed sphere of the plurality of speakers.

8. An audio data processing system, characterized in that: The system includes an audio processing device, and a speaker array disposed around a reference position, the speaker array including a plurality of speakers; in: The audio processing device is used to obtain an audio file; the audio file includes audio data and metadata of the audio data, and the metadata includes a virtual sound source position and an audio gain corresponding to the audio data; according to the reference position and the virtual sound source position, two mirror sound source positions of the virtual sound source position on the deployment boundary of the speaker array are determined; according to the virtual sound source position and the two mirror sound source positions, a first audio gain weight corresponding to each mirror sound source position is determined; for each mirror sound source position, according to the mirror sound source position and the positions of each of the speakers, at least one target speaker associated with the mirror sound source position is determined from the multiple speakers; according to the mirror sound source position, the positions of each target speaker associated with the mirror sound source position, and the first audio gain weight corresponding to the mirror sound source position, a target audio gain weight of each target speaker associated with the mirror sound source position is determined; for each of the target speakers, according to the target audio gain weight of the target speaker and the audio gain, a target audio gain of the target speaker is determined; according to the target audio gain of the target speaker and the audio data, the target audio data corresponding to the target speaker is obtained; and the corresponding target audio data is sent to each of the target speakers respectively; The target speaker is used to play the received target audio data.

9. An audio data processing device, characterized in that: The apparatus is deployed in an audio processing device in an audio data processing system, wherein the audio data processing system comprises the audio processing device and a speaker array deployed around a reference position, wherein the speaker array comprises a plurality of speakers; the apparatus comprises: An acquisition module, used to acquire an audio file; the audio file includes audio data and metadata of the audio data, and the metadata includes a virtual sound source position and an audio gain corresponding to the audio data; An image sound source localization module, used to determine two image sound source positions of the virtual sound source position on the deployment boundary of the speaker array according to the reference position and the virtual sound source position; A first weight determination module, used to determine a first audio gain weight corresponding to each image sound source position according to the virtual sound source position and the two image sound source positions; a target speaker determination module, configured to determine, for each of the image sound source positions, at least one target speaker associated with the image sound source position from the plurality of speakers according to the image sound source position and the positions of the speakers; A second weight determination module is used to determine, for each of the image sound source positions, a target audio gain weight of each target speaker associated with the image sound source position according to the image sound source position, the positions of each target speaker associated with the image sound source position, and the first audio gain weight corresponding to the image sound source position; The target audio data determination module is used to determine the target audio gain of each target speaker according to the target audio gain weight of the target speaker and the audio gain; and obtain the target audio data to be played by the target speaker according to the target audio gain of the target speaker and the audio data.

10. The device according to claim 9, characterized in that The two image sound source positions include a first image sound source position and a second image sound source position; The first weight determination module is used for: Determine a first vector pointing from the reference position to the virtual sound source position; For each image sound source position, determining a second vector pointing from the reference position to the image sound source position; Determine a first audio gain weight corresponding to each of the image sound source positions according to the first vector, the second vector corresponding to each of the image sound source positions, and the first condition; The first condition is a constraint condition satisfied by the first audio gain weights corresponding to the two image sound source positions; the adjusted vectors corresponding to the two image sound source positions and the first vector satisfy the second condition, and the adjusted vector corresponding to each of the image sound source positions is obtained by adjusting the second vector corresponding to the image sound source position based on the first audio gain weight corresponding to the image sound source position; The first condition includes: ;in, is a first audio gain weight corresponding to the first image sound source position; is the first audio gain weight corresponding to the second image sound source position; The second condition includes: ; Wherein, O represents the reference position, b represents the virtual sound source position, represents the first vector, represents the position of the first image sound source, Represents the second vector corresponding to the position of the first image sound source, represents the adjusted vector corresponding to the first image source position, Indicates the position of the second image sound source, Represents the second vector corresponding to the position of the second image sound source, Represents the adjusted vector corresponding to the position of the second image sound source.

11. The device according to claim 9, characterized in that The second weight determination module is used for: Determine a second vector pointing from the reference position to the mirror image sound source position; For each target speaker associated with the image sound source position, determining a third vector pointing from the reference position to the position of the target speaker; Determine the contribution of each target speaker to the image sound source position according to the second vector, the third vector corresponding to each target speaker, and the third condition; wherein the third condition is a constraint condition satisfied by the contribution of each target speaker to the image sound source position; the adjusted vector corresponding to each target speaker and the second vector satisfy the fourth condition, and the adjusted vector corresponding to each target speaker is obtained by adjusting the third vector corresponding to the target speaker based on the contribution of the target speaker to the image sound source position; For each target speaker associated with the image sound source position, multiplying the contribution of the target speaker to the image sound source position by the first audio gain weight corresponding to the image sound source position as the target audio gain weight of the target speaker; For each of the image sound source positions, the image sound source position is associated with n target loudspeakers, n 1, and n is an integer; The third condition includes: ; in, Indicates the contribution of the nth target loudspeaker associated with the image sound source position to the image sound source position; The fourth condition includes: ; Wherein, O represents the reference position, Indicates the position of the image sound source, Represents the second vector corresponding to the position of the image sound source, Indicates the position of the nth target speaker associated with the image sound source position, represents the third vector corresponding to the associated n-th target speaker, Represents the adjusted vector corresponding to the associated n-th target speaker.

12. The device according to claim 9, characterized in that The target speaker determination module is used for: Determine a second distance between the position of each loudspeaker and the position of the image sound source; Selecting at least one speaker whose second distance from the image sound source position meets a preset condition from among the speakers as at least one target speaker associated with the image sound source position; The preset condition includes at least one of the following: The distance between the position of the loudspeaker and the position of the image sound source is less than or equal to a first threshold; According to the order of the distance between the position of each speaker and the position of the image sound source from near to far, a preset number of speakers with the corresponding distance rankings are selected.

13. The device according to claim 9, characterized in that The image sound source localization module is used for: Determine a straight line between the reference position and the virtual sound source position; The two intersection points of the straight line and the deployment boundary are respectively used as two mirror image sound source positions corresponding to the virtual sound source position.

14. The device according to any one of claims 9 to 13, characterized in that The virtual sound source position is located inside the speaker array or on a deployment boundary.

15. The device according to claim 9, characterized in that The deployment boundary of the loudspeaker array is determined by: In the case where the plurality of speakers in the speaker array are arranged in a two-dimensional space, the deployment boundary of the speaker array is a circumscribed circle of the plurality of speakers; When the plurality of speakers in the speaker array are arranged in a three-dimensional space, the deployment boundary of the speaker array is a spherical surface of a circumscribed sphere of the plurality of speakers.

16. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein a computer program is stored in the memory, and the processor executes the computer program to implement the method according to any one of claims 1 to 7.

17. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

18. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.