Audio data transmission method, device, electronic device and storage medium

Abstract

The present application discloses an audio data transmission method, device, electronic device and storage medium. The audio data transmission method includes: obtaining N channels of audio data to be sent; selecting M channels of audio data from the N channels of audio data; extracting audio feature information of the M channels of audio data respectively; obtaining the psychoacoustic perception coefficient EP(i) corresponding to the M channels of audio data according to the audio feature information; and controlling the multiple transmission of the corresponding audio data according to the psychoacoustic perception coefficient, so that when EP(j)>EP(k), red'(j)>red'(k). The present method can realize the multiple transmission of the corresponding audio data based on the psychoacoustic perception coefficient of the audio data, using a larger multiple transmission multiple for transmitting audio data with a larger psychoacoustic perception coefficient, and using a smaller multiple transmission multiple for transmitting audio data with a smaller psychoacoustic perception coefficient, thereby ensuring the effective transmission of the audio data and improving the transmission quality of the audio data.

Description

Technical Field

[0001] The present application relates to the field of network transmission technology, and more specifically, to an audio data transmission method, device, electronic device and storage medium. Background Art

[0002] In a multi-person interactive voice communication system, the quality of multi-person calls is mainly affected by network packet loss. Packet loss occurs during the transmission process due to the instability of the transmission network, causing the sound at the receiving end to be stuck and incoherent, seriously reducing the listener's experience. Using repeated multiple transmissions, that is, copying multiple copies of the audio data packet and sending them to the receiving end at different time intervals, can effectively avoid the problem of the receiving end not receiving the corresponding audio data packet due to network packet loss.

[0003] In traditional multi-person calls, downlink multiple transmission of audio data packets is achieved based on the actual packet loss rate of the client, that is, the same multiple transmission strategy is adopted for all audio data packets. The multiple transmission strategy requires multiple times the network bandwidth as a cost while solving the network packet loss. However, in some cases where the network bandwidth is limited, multiple transmission causes the required bandwidth to increase several times, which in turn leads to more serious packet loss. Summary of the invention

[0004] In view of the above problems, the present application proposes an audio data transmission method, device, electronic device and storage medium, which can realize the multiple transmission of corresponding audio data based on the psychoacoustic perception coefficient of the audio data, and adopt a larger multiple transmission multiple for audio data with a larger psychoacoustic perception coefficient, and adopt a smaller multiple transmission multiple for audio data with a smaller psychoacoustic perception coefficient, thereby ensuring the effective transmission of audio data and improving the transmission quality of audio data.

[0005] In a first aspect, an embodiment of the present application provides an audio data transmission method, the audio data transmission method comprising: obtaining N channels of audio data to be sent, N being a positive integer greater than or equal to 1; selecting M channels of audio data from the N channels of audio data, wherein M is a positive integer and 1≤M≤N; extracting audio feature information of the M channels of audio data respectively; obtaining a psychoacoustic perception coefficient EP(i) corresponding to the M channels of audio data according to the audio feature information, wherein i represents the sequence number of the audio data and 1≤i≤M; and controlling the multiple transmission of the corresponding audio data according to the psychoacoustic perception coefficient EP(i), so that when EP(j)>EP(k), red'(j)>red'(k), wherein 1≤j≤M, 1≤k≤M, and j≠k, red'(j) and red'(k) respectively represent the multiple transmission times of the jth and kth audio data.

[0006] In a second aspect, an embodiment of the present application provides an audio data transmission device, which includes: a first acquisition module, used to acquire N channels of audio data to be sent, N is a positive integer greater than or equal to 1; a selection module, used to select M channels of audio data from the N channels of audio data, wherein M is a positive integer and 1≤M≤N; an extraction module, used to extract audio feature information of the M channels of audio data respectively; a second acquisition module, used to acquire a psychoacoustic perception coefficient EP(i) corresponding to the M channels of audio data according to the audio feature information, wherein i represents the audio data sequence number and 1≤i≤M; and a psychoacoustic perception control module, used to control the multiple transmission of the corresponding audio data according to the psychoacoustic perception coefficient EP(i), so that when EP(j)>EP(k), red'(j)>red'(k), wherein 1≤j≤M, 1≤k≤M, and j≠k, red'(j) and red'(k) respectively represent the multiple transmission times of the jth and kth audio data.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the audio data transmission method provided in the first aspect is implemented.

[0008] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a program code is stored. The program code can be called by a processor to execute the audio data transmission method provided in the first aspect above.

[0009] The solution provided by the present application obtains N channels of audio data to be sent, then selects M channels of audio data from the N channels of audio data, then extracts audio feature information of the M channels of audio data respectively, and obtains psychoacoustic perception coefficients corresponding to the M channels of audio data according to the audio feature information, and then controls the multiple transmission of the corresponding audio data according to the psychoacoustic perception coefficient. Therefore, the multiple transmission of the corresponding audio data is controlled based on the psychoacoustic perception coefficient of the audio data, and a larger multiple multiple is used for transmission of audio data with a larger psychoacoustic perception coefficient, and a smaller multiple multiple is used for transmission of audio data with a smaller psychoacoustic perception coefficient, thereby ensuring the effective transmission of the audio data and improving the transmission quality of the audio data. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0011] Figure 1 A schematic diagram of an application scenario applicable to an embodiment of the present application is shown;

[0012] Figure 2 A schematic diagram of a flow chart of an audio data transmission method provided by an embodiment of the present application is shown;

[0013] Figure 3 A schematic diagram of an acoustic equal loudness curve diagram provided in an embodiment of the present application is shown;

[0014] Figure 4 A schematic diagram of a perception weighting coefficient calculated based on a psychoacoustic perception weighting coefficient formula provided in an embodiment of the present application is shown;

[0015] Figure 5 A schematic diagram showing a flow chart of an audio data transmission method provided by another embodiment of the present application is shown;

[0016] Figure 6 A schematic diagram of a flow chart of an audio data transmission method provided by another embodiment of the present application is shown;

[0017] Figure 7 A schematic flow chart of an audio data transmission method provided by yet another embodiment of the present application is shown;

[0018] Figure 8 A schematic diagram of the structure of an audio data transmission device provided in an embodiment of the present application is shown;

[0019] Fig. 9 A structural block diagram of an electronic device provided in an embodiment of the present application is shown;

[0020] Fig.10 A storage unit for storing or carrying a program code of an audio data transmission method according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0022] Definition of terms:

[0023] Multiple transmission: When the transmission network is unstable and packet loss occurs, the original data packet is copied multiple times and sent at different time intervals.

[0024] Psychoacoustics: It is a marginal discipline that studies the relationship between sound and the hearing it causes. It is both a branch of acoustics and a branch of psychophysics.

[0025] Psychoacoustic perception coefficient: It represents the ability of sound to be perceived by the human ear. The larger the psychoacoustic perception coefficient, the easier the sound is perceived by the human ear. The smaller the psychoacoustic perception coefficient, the more difficult the sound is perceived by the human ear.

[0026] Packet loss rate: refers to the ratio of the number of lost data packets to the number of sent data groups during the test. The packet loss rate is related to the length of the data packet and the frequency of data packet sending.

[0027] Mel-frequency cepstral coefficient: proposed based on the human auditory system, simulating the human ear's perception of speech of different frequencies.

[0028] When multiple people are talking on the phone, when the server forwards the acquired audio data to the client through the network, packet loss may occur during the transmission process due to the instability of the transmission network, which in turn causes the client to receive audio data in a stuck and discontinuous manner. The repeated multiple transmission method can effectively solve the network packet loss problem.

[0029] In traditional multi-person voice communication systems, all audio data packets are transmitted at the same multiplex multiplex, and the multiplex multiple of audio data packets is determined based on the actual packet loss rate of the client. The higher the packet loss rate, the higher the multiplex multiple, and the higher the multiplex multiple, the more bandwidth consumed. Therefore, in some cases where network bandwidth is limited, multiple transmissions cause the required bandwidth to increase multiples, which in turn leads to more serious packet loss.

[0030] In response to the above problems, the inventors have conducted a long period of research and proposed the audio data transmission method, device, electronic device and storage medium provided in the embodiments of the present application. The multiple transmission of the corresponding audio data is controlled according to the psychoacoustic perception coefficient of the audio data. A larger multiple transmission multiple is used for transmitting audio data with a larger psychoacoustic perception coefficient, and a smaller multiple transmission multiple is used for transmitting audio data with a smaller psychoacoustic perception coefficient, thereby ensuring the effective transmission of the audio data and improving the transmission quality of the audio data.

[0031] The following is an introduction to the application scenarios provided by the embodiments of the present application.

[0032] See also Figure 1, which shows a schematic diagram of an application scenario provided by an embodiment of the present application, and the application scenario includes an audio data transmission system 10. The audio data transmission system 10 includes at least a server 110 and a client 120, wherein the server 110 communicates with the client 120 through a network, and the server 110 can interact with the client 120 for data transmission, thereby realizing audio data transmission. The server 110 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), big data, and artificial intelligence platforms; the client 120 can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., which is not limited here.

[0033] See also Figure 2 , which shows a flow chart of an audio data transmission method provided by an embodiment of the present application. The audio data transmission method is used to control the multiple transmission of corresponding audio data according to the psychoacoustic perception coefficient of the audio data, and adopts a larger multiple transmission multiple to transmit the audio data with a larger psychoacoustic perception coefficient, and adopts a smaller multiple transmission multiple to transmit the audio data with a smaller psychoacoustic perception coefficient, thereby ensuring the effective transmission of the audio data and improving the transmission quality of the audio data. In a specific embodiment, the audio data transmission method is applied to Figure 1 The server in the audio data transmission system shown in FIG. Figure 2 The process shown in the figure is described in detail, and the audio data transmission method may include the following steps:

[0034] Step S210: Acquire N channels of audio data to be sent.

[0035] In an embodiment of the present application, before controlling the audio data to be transmitted in multiple transmissions, the server may obtain N channels of audio data to be transmitted from the client, where N is a positive integer greater than or equal to 1.

[0036] In some implementations, the sounds recorded by N clients through microphones are converted into N channels of audio data via a sound circuit. The N clients respectively send the corresponding audio data to a server, and the server can respectively receive the N channels of audio data sent by the N clients.

[0037] In other embodiments, the sounds recorded by N clients through cameras are converted into N channels of audio data via a sound circuit. The N clients respectively send the corresponding audio data to the server, and the server can respectively receive the N channels of audio data sent by the N clients.

[0038] Step S220: Select M channels of audio data from the N channels of audio data.

[0039] In an embodiment of the present application, after obtaining N channels of audio data to be sent, the server can select M channels of audio data from the N channels of audio data according to the routing algorithm of the audio data, so that the server can transmit the selected M channels of audio data. Wherein, M is a positive integer, and 1≤M≤N. It is understandable that since the human ear has a relatively limited ability to effectively recognize mixed audio data from different sound sources at the same time, under normal circumstances, the human ear can recognize the simultaneous speech of less than 4 people. When the number of people speaking at the same time reaches or exceeds 4 people, it is difficult for the human ear to distinguish the mixed audio data of 4 or more channels. The human ear feels that the audio data is messy and cannot be heard clearly. The upper limit value of the mixed audio data that the human ear can recognize is not limited here.

[0040] In some embodiments, the routing algorithm of the audio data is associated with the energy value and the signal-to-noise ratio of the audio data. The server can select M channels of audio data whose energy value is greater than or equal to a preset energy threshold and whose signal-to-noise ratio is greater than or equal to a preset signal-to-noise ratio threshold from N channels of audio data based on the energy value and the signal-to-noise ratio of the audio data.

[0041] Step S230: extracting audio feature information of M channels of audio data respectively.

[0042] In the embodiment of the present application, after selecting M channels of audio data from N channels of audio data, the server can respectively extract audio feature information of the M channels of audio data, so as to obtain the psychoacoustic perception coefficient EP(i) corresponding to the audio data according to the audio feature information, thereby controlling the multiple transmission of the corresponding audio data according to the psychoacoustic perception coefficient EP(i). Wherein, i represents the sequence number of the audio data, and 1≤i≤M.

[0043] Step S240: Acquire the psychoacoustic perception coefficient EP(i) corresponding to the M channels of audio data according to the audio feature information.

[0044] In an embodiment of the present application, after respectively extracting audio feature information of M channels of audio data, the server can obtain the psychoacoustic perception coefficient EP(i) corresponding to the M channels of audio data based on the extracted audio feature information, so as to control the multiple transmission of the audio data according to the psychoacoustic perception coefficient.

[0045] It can be understood that the psychoacoustic perception coefficient is related to the ease of perception of audio data by the human ear. The larger the psychoacoustic perception coefficient of audio data, the easier it is for the audio data to be perceived by the human ear; the smaller the psychoacoustic perception coefficient of audio data, the more difficult it is for the audio data to be perceived by the human ear.

[0046] The main basis of auditory perception is "loudness". "Loudness" varies with the intensity of the sound, but is also affected by the frequency. That is, sounds of the same intensity but different frequencies have different auditory perceptions for the human ear, such as Figure 3 As shown in the figure, it is a schematic diagram of an acoustic equal loudness curve, which describes the relationship between the sound pressure level and the sound wave frequency under equal loudness conditions. The equal loudness curve indicates the sound pressure level that needs to be achieved for the listener to obtain consistent auditory loudness at different frequencies. To illustrate the meaning of this curve, as Figure 3 For any equal loudness curve, the lower the frequency of the mid-low frequency (below 1kHz), the greater the sound pressure intensity (energy) required for equal loudness, that is, a greater sound energy is required to give the same auditory experience to the human ear; and the equal loudness curve has different acoustic auditory perception characteristics in different frequency bands of the mid-high frequency (above 1kHz).

[0047] In some implementations, the server may calculate the psychoacoustic perception coefficients of the M channels of audio data respectively according to a psychoacoustic perception coefficient formula, where the psychoacoustic perception coefficient formula is: Wherein, i represents the audio data serial number, and 1≤i≤M, EP(i) is the psychoacoustic perception coefficient of the i-th audio data, k is the frequency point serial number, K is the total number of frequency points, p(i, k) is the absolute value of the power of the i-th audio data, and cof(k) is the psychoacoustic perception weighting coefficient of frequency point k.

[0048] The psychoacoustic perception weighting coefficient can be calculated according to the psychoacoustic perception weighting coefficient formula. The psychoacoustic perception weighting coefficient formula is: cof(freq)=(10^loud / 20) / 1000, where freq is the target frequency value, and loud is the loudness value of the target frequency. Figure 4 , which is a schematic diagram of the perceptual weighting coefficient calculated based on the psychoacoustic perceptual weighting coefficient formula.

[0049] The loudness value of the target frequency point can be calculated according to the loudness formula, which is: loud = 4.2 + afy*(dB-cfy) / (1 + bfy*(dB-cfy)), where afy, bfy and cfy are the loudness values ​​in the equal loudness curve data table, and afy, bfy and cfy can be calculated according to the following formulas respectively:

[0050] afy=af(k-1)+(freq-ff(k-1))*(af(k)-af(k-1)) / (ff(k)-(ff(k-1)));

[0051] bfy=bf(k-1)+(freq-ff(k-1))*(bf(k)-bf(k-1)) / (ff(k)-(ff(k-1)));

[0052] cfy=cf(k-1)+(freq-ff(k-1))*(cf(k)-cf(k-1)) / (ff(k)-(ff(k-1)));

[0053] Among them, af, bf, cf and ff are the frequency points in the equal loudness curve data table.

[0054] Step S250: controlling the multiple transmission of corresponding audio data according to the psychoacoustic perception coefficient EP(i).

[0055] In an embodiment of the present application, after the server obtains the psychoacoustic perception coefficient EP(i) corresponding to M channels of audio data based on the extracted audio feature information, it can control the multiple transmission of the corresponding audio data according to the psychoacoustic perception coefficient EP(i), so that when EP(j)>EP(k), red'(j)>red'(k), wherein 1≤j≤M, 1≤k≤M, and j≠k, red'(j) and red'(k) respectively represent the multiple multiples of the j-th and k-th audio data, so that the multiple transmission of the corresponding audio data can be controlled according to the psychoacoustic perception coefficient of the audio data, and the audio data with a larger psychoacoustic perception coefficient is transmitted with a larger multiple multiple, and the audio data with a smaller psychoacoustic perception coefficient is transmitted with a smaller multiple multiple, thereby ensuring the effective transmission of the audio data and improving the transmission quality of the audio data.

[0056] It can be understood that the server can control the multiple transmission of M channels of audio data according to the multiple transmission multiples of each channel of audio data, thereby controlling the transmission of audio data according to the psychoacoustic perception coefficient, so that the received audio data can be perceived by hearing, and the transmission quality of the audio data can be improved. For example, the server obtains the multiple transmission multiples of 5 channels of audio data according to the extracted audio feature information, which are: the multiple transmission multiples of audio data 1 are 8, the multiple transmission multiples of audio data 2 are 5, the multiple transmission multiples of audio data 3 are 2, the multiple transmission multiples of audio data 4 are 3, and the multiple transmission multiples of audio data 5 are 3. Then the server controls audio data 1 to copy 8 copies and send them at different time intervals, controls audio data 2 to copy 5 copies and send them at different time intervals, controls audio data 3 to copy 2 copies and send them at different time intervals, controls audio data 4 to copy 3 copies and send them at different time intervals, and controls audio data 5 to copy 3 copies and send them at different time intervals. The time interval can be 0.1ms, 1ms, or 10ms. The specific value of the time interval is not limited here.

[0057] In some embodiments, the server can determine the multiple transmission multiple of each audio data channel according to the psychoacoustic perception coefficient and the first multiple transmission multiple formula, so as to multiplex the audio data according to the corresponding multiple transmission multiple. The first multiple transmission multiple formula is: red'(i)=f(EP(i))*red_org(i), wherein red'(i) is the multiple transmission multiple of the i-th audio data channel, f(EP(i))∈[0,1], and f(EP(i)) can be monotonically increasing, and red_org(i) is the multiple transmission multiple of the i-th audio data channel calculated based on the packet loss rate.

[0058] The solution provided by the present application obtains N channels of audio data to be sent, then selects M channels of audio data from the N channels of audio data, then extracts audio feature information of the M channels of audio data respectively, and obtains psychoacoustic perception coefficients corresponding to the M channels of audio data according to the audio feature information, and then controls the multiple transmission of the corresponding audio data according to the psychoacoustic perception coefficient. Therefore, the multiple transmission of the corresponding audio data is controlled based on the psychoacoustic perception coefficient of the audio data, and a larger multiple multiple is used for transmission of audio data with a larger psychoacoustic perception coefficient, and a smaller multiple multiple is used for transmission of audio data with a smaller psychoacoustic perception coefficient, thereby ensuring the effective transmission of the audio data and improving the transmission quality of the audio data.

[0059] See also Figure 5 , which shows a flow chart of an audio data transmission method provided by another embodiment of the present application, the audio data transmission method can be applied to Figure 1 The server in the audio data transmission system shown in FIG. Figure 5The process shown in the figure is described in detail, and the audio data transmission method may include the following steps:

[0060] Step S510: Acquire N channels of audio data to be sent.

[0061] Step S520: Select M channels of audio data from the N channels of audio data.

[0062] Step S530: extracting audio feature information of M channels of audio data respectively.

[0063] In the embodiment of the present application, step S510, step S520 and step S530 can refer to the contents of the aforementioned embodiment and will not be described again here.

[0064] Step S540: Acquire Mel-frequency cepstral coefficients M(i) corresponding to the M channels of audio data according to the audio feature information.

[0065] In an embodiment of the present application, after respectively extracting audio feature information of M channels of audio data, the server can obtain the Mel-frequency cepstral coefficients M(i) corresponding to the M channels of audio data based on the extracted audio feature information, so as to control the multiple transmission of the audio data according to the Mel-frequency cepstral coefficients.

[0066] It can be understood that Mel-Frequency Cepstral Coefficients (MFCC) are widely used in the field of audio processing. Mel-Frequency Cepstral Coefficients are coefficients that make up the Mel-Frequency Cepstral, which are derived from the Cepstral of the audio segment. The Mel-Frequency Cepstral Coefficients can better approximate the human auditory system than the linearly spaced frequency bands used in the normal logarithmic cepstrum; the Mel-Frequency Cepstral (MFC) is a linear transformation of the logarithmic energy spectrum based on the nonlinear Mel-Scale of the sound frequency.

[0067] The Mel-frequency cepstral coefficient is related to the ease of human hearing perception of audio data. The larger the Mel-frequency cepstral coefficient of audio data, the easier it is for the audio data to be perceived by human hearing; the smaller the psychoacoustic perception coefficient of audio data, the more difficult it is for the audio data to be perceived by human hearing.

[0068] In some implementations, the server may calculate the Mel-frequency cepstral coefficients of the M channels of audio data respectively according to the Mel-frequency cepstral coefficient formula, where the Mel-frequency cepstral coefficient formula is: Among them, i is the audio data number, l is the filter number, L is the number of filters, T is the number of filter sampling points, F i(l) is the filtering output of the lth filter of the i-th audio data, and M(i) is the Mel-frequency cepstrum coefficient of the i-th audio data.

[0069] The filter output can be calculated according to the filter formula, which is: Among them, w i (l) is the filter coefficient of the lth filter of the i-th audio data, X i [i] is the amplitude spectrum of the lth filter of the i-th audio data.

[0070] The filter coefficient can be calculated according to the spectral formula, which is: Among them, k is the sampling number, f i (c(l)) is the lower limit frequency of the lth filter of the i-th audio data, f i (c(l)) is the center frequency of the lth filter of the i-th audio data, f i (h(l)) is the upper limit frequency of the lth filter of the i-th audio data.

[0071] The amplitude spectrum can be calculated according to the amplitude spectrum formula, which is: Among them, x i [l] is the speech sequence of the lth filter of the i-th audio data.

[0072] Step S550: controlling the multiple transmission of corresponding audio data according to the Mel-frequency cepstral coefficient M(i).

[0073] In an embodiment of the present application, after obtaining the Mel-frequency cepstral coefficient M(i) corresponding to M channels of audio data based on the extracted audio feature information, the server can control the multiple transmission of the corresponding audio data based on the Mel-frequency cepstral coefficient M(i), so that when M(j)>M(k), red'(j)>red'(k), thereby controlling the multiple transmission of the corresponding audio data based on the Mel-frequency cepstral coefficient of the audio data, transmitting audio data with larger Mel-frequency cepstral coefficients with a larger multiple transmission factor, and transmitting audio data with smaller Mel-frequency cepstral coefficients with a smaller multiple transmission factor, thereby ensuring the effective transmission of the audio data and improving the transmission quality of the audio data.

[0074] In some implementations, the server may determine the multiple transmission multiple of each audio data channel according to the Mel frequency cepstrum coefficient and the formula under the second multiple transmission multiple, so as to multiple the audio data according to the corresponding multiple transmission multiple. The formula for the second multiple transmission multiple is: red'(i)=F(M(i))*red_org(i), where red'(i) is the multiple transmission multiple of the i-th audio data channel, F(M(i))∈[0,1], and F(M(i)) is monotonically increasing, and red_org(i) is the multiple transmission multiple of the i-th audio data channel calculated based on the packet loss rate.

[0075] The solution provided by the present application obtains N channels of audio data to be sent, then selects M channels of audio data from the N channels of audio data, then extracts audio feature information of the M channels of audio data respectively, and obtains Mel-frequency cepstral coefficients corresponding to the M channels of audio data according to the audio feature information, and then controls the multiple transmission of the corresponding audio data according to the Mel-frequency cepstral coefficients. Therefore, the multiple transmission of the corresponding audio data is controlled based on the Mel-frequency cepstral coefficients of the audio data. The audio data with larger Mel-frequency cepstral coefficients is transmitted with a larger multiple multiple, and the audio data with smaller Mel-frequency cepstral coefficients is transmitted with a smaller multiple multiple, thereby ensuring the effective transmission of the audio data and improving the transmission quality of the audio data.

[0076] See also Figure 6 , which shows a flowchart of an audio data transmission method provided by another embodiment of the present application, the audio data transmission method can be applied to Figure 1 The server in the audio data transmission system shown in FIG. Figure 6 The process shown in the figure is described in detail, and the audio data transmission method may include the following steps:

[0077] Step S610: Acquire N channels of audio data to be sent.

[0078] Step S620: Select M channels of audio data from the N channels of audio data.

[0079] Step S630: extracting audio feature information of M channels of audio data respectively.

[0080] Step S640: Obtain psychoacoustic perception coefficient EP(i) corresponding to the M channels of audio data according to the audio feature information.

[0081] In the embodiment of the present application, step S610, step S620, step S630 and step S640 can refer to the contents of the aforementioned embodiment and will not be described again here.

[0082] Step S650: Control the corresponding audio data to be transmitted multiple times according to the transmission interval duration according to the psychoacoustic perception coefficient EP(i).

[0083] In the embodiment of the present application, after the server obtains the psychoacoustic perception coefficient EP(i) corresponding to the M-channel audio data according to the audio feature information, it can control the corresponding audio data to be transmitted multiple times according to the transmission interval duration according to the psychoacoustic perception coefficient EP(i), which can avoid the repeated transmission interval duration of a single-channel audio data being too short, resulting in serious network bandwidth consumption, thereby affecting the transmission of the audio data, and thus improving the transmission efficiency of the audio data. Among them, the transmission interval duration is used to indicate the interval duration between two repeated transmissions of the audio data.

[0084] It can be understood that M channels of audio data may include 5 channels of audio data, which are recorded as audio data A, audio data B, audio data C, audio data D and audio data E, respectively. The multiple transmission control parameters of the 5 channels of audio data are respectively: audio data A (5, 0.1ms), audio data B (3, 0.3ms), audio data C (6, 0.2ms), audio data D (2, 0.4ms), and audio data E (4, 0.5ms), indicating that the server controls audio data A to be transmitted at a time interval of 0.1ms and a multiple transmission multiple of 5, controls audio data B to be transmitted at a time interval of 0.3ms and a multiple transmission multiple of 3, controls audio data C to be transmitted at a time interval of 0.2ms and a multiple transmission multiple of 6, controls audio data D to be transmitted at a time interval of 0.4ms and a multiple transmission multiple of 2, and controls audio data E to be transmitted at a time interval of 0.5ms and a multiple transmission multiple of 2. The total number of audio data, the multiple transmission multiple of each channel of audio data, and the transmission interval duration are not limited here.

[0085] The solution provided by the present application obtains N channels of audio data to be sent, then selects M channels of audio data from the N channels of audio data, then extracts audio feature information of the M channels of audio data respectively, and obtains psychoacoustic perception coefficients corresponding to the M channels of audio data according to the audio feature information, and then controls the corresponding audio data to be transmitted multiple times according to the transmission interval duration according to the psychoacoustic perception coefficient EP(i). Therefore, the multiple transmission of the corresponding audio data is controlled based on the psychoacoustic perception coefficient of the audio data, and a larger multiple multiple is used for transmission of audio data with a larger psychoacoustic perception coefficient, and a smaller multiple multiple is used for transmission of audio data with a smaller psychoacoustic perception coefficient, thereby ensuring the effective transmission of the audio data and improving the transmission quality of the audio data. By controlling the corresponding audio data to be transmitted multiple times according to the transmission interval duration according to the psychoacoustic perception coefficient EP(i), it is possible to avoid the situation where the repeated transmission interval duration of a single channel of audio data is too short, which results in serious network bandwidth consumption, thereby affecting the transmission of the audio data, and thus the transmission efficiency of the audio data can be improved.

[0086] See also Figure 7, which shows a flowchart of an audio data transmission method provided by another embodiment of the present application, the audio data transmission method can be applied to Figure 1 The server in the audio data transmission system shown in FIG. Figure 7 The process shown in the figure is described in detail, and the audio data transmission method may include the following steps:

[0087] Step S710: Obtain N channels of audio data to be sent.

[0088] Step S720: Select M channels of audio data from the N channels of audio data.

[0089] Step S730: extracting audio feature information of M channels of audio data respectively.

[0090] Step S740: Obtain psychoacoustic perception coefficient EP(i) corresponding to the M channels of audio data according to the audio feature information.

[0091] In the embodiment of the present application, step S710, step S720, step S730 and step S740 can refer to the contents of the aforementioned embodiment and will not be described again here.

[0092] Step S750: in the process of controlling the multiple transmissions of the corresponding audio data according to the psychoacoustic perception coefficient EP(i), for each transmission of each channel of audio data, the packet loss rate of each channel of audio data at that time is obtained.

[0093] In an embodiment of the present application, in order to determine in real time whether the audio data is sent successfully, the server can obtain the packet loss rate of each audio data transmission at each transmission of each audio data, so as to control the subsequent transmission of the audio data based on the obtained packet loss rate of the current transmission.

[0094] Step S760: when the packet loss rate of the current transmission is obtained to be 0, the current multiple of each channel of audio data is updated to 0.

[0095] In an embodiment of the present application, when the server obtains that the packet loss rate of each audio data transmission is 0, it indicates that the audio data has been successfully received by the client. The server can update the current multiple of the audio data corresponding to the packet loss rate of 0 in the current transmission to 0, that is, stop sending the audio data that has been successfully received, thereby reducing the consumption of network bandwidth and further reducing the packet loss rate of subsequent audio data transmission, thereby improving the transmission quality of the audio data.

[0096] The solution provided by the present application obtains N channels of audio data to be sent, then selects M channels of audio data from the N channels of audio data, then extracts audio feature information of the M channels of audio data respectively, and obtains the psychoacoustic perception coefficient corresponding to the M channels of audio data according to the audio feature information, and then controls the multiple transmission of the corresponding audio data according to the psychoacoustic perception coefficient EP(i). Therefore, the multiple transmission of the corresponding audio data is controlled based on the psychoacoustic perception coefficient of the audio data, and a larger multiple multiple is used for transmission of audio data with a larger psychoacoustic perception coefficient, and a smaller multiple multiple is used for transmission of audio data with a smaller psychoacoustic perception coefficient, thereby ensuring the effective transmission of the audio data and improving the transmission quality of the audio data. In addition, by obtaining the packet loss rate of the current transmission of each channel of audio data during the multiple transmission process, when the packet loss rate of the current transmission is obtained to be 0, the current multiple multiple of the audio data with the packet loss rate of 0 in the current transmission is updated to 0, which can reduce the consumption of network bandwidth, and further reduce the packet loss rate of subsequent audio data transmission, so as to further improve the transmission quality of the audio data.

[0097] See also Figure 8 , which shows a schematic diagram of the structure of an audio data transmission device provided by an embodiment of the present application, the audio data transmission device can be applied to Figure 1 The server in the audio data transmission system shown in FIG. Figure 8 The device shown in the figure is described in detail, and the audio data transmission device 800 may include:

[0098] The first acquisition module 801 is used to acquire N channels of audio data to be sent, where N is a positive integer greater than or equal to 1;

[0099] A selection module 802 is used to select M channels of audio data from N channels of audio data, where M is a positive integer and 1≤M≤N;

[0100] An extraction module 803 is used to extract audio feature information of M channels of audio data respectively;

[0101] A second acquisition module 804 is configured to acquire a psychoacoustic perception coefficient EP(i) corresponding to M channels of audio data according to the audio feature information, wherein i represents a sequence number of the audio data, and 1≤i≤M; and

[0102] The psychoacoustic perception control module 805 is used to control the multiple transmission of the corresponding audio data according to the psychoacoustic perception coefficient EP(i), so that when EP(j)>EP(k), red'(j)>red'(k), wherein 1≤j≤M, 1≤k≤M, and j≠k, red'(j) and red'(k) respectively represent the multiple transmission times of the j-th and k-th audio data.

[0103] In some implementations, the second acquisition module 804 may include:

[0104] The first calculation unit is used to calculate the psychoacoustic perception coefficients of the M-channel audio data respectively according to the following formula: Wherein, i represents the audio data serial number, and 1≤i≤M, EP(i) is the psychoacoustic perception coefficient of the i-th audio data, k is the frequency point serial number, K is the total number of frequency points, p(i, k) is the absolute value of the power of the i-th audio data, and cof(k) is the psychoacoustic perception weighting coefficient of frequency point k.

[0105] In this embodiment, the psychoacoustic perception control module 805 may include:

[0106] The first determination unit is used to determine the multiplexing multiple of each audio data channel according to the psychoacoustic perception coefficient and the first multiplexing multiple formula, the first multiplexing multiple formula is: red'(i)=f(EP(i))*red_org(i), red'(i) is the multiplexing multiple of the i-th audio data channel, f(EP(i))∈[0,1], and f(EP(i)) is monotonically increasing, and red_org(i) is the multiplexing multiple of the i-th audio data channel calculated based on the packet loss rate.

[0107] In some other implementations, the audio data transmission device 800 may further include:

[0108] A third acquisition module is used to acquire Mel-frequency cepstral coefficients M(i) corresponding to M channels of audio data according to audio feature information;

[0109] The Mel frequency cepstrum control module is used to control the multiple transmission of the corresponding audio data according to the Mel frequency cepstrum coefficient M(i), so that when M(j)>M(k), red'(j)>red'(k).

[0110] In this implementation, the third acquisition module may include:

[0111] The second calculation unit calculates the Mel-frequency cepstrum coefficients of the M-channel audio data respectively according to the following formula: Among them, i is the audio data number, l is the filter number, L is the number of filters, T is the number of filter sampling points, F i (l) is the filtering output of the lth filter of the i-th audio data, and M(i) is the Mel-frequency cepstrum coefficient of the i-th audio data.

[0112] In this implementation, the Mel-frequency cepstrum control module may include:

[0113] The second determination unit is used to determine the multiple times of each audio data channel according to the Mel frequency cepstrum coefficient and the second multiple times formula, the second multiple times formula is: red'(i)=F(M(i))*red_org(i), wherein red'(i) is the multiple times of the i-th audio data channel, F(M(i))∈[0,1], and F(M(i)) is monotonically increasing, and red_org(i) is the multiple times of the i-th audio data channel calculated based on the packet loss rate.

[0114] In some further implementations, the psychoacoustic perception control module 805 may further include:

[0115] A control unit is used to control the corresponding audio data to be transmitted multiple times according to the psychoacoustic perception coefficient EP(i) according to the transmission interval duration, wherein the transmission interval duration is used to indicate the interval duration between two repeated transmissions of the audio data.

[0116] In some further implementations, the control module 805 may further include:

[0117] an acquisition unit, configured to acquire, for each transmission of each channel of audio data, a packet loss rate of the current transmission of each channel of audio data in a process of controlling the multiple transmissions of the corresponding audio data according to the psychoacoustic perception coefficient EP(i);

[0118] The updating unit is used to update the current multiple of each channel of audio data to 0 when the packet loss rate of the current transmission is 0.

[0119] In yet other implementations, the selection module 802 may include:

[0120] The selection unit is used to select M channels of audio data whose energy value is greater than or equal to a preset energy threshold and whose signal-to-noise ratio is greater than or equal to a preset signal-to-noise ratio threshold from N channels of audio data.

[0121] The solution provided by the present application obtains N channels of audio data to be sent, then selects M channels of audio data from the N channels of audio data, then extracts audio feature information of the M channels of audio data respectively, and obtains psychoacoustic perception coefficients corresponding to the M channels of audio data according to the audio feature information, and then controls the multiple transmission of the corresponding audio data according to the psychoacoustic perception coefficient. Therefore, the multiple transmission of the corresponding audio data is controlled based on the psychoacoustic perception coefficient of the audio data, and a larger multiple multiple is used for transmission of audio data with a larger psychoacoustic perception coefficient, and a smaller multiple multiple is used for transmission of audio data with a smaller psychoacoustic perception coefficient, thereby ensuring the effective transmission of the audio data and improving the transmission quality of the audio data.

[0122] See also Fig. 9, which shows an electronic device 900 provided by an embodiment of the present application, including a memory 910, a processor 920, and a computer program stored in the memory 910 and executable on the processor 920, and the processor 920 implements the method described in the aforementioned method embodiment when executing the computer program.

[0123] The processor 920 may include one or more processing cores. The processor 920 uses various interfaces and lines to connect various parts of the entire electronic device 900, and executes various functions and processes data of the electronic device 900 by running or executing instructions, programs, code sets or instruction sets stored in the memory 910, and calling data stored in the memory 910. Optionally, the processor 920 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 920 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the modem may not be integrated into the processor 920, but may be implemented separately through a communication chip.

[0124] The memory 910 may include a random access memory (RAM) or a read-only memory (ROM). The memory 910 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 910 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as acquisition, selection, extraction, control, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as N-channel audio data, M-channel audio data, audio feature information, psychoacoustic perception coefficients, psychoacoustic perception weighting coefficients, multiple multiples, Mel frequency cepstrum coefficients, number of filter sampling points, energy value, and signal-to-noise ratio) created by the electronic device 900 during use.

[0125] See also Fig.10, which shows a computer-readable storage medium provided by an embodiment of the present application, characterized in that a program code is stored in the computer-readable storage medium 1000, and the program code can be called by a processor to execute the method described in the aforementioned method embodiment.

[0126] The computer readable storage medium 1000 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 1000 includes a non-transitory computer-readable storage medium. The computer readable storage medium 1000 has storage space for program code 1010 that performs any method step in the above method. These program codes can be read from or written to one or more computer program products. The program code 1010 can be compressed, for example, in an appropriate form.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An audio data transmission method, characterized in that: The method comprises: Obtain N channels of audio data to be sent, where N is a positive integer greater than or equal to 1; Selecting M channels of audio data from the N channels of audio data, wherein M is a positive integer, and ; Extracting audio feature information of the M channels of audio data respectively; Acquire the psychoacoustic perception coefficient corresponding to the M channels of audio data according to the audio feature information ,in Indicates the audio data sequence number, and ;as well as According to the psychoacoustic perception coefficient Control the multiple transmission of the corresponding audio data so that hour, ,in, , ,and , and Respectively represent j , k The multiple of audio data transmission.

2. The method according to claim 1, characterized in that The step of obtaining the psychoacoustic perception coefficient corresponding to the M-channel audio data according to the audio feature information include: The psychoacoustic perception coefficients of the M channels of audio data are calculated respectively according to the following formulas: ,in, Indicates the audio data sequence number, and , For the Psychoacoustic perception coefficients of audio data, k is the frequency point number, K is the total frequency points, p(i, k) For the The absolute value of the power of the audio data. cof(k) Frequency k The psychoacoustic perceptual weighting coefficient of .

3. The method according to claim 2, characterized in that The psychoacoustic perception coefficient Controlling the multiple transmission of the corresponding audio data includes: determining the multiple transmission multiple of each channel of the audio data according to the psychoacoustic perception coefficient and a first multiple transmission multiple formula, wherein the first multiple transmission multiple formula is: , For the The multiple of audio data transmission. ,and Monotonically increasing, For the The multiple of audio data transmission based on the packet loss rate.

4. The method according to claim 1, characterized in that: The method further comprises: Acquire the Mel-frequency cepstrum coefficients corresponding to the M channels of audio data according to the audio feature information ; According to the Mel frequency cepstral coefficient Control the multiple transmission of the corresponding audio data so that hour, .

5. The method according to claim 4, characterized in that The method of acquiring the Mel-frequency cepstral coefficients corresponding to the M channels of audio data according to the audio feature information include: The Mel-frequency cepstrum coefficients of the M channels of audio data are calculated respectively according to the following formula: , ,in, is the audio data sequence number, is the filter number, L is the number of filters, T is the number of filter sampling points, For the The first channel of audio data The filtered output of the filter, For the Mel-frequency cepstral coefficients of the audio data.

6. The method according to claim 4, characterized in that The Mel frequency cepstral coefficients Control the multiple transmission of corresponding audio data, including: The multiple transmission multiple of each channel of the audio data is determined according to the Mel frequency cepstrum coefficient and a second multiple transmission multiple formula, wherein the second multiple transmission multiple formula is: ,in, For the The multiple of audio data transmission. ,and Monotonically increasing, For the The multiple of audio data transmission based on the packet loss rate.

7. The method according to claim 1, characterized in that The psychoacoustic perception coefficient Controlling the multiple transmission of corresponding audio data includes: According to the psychoacoustic perception coefficient Control the corresponding audio data to be transmitted multiple times according to the transmission interval duration, wherein the transmission interval duration is used to indicate the interval duration for repeating the transmission of the audio data twice.

8. The method according to claim 1, characterized in that The psychoacoustic perception coefficient Controlling the multiple transmission of corresponding audio data includes: According to the psychoacoustic perception coefficient In the process of controlling the multiple transmissions of the corresponding audio data, for each transmission of each channel of audio data, obtaining a packet loss rate of the current transmission of each channel of audio data; When the packet loss rate of the current transmission is obtained to be 0, the current multiple of each channel of audio data is updated to 0.

9. The method according to any one of claims 1 to 8, characterized in that: The selecting M channels of audio data from the N channels of audio data comprises: The M channels of audio data having energy values ​​greater than or equal to a preset energy threshold and signal-to-noise ratios greater than or equal to a preset signal-to-noise ratio threshold are selected from the N channels of audio data.

10. An audio data transmission device, characterized in that: The device comprises: A first acquisition module, used to acquire N channels of audio data to be sent, where N is a positive integer greater than or equal to 1; A selection module is used to select M channels of audio data from the N channels of audio data, wherein M is a positive integer, and ; An extraction module, used to extract audio feature information of the M channels of audio data respectively; A second acquisition module is used to acquire the psychoacoustic perception coefficient corresponding to the M channels of audio data according to the audio feature information. ,in Indicates the audio data sequence number, and ;as well as A psychoacoustic perception control module is used to control the Control the multiple transmission of the corresponding audio data so that hour, ,in, , ,and , and Respectively represent j , k The multiple of audio data transmission.

11. The device according to claim 10, characterized in that The second acquisition module includes: The first calculation unit is used to calculate the psychoacoustic perception coefficients of the M channels of audio data respectively according to the following formula: ,in, Indicates the audio data sequence number, and , For the Psychoacoustic perception coefficients of audio data, k is the frequency point number, K is the total frequency points, p(i, k) For the The absolute value of the power of the audio data. cof(k) Frequency k The psychoacoustic perceptual weighting coefficient of .

12. The device according to claim 11, characterized in that The psychoacoustic perception control module comprises: The first determining unit is used to determine the multiple sending multiple of each channel of the audio data according to the psychoacoustic perception coefficient and a first multiple sending multiple formula, wherein the first multiple sending multiple formula is: , For the The multiple of audio data transmission. ,and Monotonically increasing, For the The multiple of audio data transmission based on the packet loss rate.

13. The device according to claim 10, characterized in that The audio data transmission device further comprises: The third acquisition module is used to acquire the Mel frequency cepstrum coefficients corresponding to the M channels of audio data according to the audio feature information. ; Mel frequency cepstrum control module, used for Control the multiple transmission of the corresponding audio data so that hour, .

14. The device according to claim 13, characterized in that The third acquisition module includes: The second calculation unit is used to calculate the Mel-frequency cepstral coefficients of the M channels of audio data respectively according to the following formula: , ,in, is the audio data sequence number, is the filter number, L is the number of filters, T is the number of filter sampling points, For the The first channel of audio data The filtered output of the filter, For the Mel-frequency cepstral coefficients of the audio data.

15. The device according to claim 13, characterized in that The Mel frequency cepstrum control module comprises: The second determining unit is used to determine the multiple multiples of each channel of the audio data according to the Mel frequency cepstrum coefficient and a second multiple multiple formula, wherein the second multiple multiple formula is: ,in, For the The multiple of audio data transmission. ,and Monotonically increasing, For the The multiple of audio data transmission based on the packet loss rate.

16. The device according to claim 10, characterized in that The psychoacoustic perception control module comprises: control unit for determining the psychoacoustic perception coefficient Control the corresponding audio data to be transmitted multiple times according to the transmission interval duration, wherein the transmission interval duration is used to indicate the interval duration for repeating the transmission of the audio data twice.

17. The device according to claim 10, characterized in that The psychoacoustic perception control module comprises: Get unit for the psychoacoustic perception coefficient In the process of controlling the multiple transmissions of the corresponding audio data, for each transmission of each channel of audio data, obtaining a packet loss rate of the current transmission of each channel of audio data; The updating unit is used to update the current multiple of each channel of audio data to 0 when the packet loss rate of the current transmission is obtained to be 0.

18. The device according to any one of claims 10 to 17, characterized in that The selection module comprises: The selection unit is used to select the M channels of audio data whose energy value is greater than or equal to a preset energy threshold and whose signal-to-noise ratio is greater than or equal to a preset signal-to-noise ratio threshold from the N channels of audio data.

19. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 9 is implemented.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 9.

21. A computer program product, characterized in that The method comprises a program code, and when the program code is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images