Audio understanding method and system, electronic equipment and storage medium

Abstract

The invention discloses an audio understanding method and system, electronic equipment and a storage medium, and the method comprises the steps: obtaining each coding unit of each audio clip of a target audio, and carrying out the compression through discrete cosine transform based on the information entropy of each coding unit, and obtaining a compressed coding unit; polling each audio clip, and calculating the acoustic significance membership degree of each compressed coding unit and the time sequence correlation membership degree of each current fusion coding unit of the previous audio clip; calculating the contribution weight of each compression coding unit and the contribution weight of each current fusion coding unit to the compression coding unit by using the membership degree of each compression coding unit; respectively utilizing the contribution weight corresponding to each compression coding unit to carry out weighted summation on the compression coding unit and each current fusion coding unit to obtain each fusion coding unit of the current audio clip, inputting the fusion coding units into a large language model, and outputting an understanding analysis result; and fusing the understanding analysis results into a result of the target audio.

Description

Technical Field

[0001] This application relates to the field of audio processing technology, and in particular to an audio understanding method and system, electronic device, and storage medium. Background Technology

[0002] Current multimodal large-scale models can understand audio and video, but they lag behind proprietary large-scale audio understanding models in audio understanding. Therefore, current proprietary large-scale audio understanding models are mainly used for audio analysis and understanding.

[0003] Current proprietary large-scale audio understanding models require processing audio into audio coding units (audio tokens) before performing understanding and analysis. This results in a large number of audio tokens for longer audio files, exceeding the model's context window length. Therefore, long audio files are typically split into multiple audio segments, and the audio tokens for each segment are obtained separately. These individual audio tokens are then directly input into the large-scale model for understanding and analysis, resulting in the fused and outputted understanding and analysis of each audio segment.

[0004] However, this method still requires processing a large number of audio tokens, resulting in low efficiency in understanding and analyzing long audio files and causing inference delays. Furthermore, since there are relationships between audio data, directly understanding and analyzing audio segments can lead to inaccurate results due to missing information, ultimately resulting in an inaccurate final outcome. Summary of the Invention

[0005] In view of the shortcomings of the prior art, this application provides an audio understanding method and system, electronic device and storage medium to solve the problem that the prior art cannot efficiently and accurately understand and analyze audio.

[0006] To achieve the above objectives, this application provides the following technical solution:

[0007] The first aspect of this application provides an audio understanding method, including:

[0008] Obtain each coding unit of each audio segment of the target audio;

[0009] Based on the information entropy of each coding unit of each audio segment, the coding units of each audio segment are compressed by discrete cosine transform to obtain each compressed coding unit of each audio segment.

[0010] Each audio segment is polled sequentially, and the membership degree corresponding to each compressed coding unit of the current audio segment is calculated; wherein, the membership degree corresponding to the compressed coding unit includes its own acoustic saliency membership degree and its temporal correlation membership degree with each current fusion coding unit; the current fusion coding unit is each fusion coding unit of the previous audio segment of the current audio segment;

[0011] The contribution weight of each compression coding unit is calculated using the membership degree of each compression coding unit.

[0012] By utilizing the contribution weight corresponding to each compression coding unit, a weighted sum is performed on the compression coding unit and each of the current fusion coding units to obtain the fusion coding units of the current audio segment;

[0013] The fusion coding units of each audio segment are merged and input into a large language model. The understanding and analysis results of each audio segment are output and fused to obtain the understanding and analysis results of the target audio.

[0014] Optionally, in the above audio understanding method, the step of compressing the coding units of each audio segment based on the information entropy of each coding unit of each audio segment using discrete cosine transform to obtain each compressed coding unit of each audio segment includes:

[0015] For each of the audio segments, calculate the initial information entropy of each coding unit of the audio segment;

[0016] The initial information entropy of each coding unit of the audio segment is subjected to a cosine discrete transform to obtain a coefficient matrix;

[0017] Update all coefficients in the coefficient matrix that are not sorted at the beginning of the preset compression ratio to zero;

[0018] The updated coefficient matrix is ​​subjected to inverse discrete cosine transform to obtain the updated information entropy of each coding unit;

[0019] Calculate the similarity between the initial information entropy and the updated information entropy of each coding unit, and filter out each coding unit whose similarity is less than a first similarity threshold;

[0020] Based on the information entropy of the coding unit, the remaining consecutive and similar coding units are aggregated to obtain the various compressed coding units of the audio segment.

[0021] Optionally, in the above audio understanding method, the step of aggregating consecutive and similar coding units among the remaining coding units based on the information entropy of the coding unit to obtain each compressed coding unit of the audio segment includes:

[0022] The current window position is initialized to the first coding unit in the coding unit sequence; wherein, the coding unit sequence is the sequence of the remaining coding units;

[0023] Based on the similarity of adjacent coding units, the current window is continuously expanded until the similarity between the last coding unit in the current window and the next coding unit is less than a first similarity threshold, or it is expanded to the last coding unit in the coding unit sequence.

[0024] Based on the information entropy of the coding unit, the coding units in the current window are aggregated to obtain a compression coding unit;

[0025] If the current window does not expand to the last coding unit in the coding unit sequence, the current window is reset to its next coding unit, and the process of continuously expanding the current window based on the similarity of adjacent coding units is returned.

[0026] Optionally, in the above audio understanding method, the step of aggregating the various coding units in the current window based on the information entropy of the coding units to obtain a compression coding unit includes:

[0027] The sum of the products of each coding unit in the current window and its information entropy is divided by the sum of the information entropy of each coding unit, and the quotient is added to the product of the residual term and the residual coefficient of the current window to obtain a compressed coding unit.

[0028] Optionally, in the above audio understanding method, calculating the membership degree corresponding to each compressed coding unit of the current audio segment includes:

[0029] For each compression coding unit of the current audio segment, the temporal association membership degree between the compression coding unit and each current fusion coding unit is calculated based on the time difference between the compression coding unit and each current fusion coding unit.

[0030] The acoustic saliency membership degree of the compressed coding unit is calculated based on the ratio of the acoustic energy of the compressed coding unit to the global maximum acoustic energy, and the ratio of the spectral entropy of the compressed coding unit to the global maximum spectral entropy.

[0031] Optionally, in the above audio understanding method, the step of calculating the contribution weight of each compressed coding unit using the membership degree of each compressed coding unit includes:

[0032] For each of the compression coding units, the temporal association membership degree between the compression coding unit and each of the current fusion coding units is multiplied by the acoustic saliency membership degree of the fusion coding unit to obtain the absolute contribution weight of each of the current fusion coding units to the compression coding unit.

[0033] The absolute contribution weight of each current fusion coding unit to the compression coding unit is normalized to obtain the contribution weight of each current fusion coding unit to the compression coding unit.

[0034] The balance coefficient is added to the unbalance coefficient and the acoustic saliency membership degree of each compression coding unit to obtain the self-contribution weight of each compression coding unit; wherein the unbalance coefficient is 1 minus the balance coefficient.

[0035] Optionally, in the above audio understanding method, the step of using the contribution weight corresponding to each of the compression coding units to perform a weighted summation of the compression coding unit and each of the current fusion coding units to obtain the fusion coding units of the current audio segment includes:

[0036] For each of the compression coding units, the weighted sum of the compression coding unit and each of the current fusion coding units is weighted according to the contribution weight of the compression coding unit itself to obtain a fusion coding unit for the current audio segment; wherein, the weighted sum of each of the current fusion coding units is equal to the result of weighted summation of each current fusion coding unit and its contribution weight to the compression coding unit.

[0037] A second aspect of this application provides an audio understanding system, comprising:

[0038] The data acquisition unit is used to acquire each encoding unit of each audio segment of the target audio.

[0039] An audio compression unit is used to compress the coding units of each audio segment based on the information entropy of each coding unit of each audio segment through discrete cosine transform, so as to obtain each compressed coding unit of each audio segment.

[0040] The membership calculation unit is used to sequentially poll each of the audio segments and calculate the membership degree corresponding to each compressed coding unit of the current audio segment; wherein, the membership degree corresponding to the compressed coding unit includes its own acoustic saliency membership degree and its temporal correlation membership degree with each current fusion coding unit; the current fusion coding unit is each fusion coding unit of the previous audio segment of the current audio segment;

[0041] The weight calculation unit is used to calculate the contribution weight of each compressed coding unit by using the membership degree of each compressed coding unit respectively.

[0042] A weighting unit is used to perform a weighted summation of the compression coding unit and each of the current fusion coding units using the contribution weight corresponding to each compression coding unit, so as to obtain each fusion coding unit of the current audio segment;

[0043] The understanding and analysis unit is used to merge the various fusion coding units of each audio segment, input them into the large language model, output the understanding and analysis results of each audio segment, and fuse them to obtain the understanding and analysis results of the target audio.

[0044] Optionally, in the above-described audio understanding system, the audio compression unit includes:

[0045] An information entropy calculation unit is used to calculate the initial information entropy of each coding unit of each audio segment for each audio segment.

[0046] The transformation unit is used to perform a cosine discrete transformation on the initial information entropy of each coding unit of the audio segment to obtain a coefficient matrix;

[0047] The coefficient update unit is used to update each coefficient in the coefficient matrix that is not sorted at the beginning of the preset compression ratio to zero.

[0048] The inverse transform unit is used to perform an inverse discrete cosine transform on the updated coefficient matrix to obtain the updated information entropy of each coding unit.

[0049] A secondary compression unit is used to calculate the similarity between the initial information entropy and the updated information entropy of each coding unit, and to filter out each coding unit whose similarity is less than a first similarity threshold.

[0050] An aggregation unit is used to aggregate consecutive and similar coding units among the remaining coding units based on the information entropy of the coding unit, to obtain each compressed coding unit of the audio segment.

[0051] Optionally, in the above-described audio understanding system, the aggregation unit includes:

[0052] An initialization unit is used to initialize the current window position as the first encoding unit in the encoding unit sequence; wherein the encoding unit sequence is the sequence of the remaining encoding units;

[0053] An expansion unit is used to continuously expand the current window based on the similarity of adjacent coding units until the similarity between the last coding unit in the current window and the next coding unit is less than a first similarity threshold, or to expand to the last coding unit in the coding unit sequence.

[0054] A window aggregation unit is used to aggregate the various encoding units in the current window based on the information entropy of the encoding unit to obtain a compression encoding unit;

[0055] A reset unit is configured to reset the current window to its next encoding unit and return to the expansion unit when the current window has not expanded to the last encoding unit of the encoding unit sequence.

[0056] Optionally, in the above-described audio understanding system, the window aggregation unit includes:

[0057] The window aggregation subunit is used to divide the sum of the products of each coding unit and its information entropy in the current window by the sum of the information entropies of each coding unit, and add the quotient to the product of the residual term and the residual coefficient of the current window to obtain a compression coding unit.

[0058] Optionally, in the above-described audio understanding system, the membership calculation unit includes:

[0059] The first membership calculation unit is used to calculate the temporal association membership degree between each compression coding unit and each current fusion coding unit for each compression coding unit of the current audio segment, based on the time difference between the compression coding unit and each current fusion coding unit.

[0060] The second membership calculation unit is used to calculate the acoustic saliency membership degree of the compressed coding unit based on the ratio of the acoustic energy of the compressed coding unit to the global maximum acoustic energy, and the ratio of the spectral entropy of the compressed coding unit to the global maximum spectral entropy.

[0061] Optionally, in the above-described audio understanding system, the weight calculation unit includes:

[0062] The first weight calculation unit is used to multiply the temporal association membership degree between the compressed coding unit and each current fusion coding unit by the acoustic saliency membership degree of the fusion coding unit for each compressed coding unit, so as to obtain the absolute contribution weight of each current fusion coding unit to the compressed coding unit.

[0063] The second weight calculation unit is used to normalize the absolute contribution weight of each current fusion coding unit to the compression coding unit, so as to obtain the contribution weight of each current fusion coding unit to the compression coding unit.

[0064] The third weight calculation unit is used to add the unbalanced coefficient and the acoustic saliency membership degree of each of the compressed coding units to the balance coefficient to obtain the self-contribution weight of each of the compressed coding units; wherein the unbalanced coefficient is 1 minus the balance coefficient.

[0065] Optionally, in the above-described audio understanding system, the weighting unit includes:

[0066] A weighted subunit is used to weight the weighted sum of the compression coding unit and each of the current fusion coding units according to the contribution weight of the compression coding unit itself, so as to obtain a fusion coding unit of the current audio segment; wherein, the weighted sum of each of the current fusion coding units is equal to the result of weighted summation of each current fusion coding unit and its contribution weight to the compression coding unit.

[0067] A third aspect of this application provides an electronic device, comprising:

[0068] Memory and processor;

[0069] The memory is used to store programs;

[0070] The processor is used to execute the program, which, when executed, is specifically used to implement the audio understanding method as described in any of the above.

[0071] A fourth aspect of this application provides a computer storage medium for storing a computer program, which, when executed by a processor, is used to implement the audio understanding method as described in any of the preceding claims.

[0072] This application provides an audio understanding method that obtains the coding units of each audio segment of the target audio. Based on the information entropy of each coding unit in each audio segment, it compresses the coding units of each audio segment using discrete cosine transform to obtain compressed coding units for each audio segment. This effectively filters redundant and invalid information, compresses the compressed coding units of the audio segments, and thus effectively reduces the amount of data to be processed while ensuring data accuracy, improving the efficiency of audio understanding and analysis, and ensuring its timeliness. Then, each audio segment is iteratively polled, and the membership degree corresponding to each compressed coding unit of the current audio segment is calculated. The membership degree of the compressed coding unit includes its own acoustic saliency membership degree and its temporal correlation membership degree with each of the current fused coding units. The current fused coding units are the fused coding units of the previous audio segment of the current audio segment. Therefore, the membership degree reflects the contribution of the current coding unit and the past coding units to the current coding unit. Then, using the membership degree of each compressed coding unit, the contribution weight of each compressed coding unit is calculated. This contribution weight is then used to perform a weighted summation of the compressed coding units and each current fusion coding unit, resulting in the fusion coding units for the current audio segment. Based on the contribution level of each coding unit, the coding units of the current audio segment are fused with those of past audio segments, thus incorporating prior information from the previous audio segment. Finally, the fusion coding units of each audio segment are merged and input into a large language model. The output of the understanding and analysis results for each audio segment is then fused to obtain the understanding and analysis result for the target audio. Because the fusion coding units of each audio segment incorporate prior information from the previous audio segment, the accuracy of the understanding and analysis results is effectively guaranteed. Therefore, a method for efficient and accurate audio understanding and analysis is implemented. Attached Figure Description

[0073] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0074] Figure 1 A flowchart of an audio understanding method provided in an embodiment of this application;

[0075] Figure 2 A flowchart illustrating a method for compressing an audio encoding unit, as provided in an embodiment of this application;

[0076] Figure 3A flowchart illustrating a method for aggregating coding units provided in this application embodiment;

[0077] Figure 4 A flowchart illustrating a method for calculating the contribution weight corresponding to a compression coding unit, provided in an embodiment of this application;

[0078] Figure 5 This application provides an architectural diagram of an audio understanding system according to an embodiment of the present application.

[0079] Figure 6 This is a schematic diagram of the architecture of an electronic device provided in an embodiment of this application. Detailed Implementation

[0080] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0081] In this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0082] This application provides an audio understanding method, such as... Figure 1 As shown, it includes the following steps:

[0083] S101. Obtain each encoding unit of each audio segment of the target audio.

[0084] Specifically, the target audio that needs to be understood and analyzed is split into multiple audio segments. Then, the audio encoder processes the audio segments to obtain each audio segment's encoding unit (Token), which is the set of n Tokens for each audio segment: T = {t1, t2, ..., tn}. n}

[0085] S102. Based on the information entropy of each coding unit of each audio segment, the coding units of each audio segment are compressed by discrete cosine transform to obtain each compressed coding unit of each audio segment.

[0086] In order to improve the efficiency of audio understanding and analysis, the encoding unit of each audio segment is compressed to remove redundant audio tokens.

[0087] The information entropy of each coding unit in each audio segment reflects its uncertainty. Therefore, in this embodiment, the information entropy is calculated for each coding unit. Optionally, specifically for any coding unit t... i ={t i1 , t i2 , ..., t im}, that is, a vector of dimension 1*m, which is then normalized to a vector between 0 and 1. Then the coding unit t can be calculated using the following formula. i Information entropy :

[0088] .

[0089] Then, by performing a discrete cosine transform on the information entropy of each coding unit, the information is converted to the frequency domain space, which highlights the local features of speech (such as pitch and timbre), making the entropy value more accurately represent local uncertainty, thereby enabling the identification and filtering of redundant information.

[0090] Optionally, in another embodiment of this application, one specific implementation of step S102 is as follows: Figure 2 As shown, it includes the following steps:

[0091] S201. For each audio segment, calculate the initial information entropy of each coding unit of the audio segment.

[0092] S202. Perform a cosine discrete transform on the initial information entropy of each coding unit of the audio segment to obtain the coefficient matrix.

[0093] Specifically, the set of initial information entropy for each coding unit of an audio segment: By using the cosine discrete transform to transform to the frequency domain space of the token, the matrix of coefficients of each coding unit is obtained. .

[0094] S203. Update all coefficients in the coefficient matrix that are not sorted at the beginning of the preset compression ratio to zero.

[0095] It should be noted that the coefficients in the coefficient matrix are closer to low frequency and responsible for global information as they move further to the left. Therefore, a compression ratio was set according to the requirements. For example, set it to 0.1. Then retain the coefficients from the beginning of the coefficient matrix. For example, the top 10% of the coefficients are used because these coefficients correspond to important information about the token. The remaining unsorted coefficients are those with a preset compression ratio. That is, all coefficients are set to 0, so that these coefficients are no longer retained, thereby changing the information entropy corresponding to these coefficients, and thus eliminating redundant coding units.

[0096] S204. Perform inverse discrete cosine transform on the updated coefficient matrix to obtain the updated information entropy of each coding unit.

[0097] Specifically, the coefficient matrix is ​​restored to the spatiotemporal domain using the inverse discrete cosine transform, that is, restored to the audio token set space, to obtain the sequence of updated information entropy for each coding unit. .

[0098] S205. Calculate the similarity between the initial information entropy and the updated information entropy of each coding unit, and filter out each coding unit whose similarity is less than the first similarity threshold.

[0099] It should be noted that since the coefficient for unimportant information is set to 0, the information entropy of the corresponding unimportant coding units will change significantly. Therefore, the similarity between the initial and updated information entropy of each coding unit can be calculated. If the similarity between two information entropies is less than the first similarity threshold, it means that the positional information of the coding unit at that position is unimportant, so that coding unit is filtered out, ultimately resulting in a set of Q important audio coding unit tokens. Its corresponding information entropy is .

[0100] S206. Based on the information entropy of the coding unit, aggregate the continuous and similar coding units in the remaining coding units to obtain the various compressed coding units of the audio segment.

[0101] In order to further compress the coding units of audio segments, local temporal semantic aggregation is further used to aggregate adjacent and similar coding units.

[0102] Specifically, for the set of Q important audio tokens generated in the previous step, an "aggregation window" is constructed using technical temporal similarity to dynamically merge adjacent and similar tokens, thereby generating a set of secondary audio compression tokens.

[0103] Optionally, in another embodiment of this application, one specific implementation of step S206 is as follows: Figure 3 As shown, it includes:

[0104] S301. Initialize the current window position to the first coding unit in the coding unit sequence.

[0105] The coding unit sequence is the sequence of the remaining coding units.

[0106] Specifically, in order to perform similarity matching on each coding unit in turn, the starting position of the current window is initialized to the first coding unit in the coding unit sequence, that is, the initial set of tokens in the current window is W={t1}.

[0107] S302. Based on the similarity of adjacent coding units, continuously expand the current window until the similarity between the last coding unit in the current window and the next coding unit is less than the first similarity threshold, or expand to the last coding unit in the coding unit sequence.

[0108] Specifically, the similarity between the last coding unit in the current window and the next coding unit is calculated.

[0109] Specifically, calculate the last coding unit t in the current window. i Its next coding unit t i-1 Similarity S(t) i , t i-1 ).

[0110] Then determine whether the similarity between the last coding unit in the current window and the next coding unit is less than the first similarity threshold, or extend to the last coding unit in the coding unit sequence.

[0111] If the similarity between the last coding unit and the next coding unit in the current window is not less than the first similarity threshold, it means that the consecutive coding units are still similar. Therefore, the current window is expanded to include the next coding unit. In other words, the current window is enlarged so that the next coding unit of the last coding unit is included in the current window.

[0112] If the similarity between the last coding unit in the current window and the next coding unit is less than the first similarity threshold, then the continuous sequence of similar coding units has been interrupted, indicating that the window will no longer be expanded. Therefore, step S303 can be executed at this point. When the expansion reaches the last coding unit in the coding unit sequence, it means that all coding units have been processed, and the current window can no longer be expanded. Therefore, step S304 is also executed at this point.

[0113] S303. Based on the information entropy of the coding unit, aggregate the coding units in the current window to obtain a compressed coding unit.

[0114] Since the mean values ​​of the coding units in the current window are similar, the coding units in the current window are aggregated.

[0115] Alternatively, the aggregation result S of the current window W can be calculated using a weighted average based on the residual information entropy. j This ensures that highly distinguishable tokens dominate the aggregation.

[0116] Optionally, in another embodiment of this application, one specific implementation of step S304 includes:

[0117] The sum of the products of each coding unit in the current window and its information entropy is divided by the sum of the information entropy of each coding unit, and the quotient is added to the product of the residual term and the residual coefficient of the current window to obtain a compressed coding unit.

[0118] Therefore, the specific calculation method for the aggregation result is as follows:

[0119] .

[0120] in, The residual term for the current window is defined as the L2 norm of the differences between all tokens within W and the unweighted average, i.e., Res(W) = ||t||. i -avg(W)||, where avg(W) is the mean of the encoding units in the current window, used to preserve details lost during the aggregation process; The residual coefficient (0 < (≤0.25, default value is 0.15), its main function is to balance the main features and detailed information.

[0121] S304. Determine whether to extend to the last coding unit of the coding unit sequence.

[0122] Step S303 will be executed if the similarity between the last coding unit in the current window and the next coding unit is less than the first similarity threshold, or if the window is extended to the last coding unit in the coding unit sequence. Therefore, after executing step S303, in order to determine whether there are any unprocessed coding units so that they can be aggregated, it is necessary to determine whether the current window has been extended to the last coding unit in the coding unit sequence; that is, to determine whether the last coding unit in the expanded current window is indeed the last coding unit in the coding unit sequence.

[0123] If the current window has not expanded to the last coding unit of the coding unit sequence, then subsequent coding units need to be processed, so step S305 is executed. If the current window has expanded to the last coding unit of the coding unit sequence, then step S306 is executed.

[0124] S305. Reset the position of the current window to the next encoding unit.

[0125] Because aggregation of the next coding unit is required, the current window position is reset to the next coding unit t. i+1 That is, at this time W={t i+1 Then return to step S302.

[0126] S306. End the aggregation of each coding unit.

[0127] S103. Poll each audio segment in turn and calculate the membership degree of each compressed coding unit of the current audio segment.

[0128] The membership degree of a compression coding unit includes its own acoustic saliency membership degree and its temporal correlation membership degree with each of the current fusion coding units. The current fusion coding units are the fusion coding units of the previous audio segment of the current audio segment, that is, the coding units after the previous audio segment has been fused.

[0129] To ensure the accuracy of audio segment understanding and analysis by considering prior information from previous audio segments without information loss, each coding unit of each audio segment needs to be fused with the coding units of the previous audio segment.

[0130] Specifically, in the embodiments of this application, "temporal correlation" and "acoustic saliency" are quantified by fuzzy set theory, namely acoustic saliency membership degree and temporal correlation membership degree. Based on the two quantified indicators, each current audio token can effectively integrate past audio tokens, which not only retains the current timeliness, but also incorporates historical context, reduces the illusion of a large model, and improves the accuracy of content understanding.

[0131] To fuse the current token and the token of the previous audio segment based on the contribution of the current compression coding unit and the fusion coding unit of the previous audio segment, the acoustic saliency membership degree of each compression coding unit is calculated. This degree reflects the contribution of the compression coding unit itself, while the remaining contribution is the contribution of the token of the previous audio segment.

[0132] Since the contribution of each token in the previous audio segment to the current token is different, the temporal correlation between each compression coding unit and each current fusion coding unit in the current audio is calculated, that is, the temporal correlation membership degree, so as to reflect the contribution of each fusion coding unit to the compression coding unit, and then perform fusion based on it.

[0133] Optionally, a memory repository G can be defined to store the token sequence of processed audio segments, that is, the sequence of fused coding units of processed audio segments, for processing the coding units of the current audio segment.

[0134] Optionally, in another embodiment of this application, a method for calculating the temporal association membership degree between a compression coding unit and each currently fused coding unit includes:

[0135] For each compression coding unit of the current audio segment, calculate the temporal association membership degree between the compression coding unit and each current fusion coding unit based on the time difference between the compression coding unit and each current fusion coding unit.

[0136] It should be noted that since the temporal association membership degree between the compression coding unit and the current fusion coding unit needs to be calculated, it can be calculated based on their time difference. The closer the times, the higher the association degree. Specifically, the temporal association membership degree can be calculated by modeling the time difference between the i-th token in the sequence of the fusion coding unit of the previous audio segment and the k-th token in the current sequence using a Gaussian function.

[0137] Specifically, the current sequence is selected sequentially, that is, the sequence of compressed coding units of the current audio segment is selected as follows. The k-th token is The sequence of fusion coding units for the previous audio segment is... The i-th token is .

[0138] Therefore, the temporal association degree between the i-th token in the sequence of the fusion coding units of the previous audio segment and the k-th token in the sequence of the compression coding units of the current audio segment is:

[0139] .

[0140] in, The absolute timestamp of the k-th token in the current sequence is calculated using the following formula:

[0141] .

[0142] in, This is the current start time. , This represents a unit timestamp.

[0143] The absolute timestamp of the i-th token in the sequence of the current fusion coding unit is calculated as follows:

[0144] .

[0145] , The set experience value is usually set to 5, which means it covers the historical association of 5 token windows to avoid interference from too distant memories.

[0146] Optionally, in another embodiment of this application, the method for calculating the acoustic saliency membership of the compression coding unit includes:

[0147] The acoustic saliency membership degree of the compressed coding unit is calculated based on the ratio of the acoustic energy of the compressed coding unit to the global maximum acoustic energy, and the ratio of the spectral entropy of the compressed coding unit to the global maximum spectral entropy.

[0148] Specifically, the acoustic saliency membership degree of any coding unit t in the current sequence is calculated, highlighting the contribution of key tokens. The specific calculation method is as follows:

[0149] .

[0150] in, This represents the acoustic energy of the coding unit. The stronger the signal it represents, the greater the probability that it is a keyframe. The specific calculation method is as follows:

[0151] .

[0152] The spectral entropy of this coding unit represents a higher probability of being a keyframe due to its higher complexity. Its specific calculation method is as follows:

[0153] .

[0154] and These are the global maximum acoustic energy and global maximum spectral entropy of the current sequence, respectively.

[0155] S104. Calculate the contribution weight of each compressed coding unit using the membership degree of each compressed coding unit.

[0156] The contribution weight of the compression coding unit includes its own contribution weight and the contribution weight of each current fused coding unit to the compression coding unit.

[0157] Specifically, the acoustic saliency membership degree of the compressed coding unit is used to calculate its own contribution weight. A higher acoustic saliency membership degree indicates greater importance, thus resulting in a larger contribution weight. Then, based on the temporal association membership degree between the compressed coding unit and each currently fused coding unit, the contribution weight of each currently fused coding unit to the compressed coding unit is calculated. A higher temporal association membership degree between the compressed coding unit and the currently fused coding unit indicates a greater influence of the currently fused coding unit on the compressed coding unit, thus resulting in a larger contribution weight.

[0158] Optionally, in another embodiment of this application, one specific implementation of step S104 is as follows: Figure 4 As shown, it includes:

[0159] S401. For each compression coding unit, multiply the temporal association membership degree between the compression coding unit and each current fusion coding unit by the acoustic saliency membership degree of the fusion coding unit to obtain the absolute contribution weight of each current fusion coding unit to the compression coding unit.

[0160] Therefore, the absolute contribution weight of the i-th current fusion coding unit to the k-th compression coding unit is:

[0161] .

[0162] in, It represents the acoustic saliency membership degree of the i-th current fusion coding unit, calculated in the same way as the acoustic saliency membership degree of the compressed coding unit.

[0163] S402. Normalize the absolute contribution weight of each current fusion coding unit to the compression coding unit to obtain the contribution weight of each current fusion coding unit to the compression coding unit.

[0164] Since the contribution of each current fusion coding unit to the compression coding unit is relative, a relative contribution weight is needed as the final contribution weight. Therefore, specifically, the absolute contribution weight of each current fusion coding unit to the compression coding unit is divided by the sum of the absolute contribution weights of all current fusion coding units to the compression coding unit, thus normalizing the absolute contribution weights and obtaining the contribution weight of each current fusion coding unit to the compression coding unit. Therefore, the contribution weight of the i-th current fusion coding unit to the k-th compression coding unit is:

[0165] .

[0166] S403. Add the unbalanced coefficient and the acoustic saliency membership degree of each compressed coding unit to the balance coefficient to obtain the self-contribution weight of each compressed coding unit.

[0167] The unbalance coefficient is 1 minus the balance coefficient.

[0168] It should be noted that the weight of the current audio segment's compression coding unit is complementary to the total contribution of the previous audio segment's fusion coding unit. Therefore, a balancing coefficient β is introduced to control the proportion of the current information. For example, it can be set to 0.6. Therefore, the self-contribution weight of each compression coding unit is:

[0169] .

[0170] S105. Using the contribution weight corresponding to each compression coding unit, perform a weighted summation on the compression coding unit and each current fusion coding unit to obtain each fusion coding unit of the current audio segment.

[0171] It should be noted that the contribution weights corresponding to the compression coding units include the weight of the compression coding unit's contribution to itself, i.e., its own contribution weight, as well as the contribution weights of each current fusion coding unit to the compression coding. Therefore, by using the contribution weights corresponding to each compression coding unit, a weighted sum can be made on the corresponding compression coding unit and each current fusion coding unit to obtain the coding unit that combines the compression coding unit of the current audio segment with the fusion coding unit of the previous audio segment, i.e., each fusion coding unit of the current audio segment, so that the coding unit of the current audio segment incorporates the leading information of the previous audio segment.

[0172] Optionally, in another embodiment of this application, one specific implementation of step S105 includes:

[0173] For each compression coding unit, the weighted sum of the compression coding unit and all current fusion coding units is weighted according to the compression coding unit's own contribution weight to obtain a fusion coding unit for the current audio segment. The weighted sum of all current fusion coding units is equal to the weighted sum of each current fusion coding unit and its contribution weight to the compression coding unit.

[0174] Specifically, the contribution weight of each compression coding unit is its own contribution weight, which is the weight of the compression coding unit. The remaining weight, which is the difference between 1 and the compression coding unit's own contribution weight, is the weight of the previous audio segment's fusion coding unit. Therefore, the contribution weight of the compression coding unit is used to weight the current and past coding units. Since the contribution of each current fusion coding unit is different, it is necessary to use the weights corresponding to each current fusion coding unit, that is, the contribution weights of each current fusion coding unit to the compression coding unit, to perform a weighted sum of the current fusion coding units before fusing them with the compression coding unit.

[0175] Therefore, the weighted summation of the k-th compression coding unit yields the fusion coding unit for the current audio segment as follows:

[0176] .

[0177] S106. Merge the individual fusion coding units of each audio segment, input them into the large language model, output the understanding and analysis results of each audio segment, and fuse them to obtain the understanding and analysis results of the target audio.

[0178] Specifically, for each audio segment, the various fusion coding units of that segment are merged to form a set of coding units that incorporate prior information. This fused set of audio tokens, along with the token sequence of the corresponding input text, is then input into the large language model. Based on the requirements of the input text's token sequence, the audio token set is processed to understand and analyze the audio segment, outputting the corresponding understanding and analysis results. Finally, the understanding and analysis results of all audio segments are fused to obtain the understanding and analysis result of the entire target audio.

[0179] Because the token set for each audio segment is compressed, removing redundant and invalid data, the amount of data to be processed is effectively reduced, thus improving the efficiency of understanding and analysis and ensuring the timeliness of audio understanding and analysis. Furthermore, the token sets for each audio segment incorporate information from the previous audio segment, effectively guaranteeing the accuracy of the understanding and analysis of each audio segment.

[0180] This application provides an audio understanding method that obtains each coding unit of each audio segment of a target audio file. Based on the information entropy of each coding unit in each audio segment, the method compresses the coding units of each audio segment using discrete cosine transform to obtain compressed coding units for each audio segment. This effectively filters redundant and invalid information, compresses the compressed coding units of the audio segments, and thus effectively reduces the amount of data to be processed while ensuring data accuracy, improving the efficiency of audio understanding and analysis, and ensuring its timeliness. Then, each audio segment is sequentially polled, and the membership degree corresponding to each compressed coding unit of the current audio segment is calculated. The membership degree of a compressed coding unit includes its own acoustic saliency membership degree and its temporal correlation membership degree with each currently fused coding unit. The current fused coding units are the fused coding units of the previous audio segment of the current audio segment, so the membership degree reflects the contribution of the current coding unit and past coding units to the current coding unit. Then, using the membership degree of each compressed coding unit, the contribution weight of each compressed coding unit is calculated. This contribution weight is then used to perform a weighted summation of the compressed coding units and each current fusion coding unit, resulting in the fusion coding units for the current audio segment. Based on the contribution level of each coding unit, the coding units of the current audio segment are fused with those of past audio segments, thus incorporating prior information from the previous audio segment. Finally, the fusion coding units of each audio segment are merged and input into a large language model. The output of the understanding and analysis results for each audio segment is then fused to obtain the understanding and analysis result for the target audio. Because the fusion coding units of each audio segment incorporate prior information from the previous audio segment, the accuracy of the understanding and analysis results is effectively guaranteed. Therefore, a method for efficient and accurate audio understanding and analysis is implemented.

[0181] Another embodiment of this application provides an audio understanding system, such as... Figure 5 As shown, it includes:

[0182] The data acquisition unit 501 is used to acquire each encoding unit of each audio segment of the target audio.

[0183] The audio compression unit 502 is used to compress the coding units of each audio segment based on the information entropy of each coding unit of each audio segment through discrete cosine transform, so as to obtain each compressed coding unit of each audio segment.

[0184] Membership calculation unit 503 is used to poll each audio segment in turn and calculate the membership degree of each compressed coding unit of the current audio segment.

[0185] The membership degree of a compression coding unit includes its own acoustic saliency membership degree and its temporal correlation membership degree with each of the current fusion coding units. The current fusion coding units are the fusion coding units of the previous audio segment of the current audio segment.

[0186] The weight calculation unit 504 is used to calculate the contribution weight of each compressed coding unit by utilizing the membership degree of each compressed coding unit.

[0187] Weighting unit 505 is used to perform weighted summation on the compression coding unit and each current fusion coding unit by utilizing the contribution weight corresponding to each compression coding unit, so as to obtain each fusion coding unit of the current audio segment.

[0188] The understanding and analysis unit 506 is used to merge the various fusion coding units of each audio segment, input them into the large language model, output the understanding and analysis results of each audio segment, and fuse them to obtain the understanding and analysis results of the target audio.

[0189] Optionally, in another embodiment of the audio understanding system provided in this application, the audio compression unit includes:

[0190] The information entropy calculation unit is used to calculate the initial information entropy of each coding unit of each audio segment.

[0191] The transform unit is used to perform a cosine discrete transform on the initial information entropy of each coding unit of the audio segment to obtain the coefficient matrix.

[0192] The coefficient update unit is used to update each coefficient in the coefficient matrix that is not sorted at the beginning and has a preset compression ratio to zero.

[0193] The inverse transform unit is used to perform an inverse discrete cosine transform on the updated coefficient matrix to obtain the updated information entropy of each coding unit.

[0194] The secondary compression unit is used to calculate the similarity between the initial information entropy and the updated information entropy of each coding unit, and to filter out coding units whose similarity is less than the first similarity threshold.

[0195] Aggregation units are used to aggregate consecutive and similar coding units among the remaining coding units based on the information entropy of the coding units, to obtain the various compressed coding units of the audio segment.

[0196] Optionally, in another embodiment of the audio understanding system provided in this application, the aggregation unit includes:

[0197] The initialization unit is used to initialize the current window's position as the first coding unit in the coding unit sequence. The coding unit sequence is the sequence of the remaining coding units.

[0198] An extension unit is used to continuously expand the current window based on the similarity of adjacent coding units until the similarity between the last coding unit in the current window and the next coding unit is less than a first similarity threshold, or to expand to the last coding unit in the coding unit sequence.

[0199] The window aggregation unit is used to aggregate the various coding units in the current window based on the information entropy of the coding units, to obtain a compressed coding unit.

[0200] The reset unit is used to reset the current window to its next coding unit and return to the expansion unit if the current window has not been expanded to the last coding unit of the coding unit sequence.

[0201] Optionally, in another embodiment of the audio understanding system provided in this application, the window aggregation unit includes:

[0202] The window aggregation subunit is used to divide the sum of the products of each coding unit in the current window and its information entropy by the sum of the information entropy of each coding unit, and add the product of the residual term and residual coefficient of the current window to obtain a compressed coding unit.

[0203] Optionally, in another embodiment of the audio understanding system provided in this application, the membership calculation unit includes:

[0204] The first membership calculation unit is used to calculate the temporal association membership degree between each compression coding unit and each current fusion coding unit for each compression coding unit of the current audio segment, based on the time difference between the compression coding unit and each current fusion coding unit.

[0205] The second membership calculation unit is used to calculate the acoustic saliency membership of the compressed coding unit based on the ratio of the acoustic energy of the compressed coding unit to the global maximum acoustic energy, and the ratio of the spectral entropy of the compressed coding unit to the global maximum spectral entropy.

[0206] Optionally, in another embodiment of the audio understanding system provided in this application, the weight calculation unit includes:

[0207] The first weight calculation unit is used to multiply the temporal association membership degree between the compression coding unit and each current fusion coding unit by the acoustic saliency membership degree of the fusion coding unit for each compression coding unit, so as to obtain the absolute contribution weight of each current fusion coding unit to the compression coding unit.

[0208] The second weight calculation unit is used to normalize the absolute contribution weight of each current fusion coding unit to the compression coding unit, so as to obtain the contribution weight of each current fusion coding unit to the compression coding unit.

[0209] The third weighting calculation unit is used to add the unbalanced coefficient and the acoustic saliency membership degree of each compressed coding unit to the balance coefficient, respectively, to obtain the self-contribution weight of each compressed coding unit. Among them, the unbalanced coefficient is 1 minus the balance coefficient.

[0210] Optionally, in another embodiment of the audio understanding system provided in this application, the weighting unit includes:

[0211] The weighted subunit is used to weight the sum of the weighted sums of the compression coding units and all current fusion coding units according to the contribution weight of each compression coding unit, to obtain a fusion coding unit for the current audio segment. The weighted sum of all current fusion coding units is equal to the weighted sum of each current fusion coding unit and its contribution weight to the compression coding unit.

[0212] It should be noted that the specific working process of each unit provided in the above embodiments of this application can be referred to the implementation process of the corresponding steps in the above method embodiments, and will not be repeated here.

[0213] Another embodiment of this application provides an electronic device, such as... Figure 6 As shown, it includes:

[0214] Memory 601 and processor 602.

[0215] The memory 601 is used to store the program.

[0216] The processor 602 is used to execute a program stored in the memory 601, which, when executed, is specifically used to implement the audio understanding method provided in any of the above embodiments.

[0217] Another embodiment of this application provides a computer storage medium for storing a computer program, which, when executed by a processor, is used to implement the audio understanding method provided in any of the above embodiments.

[0218] Computer storage media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0219] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0220] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An audio understanding method, characterized in that, include: Obtain each coding unit of each audio segment of the target audio; Based on the information entropy of each coding unit of each audio segment, the coding units of each audio segment are compressed by discrete cosine transform to obtain each compressed coding unit of each audio segment. Each audio segment is polled sequentially, and the membership degree corresponding to each compressed coding unit of the current audio segment is calculated; wherein, the membership degree corresponding to the compressed coding unit includes its own acoustic saliency membership degree and its temporal correlation membership degree with each current fusion coding unit; the current fusion coding unit is each fusion coding unit of the previous audio segment of the current audio segment; The contribution weight of each compression coding unit is calculated using the membership degree of each compression coding unit. By utilizing the contribution weight corresponding to each compression coding unit, a weighted sum is performed on the compression coding unit and each of the current fusion coding units to obtain the fusion coding units of the current audio segment; The fusion coding units of each audio segment are merged and input into a large language model. The understanding and analysis results of each audio segment are output and fused to obtain the understanding and analysis results of the target audio.

2. The method according to claim 1, characterized in that, The compression of each audio segment's coding unit based on the information entropy of each coding unit of each audio segment, using discrete cosine transform, yields each compressed coding unit of each audio segment, including: For each of the audio segments, calculate the initial information entropy of each coding unit of the audio segment; The initial information entropy of each coding unit of the audio segment is subjected to a cosine discrete transform to obtain a coefficient matrix; Update all coefficients in the coefficient matrix that are not sorted at the beginning of the preset compression ratio to zero; The updated coefficient matrix is ​​subjected to inverse discrete cosine transform to obtain the updated information entropy of each coding unit; Calculate the similarity between the initial information entropy and the updated information entropy of each coding unit, and filter out each coding unit whose similarity is less than a first similarity threshold; Based on the information entropy of the coding unit, the remaining consecutive and similar coding units are aggregated to obtain the various compressed coding units of the audio segment.

3. The method according to claim 2, characterized in that, The aggregation of consecutive and similar coding units among the remaining coding units, based on the information entropy of the coding unit, to obtain the various compressed coding units of the audio segment includes: The current window position is initialized to the first coding unit in the coding unit sequence; wherein, the coding unit sequence is the sequence of the remaining coding units; Based on the similarity of adjacent coding units, the current window is continuously expanded until the similarity between the last coding unit in the current window and the next coding unit is less than a first similarity threshold, or it is expanded to the last coding unit in the coding unit sequence. Based on the information entropy of the coding unit, the coding units in the current window are aggregated to obtain a compression coding unit; If the current window does not expand to the last coding unit in the coding unit sequence, the current window is reset to its next coding unit, and the process of continuously expanding the current window based on the similarity of adjacent coding units is returned.

4. The method according to claim 3, characterized in that, The aggregation of the various coding units in the current window based on the information entropy of the coding unit to obtain a single compression coding unit includes: The sum of the products of each coding unit in the current window and its information entropy is divided by the sum of the information entropy of each coding unit, and the quotient is added to the product of the residual term and the residual coefficient of the current window to obtain a compressed coding unit.

5. The method according to claim 1, characterized in that, The calculation of the membership degree corresponding to each compressed coding unit of the current audio segment includes: For each compression coding unit of the current audio segment, the temporal association membership degree between the compression coding unit and each current fusion coding unit is calculated based on the time difference between the compression coding unit and each current fusion coding unit. The acoustic saliency membership degree of the compressed coding unit is calculated based on the ratio of the acoustic energy of the compressed coding unit to the global maximum acoustic energy, and the ratio of the spectral entropy of the compressed coding unit to the global maximum spectral entropy.

6. The method according to claim 1, characterized in that, The step of calculating the contribution weight of each compression coding unit using the membership degree of each compression coding unit includes: For each of the compression coding units, the temporal association membership degree between the compression coding unit and each of the current fusion coding units is multiplied by the acoustic saliency membership degree of the fusion coding unit to obtain the absolute contribution weight of each of the current fusion coding units to the compression coding unit. The absolute contribution weight of each current fusion coding unit to the compression coding unit is normalized to obtain the contribution weight of each current fusion coding unit to the compression coding unit. The balance coefficient is added to the unbalance coefficient and the acoustic saliency membership degree of each compression coding unit to obtain the self-contribution weight of each compression coding unit; wherein the unbalance coefficient is 1 minus the balance coefficient.

7. The method according to claim 1, characterized in that, The step of using the contribution weight corresponding to each of the compression coding units to perform a weighted summation of the compression coding unit and each of the current fusion coding units to obtain the fusion coding units of the current audio segment includes: For each of the compression coding units, the weighted sum of the compression coding unit and each of the current fusion coding units is weighted according to the contribution weight of the compression coding unit itself to obtain a fusion coding unit for the current audio segment; wherein, the weighted sum of each of the current fusion coding units is equal to the result of weighted summation of each current fusion coding unit and its contribution weight to the compression coding unit.

8. An audio understanding system, characterized in that, include: The data acquisition unit is used to acquire each encoding unit of each audio segment of the target audio. An audio compression unit is used to compress the coding units of each audio segment based on the information entropy of each coding unit of each audio segment through discrete cosine transform, so as to obtain each compressed coding unit of each audio segment. The membership calculation unit is used to sequentially poll each of the audio segments and calculate the membership degree corresponding to each compressed coding unit of the current audio segment; wherein, the membership degree corresponding to the compressed coding unit includes its own acoustic saliency membership degree and its temporal correlation membership degree with each current fusion coding unit; the current fusion coding unit is each fusion coding unit of the previous audio segment of the current audio segment; The weight calculation unit is used to calculate the contribution weight of each compressed coding unit by using the membership degree of each compressed coding unit respectively. A weighting unit is used to perform a weighted summation of the compression coding unit and each of the current fusion coding units using the contribution weight corresponding to each compression coding unit, so as to obtain each fusion coding unit of the current audio segment; The understanding and analysis unit is used to merge the various fusion coding units of each audio segment, input them into the large language model, output the understanding and analysis results of each audio segment, and fuse them to obtain the understanding and analysis results of the target audio.

9. An electronic device, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is used to execute the program, which, when executed, is specifically used to implement the audio understanding method as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, is used to implement the audio understanding method as described in any one of claims 1 to 7.

Images