An audio data processing system and method

Abstract

The application provides an audio data processing system and method, and relates to the technical field of digital audio processing.The system comprises: a feature extraction unit for obtaining audio data and performing multi-modal feature extraction to obtain a plurality of modal features; a feature coding unit for dimension reduction processing and coding the dimension-reduced modal features to obtain a feature code stream corresponding to each modal; a compression strategy unit for predicting the feature code stream of each modal feature to obtain a complexity value of the feature code stream, and determining an information loss risk value of the modal feature in combination with a compression requirement of the audio data to adjust the compression rate of each modal feature in the audio data; obtaining a compression strategy of the audio data according to the compression rates of all the modal features; and a data processing unit for compressing the feature code stream of each modal feature according to the compression strategy to obtain a compressed file of the audio data.The application flexibly determines a suitable compression mode for each kind of audio, and improves the compression effect of the audio.

Description

Technical Field

[0001] This invention relates to the field of digital audio processing technology, and more specifically, to an audio data processing system and method. Background Technology

[0002] With the widespread adoption of digital music, the demand for audio data storage and transmission is constantly increasing. Traditional audio data compression technology mainly reduces the amount of audio data by analyzing and compressing the audio spectrum. It typically utilizes the auditory masking effect of the human ear, converting the audio signal to the frequency domain based on the audio spectrum and removing frequency components that are not easily perceived by the human ear, thereby achieving compression.

[0003] In related technologies, for audio data with high sound quality requirements, such as high-quality music audio data, traditional audio data compression technology relies solely on the single modal characteristics of the audio spectrum for compression. This leads to the easy loss of a large amount of musical details in the music data during the compression process, resulting in distortion in the restored music and failing to meet users' demand for high-fidelity music. Summary of the Invention

[0004] The problem solved by this invention is how to improve the compression effect of audio data while ensuring the sound quality after decompression.

[0005] To address the above problems, the present invention provides an audio data processing system and method.

[0006] In a first aspect, the present invention provides an audio data processing system comprising: The feature extraction unit is used to acquire audio data and perform multimodal feature extraction on the audio data to obtain multiple modal features of the audio data; The feature encoding unit is used to perform dimensionality reduction processing on the modal features of each modality using a preset dimensionality reduction algorithm, and to encode the dimensionality-reduced modal features to obtain the feature code stream corresponding to each modality; A compression strategy unit is used to predict the complexity value of the feature bitstream of each modal feature using a deep learning model; determine the information loss risk value of the modal feature based on the complexity value of the feature bitstream and the compression requirements of the audio data; dynamically adjust the compression ratio of each modal feature in the audio data based on the information loss risk value; and obtain the compression strategy of the audio data based on the compression ratio of all modal features. The data processing unit is used to compress the feature bitstream of each modality feature according to the compression strategy to obtain a compressed file of the audio data.

[0007] Optionally, the feature extraction unit is specifically used for: Obtain the raw audio data, perform preprocessing on the raw audio data to obtain the preprocessed raw audio data, and use the preprocessed raw audio data as the audio data; The audio data is converted from the time domain to the frequency domain by using a short-time Fourier transform to obtain the spectral characteristics of the audio data. The rhythmic characteristics of the audio data are obtained by calculating the zero-crossing rate based on the audio data. The pitch sequence of the audio data is extracted using a pitch detection algorithm, and the melody features of the audio data are determined based on the pitch sequence. Chord recognition is performed on the audio data to obtain the harmonic features of the audio data; The spectral features, rhythmic features, melodic features, and harmonic features are used as the modal features.

[0008] Optionally, the feature encoding unit is specifically used for: Each modal feature is standardized to obtain the standardized modal features; Principal component analysis is performed on the standardized modal features to obtain the covariance matrix of the modal features; The modal features are obtained by projecting them onto the covariance matrix of the modal features; Probability calculations are performed on the dimensionality-reduced modal features to obtain the probability of occurrence of each feature value in each modal feature; Based on the occurrence probability, the probability distribution of the modal feature is obtained; A Huffman tree is constructed based on the probability distribution, and the feature code stream corresponding to the modality feature is obtained by assigning a Huffman code to each feature value.

[0009] Optionally, the deep learning model includes a long short-term memory layer and a fully connected layer, and the compression strategy unit is specifically used for: The feature stream of each modality feature is divided into multiple input sequences according to the window size of the deep learning model; The long short-term memory layer captures long-term dependencies for each input sequence to obtain the hidden state sequence of the input sequence, and uses the hidden state sequence as the output sequence of the long short-term memory layer. The complexity value of the feature code stream is obtained by performing feature integration and nonlinear transformation on all the output sequences through the fully connected layer.

[0010] Optionally, the compression strategy unit is further configured to: The initial information loss risk value of the modal feature is obtained by mapping the complexity value of the feature code stream. Based on the compression requirements, determine the calibration coefficient for the initial information loss risk value; The initial information loss risk value is updated based on the calibration coefficient to obtain the information loss risk value of the modal feature.

[0011] Optionally, the compression strategy unit is further configured to: Based on the compression requirements, determine the general compression ratio of the audio data; Based on the information loss risk value of each modal feature, determine the importance weight of the modal feature in the audio data; The compression ratio of each modal feature is obtained by combining the importance weight of each modal feature with the general compression ratio.

[0012] Optionally, the compression strategy unit is further configured to: Based on the compression ratio of all the modal features, a compression sub-strategy corresponding to each modal feature is obtained; The compression sub-strategies of all the modal features are integrated to obtain the compression strategy of the audio data.

[0013] Optionally, the data processing unit is specifically used for: The feature bitstream of each modality feature is compressed according to the compression strategy through a distributed network to obtain a compressed file of the audio data. The distributed network includes multiple nodes. Each node is assigned a feature code stream with a different modal feature. By using each node, compression is performed according to the compression sub-strategy of the modal feature corresponding to the node, resulting in a compressed file corresponding to the compressed modal feature; The compressed files corresponding to each modal feature are integrated to obtain the compressed file of the audio data.

[0014] Optionally, the system further includes a recovery unit, the recovery unit being used for: Based on the compression ratio of each modal feature in the audio data, the compressed file of each modal feature is decompressed and decoded to obtain the decompressed and decoded modal feature; The audio data is recovered by reconstructing all the decompressed and decoded modal features using an audio synthesis model.

[0015] Secondly, the present invention provides an audio data processing method, comprising: Acquire audio data and perform multimodal feature extraction on the audio data to obtain multiple modal features of the audio data; The modal features of each modality are reduced using a preset dimensionality reduction algorithm, and the reduced modal features are encoded to obtain the feature code stream corresponding to each modality. The complexity value of the feature code stream for each modality feature is obtained by predicting the feature code stream based on the feature code stream of each modality feature using a deep learning model. Based on the complexity value of the feature bitstream and the compression requirements of the audio data, the information loss risk value of the modal feature is determined; Based on the information loss risk value, the compression rate of each modal feature in the audio data is dynamically adjusted; Based on the compression ratio of all the modal features, the compression strategy of the audio data is obtained; The feature bitstream of each modality feature is compressed according to the compression strategy to obtain the compressed file of the audio data.

[0016] In the audio data processing system of this invention, the feature extraction unit provides the feature encoding unit with a rich variety of features. Traditional audio data compression techniques rely solely on the single modal feature of the audio spectrum, easily losing a significant amount of musical detail. This invention, by introducing a feature extraction unit, can acquire multiple modal features of the audio data, allowing for observation and understanding of the audio data from multiple perspectives, rather than being limited to the spectrum alone, thus preserving more musical detail. For example, in addition to spectral features, rhythmic features (such as beat and rhythmic patterns) and timbre features (such as the timbre differences between different instruments) can also be extracted. The feature encoding unit utilizes these multimodal features for dimensionality reduction and encoding, converting high-dimensional and complex features into a feature bitstream. The extraction of multimodal features enables dimensionality reduction and encoding to more accurately preserve key information. For example, for music containing rich harmonies, the extraction of timbre features helps preserve subtle timbre differences between different instruments during encoding, thus allowing for a more realistic restoration of the music's rich layers during subsequent decoding and reconstruction. This provides a high-quality, information-rich feature bitstream foundation for subsequent compression strategies.

[0017] The feature bitstream generated by the feature encoding unit serves as input to the compression strategy unit. For each modality's features, the feature encoding unit performs dimensionality reduction using a pre-defined dimensionality reduction algorithm before encoding to obtain the feature bitstream. Dimensionality reduction removes redundant information, preparing for subsequent compression to some extent, while the encoding process transforms the features into a form more suitable for compression. The compression strategy unit predicts the complexity of the feature bitstream using a deep learning model. The complexity value reflects the amount of information and complexity contained in the feature bitstream for that modality. For example, for music containing complex melodic variations, the complexity value of the feature bitstream for its melodic modality may be high. Then, combined with the compression requirements of the audio data, such as the target compression ratio and the audio quality loss threshold, the information loss risk value is determined. This is equivalent to performing a risk assessment on the feature bitstream of each modality. If the feature bitstream complexity of a modality is high and the compression requirements are strict (such as requiring minimal audio quality loss), then its information loss risk value is high. The compression strategy unit dynamically adjusts the compression ratio of each modality based on this risk value. For feature streams with high complexity and a high risk of information loss, a lower compression ratio is assigned to reduce information loss; for feature streams with low complexity and a low risk of information loss, a higher compression ratio can be assigned. This application makes the entire compression process more refined and efficient through risk-based dynamic compression ratio adjustment. For example, for modal features of background noise (which are generally low in complexity and have a low risk of information loss), a higher compression ratio can be used, while for modal features of the main melody of music (which are high in complexity and have a high risk of information loss), a lower compression ratio is used. This ensures the overall compression effect while preserving the main details of the music to the greatest extent and reducing distortion.

[0018] The data processing unit ultimately performs actual compression operations on the feature stream of each modality according to the compression strategy, outputting a compressed audio file. This ensures that the final compression effect not only theoretically meets the requirements of high quality and high efficiency, but also achieves the expected results in terms of actual file size and sound quality restoration. For example, the feature stream, after multimodal extraction, encoding, and strategy adjustment, is compressed into a file that meets the user's expectations under the action of the data processing unit, enabling both fast storage and transmission, and providing a high-quality music experience after decompression. Through the close correlation between various technical features, this invention achieves a significant improvement in sound quality fidelity while increasing compression efficiency, effectively solving the problems existing in traditional audio data compression technologies.

[0019] From the perspective of data compression efficiency, the technical solution of this application, through multimodal feature extraction and subsequent targeted compression, can effectively reduce the storage size and transmission bandwidth of audio data while ensuring sound quality. Compared with traditional single-modal compression technology, it can achieve a higher compression ratio. For example, for high-quality music files, traditional methods may only achieve a certain compression ratio, while this invention can further increase the compression ratio without significantly reducing sound quality, making the audio file smaller, easier to store, and faster to transmit. Furthermore, it fully considers multimodal features and dynamically adjusts the compression ratio to reduce the risk of information loss, ensuring better sound quality of the restored music. It can retain more musical details, such as subtle timbre changes of instruments and nuances of vocals. When users play audio compressed by this invention, they can experience a listening experience closer to the original high-quality music, satisfying their demand for high-fidelity music. This system can dynamically adjust the compression strategy according to different audio data and compression requirements. Different types of audio (such as classical music, pop music, electronic music, etc.) may have different modal characteristics and sensitivities to compression. This invention addresses these differences by flexibly determining a suitable compression method for each type of audio, thereby improving the compression effect and demonstrating strong adaptability. Attached Figure Description

[0020] Figure 1 This is one of the structural block diagrams of the audio data processing system according to an embodiment of the present invention; Figure 2 This is a second structural block diagram of the audio data processing system according to an embodiment of the present invention; Figure 3 This is a flowchart of an audio data processing method according to an embodiment of the present invention. Detailed Implementation

[0021] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Although some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the accompanying drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0022] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0023] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to"; the term "based on" means "at least partially based on"; the term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments"; and the term "optionally" means "optional embodiments". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first," "second," etc., mentioned in this invention are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0024] It should be noted that the terms "a" and "a plurality of" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0025] The names of the messages or information exchanged between the multiple devices in the embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0026] In today's digital age, digital music has become an indispensable part of people's daily lives. From the widespread use of online music streaming platforms to the continuous expansion of personal digital music libraries, the demand for audio data storage and transmission has exploded. This trend has placed higher demands on audio data compression technology.

[0027] Traditional audio data compression techniques primarily rely on audio spectrum analysis. The basic principle is to utilize the auditory masking effect of the human ear, converting the audio signal to the frequency domain and removing frequency components that are not easily perceived by the human ear, thereby reducing the data size. For example, MP3 compression technology is based on this principle, analyzing the audio spectrum to remove frequency components that are outside the range of human hearing or masked by other sounds, thus achieving compression.

[0028] However, as users' demands for sound quality continue to rise, especially the growing need for high-quality music, the limitations of traditional audio data compression technologies are becoming increasingly apparent. Traditional techniques rely solely on the single modal characteristic of the audio spectrum for compression, leading to the loss of a significant amount of musical detail during the compression process. For example, when compressing classical music, traditional techniques may remove subtle instrumental harmonies and background sound effects—details crucial to the overall expressiveness and emotional delivery of the music.

[0029] When these audio data with lost details are recovered, the music exhibits distortion. This manifests as a monotonous timbre, a reduced dynamic range, and a diminished sense of space. For example, music processed using traditional compression techniques may lose the crispness of high frequencies, and low frequencies may become muddy, significantly reducing the overall depth and three-dimensionality of the music. This distortion fails to meet users' demands for high-fidelity music, especially in scenarios such as professional music production, music appreciation, and the use of high-end audio equipment, where users have even more stringent requirements for audio quality.

[0030] Furthermore, with the diversification and complexity of audio content, single-modal compression methods struggle to handle different types of audio data. For example, for modern pop music containing complex vocals, instrumental ensembles, and special sound effects, traditional compression techniques may inadvertently remove some important sound components, resulting in damage to the integrity and artistry of the music. Moreover, as the volume of audio data continues to increase, traditional compression techniques face challenges in balancing compression efficiency and compression quality.

[0031] To address the problems existing in the aforementioned related technologies, this embodiment provides an audio data processing system and method.

[0032] Combination Figure 1 As shown, an audio data processing system provided in this embodiment of the invention includes: The feature extraction unit is used to acquire audio data and perform multimodal feature extraction on the audio data to obtain multiple modal features of the audio data.

[0033] Specifically, the feature extraction unit is used to acquire audio data and perform multimodal feature extraction. It receives audio data from various sources, such as reading from audio files (commonly MP3 and WAV formats) or real-time acquisition of audio streams input from microphones. Taking a live concert as an example, this unit can acquire live recorded audio data, analyze the audio data, extract multiple modal features, convert the audio signal to the frequency domain, and calculate the amplitude and phase information of different frequency components. For example, in a symphony performance, analysis can reveal the distribution of frequency components of each instrument, such as the high-frequency harmonic components of the violin and the low-frequency fundamental frequency components of the cello. It also identifies beat and rhythm patterns in audio. For example, in a pop song, it can extract the rhythmic pattern of drum beats, determine the beats per minute (BPM), and identify the positions of strong and weak beats. If the song is in 4 / 4 time, it can identify the four beat positions in each measure and rhythmic characteristics such as accents on the first and third beats. Finally, it analyzes the timbre differences between different instruments or vocals. Taking the same melody played on the piano and guitar as an example, the feature extraction unit can extract the crisp and bright tone and rich overtones of the piano and the warm and soft tone and woody texture of the guitar by analyzing the audio signal.

[0034] The feature encoding unit is used to perform dimensionality reduction processing on the modal features of each modality using a preset dimensionality reduction algorithm, and to encode the dimensionality-reduced modal features to obtain the feature code stream corresponding to each modality.

[0035] Specifically, the feature encoding unit receives multiple modal features output by the feature extraction unit, performs dimensionality reduction processing and encoding on these features using a preset dimensionality reduction algorithm, and converts the high-dimensional and complex features into a feature code stream, which facilitates subsequent compression operations.

[0036] For example, common dimensionality reduction algorithms include Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA). Taking PCA as an example, for the extracted spectral features (which may contain amplitude information at thousands of frequency points), PCA can identify the main directions of change at these frequency points, mapping high-spectral features to a low-dimensional space. For instance, spectral features that originally had 1000 frequency points may only require 100 dimensions of features after dimensionality reduction using PCA to retain most of the key information.

[0037] The dimensionality-reduced features are encoded to generate a feature bitstream. For example, Huffman coding, an isoentropy coding method, can be used. Assuming the dimensionality-reduced rhythmic feature data is a series of discrete values, Huffman coding assigns different length codes based on the frequency of each value; frequently occurring values ​​are represented by shorter codes, and vice versa. This converts the dimensionality-reduced features into a feature bitstream, facilitating subsequent transmission and storage, and also providing a data format for further processing by the data processing unit.

[0038] The compression strategy unit is used to predict the complexity value of the feature bitstream of each modal feature by using a deep learning model; determine the information loss risk value of the modal feature based on the complexity value of the feature bitstream and the compression requirements of the audio data; dynamically adjust the compression ratio of each modal feature in the audio data based on the information loss risk value; and obtain the compression strategy of the audio data based on the compression ratio of all modal features.

[0039] Specifically, the compression strategy unit is the core decision-making module of this system. By using a deep learning model to analyze the feature bitstream, it determines the compression rate of each modal feature based on the complexity of the feature bitstream and the compression requirements of the audio data, thereby formulating a reasonable audio data compression strategy.

[0040] Specifically, deep learning models (such as trained neural networks) are used to analyze the feature streams of each modality and predict their complexity values. The complexity value reflects the richness and complexity of the information in the feature stream. For example, for a classical music passage containing complex harmonies and rapid rhythmic changes, the complexity values ​​of its timbre feature stream, rhythm feature stream, and spectral feature stream may all be high; while for a simple piece of pure music (such as a long note played by a single instrument), the complexity values ​​of each modality's feature stream are relatively low.

[0041] Based on the complexity value of the feature bitstream and the compression requirements of the audio data (such as target compression ratio, audio quality loss limits, etc.), the information loss risk value of each modal feature is determined. For example, in cases with high compression requirements (such as needing to reduce the audio file size to half its original size), if the complexity value of a certain modal feature is high, then over-compression of it during the compression process may lead to significant information loss, thus determining that it has a high information loss risk value.

[0042] The compression ratio of each modal feature is dynamically adjusted based on its information loss risk value. Modal features with a high information loss risk value are assigned a lower compression ratio to reduce information loss; conversely, modal features with a low information loss risk value can be assigned a higher compression ratio. For example, in the compression of the classical music excerpt mentioned above, for timbre features with high complexity and a high risk of information loss, a relatively high fidelity is maintained, and a lower compression ratio is used; while for relatively simple rhythmic background features, the compression ratio is appropriately increased. In this way, by combining the compression ratios of all modal features, an audio data compression strategy is obtained, ensuring that the compressed audio meets file size requirements while preserving as much of the original audio's sound quality and detail as possible.

[0043] The data processing unit is used to compress the feature bitstream of each modality feature according to the compression strategy to obtain a compressed file of the audio data.

[0044] Specifically, the data processing unit formulates a compression strategy based on the compression strategy unit, performs actual compression operations on the feature bitstream of each modality feature, and finally generates a compressed audio data file.

[0045] Specifically, according to the compression ratio of each modal feature determined by the compression strategy unit, the corresponding compression algorithm is used to compress the feature bitstream. Common compression algorithms include arithmetic coding and the LZ77 series compression algorithms. For example, for a timbre feature bitstream, if the compression strategy determines its compression ratio to be 30%, then a suitable compression algorithm is used to compress the timbre feature bitstream, removing redundant information and reducing the data volume. During the compression process, compression efficiency and information preservation are balanced according to the compression ratio requirements to ensure that the compressed feature bitstream can restore the quality of the original audio as much as possible after decompression.

[0046] The compressed bitstreams of various modal features are integrated and encapsulated to generate the final compressed audio data file. For example, the compressed spectral feature bitstream, rhythm feature bitstream, timbre feature bitstream, etc., are combined according to a certain file format (such as a designed dedicated audio compression format), and necessary file header information (such as compression algorithm identifier, parameters of each modal feature, etc.) is added to form a complete, decompressible, and playable audio file. Compared with the original audio file, this compressed file significantly reduces storage space and transmission bandwidth requirements. At the same time, due to the use of multimodal feature processing and targeted compression strategies, the original audio can be restored with high fidelity after decompression, meeting users' needs for high-quality audio. The audio data processing system provided in this embodiment of the invention effectively solves the problem of musical detail loss and sound quality distortion caused by traditional audio compression technology relying solely on a single modal feature through the collaborative work of the above units. It achieves high-quality audio data compression and adapts to the high-efficiency and high-fidelity requirements of audio storage and transmission in the digital music era.

[0047] The audio data processing system of this invention provides a rich and diverse set of features to the feature encoding unit through its feature extraction unit. Traditional audio data compression techniques rely solely on the single modal feature of the audio spectrum, easily resulting in the loss of significant musical details. This invention, however, introduces a feature extraction unit that can acquire multiple modal features of the audio data, allowing for observation and understanding of the audio data from multiple perspectives, rather than being limited to the spectrum alone, thus preserving more musical detail. For example, in addition to spectral features, rhythmic features (such as beat and rhythmic patterns) and timbre features (such as the timbre differences between different instruments) can also be extracted. The feature encoding unit utilizes these multimodal features for dimensionality reduction and encoding, converting high-dimensional and complex features into a feature bitstream. The extraction of multimodal features enables dimensionality reduction and encoding to more accurately preserve key information. For instance, for music containing rich harmonies, the extraction of timbre features helps preserve subtle timbre differences between different instruments during encoding, allowing for a more realistic restoration of the music's rich layers during subsequent decoding and reconstruction. This provides a high-quality, information-rich feature bitstream foundation for subsequent compression strategies.

[0048] The feature bitstream generated by the feature encoding unit serves as input to the compression strategy unit. For each modality's features, the feature encoding unit performs dimensionality reduction using a pre-defined dimensionality reduction algorithm before encoding to obtain the feature bitstream. Dimensionality reduction removes redundant information, preparing for subsequent compression to some extent, while the encoding process transforms the features into a form more suitable for compression. The compression strategy unit predicts the complexity of the feature bitstream using a deep learning model. The complexity value reflects the amount of information and complexity contained in the feature bitstream for that modality. For example, for music containing complex melodic variations, the complexity value of the feature bitstream for its melodic modality may be high. Then, combined with the compression requirements of the audio data, such as the target compression ratio and the audio quality loss threshold, the information loss risk value is determined. This is equivalent to performing a risk assessment on the feature bitstream of each modality. If the feature bitstream complexity of a modality is high and the compression requirements are strict (such as requiring minimal audio quality loss), then its information loss risk value is high. The compression strategy unit dynamically adjusts the compression ratio of each modality based on this risk value. For feature streams with high complexity and a high risk of information loss, a lower compression ratio is assigned to reduce information loss; for feature streams with low complexity and a low risk of information loss, a higher compression ratio can be assigned. This application makes the entire compression process more refined and efficient through risk-based dynamic compression ratio adjustment. For example, for modal features of background noise (which are generally low in complexity and have a low risk of information loss), a higher compression ratio can be used, while for modal features of the main melody of music (which are high in complexity and have a high risk of information loss), a lower compression ratio is used. This ensures the overall compression effect while preserving the main details of the music to the greatest extent and reducing distortion.

[0049] The data processing unit ultimately performs actual compression operations on the feature stream of each modality according to the compression strategy, outputting a compressed audio file. This ensures that the final compression effect not only theoretically meets the requirements of high quality and high efficiency, but also achieves the expected results in terms of actual file size and sound quality restoration. For example, the feature stream, after multimodal extraction, encoding, and strategy adjustment, is compressed into a file that meets the user's expectations under the action of the data processing unit, enabling both fast storage and transmission, and providing a high-quality music experience after decompression. Through the close correlation between various technical features, this invention achieves a significant improvement in sound quality fidelity while increasing compression efficiency, effectively solving the problems existing in traditional audio data compression technologies.

[0050] From the perspective of data compression efficiency, the technical solution of this application, through multimodal feature extraction and subsequent targeted compression, can effectively reduce the storage size and transmission bandwidth of audio data while ensuring sound quality. Compared with traditional single-modal compression technology, it can achieve a higher compression ratio. For example, for high-quality music files, traditional methods may only achieve a certain compression ratio, while this invention can further increase the compression ratio without significantly reducing sound quality, making the audio file smaller, easier to store, and faster to transmit. Furthermore, it fully considers multimodal features and dynamically adjusts the compression ratio to reduce the risk of information loss, ensuring better sound quality of the restored music. It can retain more musical details, such as subtle timbre changes of instruments and nuances of vocals. When users play audio compressed by this invention, they can experience a listening experience closer to the original high-quality music, satisfying their demand for high-fidelity music. This system can dynamically adjust the compression strategy according to different audio data and compression requirements. Different types of audio (such as classical music, pop music, electronic music, etc.) may have different modal characteristics and sensitivities to compression. This invention addresses these differences by flexibly determining a suitable compression method for each type of audio, thereby improving the compression effect and demonstrating strong adaptability.

[0051] Optionally, the feature extraction unit is specifically used for: Obtain the raw audio data, perform preprocessing on the raw audio data to obtain the preprocessed raw audio data, and use the preprocessed raw audio data as the audio data; The audio data is converted from the time domain to the frequency domain by using a short-time Fourier transform to obtain the spectral characteristics of the audio data. The rhythmic characteristics of the audio data are obtained by calculating the zero-crossing rate based on the audio data. The pitch sequence of the audio data is extracted using a pitch detection algorithm, and the melody features of the audio data are determined based on the pitch sequence. Chord recognition is performed on the audio data to obtain the harmonic features of the audio data; The spectral features, rhythmic features, melodic features, and harmonic features are used as the modal features.

[0052] Specifically, the feature extraction unit first acquires the raw audio data, which may originate from various sources, such as audio files or real-time audio streams from microphone input. To ensure accuracy in subsequent processing, preprocessing is performed on the raw audio data. Preprocessing includes removing background noise and normalizing volume. For example, in recorded live concert audio, there may be excessive background noise; preprocessing can reduce this noise interference and adjust the volume to bring the audio data within a suitable range, resulting in preprocessed raw audio data, which is then used as the audio data for subsequent processing. The audio data is then converted from the time domain to the frequency domain using a Short-Time Fourier Transform (STFT), thus obtaining the spectral characteristics of the audio data. For example, for a piece of music played by multiple instruments, STFT can decompose the audio signal into the relationship between different frequency components and time. This clearly shows the amplitude of the frequency components corresponding to each instrument in each time period of the music, such as the harmonic components of the violin in the high-frequency range and their intensity variations during performance. Zero-crossing rate calculation is performed on the audio data to obtain its rhythmic characteristics. For example, the zero-crossing rate refers to the number of times the audio signal crosses zero per unit time. The calculation steps are as follows: First, the audio signal is preprocessed, including removing the DC component and normalization. Removing the DC component eliminates the DC offset in the signal, preventing it from affecting the zero-crossing rate calculation. Normalization adjusts the signal amplitude to a suitable range, typically [-1, 1]. Then, zero-crossing detection is performed on the preprocessed audio signal. Specifically, the method involves detecting whether adjacent sampling points of the signal have opposite signs. If the signs are opposite, it is determined that a zero-crossing point exists between these two sampling points. The number of zero-crossing points within a certain time period is counted and divided by the length of that time interval to obtain the zero-crossing rate. For example, in an audio signal with a sampling rate of 44100Hz, the zero-crossing rate of a 1-second audio segment is calculated. First, the number of zero-crossing points within that 1-second audio segment is counted, assuming it is 1000, then the zero-crossing rate is 1000 times / second.

[0053] For example, taking a popular song as an example, the zero-crossing rate reflects the frequency at which the audio signal crosses zero, which is closely related to the rhythm of the music. By calculating the zero-crossing rate, rhythmic information such as drum beats and tempo can be identified in the song, the beats per minute (BPM) can be determined, and the positions of strong and weak beats can be identified, such as whether the song is in 4 / 4 or 3 / 4 time, and the specific time point of each beat can be located.

[0054] Pitch detection algorithms extract pitch sequences from audio data, reflecting the changes in pitch over time. For example, the YIN algorithm detects periodicity by calculating the autocorrelation function of the audio signal, thus determining the pitch. The specific steps are as follows: First, the audio signal is preprocessed, including removing DC components and normalization. Next, the autocorrelation function is calculated to measure the similarity of the signal to itself at different time delays. Then, the difference function is calculated to measure the periodicity of the signal at different delays. Next, the cumulative difference function is calculated to accumulate the difference values ​​and improve the robustness of detection. Then, a threshold is set to find the first minimum delay that meets the conditions. Interpolation is used to refine the delay and improve accuracy. Finally, the pitch frequency is calculated based on the refined delay. For example, for a simple sine wave signal, the YIN algorithm can accurately extract its pitch sequence, reflecting the changes in pitch over time. The YIN algorithm can effectively process various audio signals, extracting accurate pitch information, and is suitable for various scenarios such as speech and music. For example, in a solo piece of music, the pitch and duration of each note sung by the singer can be determined based on the pitch sequence. Then, based on this pitch sequence, the melodic features of the audio data are determined, i.e., the main melody line of the music. This helps to preserve the core melody of the music in subsequent processing, avoiding melody loss or distortion. Chord recognition is performed based on the audio data to obtain the harmonic features of the audio data. In a preferred embodiment of the invention, the audio signal is first preprocessed, including removing background noise and normalizing the volume. Taking a live performance audio as an example, preprocessing can reduce interference such as audience noise, making the audio data more suitable for subsequent analysis. Next, the audio signal is converted from the time domain to the frequency domain using Short-Time Fourier Transform (STFT) to obtain the spectrum. For example, for an audio clip containing guitar playing, STFT can decompose the audio signal of the guitar playing into the relationship between different frequency components and the changes over time. This can clearly show the frequency components and amplitude changes of each note contained in each chord in the guitar playing. The peak positions of the spectrum are analyzed to identify the individual notes in the chord. If the spectrum shows that the audio has significant peaks at the frequencies of C (261.63Hz), E (329.63Hz), and G (392.00Hz) within a certain time period, these notes constitute a chord. Based on the identified notes, the chord type is determined using music theory. For example, if the C, E, and G notes satisfy the rules for constructing a C major triad (root C, major third E, fifth G), then it can be determined that a C major triad is being played. Finally, the chord types identified in each time period are integrated into a sequence as the harmonic characteristics of the audio. For example, if the chord sequence identified for a popular song is CG-Am-F, this reflects the song's harmonic progression and provides harmonic information for subsequent audio analysis, compression, and other processing.Through the steps described above, the chord recognition function in the feature extraction unit can accurately extract harmonic features from audio data. This process, combined with other functions of the unit, achieves comprehensive feature extraction from audio data. In a piece of music played by an orchestra, chord recognition can determine the chord types formed by the combination of various instruments at different times, such as C major chords, G minor chords, etc. Harmony is an important component of music, and these harmonic features are crucial for restoring the richness and three-dimensionality of music, providing support and background for the melody.

[0055] In this optional embodiment, the audio data processing system performs comprehensive feature extraction on the audio data through a feature extraction unit, including spectral features, rhythmic features, melodic features, and harmonic features, covering multiple important aspects of audio. This multimodal feature extraction method can more comprehensively and accurately characterize the essential features of audio data, avoiding the loss of musical details caused by traditional technologies relying solely on single spectral features. For example, when compressing classical music, traditional technologies may lose some subtle rhythmic variations and harmonic layers because they only focus on the spectrum. However, this system, by extracting multiple modal features, can completely preserve these details, enabling the maximum restoration of the original audio's sound quality and emotional expression during subsequent compression and restoration processes. This meets users' demands for high-quality audio and improves the performance and effectiveness of audio compression technology in high-fidelity music processing.

[0056] Optionally, the feature encoding unit is specifically used for: Each modal feature is standardized to obtain the standardized modal features; Principal component analysis is performed on the standardized modal features to obtain the covariance matrix of the modal features; The modal features are obtained by projecting them onto the covariance matrix of the modal features; Probability calculations are performed on the dimensionality-reduced modal features to obtain the probability of occurrence of each feature value in each modal feature; Based on the occurrence probability, the probability distribution of the modal feature is obtained; A Huffman tree is constructed based on the probability distribution, and the feature code stream corresponding to the modality feature is obtained by assigning a Huffman code to each feature value.

[0057] Specifically, the feature encoding unit is responsible for processing the various modal features extracted by the feature extraction unit to generate the corresponding feature code stream. Specifically, each modal feature is standardized to eliminate the influence of differences in dimensions and numerical ranges between different modal features, so that each modal feature is comparable and processable.

[0058] For example, the amplitude data in the spectral features may have a large numerical range, while the zero-crossing rate data in the rhythm features may have a relatively small numerical range. Standardization can transform these different modal features into the same numerical range, such as [0,1] or [-1,1]. For instance, if the amplitude of a frequency component in the original spectral features is 1000, it may become 0.8 after standardization; while if the zero-crossing rate in the original rhythm features is 0.5, it may become 0.4 after standardization.

[0059] Principal component analysis (PCA) is performed on the standardized modal features to calculate their covariance matrix. Taking melody features as an example, assuming a melody feature is a vector composed of multiple pitch values, calculating its covariance matrix after standardization reveals the correlation between these pitch values. A larger value in the covariance matrix indicates a strong correlation between the corresponding two pitch values. For instance, in a melody primarily in the key of C, the C note and the G note a fifth above it exhibit a high correlation, which will be reflected in the covariance matrix.

[0060] Dimensionality reduction is achieved by projecting modal features onto the covariance matrix. Taking melody features as an example, a projection matrix is ​​constructed by selecting the main eigenvectors (those with larger eigenvalues) from the covariance matrix, projecting the original melody feature vectors into a lower-dimensional space. Assuming the original melody feature vectors are 10-dimensional, they may become 3-dimensional after projection reduction. 3-dimensional features can retain the main trends and key information in the original melody features, such as the overall pitch trend and main pitch components.

[0061] The probabilities of each feature value within the dimensionality-reduced modal features are calculated. Taking harmonic features as an example, assuming the dimensionality-reduced harmonic features contain several major chord types (such as C major, G major, Am minor, etc.), the frequency of each chord type in the entire audio is counted, and then divided by the total number of chords to obtain the probability of each chord type. For example, the probability of a C major chord is 0.4, a G major chord is 0.3, and an Am minor chord is 0.2, etc. Based on these probabilities, the probability distribution of the modal feature is constructed, which provides a basis for subsequent encoding.

[0062] A Huffman tree is constructed based on the probability distribution, and a Huffman code is assigned to each feature value to obtain the feature bitstream corresponding to the modal feature. Taking rhythm features as an example, assuming that after dimensionality reduction and probability calculation, several main rhythm patterns are obtained, each rhythm pattern corresponds to a feature value and its probability of occurrence. A Huffman tree is constructed according to the order of occurrence probability from high to low. Feature values ​​with high occurrence probability (such as the common quarter note rhythm pattern) are assigned shorter codes, such as "0"; feature values ​​with low occurrence probability (such as the complex triplet rhythm pattern) are assigned longer codes, such as "110". In this way, the feature value of each modal feature is converted into the corresponding Huffman code, forming a feature bitstream, which facilitates subsequent compression, storage, and transmission.

[0063] In this optional embodiment, the feature encoding unit effectively encodes and reduces the dimensionality of the modal features extracted by the feature extraction unit, ultimately generating a feature bitstream, providing strong support for efficient audio data compression. First, standardization ensures fairness and processability among different modal features, eliminating the adverse effects of differences in units and numerical ranges, making subsequent analysis and processing more accurate. Principal component analysis and projection dimensionality reduction extract key information from the modal features, removing redundant components, preserving the main features of the audio data while reducing the data volume and improving the efficiency of subsequent processing. The probability calculation and distribution construction, as well as the construction and encoding of the Huffman tree, fully consider the frequency characteristics of feature values, assigning shorter codes to feature values ​​with high occurrence probabilities, thereby achieving efficient encoding and compression of modal features while ensuring information integrity. This feature probability distribution-based encoding method in this embodiment can further reduce the data volume of the feature bitstream and improve the efficiency and quality of audio data compression. Overall, the feature coding unit's processing fully explores and utilizes the inherent statistical characteristics of audio data, achieving effective encoding and compression of multimodal features, thus laying a solid foundation for the efficient operation of the entire audio data processing system and high-quality audio compression.

[0064] Optionally, the deep learning model includes a long short-term memory layer and a fully connected layer, and the compression strategy unit is specifically used for: The feature stream of each modality feature is divided into multiple input sequences according to the window size of the deep learning model; The long short-term memory layer captures long-term dependencies for each input sequence to obtain the hidden state sequence of the input sequence, and uses the hidden state sequence as the output sequence of the long short-term memory layer. The complexity value of the feature code stream is obtained by performing feature integration and nonlinear transformation on all the output sequences through the fully connected layer.

[0065] Specifically, the feature stream of each modality is divided into multiple input sequences according to the window size set by the deep learning model. For example, assuming the feature stream is a sequence of length 1000 and the window size is set to 10, it will be divided into 100 input sequences, each containing 10 feature stream elements. LSTM is used to capture long-term dependencies in each input sequence. LSTM can effectively learn temporal dependencies in sequence data, and is particularly good at capturing long-term contextual information. Taking the spectral feature stream of audio data as an example, audio signals have continuity and correlation in the time domain. LSTM can analyze the relationship between the spectral features at the current moment and the spectral features at previous moments. For example, in the climax of a piece of music, the spectral features may show specific trends, and LSTM can capture these trends and reflect them in the output hidden state sequence. The hidden state sequence contains key feature information of the input sequence at different moments, reflecting the inherent regularity and pattern of the sequence, and is passed to the next layer as the output sequence of LSTM. The fully connected layer receives all LSTM output sequences and performs feature integration and nonlinear transformation on them. Feature integration involves combining the hidden state information from different positions to mine higher-level feature representations. Nonlinear transformations introduce nonlinear factors through activation functions (such as ReLU and sigmoid), enabling the model to learn more complex feature mapping relationships. Taking rhythmic feature streams as an example, fully connected layers can comprehensively analyze the hidden states of various input sequences, extracting features that reflect the overall rhythmic regularity and complexity. Ultimately, the fully connected layer outputs a complexity value for the feature stream, which quantifies the information complexity within the feature stream, providing a basis for subsequently determining the risk of information loss and dynamically adjusting the compression ratio.

[0066] In a preferred embodiment of the present invention, the deep learning model employs a Long Short-Term Memory-Fully Connected (LSTM-FC) model. It includes a LSTM layer and a fully connected layer to address the technical problem of improving audio data compression. The specific implementation is as follows: The LSTM layer contains multiple LSTM units, each consisting of an input gate, a forget gate, and an output gate. When processing audio data feature streams, the LSTM layer, through its internal gating mechanism, can effectively capture long-term dependencies in the sequence. For example, when processing music audio data, the LSTM layer can capture long-term trends and patterns in the feature streams of melody, rhythm, etc. When processing a melody feature stream containing multiple notes, the LSTM unit can selectively receive the current note feature stream information through the input gate, discard information that is no longer useful in the long-term state through the forget gate, and pass the updated long-term state information to subsequent units through the output gate. The fully connected layer (FC layer) receives the hidden state sequence of the feature stream sequence output by the LSTM layer and performs feature integration and nonlinear transformation on it. The fully connected layer consists of multiple neurons, each neuron connected to all neurons in the previous layer. When processing audio data feature streams, fully connected layers integrate hidden states at different locations to extract information reflecting the overall features of the entire sequence. Simultaneously, nonlinear transformations are applied through activation functions, introducing nonlinear factors into the model and enabling it to learn more complex feature mapping relationships.

[0067] The LSTM-FC model achieves accurate assessment of the complexity of audio data feature streams by capturing long-term dependencies in LSTM layers and integrating features and performing nonlinear transformations in fully connected layers. This effectively solves the problems of detail loss and sound quality distortion caused by the inability to accurately assess feature stream complexity in traditional audio data compression techniques. It can more comprehensively and accurately reflect the information content and complexity of the feature stream, thus making the information loss risk value and dynamic compression ratio determined based on the complexity value more reasonable, improving the scientific rigor and effectiveness of audio data compression.

[0068] In this optional embodiment, the compression strategy unit employs a deep learning model containing a long short-term memory layer and a fully connected layer. This model can accurately evaluate the complexity of the feature bitstream, thus providing a scientific basis for formulating audio data compression strategies. First, by dividing the feature bitstream into multiple input sequences, the model can process the sequence data segment by segment, ensuring both high processing efficiency and full consideration of the sequence's local features. The introduction of the long short-term memory layer effectively solves the problem of traditional models struggling to capture long-term dependencies when processing sequence data. It can deeply explore the intrinsic connections and patterns in the feature bitstream that change over time, which is particularly important for signal processing of audio data, which has significant temporal continuity. For example, in analyzing the melody direction, rhythmic changes, and harmonic development in music, LSTM can extract key time-dependent features. The fully connected layer further integrates and nonlinearly transforms the features output by the LSTM, fusing the sequence's local features and time-dependent features into a numerical value that can characterize the overall complexity of the feature bitstream. Compared to traditional simple statistical methods, this deep learning-based complexity assessment method can more comprehensively and accurately reflect the information content and complexity of the feature bitstream, thereby making the information loss risk value and dynamic compression rate determined based on the complexity value more reasonable, and improving the scientificity and effectiveness of audio data compression.

[0069] Optionally, the compression strategy unit is further configured to: The initial information loss risk value of the modal feature is obtained by mapping the complexity value of the feature code stream. Based on the compression requirements, determine the calibration coefficient for the initial information loss risk value; The initial information loss risk value is updated based on the calibration coefficient to obtain the information loss risk value of the modal feature.

[0070] Specifically, the compression strategy unit maps the complexity value of the feature bitstream using a pre-designed mapping function or a lookup table to obtain the initial information loss risk value of the modal feature. For example, assuming the complexity value ranges from [0, 1], the mapping function can be designed as: initial information loss risk value = 1 - complexity value. If the complexity value of a certain modal feature is 0.8, then its initial information loss risk value is 0.2. This means that the higher the complexity value, the lower the initial information loss risk value, because a feature bitstream with higher complexity contains more important information and is less likely to be lost during compression.

[0071] Based on compression requirements, a calibration coefficient is determined for the initial information loss risk value. Compression requirements include factors such as the target compression ratio and limitations on audio quality loss. For example, if the target compression ratio is high (e.g., requiring audio file size to be compressed to 10% of its original size), the calibration coefficient might be greater than 1, such as 1.5; if the audio quality requirements are extremely high and the target compression ratio is low (e.g., only compressing to 90% of the original size), the calibration coefficient might be less than 1, such as 0.8. A correspondence between compression requirements and calibration coefficients can be pre-established. When processing audio data, the corresponding calibration coefficient is looked up in this correspondence based on the current compression requirements. Because a higher target compression ratio means a stricter control over information loss risk, the initial information loss risk value needs to be amplified, allowing for more cautious handling of information loss during subsequent dynamic adjustments to the compression ratio. The initial information loss risk value is updated based on the calibration coefficient to obtain the information loss risk value for the modal features. Taking the above example, if the initial information loss risk value is 0.2, when the target compression ratio is high and the calibration coefficient is 1.5, the updated information loss risk value is 0.2 × 1.5 = 0.3; if the target compression ratio is low and the calibration coefficient is 0.8, the updated information loss risk value is 0.2 × 0.8 = 0.16. In this way, the information loss risk value can be flexibly adjusted according to different compression requirements, allowing the compression strategy to better meet actual needs.

[0072] In this optional embodiment, the compression strategy unit obtains an initial information loss risk value by mapping the complexity value of the feature bitstream, and updates the information loss risk value by determining a calibration coefficient based on compression requirements, thus achieving precise control over the information loss risk. First, the process of mapping the complexity value to the initial information loss risk value provides a basic benchmark for subsequent dynamic compression ratio adjustments, reflecting the information importance of the modal features themselves. Then, the calibration coefficient is determined according to the compression requirements, enabling the system to flexibly adjust the assessment result of the information loss risk based on different application scenarios. For example, in scenarios requiring a high compression ratio, the initial information loss risk value is amplified by the calibration coefficient, prompting the system to pay more attention to protecting key information during compression and avoiding distortion caused by over-compression; while in scenarios with extremely high sound quality requirements, the initial information loss risk value is reduced by the calibration coefficient, allowing the compression process to appropriately increase the compression degree of secondary information to meet sound quality requirements. This flexible calibration mechanism allows the compression strategy unit to comprehensively consider the characteristics of the audio data and the actual needs of the user to formulate the optimal compression scheme, effectively improving the quality and effect of audio data compression.

[0073] Optionally, the compression strategy unit is further configured to: Based on the compression requirements, determine the general compression ratio of the audio data; Based on the information loss risk value of each modal feature, determine the importance weight of the modal feature in the audio data; The compression ratio of each modal feature is obtained by combining the importance weight of each modal feature with the general compression ratio.

[0074] Specifically, the compression strategy unit determines the general compression ratio of the audio data based on compression requirements. Compression requirements are typically set by the user or system according to actual needs, and different general compression ratios can be pre-set for different requirements. For example, with limited storage space, it might be necessary to compress the audio file size to 20% of its original size; in this case, the general compression ratio would be set to 0.2. Based on the information loss risk value of each modal feature, its importance weight in the audio data is determined. For example, suppose there are three modal features A, B, and C, with information loss risk values ​​of 0.3, 0.5, and 0.2, respectively. Modal feature B has the highest information loss risk value, meaning it contains more important information and should be preserved as much as possible during compression; therefore, its importance weight can be determined as 0.5 / (0.3+0.5+0.2)=0.5. Similarly, the importance weight of A is 0.3 / 1=0.3, and the importance weight of C is 0.2 / 1=0.2. Based on the importance weight of each modal feature and the general compression ratio, the compression ratio of each modal feature is obtained.

[0075] For example, given the general compression ratio of 0.2 and the importance weights of each modality feature, the compression ratio of modality feature A = 1 - general compression ratio × (1 - importance weight) = 1 - 0.2 × (1 - 0.3) = 0.86; the compression ratio of modality feature B = 1 - 0.2 × (1 - 0.5) = 0.9; and the compression ratio of modality feature C = 1 - 0.2 × (1 - 0.2) = 0.84. This indicates that modality feature B achieves a higher compression ratio due to its higher importance weight, meaning that its information is preserved to a greater extent during the compression process.

[0076] In this optional embodiment, the compression strategy unit achieves personalized compression of different modal features in the audio data by determining a general compression ratio, the importance weight of modal features, and calculating the compression ratio of modal features, effectively improving the quality and effect of audio data compression. First, determining the general compression ratio provides an overall target and benchmark for the compression of the entire audio data, ensuring that the size of the compressed audio file meets actual needs. Then, the importance weight of each modal feature is determined based on its information loss risk value. This process fully considers the relative importance of different modal features in the audio data, enabling the subsequent compression process to target different features with varying degrees of compression. For example, modal features containing key melodic information receive a higher importance weight due to their higher information loss risk value, thus preserving them to a greater extent during compression and avoiding the loss of musical details and distortion of sound quality. Finally, by combining the general compression ratio and the importance weight, the compression ratio of each modal feature is calculated, achieving fine-grained compression control of each modal feature while meeting the overall compression target. This compression method not only improves compression efficiency but also preserves the key information and sound quality characteristics of the audio data to the greatest extent.

[0077] Optionally, the compression strategy unit is further configured to: Based on the compression ratio of all the modal features, a compression sub-strategy corresponding to each modal feature is obtained; The compression sub-strategies of all the modal features are integrated to obtain the compression strategy of the audio data.

[0078] Specifically, the compression strategy unit formulates a corresponding compression sub-strategy for each modal feature based on the compression ratio of all modal features. For example, for modal feature A (spectral feature), with a compression ratio of 0.8, the corresponding compression sub-strategy might be to use medium-intensity quantization compression to reduce the bit depth of the spectral data while retaining most frequency components. For modal feature B (rhythmic feature), with a compression ratio of 0.9, the corresponding compression sub-strategy might be to retain only the key beat positions and rhythmic patterns, removing some subtle rhythmic variations. The compression sub-strategies for all modal features are integrated to form a complete compression strategy for the audio data. The integration process considers the interrelationships between various modal features and their impact on the overall audio quality. For example, during integration, it is necessary to ensure that the compression of spectral features does not conflict with the compression of rhythmic features, resulting in a mismatch between the melody and rhythm of the music. The integrated compression strategy is a comprehensive scheme that guides how to collaboratively compress the various modal features of the audio data to achieve the desired compression effect and sound quality requirements. For example, the integrated compression strategy might stipulate: first, perform moderate-intensity quantization compression on the spectral features; then, retain the key beats on the rhythmic features; next, simplify the melodic features while preserving the main melody; and finally, appropriately reduce the complexity of the chords on the harmonic features. This integration method ensures that the various modal features of the compressed audio data are consistent, resulting in high-quality audio.

[0079] In this optional embodiment, the compression strategy unit achieves refined compression management of audio data by formulating and integrating compression sub-strategies for each modal feature. First, a compression sub-strategy is determined based on the compression ratio of each modal feature, fully considering the role and importance of different modal features in the audio data, and tailoring a suitable compression method for each feature. For example, a more conservative compression sub-strategy is used for key melodic features to preserve the core expressiveness of the music; while a slightly more aggressive compression method can be used for relatively minor harmonic features to reduce data volume while minimizing the impact on sound quality. Then, all sub-strategies are integrated into a complete compression strategy, ensuring the synergy and consistency of each modal feature during the compression process, avoiding audio data incoherence or distortion caused by compressing individual features separately. This integrated compression strategy can optimize the overall compression effect of audio data, satisfying users' requirements for audio file size while preserving audio quality and detail to the greatest extent.

[0080] Optionally, the data processing unit is specifically used for: The feature bitstream of each modality feature is compressed according to the compression strategy through a distributed network to obtain a compressed file of the audio data. The distributed network includes multiple nodes. Each node is assigned a feature code stream with a different modal feature. By using each node, compression is performed according to the compression sub-strategy of the modal feature corresponding to the node, resulting in a compressed file corresponding to the compressed modal feature; The compressed files corresponding to each modal feature are integrated to obtain the compressed file of the audio data.

[0081] Specifically, the data processing unit utilizes a distributed network for compression operations, which comprises multiple nodes. In a preferred embodiment of the invention, a distributed network consisting of five nodes is constructed. These nodes can be different servers or computing units, distributed across different geographical locations or different racks within the same data center. Different modal feature bitstreams are assigned to each node. For example, in a five-node distributed network, the spectral feature bitstream is assigned to node 1, the rhythm feature bitstream to node 2, the melody feature bitstream to node 3, the harmony feature bitstream to node 4, and other possible modal features (such as timbre features) to node 5. Each node compresses the data according to its corresponding modal feature compression sub-strategy. For example, node 1 receives the spectral feature bitstream and its corresponding compression sub-strategy (e.g., using medium-intensity quantization compression to reduce the bit depth of the spectral data), and compresses the spectral feature bitstream according to the sub-strategy to obtain a compressed spectral feature file; node 2 compresses the rhythm feature bitstream according to the rhythm feature compression sub-strategy (e.g., preserving key beat positions), obtaining a compressed rhythm feature file, and so on for other nodes.

[0082] The compressed modal feature files generated by each node are integrated to obtain the final compressed audio data file. The integration process needs to ensure the consistency of the format and the synchronization of the time sequence of each modal feature file. For example, the compressed spectral feature files, rhythm feature files, melody feature files, harmony feature files, etc., can be combined sequentially according to the preset file format specifications, and corresponding identification and index information can be added to generate a complete audio compressed file that can be decompressed and played.

[0083] In this optional embodiment, the data processing unit employs a distributed network to compress audio data, achieving efficient, flexible, and scalable audio data compression processing. By allocating feature streams of different modal characteristics to different nodes in the distributed network for parallel compression, compression efficiency is significantly improved, and compression processing time is shortened. For example, when processing large audio datasets or high-resolution audio files, traditional single-machine compression methods may take a long time, while the distributed network architecture can fully utilize the computing resources of multiple nodes to achieve parallel processing, greatly accelerating the compression speed. Simultaneously, this approach also possesses good scalability, allowing the number of nodes to be increased or decreased according to actual needs to adapt to audio data compression tasks of different scales. Furthermore, each node independently compresses according to its corresponding compression sub-strategy, ensuring quality control of each modal feature during the compression process. Finally, by integrating the compressed modal feature files, the integrity and consistency of the audio data are guaranteed. This distributed compression method not only improves compression efficiency but also ensures the quality of the compressed audio, meeting the demands of digital music storage and transmission for efficient and high-quality audio compression.

[0084] Optionally, combined Figure 2 As shown, the system further includes a recovery unit, which is used for: Based on the compression ratio of each modal feature in the audio data, the compressed file of each modal feature is decompressed and decoded to obtain the decompressed and decoded modal feature; The audio data is recovered by reconstructing all the decompressed and decoded modal features using an audio synthesis model.

[0085] Specifically, the recovery unit decompresses and decodes the compressed file of each modal feature based on its compression ratio in the audio data, obtaining the decompressed and decoded modal features. For example, for a spectral feature file with a compression ratio of 0.8, the decompression process reverses the operation based on the compression algorithm (such as quantization compression algorithm) to recover the original spectral feature data; for feature streams using Huffman coding, the original modal features are obtained through Huffman decoding. All modal features are then reconstructed using an audio synthesis model to obtain the recovered audio data. This audio synthesis model can be a deep learning model, such as a Generative Adversarial Network (GAN). For example, the recovered spectral features, rhythm features, melody features, and harmony features are input into a pre-trained GAN model, and the GAN's generator generates the recovered audio waveform based on these features. During the generation process, the model comprehensively considers the relationships between various modal features, such as ensuring harmonious matching between melody features and harmony features, and synchronization between rhythm features and spectral features, ultimately outputting recovered audio data that is highly similar to the original audio.

[0086] In a preferred embodiment of the invention, the audio synthesis model employs a Generative Adversarial Network (GAN), consisting of a generator and a discriminator. This GAN reconstructs all modal features to obtain the restored audio data. The generator receives the restored spectral, rhythmic, melodic, and harmonic features as input. Its structure typically includes multiple fully connected layers, convolutional layers, and deconvolutional layers. For example, the input spectral features are first processed through a fully connected layer for feature integration, mapping features of different dimensions to a suitable feature space. Then, deconvolutional layers progressively upsample the data to generate a higher-resolution audio waveform feature map. During this process, the generator comprehensively considers the relationships between various modal features, such as ensuring harmonious matching between melodic and harmonic features, and synchronization between rhythmic and spectral features. For instance, when generating an audio waveform for a piece of music, the generator determines the pitch and duration of the main note based on the melody features, adds corresponding accompaniment chords based on the harmonic features, and adjusts the beat position of the notes based on the rhythmic features to ensure that the generated audio waveform is consistent in melody, harmony, and rhythm.

[0087] The discriminator distinguishes between the audio waveform generated by the generator and the real original audio waveform. It consists of multiple convolutional and fully connected layers. For example, the discriminator receives both the generated and real audio waveforms as input. The convolutional layers extract local features from the audio waveform, such as frequency components and waveform morphology. Then, the fully connected layers integrate and analyze these local features, ultimately outputting a probability value representing the probability that the input waveform is real audio. If the generated audio waveform is highly similar to the real audio in terms of melody, harmony, rhythm, and spectrum, the discriminator will output a higher probability value; conversely, if there are significant differences, the probability value will be lower.

[0088] During training, the generator and discriminator engage in adversarial training. The generator aims to produce audio waveforms that are as realistic as possible to deceive the discriminator; the discriminator aims to accurately distinguish between the generated waveforms and the real waveforms. Through continuous iterative training, the generator gradually learns how to generate high-quality audio waveforms based on the input modal features. For example, in the early stages of training, the audio waveforms generated by the generator may not be smooth enough in terms of melodic lines, harmony, or rhythm. The discriminator can easily identify these differences between the generated waveforms and the real waveforms. As training progresses, the generator continuously adjusts its parameters, optimizing the generated waveforms to make the melody smoother and more natural, the harmony richer and more harmonious, and the rhythm more precise and stable. At the same time, the discriminator also continuously improves its discrimination ability, prompting the generator to produce higher-quality audio waveforms. Ultimately, a fully trained generator can generate reconstructed audio data that is highly similar to the original audio based on the input modal features.

[0089] In a preferred embodiment of this invention, a Generative Adversarial Network (GAN) is employed as the audio synthesis model to achieve high-quality restoration of compressed audio data. The GAN generator comprehensively considers the relationships between various modal features, such as the harmonious matching of melody and harmony, and the synchronization of rhythm and spectrum, thereby generating natural, coherent audio waveforms that are highly similar to the original audio. The discriminator, through adversarial training, continuously prompts the generator to improve the quality of the generated waveform, ensuring that the restored audio data is as close as possible to the original audio in terms of sound quality and detail. This deep learning-based audio synthesis method overcomes the limitations of traditional audio restoration techniques, effectively compensating for information that may be lost during compression and improving the fidelity and quality of audio restoration. For example, when restoring music audio processed with a high compression rate, the GAN model can reasonably restore the harmonic background and timbre details of the music based on the retained main melody features and key rhythmic features, making the restored music sound richer and more complete.

[0090] In this optional embodiment, the recovery unit achieves efficient and high-quality recovery of compressed audio data through decompression, decoding, and reconstruction of the audio synthesis model. First, decompression and decoding are performed according to the compression ratio of each modal feature, accurately recovering the original data of each modal feature and providing an accurate foundation for subsequent audio synthesis. Then, the audio synthesis model is used to comprehensively process and reconstruct all modal features, generating natural and coherent audio data. This model-based synthesis method fully explores and utilizes the inherent relationships between various modal features, compensating for details that may be lost during compression, thereby improving the quality and fidelity of the recovered audio. For example, when recovering music audio processed with a high compression ratio, the audio synthesis model can reasonably restore the harmonic background and timbre details of the music based on the retained main melody features and key rhythmic features, making the recovered music sound richer and more complete. Overall, the design of the recovery unit ensures the recoverability of audio data after compression, storage, or transmission, guaranteeing the continuity and high quality of the user experience.

[0091] Combination Figure 3 As shown, the present invention provides an audio data processing method, comprising: Acquire audio data and perform multimodal feature extraction on the audio data to obtain multiple modal features of the audio data; The modal features of each modality are reduced using a preset dimensionality reduction algorithm, and the reduced modal features are encoded to obtain the feature code stream corresponding to each modality. The complexity value of the feature code stream for each modality feature is obtained by predicting the feature code stream based on the feature code stream of each modality feature using a deep learning model. Based on the complexity value of the feature bitstream and the compression requirements of the audio data, the information loss risk value of the modal feature is determined; Based on the information loss risk value, the compression rate of each modal feature in the audio data is dynamically adjusted; Based on the compression ratio of all the modal features, the compression strategy of the audio data is obtained; The feature bitstream of each modality feature is compressed according to the compression strategy to obtain the compressed file of the audio data.

[0092] The audio data processing method of the present invention has the same advantages over the prior art as the audio data processing system described above, and will not be repeated here.

[0093] While the present invention has been disclosed above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, and all such changes and modifications will fall within the scope of protection of the present invention.

Claims

1. An audio data processing system, characterized in that, include: The feature extraction unit is used to acquire audio data and perform multimodal feature extraction on the audio data to obtain multiple modal features of the audio data; The feature encoding unit is used to perform dimensionality reduction processing on the modal features of each modality using a preset dimensionality reduction algorithm, and to encode the dimensionality-reduced modal features to obtain the feature code stream corresponding to each modality; A compression strategy unit is used to predict the complexity value of the feature bitstream of each modal feature using a deep learning model; determine the information loss risk value of the modal feature based on the complexity value of the feature bitstream and the compression requirements of the audio data; dynamically adjust the compression ratio of each modal feature in the audio data based on the information loss risk value; and obtain the compression strategy of the audio data based on the compression ratio of all modal features. The data processing unit is used to compress the feature bitstream of each modality feature according to the compression strategy to obtain a compressed file of the audio data.

2. The audio data processing system according to claim 1, characterized in that, The feature extraction unit is specifically used for: Obtain the raw audio data, perform preprocessing on the raw audio data to obtain the preprocessed raw audio data, and use the preprocessed raw audio data as the audio data; The audio data is converted from the time domain to the frequency domain by using a short-time Fourier transform to obtain the spectral characteristics of the audio data. The rhythmic characteristics of the audio data are obtained by calculating the zero-crossing rate based on the audio data. The pitch sequence of the audio data is extracted using a pitch detection algorithm, and the melody features of the audio data are determined based on the pitch sequence. Chord recognition is performed on the audio data to obtain the harmonic features of the audio data; The spectral features, rhythmic features, melodic features, and harmonic features are used as the modal features.

3. The audio data processing system according to claim 2, characterized in that, The feature coding unit is specifically used for: Each modal feature is standardized to obtain the standardized modal features; Principal component analysis is performed on the standardized modal features to obtain the covariance matrix of the modal features; The modal features are obtained by projecting them onto the covariance matrix of the modal features; Probability calculations are performed on the dimensionality-reduced modal features to obtain the probability of occurrence of each feature value in each modal feature; Based on the occurrence probability, the probability distribution of the modal feature is obtained; A Huffman tree is constructed based on the probability distribution, and the feature code stream corresponding to the modality feature is obtained by assigning a Huffman code to each feature value.

4. The audio data processing system according to claim 1, characterized in that, The deep learning model includes a long short-term memory layer and a fully connected layer. The compression strategy unit is specifically used for: The feature stream of each modality feature is divided into multiple input sequences according to the window size of the deep learning model; The long short-term memory layer captures long-term dependencies for each input sequence to obtain the hidden state sequence of the input sequence, and uses the hidden state sequence as the output sequence of the long short-term memory layer. The complexity value of the feature code stream is obtained by performing feature integration and nonlinear transformation on all the output sequences through the fully connected layer.

5. The audio data processing system according to claim 1, characterized in that, The compression strategy unit is further configured to: The initial information loss risk value of the modal feature is obtained by mapping the complexity value of the feature code stream. Based on the compression requirements, determine the calibration coefficient for the initial information loss risk value; The initial information loss risk value is updated based on the calibration coefficient to obtain the information loss risk value of the modal feature.

6. The audio data processing system according to claim 5, characterized in that, The compression strategy unit is further configured to: Based on the compression requirements, determine the general compression ratio of the audio data; Based on the information loss risk value of each modal feature, determine the importance weight of the modal feature in the audio data; The compression ratio of each modal feature is obtained by combining the importance weight of each modal feature with the general compression ratio.

7. The audio data processing system according to claim 1, characterized in that, The compression strategy unit is further configured to: Based on the compression ratio of all the modal features, a compression sub-strategy corresponding to each modal feature is obtained; The compression sub-strategies of all the modal features are integrated to obtain the compression strategy of the audio data.

8. The audio data processing system according to claim 7, characterized in that, The data processing unit is specifically used for: The feature bitstream of each modality feature is compressed according to the compression strategy through a distributed network to obtain a compressed file of the audio data. The distributed network includes multiple nodes. Each node is assigned a feature code stream with a different modal feature. By using each node, compression is performed according to the compression sub-strategy of the modal feature corresponding to the node, resulting in a compressed file corresponding to the compressed modal feature; The compressed files corresponding to each modal feature are integrated to obtain the compressed file of the audio data.

9. The audio data processing system according to claim 1, characterized in that, The system further includes a recovery unit, the recovery unit being used for: Based on the compression ratio of each modal feature in the audio data, the compressed file of each modal feature is decompressed and decoded to obtain the decompressed and decoded modal feature; The audio data is recovered by reconstructing all the decompressed and decoded modal features using an audio synthesis model.

10. An audio data processing method, characterized in that, include: Acquire audio data and perform multimodal feature extraction on the audio data to obtain multiple modal features of the audio data; The modal features of each modality are reduced using a preset dimensionality reduction algorithm, and the reduced modal features are encoded to obtain the feature code stream corresponding to each modality. The complexity value of the feature code stream for each modality feature is obtained by predicting the feature code stream based on the feature code stream of each modality feature using a deep learning model. Based on the complexity value of the feature bitstream and the compression requirements of the audio data, the information loss risk value of the modal feature is determined; Based on the information loss risk value, the compression rate of each modal feature in the audio data is dynamically adjusted; Based on the compression ratio of all the modal features, the compression strategy of the audio data is obtained; The feature bitstream of each modality feature is compressed according to the compression strategy to obtain the compressed file of the audio data.

Images