Encoding / decoding model training methods, computer equipment, and computer storage media

Abstract

This application discloses a method for training an encoding / decoding model, a computer device, and a computer storage medium. By introducing a dual-path design that separates the main encoding structure and the detail encoding structure, differentiated encoding of the main information and detail information in the audio signal is achieved. The main encoding structure focuses on processing the raw spectral data that carries the basic outline and core content of the audio, ensuring accurate capture and efficient encoding of the main information; the detail encoding structure is specifically designed for fine processing of the detail spectral data extracted from the raw spectrum, focusing on preserving subtle features that are crucial to the auditory experience, such as high-frequency overtones, transient changes, and weak environmental noise. This clearly defined structural design avoids the problems of insufficient learning or overfitting that easily occur when a single encoding path processes a large amount of main and detail information simultaneously at high bit rates, thereby effectively reducing the "illusion" phenomenon caused by insufficient learning of detail information or scarcity of training data.

Description

Technical Field

[0001] This application relates to the field of audio processing, specifically to a method for training a codec model, a computer device, and a computer storage medium. Background Technology

[0002] Audio encoding and decoding has always been an important research branch in the audio field. Traditional audio encoding and decoding compresses audio signals based on signal processing and psychoacoustic models, using fixed rules to drive lossy encoding compression. This traditional approach can reduce the recording and transmission of redundant data to some extent, but it still compresses some useful data and retains some data that still has room for compression. Therefore, rule-based compression methods exhibit inconsistent results when applied to different types of audio compression.

[0003] AI codecs, being data-driven encoding and decoding models, hold greater potential for exploration in terms of both compression ratio and fidelity. AI codecs utilize artificial intelligence algorithms, particularly deep learning models, to analyze, learn, and model audio signals, thereby achieving audio encoding and decoding. Unlike traditional encoding and decoding methods that rely on manually designed fixed rules, AI codecs learn the inherent characteristics and statistical patterns of audio signals from large amounts of audio data, constructing encoding and decoding models that can adapt to different audio types.

[0004] However, AI codec models are prone to illusions. In high-bitrate scenarios, the model tends to generate more details to match the complexity of the input signal. However, due to the lack of sufficient high-fidelity, high-complexity audio samples in the training data, or the model's insufficient learning of complex audio features, the model may incorrectly generate false details or noise that are not present in the original audio during decoding and reconstruction. These "illusions" not only fail to improve audio quality but also interfere with the accurate restoration of the original signal, resulting in distortion, noise, or unnatural timbre in the listening experience, thereby reducing the overall effect of encoding and decoding. Summary of the Invention

[0005] This application provides a method for training a codec model, a computer device, and a computer storage medium to improve the audio reproduction quality of AI codec models in high bitrate scenarios and reduce the occurrence of "hallucination" phenomena.

[0006] The first aspect of this application provides a method for training an encoding / decoding model, the method comprising:

[0007] Acquire raw spectral data of the audio signal and detailed spectral data for characterizing the audio signal;

[0008] The original spectral data is encoded using the main coding structure of the initial coding-decoding model to obtain the main coding features, and the detailed spectral data is encoded using the detailed coding structure of the initial coding-decoding model to obtain the detailed coding features.

[0009] Based on the main coding features, a first decoding process is performed to obtain a first decoding result;

[0010] A second decoding process is performed based on the aforementioned detail coding features to obtain a second decoding result;

[0011] The initial codec model is optimized based on the first decoding result and the second decoding result until the convergence condition is met, thereby obtaining the target codec model.

[0012] A second aspect of this application provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method of the first aspect described above.

[0013] A third aspect of this application provides a computer storage medium storing instructions that, when executed on a computer, cause the computer to perform the method described in the first aspect.

[0014] A fourth aspect of this application provides a computer program product that, when run on a computer device, causes the computer device to perform the method described in the first aspect.

[0015] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0016] By introducing a dual-path design that separates the main coding structure and the detail coding structure, differentiated encoding of the main and detail information in the audio signal is achieved. The main coding structure focuses on processing the raw spectral data that carries the basic outline and core content of the audio, ensuring accurate capture and efficient encoding of the main information. The detail coding structure, on the other hand, is specifically designed for fine processing of the detail spectral data extracted from the raw spectrum, focusing on preserving subtle features crucial to the auditory experience, such as high-frequency overtones, transient changes, and weak environmental noise. This clearly defined structural design avoids the problems of insufficient learning or overfitting that can easily occur when a single coding path processes a large amount of main and detail information simultaneously at high bit rates, thus effectively reducing the "illusion" phenomenon caused by insufficient learning of detail information or scarcity of training data. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the network framework in an embodiment of this application;

[0018] Figure 2A schematic diagram of an exemplary model structure for the encoding / decoding model in the relevant scheme;

[0019] Figure 3 This is a flowchart illustrating the encoding / decoding model training method in the embodiments of this application;

[0020] Figure 4 This is a flowchart illustrating an exemplary application scenario of the encoding / decoding model training method in this application.

[0021] Figure 5 This is a schematic diagram of a computer device in an embodiment of this application. Detailed Implementation

[0022] This application provides a method for training a codec model, a computer device, and a computer storage medium to improve the audio reproduction quality of AI codec models in high bitrate scenarios and reduce the occurrence of "hallucination" phenomena.

[0023] Please see Figure 1 The network framework in this embodiment includes:

[0024] The business server 100 and the terminal cluster; the terminal cluster may include: terminal devices 200a, terminal devices 200b, terminal devices 200c, ..., terminal devices 200n and other terminal devices.

[0025] The aforementioned business server 100 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud databases, cloud services, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal devices (including terminal devices 200a, 200b, 200c, ..., 200n) can be smartphones, tablets, laptops, desktop computers, PDAs, mobile internet devices (MIDs), wearable devices (such as smartwatches, smart bracelets, etc.), smart computers, smart in-vehicle systems, and other intelligent terminals.

[0026] The service server 100 can establish communication connections with each terminal device in the terminal cluster, and the terminal devices in the terminal cluster can also establish communication connections with each other. In other words, the service server 100 can establish communication connections with each terminal device among terminal devices 200a, 200b, 200c, ..., 200n. For example, terminal device 200a can establish a communication connection with the service server 100. Terminal devices 200a and 200b can establish a communication connection, and terminal devices 200a and 200c can also establish a communication connection. The communication connection method is not limited; it can be established directly or indirectly through wired communication or wireless communication, etc., depending on the actual application scenario. This application does not impose any restrictions on this.

[0027] It should be understood that, such as Figure 1 Each terminal device in the terminal cluster shown can have an application client installed. When the application client runs on each terminal device, it can interact with the business server 100, allowing the business server 100 to receive business data from each terminal device (such as user identity data uploaded by users through the terminal devices). This application client can be a music player, karaoke app, browser, social networking app, instant messaging app, live streaming app, game app, short video app, video app, shopping app, novel app, payment app, or any other application client capable of displaying text, images, audio, and video data. The specific application client can be determined based on the actual application scenario requirements and is not limited here. This application client can be a standalone client or an embedded sub-client integrated into a client (such as a music app or karaoke app), depending on the actual application scenario and is not limited here.

[0028] The applications of audio encoding and decoding technologies are wide-ranging and diverse. In music streaming services, efficient compression of massive amounts of music files is required to reduce storage costs and transmission bandwidth while ensuring a high-quality listening experience for users. In online conferencing scenarios, real-time audio encoding and decoding ensures clear transmission of voice signals, reduces latency and stuttering, and improves communication efficiency. In the fields of virtual reality (VR) and augmented reality (AR), high-quality spatial audio encoding and decoding are crucial for creating immersive auditory environments, allowing users to accurately perceive the location and distance of sounds. For podcasts, audiobooks, and other content, encoding and decoding technologies can compress file sizes while ensuring the integrity of the audio content, facilitating downloads and offline listening. Furthermore, audio encoding and decoding technologies play an indispensable role in broadcasting, voice assistants, and intelligent in-vehicle systems, directly impacting the efficiency and quality of audio information transmission and the overall user experience.

[0029] Audio encoding and decoding technologies include AI codec technology, which refers to the technology of encoding and decoding audio using deep neural networks. Existing high-performance AI codec models come from Descriptive Audio Codecs (DACs), such as... Figure 2 This document illustrates an exemplary model structure, which utilizes four main modules—a convolutional encoder, a quantizer, a deconvolutional decoder, and a discriminator—to implement the compression, encoding, and decoding processes of audio signals. Specifically, the processing steps are as follows:

[0030] 1. First, the encoder part performs convolutional encoding on the input audio signal. By controlling the step size of the convolutional encoder, the purpose of downsampling compression is achieved.

[0031] 2. For the quantization part, RVQ (Residual Vector Quantization) is used to quantize the audio signal downsampled by the encoder into a smaller integer sequence. Specifically, the high-dimensional input feature vector is progressively quantized through multiple quantizer levels. Each quantizer level matches the input vector with codewords in the codebook, outputting a quantization index and a residual vector. The residual vector serves as the input to the next quantizer layer. After multiple quantization layers, the final integer sequence is obtained, which becomes the encoded bitstream, enabling compressed storage and transmission of the data.

[0032] 3. The quantized integer sequence is input into the decoder network, and the decoder gradually recovers the audio signal through deconvolution.

[0033] 4. Input the audio signal output from the decoder and the original audio signal into the discriminator. The discriminator's function is to make the output signal closer to the original real signal.

[0034] 5. Calculate network losses: Ls is the spectral loss, used to measure the spectral similarity of the audio signal before and after encoding; Lt is the temporal loss, used to measure the temporal similarity of the audio signal before and after encoding; Lw is the quantization loss of the quantizer, used to measure the similarity between the codebook and the signal before quantization; Ll is the entropy coding loss of the quantizer, used to measure the similarity of the encoded vector before and after secondary compression; Ld and Lg are the discriminator and generator losses, respectively, used to measure the similarity between the generated samples and the original samples.

[0035] While existing open-source DAC models can achieve decent compression results, this performance is limited to low-bitrate scenarios. In high-bitrate scenarios, their performance is often inferior to traditional codecs. This is because, at high bitrates, the model needs to retain more audio details to achieve higher fidelity. However, existing DAC models primarily rely on a single encoding path to process the entire audio spectrum, failing to differentiate the encoding of information of varying importance within the audio signal. As the bitrate increases, the model attempts to process both the main structural information and rich detail information simultaneously within a single encoding structure, which can easily lead to insufficient learning of detail information or overfitting to specific patterns in the training data.

[0036] Furthermore, since high-fidelity, high-complexity audio samples may be relatively scarce in the training data, when decoding and reconstructing high-bitrate audio, the model cannot accurately generate complex and subtle features in the original audio (such as instrument overtones, subtle emotional changes in human voices, and weak details in ambient sounds) from the statistical patterns learned during training. Instead, it may generate false details that do not match the original signal based on insufficient learning, i.e., producing an "illusion" phenomenon. These false details are superimposed on the decoded signal, causing the audio to sound like high-frequency noise, timbre distortion, phase misalignment, or unnatural audio segments, seriously affecting the audio restoration quality at high bitrates and resulting in performance inferior to traditional codecs that have been optimized for this scenario over a long period of time.

[0037] To address the aforementioned issues, this application proposes a codec model training method to improve the audio reproduction quality of AI codec models in high bitrate scenarios and reduce the occurrence of "hallucination" phenomena.

[0038] The following will be discussed in the preceding text. Figure 1 Based on the network framework shown, embodiments of this application are described in further detail. Please refer to [link / reference]. Figure 3 Another embodiment of the encoding / decoding model training method in this application includes:

[0039] 301. Obtain the raw spectral data of the audio signal and the detailed spectral data used to characterize the detailed information of the audio signal;

[0040] The method of this embodiment can be applied to a computer device, which may be... Figure 1The network framework shown includes a service server 100 or various terminal devices. In this step, the computer device can obtain an initial encoding / decoding model. This model differs from the traditional single-encoding path structure in its innovative inclusion of two independent encoding structures: a main encoding structure and a detail encoding structure. The main encoding structure is primarily responsible for encoding the spectral components of the audio signal that carry the main information and basic outline, while the detail encoding structure focuses on capturing and encoding the subtle, complex, but crucial details in the audio signal that are essential for the auditory experience. This dual-encoding structure design aims to achieve differentiated and more refined processing of information with different importance and characteristics in the audio signal, laying the foundation for improving audio reproduction quality in high-bitrate scenarios.

[0041] Audio signals used as training data for models can be converted into spectral data, for example, by using methods such as Short Time Fourier Transform (STFT) to convert time-domain audio signals into raw spectral data containing frequency, amplitude, and phase information. Furthermore, in order for the model to learn detailed information in the audio signal, detailed spectral data that can characterize the detailed features can be extracted from the raw spectral data.

[0042] The detailed information of an audio signal refers to the subtle features in the audio signal that have a significant impact on the richness, realism, and delicacy of the auditory experience. These features include, but are not limited to, the following categories:

[0043] First, there are the subtle changes in the structure and intensity of overtones produced during instrument playing. For example, the layering and decay characteristics of overtones when the violin plays in the high register, or the details of the burst and continuation of overtones when the piano strikes the string. Second, there are the subtle emotional characteristics in human vocal expression, such as the slight vibrato of the vocal cords when a speaker or singer produces sound, the changes in the strength of breath, the subtle friction of labiodental consonants, and the instantaneous fine-tuning of pitch and timbre caused by emotional fluctuations.

[0044] Third, subtle details and spatial information in ambient sound, such as faint wind sounds, distant footsteps, and slight echoes of objects colliding in the recording environment, as well as the spatial orientation and reverberation characteristics carried by these sounds; fourth, transient details in the time domain of audio signals, such as the initial transient impact of drumbeats, the instantaneous explosive force of plucking strings, and the fine dynamic range in the gradual process of sound from nothing to something or from something to nothing; fifth, subtle interactions and separation characteristics between different sound sources in complex audio scenarios, such as the unique timbre of each instrument and its proportion and subtle differences in spatial position in the overall mix when multiple instruments are played simultaneously in a symphony.

[0045] These details together constitute the unique texture and realism of the audio signal, which is the key to improving audio reproduction quality in high bitrate scenarios.

[0046] Specifically, the original spectral data can be decomposed into sub-spectrums of different frequency bands using multi-scale methods, such as wavelet transform or empirical mode decomposition. High-frequency bands or sub-spectrums with rapidly changing characteristics typically contain more detailed information, such as instrument overtones and transient changes in sound. Alternatively, the residual between the original spectral data and the smoothed spectral data after low-pass filtering can be calculated and used as detailed spectral data. This residual effectively reflects the subtle fluctuations and high-frequency components that were smoothed out in the original spectrum, thus accurately capturing the detailed features of the audio signal. Detailed spectral data obtained in this way can provide targeted learning targets for training detailed coding structures.

[0047] The operation of generating the raw spectrum data and detailed spectrum data of the audio signal can be performed by the computer device, or it can be pre-processed by other devices and provided to the computer device. The specific method can be determined according to the division of labor and resource configuration of data processing in the actual application scenario. This application does not impose strict restrictions on this.

[0048] 302. The original spectrum data is encoded using the main coding structure of the initial coding and decoding model to obtain the main coding features, and the detail spectrum data is encoded using the detail coding structure of the initial coding and decoding model to obtain the detail coding features;

[0049] In this step, the computer device can input the acquired raw spectral data and detailed spectral data into the main coding structure and detailed coding structure of the initial encoding / decoding model, respectively. After receiving the raw spectral data, the main coding structure performs feature extraction and dimensionality reduction coding on the raw spectrum through its internal network components such as convolutional layers and pooling layers. It focuses on preserving the main outline and key information of the audio, such as the fundamental frequency, main harmonic structure, and overall energy distribution. Finally, it outputs the coded features of the raw spectral data (i.e., the main coding features), which contain the core content of the audio signal.

[0050] Meanwhile, the detail coding structure receives detail spectral data. This structure can precisely capture information such as high-frequency overtones, transient impulses, weak environmental noise, and subtle changes in timbre contained in the detail spectral data, and encodes this detailed information to output the coded features of the detail spectral data (i.e., detail coding features). These two coded features will serve as inputs to the subsequent decoding process, providing a foundation for achieving high-quality audio reconstruction at high bit rates.

[0051] 303. Perform a first decoding process based on the main encoding features to obtain a first decoding result;

[0052] 304. Perform a second decoding process based on the aforementioned detail coding features to obtain a second decoding result;

[0053] The decoding process refers to progressively upsampling and restoring the encoded feature vectors, corresponding to the dimensionality reduction operation in the encoding stage. Through network operations such as deconvolution or transposed convolution, the compressed features are restored to a data form with the same dimension as the input spectrum, thereby reconstructing the corresponding spectral components. The first decoding result obtained by the first decoding process of the main encoded features corresponds to the reconstruction of the main spectral information of the audio signal, containing the core outline and basic content of the audio; the second decoding result obtained by the second decoding process of the detail encoded features corresponds to the reconstruction of the detail spectral components of the audio signal, carrying various subtle feature information of the audio.

[0054] In this implementation, the decoder performing the first decoding process based on the master encoding features and the decoder performing the second decoding process based on the detail encoding features can be the same decoder, or they can be two different decoders with the same structure but independently trained parameters. Both implementation methods can achieve the desired results. The implementation method using the same decoder with shared parameters can reduce the number of parameters in the overall model, reduce the computational overhead of model training and inference, and is more suitable for edge deployment scenarios. On the other hand, using two decoders with independent parameters allows the two modules to learn the decoding capabilities corresponding to different features, which is more adaptable to the distribution characteristics of different features and usually achieves better detail restoration results. The specific choice can be flexibly made according to the actual computing power limitations and effect requirements of the deployment.

[0055] 305. Optimize the initial codec model based on the first decoding result and the second decoding result until the convergence condition is met to obtain the target codec model;

[0056] In this step, a spectral loss can be constructed based on the coding features of the original spectral data and the coding features of the detailed spectral data. This loss is used to measure the spectral similarity of the audio signal before and after encoding. For example, it measures the difference between the spectrum reconstructed from the coding features of the original spectral data after decoding and the original spectral data, as well as the difference between the spectrum reconstructed from the coding features of the detailed spectral data after decoding and the detailed spectral data. Simultaneously, a temporal loss can be introduced. By converting the decoded spectral data back to a time-domain audio signal and comparing it with the original time-domain audio signal, the mean square error and other metrics in the time-domain waveform are calculated to ensure that the model can accurately reconstruct the audio signal not only at the spectral level but also at the time-domain level.

[0057] Furthermore, quantization loss and entropy coding loss can be set separately for the main coding structure and the detail coding structure to optimize the quantization accuracy and compression efficiency of their respective encoded features. For the main coding structure, the quantization loss can focus on ensuring accurate quantization of the main information and avoiding distortion of the main outline due to quantization errors; while the quantization loss for the detail coding structure can focus more on preserving detailed information, allowing as many subtle features as possible to be transmitted under a certain compression ratio. The entropy coding loss is used to ensure that the information loss of the encoded features is minimized during the secondary compression process, further improving the overall coding efficiency.

[0058] The quantization loss is determined as follows: for the main coding structure, its output coding features are matched with the corresponding codebook, and the distance between them (such as Euclidean distance, cosine distance, etc.) is calculated as the quantization loss. The smaller the loss value, the more accurate the quantization of the main coding structure is in terms of the main information and the lower the risk of distortion of the main outline. For the detail coding structure, its coding features are also matched with the corresponding codebook, and the distance is calculated as the quantization loss. However, the weight of high-frequency detail features can be appropriately increased during the calculation to prioritize the preservation of detail information in the quantization process.

[0059] The entropy coding loss is determined by first performing entropy coding (such as Huffman coding, arithmetic coding, etc.) on the encoded features output by the main coding structure and the detail coding structure respectively to obtain a second-compressed bitstream. Then, the bitstream is decoded back into a feature vector, and the similarity (such as cross-entropy, mean square error, etc.) between the decoded feature vector and the original encoded features is calculated. The lower the similarity, the greater the entropy coding loss. This is to encourage the model to minimize information loss when performing entropy coding and improve the overall coding compression efficiency.

[0060] By comprehensively considering the aforementioned spectral loss, temporal loss, quantization loss and entropy coding loss of the main coding structure, and quantization loss and entropy coding loss of the detail coding structure, a multi-objective total loss function is constructed. Then, using the backpropagation algorithm, the network parameters of the main coding structure and detail coding structure in the initial codec model are continuously adjusted, including convolutional kernel weights, bias terms, and quantizer codebooks, so that the value of the total loss function gradually decreases. When the value of the total loss function tends to stabilize and no longer decreases significantly, or reaches the preset upper limit of the number of iterations, the model is considered to have reached the convergence condition. The model obtained at this time is the target codec model that can effectively improve the audio restoration quality and reduce the "hallucination" phenomenon in high bitrate scenarios.

[0061] Therefore, in this embodiment, a dual-path design with separate main coding structure and detail coding structure is introduced to achieve differentiated encoding of the main information and detail information in the audio signal. The main coding structure focuses on processing the raw spectral data that carries the basic outline and core content of the audio, ensuring accurate capture and efficient encoding of the main information; the detail coding structure is specifically designed for fine processing of the detail spectral data extracted from the raw spectrum, focusing on preserving subtle features that are crucial to the auditory experience, such as high-frequency overtones, transient changes, and weak environmental noise. This clearly defined structural design avoids the problem of insufficient learning or overfitting that easily occurs when a single coding path processes a large amount of main and detail information at high bit rates, thereby effectively reducing the "illusion" phenomenon caused by insufficient learning of detail information or scarcity of training data.

[0062] During model optimization, quantization loss and entropy coding loss were applied to the main coding structure and detail coding structure, respectively. A multi-objective total loss function was constructed by combining spectral loss and temporal loss. The network parameters were adjusted using backpropagation. This allows the main coding structure to optimize compression efficiency while maintaining the quantization accuracy of the main information, while the detail coding structure prioritizes the preservation of detailed features during quantization. The resulting target codec model can more accurately reproduce the complex and subtle features of the original audio in high-bitrate scenarios, significantly improving audio reproduction quality and enabling it to compete with or even outperform traditional codecs in high-bitrate applications.

[0063] based on Figure 3 In one optional implementation of the embodiment shown, the initial codec model is optimized based on the first decoding result and the second decoding result. Specifically, the optimization method may be to construct the main spectrum loss based on the original spectrum data and its corresponding first decoding result, and to construct the detail spectrum loss based on the detail spectrum data and its corresponding second decoding result, and then optimize the initial codec model based on the main spectrum loss and the detail spectrum loss.

[0064] The primary spectral loss is used to measure the encoding and decoding reconstruction effect of the primary coding structure on the original spectral data. Specifically, it is determined by calculating the difference between the original spectral data and the reconstructed spectral data obtained after decoding the encoded features output by the primary coding structure. For example, metrics such as mean square error (MSE), spectral angle distance (SAD), or logarithmic spectral distortion (LSD) can be used to quantify the degree of deviation between the two in amplitude, phase, or overall spectral shape. The primary spectral loss is designed to ensure that the primary coding structure can accurately capture and reconstruct the main contours and core spectral information of the audio signal, providing a solid foundation for high-quality audio restoration.

[0065] The main spectral loss can be Mel loss. Mel loss refers to converting the original spectral data and the decoded and reconstructed spectral data into Mel spectra respectively, and then measuring the difference between the two Mel spectra to assess the encoding and reconstruction effect of the main coding structure on the main audio information. Specifically, firstly, Mel frequency cepstral analysis (MFCC) is performed on the original time-domain audio signal and the reconstructed time-domain audio signal, or Mel spectral features are directly extracted. The linear spectrum is mapped onto the Mel frequency axis, which conforms to the characteristics of human hearing, to obtain a spectral representation with human auditory perception. Next, methods such as mean square error (MSE), KL divergence, or cosine similarity are used to calculate the distance between the original Mel spectrum and the reconstructed Mel spectrum. This distance value is the main spectral loss.

[0066] Since the Mel spectrum is closer to the human ear's perception of sound frequencies, using Mel loss as the main spectral loss can more directly reflect the distortion of auditory subject information, thereby guiding the main coding structure to prioritize the optimization of the subject spectral features that the human ear is sensitive to, and improving the perceptual restoration quality of the audio subject contour at high bit rates.

[0067] The detail spectrum loss is used to evaluate the encoding and decoding reconstruction effect of the detail coding structure on the detail spectrum data. Its calculation method is similar to the main spectrum loss, comparing the difference between the detail spectrum data and the decoded and reconstructed detail spectrum data. However, when calculating the detail spectrum loss, higher weights can be assigned to high-frequency bands or specific sensitive frequency regions based on the characteristics of the detail information, highlighting the accuracy requirements for restoring these subtle features. For example, when calculating the mean square error, the error in the high-frequency part is multiplied by a coefficient greater than 1, causing the model to pay more attention to the accurate reconstruction of high-frequency details during optimization, thereby effectively preserving key details such as overtones and transients in the audio and reducing the occurrence of "illusion" phenomena.

[0068] Specifically, when optimizing the initial encoding / decoding model based on the main spectrum loss and detail spectrum loss, the main spectrum loss and detail spectrum loss can be weighted and summed according to a preset weight ratio to obtain the fusion loss. For example, based on the relative importance of main information and detail information in audio perception, the weight of the main spectrum loss can be set to 0.6, and the weight of the detail spectrum loss to 0.4. Of course, this weight ratio can be dynamically adjusted according to the actual application scenario and specific requirements for audio quality. Then, this fusion loss is used as one of the loss functions for model optimization, together with the aforementioned temporal loss, quantization loss, and entropy coding loss to form the total loss function, and the model parameters are adjusted through the backpropagation algorithm.

[0069] Furthermore, more complex loss fusion strategies can be employed. For example, a dynamic weighting mechanism can be introduced to automatically adjust the proportions of the main spectral loss and the detail spectral loss in the fusion loss based on the current training epoch or the model's learning level of subject and detail information. In the early stages of model training, learning subject information is more critical, so the main spectral loss can be given a higher weight to ensure that the model can accurately grasp the basic outline of the audio. As training progresses and the encoding and reconstruction of subject information stabilizes, the weight of the detail spectral loss can be gradually increased, guiding the model to devote more effort to the fine capture and reconstruction of detail information. This dynamically adjusted weighting mechanism allows the model to learn different levels of information with emphasis at different training stages, thereby further improving the overall performance of the model.

[0070] In this way, the model can simultaneously ensure the accurate transmission of main information and the fine restoration of detailed information during the optimization process, enabling the main coding structure and the detailed coding structure to work together to improve the overall audio restoration quality in high bitrate scenarios.

[0071] Of course, besides the aforementioned method of weighted summation of the principal spectrum loss and detail spectrum loss to construct the fusion loss, the principal spectrum loss and detail spectrum loss can also be treated as independent optimization objectives. Multi-objective optimization algorithms (such as Pareto optimization) can be used to find model parameters that perform well on both objectives. This method does not directly merge the two losses, but rather tracks the model's performance on the principal spectrum loss and detail spectrum loss during training, attempting to find a set of parameters that minimizes the principal spectrum loss while keeping the detail spectrum loss at an acceptable low level, thus achieving a balance between the reconstruction quality of principal and detail information.

[0072] For example, after each iteration, record the principal spectrum loss value and detail spectrum loss value corresponding to the current model parameters. If both loss values ​​corresponding to the new parameters are better than a certain parameter combination in history, or if one loss value is significantly reduced while the other loss value does not deteriorate significantly, then update the current optimal parameter set.

[0073] This multi-objective optimization strategy can provide more diverse model choices for different application scenarios. For example, some scenarios may focus more on the clarity of the main information, while other scenarios have higher requirements for the richness of details. Through the Pareto optimal solution set, appropriate model parameter configurations can be selected according to specific needs.

[0074] Optionally, when decoding the main coding feature to obtain the first decoding result, the coding features of the original spectral data (i.e., the main coding feature) and the coding features of the detail spectral data (i.e., the detail coding feature) can be concatenated to obtain a concatenated coding feature. This concatenated coding feature is then input into the initial encoding / decoding model. The quantizer structure in the initial encoding / decoding model quantizes the concatenated coding feature into an integer sequence, and the decoding structure in the initial encoding / decoding model decodes this integer sequence to obtain the decoding result corresponding to the concatenated coding feature. Furthermore, when constructing the main spectral loss, it can be constructed based on the original spectral data and the decoding result corresponding to the concatenated coding feature.

[0075] The concatenated features are input into a quantizer, which maps the high-dimensional continuous feature vector into a discrete integer sequence to facilitate subsequent entropy encoding and transmission. Specifically, the quantizer uses Residual Vector Quantization (RVQ), progressively approximating the concatenated feature vector through multiple quantization stages. Each stage compares the input vector with vectors in the codebook, selects the closest code vector as the quantization result for that stage, and calculates the residual, passing it to the next stage. The final result is an integer sequence composed of multiple codebook indices. This integer sequence constitutes the core of the encoded bitstream, significantly reducing the data volume and achieving efficient compression of the audio signal.

[0076] For example, the encoded features of the original spectral data and the encoded features of the detailed spectral data are concatenated along the feature dimension to form a concatenated encoded feature vector that integrates both main and detailed information. This concatenated encoded feature vector is then input into the quantizer structure of the initial codec model. The quantizer structure maps the continuous concatenated encoded feature vector into a discrete integer sequence according to a pre-defined codebook; this step is crucial for data compression and preparing for subsequent entropy coding. Next, the decoding structure in the initial codec model receives this integer sequence and reconstructs it into spectral data with the same dimension as the original spectral data through a series of inverse transformation operations, such as dequantization, deconvolution, and upsampling. This is the decoding result corresponding to the concatenated encoded features.

[0077] Then, the decoding result is compared with the original spectral data, and the main spectral loss is constructed by calculating the difference between the two (e.g., using mean squared error, spectral angular distance, etc.). This approach allows the calculation of the main spectral loss to comprehensively consider the overall performance of both main and detailed information in the decoding and reconstruction process. This ensures that when optimizing the model, it not only focuses on restoring the outline of the main audio subject but also takes into account the impact of detailed features on the overall spectral similarity, thereby more comprehensively evaluating the degree to which the encoding and decoding process preserves the core content of the original audio.

[0078] Of course, in other alternative methods, the coding features of the main coding structure and the detail coding structure can be quantized and decoded independently, and then the decoding results of the two can be fused and compared with the original spectral data to construct the main spectral loss. For example, the coding features output by the main coding structure are quantized and decoded to obtain the main reconstructed spectrum, and the coding features output by the detail coding structure are quantized and decoded to obtain the detail reconstructed spectrum. The main reconstructed spectrum and the detail reconstructed spectrum are combined according to a certain fusion strategy (such as weighted superposition based on feature importance) to generate the final fused reconstructed spectrum, and then the difference between the fused reconstructed spectrum and the original spectral data is calculated as the main spectral loss.

[0079] This independent processing and fusion approach can clearly separate the contributions of main and detail information in the reconstruction process, making it easier for the model to optimize the performance of the main coding structure and detail coding structure separately. At the same time, through the design of the fusion strategy, it ensures that the two can effectively complement each other in the final spectrum reconstruction, thereby improving the overall accuracy of main information restoration.

[0080] Alternatively, instead of concatenating the coding features of the original spectral data and the coding features of the detail spectral data, one can directly compare the coding features output by the main coding structure with the corresponding decoding results output by the decoding structure to construct the main spectral loss, and at the same time compare the coding features output by the detail coding structure with the corresponding decoding results output by the decoding structure to construct the detail spectral loss.

[0081] Specifically, the master coding structure encodes the original spectral data to obtain master coding features, which are then quantized and decoded to obtain the master decoded spectrum. Similarly, the detail coding structure encodes the detail spectral data to obtain detail coding features, which are then quantized and decoded to obtain the detail decoded spectrum. Then, the difference between the original spectral data and the master decoded spectrum is calculated as the master spectral loss, and the difference between the detail spectral data and the detail decoded spectrum is calculated as the detail spectral loss.

[0082] In this approach, the calculation of the main spectrum loss and the detail spectrum loss is more independent, which can directly reflect the encoding and reconstruction effect of their respective coding structures on the corresponding input data. This makes it easier for the model to accurately optimize for the characteristics of the main coding structure and the detail coding structure, ensuring the restoration quality of the main information and the detail information.

[0083] based on Figure 3 In another optional implementation of the embodiment shown, the detailed spectrum data can be obtained by acquiring the base spectrum data corresponding to the original spectrum data, which is used to characterize the main structural information of the audio signal, and subtracting the base spectrum data from the original spectrum data to obtain the detailed spectrum data.

[0084] The core structural information refers to the key elements in the audio signal that determine the overall contour and main auditory perception, such as the fundamental frequency, main harmonic components, overall energy distribution trend, and stable spectral structure over a long period. This information constitutes the basic framework of the audio and is the key basis for listeners to identify sound type (such as vocals, instrumental sounds), pitch, rhythm, and melody. The base spectrum data is the extraction and representation of the core structural information from the original spectrum data. It retains the spectral portion that reflects the main characteristics of the audio by filtering out high-frequency transients, weak overtones, random noise, and other detailed components.

[0085] For example, for a human voice audio clip, the fundamental spectrum data may mainly contain the fundamental frequency and its main harmonics generated by vocal cord vibration, while the detail spectrum data may include subtle articulation details such as breath sounds, sibilants, and labial sounds, as well as slight background noise. By subtracting the fundamental spectrum data from the original spectrum data, the detail spectrum data that enriches the audio's expressiveness and enhances the realism of the sound can be separated, providing a data foundation for the subsequent detail coding structure to accurately capture and encode subtle features.

[0086] One method for obtaining the base spectrum data is to sequentially slide a preset filter operator across the original spectrum data. For each sliding operation targeting a center spectrum point, the amplitude of that center spectrum point and its neighboring spectrum points can be weighted and summed based on the weights represented by the filter operator. The calculated result is then used as the new amplitude of that center spectrum point. Therefore, the base spectrum data can be obtained by acquiring the new amplitude of each spectrum point in the original spectrum data.

[0087] For example, the filtering operator can be a low-pass filter kernel with Gaussian distribution characteristics, and its weights exhibit a smooth transition characteristic of high at the center and low at the edges. In specific calculations, the center of the filtering operator is aligned with a certain spectral point in the original spectral data. Taking that point as the center, the amplitudes of spectral points within a certain range (such as a 3×3 or 5×5 neighborhood) are taken, multiplied by the weights of the corresponding positions of the filtering operator, and then summed. The result is the amplitude of that central spectral point in the base spectral data.

[0088] Specifically, the original spectral data is a two-dimensional matrix, where each element represents the audio signal energy or amplitude at a specific time point t and frequency point f. During Gaussian blurring, a Gaussian kernel is convolved with the neighborhood centered at point (t, f) in the original spectral data (i.e., a weighted average). The result replaces the amplitude value at that point (t, f) in the original spectral data, thus forming a new amplitude value at the corresponding position in the base spectral data. Compared to the original value, this new value, due to neighborhood smoothing, has its detailed high-frequency components weakened, reflecting more of the overall trend or "base" characteristics of that position and its surrounding area.

[0089] When using a Gaussian kernel to perform Gaussian blurring on the original spectral data, the size and standard deviation of the Gaussian kernel determine the degree of blurring. The corresponding formula for the two-dimensional Gaussian function can be expressed as follows:

[0090] ;

[0091] In this formula, (x, y) represents the coordinates of a certain position in the filter kernel relative to the center, and G(x, y) represents the weight value of the Gaussian kernel at coordinates (x, y). σ is the standard deviation of the Gaussian function, which determines the rate at which the weight decays with distance from the center. The larger the value of σ, the slower the weight decays, the smoother the filtered spectral data, and the more macroscopic the main structural information retained in the base spectrum data; the smaller the value of σ, the faster the weight decays, the weaker the filtering effect, and the more relatively detailed main features are contained in the base spectrum data.

[0092] By selecting appropriate Gaussian kernel size and σ value, it is possible to effectively extract the basis spectrum data that conforms to the definition of the main structure information from the original spectrum data, providing a reliable basis for the separation of detailed spectrum data. For example, when processing instrument audio containing rich high-frequency overtones, a smaller σ value can be selected to avoid excessive smoothing and loss of the instrument's unique main harmonic structure; while for audio that is dominated by low-frequency energy and has relatively little detailed information (such as deep drumbeats), a larger σ value can be selected to more thoroughly filter out high-frequency noise and highlight the low-frequency main contour.

[0093] Therefore, by performing such a sliding weighted summation operation on each spectral point of the original spectral data, the original spectral data can be smoothed, high-frequency fluctuations and subtle noise can be filtered out, thereby extracting the base spectral data that reflects the main structure. This sliding window weighted averaging method can effectively preserve the low-frequency main components and overall trends in the original spectrum, while weakening the local fluctuations caused by high-frequency details, providing a clear benchmark for the separation of detailed spectral data.

[0094] Besides using low-pass filtering to obtain detailed spectral data, high-pass filtering can also be used. Specifically, the raw spectral data can be input into a preset high-pass filter. The high-pass filter can filter the raw spectral data based on the cutoff frequency corresponding to the detailed frequency band characteristics of the audio signal, thereby preserving the high-frequency detail components in the raw spectral data and obtaining detailed spectral data.

[0095] For example, for a violin audio clip containing high-frequency overtones, the cutoff frequency of the high-pass filter can be set to 5kHz. This preserves the high-frequency overtone components above 5kHz (such as the subtle friction of vibrating strings and higher harmonics in the overtone series), while filtering out the low-frequency main melody below 5kHz. This directly yields detailed spectral data focused on high-frequency details. This approach is suitable for scenarios with a clear need to extract detailed information from specific frequency bands in the audio. By adjusting the cutoff frequency, detailed components in different frequency bands can be flexibly separated, providing targeted training data for the detail coding structure.

[0096] based on Figure 3 In another optional implementation of the embodiments shown and the various alternative implementations described above, the main coding structure and the detail coding structure are identical in network structure but their network parameters are independent. That is, the coding network through which the detail spectral data passes is identical to the main branch coding network of the original spectrum input in terms of network layer type (e.g., convolutional layers, activation functions), layer order, number of neurons, and other structural design. However, the weights, biases, and other parameters learned by these two branch networks during training are independent and not shared. The main branch coding network learns and optimizes parameters based on the overall features of the original spectrum, while the detail branch coding network independently learns and adjusts parameters specifically for the features of the detail spectrum separated from the original spectrum. The two do not interfere with each other or share parameters, thus enabling more targeted capture of the unique information of the detail spectrum.

[0097] The advantage of this setup is that the main coding structure can focus on learning deep feature representations of the main structural information in the raw spectral data, such as how to efficiently encode key elements that determine the basic outline of the audio, including the fundamental frequency, major harmonics, and overall energy distribution. Meanwhile, the detail coding structure can independently optimize for the characteristics of the detail spectral data, such as learning unique patterns of subtle features like breath sounds and sibilance, and how to better preserve this information, crucial for auditory perception, during compression. Because the parameters are independent, the optimization of the main coding structure is not affected by the learning process of the detail coding structure, and vice versa.

[0098] For example, when the model optimizes the main spectral loss to improve the reproduction of main information, the parameters of the detail coding structure are not adjusted in this process, thus ensuring that its ability to capture detailed features is not weakened by the main optimization objective. Similarly, when optimizing the detail spectral loss to enhance detail representation, the main coding ability already learned by the main coding structure remains stable. This independence allows the two coding structures to perform their respective functions, achieving better coding results in their respective information dimensions, thereby improving the overall quality of the encoding and decoding model in reproducing audio signals from main to detail.

[0099] For example, Figure 4 The flowchart illustrates an exemplary application scenario of the method in this embodiment. As shown in the figure, after the audio signal undergoes a Fourier transform to obtain spectral data, it is divided into a main branch and detail branches. In the main branch, the original spectral data is directly input to the main coding structure. This main coding structure has the same network architecture as the detail coding structure, such as containing several convolutional layers, pooling layers, and activation function layers, but their network parameters are independent. The main coding structure extracts and compresses features from the original spectral data, outputting the main coding features.

[0100] Meanwhile, in the detail branch, the original spectrum data is first extracted by means of Gaussian low-pass filtering, and then the difference between the original spectrum data and the base spectrum data is obtained to obtain the detail spectrum data. This detail spectrum data is then input into the detail coding structure with independent parameters to encode the detail features and output the detail coded features.

[0101] The master coding features and detail coding features are concatenated and fused. The resulting concatenated coding features are then processed into integer sequences by quantizers (e.g., using RVQ) and input into the decoder to obtain the decoding result of the concatenated coding features. The decoding result of the concatenated coding features is compared with the original spectral data, and the difference between the two is calculated to construct the master spectral loss (e.g., master Mel loss).

[0102] Similarly, detail-encoded features are input separately into the quantizer and decoder to obtain the decoded detail spectrum. Detail spectrum loss (e.g., detail Mel-ray loss) is calculated between the decoded and original detail spectra. The main spectrum loss and detail spectrum loss are weighted and summed according to a preset weight ratio (e.g., setting the main spectrum loss weight to 0.7 and the detail spectrum loss weight to 0.3 based on the different importance of main and detail information to audio perception) to obtain the fusion loss function. During training, the model continuously adjusts the network parameters of the main encoding structure, detail encoding structure, and decoding structure through backpropagation to minimize this fusion loss function.

[0103] In this way, the model can optimize the accuracy of main information restoration while preserving detailed features, thereby achieving a more comprehensive and delicate encoding and decoding effect for audio signals. For example, when training an audio clip containing vocals and piano accompaniment, the optimization of the main spectrum loss ensures that the fundamental frequency of the vocals, the melodic direction, and the main harmonic structure of the piano are accurately restored, while the optimization of the detail spectrum loss focuses on preserving the breaths and vibrato details of the vocals, as well as the subtle overtones of the piano keys and pedal resonance, so that the decoded and reconstructed audio can closely resemble the original audio in terms of overall outline and detailed texture.

[0104] The main coding structure and the detail coding structure described above use the same network structure. Alternatively, the detail coding structure can also employ a network design with higher resolution or a greater focus on local features. For example, it can use smaller convolutional kernels, deeper network layers, or attention mechanisms to precisely capture information such as high-frequency overtones, transient impulses, weak environmental noise, and subtle changes in timbre contained in the detail spectral data, and encode this detailed information to output the encoded features of the detail spectral data. This embodiment does not limit this approach; the network architecture of the detail coding structure can be specifically designed according to the characteristics of the detail spectral data.

[0105] The computer device in the embodiments of this application is described below. Please refer to [link / reference]. Figure 5 One embodiment of the computer device in this application includes:

[0106] The computer device 500 may include one or more central processing units (CPUs) 501 and a memory 505, in which one or more applications or data are stored.

[0107] The memory 505 can be volatile or persistent storage. The program stored in the memory 505 can include one or more modules, each module including a series of instruction operations on the computer device. Furthermore, the central processing unit 501 can be configured to communicate with the memory 505 and execute the series of instruction operations stored in the memory 505 on the computer device 500.

[0108] Computer device 500 may also include one or more power supplies 502, one or more wired or wireless network interfaces 503, one or more input / output interfaces 504, and / or one or more operating systems, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0109] The central processing unit 501 can perform the aforementioned... Figure 3 The operations performed by the computer device in the illustrated embodiments and their various alternative embodiments are not described in detail here.

[0110] This application also provides a computer storage medium, one embodiment of which includes: the computer storage medium storing instructions, which, when executed on a computer, cause the computer to perform the aforementioned... Figure 3 The operations performed by the computer device in the illustrated embodiments and their various alternative embodiments.

[0111] This application also provides a computer program product, one embodiment of which includes: when the computer program product is run on a computer device, causing the computer device to perform the aforementioned... Figure 3 The operations performed by the computer device in the illustrated embodiments and their various alternative embodiments.

[0112] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0113] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0114] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0115] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0116] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for training an encoding / decoding model, characterized in that, The method includes: Acquire raw spectral data of the audio signal and detailed spectral data for characterizing the audio signal; The original spectral data is encoded using the main coding structure of the initial coding-decoding model to obtain the main coding features, and the detailed spectral data is encoded using the detailed coding structure of the initial coding-decoding model to obtain the detailed coding features. Based on the main coding features, a first decoding process is performed to obtain a first decoding result; A second decoding process is performed based on the aforementioned detail coding features to obtain a second decoding result; The initial codec model is optimized based on the first decoding result and the second decoding result until the convergence condition is met, thereby obtaining the target codec model.

2. The method according to claim 1, characterized in that, The optimization of the initial encoding / decoding model based on the first decoding result and the second decoding result includes: The main spectrum loss is constructed based on the original spectrum data and its corresponding first decoding result; Construct a detailed spectrum loss based on the detailed spectrum data and its corresponding second decoding result; The initial encoding / decoding model is optimized based on the main spectrum loss and the detail spectrum loss.

3. The method according to claim 2, characterized in that, The optimization of the initial encoding / decoding model based on the main spectrum loss and the detail spectrum loss includes: The main spectrum loss and the detailed spectrum loss are weighted and fused to obtain the fusion loss; The initial encoding / decoding model is optimized based on the fusion loss.

4. The method according to claim 2, characterized in that, The first decoding process based on the main coding feature to obtain the first decoding result includes: The main coding feature and the detail coding feature are concatenated to obtain the concatenated coding feature; The concatenated coding feature is input into the initial encoding and decoding model, so that the quantizer structure in the initial encoding and decoding model quantizes the concatenated coding feature into an integer sequence, and the decoding structure in the initial encoding and decoding model decodes the integer sequence to obtain the decoding result corresponding to the concatenated coding feature; The construction of the main spectrum loss based on the original spectrum data and its corresponding first decoding result includes: The main spectrum loss is constructed based on the original spectrum data and the decoding results corresponding to the splicing coding features.

5. The method according to claim 1, characterized in that, Obtaining detailed spectral data for characterizing the audio signal includes: Obtain the base spectrum data corresponding to the original spectrum data, which is used to characterize the main structural information of the audio signal; The detailed spectrum data is obtained by subtracting the original spectrum data from the base spectrum data.

6. The method according to claim 5, characterized in that, The step of obtaining the base spectrum data corresponding to the original spectrum data for characterizing the main structural information of the audio signal includes: The preset filtering operator is slid sequentially in the original spectrum data. For the center spectrum point targeted by the filtering operator at each slide, the amplitude of the center spectrum point and its neighboring spectrum points is calculated by weighted summation based on the weight represented by the filtering operator. The calculation result is used as the new amplitude of the center spectrum point. The base spectrum data is obtained when the new amplitude of each spectrum point in the original spectrum data is obtained.

7. The method according to claim 1, characterized in that, Obtaining detailed spectral data for characterizing the audio signal includes: The original spectrum data is input to a preset high-pass filter, so that the high-pass filter filters the original spectrum data based on the cutoff frequency corresponding to the detail frequency band characteristics of the audio signal, so as to retain the high-frequency detail components in the original spectrum data and obtain the detail spectrum data.

8. The method according to any one of claims 1 to 7, characterized in that, The main coding structure and the detailed coding structure are identical in network structure, but their network parameters are independent of each other.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 8.

10. A computer storage medium, characterized in that, The computer storage medium stores instructions that, when executed on the computer, cause the computer to perform the method as described in any one of claims 1 to 8.

Images