Encoding and decoding method, training method, encoding device, decoding device and storage medium

Abstract

The invention discloses a coding and decoding method, a training method, a coding device, a decoding device and a storage medium. The method comprises the following steps: acquiring audio characteristics of first audio data; encoding the audio features to obtain first encoded data; wherein the encoder belongs to a first network model, and the first network model is obtained by performing self-supervision training based on a second network model. In the embodiment of the invention, the first network model is obtained by training in a self-supervision mode, and the encoder can keep semantic understanding ability in the processing process, so that the generated first coded data has better semantic expression ability, and semantic richness is improved while the audio reconstruction quality is ensured, thereby improving the audio processing effect.

Description

Technical Field

[0001] The present disclosure relates to the field of communication technology, and in particular to a coding and decoding method, a training method, a coding device, a decoding device, and a storage medium. Background Art

[0002] With the continuous advancement of audio processing technology, audio codecs play a crucial role in signal compression, transmission, and reconstruction. Audio codecs optimize acoustic fidelity through time-frequency transformation and quantization techniques, primarily focusing on minimizing distortion at the signal level. This results in the need to train separate models for downstream tasks such as speech recognition and sentiment analysis, leading to lower efficiency. Summary of the Invention

[0003] The embodiments of the present disclosure provide a coding and decoding method, a training method, a coding device, a decoding device, and a storage medium.

[0004] According to a first aspect of an embodiment of the present disclosure, a coding method is proposed, which is performed by an encoder. The method includes:

[0005] Acquire audio features of the first audio data;

[0006] Encoding the audio feature to obtain first encoded data;

[0007] The encoder belongs to a first network model, and the first network model is obtained by self-supervised training based on a second network model.

[0008] According to a second aspect of an embodiment of the present disclosure, a decoding method is proposed, which is performed by a decoder. The method includes:

[0009] Decoding the first encoded data to obtain second audio data;

[0010] The decoder belongs to a first network model, and the first network model is obtained by self-supervised training based on a second network model.

[0011] According to a third aspect of the embodiments of the present disclosure, a model training method is proposed, the method comprising:

[0012] Acquire a training sample set, wherein the training sample set includes a plurality of sample audio data;

[0013] Encoding the sample audio data by an encoder to obtain first sample encoded data;

[0014] Decoding the first sample coded data by a decoder to obtain sample decoded data;

[0015] Determining a first loss value according to the sample audio data, the sample decoded data, and a first loss function;

[0016] Parameters of the encoder and the decoder are updated according to the first loss value.

[0017] According to a fourth aspect of the embodiments of the present disclosure, a coding and decoding method is proposed, the method comprising:

[0018] The encoder obtains audio features of the first audio data;

[0019] The encoder encodes the audio feature to obtain first encoded data;

[0020] The decoder decodes the first encoded data to obtain second audio data;

[0021] The encoder and the decoder belong to a first network model, and the first network model is obtained by self-supervised training based on the second network model.

[0022] According to a fifth aspect of the embodiments of the present disclosure, a coding device is provided, the coding device including:

[0023] a processing module, configured to obtain audio features of the first audio data;

[0024] The processing module is further configured to encode the audio feature to obtain first encoded data;

[0025] The encoder belongs to a first network model, and the first network model is obtained by self-supervised training based on a second network model.

[0026] According to a sixth aspect of the embodiments of the present disclosure, a decoding device is provided, the decoding device including:

[0027] a processing module, configured to decode the first encoded data to obtain second audio data;

[0028] The decoder belongs to a first network model, and the first network model is obtained by self-supervised training based on a second network model.

[0029] According to a seventh aspect of an embodiment of the present disclosure, a coding device is proposed, which may include: one or more processors; wherein the coding device can be used to execute the optional implementation method of the first aspect.

[0030] According to an eighth aspect of an embodiment of the present disclosure, a decoding device is proposed, which may include: one or more processors; wherein the decoding device can be used to execute the optional implementation method of the second aspect.

[0031] According to the ninth aspect of the embodiments of the present disclosure, a coding and decoding system is proposed, comprising an encoder and a decoder, wherein the encoder is configured to execute the method described in the optional implementation manner of the first aspect, and the decoder is configured to execute the method described in the optional implementation manner of the second aspect.

[0032] According to a tenth aspect of an embodiment of the present disclosure, a storage medium is proposed, which stores instructions. When the instructions are executed on a communication device, the communication device executes the method described in the optional implementation of the first aspect or the second aspect.

[0033] According to the eleventh aspect of the embodiments of the present disclosure, a program product is proposed, comprising at least one of a program and an instruction, wherein when the at least one of the program and the instruction is executed by a communication device, the method described in the optional implementation manner of the first aspect or the second aspect is implemented.

[0034] The technical solution provided by the embodiments of the present disclosure can produce the following beneficial effects: obtaining audio features of first audio data; encoding the audio features to obtain first encoded data; wherein the encoder belongs to a first network model, which is self-supervised training based on a second network model. The first network model is trained using a self-supervised method, and the encoder is able to retain its semantic understanding capabilities during processing, resulting in better semantic expression of the generated first encoded data. This improves the semantic richness while maintaining the quality of audio reconstruction, thereby enhancing the audio processing effect.

[0035] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following drawings required for describing the embodiments are introduced. The following drawings are merely some embodiments of the present disclosure and do not impose specific limitations on the protection scope of the present disclosure.

[0037] Figure 1 1 is a schematic diagram of the architecture of a coding and decoding system according to an embodiment of the present disclosure.

[0038] Figure 2A This is an interactive schematic diagram of a coding and decoding method according to an embodiment of the present disclosure.

[0039] Figure 2B It is a flowchart of a model training method according to an embodiment of the present disclosure.

[0040] Figure 3A It is a flowchart of an encoding method according to an embodiment of the present disclosure.

[0041] Figure 3B It is a flowchart of an encoding method according to an embodiment of the present disclosure.

[0042] Figure 4A It is a flowchart of a decoding method according to an embodiment of the present disclosure.

[0043] Figure 4B It is a flowchart of a decoding method according to an embodiment of the present disclosure.

[0044] Figure 5A It is a schematic diagram of a model architecture according to an embodiment of the present disclosure.

[0045] Figure 5B The figure is a flowchart of a quantization process according to an embodiment of the present disclosure.

[0046] Figure 5C It is a schematic diagram of a decoding process according to an embodiment of the present disclosure.

[0047] Figure 5D 1 is a schematic diagram of the architecture of a decoder according to an embodiment of the present disclosure.

[0048] Figure 6A It is a structural diagram of an encoding device proposed in an embodiment of the present disclosure.

[0049] Figure 6B It is a structural diagram of a decoding device proposed in an embodiment of the present disclosure.

[0050] Figure 7A It is a structural diagram of the communication device proposed in the embodiment of the present disclosure.

[0051] Figure 7B It is a schematic diagram of the structure of the chip proposed in the embodiment of the present disclosure. DETAILED DESCRIPTION

[0052] The embodiments of the present disclosure provide a coding and decoding method, a training method, a coding device, a decoding device, and a storage medium.

[0053] In a first aspect, an embodiment of the present disclosure provides an encoding method, which is performed by an encoder. The method includes:

[0054] Acquire audio features of the first audio data;

[0055] Encoding the audio feature to obtain first encoded data;

[0056] The encoder belongs to a first network model, and the first network model is obtained by self-supervised training based on a second network model.

[0057] In the above embodiment, the first network model is obtained through self-supervision training, and the encoder can retain the semantic understanding ability during the processing process, so that the generated first encoded data has better semantic expression ability, while ensuring the quality of audio reconstruction, the semantic richness is improved, thereby improving the audio processing effect.

[0058] In conjunction with some embodiments of the first aspect, in some embodiments, the method further includes:

[0059] The first encoded data is sent to a decoder.

[0060] In the above embodiment, the first encoded data is sent to the decoder for decoding, which can effectively enhance the semantic expression capability of the decoded output audio while ensuring the quality of audio reconstruction, thereby achieving a better audio processing effect.

[0061] In conjunction with some embodiments of the first aspect, in some embodiments, encoding the audio feature to obtain first encoded data includes:

[0062] Performing masking processing on the audio feature to obtain a masked feature;

[0063] The first encoded data is generated according to the mask feature.

[0064] In the above embodiment, mask features are obtained by masking the audio features, so that the encoder can learn the key semantic information of the audio features and enhance the abstract expression capability of the encoded data for the audio features.

[0065] In conjunction with some embodiments of the first aspect, in some embodiments, generating the first encoded data according to the mask feature includes:

[0066] generating a latent representation of the mask features;

[0067] The latent representation is quantized to obtain the first encoded data.

[0068] In the above embodiment, by generating a latent representation of the mask feature and performing quantization processing to obtain the first encoded data, the core semantic information of the audio can be effectively refined, thereby improving the accuracy of the semantic expression.

[0069] In combination with some embodiments of the first aspect, in some embodiments, the first network model is a student model, and the second network model is a teacher model.

[0070] In the above embodiment, the second network model is used as the teacher model to perform self-supervisory training on the student model (the first network model), so that the student model can efficiently learn the knowledge of the teacher model and has stronger semantic expression ability, thereby improving the audio processing effect and semantic richness while ensuring the quality of audio reconstruction.

[0071] In a second aspect, an embodiment of the present disclosure provides a decoding method, which is performed by a decoder. The method includes:

[0072] Decoding the first encoded data to obtain second audio data;

[0073] The decoder belongs to a first network model, and the first network model is obtained by self-supervised training based on a second network model.

[0074] In the above embodiment, the first encoded data is decoded using the decoder in the first network model obtained based on self-supervised training of the second network model to obtain the second audio data, which effectively ensures the quality of audio reconstruction and improves the semantic richness and audio processing effect.

[0075] In conjunction with some embodiments of the second aspect, in some embodiments, the method further includes:

[0076] The first encoded data sent by the encoder is received.

[0077] In conjunction with some embodiments of the second aspect, in some embodiments, decoding the first encoded data to obtain the second audio data includes:

[0078] generating a short-time Fourier transform (STFT) coefficient sequence according to the first coded data;

[0079] Perform an inverse short-time Fourier transform on the STFT coefficient sequence to obtain the second audio data.

[0080] In the above embodiment, the first encoded data is converted into an STFT coefficient sequence, and the second audio data is obtained by inverse short-time Fourier transform. This can fully utilize the semantic and spectral information in the encoded data, accurately restore the audio time domain waveform, and improve the quality of audio reconstruction and audio processing effects.

[0081] In conjunction with some embodiments of the second aspect, in some embodiments, the inverse short-time Fourier transform is performed using the following formula:

[0082]

[0083] in, is the second audio data, m is the frame index, H is the frame shift, n is the time index, S[m] is the STFT coefficient vector of the mth frame in the STFT coefficient sequence, ISTFT() represents the inverse short-time Fourier transform, and w[n-mH] is the synthesis window function.

[0084] In the above embodiment, the STFT coefficient sequence generated by the first encoded data is subjected to an inverse short-time Fourier transform, which takes into account factors such as frame index, frame shift, time index, and synthesis window function, and can accurately convert the STFT coefficient sequence, thereby improving the accuracy and completeness of audio reconstruction.

[0085] In conjunction with some embodiments of the second aspect, in some embodiments, generating an STFT coefficient sequence according to the first encoded data includes:

[0086] Performing embedding processing on the first encoded data to obtain an embedded feature representation;

[0087] extracting a deep feature representation based on the embedded feature representation;

[0088] Performing linear projection on the deep feature representation to generate first parameters related to amplitude and phase;

[0089] The STFT coefficient sequence is determined according to the first parameter.

[0090] In the above embodiment, by sequentially performing embedding processing, deep feature extraction, and linear projection on the first encoded data to obtain parameters related to amplitude and phase, and then determining the STFT coefficient sequence, the audio information in the encoded data can be deeply extracted, and the frequency domain feature representation can be accurately constructed, thereby further improving the accuracy of audio reconstruction and the sound quality effect.

[0091] In conjunction with some embodiments of the second aspect, in some embodiments, extracting a deep feature representation based on the embedded feature representation includes:

[0092] Performing contextual relationship modeling on the embedded feature representation to obtain intermediate features;

[0093] Time-frequency information is extracted according to the intermediate features to obtain the deep feature representation.

[0094] In the above embodiment, by performing contextual relationship modeling on the embedded feature representation to obtain intermediate features, and extracting time-frequency information based on the intermediate features to obtain a deep feature representation, the contextual dependencies and local details of the audio features can be effectively captured, thereby further improving the accuracy and sound quality of audio reconstruction.

[0095] In combination with some embodiments of the second aspect, in some embodiments, the first network model is a student model and the second network model is a teacher model.

[0096] In a third aspect, the present disclosure provides a model training method, comprising:

[0097] Acquire a training sample set, wherein the training sample set includes a plurality of sample audio data;

[0098] Encoding the sample audio data by an encoder to obtain first sample encoded data;

[0099] Decoding the first sample coded data by a decoder to obtain sample decoded data;

[0100] Determining a first loss value according to the sample audio data, the sample decoded data, and a first loss function;

[0101] Parameters of the encoder and the decoder are updated according to the first loss value.

[0102] In the above embodiment, the loss value is determined by using sample audio data, sample decoded data and the first loss function, and the encoder and decoder parameters are updated, thereby improving the accuracy of the model in extracting and reconstructing audio features.

[0103] In conjunction with some embodiments of the third aspect, in some embodiments, the first loss function is:

[0104]

[0105] in, is the first loss value, X is the sample audio data, is the sample decoded data, λ is the spectral loss weight, and STFT() represents short-time Fourier transform.

[0106] In the above embodiment, the first loss value calculated using the above formula can measure the error of the model encoding and decoding audio data from two dimensions: time domain and frequency domain, effectively constrain the update direction of the encoder and decoder parameters, improve the model's reconstruction accuracy of the audio data, and enhance the audio processing effect.

[0107] In conjunction with some embodiments of the third aspect, in some embodiments, the method further includes:

[0108] Predicting mask sample features of the sample audio data through a first network model and quantizing the mask sample features to obtain second sample encoded data, wherein the first network model includes an encoder and a decoder;

[0109] Predicting global sample features of the sample audio data using a second network model, where the second network model includes an encoder and does not include a decoder, and the encoder of the second network model has the same structure as the encoder of the first network model;

[0110] Determining a second loss value according to the mask sample feature, the global sample feature, and the second sample encoding data;

[0111] Update parameters of the encoder of the first network model according to the second loss value.

[0112] In the above embodiment, the second loss value is determined by masking the sample features, the global sample features and the second sample encoding data, and the parameters of the first network model encoder are updated according to the second loss value. This can enhance the first network model encoder's ability to learn the local and global features of the audio data, optimize the encoder's extraction and representation of audio features, and improve the performance and semantic understanding capabilities of the first network model in audio processing tasks.

[0113] In conjunction with some embodiments of the third aspect, in some embodiments, determining the second loss value based on the mask sample feature, the global sample feature, and the second sample encoding data includes:

[0114] Determining a third loss value according to the mask sample feature, the global sample feature, and the second loss function;

[0115] determining a fourth loss value according to the second sample encoded data and a third loss function;

[0116] The second loss value is determined according to the third loss value and the fourth loss value.

[0117] In the above embodiment, the determined second loss value can comprehensively measure the model's prediction and encoding errors of audio features from different angles, more carefully guide the parameter update direction of the first network model encoder, and enhance the encoder's learning and representation capabilities of audio data features, thereby improving the model's audio processing performance.

[0118] In conjunction with some embodiments of the third aspect, in some embodiments, the second loss function is:

[0119]

[0120] Among them, L smooth (x, y) is the third loss value, x is the mask sample feature, y is the global sample feature, and β is the smoothing coefficient.

[0121] In the above embodiment, the third loss value is calculated by the above formula, which can smoothly measure the difference between the mask sample features and the global sample features, improve the model's ability to extract and represent audio features, and thus improve the model's audio processing effect.

[0122] In conjunction with some embodiments of the third aspect, in some embodiments, the third loss function is:

[0123]

[0124] Among them, L vq is the fourth loss value, Z is the second sample encoding data, is the quantized output of the lth layer, c q(z) is the codebook vector with index q(z) at the lth level, and L is the number of quantization levels.

[0125] In the above embodiment, the difference between the second sample encoded data and the sum of the quantized outputs of each layer, as well as the difference between the quantized output of each layer and the corresponding codebook vector, can accurately measure the error in the quantization process, provide precise direction guidance for updating the parameters of the first network model encoder, optimize the model's quantization processing of audio features, improve the quality and expression ability of the encoded data, and thus improve the performance and effect of the entire audio processing model.

[0126] In conjunction with some embodiments of the third aspect, in some embodiments, the method further includes:

[0127] The parameters of the encoder of the second network model are updated according to the parameters of the encoder of the first network model.

[0128] In the above embodiment, the parameters of the second network model encoder with the same structure are updated according to the parameters of the first network model encoder, which can share the experience of the first network model encoder in extracting and representing audio data features, enhance the second network model's ability to predict the global features of audio data, and reduce training costs and time.

[0129] In conjunction with some embodiments of the third aspect, in some embodiments, the parameters of the encoder of the second network model are updated by the following formula:

[0130]

[0131] in, is the parameter of the encoder of the second network model during training at step t, is the parameter of the encoder of the second network model during training at step t-1, is the parameter of the encoder of the first network model during training at the tth step, and α is the smoothing coefficient.

[0132] In the above embodiment, by updating the parameters of the encoder of the second network model through the above formula, the encoder of the second network model can gradually absorb the optimization results of the encoder of the first network model, effectively improve its ability to extract and represent audio data features, and enhance the stability and performance of the model.

[0133] In conjunction with some embodiments of the third aspect, in some embodiments, the method further includes:

[0134] Inputting the sample decoded data and the sample audio data into a discriminator to obtain a fifth loss value;

[0135] The parameters of the decoder are updated according to the fifth loss value.

[0136] In the above embodiment, the sample decoded data and the sample audio data are input into the discriminator to obtain the fifth loss value, and the parameters of the decoder are updated according to the fifth loss value. With the help of the discriminator's discrimination feedback on the sample decoded data and the original sample audio data, the decoder can be effectively guided to optimize its own parameters, thereby improving the decoder's decoding ability of the encoded data, thereby improving the quality and accuracy of audio reconstruction.

[0137] In a fourth aspect, an embodiment of the present disclosure provides a coding and decoding method, the method comprising:

[0138] The encoder obtains audio features of the first audio data;

[0139] The encoder encodes the audio feature to obtain first encoded data;

[0140] The decoder decodes the first encoded data to obtain second audio data;

[0141] The encoder and the decoder belong to a first network model, and the first network model is obtained by self-supervised training based on the second network model.

[0142] In a fifth aspect, an embodiment of the present disclosure proposes a coding device, which may include at least one of a transceiver module and a processing module; wherein the coding device can be used to execute the optional implementation method of the first aspect.

[0143] In a sixth aspect, an embodiment of the present disclosure proposes a decoding device, which may include at least one of a transceiver module and a processing module; wherein the decoding device may be used to execute the optional implementation method of the second aspect.

[0144] In a seventh aspect, an embodiment of the present disclosure proposes a coding device, which may include: one or more processors; wherein, the coding device can be used to execute the optional implementation method of the first aspect.

[0145] In an eighth aspect, an embodiment of the present disclosure proposes a decoding device, which may include: one or more processors; wherein, the decoding device can be used to execute the optional implementation method of the second aspect.

[0146] In the ninth aspect, an embodiment of the present disclosure proposes a coding and decoding system, which may include: an encoder and a decoder; wherein, the encoder is configured to execute the method described in the optional implementation of the first aspect, and the decoder is configured to execute the method described in the optional implementation of the second aspect.

[0147] In a tenth aspect, an embodiment of the present disclosure proposes a storage medium storing instructions, which, when executed on a communication device, enables the communication device to execute the method described in the optional implementation of the first aspect or the second aspect.

[0148] In an eleventh aspect, an embodiment of the present disclosure proposes a program product, which, when executed by a communication device, enables the communication device to execute the method described in the optional implementation manner of the first aspect or the second aspect.

[0149] In a twelfth aspect, an embodiment of the present disclosure proposes a computer program, which, when executed on a computer, enables the computer to execute the method described in the optional implementation of the first or second aspect.

[0150] In a thirteenth aspect, an embodiment of the present disclosure provides a chip or a chip system, wherein the chip or chip system includes a processing circuit configured to execute the method described in the optional implementation of the first aspect or the second aspect.

[0151] It is understandable that the above-mentioned encoders, decoders, encoding devices, decoding devices, storage media, program products, computer programs, chips, or chip systems can all be used to perform the methods proposed in the embodiments of the present disclosure. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects of the corresponding methods and will not be repeated here.

[0152] The present disclosure provides a coding and decoding method, training method, coding device, decoding device, and storage medium. In some embodiments, the terms coding and decoding method, audio processing method, information coding and decoding method, etc., are interchangeable; the terms coding and decoding device, audio processing device, information coding and decoding device, communication device, etc., are interchangeable; and the terms coding and decoding system, information processing system, etc., are interchangeable.

[0153] The embodiments of the present disclosure are not exhaustive and are merely illustrative of some embodiments, and are not intended to be a specific limitation on the scope of protection of the present disclosure. In the absence of contradiction, each step in a certain embodiment can be implemented as an independent embodiment, and the steps can be arbitrarily combined. For example, a solution after removing some steps in a certain embodiment can also be implemented as an independent embodiment, and the order of the steps in a certain embodiment can be arbitrarily exchanged. In addition, the optional implementation methods in a certain embodiment can be arbitrarily combined; in addition, the embodiments can be arbitrarily combined. For example, some or all steps of different embodiments can be arbitrarily combined, and a certain embodiment can be arbitrarily combined with the optional implementation methods of other embodiments.

[0154] In each embodiment of the present disclosure, unless otherwise specified or provided for by logic, the terms and / or descriptions between the embodiments are consistent and can be referenced by each other. The technical features in different embodiments can be combined to form a new embodiment based on their inherent logical relationships.

[0155] The terms used in the embodiments of the present disclosure are only for the purpose of describing specific embodiments and are not intended to limit the present disclosure.

[0156] In the embodiments of the present disclosure, unless otherwise specified, elements expressed in the singular, such as "a", "an", "the", "above", "said", "the", "the", etc., may mean "one and only one", or "one or more", "at least one", etc. For example, when using articles such as "a", "an", "the" in English in translation, the noun following the article may be understood as a singular expression or a plural expression.

[0157] In some embodiments, "plurality" may refer to two or more.

[0158] In some embodiments, the terms "at least one," "one or more," "a plurality of," "multiple," etc. may be used interchangeably.

[0159] In some embodiments, descriptions such as "at least one of A and B," "A and / or B," "A in one case, B in another case," or "in response to one case A, in response to another case B" may include the following technical solutions depending on the situation: in some embodiments, A (A is executed independently of B); in some embodiments, B (B is executed independently of A); in some embodiments, execution is selected from A and B (A and B are selectively executed); and in some embodiments, A and B (both A and B are executed). The above is also applicable when there are more branches such as A, B, and C.

[0160] In some embodiments, "A or B" and other descriptions may include the following technical solutions depending on the situation: in some embodiments, A (A is executed independently of B); in some embodiments, B (B is executed independently of A); in some embodiments, execution is selected from A and B (A and B are selectively executed). The above is also applicable when there are more branches such as A, B, C, etc.

[0161] The prefixes such as "first" and "second" in the embodiments of the present disclosure are only used to distinguish different description objects and do not constitute any restriction on the position, order, priority, quantity or content of the description objects. For the statement of the description object, please refer to the description in the context of the claims or embodiments, and no unnecessary restriction should be constituted due to the use of prefixes. For example, if the description object is a "field", the ordinal number before the "field" in the "first field" and the "second field" does not limit the position or order between the "fields". "First" and "second" do not limit whether the "fields" they modify are in the same message, nor do they limit the order of the "first field" and the "second field". For another example, if the description object is a "level", the ordinal number before the "level" in the "first level" and the "second level" does not limit the priority between the "levels". For another example, the number of description objects is not limited by the ordinal number and can be one or more. Taking "first device" as an example, the number of "devices" can be one or more. In addition, the objects modified by different prefixes can be the same or different. For example, if the description object is "device", then the "first device" and the "second device" can be the same device or different devices, and their types can be the same or different; for another example, if the description object is "information", then the "first information" and the "second information" can be the same information or different information, and their contents can be the same or different.

[0162] In some embodiments, “including A,” “comprising A,” “used to indicate A,” and “carrying A” can be interpreted as directly carrying A or indirectly indicating A.

[0163] In some embodiments, terms such as "in response to...", "in response to determining...", "in the case of...", "at the time of...", "when...", "if...", "if...", etc. can be used interchangeably.

[0164] In some embodiments, terms such as "greater than", "greater than or equal to", "not less than", "more than", "more than or equal to", "not less than", "higher than", "higher than or equal to", "not less than", and "above" can be replaced with each other, and terms such as "less than", "less than or equal to", "not greater than", "less than", "less than or equal to", "not more than", "lower than", "lower than or equal to", "not higher than", and "below" can be replaced with each other.

[0165] In some embodiments, devices and the like can be interpreted as physical or virtual, and their names are not limited to those described in the embodiments. Terms such as "device," "equipment," "device," "circuit," "network element," "node," "function," "unit," "section," "system," "network," "chip," "chip system," "entity," and "subject" can be used interchangeably.

[0166] In some embodiments, "network" can be interpreted as devices included in the network (eg, access network equipment, core network equipment, etc.).

[0167] In some embodiments, the terms "Access Network Device (AN Device)", "Radio Access Network Device (RAN Device)", "Base Station (BS)", "Radio Base Station (Radio Base Station)", "Fixed Station (Fixed Station)", "Node (Node)", "Access Point (Access Point)", "Transmission Point (TP)", "Reception Point (RP)", "Transmission and / or Reception Point (TRP))", "Panel (Panel)", "Antenna Panel (Antenna Panel)", "Antenna Array (Antenna Array)" "Cell (Cell)", "Macro Cell (Macro Cell)", "Small Cell (Small Cell)", "Femto Cell (Femto Cell)", "Pico Cell (Pico Cell)" "Sector (Sector)", "Cell Group (Cell Group)", "Serving Cell", "Carrier (Carrier)", "Component Carrier (Component Carrier)", "Bandwidth Part (BWP)" and the like can be used interchangeably.

[0168] In some embodiments, the terms "terminal", "terminal device", "user equipment (UE)", "user terminal" "mobile station (MS)", "mobile terminal (MT)", subscriber station (Subscriber Station), mobile unit (Mobile Unit), subscriber unit (Subscriber Unit), wireless unit (Wireless Unit), remote unit (Remote Unit), mobile device (Mobile Device), wireless device (Wireless Device), wireless communication device (Wireless Communication Device), remote device (Remote Device), mobile subscriber station (Mobile Subscriber Station), access terminal (Access Terminal), mobile terminal (Mobile Terminal), wireless terminal (Wireless Terminal), remote terminal (Remote Terminal), handset (Handset), user agent (User Agent), mobile client (Mobile Client), client (Client) and the like can be used interchangeably.

[0169] In some embodiments, the access network device, the core network device, or the network device can be replaced by a terminal. For example, the various embodiments of the present disclosure can also be applied to a structure in which the communication between the access network device, the core network device, or the network device and the terminal is replaced by communication between multiple terminals (for example, device-to-device (D2D), vehicle-to-everything (V2X), etc.). In this case, it is also possible to set the structure in which the terminal has all or part of the functions of the access network device. In addition, terms such as "uplink" and "downlink" can also be replaced by terms corresponding to communication between terminals (for example, "side"). For example, uplink channels, downlink channels, etc. can be replaced by side channels or direct channels, and uplinks, downlinks, etc. can be replaced by side links or direct links.

[0170] In some embodiments, the terminal may be replaced by an access network device, a core network device, or a network device. In this case, the access network device, the core network device, or the network device may have a structure that has all or part of the functions of the terminal.

[0171] In some embodiments, obtaining data, information, etc. may comply with the laws and regulations of the country where the data is obtained.

[0172] In some embodiments, data, information, etc. may be obtained with the user's consent.

[0173] In addition, each element, each row, or each column in the table of the embodiment of the present disclosure can be implemented as an independent embodiment, and the combination of any elements, any rows, and any columns can also be implemented as an independent embodiment.

[0174] Figure 1 FIG. 1 is a schematic diagram of an architecture of a coding and decoding system according to an embodiment of the present disclosure. Figure 1 As shown, the codec system 100 may include an encoder 101 and a decoder 102. It should be noted that the codec system 100 may also include other devices, and the present disclosure does not limit the devices included in the codec system 100.

[0175] In some embodiments, the encoder 101 and the decoder 102 may be provided in a terminal.

[0176] In some embodiments, the terminal may include at least one of a mobile phone, a wearable device, an Internet of Things device, a car with communication function, a smart car, a tablet computer, a computer with wireless transceiver function, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal device in industrial control, a wireless terminal device in self-driving, a wireless terminal device in remote medical surgery, a wireless terminal device in smart grid, a wireless terminal device in transportation safety, a wireless terminal device in smart city, and a wireless terminal device in smart home, but is not limited thereto.

[0177] It can be understood that the encoding and decoding system described in the embodiment of the present disclosure is for the purpose of more clearly illustrating the technical solution of the embodiment of the present disclosure, and does not constitute a limitation on the technical solution proposed in the embodiment of the present disclosure. Ordinary technicians in this field can know that with the evolution of the system architecture and the emergence of new business scenarios, the technical solution proposed in the embodiment of the present disclosure is also applicable to similar technical problems.

[0178] The following embodiments of the present disclosure can be applied to Figure 1The coding system 100 or a part thereof is shown, but is not limited thereto. Figure 1 The various subjects shown are examples, and the encoding and decoding system may include Figure 1 All or part of the subject, and may also include Figure 1 The number and form of other entities are arbitrary, each entity can be physical or virtual, the connection relationship between the entities is an example, the entities can be connected or disconnected, and the connection can be in any way, which can be direct or indirect, wired or wireless.

[0179] The embodiments of the present disclosure can be applied to Long Term Evolution (LTE), LTE-Advanced (LTE-A), LTE-Beyond (LTE-B), SUPER 3G, IMT-Advanced, 4th generation mobile communication system (4G), 5th generation mobile communication system (5G), 5G New Radio (NR), Future Radio Access (FRA), New-Radio Access Technology (RAT), New Radio (NR), New Radio Access (NX), Future generation radio access (FX), Global System for Mobile communications (GSM (registered trademark)), CDMA2000, Ultra Mobile Broadband (UMB), IEEE 802.11 (Wi-Fi (registered trademark)), IEEE 802.16 (WiMAX (registered trademark)), IEEE 802.20, Ultra-WideBand (UWB), Bluetooth (registered trademark), Public Land Mobile Network (PLMN) networks, Device-to-Device (D2D) systems, Machine-to-Machine (M2M) systems, Internet of Things (IoT) systems, Vehicle-to-Everything (V2X), systems utilizing other communication methods, and next-generation systems based on and extending these methods. Furthermore, multiple systems may be combined (for example, a combination of LTE or LTE-A with 5G).

[0180] In some embodiments of the present disclosure, audio codecs optimize acoustic fidelity through time-frequency transformation and quantization techniques, primarily focusing on minimizing distortion at the signal level. However, these approaches lack the ability to model semantic information such as speech content, emotion, and context. This results in the need to train separate models for downstream tasks such as speech recognition and sentiment analysis, resulting in low efficiency.

[0181] In some embodiments, self-supervised learning techniques have made significant progress in the field of audio processing, capable of extracting rich semantic features from unlabeled data. However, these techniques have not yet been effectively combined with the compression and reconstruction process of audio codecs, and cannot directly generate high-fidelity audio, nor do they solve the problem of joint optimization of semantic features and acoustic reconstruction. Therefore, it is difficult to simultaneously achieve the reconstruction quality and semantic richness of audio signals under high compression rates, limiting their performance in application scenarios that require high-quality audio and semantic understanding, such as speech synthesis, audio editing, and artificial intelligence (AI)-driven audio analysis.

[0182] Therefore, there is an urgent need to provide an audio codec system that can balance audio reconstruction quality and semantic expression capabilities at a high compression rate to meet the requirements of audio processing technology for efficiency and versatility.

[0183] Figure 2A FIG. 1 is an interactive diagram illustrating a coding and decoding method according to an embodiment of the present disclosure. The method can be executed by the above coding and decoding system. Figure 2A As shown, the method may include:

[0184] Step S2101: The encoder obtains audio features of first audio data.

[0185] In some embodiments, the encoder belongs to a first network model, which is obtained by self-supervised training based on a second network model.

[0186] In some embodiments, the first network model is a student model and the second network model is a teacher model. It should be understood that the encoder is trained based on the teacher-student model architecture.

[0187] In some embodiments, the first audio data may be collected by an encoder, or may be collected by a collection device and then sent to the encoder.

[0188] In some embodiments, the first audio data may include multiple frames of audio data.

[0189] In some embodiments, the encoder may obtain a frame of audio data at intervals. Optionally, the time period for the encoder to obtain audio data may be set by the encoder or may be a protocol agreement, which is not limited in the embodiments of the present disclosure.

[0190] In some embodiments, the first audio data may be a mono audio waveform signal with a sampling rate of 16 kHz.

[0191] In some embodiments, the encoder may include a pre-processing unit, and the audio features of the first audio data are obtained through the pre-processing unit.

[0192] In some embodiments, the preprocessing unit may include a 7-layer 1D convolutional neural network (CNN) feature extractor, the convolution kernel size of each CNN layer is 3, the step size is 2, the number of channels is [256, 256, 512, 512, 512, 512, 512] in sequence, each layer is group normalized (GroupNorm), the number of groups is 32, and it is normalized along the channel dimension.

[0193] In some embodiments, group normalization may be followed by processing using an activation function ReLU to obtain frame-level audio features.

[0194] In some embodiments, the formula for ReLU is:

[0195] ReLU(x)=max(0,x) (1)

[0196] Among them, ReLU(x) is the audio feature, and x is the normalized output.

[0197] For example, after the first audio data is input into a 7-layer 1D convolutional neural network, each time frame is converted into a 512-dimensional vector, where each dimension corresponds to an abstract feature such as pitch, timbre, or spectral energy distribution. The output audio features after ReLU processing are 512-dimensional time series embeddings (frame-level embeddings) with a temporal resolution of 1 / 10 the original waveform (approximately 10ms frame length).

[0198] Step S2102: The encoder performs mask processing on the audio features to obtain mask features.

[0199] In some embodiments, after extracting the audio features of the first audio data, the encoder may randomly select a mask starting position based on a Bernoulli distribution with a specified probability p, and continuously mask for L time steps to generate a set of masked regions. Optionally, the value of p may be 0.1. Optionally, the value of L may be 5.

[0200] In some embodiments, the mask region set is:

[0201]

[0202] Among them, M is the mask area set, m i Indicates whether the i-th time step (frame) is masked.

[0203] For example, when p=0.1 and L=5, the encoder can first traverse all time steps, and each position is marked as a candidate starting position with a probability of p=0.1, and the position that meets the continuous mask length is screened from the candidate starting positions. For example, if the candidate starting position is j, it is determined whether the five positions j to j+4 are all unoccupied. If so, it is determined that the candidate starting position is j and meets the continuous mask length, and the audio features corresponding to the five positions j to j+4 are masked, and the audio feature m is converted to j 、m j+1 、m j+2 、m j+3 、m j+4 Set to 1.

[0204] In some embodiments, the mask region set M may be a binary sequence indicating the masked time steps. For example, M = {m1, m2, ..., m T}, m i ∈{0,1},m i =1 means that the audio features of the i-th frame are masked, that is, all the audio features of the i-th frame are blocked, m i = 0 means that the audio feature of the i-th frame is not masked. Taking the mask region set M = {0, 1, 1, 1, 1, 1, 0, 0} as an example, the frame indexes of the masked audio features are 1, 2, 3, 4, and 5.

[0205] In some embodiments, the encoder can perform masking on the audio features according to the generated mask region set, replacing the audio features indicated by the mask region set to be masked with specific values, such as 0 or noise. Taking the mask region set M = {0, 1, 1, 1, 1, 0, 0} as an example, the audio features with frame indices 1, 2, 3, 4, and 5 are replaced with 0, while the audio features with frame indices 0, 6, and 7 remain unchanged, thereby obtaining the masked features after masking.

[0206] Step S2103: The encoder generates a latent representation of the mask feature.

[0207] In some embodiments, the encoder can project the mask feature into the latent space through a linear mapping of the embedding layer to obtain the audio embedding feature. For example, when projecting a 512-dimensional mask feature into a 512-dimensional latent space, the weight matrix dimension of the linear mapping is [512, 512].

[0208] In some embodiments, the encoder may generate a position encoding vector for each time step based on the time step index and the dimension index.

[0209] In some embodiments, the position vector may be generated by the sine-cosine method:

[0210]

[0211] Among them, formula (3) is used for even dimensions, PE(pos,2i) represents the position encoding vector of even dimensions, formula (4) is used for odd dimensions, PE(pos,2i+1) represents the position encoding vector of odd dimensions, pos is the time step index, i is the dimension index, d model Indicates the dimension of the model, and its value can be 512, for example.

[0212] In some embodiments, the encoder may add the position encoding vector to the corresponding audio embedding feature to obtain an input feature with position information, and process the input feature through the Transformer module to obtain a latent representation of the mask feature.

[0213] In some embodiments, the encoder can be configured with 8 Transformer layers, each with 8 attention heads, and the attention calculation formula is:

[0214] Among them, d k is the query dimension, the value can be 64, for example, Q is the query matrix, K is the key matrix, and V is the value matrix.

[0215] In some embodiments, the input features are projected into Q, K, and V through the Transformer module and split into 8 attention heads. The output of each attention head is calculated by formula (5), the outputs of the 8 attention heads are concatenated, and the attention result is obtained by a linear layer. The attention result is residually connected with the input feature, and then layer normalization is performed to obtain a normalized tensor. The normalized tensor is mapped to a 2048-dimensional hidden layer feature through the activation function GELU of the feedforward network, and then the 2048-dimensional hidden layer feature is mapped to 512 dimensions through a linear layer to obtain the output feature of the feedforward network. The output features of the feedforward network are residually connected with the input features of the current layer (y = x + f(x), y is the output after residual connection, x is the input feature of the current layer, and f(x) is the output feature of the feedforward network), and then layer normalization processing (LayerNorm(y)) is performed to obtain the output of the Transformer layer. The output of this Transformer layer can be used as the input of the output layer of the next Transformer layer. Continue the above processing and obtain the potential representation of the mask feature after 8 layers of Transformer processing.

[0216] In some embodiments, the formula of GELU is:

[0217] GELU(x)=x·Φ(x) (6)

[0218] Among them, GELU(x) is the 2048-dimensional hidden layer feature, x is the normalized tensor, and Φ(x) represents the standard normal distribution.

[0219] It should be noted that the values ​​of the above parameters are for illustrative purposes only and are not limited in the embodiments of the present disclosure.

[0220] Step S2104: The encoder quantizes the latent representation to obtain first encoded data.

[0221] In some embodiments, the encoder may discretize the latent representation into a token sequence to obtain the first encoded data.

[0222] In some embodiments, the encoder may include a vector quantization module, which performs quantization processing on the latent representation to obtain the first encoded data. For example, the vector quantization module may be a multi-layer residual vector quantization module.

[0223] In some embodiments, the vector quantization module can be configured as 4 layers, each layer contains 1024 entries, and the entry dimension is 512.

[0224] In some embodiments, a sample can be randomly selected from the potential representation as the first center point of the first layer codebook. According to P(x)∝D(x) 2 The probability of selecting a new center point is: x is a sample, P(x) represents the probability of x being selected as the new center point, and D(x) 2 Represents the minimum squared Euclidean distance from x to the selected center point of the current layer. After the center point is selected, it can be iteratively updated using the k-means algorithm to form the first-layer codebook.

[0225] In some embodiments, for each layer of codebook, the original potential vector z is quantized using the codebook of that layer, the distance from z to all center points of the codebook of that layer is calculated, and the nearest codebook entry is selected and quantized. For example, z can be quantized using formula (7) and formula (8):

[0226] q(z)=argmin j ||zc j ||2 (7)

[0227] Q(z)=c q(z) (8)

[0228] Where q(z) represents the index of the codebook entry with the smallest distance to z, c j represents the codebook entry, which is the jth vector in the codebook; c q(z) It is the codebook entry taken from the codebook according to the index q(z), Q(z) represents the quantized vector, that is, the codebook entry c q(z)Approximately represents the original latent vector z.

[0229] In some embodiments, after Q(z) is calculated, it can be quantified using the following formula:

[0230]

[0231] in, is the quantization result of the lth layer, is the residual vector of the l-1th layer, Q l is the projection operation of the quantizer of the first layer, which is to find the codebook entry closest to the input residual from the codebook of the first layer. It should be noted that the initial residual of the first layer quantization is

[0232] In some embodiments, the quantization result of the first layer is calculated Then, the residual is updated by the following formula:

[0233]

[0234] in, is the updated residual vector of the lth layer.

[0235] In some embodiments, the final quantization result can be expressed as:

[0236]

[0237] Among them, z g is the first coded data after quantization, L is the number of layers, and the value of L can be 4, for example.

[0238] Step S2105: The encoder sends first encoded data to the decoder.

[0239] In some embodiments, the decoder belongs to a first network model, which is obtained by self-supervised training based on a second network model.

[0240] In some embodiments, the first network model is a student model and the second network model is a teacher model.

[0241] In some embodiments, the student model includes an encoder and a decoder, and the teacher model includes an encoder but not a decoder.

[0242] Step S2106: The decoder performs embedding processing on the first encoded data to obtain an embedded feature representation.

[0243] In some embodiments, the decoder belongs to a first network model, which is obtained by self-supervised training based on a second network model.

[0244] In some embodiments, the decoder may perform embedding processing on the first encoded data using methods of related technologies to obtain an embedded feature representation, which will not be described in detail here.

[0245] Step S2107: The decoder performs context modeling on the embedded feature representation to obtain intermediate features.

[0246] In some embodiments, the embedded feature representation is input into a transformer module for context-critical modeling to obtain the intermediate feature. The transformer module may include 6 layers, with 8 attention heads in each layer.

[0247] In some embodiments, the context modeling window is 3s, single-sideband spectrum parameterization is adopted, and the spectrum coefficient dimension is nfft / 2+1, nfft=1024.

[0248] Step S2108: The decoder extracts time-frequency information based on the intermediate features to obtain a deep feature representation.

[0249] In some embodiments, the intermediate features are input into a ConvNeXt module to extract time-frequency information and obtain a deep feature representation. The ConvNeXt module contains 4 blocks with the number of channels [256, 128, 64, 32]. Each block includes a 7x7 depthwise convolution, layer normalization, and a 1x1 convolution.

[0250] Step S2109: The decoder performs linear projection on the deep feature representation to generate first parameters related to amplitude and phase.

[0251] In some embodiments, a linear layer is used to perform linear mapping on the deep feature representation to generate a first parameter related to amplitude and phase. The specific mapping method can be referred to the relevant technical records and will not be repeated here.

[0252] Step S2110: The decoder determines the STFT coefficient sequence according to the first parameter.

[0253] In some embodiments, the STFT coefficient is a complex number consisting of a real part and an imaginary part, which can be obtained by amplitude and phase conversion. The specific conversion method can be referred to the records of related technologies and will not be repeated here.

[0254] Step S2111: The decoder performs an inverse short-time Fourier transform on the STFT coefficient sequence to obtain second audio data.

[0255] In some embodiments, the inverse short-time Fourier transform is performed on the STFT coefficient sequence using the following formula:

[0256]

[0257] in, is the second audio data, M is the frame index, H is the frame shift, n is the time index, S[m] is the STFT coefficient vector of the mth frame in the STFT coefficient sequence, I S T F T () represents the inverse short-time Fourier transform, and w[n-mH] is the synthesis window function. For example, the frame shift H can take a value of 256.

[0258] In some embodiments, the STFT coefficient vector of each frame is subjected to the ISTFT by formula (12) to obtain the time domain frame y m [n], the length is the window length N, the value of N can be 1024, for example, m [n] is multiplied by the synthesis window w[n-mH], positioned at the position mH on the time axis, and then all the windowed time domain frames are superimposed on the time axis to obtain the reconstructed second audio data

[0259] Using the above method, the audio features are masked by the encoder to obtain mask features, and the first encoded data is generated based on the mask features. Since the encoder is trained through self-supervision, it can learn the key semantic information of the audio features, so that the generated first encoded data can have better semantic expression capabilities. Furthermore, the first encoded data is decoded by the decoder, which can effectively ensure the quality of audio reconstruction and improve the semantic richness and audio processing effect.

[0260] The method involved in the embodiment of the present disclosure may include at least one of the above-mentioned steps S2101 to S2111. For example, step S2101 can be implemented as an independent embodiment, step S2102 can be implemented as an independent embodiment, step S2103 can be implemented as an independent embodiment, step S2104 can be implemented as an independent embodiment, step S2105 can be implemented as an independent embodiment, step S2106 can be implemented as an independent embodiment, step S2107 can be implemented as an independent embodiment, step S2108 can be implemented as an independent embodiment, step S2111 can be implemented as an independent embodiment, step S2102+step S2103 can be implemented as an independent embodiment, step S2103+step S2104 can be implemented as an independent embodiment, step S2101+step S2102+step S2103+step S2104 can be implemented as an independent embodiment, step S2106+step S2107+step S2108+step S2109+step S2110+step S2111 can be implemented as independent embodiments, but is not limited thereto.

[0261] In some embodiments, steps S2101 to S2111 are optional, and one or more of these steps may be omitted or replaced in different embodiments.

[0262] In some embodiments, see Figure 2A Other optional implementations recorded before or after the corresponding description.

[0263] In some embodiments, the names of information, etc. are not limited to the names described in the embodiments, and terms such as "information", "message", "signal", "signaling", "report", "configuration", "indication", "instruction", "command", "channel", "parameter", "domain", "field", "symbol", "symbol", "codeword", "codebook", "codeword", "codepoint", "bit", "data", "program", and "chip" can be used interchangeably.

[0264] In some embodiments, the terms "codebook," "codeword," and "precoding matrix" may be used interchangeably. For example, a codebook may be a collection of one or more codewords / precoding matrices.

[0265] In some embodiments, terms such as "moment", "time point", "time", and "time position" can be replaced with each other, and terms such as "duration", "period", "time window", "window", and "time" can be replaced with each other.

[0266] In some embodiments, the terms "precoding", "precoder", "weight", "precoding weight", "Quasi-Co-Location (QCL)", "Transmission Configuration Indication (TCI) state", "Spatial Relation", "Spatial Domain Filter", "Transmission Power", "Phase Rotation", "Antenna Port", "Antenna Port Group", "Layer", "the number of layers", "Rank", "Resource", "Resource Set", "Resource Group", "Beam", "Beam Width", "Beam Angular Degree", "Antenna", "Antenna Element", "Panel" and the like can be used interchangeably.

[0267] In some embodiments, terms such as "frame", "radio frame", "subframe", "slot", "sub-slot", "mini-slot", "symbol", "symbol", and "transmission time interval (TTI)" can be used interchangeably.

[0268] In some embodiments, the terms "obtain," "get," "obtain," "receive," "transmit," "bidirectionally transmit," and "send and / or receive" are interchangeable and can be interpreted as meaning receiving from another entity, obtaining from a protocol, obtaining from a higher layer, obtaining through self-processing, or autonomous implementation. For example, the protocol includes at least one of a 3GPP protocol, a Wi-Fi protocol, and an audio and / or video protocol.

[0269] In some embodiments, terms such as "send", "transmit", "report", "download", "transmit", "bidirectional transmission", "send and / or receive" can be used interchangeably.

[0270] In some embodiments, terms such as "certain", "preset", "preset", "setting", "indicated", "a certain", "any", and "first" can be interchangeable. "Specific A", "preset A", "preset A", "setting A", "indicated A", "a certain A", "any A", and "first A" can be interpreted as A pre-specified in a protocol, etc., or as A obtained through setting, configuration, or indication, etc., or as specific A, a certain A, any A, or first A, etc., but not limited to this.

[0271] In some embodiments, the determination or judgment can be performed by a value represented by 1 bit (0 or 1), or by a true or false value (Boolean value) represented by true or false, or by comparison of numerical values ​​(for example, comparison with a predetermined value), but is not limited thereto.

[0272] In some embodiments, "not expecting to receive" can be interpreted as not receiving on time domain resources and / or frequency domain resources, or it can be interpreted as not performing subsequent processing on the data and / or instructions after receiving the data and / or instructions; "not expecting to send" can be interpreted as not sending, or it can be interpreted as sending but not expecting the recipient to respond to the content sent.

[0273] In some embodiments, if the arrow in the interaction diagram indicating the sending of information, signaling, etc. from one subject to another passes through other subjects, it can be interpreted as forwarding from one subject to another via other subjects, or it can be interpreted as sending from one subject to another without passing through other subjects.

[0274] Figure 2B FIG. 1 is a flow chart of a model training method according to an embodiment of the present disclosure. Figure 2B As shown, the method may include:

[0275] Step S2201: Obtain a training sample set.

[0276] In some embodiments, the training sample set may include a plurality of sample audio data, where the sample audio data may be audio segments with a sampling rate of 16 kHz and randomly cut into 10 s segments.

[0277] Step S2202: Encode the sample audio data through an encoder to obtain first sample encoded data.

[0278] In some embodiments, the encoder includes an encoder for a first network element model and an encoder for a second network model, the first network model may be a student model, the second network model may be a teacher model, and the encoder for the second network model has the same structure as the encoder for the first network model.

[0279] In some embodiments, the sample audio data is input into the encoder, processed by the CNN feature extractor and Transformer module of the encoder, and the sample potential representation is output. The specific structure of the CNN feature extractor can be referred to Figure 2A In step S2101 of the embodiment shown, the specific structure of the Transformer module can be referred to Figure 2A Step S2103 of the illustrated embodiment will not be described in detail here.

[0280] In some embodiments, the sample potential representation generates a token sequence through a multi-layer residual vector quantization module to obtain the first sample encoded data.

[0281] Step S2203: decode the first sample encoded data through a decoder to obtain sample decoded data.

[0282] In some embodiments, the decoder belongs to the first network model. It should be understood that the first network model includes an encoder and a decoder, and the second network model includes an encoder but does not include a decoder.

[0283] In some embodiments, the first sample encoded data output by the encoder is input into the decoder, decoded by the decoder, and sample decoded data is output.

[0284] It should be noted that the specific structure of the decoder can be referred to Figure 2A The specific implementation of steps S2107 and S2108 of the embodiment shown in the figure, in which the sample decoding data is obtained by the decoder, can be referred to. Figure 2A Steps S2106 to S2111 of the illustrated embodiment are not described in detail here.

[0285] Step S2204: Determine a first loss value according to the sample audio data, the sample decoded data, and the first loss function.

[0286] In some embodiments, the first loss function is:

[0287]

[0288] in, is the first loss value, X is the sample audio data, is sample decoding data, STFT() represents short-time Fourier transform, λ is spectral loss weight, and the value of λ may be 0.5, for example.

[0289] In some embodiments, the sample audio data and the sample decoded data are substituted into formula (13) to calculate a first loss value.

[0290] Step S2205: Update the parameters of the encoder and decoder according to the first loss value.

[0291] In some embodiments, after calculating the first loss value, the parameters of the encoder and decoder can be updated separately through back propagation. It should be noted that in the process of updating the parameters of the encoder and decoder, the parameters of the quantizer remain frozen and are not updated to avoid training oscillations and affect the training effect.

[0292] In some embodiments, the hyperparameters in the model training process may be: the learning rate uses cosine annealing scheduling, the annealing period is 50 training rounds (epochs), and the training is a total of 50 epochs.

[0293] Step S2206: Input the sample decoded data and the sample audio data into the discriminator to obtain a fifth loss value.

[0294] In some embodiments, the discriminator is a multi-scale discriminator and may include at least one of the following:

[0295] Multi-resolution discriminator (MRD): consists of 5 convolutional layers, each with a kernel size of [5,5], a stride of [2,2], and a number of channels of [32,64,128,256,512];

[0296] Multi-cycle Discriminator (MPD): The input audio is segmented into [2, 3, 5, 7, 11] different cycle lengths and one-dimensional convolution is performed on each segment.

[0297] STFT spectral discriminator: Calculates the L1 loss of Mel spectrum and linear spectrum, with 80 frequency bands.

[0298] In some embodiments, the sample decoded data and sample audio data are downsampled by different multiples, for example, 22.05kHz, 11.025kHz, and 5.5125kHz, and the sample decoded data and sample audio data of different resolutions obtained after sampling are input into the discriminator of the corresponding resolution, and the adversarial losses corresponding to the different resolutions are calculated, and multiple adversarial losses are fused, for example, by direct addition or weighted summation, to obtain the total MRD loss.

[0299] In some embodiments, the sample decoded data and the sample audio data are divided into sub-segments of different lengths, for example, 128ms, 256ms, and 512ms. The sub-segments of different lengths are input into the discriminator of the corresponding period, and the adversarial losses corresponding to the different periods are calculated. The multiple adversarial losses are fused, for example, by direct addition or weighted summation, to obtain the total MPD loss.

[0300] In some embodiments, the STFT spectrum of the sample decoded data and the sample audio data is calculated, the Mel spectrum is obtained as the input of the STFT spectrum discriminator, and the STFT loss is calculated.

[0301] In some embodiments, a fifth loss value is obtained by performing weighted summation or direct summation on the MRD total loss, the MPD total loss, and the STFT loss.

[0302] Step S2207: Update the decoder parameters according to the fifth loss value.

[0303] In some embodiments, parameters of the decoder are updated by back-propagation according to the fifth loss value.

[0304] In some embodiments, steps S2206 to S2207 may be performed before step S2204 or after step S2204, which is not limited in the embodiments of the present disclosure.

[0305] Step S2208: Predict the mask sample features of the sample audio data through the first network model, and quantize the mask sample features to obtain second sample encoded data.

[0306] In some embodiments, the sample audio data is input into the encoder of the first network model, the sample audio data is masked to obtain masked sample features, and multiple masked sample features are subjected to multi-layer residual vector quantization processing to obtain second sample encoded data.

[0307] It should be noted that the specific implementation method of generating the second sample coded data can refer to Figure 2A The implementation method of generating the first coded data in steps S2101 to S2105 of the illustrated embodiment will not be repeated here.

[0308] Step S2209: predict global sample features of the sample audio data using the second network model.

[0309] In some embodiments, the sample audio data is input into an encoder of a second network model for encoding processing, and a global sample feature is output. The second network model is a teacher model that receives the complete input sequence, i.e., does not perform masking processing, to generate the global sample feature.

[0310] Step S2210: Determine a third loss value based on the mask sample features, the global sample features, and the second loss function.

[0311] In some embodiments, the third loss value is a smoothed L1 loss for mask prediction, corresponding to the semantic enhancement objective of the teacher-student model.

[0312] In some embodiments, the second loss function is:

[0313]

[0314] Among them, L smooth (x, y) is the third loss value, x is the mask sample feature, y is the global sample feature, β is the smoothing coefficient, and the value of β can be 1.0, for example.

[0315] In some embodiments, the mask sample features and the global sample features are substituted into formula (14) to calculate a third loss value.

[0316] Step S2211: Determine a fourth loss value based on the second sample encoded data and the third loss function.

[0317] In some embodiments, the fourth loss value is a vector quantization loss, corresponding to a low bit rate compression target of the quantization module.

[0318] In some embodiments, the third loss function is:

[0319]

[0320] Among them, L vq is the fourth loss value, Z is the second sample encoding data, is the quantized output of the lth layer, c q(z) is the codebook vector with the index q(z) of the lth layer, L is the number of quantization levels, and the value of L can be 4.

[0321] In some embodiments, c q(z) The definition of L can refer to Figure 2A Step S2104 of the illustrated embodiment will not be described again here.

[0322] In some embodiments, the parameters corresponding to each layer of quantization are substituted into formula (15) to calculate the fourth loss value.

[0323] Step S2212: Determine a second loss value according to the third loss value and the fourth loss value.

[0324] In some embodiments, the third loss value and the fourth loss value may be directly summed or weighted summed to obtain the second loss value.

[0325] Step S2213: Update the parameters of the encoder of the first network model according to the second loss value.

[0326] In some embodiments, parameters of the encoder of the first network model are updated by backpropagation according to the second loss value.

[0327] In some embodiments, for the vector quantization module, the codebook may be updated every 10 batches, and the usage frequency of codebook entries may be checked every 1000 steps. If the usage frequency is less than 0.01%, the high-frequency entries may be replaced with Gaussian noise (mean 0, variance 0.01).

[0328] In some embodiments, during backpropagation, since the quantization operation is not differentiable, the gradient can be passed to the input through the pass-through estimator:

[0329]

[0330] In some embodiments, the codebook may be updated using online clustering, and the update formula is:

[0331]

[0332] in, represents the jth codebook entry after the tth iteration, c j represents the jth codebook entry, z i Indicates allocation in c j The potential representation vector of , N is the number of samples in the training sample set. i ∈c j ) is defined as:

[0333]

[0334] In some embodiments, the hyperparameters in the model training process can be: the learning rate is adjusted to 5×10 -5 , train for 30 epochs, cosine annealing scheduling.

[0335] It should be noted that, during the process of training the encoder of the first network model through steps S2208 to S2213, the parameters of the decoder can be frozen for outputting sample decoded data.

[0336] Step S2214: Update the parameters of the encoder of the second network model according to the parameters of the encoder of the first network model.

[0337] In some embodiments, the parameters of the second network model are dynamically updated by exponential moving average (EMA). It should be understood that after the parameters of the first network model are updated, the parameters of the second network model can be dynamically updated, with one training iteration per batch.

[0338] In some embodiments, the parameters of the encoder of the second network model are updated by the following formula:

[0339]

[0340] in, is the parameter of the encoder of the second network model during the t-th step training, is the parameter of the encoder of the second network model during the training of step t-1, is the parameter of the encoder of the first network model during the t-th step training, α is the smoothing coefficient, α∈[0.9,0.999].

[0341] Using the above method, a phased training method is used to solve the balance between acoustic fidelity and semantic expression ability in codec design. By first optimizing acoustic reconstruction and then freezing the decoder to jointly optimize the encoder and quantizer, mutual interference between the two objectives is avoided, and the accuracy of audio feature extraction and reconstruction of the codec is improved.

[0342] The method involved in the embodiment of the present disclosure may include at least one of the above steps S2201 to S2214. For example, step S2202 can be implemented as an independent embodiment, step S2203 can be implemented as an independent embodiment, step S2208 can be implemented as an independent embodiment, step S2209 can be implemented as an independent embodiment, step S2214 can be implemented as an independent embodiment, step S2201+step S2202+step S2203+step S2204+step S2205 can be implemented as an independent embodiment, step S2201+step S2202+step S2203+step S2206+step S2207 can be implemented as an independent embodiment. For example, step S2201+step S2202+step S2203+step S2206+step S2207+step S2214 can be implemented as independent embodiments, step S2201+step S2208+step S2209+step S2210+step S2211+step S2212+step S2213, step S2201+step S2208+step S2209+step S2210+step S2211+step S2212+step S2213+step S2214 can be implemented as independent embodiments, but are not limited to this.

[0343] In some embodiments, any two steps in steps S2201 to S2214 can be executed in an order swapped or simultaneously. For example, step S2206 and step S2214 can be executed in an order swapped or simultaneously.

[0344] In some embodiments, step S2206 and step S2207 are optional, and one or more of these steps may be omitted or replaced in different embodiments.

[0345] In some embodiments, steps S2208 to S2214 are optional, and one or more of these steps may be omitted or replaced in different embodiments.

[0346] In some embodiments, steps S2202 to S2207 are optional, and one or more of these steps may be omitted or replaced in different embodiments.

[0347] In some embodiments, step S2214 is optional, and one or more of these steps may be omitted or replaced in different embodiments.

[0348] Figure 3A FIG. 1 is a flow chart of an encoding method according to an embodiment of the present disclosure. Figure 3A As shown, the embodiment of the present disclosure relates to an encoding method, which can be performed by an encoder. The method may include:

[0349] Step S3101: Acquire audio features of first audio data.

[0350] The optional implementation of step S3101 can be found in Figure 2A Optional implementation of step S2101, and Figure 2A Other related parts in the embodiments involved will not be described in detail here.

[0351] Step S3102: Mask the audio features to obtain mask features.

[0352] The optional implementation of step S3102 can be found in Figure 2A Optional implementation of step S2102, and Figure 2A Other related parts in the embodiments involved will not be described in detail here.

[0353] Step S3103: Generate first encoded data according to the mask features.

[0354] The optional implementation of step S3103 can be found in Figure 2A Optional implementation of steps S2103 to S2104, and Figure 2AOther related parts in the embodiments involved will not be described in detail here.

[0355] Step S3104: Send the first coded data.

[0356] The optional implementation of step S3104 can be found in Figure 2A Optional implementation of step S2105, and Figure 2A Other related parts in the embodiments involved will not be described in detail here.

[0357] In some embodiments, the above steps are all optional steps.

[0358] Figure 3B FIG. 1 is a flow chart of an encoding method according to an embodiment of the present disclosure. Figure 3B As shown, the embodiment of the present disclosure relates to an encoding method, which can be performed by an encoder. The method may include:

[0359] Step S3201: Acquire audio features of first audio data.

[0360] The optional implementation of step S3201 can be found in Figure 2A Optional implementation of step S2101, and Figure 2A Other related parts in the embodiments involved will not be described in detail here.

[0361] Step S3202: Encode the audio features to obtain first encoded data.

[0362] The optional implementation of step S3202 can be found in Figure 2A Optional implementations of steps S2102 to S2104, and Figure 2A Other related parts in the embodiments involved will not be described in detail here.

[0363] In some embodiments, the method further comprises:

[0364] The first encoded data is sent to a decoder.

[0365] In some embodiments, encoding the audio feature to obtain first encoded data includes:

[0366] Performing masking processing on the audio feature to obtain a masked feature;

[0367] The first encoded data is generated according to the mask feature.

[0368] In some embodiments, generating the first encoded data according to the mask feature includes:

[0369] generating a latent representation of the mask features;

[0370] The latent representation is quantized to obtain the first encoded data.

[0371] In some embodiments, the first network model is a student model and the second network model is a teacher model.

[0372] Figure 4A FIG. 1 is a flowchart of a decoding method according to an embodiment of the present disclosure. Figure 4A As shown, the embodiment of the present disclosure relates to a decoding method, which can be performed by a decoder. The method may include:

[0373] Step S4101: Receive first coded data.

[0374] The optional implementation of step S4101 can be found in Figure 2A Optional implementation of step S2105, and Figure 2A Other related parts in the embodiments involved will not be described in detail here.

[0375] Step S4102: Generate an STFT coefficient sequence according to the first encoded data.

[0376] In some embodiments, the first encoded data is embedded to obtain an embedded feature representation, a deep feature representation is extracted based on the embedded feature representation, the deep feature representation is linearly projected to generate a first parameter related to the amplitude and phase, and an STFT coefficient sequence is determined based on the first parameter.

[0377] The optional implementation of step S4102 can be found in Figure 2A Optional implementation of steps S2106 to S2110, and Figure 2A Other related parts in the embodiments involved will not be described in detail here.

[0378] Step S4103: Perform inverse short-time Fourier transform on the STFT coefficient sequence to obtain second audio data.

[0379] The optional implementation of step S4103 can be found in Figure 2A Optional implementation of step S2111, and Figure 2A Other related parts in the embodiments involved will not be described in detail here.

[0380] In some embodiments, the above steps are all optional steps.

[0381] Figure 4B FIG. 1 is a flowchart of a decoding method according to an embodiment of the present disclosure. Figure 4BAs shown, the embodiment of the present disclosure relates to a decoding method, which can be performed by a decoder. The method may include:

[0382] Step S4201: Decode the first encoded data to obtain second audio data.

[0383] In some embodiments, the method further comprises:

[0384] The first encoded data sent by the encoder is received.

[0385] In conjunction with some embodiments of the second aspect, in some embodiments, decoding the first encoded data to obtain the second audio data includes:

[0386] generating a short-time Fourier transform (STFT) coefficient sequence according to the first coded data;

[0387] Perform an inverse short-time Fourier transform on the STFT coefficient sequence to obtain the second audio data.

[0388] In some embodiments, the inverse short-time Fourier transform is performed by the following formula:

[0389]

[0390] in, is the second audio data, m is the frame index, H is the frame shift, n is the time index, S[m] is the STFT coefficient vector of the mth frame in the STFT coefficient sequence, ISTFT() represents the inverse short-time Fourier transform, and w[n-mH] is the synthesis window function.

[0391] In some embodiments, generating an STFT coefficient sequence according to the first encoded data includes:

[0392] Performing embedding processing on the first encoded data to obtain an embedded feature representation;

[0393] extracting a deep feature representation based on the embedded feature representation;

[0394] Performing linear projection on the deep feature representation to generate first parameters related to amplitude and phase;

[0395] The STFT coefficient sequence is determined according to the first parameter.

[0396] In some embodiments, extracting a deep feature representation based on the embedded feature representation includes:

[0397] Performing contextual relationship modeling on the embedded feature representation to obtain intermediate features;

[0398] Time-frequency information is extracted according to the intermediate features to obtain the deep feature representation.

[0399] In some embodiments, the first network model is a student model and the second network model is a teacher model.

[0400] The embodiment of the present disclosure provides a semantically enhanced audio codec system, which belongs to the field of audio processing technology. In order to solve the problem of insufficient audio reconstruction quality and semantic expression ability at high compression rates, the system includes: an encoder module, which adopts a teacher-student model architecture to generate semantically enhanced latent representations through self-supervised mask prediction tasks; a multi-layer residual vector quantization module, which initializes the codebook based on the k-means++ algorithm, and compresses the latent representation into a low-bitrate token sequence through residual transfer and online clustering update mechanism; a decoder module, which combines the inverse short-time Fourier transform (STFT) with a multi-scale discriminator to reconstruct high-fidelity audio waveforms. The system optimizes semantic learning and acoustic reconstruction goals through staged training. The present disclosure achieves high-quality reconstruction of audio signals at low bit rates while improving semantic richness, and is suitable for scenarios such as speech synthesis and audio editing.

[0401] Embodiment 1: A semantically enhanced audio coding and decoding system, comprising:

[0402] An encoder module for generating semantically enhanced latent representations via a self-supervised mask prediction task;

[0403] A multi-layer residual vector quantization module to discretize the latent representation into a low-bitrate token sequence;

[0404] The decoder module combines the inverse short-time Fourier transform (STFT) with a multi-scale discriminator to reconstruct the token sequence into a high-fidelity audio waveform.

[0405] Embodiment 2, based on embodiment 1, the encoder module includes:

[0406] (a) Teacher-student model architecture, where the weights of the student model are dynamically updated by the teacher model through the exponential moving average (EMA), and the update formula is:

[0407]

[0408] Among them, α∈[0.9,0.999] is the smoothing coefficient, and the update frequency is once per batch training iteration;

[0409] (b) The mask strategy is based on Bernoulli distribution, randomly selecting the mask starting position with probability p = 0.1, and continuously masking for L = 5 time steps to generate a set of masked regions.

[0410] Embodiment 3, based on embodiment 1, the multi-layer residual vector quantization module includes:

[0411] (a) The codebook is initialized using the k-means++ algorithm. The number of codebook entries per layer is 1024, and the number of layers can be configured from 2 to 4.

[0412] (b) The residual transfer formula is:

[0413]

[0414] Among them, Q l represents the projection operation of the l-th layer quantizer, is the original latent representation output by the encoder;

[0415] (c) The codebook update mechanism adopts online clustering based on nearest neighbor assignment, and the update formula is:

[0416]

[0417] Among them, c j is the jth entry in the codebook, z i is the potential vector assigned to this position.

[0418] Embodiment 4, based on embodiment 1, the decoder module includes:

[0419] (a) The inverse short-time Fourier transform (STFT) uses a Hamming window function with a window length of N = 1024 and a step size of M = 256. The reconstruction formula is:

[0420]

[0421] (b) The decoder consists of a cascaded Transformer attention module and a ConvNeXt module, where the context modeling window of the attention module is 3 seconds, single-sideband spectrum parameterization is used, and the spectral coefficient dimension is nfft / 2+1, nfft=1024.

[0422] (c) The multi-scale discriminator includes:

[0423] Multi-resolution discriminator (MRD): consists of 5 convolutional layers, each with a kernel size of [5,5], a stride of [2,2], and a number of channels of [32,64,128,256,512];

[0424] Multi-cycle Discriminator (MPD): The input audio is segmented into [2, 3, 5, 7, 11] different cycle lengths and one-dimensional convolution is performed on each segment.

[0425] STFT spectral discriminator: Calculates the L1 loss of Mel spectrum and linear spectrum, with 80 frequency bands.

[0426] Example 5: Based on Example 1, the training method includes a phased strategy:

[0427] (a) The first stage trains the acoustic reconstruction objective, and the loss function is:

[0428]

[0429] Among them, λ = 0.5 is the spectral loss weight;

[0430] (b) The second stage freeze decoder jointly optimizes the encoder and quantizer, and the loss function is:

[0431]

[0432] in, is the smoothed L1 loss for mask prediction, corresponding to the semantic enhancement goal of the teacher-student model, is the vector quantization loss, corresponding to the low bit rate compression target of the multi-layer residual vector quantization module.

[0433] The semantically enhanced audio codec system of the disclosed embodiment achieves dual optimization of the reconstruction quality and semantic richness of audio signals under high compression rate conditions by integrating self-supervised learning, efficient quantization and high-quality reconstruction technology.

[0434] In some embodiments, the semantically enhanced audio codec system of the present disclosure includes:

[0435] Encoder module: used to generate semantically enhanced latent representations through a self-supervised mask prediction task;

[0436] Multi-layer residual vector quantization module: used to discretize the potential representation into a low-bitrate token sequence;

[0437] Decoder module: Combines the inverse short-time Fourier transform (ISTFT) with a multi-scale discriminator to reconstruct the token sequence into a high-fidelity audio waveform.

[0438] In some embodiments, the encoder module includes the following features:

[0439] (1) Figure 5A This is a schematic diagram of a model architecture according to an embodiment of the present disclosure. Figure 5A As shown in Figure 2, a teacher-student model architecture is adopted, where the student encoder is the encoder of the student model, the student decoder is the decoder of the student model, and the teacher encoder is the encoder of the teacher model. Semantically enhanced latent representations are generated through a self-supervised mask prediction task. The weights of the student model are dynamically updated by the teacher model through the exponential moving average (EMA). The update formula is:

[0440]

[0441] Among them, α∈[0.9,0.999] is the smoothing coefficient, and the update frequency is once per batch training iteration;

[0442] (2) Using a Bernoulli distribution-based masking strategy, we randomly select the mask starting position with probability p = 0.1 and continuously mask for L = 5 time steps to generate a set of masked regions:

[0443]

[0444] Through the mask prediction task, the encoder can capture the contextual dependencies and semantic information of the audio signal and improve the semantic richness of the potential representation.

[0445] In some embodiments, the multi-layer residual vector quantization module includes the following features:

[0446] (1) Figure 5B FIG. 1 is a flow chart of a quantization process according to an embodiment of the present disclosure. Figure 5B As shown in the figure, the k-means++ algorithm is used to initialize the codebook. The number of codebook entries in each layer is 1024, and the number of layers can be configured to 4.

[0447] (2) The residual transfer formula is:

[0448]

[0449] Among them, Q l represents the projection operation of the l-th layer quantizer, is the original latent representation output by the encoder.

[0450] (3) Codebook update mechanism: Online clustering based on nearest neighbor allocation is adopted, and the update formula is:

[0451]

[0452] Among them, c j is the jth entry in the codebook, z i is the potential vector assigned to this position.

[0453] In some embodiments, the module uses multi-layer residual quantization to preserve the subtle structure of the audio signal at a low bit rate, thereby improving compression efficiency and reconstruction quality.

[0454] In some embodiments, the decoder module includes the following features:

[0455] (1) Figure 5C FIG. 1 is a schematic diagram of a decoding process according to an embodiment of the present disclosure. Figure 5CAs shown in the figure, the inverse short-time Fourier transform (ISTFT) is combined with a multi-scale discriminator to reconstruct the token sequence into a high-fidelity audio waveform. The STFT uses a Hamming window function with a window length of N = 1024 and a step size of M = 256. The reconstruction formula is:

[0456]

[0457] in, is the second audio data, m is the frame index, H is the frame shift, S[m] is the STFT coefficient vector of the mth frame in the STFT coefficient sequence, ISTFT() represents the inverse short-time Fourier transform, w[n-mH] is the synthesis window function, and n is the time index.

[0458] (2) The Transformer attention module contains 6 self-attention layers, each with 8 attention heads, processing a 3-second context window; the ConvNeXt module uses a 7×7 depthwise separable convolution and an inverted bottleneck structure, with a channel expansion ratio of 4.

[0459] (3) Multi-scale discriminator includes:

[0460] Multi-resolution discriminator (MRD): consists of 5 convolutional layers, each with a kernel size of [5,5], a stride of [2,2], and a number of channels of [32,64,128,256,512];

[0461] Multi-cycle Discriminator (MPD): The input audio is segmented into [2, 3, 5, 7, 11] different cycle lengths and one-dimensional convolution is performed on each segment.

[0462] STFT spectral discriminator: Calculates the L1 loss of Mel spectrum and linear spectrum, with 80 frequency bands.

[0463] In some embodiments, the system is trained using a phased strategy:

[0464] The first stage trains the acoustic reconstruction objective, and the loss function is:

[0465]

[0466] Wherein, λ=0.5 is the spectral loss weight.

[0467] In some embodiments, the second stage freezes the decoder and jointly optimizes the encoder and quantizer, with the loss function being:

[0468]

[0469] in, is the smoothed L1 loss for mask prediction, is the vector quantization loss.

[0470] In some embodiments, the present disclosure enhances the semantic richness of the potential representation through self-supervised mask prediction tasks, providing more effective feature support for downstream tasks (such as speech recognition and sentiment analysis). Multi-layer residual vector quantization technology is used to retain the subtle structure of the audio signal at a low bit rate, thereby improving compression efficiency and reconstruction quality. Combining the inverse short-time Fourier transform with a multi-scale discriminator, high-fidelity audio waveform reconstruction is achieved, especially under high compression rate conditions. The phased training strategy effectively avoids the training conflict between reconstruction quality and semantic learning objectives, ensuring the optimization of the overall system performance.

[0471] Example 1: Overall workflow of a semantically enhanced audio codec system:

[0472] The embodiment of the present disclosure describes in detail the implementation process of the semantically enhanced audio coding and decoding system, which includes an encoder module, a multi-layer residual vector quantization module and a decoder module. Figure 5C As shown in the figure, the system takes mono audio with a sampling rate of 16kHz as input, generates semantically enhanced latent representations through the encoder module, discretizes them into low-bitrate (6kbps) token sequences through the multi-layer residual vector quantization module, and finally reconstructs them into high-fidelity audio waveforms through the decoder module.

[0473] 1. Encoder module:

[0474] like Figure 5A As shown in Figure 2, the encoder module adopts a self-supervised learning architecture, which includes the following submodules:

[0475] (1) Preprocessing unit:

[0476] The pre-processing unit receives a mono audio waveform signal with a sampling rate of 16kHz and uses the front-end processing method of related technologies to extract features. The specific steps are as follows:

[0477] The input audio waveform is directly fed into a 7-layer 1D convolutional neural network (CNN) feature extractor.

[0478] The CNN structure is as follows: the convolution kernel size of each layer is 3, the stride is 2, the number of channels is [256, 256, 512, 512, 512, 512, 512], and each layer is followed by group normalization (GroupNorm), the number of groups is 32, and normalization is performed along the channel dimension.

[0479] The activation function is ReLU, and the calculation formula is:

[0480] ReLU(x)=max(0,x)

[0481] The output is 512-dimensional time series embeddings (frame-level embeddings) with a time resolution of 1 / 10 of the original waveform (about 10ms frame length) for subsequent Transformer processing.

[0482] (2) Transformer encoder:

[0483] The frame-level features generated by preprocessing are input into the Transformer encoder, and the structure is as follows:

[0484] The input embedding layer projects the 512-dimensional features into a 512-dimensional latent space through a linear mapping, and the weight matrix dimension is [512, 512].

[0485] Position encoding uses the sine-cosine method, the formula is:

[0486]

[0487] Among them, pos is the time step index, i is the dimension index, d model =512.

[0488] The Transformer layer contains 8 self-attention layers, each with 8 attention heads. The attention calculation formula is:

[0489]

[0490] Among them, d k =64.

[0491] The hidden layer dimension of the feedforward network is 2048, and the activation function is GELU:

[0492] GELU(x)=x·Φ(x)

[0493] Each layer is followed by a residual connection y=x+f(x) and layer normalization LayerNorm(y).

[0494] In some embodiments, the disclosed embodiments employ a self-supervised learning mechanism, and the encoder employs a teacher-student architecture and a mask prediction task, specifically implemented as follows:

[0495] Teacher model: Contains only the encoder and uses the same structure as the Transformer encoder, i.e., an 8-layer Transformer with 8 attention heads per layer and a hidden layer dimension of 512. The teacher model receives the complete input sequence and generates the target feature representation Z M .

[0496] Student model: consists of two parts: encoder and decoder.

[0497] Encoder: Same structure as the teacher model, i.e., 8-layer Transformer, 8 attention heads per layer, hidden layer dimension of 512, receiving masked input sequence.

[0498] Decoder: uses a 6-layer Transformer structure with 8 attention heads per layer and a hidden layer dimension of 512 to predict the feature representation of the mask area

[0499] Masking strategy: Based on Bernoulli distribution, probability p = 0.1, continuous masking L = 5 time steps, the set of masked regions is:

[0500]

[0501] Mask prediction task: The student model predicts mask region features The teacher model generates the target feature Z M , the loss function is smooth L1 loss:

[0502]

[0503] Where β = 1.0, Predict features for the student model, y = Z M is the target feature of the teacher model.

[0504] EMA update: The teacher model parameters are dynamically updated through the exponential moving average (EMA), and the update formula is:

[0505]

[0506] Among them, α∈[0.9,0.999] is the smoothing coefficient, and the update frequency is once per batch training iteration;

[0507] 2. Multi-layer residual vector quantization module:

[0508] like Figure 5B As shown in Figure 2, the multi-layer residual vector quantization module is responsible for discretizing the continuous potential representation Z output by the encoder into a token sequence.

[0509] Codebook structure and initialization:

[0510] The codebook is configured as 4 layers, each layer contains K = 1024 entries, and the entry dimension is 512. The codebook is initialized using the k-means++ algorithm: the first center point is randomly selected, and the probability of selecting a new center point is P(x) ∝ D(x) 2 , repeat until K center points are selected and then run k-means.

[0511] The multi-layer residual quantization process includes:

[0512] Initial Input Quantization calculation of each layer:

[0513]

[0514] The final quantized representation is:

[0515]

[0516] Among them, L=4.

[0517] Quantitative operations:

[0518] Find the most recent codebook entry:

[0519] q(z)=argmin j ||zc j ||2

[0520] Q(z)=c q(z)

[0521] Use the straight-through estimator when training:

[0522]

[0523] Codebook update:

[0524] The update formula is:

[0525]

[0526] Every 1000 steps, entries with a usage frequency below 0.01% are checked and replaced with high-frequency entries plus perturbations.

[0527] 3. Decoder module:

[0528] The decoder module reconstructs the audio waveform, Figure 5D FIG. 1 is a schematic diagram of a decoder structure according to an embodiment of the present disclosure. Figure 5D As shown, the decoder module includes Transformer and ConvNeXt.

[0529] Decoder network structure:

[0530] The input layer transforms z into q Mapped to [T,512] features.

[0531] The Transformer encoder consists of 6 layers, with 8 attention heads in each layer. The attention formula is:

[0532]

[0533] The ConvNeXt module contains 4 blocks with the number of channels [256, 128, 64, 32]. Each block includes 7x7 depthwise convolution, layer normalization and 1x1 convolution.

[0534] The ConvNeXt module processes the deep feature representation, which is then projected through a linear layer to generate parameters related to the amplitude and phase of the target STFT spectrum.

[0535] Reconstructed waveform:

[0536]

[0537] The multi-scale discriminator includes:

[0538] Multi-resolution discriminator (MRD): consists of 5 convolutional layers, each with a kernel size of [5,5], a stride of [2,2], and a number of channels of [32,64,128,256,512];

[0539] Multi-cycle Discriminator (MPD): The input audio is segmented into [2, 3, 5, 7, 11] different cycle lengths and one-dimensional convolution is performed on each segment.

[0540] STFT spectral discriminator: Calculates the L1 loss of Mel spectrum and linear spectrum, with 80 frequency bands.

[0541] Example 2: Training process of semantically enhanced audio codec system:

[0542] The embodiment of the present disclosure describes in detail the training process of the semantically enhanced audio codec system, which uses a phased training strategy to optimize the performance of the encoder module, multi-layer residual vector quantization module and decoder module. Figure 5C As shown in Figure 2, the training process is based on a mono audio dataset with a sampling rate of 16kHz, and the goal is to achieve high-quality audio reconstruction and semantic enhancement at a low bit rate (6kbps).

[0543] 1. Training preparation:

[0544] Dataset: The LibriSpeech dataset is used, which contains approximately 960 hours of English pronunciation audio with a sampling rate of 16 kHz. The audio is randomly cut into 10-second segments as training samples.

[0545] Software framework: The initial learning rate is 1×10 -4 , the weight decay is 1×10 -2 .

[0546] 2. Phase 1 (Acoustic Reconstruction Training):

[0547] The first phase focuses on training the decoder module and the front-end of the encoder module, with the goal of optimizing the reconstruction quality of the audio waveform. The specific steps are as follows:

[0548] Training objective: minimize reconstruction loss and spectral loss, the loss function is:

[0549]

[0550] Among them, X is the original audio waveform, To reconstruct the waveform, λ = 0.5 is the spectral loss weight, STFT represents short-time Fourier transform, window length N = 1024, and frame shift M = 256.

[0551] Training process:

[0552] The input audio waveform passes through the CNN feature extractor of the encoder module (7-layer 1D CNN, convolution kernel size 3, stride 2, channel number [256, 256, 512, 512, 512, 512], followed by group normalization in each layer) and the Transformer encoder (8 layers, 8 attention heads) to generate the latent representation Z.

[0553] The latent representation Z generates a token sequence through a multi-layer residual vector quantization module.

[0554] The decoder module (containing 6 layers of Transformer decoder and 4 ConvNeXt modules) reconstructs the token sequence into a waveform The final waveform is generated by the inverse short-time Fourier transform (ISTFT):

[0555]

[0556] Calculating losses The encoder and decoder parameters are updated via back-propagation, while the quantizer parameters are kept frozen.

[0557] Hyperparameters: The learning rate is scheduled using cosine annealing, the annealing period is 50 epochs, and the training is performed for a total of 50 epochs.

[0558] 3. Second stage (joint optimization training)

[0559] The second stage freezes the decoder parameters and jointly optimizes the encoder module and the multi-layer residual vector quantization module, with the goal of enhancing semantic representation and quantization accuracy. The specific steps are as follows:

[0560] Training objective: minimize mask prediction loss and vector quantization loss. The total loss function is:

[0561]

[0562] in Smoothed L1 loss for mask prediction:

[0563]

[0564] Where β = 1.0. is the vector quantization loss, and the calculation formula is:

[0565]

[0566] Where Z is the encoder output, is the quantized output of the lth layer, is the most recent codebook entry, and L=4.

[0567] Training process:

[0568] Freeze the decoder parameters and use the decoder only to generate the reconstructed waveform.

[0569] The input audio waveform passes through the encoder module and applies the masking strategy to generate a masked input sequence with probability p = 0.1. The masking is continuous for L = 5 time steps. The set of masked regions is:

[0570]

[0571] Student model predicts mask region features The teacher model generates the target feature Z M , the loss function is smooth L1 loss:

[0572]

[0573] Where β = 1.0.

[0574] The multi-layer residual vector quantization module quantizes the potential representation Z. The quantization process is as follows:

[0575]

[0576] The final quantized representation is:

[0577]

[0578] Where L = 4. The quantization operation uses a straight-through estimator:

[0579]

[0580] The codebook is updated using online clustering, and the update formula is:

[0581]

[0582] The codebook is updated every 10 batches, and the usage frequency of codebook entries is checked every 1000 steps. If it is less than 0.01%, it is replaced with high-frequency entries plus Gaussian noise (mean 0, variance 0.01).

[0583] Jointly optimize encoder and quantizer parameters.

[0584] Hyperparameters: Learning rate is adjusted to 5×10 -5 , train for 30 epochs, cosine annealing scheduling.

[0585] In some embodiments of the present disclosure, a coding and decoding system is provided, which may include an encoder and a decoder, wherein the encoder can execute the coding and decoding method executed by the encoder in the aforementioned embodiments of the present disclosure; the decoder can execute the coding and decoding method executed by the decoder in the aforementioned embodiments of the present disclosure.

[0586] The present disclosure also provides an apparatus for implementing any of the above methods. For example, a device is provided that includes units or modules for implementing each step performed by an encoder in any of the above methods. For another example, another device is provided that includes units or modules for implementing each step performed by a decoder in any of the above methods.

[0587] It should be understood that the division of the various units or modules in the above device is merely a division of logical functions. In actual implementation, they may be fully or partially integrated into a physical entity, or they may be physically separated. In addition, the units or modules in the device may be implemented in the form of a processor calling software: for example, the device includes a processor, the processor is connected to a memory, and the memory stores instructions. The processor calls the instructions stored in the memory to implement any of the above methods or implement the functions of the various units or modules of the above device, wherein the processor is, for example, a general-purpose processor, such as a central processing unit (CPU) or a microprocessor, and the memory is a memory within the device or a memory outside the device. Alternatively, the units or modules in the device can be implemented in the form of hardware circuits, and the functions of some or all of the units or modules can be realized by designing the hardware circuits. The above-mentioned hardware circuits can be understood as one or more processors; for example, in one implementation, the above-mentioned hardware circuit is an application-specific integrated circuit (ASIC), and the functions of some or all of the above units or modules are realized by designing the logical relationship of the components in the circuit; for example, in another implementation, the above-mentioned hardware circuit can be realized by a programmable logic device (PLD). Taking a field programmable gate array (FPGA) as an example, it can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by configuring the configuration file, thereby realizing the functions of some or all of the above units or modules. All units or modules of the above devices can be realized in the form of software called by the processor, or in the form of hardware circuits, or in part by the form of software called by the processor, and the rest by hardware circuits.

[0588] In the embodiments of the present disclosure, the processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with instruction reading and execution capabilities, such as a central processing unit (CPU), a microprocessor, a graphics processing unit (GPU) (which can be understood as a microprocessor), or a digital signal processor (DSP). In another implementation, the processor can implement certain functions through the logical relationship of a hardware circuit. The logical relationship of the above-mentioned hardware circuit is fixed or reconfigurable. For example, the processor is a hardware circuit implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units or modules. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as ASIC, such as a neural network processing unit (NPU), a tensor processing unit (TPU), a deep learning processing unit (DPU), etc.

[0589] Figure 6A This is a schematic diagram of the structure of an encoding device proposed in an embodiment of the present disclosure. Figure 6AAs shown, the encoding device 6100 may include at least one of a processing module 6101, a transceiver module 6102, etc. In some embodiments, the processing module 6101 is configured to obtain audio features of the first audio data; encode the audio features to obtain first encoded data; wherein the encoder belongs to a first network model, and the first network model is obtained by self-supervised training based on the second network model. Optionally, the transceiver module 6101 can be used to perform at least one of the communication steps such as sending and / or receiving performed by the encoder in any of the above methods (for example, step S2105, step S3104, but not limited to this), which will not be repeated here. Optionally, the processing module 6102 can be used to execute at least one of the other steps performed by the encoder in any of the above methods (for example, step S2101, step S2102, step S2103, step S2104, step S3101, step S3102, step S3103, step S3201, step S3202, but not limited to these), which are not repeated here.

[0590] Figure 6B This is a schematic diagram of the structure of a decoding device proposed in an embodiment of the present disclosure. Figure 6B As shown, the decoding device 6200 may include: at least one of a processing module 6201, a transceiver module 6202, etc. In some embodiments, the processing module 6201 is configured to decode the first encoded data to obtain the second audio data; wherein the decoder belongs to a first network model, and the first network model is obtained by self-supervised training based on the second network model. Optionally, the transceiver module 6201 can be used to perform at least one of the communication steps such as sending and / or receiving performed by the decoder in any of the above methods (for example, step S2105, step S4101, but not limited to this), which will not be repeated here. Optionally, the processing module 6202 can be used to perform at least one of the other steps performed by the decoder in any of the above methods (for example, step S2106, step S2107, step S2108, step S2109, step S2110, step S2111, step S4102, step S4103, step S4201, but not limited to this), which will not be repeated here.

[0591] In some embodiments, the transceiver module may include a transmitting module and / or a receiving module, and the transmitting module and the receiving module may be separate or integrated. Optionally, the transceiver module may be interchangeable with the transceiver.

[0592] In some embodiments, the processing module can be a single module or can include multiple submodules. Optionally, the multiple submodules respectively execute all or part of the steps required to be executed by the processing module. Optionally, the processing module can be interchangeable with the processor.

[0593] Figure 7A 7 is a schematic diagram of the structure of a communication device 7100 proposed in an embodiment of the present disclosure. Communication device 7100 can be an encoder or a decoder, or a chip, chip system, or processor that supports an encoder in implementing any of the above methods. It can also be a chip, chip system, or processor that supports a decoder in implementing any of the above methods. Communication device 7100 can be used to implement the methods described in the above method embodiments. For details, please refer to the description of the above method embodiments.

[0594] like Figure 7A As shown, communication device 7100 includes one or more processors 7101. Processor 7101 can be a general-purpose processor or a dedicated processor, for example, a baseband processor or a central processing unit. The baseband processor can be used to process communication protocols and communication data, while the central processing unit can be used to control the codec device, execute programs, and process program data. Communication device 7100 is configured to perform any of the above methods.

[0595] In some embodiments, the communication device 7100 further includes one or more memories 7102 for storing instructions. Optionally, all or part of the memories 7102 may be located outside the communication device 7100.

[0596] In some embodiments, the communication device 7100 further includes one or more transceivers 7103. When the communication device 7100 includes one or more transceivers 7103, the transceiver 7103 performs at least one of the communication steps such as sending and / or receiving in the above method (for example, step S2105, step S3104, step S4101, but not limited thereto), and the processor 7101 performs at least one of the other steps (for example, step S2101, step S2102, step S2103, step S2104, step S3101, step S3102, step S3103, step S3201, step S3202, step S2106, step S2107, step S2108, step S2109, step S2110, step S2111, step S4102, step S4103, step S4201, but not limited thereto).

[0597] In some embodiments, a transceiver may include a receiver and / or a transmitter. The receiver and transmitter may be separate or integrated. Optionally, the terms transceiver, transceiver unit, transceiver, and transceiver circuit may be used interchangeably; the terms transmitter, transmitting unit, transmitter, and transmitting circuit may be used interchangeably; and the terms receiver, receiving unit, receiver, and receiving circuit may be used interchangeably.

[0598] In some embodiments, the communication device 7100 may include one or more interface circuits. Optionally, the interface circuits are connected to the memory 7102 and may be used to receive signals from the memory 7102 or other devices, or to send signals to the memory 7102 or other devices. For example, the interface circuits may read instructions stored in the memory 7102 and send the instructions to the processor 7101.

[0599] The communication device 7100 described in the above embodiments may be an encoder or a decoder, but the scope of the communication device 7100 described in the present disclosure is not limited thereto, and the structure of the communication device 7100 may not be limited thereto. Figure 7A The communication device may be an independent device or a part of a larger device. For example, the communication device may be: 1) an independent integrated circuit IC, or a chip, or a chip system or subsystem; (2) a collection of one or more ICs, optionally, the above IC collection may also include a storage component for storing data and programs; (3) an ASIC, such as a modem; (4) a module that can be embedded in other devices; (5) a receiver, an encoder, a decoder, an encoder device, a decoder device, a cellular phone, a wireless device, a handheld device, a mobile unit, a vehicle-mounted device, a cloud device, an artificial intelligence device, etc.; (6) others, etc.

[0600] Figure 7B 7200 is a schematic diagram of the structure of the chip 7200 proposed in the embodiment of the present disclosure. For the case where the communication device 7100 can be a chip or a chip system, please refer to Figure 7B The structure of the chip 7200 is shown, but is not limited to this.

[0601] The chip 7200 includes one or more processors 7201 , and the chip 7200 is configured to execute any of the above methods.

[0602] In some embodiments, the chip 7200 further includes one or more interface circuits 7203. Optionally, the interface circuit 7203 is connected to the memory 7202. The interface circuit 7203 can be used to receive signals from the memory 7202 or other devices, and can be used to send signals to the memory 7202 or other devices. For example, the interface circuit 7203 can read instructions stored in the memory 7202 and send the instructions to the processor 7201.

[0603] In some embodiments, the interface circuit 7203 performs at least one of the communication steps such as sending and / or receiving in the above method (for example, step S2105, step S3104, step S4101, but not limited to this), and the processor 7201 performs at least one of the other steps (for example, step S2101, step S2102, step S2103, step S2104, step S3101, step S3102, step S3103, step S3201, step S3202, step S2106, step S2107, step S2108, step S2109, step S2110, step S2111, step S4102, step S4103, step S4201, but not limited to this).

[0604] In some embodiments, terms such as interface circuit, interface, transceiver pin, and transceiver may be used interchangeably.

[0605] In some embodiments, the chip 7200 further includes one or more memories 7202 for storing instructions. Alternatively, all or part of the memory 7202 may be external to the chip 7200.

[0606] The embodiments of the present disclosure further provide a storage medium having instructions stored thereon. When the instructions are executed on the communication device 7100, the communication device 7100 executes any of the above methods. Optionally, the storage medium is an electronic storage medium. Optionally, the storage medium is a computer-readable storage medium, but is not limited thereto, and may also be a storage medium readable by other devices. Optionally, the storage medium may be a non-transitory storage medium, but is not limited thereto, and may also be a temporary storage medium.

[0607] The present disclosure also provides a program product, which, when executed by the communication device 7100, enables the communication device 7100 to perform any of the above methods. Optionally, the program product may be a computer program product.

[0608] The embodiments of the present disclosure also provide a computer program, which, when executed on a computer, enables the computer to execute any one of the above methods.

Claims

1. A coding method, characterized in that: Executed by an encoder, the method includes: Acquire audio features of the first audio data; Encoding the audio feature to obtain first encoded data; The encoder belongs to a first network model, and the first network model is obtained by self-supervised training based on a second network model.

2. The method according to claim 1, characterized in that The method further comprises: The first encoded data is sent to a decoder.

3. The method according to claim 1 or 2, characterized in that The encoding of the audio feature to obtain first encoded data includes: Performing masking processing on the audio feature to obtain a masked feature; The first encoded data is generated according to the mask feature.

4. The method according to claim 3, characterized in that Generating the first coded data according to the mask feature includes: generating a latent representation of the mask features; The latent representation is quantized to obtain the first encoded data.

5. The method according to any one of claims 1 to 4, characterized in that The first network model is a student model, and the second network model is a teacher model.

6. A decoding method, characterized in that: Executed by a decoder, the method includes: Decoding the first encoded data to obtain second audio data; The decoder belongs to a first network model, and the first network model is obtained by self-supervised training based on a second network model.

7. The method according to claim 6, characterized in that The method further comprises: The first encoded data sent by the encoder is received.

8. The method according to claim 6 or 7, characterized in that The decoding of the first encoded data to obtain the second audio data includes: generating a short-time Fourier transform (STFT) coefficient sequence according to the first coded data; Perform an inverse short-time Fourier transform on the STFT coefficient sequence to obtain the second audio data.

9. The method according to claim 8, characterized in that The inverse short-time Fourier transform is performed by the following formula: in, is the second audio data, m is the frame index, H is the frame shift, n is the time index, S[m] is the STFT coefficient vector of the mth frame in the STFT coefficient sequence, ISTFT() represents the inverse short-time Fourier transform, and w[n-mH] is the synthesis window function.

10. The method according to claim 8 or 9, characterized in that Generating an STFT coefficient sequence according to the first coded data includes: Performing embedding processing on the first encoded data to obtain an embedded feature representation; extracting a deep feature representation based on the embedded feature representation; Performing linear projection on the deep feature representation to generate first parameters related to amplitude and phase; The STFT coefficient sequence is determined according to the first parameter.

11. The method according to claim 10, characterized in that The extracting a deep feature representation according to the embedded feature representation comprises: Performing contextual relationship modeling on the embedded feature representation to obtain intermediate features; Time-frequency information is extracted according to the intermediate features to obtain the deep feature representation.

12. The method according to any one of claims 6 to 11, characterized in that: The first network model is a student model, and the second network model is a teacher model.

13. A model training method, characterized in that: The method comprises: Acquire a training sample set, wherein the training sample set includes a plurality of sample audio data; Encoding the sample audio data by an encoder to obtain first sample encoded data; Decoding the first sample coded data by a decoder to obtain sample decoded data; Determining a first loss value according to the sample audio data, the sample decoded data, and a first loss function; Parameters of the encoder and the decoder are updated according to the first loss value.

14. The method according to claim 13, characterized in that The first loss function is: in, is the first loss value, X is the sample audio data, is the sample decoded data, λ is the spectral loss weight, and STFT() represents short-time Fourier transform.

15. The method according to claim 13 or 14, characterized in that The method further comprises: Predicting mask sample features of the sample audio data through a first network model and quantizing the mask sample features to obtain second sample encoded data, wherein the first network model includes an encoder and a decoder; Predicting global sample features of the sample audio data using a second network model, where the second network model includes an encoder and does not include a decoder, and the encoder of the second network model has the same structure as the encoder of the first network model; Determining a second loss value according to the mask sample feature, the global sample feature, and the second sample encoding data; Update parameters of the encoder of the first network model according to the second loss value.

16. The method according to claim 15, characterized in that The determining of the second loss value according to the mask sample feature, the global sample feature, and the second sample encoding data includes: Determining a third loss value according to the mask sample feature, the global sample feature, and the second loss function; determining a fourth loss value according to the second sample encoded data and a third loss function; The second loss value is determined according to the third loss value and the fourth loss value.

17. The method according to claim 16, characterized in that The second loss function is: Among them, L smooth (x, y) is the third loss value, x is the mask sample feature, y is the global sample feature, and β is the smoothing coefficient.

18. The method according to claim 16 or 17, characterized in that The third loss function is: Among them, L vq is the fourth loss value, Z is the second sample encoding data, is the quantized output of layer I, c q(z) is the codebook vector with index q(z) at the lth level, and L is the number of quantization levels.

19. The method according to any one of claims 15 to 18, characterized in that: The method further comprises: The parameters of the encoder of the second network model are updated according to the parameters of the encoder of the first network model.

20. The method according to claim 19, characterized in that The parameters of the encoder of the second network model are updated by the following formula: in, is the parameter of the encoder of the second network model during training at step t, is the parameter of the encoder of the second network model during training at step t-1, is the parameter of the encoder of the first network model during training at the tth step, and α is the smoothing coefficient.

21. The method according to any one of claims 13 to 20, characterized in that: The method further comprises: Inputting the sample decoded data and the sample audio data into a discriminator to obtain a fifth loss value; The parameters of the decoder are updated according to the fifth loss value.

22. A coding and decoding method, characterized in that: The method comprises: The encoder obtains audio features of the first audio data; The encoder encodes the audio feature to obtain first encoded data; The decoder decodes the first encoded data to obtain second audio data; The encoder and the decoder belong to a first network model, and the first network model is obtained by self-supervised training based on the second network model.

23. An encoding device, characterized in that The encoding device comprises: a processing module, configured to obtain audio features of the first audio data; The processing module is further configured to encode the audio feature to obtain first encoded data; The encoder belongs to a first network model, and the first network model is obtained by self-supervised training based on a second network model.

24. A decoding device, characterized in that: The decoding device comprises: a processing module, configured to decode the first encoded data to obtain second audio data; The decoder belongs to a first network model, and the first network model is obtained by self-supervised training based on a second network model.

25. An encoding device, characterized in that The encoding device comprises: one or more processors; The processor is configured to execute the method according to any one of claims 1 to 5.

26. A decoding device, characterized in that: The decoding device comprises: one or more processors; The processor is configured to execute the method according to any one of claims 6 to 12.

27. A coding and decoding system, characterized in that: The system comprises an encoder and a decoder, wherein the encoder is configured to implement the method according to any one of claims 1 to 5, and the decoder is configured to implement the method according to any one of claims 6 to 12.

28. A storage medium storing instructions, characterized in that: When the instruction is executed on a communication device, the communication device is caused to execute the method according to any one of claims 1-5, 6-12, and 13-21.

29. A program product, comprising at least one of a program and instructions, characterized in that: When at least one of the program and the instruction is executed by the communication device, the steps of the method described in any one of claims 1-5, 6-12, and 13-21 are implemented.

Images