Audio coding method and device, equipment, storage medium and product

Abstract

The embodiment of the invention provides an audio coding method and device, equipment, a storage medium and a product. According to the technical scheme provided by the embodiment of the invention, the detection voice feature of the preset time duration is extracted from the to-be-coded voice, the voice feature matching is performed in the voice feature library according to the detection voice feature, the target voice feature is determined according to the matching result, and the target voice feature is added to the trained coding network, so that the coding efficiency is improved. And performing voice coding processing according to the to-be-coded voice and the target voice feature through the coding network to obtain a target audio feature, performing residual vector quantization processing on the target audio feature to obtain a target coding result, and performing voice coding processing on the target voice feature and the to-be-coded voice by taking the detection voice feature or the historical voice feature as the target voice feature. According to the invention, audio coding is carried out in combination with the voice features related to the to-be-coded voice, so that the audio compression rate is improved, the voice fidelity is improved, and the audio coding quality is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of audio processing technology, and in particular, to an audio encoding method, apparatus, device, storage medium and product. Background Art

[0002] Audio coding is generally used to encode and decode digital audio signals. With the development of digital audio technology, audio coding has become increasingly important in various applications, such as in mobile devices, computers, multimedia players, and communication devices. Audio coding refers to converting the original audio signal into a compressed format to reduce the amount of data and facilitate the storage and transmission of audio data. It usually involves removing redundant information and imperceptible audio components to achieve higher compression efficiency.

[0003] Currently, audio encoding is generally performed through traditional encoding or artificial intelligence encoding (AI Codec). Traditional audio encoders are suitable for real-time communication and are widely used in VoIP, video conferencing, and online games. However, traditional encoders have the disadvantages of low compression rate and cannot work in low bit rate scenarios. Artificial intelligence encoding uses deep learning and machine learning algorithms to achieve more efficient audio compression and decoding while ensuring sound quality. However, the artificial intelligence encoding method has the disadvantage of low voice fidelity and poor audio encoding quality. Summary of the invention

[0004] The embodiments of the present application provide an audio coding method, apparatus, device, storage medium and product to solve the technical problems of low audio coding speech fidelity and poor audio coding quality in related technologies. It can improve the compression rate of audio while improving speech fidelity and audio coding quality.

[0005] In a first aspect, an embodiment of the present application provides an audio encoding method, comprising:

[0006] Extracting detection speech features of a preset time length from the speech to be encoded;

[0007] Performing voice feature matching in a voice feature library according to the detected voice feature, and determining a target voice feature according to the matching result, wherein the voice feature library records one or more historical voice features, and the target voice feature is the detected voice feature or the historical voice feature;

[0008] The target speech feature is added to the trained coding network, and the coding network performs speech coding processing according to the speech to be coded and the target speech feature to obtain the target audio feature;

[0009] The target audio feature is subjected to residual vector quantization processing to obtain a target encoding result.

[0010] In a second aspect, an embodiment of the present application provides an audio encoding device, including a speech detection module, a feature determination module, a feature encoding module and a vector processing module, wherein:

[0011] The speech detection module is configured to extract detection speech features of a preset time length from the speech to be encoded;

[0012] The feature determination module is configured to perform voice feature matching in a voice feature library according to the detected voice feature, and determine a target voice feature according to the matching result, wherein the voice feature library records one or more historical voice features, and the target voice feature is the detected voice feature or the historical voice feature;

[0013] The feature encoding module is configured to add the target speech feature to the trained encoding network, and perform speech encoding processing according to the to-be-encoded speech and the target speech feature through the encoding network to obtain the target audio feature;

[0014] The vector processing module is configured to perform residual vector quantization processing on the target audio feature to obtain a target encoding result.

[0015] In a third aspect, an embodiment of the present application provides an audio encoding device, including: a memory and one or more processors;

[0016] The memory is used to store one or more programs;

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the audio encoding method as described in the first aspect.

[0018] In a fourth aspect, an embodiment of the present application provides a non-volatile storage medium storing computer executable instructions, wherein the computer executable instructions are used to execute the audio encoding method as described in the first aspect when executed by a computer processor.

[0019] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor of a device reads and executes the computer program from the computer-readable storage medium, so that the device performs the audio encoding method as described in the first aspect.

[0020] The embodiment of the present application extracts detected speech features of a preset time length from the speech to be encoded, performs speech feature matching in a speech feature library based on the detected speech features, determines target speech features based on the matching results, adds the target speech features to a trained coding network, performs speech coding processing based on the speech to be encoded and the target speech features through the coding network to obtain target audio features, performs residual vector quantization processing on the target audio features to obtain target coding results, performs speech coding processing with the detected speech features or historical speech features as target speech features and the speech to be encoded, and performs audio coding in combination with speech features related to the speech to be encoded, thereby improving the compression rate of the audio while improving the speech fidelity and the audio coding quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flowchart of an audio encoding method provided by an embodiment of the present application;

[0022] Figure 2 is a flowchart of another audio encoding method provided by an embodiment of the present application;

[0023] Figure 3 It is a structural diagram of a speech feature extraction network;

[0024] Figure 4 It is a schematic diagram of a target audio feature generation process;

[0025] Figure 5 It is a schematic diagram of the structure of a coding network;

[0026] Figure 6 It is a schematic diagram of the structure of a decoding network;

[0027] Figure 7 is a structural diagram of an audio encoding device provided in an embodiment of the present application;

[0028] Figure 8 It is a structural diagram of an audio encoding device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical scheme and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for the convenience of description, only the part related to the present application but not all the contents are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow chart describes each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. The above process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The above process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0030] The audio coding method provided in this application can be applied to audio coding scenarios, and aims to use detected speech features or historical speech features as target speech features to perform speech coding processing with the speech to be encoded, and to perform audio coding in combination with speech features related to the speech to be encoded, so as to improve the audio coding quality.

[0031] In the existing audio coding scheme, it is generally carried out through traditional coding or artificial intelligence coding. Traditional coding is generally suitable for audio encoders for real-time communication and is widely used in VoIP, video conferencing and online games, but it has the disadvantages of low compression rate and inability to work in low bit rate scenarios. Artificial intelligence coding uses deep learning and machine learning algorithms to achieve more efficient audio compression and decoding while ensuring sound quality, but artificial intelligence coding has problems such as insufficient voice fidelity and compression rate, and excessive model calculation and parameter volume. When processing audio signals, artificial intelligence coding audio codecs usually use general coding algorithms. Although these algorithms can provide good sound quality and compression efficiency in most cases, they often cannot achieve the best results when facing low bit rate and specific speaker's voice. At present, artificial intelligence coding cannot optimize the voice characteristics of a specific speaker, resulting in distorted sound quality and insufficient compression rate. It is impossible to improve the encoding and decoding effect by learning and accumulating the voice characteristics of a specific speaker. In different call scenarios, it is impossible to dynamically adjust according to the voice characteristics detected in real time, resulting in unstable encoding and decoding effects. Based on this, an audio coding method of an embodiment of the present application is provided to solve the technical problems of low speech fidelity and poor audio coding quality in existing audio coding.

[0032] Figure 1A flowchart of an audio encoding method provided in an embodiment of the present application is given. The audio encoding method provided in an embodiment of the present application can be executed by an audio encoding device, which can be implemented by hardware and / or software and integrated in an audio encoding device.

[0033] The following description is made by taking an audio encoding device executing an audio encoding method as an example. Figure 1 , the audio encoding method comprises:

[0034] S110: extracting detected speech features of a preset time length from the speech to be encoded.

[0035] The speech to be encoded is obtained, and feature extraction is performed on the speech to be encoded, so as to extract detection speech features of a preset time length (the preset time length may be 1-10s, such as 5s) from the speech to be encoded.

[0036] For example, during the audio encoding process, the detected speech features of speech segments of a preset time length are continuously extracted from the speech to be encoded, and the target speech features are determined based on each detected speech feature, and encoding processing and residual vector quantization processing are performed based on the speech to be encoded and the target speech features.

[0037] Optionally, the detection speech features of a preset time length can be extracted from the speech to be encoded by a preset feature extraction algorithm, or the detection speech features of a preset time length can be extracted from the speech to be encoded by a trained speech feature extraction network.

[0038] S120: performing voice feature matching in a voice feature library according to the detected voice feature, and determining a target voice feature according to the matching result, wherein the voice feature library records one or more historical voice features, and the target voice feature is a detected voice feature or a historical voice feature.

[0039] The speech feature library provided by this solution records historical speech features corresponding to historical speech of one or more preset time lengths, where different historical speech features can correspond to different objects (different speakers), that is, the speech feature library can record historical speech features corresponding to historical speech of different objects with preset time lengths.

[0040] Optionally, the historical speech features in the speech feature library can be obtained by pre-collection, or by collecting them from the speech to be encoded in a previous audio encoding process. For example, the detected speech features collected from the speech to be encoded in the audio encoding process are added to the speech feature library as historical speech features, or the historical speech features of the same object in the speech feature library are replaced (for example, speech features whose vector distance is less than a preset distance threshold or whose similarity is greater than a similarity threshold are used as speech features of the same object).

[0041] Exemplarily, voice feature matching is performed in a voice feature library according to the detected voice features extracted as described above, and a historical voice feature matching the detected voice features is determined from the voice feature library. Optionally, a historical voice feature whose vector distance (e.g., cosine distance, Hamming distance, Euclidean distance, etc.) to the detected voice features is less than a preset distance threshold, or whose similarity (e.g., cosine similarity, Pearson similarity, Jaccard similarity, etc.) to the detected voice features is greater than a similarity threshold can be used as a historical voice feature matching the detected voice features.

[0042] Optionally, when a historical speech feature that matches the detected speech feature is matched, the matched historical speech feature can be used as the target speech feature, or the detected speech feature and the matched historical speech feature can be fused to obtain the target speech feature (for example, dot multiplication, addition, concatenation, etc. of the detected speech feature and the matched historical speech feature). When a historical speech feature that matches the detected speech feature is not matched, the detected speech feature can be used as the target speech feature.

[0043] S130: adding the target speech features to the trained coding network, and performing speech coding processing according to the to-be-coded speech and the target speech features through the coding network to obtain the target audio features.

[0044] The encoding network provided by this scheme can be trained based on the speech samples to be encoded, the target speech feature samples and the corresponding human voice samples, wherein the target speech feature samples can be obtained by extracting speech features of a preset time length from the human voice samples (for example, the speech features corresponding to 5s speech at a random position in the human voice sample).

[0045] Optionally, the encoding network, the corresponding decoding network, the residual vector quantization network for residual vector quantization processing, and the speech feature extraction network for extracting speech features provided by the present solution can be trained together based on sample data to improve the data collaborative processing capabilities between the networks and improve the audio encoding and decoding quality.

[0046] Exemplarily, the target speech features determined above are added to the trained coding network, so that speech coding processing is performed according to the speech to be coded and the target speech features through the coding network, and the speech coding processing result is used as the target audio feature.

[0047] S140: Perform residual vector quantization processing on the target audio feature to obtain a target encoding result.

[0048] Exemplarily, after obtaining the target audio features, the target audio features can be input into a trained residual vector quantization network, and the target audio features can be subjected to residual vector quantization (RVQ) processing by the residual vector quantization network to obtain the target encoding result. The residual vector quantization process uses a neural network and a cascade codebook to achieve effective quantization of high-dimensional vectors, which can reduce computational costs while maintaining audio quality.

[0049] In one embodiment, in obtaining the target coding result, the target coding result can be sent to the target terminal (e.g., a call terminal during a real-time call). After receiving the target coding result, the target terminal can submit the target coding result to the decoder, and the decoder decodes the target coding result through the trained decoding network to obtain a decoding result, and plays the corresponding sound based on the decoding result. It can be understood that since the target coding result is obtained by audio encoding in combination with the speech features related to the speech to be encoded, while ensuring the audio compression rate of the target coding result, the speech fidelity is effectively improved. Correspondingly, the decoding result obtained by decoding the target coding result can more accurately restore the original sound, effectively improving the decoding quality.

[0050] In the above, by extracting detected speech features of a preset time length from the speech to be encoded, performing speech feature matching in a speech feature library according to the detected speech features, and determining target speech features according to the matching results, the target speech features are added to the trained coding network, and speech coding processing is performed according to the speech to be encoded and the target speech features through the coding network to obtain target audio features, and residual vector quantization processing is performed on the target audio features to obtain target coding results, and by using detected speech features or historical speech features as target speech features to perform speech coding processing with the speech to be encoded, and combining audio encoding with speech features related to the speech to be encoded, it is possible to improve the compression rate of the audio while improving the speech fidelity and the audio coding quality.

[0051] Based on the above embodiments, Figure 2 A flowchart of another audio encoding method provided in an embodiment of the present application is given, and the audio encoding method is a specific implementation of the above audio encoding method. Figure 2 , the audio encoding method comprises:

[0052] S210: Perform voice activity detection on the speech to be encoded to obtain a voice detection result of the speech to be encoded within a preset time length.

[0053] S220: When the speech detection result reaches a preset threshold, extracting a detected speech feature of a preset time length from the speech to be encoded.

[0054] Exemplarily, after obtaining the speech to be encoded, voice activity detection (VAD, Voiceactivity detection) can be performed on the speech to be encoded, and the speech detection result of the speech to be encoded within a preset time length can be determined according to the voice activity detection result of the speech to be encoded. For example, voice activity detection is continuously performed on the speech to be encoded to determine the voice activity detection value of the speech to be encoded within a continuous preset time length, for example, based on the voice activity detection of the speech to be encoded, it is determined whether there is a human voice in each speech frame within a preset time length (from the current time period to the preset time length), and the proportion of speech frames with human voice in the speech frames within the preset time length is calculated, and the proportion can be used as the speech detection result.

[0055] Exemplarily, the above-determined speech detection result is compared with a preset threshold (the preset threshold may be 0.6-0.9, for example, the preset threshold is set to 0.8). When the speech detection result reaches the preset threshold, it can be considered that the speech component in the speech to be encoded is high and is suitable as audio for speech feature extraction, and the detection speech feature of the preset time length can be extracted from the speech to be encoded. When the speech detection result does not reach the preset threshold, it can be considered that the speech component in the speech to be encoded is low and is not suitable as audio for speech feature extraction, and the speech to be encoded can be directly sent to the encoding network for encoding processing, or encoded through other preset encoding networks.

[0056] In one embodiment, the detected speech features of a speech segment of a preset time length in the speech to be encoded can be extracted through a trained speech feature extraction network. The speech feature extraction network can perform discrete Fourier transform, convolution processing, feature compression processing, shape conversion (reshape) and averaging processing (for example, averaging in the time dimension, that is, dividing the output of the shape conversion by the preset time length) on the speech segment of the preset time length in the speech to be encoded to obtain the speech features to be encoded.

[0057] For example, Figure 3As shown in the structural schematic diagram of a speech feature extraction network provided, the speech feature extraction network includes a discrete Fourier transform block (DFT), a convolution processing block (Conv2D), a multi-layer feature compression processing block (EncoderBlock, described using two layers as an example), a shape conversion block (reshape) and an average processing block (average), which are respectively used to perform discrete Fourier transform, convolution processing, feature compression processing, shape conversion (reshape) and average processing (for example, averaging in the time dimension, i.e. dividing the output of the shape conversion by the preset time length) on an input speech segment of a preset time length. Obtain the speech feature to be encoded. Among them, each feature compression processing block includes multiple layers of residual blocks (ResBlock2D, taking two layers as an example) connected in sequence, and a first convolution block (Conv2D). Each residual block includes a second convolution block (Conv2D(3×3, dilation=3)) and a third convolution block (Conv2D(1×1)) connected in sequence, and a fourth convolution block (Conv2D(1×1)) connected in parallel with the second convolution block and the third convolution block. The outputs of the third convolution block and the fourth convolution block are added as the output of the residual block.

[0058] For example, the 5-second speech time domain data (i.e., the speech segment of a preset time length in the speech to be encoded) is framed and windowed, and passed through a 512-point discrete Fourier transform block DFT to obtain the time-frequency domain feature feat1, whose dimension is: 3*256*t. The frequency domain feature feat1 undergoes a two-dimensional convolution Conv2D (3,32, kernel_size=(3,3), stride=(1,1)) through the convolution processing block to obtain feature feat2, whose dimension is: 32*256*t. Feature feat2 undergoes a two-layer feature compression processing block Encoder Block to obtain feature feat3, whose dimension is: 128*1*t. The structure of the Encoder Block is as follows Figure 3 As shown in the upper right corner, it consists of 2 layers of ResBlock2D and 1 layer of Conv2D, where the ResBlock2D structure is as follows Figure 3 As shown in the lower right corner, it is composed of a convolutional network of Conv2D (3×3, dilation=3) and Conv2D (1×1) connected in series, and skip is Conv2D (1×1). Feature feat3 is reshaped by the shape conversion block to obtain feature feat4, whose dimension is: 128*t. Feature feat4 is averaged in the time dimension in the average processing block to obtain a 128*1 dimensional speech feature, which is the detected speech feature. Among them, the time average vector of the last layer of speech feature extraction in the speech feature extraction network is used as the feature of the current speech, which is a 128-dimensional vector with a fixed length.

[0059] This solution obtains a speech detection result of a preset time length by performing speech activity detection on the speech to be encoded, and when the speech detection result reaches a preset threshold, extracts the detection speech features of the preset time length from the speech to be encoded, accurately determines the audio suitable for speech feature extraction and extracts the detection speech features, which can accurately determine the target speech features and improve the speech fidelity of audio encoding.

[0060] S230: performing voice feature matching in a voice feature library according to the detected voice feature, and determining a target voice feature according to the matching result, wherein the voice feature library records one or more historical voice features, and the target voice feature is a detected voice feature or a historical voice feature.

[0061] In a possible embodiment, the audio coding method provided by the present solution performs voice feature matching in a voice feature library according to the detected voice features, and determines the target voice features according to the matching results, which may include:

[0062] S231: Determine similarity information between the detected speech feature and one or more historical speech features in the speech feature library.

[0063] S232: When there is a historical speech feature whose similar information reaches a set similarity threshold, and the signal-to-noise ratio of the speech to be encoded is less than or equal to a preset signal-to-noise ratio threshold, the historical speech feature is determined as the target speech feature.

[0064] S233: When there is no historical speech feature whose similar information reaches the set similarity threshold, or the signal-to-noise ratio of the speech to be encoded is greater than a preset signal-to-noise ratio threshold, the detected speech feature is determined as the target speech feature.

[0065] Exemplarily, after extracting the detected speech features of a preset time length, similarity information between the detected speech features and each historical speech feature in the speech feature library is calculated, and the signal-to-noise ratio (SNR) of the speech to be encoded is determined. The similarity information may be a vector distance between the detected speech features and the historical speech features, or may be a similarity between the detected speech features and the historical speech features.

[0066] After determining the similarity information between the detected speech feature and the historical speech feature, it can be determined whether the similarity information reaches the set similarity threshold, and whether the signal-to-noise ratio of the speech to be encoded is less than or equal to the preset signal-to-noise ratio threshold (the preset signal-to-noise ratio threshold can be 20-40 decibels, for example, the preset signal-to-noise ratio threshold can be set to 30 decibels). Optionally, when the vector distance between the detected speech feature and the historical speech feature is less than or equal to the preset distance threshold, or the similarity between the detected speech feature and the historical speech feature is greater than or equal to the preset similarity threshold, it can be considered that the similarity information reaches the set similarity threshold; when the vector distance between the detected speech feature and the historical speech feature is greater than the preset distance threshold, or the similarity between the detected speech feature and the historical speech feature is less than the preset similarity threshold, it can be considered that the similarity information does not reach the set similarity threshold.

[0067] In one embodiment, when there are historical speech features whose similar information reaches a set similarity threshold, and the signal-to-noise ratio of the speech to be encoded is less than or equal to a preset signal-to-noise ratio threshold, the historical speech features whose similar information reaches the set similarity threshold can be determined as matching historical speech features, and the historical speech features can be determined as target speech features. By replacing the detected speech features with lower signal-to-noise ratio with historical speech features of better quality as target speech features, the quality of speech coding can be effectively improved.

[0068] In one embodiment, when there is no historical speech feature whose similar information reaches a set similarity threshold (i.e., no historical speech feature is matched in the speech feature library), or the signal-to-noise ratio of the speech to be encoded is greater than a preset signal-to-noise ratio threshold, the detected speech feature is determined as the target speech feature. By determining the detected speech feature as the target speech feature, the influence of historical speech features with a lower matching degree on speech coding is reduced.

[0069] This solution uses higher quality speech features for speech encoding based on the similarity between the detected speech features and the historical speech features, and the signal-to-noise ratio of the speech to be encoded is less than the target speech features to accurately determine the target speech features, and effectively improves the speech encoding quality. It can improve the sound quality of AI audio encoding and decoding while ensuring compression efficiency (low bit rate), and realize personalized audio encoding and decoding for specific speakers, solving the problems of insufficient personalization, insufficient utilization of historical data, and poor dynamic adaptability in the existing technology.

[0070] In a possible embodiment, the audio coding method provided by the present scheme can, after extracting the detected speech features of a preset time length from the speech to be encoded, also update the detected speech features as historical speech features into the speech feature library when the signal-to-noise ratio of the speech to be encoded is greater than a preset signal-to-noise ratio threshold.

[0071] Exemplarily, after extracting the detection speech features of a preset time length from the speech to be encoded, it can be determined whether the signal-to-noise ratio of the speech to be encoded is greater than a preset signal-to-noise ratio threshold. When the signal-to-noise ratio of the speech to be encoded is greater than the preset signal-to-noise ratio threshold, the speech to be detected of the preset time length can be considered to be a clean human voice, and the detection speech features corresponding to this segment of speech can be updated as historical speech features into the speech feature library.

[0072] Optionally, when updating the detected speech feature as a historical speech feature to the speech feature library, if there is already a historical speech feature in the speech feature library whose similar information to the detected speech feature reaches a set similarity threshold, the detected speech feature can be used to replace the similar historical speech feature to achieve real-time updating of the historical speech features of the same object; if there is no historical speech feature in the speech feature library whose similar information to the detected speech feature reaches a set similarity threshold, the detected speech feature can be directly added to the speech feature library.

[0073] This solution updates the detected speech features corresponding to the speech to be encoded and determines whether the signal-to-noise ratio is greater than a preset signal-to-noise ratio threshold into a speech feature library, continuously updates and expands the speech feature library, forms a closed loop for dynamically optimizing speech coding, and continuously optimizes the audio coding quality.

[0074] S240: adding the target speech feature to the trained coding network, performing speech coding processing according to the to-be-coded speech and the target speech feature through the coding network, and obtaining the target audio feature.

[0075] In one possible embodiment, Figure 4 As shown in the schematic diagram of a target audio feature generation process provided, the coding network provided in this solution performs speech coding processing according to the speech to be coded and the target speech features to obtain the target audio features, including:

[0076] S241: Perform discrete Fourier transform, convolution processing and feature compression processing on the speech to be encoded to obtain the features of the speech to be encoded.

[0077] S242: Fusing the to-be-encoded speech feature and the target speech feature to obtain a fused audio feature.

[0078] S243: Perform feature compression and convolution processing on the fused audio features to obtain target audio features.

[0079] Exemplarily, the coding network sequentially performs discrete Fourier transform (DFT), convolution processing (e.g., two-dimensional convolution processing) and feature compression processing (e.g., feature compression processing through one or more encoder blocks) on the speech to be coded to obtain the features of the speech to be coded. In one embodiment, after the speech to be coded is subjected to discrete Fourier transform, convolution processing and feature compression processing, shape conversion (reshape) and averaging processing (e.g., averaging in the time dimension) may be sequentially performed to obtain the features of the speech to be coded. Among them, the discrete Fourier transform, convolution processing and feature compression processing of the speech to be coded may refer to the discrete Fourier transform, convolution processing and feature compression processing of the speech segment of the preset time length extracted from the speech to be coded for the detected speech features of the preset time length, and this solution will not be repeated.

[0080] In one embodiment, after obtaining the speech feature to be encoded, the speech feature to be encoded and the target speech feature may be fused to obtain a fused audio feature. Optionally, the speech feature to be encoded and the target speech feature may be fused by dot-multiplying, adding or concatenating the detected speech feature and the matched historical speech feature, and using the dot-multiplication result, addition result or concatenation result as the fused audio feature.

[0081] In a possible embodiment, the audio coding method provided by the present solution fuses the speech feature to be encoded and the target speech feature to obtain a fused audio feature, which may be: performing a dot product process on the speech feature to be encoded and the target speech feature, and using the dot product process result as the fused audio feature.

[0082] Exemplarily, after obtaining the speech features to be encoded, the speech features to be encoded and the target speech features can be subjected to dot product processing, and the dot product processing result can be used as the fused audio features. The fused audio features can be obtained by performing dot product processing on the speech features to be encoded and the target speech features, and the target speech features can be effectively fused into the speech features to be encoded. The fused audio features can more accurately reflect the voice characteristics of the speaker and effectively improve the sound fidelity of audio coding.

[0083] In one embodiment, after obtaining the fused audio features, the fused audio features may be further subjected to feature compression (e.g., feature compression processing by an encoder block) and convolution processing (e.g., convolution processing by a convolution block Conv1D) to obtain target audio features. Optionally, after the fused audio features are subjected to feature compression, the fused audio features after feature compression may be subjected to shape transformation processing, and then convolution processing may be performed to obtain target audio features.

[0084] In a possible embodiment, the audio encoding method provided by the present solution may further fuse the preset control input information into the fused audio feature before performing feature compression and convolution processing on the fused audio feature to obtain the target audio feature. Correspondingly, the present solution performs feature compression and convolution processing on the fused audio feature to obtain the target audio feature, which may be: when the control input information is the first preset input, the fused audio feature that is fused with the control input information is subjected to feature compression and convolution processing to obtain the target audio feature.

[0085] This scheme can set whether to merge the target speech feature into the speech feature to be encoded by controlling the input information. For example, when the target speech feature needs to be merged into the speech feature to be encoded, the control input information can be set to the first preset input (for example, the control input information is set to 1); when the target speech feature does not need to be merged into the speech feature to be encoded, the control input information can be set to the second preset input (for example, the control input information is set to 0).

[0086] Exemplarily, before the fused audio features are subjected to feature compression and convolution processing to obtain the target audio features, the preset control input information is fused into the fused audio features. For example, the control input information is spliced ​​into the fused audio features. Correspondingly, when the control input information is a first preset input, the fused audio features fused with the control input information may be subjected to feature compression and convolution processing to obtain the target audio features, and when the control input information is a second preset input, the speech features to be encoded may be subjected to feature compression and convolution processing to obtain the target audio features. This solution flexibly controls whether to integrate the control input information into the speech features to be encoded for encoding processing through the control input information, thereby improving the flexibility of speech encoding.

[0087] like Figure 5 As shown in the structural schematic diagram of a coding network provided, the coding network includes a discrete Fourier transform block (DFT), a convolution processing block (Conv2D), a multi-layer feature compression processing block (Encoder Block, described by taking two layers as an example), a connection block (concat), a feature compression processing block (Encoder Block), a shape conversion block (reshape), a convolution processing block (Conv1D) and a replication block connected in sequence, wherein the replication block receives speech segment features (i.e., target speech features) and performs a dot multiplication with the output of the last layer in the multi-layer feature compression processing block to obtain a fused audio feature, and the connection block receives the fused audio feature and control input information, and fuses the control input information into the fused audio feature.

[0088] For example, the speech to be encoded is processed by the discrete Fourier transform block (DFT), the convolution processing block (Conv2D), and the two-layer feature compression processing block (Encoder Block) to obtain feature FEAT3, whose dimension is: 128*1*T. The 128-dimensional speech segment feature (i.e., the target speech feature) is copied and expanded in the time dimension through the replication block repeat to obtain feature FEAT4 with a dimension of 128*1*T. Feature FEAT3 is dot-multiplied with feature FEAT4 to achieve feature enhancement to obtain feature FEAT5, whose dimension is: 128*1*T. The control input information and feature FEAT5 are fused and concat to obtain feature FEAT6, whose dimension is: 129*1*T. When the control input information is 0, no special processing is performed on the audio. When the control input information is 1, the target speech feature is fused into the speech feature to be encoded to suppress noise and interference in the audio. Feature FEAT6 is processed through the feature compression processing block Encoder Block to obtain feature FEAT7, whose dimension is: 128*1*T. The feature FEAT7 is reshaped by the shape conversion block to obtain the feature FEAT8, whose dimension is: 1*128*T. The feature FEAT8 is processed by the convolution processing block Conv1D to obtain the audio feature (ie, the target audio feature), whose dimension is: 1*128*T.

[0089] This scheme obtains the features of the speech to be encoded by performing discrete Fourier transform, convolution processing and feature compression processing on the speech to be encoded, fuses the features of the speech to be encoded and the features of the target speech into fused audio features, and performs feature compression and convolution processing on the fused audio features to obtain the target audio features. The fused audio features can more accurately reflect the voice characteristics of the speaker and effectively improve the sound fidelity of audio coding.

[0090] The decoding network corresponds to the encoding network. For example, the decoding network may include a convolution processing block, a shape conversion block, a multi-layer decoding block (taking three layers as an example), a shape conversion block, and an inverse discrete Fourier transform block connected in sequence. Figure 6As shown in the structural diagram of a decoding network provided, the decoding network includes a sequentially connected convolution processing block (Conv1D), a shape conversion block (reshape), a multi-layer decoding block (taking a three-layer Decoder Block as an example), a shape conversion block (reshap) and an inverse discrete Fourier transform block (IDFT), wherein the decoding block is similar to the encoding block, and the decoding block includes sequentially connected multi-layer residual blocks (ResBlock2D, taking a two-layer residual block as an example) and a convolution block (Conv2D), each residual block includes sequentially connected convolution blocks (Conv2D(3×3, dilation=3)), a convolution block (Conv2D(1×1)), and a convolution block (Conv2D(1×1)) connected in parallel with two convolution blocks, and the outputs of the convolution blocks are added as the output of the residual block.

[0091] For example, the input of the decoding network is an audio feature (such as the target encoding result), whose dimension is: 1*128*T. The audio feature is processed by the convolution processing block Conv1D to obtain the feature FEAT9, whose dimension is: 1*128*T. The feature FEAT9 is reshaped by the shape conversion block to obtain the feature FEAT10, whose dimension is: 128*1*T. The feature FEAT10 is processed by the 3-layer decoding block DecoderBlock to obtain the feature FEAT11, whose dimension is: 32*256*T. The feature FEAT11 is reshaped by the shape conversion block to obtain the audio feature feature FEAT12, whose dimension is: 1*256*T. The feature FEAT12 is inversely transformed by the inverse discrete Fourier transform block to obtain the decoded audio.

[0092] S250: Perform residual vector quantization processing on the target audio feature to obtain a target encoding result.

[0093] In the above, by extracting the detected speech features of a preset time length from the speech to be encoded, matching the speech features in the speech feature library according to the detected speech features, and determining the target speech features according to the matching results, the target speech features are added to the trained coding network, and the speech coding processing is performed according to the speech to be encoded and the target speech features through the coding network to obtain the target audio features, and the target audio features are subjected to residual vector quantization processing to obtain the target coding result, and the detected speech features or historical speech features are used as the target speech features to perform speech coding processing with the speech to be encoded, and the audio coding is performed in combination with the speech features related to the speech to be encoded, the compression rate of the audio can be improved while the speech fidelity is improved, and the audio coding quality can be improved. At the same time, by accurately determining the target speech features according to the similar information of the detected speech features and the historical speech features, and the signal-to-noise ratio of the speech to be encoded is less than that of the target speech features, a higher quality speech feature is used for speech coding, and the speech coding quality is effectively improved. It can improve the sound quality of AI audio coding and decoding under the premise of ensuring compression efficiency (low bit rate), and realize personalized audio coding and decoding for specific speakers, which solves the problems of insufficient personalization, insufficient use of historical data and poor dynamic adaptability in the prior art. By establishing a historical voice feature library and comparing it in real time during a call, the voice features of a specific speaker can be identified and embedded. This personalized processing significantly improves the encoding fidelity, which is better than traditional general AI encoding, ensuring high-quality audio output at low bit rates. By embedding the historical voice features of a specific speaker, not only the sound quality is improved, but also the target voice can be effectively identified, the noise and interference in the audio can be suppressed, the target voice can be accurately identified, and background noise and interference can be reduced, thereby improving the call quality, allowing the codec to better adapt to the voice features of specific users and provide more accurate audio processing. In addition, in each call, it can dynamically choose whether to embed historical voice features based on the real-time detected voice features. If the similarity does not reach the threshold, run general AI encoding, and select appropriate detected voice features during the call and store them in the voice feature library. The flexibility and adaptability of audio encoding are ensured through dynamic adjustment and closed-loop optimization mechanisms, and the encoding and decoding effect can be continuously optimized as the user's voice features change.

[0094] Figure 7 is a structural diagram of an audio encoding device provided in an embodiment of the present application. Figure 7 The audio encoding device includes a speech detection module 71, a feature determination module 72, a feature encoding module 73 and a vector processing module 74.

[0095] Among them, the speech detection module 71 is configured to extract the detection speech features of a preset time length from the speech to be encoded; the feature determination module 72 is configured to perform speech feature matching in a speech feature library according to the detection speech features, and determine the target speech features according to the matching results, the speech feature library records one or more historical speech features, and the target speech features are the detection speech features or the historical speech features; the feature encoding module 73 is configured to add the target speech features to the trained encoding network, and perform speech encoding processing according to the speech to be encoded and the target speech features through the encoding network to obtain the target audio features; the vector processing module 74 is configured to perform residual vector quantization processing on the target audio features to obtain the target encoding result.

[0096] In the above, by extracting detected speech features of a preset time length from the speech to be encoded, performing speech feature matching in a speech feature library according to the detected speech features, and determining target speech features according to the matching results, the target speech features are added to the trained coding network, and speech coding processing is performed according to the speech to be encoded and the target speech features through the coding network to obtain target audio features, and residual vector quantization processing is performed on the target audio features to obtain target coding results, and by using detected speech features or historical speech features as target speech features to perform speech coding processing with the speech to be encoded, and combining audio encoding with speech features related to the speech to be encoded, it is possible to improve the compression rate of the audio while improving the speech fidelity and the audio coding quality.

[0097] In a possible embodiment, the audio encoding device further includes a speech detection module, which is configured to: perform speech activity detection on the speech to be encoded to obtain a speech detection result of the speech to be encoded in a preset time length;

[0098] Extracting detection speech features of a preset time length from the speech to be encoded, including:

[0099] When the speech detection result reaches a preset threshold, detected speech features of a preset time length are extracted from the speech to be encoded.

[0100] The speech detection module 71 performs speech feature matching in the speech feature library according to the detected speech features, and determines the target speech features according to the matching results, and is configured as follows:

[0101] Determining similarity information between the detected speech feature and one or more historical speech features in the speech feature library;

[0102] In the case where there are historical speech features whose similar information reaches a set similarity threshold, and the signal-to-noise ratio of the speech to be encoded is less than or equal to a preset signal-to-noise ratio threshold, determining the historical speech features as target speech features;

[0103] In the case that there is no historical speech feature whose similar information reaches the set similarity threshold, or the signal-to-noise ratio of the speech to be encoded is greater than the preset signal-to-noise ratio threshold, the detected speech feature is determined as the target speech feature.

[0104] In a possible embodiment, the feature encoding module 73 encoding network performs speech encoding processing according to the to-be-encoded speech and the target speech features to obtain the target audio features, and is configured as follows:

[0105] Performing discrete Fourier transform, convolution processing and feature compression processing on the speech to be encoded to obtain the features of the speech to be encoded;

[0106] The to-be-encoded speech feature and the target speech feature are fused to obtain a fused audio feature;

[0107] The fused audio features are compressed and convolved to obtain the target audio features.

[0108] In a possible embodiment, the feature encoding module 73 fuses the to-be-encoded speech feature and the target speech feature to obtain a fused audio feature, which is configured as follows:

[0109] The speech feature to be encoded and the target speech feature are subjected to dot multiplication processing, and the dot multiplication result is used as the fused audio feature.

[0110] In a possible embodiment, before the feature encoding module 73 performs feature compression and convolution processing on the fused audio feature to obtain the target audio feature, it is further configured as follows:

[0111] Fusing the preset control input information into the fused audio feature;

[0112] Correspondingly, the feature encoding module 73 performs feature compression and convolution processing on the fused audio features to obtain target audio features, and is configured as follows:

[0113] When the control input information is the first preset input, feature compression and convolution processing are performed on the fused audio features fused with the control input information to obtain the target audio features.

[0114] In a possible embodiment, the audio encoding device further includes a feature updating module, and the feature updating module is configured to:

[0115] When the signal-to-noise ratio of the to-be-encoded speech is greater than a preset signal-to-noise ratio threshold, the detected speech features are updated into the speech feature library as historical speech features.

[0116] It is worth noting that in the embodiment of the above-mentioned audio encoding device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present application.

[0117] The embodiment of the present application also provides an audio encoding device, which can integrate the audio encoding device provided in the embodiment of the present application. Figure 8 is a structural diagram of an audio encoding device provided in an embodiment of the present application. Figure 8 The audio encoding device includes: an input device 83, an output device 84, a memory 82 and one or more processors 81; the memory 82 is used to store one or more programs; when one or more programs are executed by one or more processors 81, the one or more processors 81 implement the audio encoding method provided in the above embodiment. The audio encoding device, equipment and computer provided above can be used to execute the audio encoding method provided in any of the above embodiments, and have corresponding functions and beneficial effects.

[0118] The embodiment of the present application also provides a non-volatile storage medium storing computer executable instructions, which are used to execute the audio encoding method provided in the above embodiment when executed by a computer processor. Of course, the non-volatile storage medium storing computer executable instructions provided in the embodiment of the present application, whose computer executable instructions are not limited to the audio encoding method provided above, can also execute the related operations in the audio encoding method provided in any embodiment of the present application. The audio encoding device, equipment and storage medium provided in the above embodiments can execute the audio encoding method provided in any embodiment of the present application. For technical details not described in detail in the above embodiments, please refer to the audio encoding method provided in any embodiment of the present application.

[0119] On the basis of the above embodiments, the embodiments of the present application also provide a computer program product. The technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer program product is stored in a storage medium and includes a number of instructions for enabling a computer device, a mobile terminal or a processor therein to execute all or part of the steps of the audio encoding method provided in each embodiment of the present application.

Claims

1. An audio encoding method, characterized in that: include: Extracting detection speech features of a preset time length from the speech to be encoded; Performing voice feature matching in a voice feature library according to the detected voice feature, and determining a target voice feature according to the matching result, wherein the voice feature library records one or more historical voice features, and the target voice feature is the detected voice feature or the historical voice feature; The target speech feature is added to the trained coding network, and the coding network performs speech coding processing according to the speech to be coded and the target speech feature to obtain the target audio feature; The target audio feature is subjected to residual vector quantization processing to obtain a target encoding result.

2. The audio encoding method according to claim 1, characterized in that Before extracting the detected speech features of the preset time length from the speech to be encoded, the method further includes: Performing voice activity detection on the speech to be encoded to obtain a voice detection result of the speech to be encoded within a preset time length; The step of extracting the detected speech features of a preset time length from the speech to be encoded comprises: When the speech detection result reaches a preset threshold, detected speech features of a preset time length are extracted from the speech to be encoded.

3. The audio encoding method according to claim 1, characterized in that The performing speech feature matching in a speech feature library according to the detected speech feature, and determining the target speech feature according to the matching result, comprises: Determining similarity information between the detected speech feature and one or more historical speech features in a speech feature library; If there is a historical speech feature whose similar information reaches a set similarity threshold, and the signal-to-noise ratio of the speech to be encoded is less than or equal to a preset signal-to-noise ratio threshold, determining the historical speech feature as a target speech feature; In the case that there is no historical speech feature whose similar information reaches the set similarity threshold, or the signal-to-noise ratio of the speech to be encoded is greater than a preset signal-to-noise ratio threshold, the detected speech feature is determined as the target speech feature.

4. The audio encoding method according to claim 1, characterized in that: The coding network performs speech coding processing according to the to-be-coded speech and the target speech feature to obtain the target audio feature, including: Performing discrete Fourier transform, convolution processing and feature compression processing on the speech to be encoded to obtain the features of the speech to be encoded; Fusing the to-be-encoded speech feature and the target speech feature to obtain a fused audio feature; The fused audio features are subjected to feature compression and convolution processing to obtain target audio features.

5. The audio encoding method according to claim 4, characterized in that: The step of fusing the to-be-encoded speech feature and the target speech feature to obtain a fused audio feature includes: The speech feature to be encoded and the target speech feature are subjected to a dot product process, and the dot product process result is used as a fused audio feature.

6. The audio encoding method according to claim 4, characterized in that: Before performing feature compression and convolution processing on the fused audio features to obtain target audio features, the method further includes: fusing the preset control input information into the fused audio feature; Correspondingly, performing feature compression and convolution processing on the fused audio features to obtain target audio features includes: In a case where the control input information is the first preset input, feature compression and convolution processing are performed on the fused audio features fused with the control input information to obtain target audio features.

7. The audio encoding method according to claim 1, characterized in that: After extracting the detected speech features of the preset time length from the speech to be encoded, the method further includes: When the signal-to-noise ratio of the to-be-encoded speech is greater than a preset signal-to-noise ratio threshold, the detected speech feature is updated into the speech feature library as a historical speech feature.

8. An audio encoding device, characterized in that: It includes a speech detection module, a feature determination module, a feature encoding module and a vector processing module, wherein: The speech detection module is configured to extract detection speech features of a preset time length from the speech to be encoded; The feature determination module is configured to perform voice feature matching in a voice feature library according to the detected voice feature, and determine a target voice feature according to the matching result, wherein the voice feature library records one or more historical voice features, and the target voice feature is the detected voice feature or the historical voice feature; The feature encoding module is configured to add the target speech feature to the trained encoding network, and perform speech encoding processing according to the to-be-encoded speech and the target speech feature through the encoding network to obtain the target audio feature; The vector processing module is configured to perform residual vector quantization processing on the target audio feature to obtain a target encoding result.

9. An audio encoding device, characterized in that include: memory and one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the audio encoding method as described in any one of claims 1-7.

10. A non-volatile storage medium storing computer executable instructions, characterized in that: When the computer executable instructions are executed by a computer processor, they are used to perform the audio encoding method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the audio encoding method according to any one of claims 1 to 7 is implemented.

Images