Audio communication method, audio conversion method, apparatus, electronic device, computer-readable storage medium, and computer program product

Abstract

An audio communication method, an audio conversion method, a bitstream processing method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product. The audio communication method comprises: in response to a first communication request for an audio signal, acquiring, from a plurality of communication modes, a voice transformation mode for the audio signal (101); performing feature coding on the audio signal, so as to obtain a coded feature of the audio signal (102); acquiring a target timbre corresponding to the voice transformation mode, and determining a timbre feature of the target timbre (103); performing timbre conversion on the coded feature on the basis of the timbre feature, so as to obtain a target coded feature (104); and performing signal coding on the target coded feature, so as to obtain a target audio bitstream conforming to the target timbre, and transmitting the target audio bitstream to a decoding terminal (105).

Description

Audio communication method, audio conversion method, device, electronic device, computer readable storage medium and computer program product

[0001] Cross-reference to Related Applications

[0002] Embodiments of the present application are based on and claim priority from Chinese Patent Application No. 202410619509.8 filed on May 14, 2024, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0003] The present application relates to the field of artificial intelligence, and in particular to an audio communication method, an audio conversion method, a device, an electronic device, a computer readable storage medium and a computer program product. BACKGROUND

[0004] Audio coding technology is one of the important applications in the field of artificial intelligence, and is a core technology in communication services including remote audio and video calls. Simply put, speech coding technology is to use less network bandwidth resources to transmit as much voice information as possible. From the perspective of Shannon's information theory, speech coding is a kind of source coding, and the purpose of source coding is to compress the amount of data of the information we want to transmit as much as possible at the encoding end, and remove the redundancy in the information, while at the decoding end it can be restored without loss (or close to lossless).

[0005] In related technologies, the form of the audio code stream generated by the encoding end is relatively single, which cannot meet the user's demand. SUMMARY

[0006] Embodiments of the present application provide an audio communication method, an audio conversion method, a device, an electronic device, a computer readable storage medium and a computer program product, which can generate audio code streams of different timbres while ensuring audio coding efficiency.

[0007] The technical scheme of the embodiments of the present application is implemented as follows:

[0008] The present application provides an audio communication method, applied to an encoding terminal, the method comprising:

[0009] In response to a first communication request for an audio signal, a voice changing mode for the audio signal is obtained from a plurality of communication modes;

[0010] The audio signal is feature encoded to obtain the encoded features of the audio signal;

[0011] The target timbre corresponding to the voice changing mode is obtained, and the timbre features of the target timbre are determined;

[0012] performing timbre conversion on the encoded feature based on the timbre feature to obtain a target encoded feature;

[0013] performing signal encoding on the target encoded feature to obtain a target audio bitstream conforming to the target timbre, and transmitting the target audio bitstream to a decoding terminal.

[0014] An audio communication method is provided in an embodiment of the application, and the method is applied to a decoding terminal, and the method comprises the following steps of:

[0015] receiving a target audio bitstream transmitted by an encoding terminal, wherein the target audio bitstream is obtained by performing timbre conversion on an audio signal based on a target timbre corresponding to a voice mode;

[0016] performing signal decoding on the target audio bitstream to obtain a target encoded feature estimation value corresponding to the target audio bitstream;

[0017] performing feature decoding on the target encoded feature estimation value to obtain a reconstructed audio signal conforming to the target timbre.

[0018] An audio conversion method is provided in an embodiment of the application, and the method is applied to a terminal, and the method comprises the following steps of:

[0019] performing feature encoding on an audio signal to obtain an encoded feature of the audio signal;

[0020] obtaining a target timbre corresponding to a voice mode and determining a timbre feature of the target timbre;

[0021] performing timbre conversion on the encoded feature based on the timbre feature to obtain a target encoded feature;

[0022] performing signal encoding on the target encoded feature to obtain a target audio bitstream conforming to the target timbre;

[0023] performing signal decoding on the target audio bitstream to obtain a target encoded feature estimation value corresponding to the target audio bitstream;

[0024] performing feature decoding on the target encoded feature estimation value to obtain a reconstructed audio signal conforming to the target timbre.

[0025] A method for processing a bitstream is provided in an embodiment of the application, wherein the bitstream is generated based on the audio communication method or the audio conversion method provided in the embodiments of the application or is decoded based on the audio communication method or the audio conversion method provided in the embodiments of the application.

[0026] An audio communication device is provided in an embodiment of the application, and the device comprises the following steps of:

[0027] The acquisition module is configured to acquire a voice changing mode for the audio signal from a plurality of communication modes in response to a first communication request for the audio signal.

[0028] The feature encoding module is configured to perform feature encoding on the audio signal to obtain encoded features of the audio signal.

[0029] The determination module is configured to obtain a target timbre corresponding to the voice changing mode and determine timbre features of the target timbre.

[0030] The conversion module is configured to perform timbre conversion on the encoded features based on the timbre features to obtain target encoded features.

[0031] The information encoding module is configured to perform signal encoding on the target encoded features to obtain a target audio bitstream conforming to the target timbre and transmit the target audio bitstream to a decoding terminal.

[0032] An embodiment of the present application provides an audio communication device, comprising:

[0033] The receiving module is configured to receive a target audio bitstream transmitted by an encoding terminal, wherein the target audio bitstream is obtained by performing timbre conversion on an audio signal based on a target timbre corresponding to a voice changing mode.

[0034] The information decoding module is configured to perform signal decoding on the target audio bitstream to obtain a target encoded feature estimation value corresponding to the target audio bitstream.

[0035] The feature decoding module is configured to perform feature decoding on the target encoded feature estimation value to obtain a reconstructed audio signal conforming to the target timbre.

[0036] An embodiment of the present application provides an audio conversion device, comprising:

[0037] The feature encoding module is configured to perform feature encoding on an audio signal to obtain encoded features of the audio signal.

[0038] The determination module is configured to obtain a target timbre corresponding to a voice changing mode and determine timbre features of the target timbre.

[0039] The conversion module is configured to perform timbre conversion on the encoded features based on the timbre features to obtain target encoded features.

[0040] The information encoding module is configured to perform signal encoding on the target encoded features to obtain a target audio bitstream conforming to the target timbre.

[0041] The information decoding module is configured to perform signal decoding on the target audio bitstream to obtain a target encoded feature estimation value corresponding to the target audio bitstream.

[0042] a feature decoding module, configured to perform feature decoding on the target coding feature estimation value to obtain a reconstructed audio signal conforming to the target timbre.

[0043] An electronic device is provided in an embodiment of the present application, and the electronic device comprises:

[0044] a memory, configured to store a computer program or computer executable instructions;

[0045] a processor, configured to execute the computer program or computer executable instructions stored in the memory to implement the audio communication method or the audio conversion method provided in the embodiments of the present application.

[0046] A computer readable storage medium is provided in an embodiment of the present application, and the computer readable storage medium stores a computer program or computer executable instructions, and the computer program or computer executable instructions are executed by a processor to implement the audio communication method or the audio conversion method provided in the embodiments of the present application.

[0047] A computer readable storage medium is provided in an embodiment of the present application, and the computer readable storage medium stores a code stream formed by a computer program or computer executable instructions, and the computer program or computer executable instructions are executed by a processor to implement the audio communication method or the audio conversion method provided in the embodiments of the present application.

[0048] A computer program product is provided in an embodiment of the present application, and the computer program product comprises a computer program or computer executable instructions, and the computer program or computer executable instructions are executed by a processor to implement the audio communication method or the audio conversion method provided in the embodiments of the present application.

[0049] The embodiments of the present application have the following beneficial effects:

[0050] The timbre features based on the target timbre are used to perform timbre conversion on the coding features of the audio signal, and the coding features are obtained by performing feature coding on the audio signal, so as to ensure that the target coding features conform to the target timbre, and reduce the feature dimension of the target coding features, thereby performing signal coding on the target coding features to obtain a target audio code stream conforming to the target timbre, so as to generate target audio code streams with different timbres in the case of ensuring the audio coding efficiency, implement streaming voice changing, diversify the target audio code stream, and thus meet the actual application requirements of users. BRIEF DESCRIPTION OF DRAWINGS

[0051] FIG. 1 is a system block diagram of voice conversion provided by the related art;

[0052] FIG. 2 is an architecture schematic diagram of an audio coding and decoding system provided in an embodiment of the present application;

[0053] FIGS. 3A-3C are structural schematic diagrams of an electronic device according to embodiments of the present disclosure;

[0054] FIGS. 4A-4F are flow diagrams of an audio communication method according to embodiments of the present disclosure;

[0055] FIGS. 5A-5C are flow diagrams of an audio communication method according to embodiments of the present disclosure;

[0056] FIG. 6A is a structural schematic diagram of an audio conversion system according to embodiments of the present disclosure;

[0057] FIG. 6B is a structural schematic diagram of a multi-task voice communication system according to embodiments of the present disclosure;

[0058] FIG. 7A is a schematic diagram of a general convolutional network according to embodiments of the present disclosure;

[0059] FIG. 7B is a schematic diagram of a hole convolutional network according to embodiments of the present disclosure;

[0060] FIG. 8 is a flow diagram of a multi-task coding method according to embodiments of the present disclosure;

[0061] FIG. 9 is a schematic diagram of a first neural network according to embodiments of the present disclosure;

[0062] FIG. 10A is a structural schematic diagram of a residual block used in an encoding block according to embodiments of the present disclosure;

[0063] FIG. 10B is a structural schematic diagram of a residual unit according to embodiments of the present disclosure;

[0064] FIG. 11 is a structural schematic diagram of a conversion network according to embodiments of the present disclosure;

[0065] FIG. 12 is a schematic diagram of a second neural network according to embodiments of the present disclosure;

[0066] FIGS. 13-14 are schematic diagrams of a conversion network according to embodiments of the present disclosure;

[0067] FIG. 15 is a flow diagram of an audio conversion method according to embodiments of the present disclosure.

[0068] It should be noted that the above-mentioned "first" and "second" are only used to distinguish different schemes, and do not represent the advantages or disadvantages of the schemes or the priority in the implementation process. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be described in further detail below with reference to the drawings, and the described embodiments should not be regarded as limiting the present application. All other embodiments obtained by a person of ordinary skill in the art without making creative efforts fall within the scope of protection of the present application.

[0070] In the following description, the terms "first\second" are merely to distinguish similar objects, and do not represent a specific order of the objects. It can be understood that the "first\second" can be interchanged in a specific order or sequence as allowed, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein.

[0071] In the following description, "some embodiments" are described, which describe a subset of all possible embodiments. It can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0072] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works with other related parts to achieve a predetermined target, and can be implemented entirely or partially by using software, hardware (such as a processing circuit or a memory) or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an integral module or unit that includes the functions of the module or unit.

[0073] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0074] Before the embodiments of the present application are further described in detail, the terms and phrases involved in the embodiments of the present application are explained, and the terms and phrases involved in the embodiments of the present application are applicable to the following explanations.

[0075] 1) Neural Network (NN): It is a kind of algorithm mathematical model that simulates the behavior characteristics of animal neural network and carries out distributed parallel information processing. This network relies on the complexity of the system to adjust the relationship between a large number of nodes connected to each other, so as to achieve the purpose of processing information.

[0076] 2) Deep Learning (DL): It is a new research direction in the field of Machine Learning (ML), and deep learning is to learn the internal rules and representation levels of sample data. The information obtained in these learning processes is very helpful for the interpretation of data such as text, images and sound. Its ultimate goal is to enable machines to have analysis and learning ability like people, and to recognize text, images and sound data.

[0077] 3) Quantization: refers to the process of approximating the continuous values (or a large number of discrete values) of a signal to a finite number (or fewer) of discrete values. Among them, quantization includes vector quantization (VQ, Vector Quantization) and scalar quantization.

[0078] Among them, vector quantization is an effective lossy compression technology, and its theoretical basis is Shannon's rate distortion theory. The basic principle of vector quantization is to replace the input vector with the index (also known as the quantization value) of the code word in the code book that best matches the input vector for transmission and storage, and only simple table lookup operation is needed for decoding. For example, a number of scalar data form a vector space, and the vector space is divided into a number of small regions. When quantizing, the vector falling into the small region is replaced by the corresponding index instead of the input vector.

[0079] Scalar quantization is the quantization of scalar, that is, one-dimensional vector quantization, which divides the dynamic range into a number of small intervals, each interval has a representative value (i.e. index). When the input signal falls into a certain interval, the input signal is quantized into the representative value.

[0080] 4) Entropy coding: a lossless coding method in the coding process according to the entropy principle without losing any information, which is also a key module in lossy coding and is at the end of the encoder. Entropy coding includes Shannon coding, Huffman coding, Exp-Golomb coding and arithmetic coding.

[0081] 5) Quadrature Mirror Filters (QMF): a filter pair containing analysis-synthesis, wherein the QMF analysis filter is used for sub-band signal decomposition to reduce the signal bandwidth so that each sub-band signal can be processed smoothly through the respective channel; the QMF synthesis filter is used to synthesize the sub-band signals recovered at the decoding end, such as through zero interpolation and bandpass filtering to reconstruct the original audio signal.

[0082] 6) Voice conversion: a process of changing the characteristics of an audio signal through technical means, thereby changing the timbre, pitch and formant characteristics of the sound. Voice conversion can include adjusting the frequency, amplitude, and time and frequency domain characteristics of the original audio signal to achieve voice pitch, tone or voice conversion effects. Voice conversion technology can achieve male-to-female and female-to-male voice conversion, as well as more unusual voice conversion effects such as child voice, electronic voice, etc. In some entertainment software, users can use voice conversion to increase the fun of voice communication. In professional applications, such as court network trials or online education, voice conversion technology can be used to protect the privacy of the speaker.

[0083] 7) Wideband: reflects the resolution or sampling rate of an audio signal. According to the definition of standard organizations such as International Telecommunication Union-Telecommunication Standardization Sector (ITU-T) and 3rd Generation Partnership Project (3GPP), the sampling rate of wideband is 16000 Hz, and the effective bandwidth is up to 8000 Hz. The transmission rate of wideband is usually above 1.54 Mbps (Megabits per second).

[0084] 8) Ultra-wideband: reflects the resolution or sampling rate of an audio signal. According to the definition of standard organizations such as ITU-T and 3GPP, the sampling rate of ultra-wideband is 32000 Hz, and the effective bandwidth is up to 16000 Hz. Ultra-wideband is a wireless communication technology. In general, ultra-wideband refers to an audio signal with a bandwidth of at least 500 MHz, or a ratio of audio signal bandwidth to center frequency exceeding 20%.

[0085] As shown in the system block diagram of voice conversion of the related art in FIG. 1, for the speech signal of the source speaker (i.e., source speaker signal), the content feature related to the content but irrelevant to the characteristics of the source speaker is obtained through the feature extraction network 101 analysis; at the same time, the speaker feature information of the target speaker (i.e., target speaker information) is obtained in advance. Then, the above two groups of features (i.e., content features and target speaker information) pass through an acoustic decoder 102 to obtain acoustic features. Finally, the acoustic features pass through a signal synthesis network 103 (i.e., vocoder) to synthesize the acoustic features into a speech signal (i.e., target speaker signal) consistent with the timbre of the target speaker. The content of the target speaker signal is consistent with the content of the speech signal of the source speaker, but the timbre is converted to the target speaker.

[0086] Voice conversion is a very popular application, such as digital people, voice changing software, etc. If real-time interaction is desired, as shown in FIG. 1, the target speaker signal also needs to be compressed and transmitted through the speech coding and decoding system. However, the above scheme has the following problems:

[0087] 1) The classic voice conversion is not a communication problem but a sound-like effect problem. It extracts the content information in the source speaker signal, then combines the features related to the "voiceprint" of the target speaker, generates acoustic features through the acoustic decoder, and finally synthesizes the waveform of the target speaker through the vocoder.

[0088] 2) Based on deep learning technology, it is essentially a waveform-to-waveform (wav2wav) mapping process with high complexity. The delay is high, even hundreds of milliseconds (ms), while the general real-time communication delay is tens of ms. In this way, the delay is too large, not a real "streaming voice change", which will also affect the experience of real-time communication scenarios.

[0089] 3) In terms of calculation, voice conversion and voice compression actually exist independently, with great redundancy. If voice conversion is embedded into a communication system, a level connection of an encoder system is also needed, which is very complex and has a large delay, making it difficult to deploy in real time, especially on mobile terminals.

[0090] To solve the above problems, the embodiments of the present application provide an audio communication method, an audio conversion method, a device, an electronic device, a computer readable storage medium and a computer program product. The following describes an exemplary application of the electronic device provided by the embodiments of the present application. The electronic device provided by the embodiments of the present application can be implemented as a terminal device, or can be implemented as a server, or can be implemented in cooperation with a terminal device and a server. The following describes an example in which the electronic device is implemented as a terminal device.

[0091] For example, referring to FIG. 2, FIG. 2 is an architecture schematic diagram of an audio codec system 10 provided by the embodiments of the present application. The audio codec system 10 includes a server 200, a network 300, a terminal device 400 (i.e., an encoding end) and a terminal device 500 (i.e., a decoding end). The network 300 can be a local area network, or a wide area network, or a combination of the two.

[0092] In some embodiments, a client 410 runs on the terminal device 400. The client 410 can be various types of clients, such as an instant messaging client, a network conference client, a live broadcast client, a browser, etc. The client 410 calls a microphone of the terminal device 400 to collect an audio signal in response to an audio collection instruction triggered by a sender (e.g., an initiator of a network conference, a host, an initiator of a voice call, etc.), and performs audio encoding processing on the collected audio signal to obtain a target audio bitstream.

[0093] For example, the client 410 performs audio encoding on the collected audio signal by calling the audio communication method provided by the embodiments of the present application, that is, in response to a communication request for the audio signal, a voice change mode for the audio signal is obtained from a plurality of communication modes; a feature encoding processing is performed on the audio signal to obtain an encoded feature of the audio signal; a target timbre corresponding to the voice change mode is obtained, and a timbre feature of the target timbre is determined; a timbre conversion processing is performed on the encoded feature of the audio signal based on the timbre feature of the target timbre to obtain a target encoded feature; a signal encoding processing is performed on the target encoded feature to obtain a target audio bitstream that conforms to the target timbre and has the same audio content as the audio signal.

[0094] The client 410 can send the target audio bitstream to the server 200 through the network 300, so that the server 200 sends the target audio bitstream to the terminal device 500 associated with the receiving party (for example, a participant of a network meeting, an audience, a receiver of a voice call, etc.).

[0095] After receiving the target audio bitstream sent by the server 200, the client 510 (for example, an instant messaging client, a network meeting client, a live broadcast client, a browser, etc.) running on the terminal device 500 can perform audio decoding on the target audio bitstream to obtain a reconstructed audio signal, thereby realizing audio communication.

[0096] For example, the client 510 calls the audio communication method provided by the embodiments of the present application to perform audio decoding on the received target audio bitstream, that is, to perform signal decoding processing on the target audio bitstream to obtain a target encoding feature estimation value corresponding to the target audio bitstream; and to perform feature decoding processing on the target encoding feature estimation value corresponding to the target audio bitstream to obtain a reconstructed audio signal that conforms to the target timbre and is the same as the audio content of the audio signal.

[0097] The audio communication method provided by the embodiments of the present application can be applied to various scenes requiring voice changing, such as game entertainment, social live broadcast, etc., which are illustrated below.

[0098] 1) Game entertainment, for example, in a multiplayer online game, a player generates his / her voice into a target audio bitstream conforming to a target timbre through the audio communication method, and sends the target audio bitstream to other players, and other players generate a reconstructed audio signal that conforms to the target timbre and is the same as the audio content of the voice through the audio communication method based on the target audio bitstream, so that players use voice changing technology to add fun to the game, and use different voices to perform role-playing during the game.

[0099] 2) Social live broadcast, for example, on a live broadcast platform, an anchor generates his / her voice into a target audio bitstream conforming to a target timbre through the audio communication method, and sends the target audio bitstream to an audience, and the audience generates a reconstructed audio signal that conforms to the target timbre and is the same as the audio content of the anchor's voice through the audio communication method based on the target audio bitstream, so that the anchor uses voice changing technology to increase the interactivity and fun of the live broadcast content.

[0100] For example, the server 200 shown in FIG. 2 can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and basic cloud computing services such as big data and artificial intelligence platforms. The terminal device 400 and the terminal device 500 shown in FIG. 2 can be smart phones, tablet computers, notebook computers, desktop computers, smart speakers, smart watches, vehicle-mounted terminals, and the like, but are not limited thereto. The terminal device (for example, the terminal device 400 and the terminal device 500) and the server 200 can be directly or indirectly connected through wired or wireless communication, which is not limited in the embodiments of the present application.

[0101] In some embodiments, the terminal device or the server 200 can also implement the audio communication method provided in the embodiments of the present application by running a computer program. For example, the computer program can be a native program or a software module in the operating system; can be a native application program (APP), that is, a program that needs to be installed in the operating system to run, such as a live broadcast APP, a network meeting APP, or an instant messaging APP; and can also be a small program that can be embedded into any APP. In summary, the above computer program can be any form of application program, module or plug-in.

[0102] Referring to FIG. 3A, FIG. 3A is a structural schematic diagram of the electronic device 500 provided in the embodiments of the present application. Taking the electronic device 500 as a terminal device for example, the electronic device 500 shown in FIG. 3A includes at least one processor 520, a memory 550, at least one network interface 530, and a user interface 540. The various components in the electronic device 500 are coupled together through a bus system 550. It can be understood that the bus system 550 is used to realize the connection and communication between the components. In addition to the data bus, the bus system 550 also includes a power bus, a control bus, and a status signal bus. However, for the purpose of clear illustration, all kinds of buses are marked as the bus system 550 in FIG. 3A.

[0103] The processor 520 can be an integrated circuit chip with signal processing capability, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., wherein the general-purpose processor can be a microprocessor or any conventional processor.

[0104] The memory 550 can be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, and the like. The memory 550 optionally includes one or more storage devices remotely located from the processor 520.

[0105] The memory 550 includes volatile memory or nonvolatile memory, and can also include both volatile and nonvolatile memory. Nonvolatile memory can be read only memory (ROM), volatile memory can be random access memory (RAM). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory.

[0106] In some embodiments, the memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, which are exemplarily illustrated below.

[0107] The operating system 551 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, and the like, for implementing various basic services and processing hardware-based tasks.

[0108] The network communication module 552 is used to communicate with other computing devices via one or more (wired or wireless) network interfaces 530, exemplary network interfaces 530 include Bluetooth, wireless compatibility certification (WiFi), and universal serial bus (USB), and the like.

[0109] In some embodiments, the audio communication device provided by the embodiments of the present application can be implemented in a software manner, and FIG. 3A shows an audio communication device 555 stored in the memory 550, which can be software in the form of programs and plug-ins, etc., including the following software modules: an acquisition module 5551, a feature encoding module 5552, a determination module 5553, a conversion module 5554, and an information encoding module 5555, wherein the acquisition module 5551, the feature encoding module 5552, the determination module 5553, the conversion module 5554, and the information encoding module 5555 are used to implement the audio encoding function, these modules are logical, and therefore can be combined or further split according to the implemented function.

[0110] Referring to FIG. 3B, FIG. 3B is a structural schematic diagram of an electronic device 600 provided by an embodiment of the present application. The electronic device 600 is taken as an example of a terminal device. The electronic device 600 shown in FIG. 3B includes at least one processor 620, a memory 650, at least one network interface 630, and a user interface 640. The various components in the electronic device 600 are coupled together by a bus system 650. The memory 650 includes an operating system 651, a network communication module 652. It should be noted that the functions of the structure in FIG. 3B are similar to the functions of the structure in FIG. 3A. The audio communication apparatus provided by an embodiment of the present application can be implemented in a software manner. FIG. 3B shows an audio communication apparatus 655 stored in the memory 650, which can be software in the form of programs and plug-ins, and includes the following software modules: a receiving module 6551, a signal decoding module 6552, and a feature decoding module 6553. The receiving module 6551, the signal decoding module 6552, and the feature decoding module 6553 are used to implement the audio decoding function. These modules are logical, and thus can be combined or further split according to the implemented function.

[0111] Referring to FIG. 3C, FIG. 3C is a structural schematic diagram of an electronic device 700 provided by an embodiment of the present application. The electronic device 700 is taken as an example of a terminal device. The electronic device 700 shown in FIG. 3B includes at least one processor 720, a memory 750, at least one network interface 730, and a user interface 740. The various components in the electronic device 700 are coupled together by a bus system 750. The memory 750 includes an operating system 751, a network communication module 752. It should be noted that the functions of the structure in FIG. 3C are similar to the functions of the structure in FIG. 3A. The audio conversion apparatus provided by an embodiment of the present application can be implemented in a software manner. FIG. 3C shows an audio conversion apparatus 755 stored in the memory 750, which can be software in the form of programs and plug-ins, and includes the following software modules: a feature encoding module 7551, a determination module 7552, a conversion module 7553, an information encoding module 7554, a signal decoding module 7555, and a feature decoding module 7556. The feature encoding module 7551, the determination module 7552, the conversion module 7553, the information encoding module 7554, the signal decoding module 7555, and the feature decoding module 7556 are used to implement the audio conversion function. These modules are logical, and thus can be combined or further split according to the implemented function.

[0112] Before specifically introducing the audio communication method provided by an embodiment of the present application, the empty convolution network is introduced.

[0113] Referring to FIG. 7A and FIG. 7B, FIG. 7A is a schematic diagram of a normal convolution (for example, causal convolution) network provided by an embodiment of the present application, and FIG. 7B is a schematic diagram of a dilated convolution network provided by an embodiment of the present application. Compared with the normal convolution network, the dilated convolution can increase the receptive field while keeping the size of the feature map unchanged, and can also avoid errors caused by upsampling and downsampling. Although the kernel size shown in FIG. 7A and FIG. 7B is 3x3; however, the receptive field 701 of the normal convolution shown in FIG. 7A is only 3, while the receptive field 702 of the dilated convolution shown in FIG. 7B reaches 5. That is, for a convolution kernel with a size of 3x3, the receptive field of the normal convolution shown in FIG. 7A is 3, and the dilation rate (the number of intervals of points in the convolution kernel) is 1; while the receptive field of the dilated convolution shown in FIG. 7B is 5, and the dilation rate is 2.

[0114] The convolution kernel can also move on a plane similar to FIG. 7A or FIG. 7B, which involves the concept of stride rate (step length). For example, the convolution kernel moves 1 grid each time, and the corresponding stride rate is 1.

[0115] In addition, there is the concept of the number of convolution channels, that is, how many parameters corresponding to the convolution kernel are used for convolution analysis. In theory, the more the number of channels, the more comprehensive the analysis of the signal and the higher the accuracy; however, the higher the channel, the higher the complexity. For example, a 1x320 tensor can use 24-channel convolution operation, and the output is a 24x320 tensor.

[0116] It should be noted that the size of the dilated convolution kernel (for example: for a speech signal, the size of the convolution kernel can be set to 1x3), the dilation rate, the stride rate and the number of channels can be defined by the actual application needs, and the embodiments of the present application do not make specific limitations.

[0117] As described above, the audio communication method provided by the embodiments of the present application can be implemented by various types of electronic devices, such as terminals, servers, or a combination of the two. Therefore, the execution subject of each step will not be repeated in the following description. Referring to FIG. 4A, FIG. 4A is a flowchart of an audio communication method provided by an embodiment of the present application. Taking the execution subject of the audio communication method shown in FIG. 4A as an encoding terminal as an example, the steps shown in FIG. 4A are described.

[0118] Before describing the following steps, two encoding methods (that is, the signal encoding method in signal processing technology and the feature encoding in artificial intelligence technology) involved in the audio communication method provided by the embodiments of the present application are explained. The feature encoding in artificial intelligence technology is described in detail in the following step 102, and the signal encoding method in signal processing technology is described in detail in the following step 105.

[0119] In step 101, a voice changing mode for the audio signal is acquired from a plurality of communication modes in response to a first communication request for the audio signal.

[0120] The first communication request is used to indicate that the audio signal is audio encoded in the voice changing mode. The plurality of communication modes includes a natural voice mode and the voice changing mode. The natural voice mode is used to indicate that the audio signal is directly audio encoded to obtain an audio stream conforming to a source timbre of the audio signal, and the audio stream is transmitted to a decoding terminal. The voice changing mode is used to indicate that the audio signal is voice changed in the process of audio encoding of the audio signal to obtain a voice changed audio stream (i.e., a target audio stream), and the voice changed audio stream is transmitted to the decoding terminal.

[0121] As an example of the voice changing mode, the encoding terminal acquires the audio signal (also referred to as an input signal) by calling a microphone of a terminal device of the encoding terminal in response to an audio acquisition instruction triggered by a sender (e.g., an initiator of a network conference, an anchor, an initiator of a voice call, etc.), and selects the voice changing mode. Then, a first communication request for the audio signal is generated, which is used to indicate that the audio signal is audio encoded in the voice changing mode. Then, the encoding terminal acquires the voice changing mode for the audio signal from the plurality of communication modes in response to the first communication request for the audio signal.

[0122] As an example of the natural voice mode, the encoding terminal acquires the audio signal (also referred to as an input signal) by calling a microphone of a terminal device of the encoding terminal in response to an audio acquisition instruction triggered by a sender (e.g., an initiator of a network conference, an anchor, an initiator of a voice call, etc.), and selects the natural voice mode. Then, a second communication request for the audio signal is generated, which is used to indicate that the audio signal is audio encoded in the natural voice mode. Then, the encoding terminal acquires the natural voice mode for the audio signal from the plurality of communication modes in response to the second communication request for the audio signal, encodes a feature of the encoding terminal based on the natural voice mode to obtain an audio stream of the audio signal (i.e., an audio stream conforming to a source timbre of the audio signal and being the same as an audio content of the audio signal), and transmits the audio stream to a decoding terminal.

[0123] To this end, the embodiments of the present application support the natural voice mode and the voice changing mode. A user can select to retain a source timbre or convert to a target timbre according to a requirement. In the voice changing mode, a target audio stream of different timbres can be generated in a case where audio encoding efficiency is guaranteed, streaming voice changing is implemented, and diversified target audio streams are obtained, thereby meeting actual application requirements of a user. In step 102, a feature of the audio signal is encoded to obtain an encoding feature of the audio signal.

[0124] The audio signal can be a wideband signal or an ultrawideband signal. When the audio signal is an ultrawideband signal, the ultrawideband signal can be first subband-decomposed to obtain a low-frequency subband signal and a high-frequency subband signal, and then the low-frequency subband signal and the high-frequency subband signal are respectively encoded to obtain low-frequency encoded features and high-frequency encoded features, both of which are encoded features of the audio signal. When the audio signal is a wideband signal, the wideband signal can be directly encoded to obtain the encoded features of the audio signal without subband decomposition. The data dimension of the encoded features is smaller than that of the audio signal.

[0125] The encoded features are representations extracted from the audio signal by encoding methods (such as neural networks and vocoders). These features can include information such as the content, timbre, and tone of the audio signal. The encoded features can include mel-frequency spectra, linear predictive coding (LPC) coefficients, and deep neural network-extracted hidden layer features.

[0126] In some embodiments, artificial intelligence technology can be used to directly extract the encoded features of the audio signal, that is, a first neural network is called to encode the features of the audio signal to obtain the encoded features of the audio signal, so as to subsequently generate a corresponding audio bitstream based on the encoded features of the audio signal. It should be noted that the embodiments of the present application are not limited to the structure of the first neural network, and the first neural network can be a convolutional neural network or a deep neural network.

[0127] In some embodiments, referring to FIG. 4B, which is a flowchart of an audio communication method according to an embodiment of the present application, step 102 in FIG. 4A can be implemented by steps 1021-1022.

[0128] In step 1021, the audio signal is feature-extracted to obtain audio features of the audio signal.

[0129] Here, the first neural network (NN) can be called based on the audio signal to extract the audio features from the audio signal by the first neural network, so as to subsequently continue feature refinement based on the audio features. It should be noted that the embodiments of the present application are not limited to the structure of the first NN, and the first NN can be a convolutional neural network or a deep neural network.

[0130] In some embodiments, step 1021 can be implemented by: performing causal convolution processing on the audio signal to obtain causal convolution features; and performing pooling processing on the causal convolution features to obtain the audio features of the audio signal.

[0131] In step 1022, at least one residual unit is used to perform residual processing on the audio features to obtain encoded features of the audio signal.

[0132] In a neural network model, a residual unit refers to a special structure used to construct a residual network (ResNet). The residual unit aims to solve the gradient vanishing and gradient explosion problems in the training process of a deep neural network, and help the network better learn features. The residual unit introduces a skip connection, that is, the input is directly added to the output instead of simply passing through the layers, and this skip connection enables the network to learn the residual function, that is, the difference between the input and the output, instead of directly learning the mapping relationship. This design makes the network easier to optimize, and also helps to alleviate the gradient vanishing problem.

[0133] Here, by performing residual processing on the audio feature on the encoding side, based on the characteristics of the residual processing, it is ensured that the audio feature is comprehensively learned, while the shallow feature information of the audio feature can be better utilized, and the shallow feature information of the audio feature is avoided to be missed.

[0134] Based on the characteristics of the residual unit, the residual processing in step 1022 is used to calculate the residual of the audio feature on the encoding side, and the residual of the audio feature is determined as the encoding feature for subsequent signal encoding. For example, the residual of the audio feature is obtained by adding the audio feature and the output of the residual unit, that is, the audio feature is taken as the input of the residual unit, the audio feature is processed through the residual unit to obtain the output of the residual unit, and the input of the residual unit and the output of the residual unit are added through the characteristics of the skip connection of the residual unit, so that the residual of the audio feature is obtained.

[0135] In some embodiments, referring to FIG. 4C, which is a flow diagram of an audio communication method provided by an embodiment of the present application, FIG. 4C shows that step 1022 in FIG. 4B can be implemented by steps 10221-10222.

[0136] In step 10221, the feature residual processing is performed on the audio feature by at least one residual unit to obtain the residual feature of the audio signal.

[0137] In step 10221, the feature residual processing is performed on the audio feature by at least one residual unit to obtain the residual feature of the audio signal.

[0138] In some embodiments, when the at least one residual unit is one residual unit, step 10221 can be implemented by the following manner: performing one residual processing on the audio feature by one residual unit to obtain the residual feature of the audio signal. In one residual processing of one residual unit, one residual corresponding to the audio signal is calculated on the encoding side.

[0139] In some embodiments, when the at least one residual unit is a plurality of cascaded residual units, step 10221 can be implemented by: performing, by a first residual unit of the plurality of cascaded residual units, one residual processing on the audio feature; outputting a residual result output by the first residual unit to a subsequent cascaded residual unit, and continuing to perform one residual processing and output of the residual result by the subsequent cascaded residual unit; and outputting a residual result output by a last residual unit as the residual feature of the audio signal.

[0140] In some embodiments, the processing procedure of the residual unit is as follows: performing, by a kth residual unit of the plurality of cascaded residual units, the following processing: performing convolution processing on an input of the kth residual unit to obtain a convolution result of the kth residual unit; and performing addition processing on the convolution result of the kth residual unit and the input of the kth residual unit to obtain a residual result output by the kth residual unit, where k is a positive integer that increases sequentially, 1≤k≤J, J is the number of residual units, when k is 1, the input of the kth residual unit is the audio feature, and when k is not 1, the input of the kth residual unit is a residual result output by a (k-1)th residual unit. That is, performing, by a first residual unit of the plurality of cascaded residual units, the following processing: performing convolution processing on the audio feature to obtain a convolution result of the first residual unit; and performing addition processing on the convolution result of the first residual unit and the audio feature to obtain a residual result output by the first residual unit. Continuing to perform residual processing and output of the residual result by the subsequent cascaded residual unit can be implemented by: performing, by a jth residual unit of the plurality of cascaded residual units, the following processing: performing convolution processing on a residual result output by a (j-1)th residual unit to obtain a convolution result of the jth residual unit; performing addition processing on the convolution result of the jth residual unit and the residual result output by the (j-1)th residual unit to obtain a residual result output by the jth residual unit; and outputting the residual result output by the jth residual unit to a (j+1)th residual unit, where j is a positive integer that increases sequentially, 1

[0141] In the foregoing embodiment, each residual unit comprises a dilated convolution operator; the following processing is performed by the kth residual unit of the plurality of cascaded residual units: performing convolution processing on the input of the kth residual unit to obtain a convolution result of the kth residual unit, which can be achieved by the following processing: performing dilated convolution processing on the input of the kth residual unit to obtain a convolution result of the kth residual unit. That is, the audio feature is subjected to dilated convolution processing by the dilated convolution operator comprised in the first residual unit to obtain a dilated convolution result of the first residual unit. The following processing is performed by the jth residual unit of the plurality of cascaded residual units: performing dilated convolution processing on the residual result output by the (j-1)th residual unit by the dilated convolution operator comprised in the jth residual unit to obtain a dilated convolution result of the jth residual unit, where j is a positive integer that increases sequentially, 1

[0142] In the foregoing embodiment, each residual unit comprises a dilated convolution operator; the following processing is performed by the kth residual unit of the plurality of cascaded residual units: performing convolution processing on the input of the kth residual unit to obtain a convolution result of the kth residual unit, which can be achieved by the following processing: performing dilated convolution processing on the input of the kth residual unit to obtain a convolution result of the kth residual unit. That is, the audio feature is subjected to dilated convolution processing by the dilated convolution operator comprised in the first residual unit to obtain a dilated convolution result of the first residual unit. The following processing is performed by the jth residual unit of the plurality of cascaded residual units: performing dilated convolution processing on the residual result output by the (j-1)th residual unit by the dilated convolution operator comprised in the jth residual unit to obtain a dilated convolution result of the jth residual unit, where j is a positive integer that increases sequentially, 1

[0143] In neural network models, causal convolution is a special type when dealing with time series data (audio signals are a kind of time series data), which can ensure that the output of the neural network network only depends on the current and previous time steps, thereby maintaining the causal relationship in time. In practical applications, causal convolution can be achieved by adjusting the size of the convolution kernel to ensure that the convolution kernel does not cross the area before the current time step. This can effectively capture long-term dependencies in time series while avoiding the problem of gradient vanishing or explosion caused by confusing future information. Causal convolution is particularly important in natural language processing, speech recognition, and time series prediction, because causal convolution follows the time order of data, avoids confusion of past information, and can effectively process and predict long time series data. In speech recognition, time series prediction and other tasks, causal convolution exhibits superior performance due to its ability to maintain time order.

[0144] In some embodiments, when applying grouped convolution to the residual unit including the causal convolution operator, the audio feature is processed by the causal convolution, which can be achieved by grouping the input channels of the audio feature to obtain a plurality of groups, wherein each group includes at least two channel corresponding first elements (i.e. first feature values) in the audio feature; and performing causal convolution on the first elements in each group. When applying grouped convolution to the residual unit including the causal convolution operator, the obtained result of the dilated convolution is processed by the causal convolution, which can be achieved by grouping the input channels of the dilated convolution result to obtain a plurality of groups, wherein each group includes at least two channel corresponding second elements (i.e. second feature values) in the dilated convolution result; and performing causal convolution on the second elements in each group.

[0145] For example, the grouped convolution can be applied to the convolutional operator of the residual unit (including the dilated convolutional operator and the causal convolutional operator), that is, the input channels are divided into multiple groups for convolution operation, and only the input channels and output channels in each group are associated. It should be noted that when the input channels are divided into multiple groups, the corresponding output channels are also divided into multiple groups, that is, the number of groups of input channels is the same as the number of groups of output channels, so that only the input channels and output channels in each group are associated after convolution in the group. Here, it is assumed that the input channels of the features input to a certain convolutional operator are 4, and the output channels are 4. If the number of groups is 1, each input channel is associated with 4 output channels. If the number of groups is 2, the 4 input channels are first divided into two groups 0-1 and 2-3, and in the two groups, the input channels are associated with the output channels in the group, for example, the input channels 0-1 in the first group are associated with the output channels 0-1, and the input channels 2-3 in the second group are associated with the output channels 2-3. As shown in FIG. 6A, when the grouped convolution scheme is not used, each input channel is associated with 4 output channels; as shown in FIG. 6B, when the grouped convolution scheme is not used, the 0th output channel is only associated with the 0-1 input channel and is not associated with the 2-3 input channel, and the 2nd output channel is only associated with the 2-3 input channel and is not associated with the 0-1 input channel. By comparison, it can be seen that the introduction of the grouped convolution can avoid the association of any input channel with all output channels, reduce the number of connections, and reduce the complexity.

[0146] In step 10221, the residual feature is encoded to obtain the encoded feature of the audio signal.

[0147] Here, the residual feature is encoded to obtain the encoded feature of the audio signal, so that subsequent signal encoding processing is performed based on the encoded feature to obtain the audio bitstream of the audio signal.

[0148] In some embodiments, step 10222 can be implemented by: performing convolution processing on the residual feature to obtain a convolution feature, wherein the number of channels of the convolution feature is greater than the number of channels of the residual feature; and performing pooling processing on the convolution feature to obtain the encoded feature of the audio signal.

[0149] In some embodiments, the third neural network for audio encoding includes a plurality of cascaded encoding blocks, each encoding block including at least one residual unit and a feature encoding block; step 10222 is implemented through the plurality of cascaded encoding blocks, and step 10222 can be implemented by: performing residual processing on the audio feature through at least one residual unit in the plurality of cascaded encoding blocks to obtain residual features of the audio signal; and performing feature encoding processing on the residual features through the feature encoding block in the plurality of cascaded encoding blocks to obtain encoded features of the audio signal.

[0150] In some embodiments, performing residual processing on the audio feature through at least one residual unit in the plurality of cascaded encoding blocks to obtain residual features of the audio signal can be implemented by: performing residual processing on the audio feature through at least one residual unit in a first encoding block of the plurality of cascaded encoding blocks, and outputting residual results output by the at least one residual unit in the first encoding block to the feature encoding block in the first encoding block; performing residual processing on the encoding results output by the feature encoding block in the i-1th encoding block through at least one residual unit in the ith encoding block of the plurality of cascaded encoding blocks, and outputting residual results output by the at least one residual unit in the ith encoding block to the feature encoding block in the ith encoding block; and outputting the residual results output by the at least one residual unit in the last encoding block as the residual features of the audio signal; wherein i is a positive integer that increases sequentially, 1 < i ≤ I, and I is the number of encoding blocks. Performing feature encoding processing on the residual features through the feature encoding block in the plurality of cascaded encoding blocks to obtain encoded features of the audio signal can be implemented by: performing feature encoding processing on the residual features through the feature encoding block in the last encoding block of the plurality of cascaded encoding blocks to obtain the encoded features of the audio signal. The encoded features are obtained by performing the following processing on the residual features through the last encoding block of the plurality of cascaded encoding blocks: performing convolution processing on the residual features to obtain convolution features, wherein the number of channels of the convolution features is greater than the number of channels of the residual features; and performing pooling processing on the convolution features to obtain the encoded features of the audio signal.

[0151] In step 10222, the processing process of the residual unit is similar to the processing process of the residual unit described above, and will not be described here.

[0152] In step 103, the target timbre corresponding to the voice changing mode is obtained, and the timbre features of the target timbre are determined.

[0153] For example, when the first communication request is used to indicate that the audio signal is audio encoded in a voice mode with a target timbre, and the plurality of communication modes include a natural voice mode and a voice mode with different timbres, the voice mode with the target timbre is obtained from the plurality of communication modes in response to the first communication request, so as to obtain the target timbre corresponding to the voice mode. The audio after voice conversion in the voice mode with different timbres has different timbres. For example, the voice mode of a loli corresponds to a loli timbre, and the voice mode of an uncle corresponds to an uncle timbre. As an example of obtaining the target timbre corresponding to the voice mode, the user can select the voice mode from the plurality of communication modes displayed in the encoding terminal, and select the target timbre from the plurality of timbres corresponding to the voice mode, so as to determine the voice mode with the target timbre selected by the user.

[0154] The timbre features of the target timbre are used to reflect the timbre of the target speaker, describe the key attributes of the target timbre, such as spectral envelope, formant, fundamental frequency, etc., and these features can distinguish the voice characteristics of different speakers and are the basis for timbre conversion. The dimension of the timbre features of the target timbre can be multi-dimensional, and the embodiments of the present application are not limited to the dimension of the timbre features. As for the generation process of the timbre features, in traditional speech signal processing, there are many feature extraction methods to extract the timbre information of the user. The timbre features of the target timbre include but are not limited to F0 fundamental frequency information related features, loudness features, mel-frequency cepstral coefficient (MFCC) spectrum information, etc. The feature information in the timbre features of the target timbre can be extracted by frame, and then the mean, variance and other statistical values of several seconds of data are calculated, so that the timbre features of the target timbre are expressed by multi-dimension. In the training data, for any target speaker, multiple files can be extracted, and the multi-dimensional features corresponding to each file are calculated; then, the above multi-dimensional features are averaged in each dimension to obtain the timbre features. Each target speaker corresponds to a multi-dimensional feature, which is identified by an identification (ID) index.

[0155] For example, when the first communication request is used to indicate that the audio signal is audio encoded in a voice mode with a target timbre, and the plurality of communication modes include a natural voice mode and a voice mode with different timbres, the voice mode with the target timbre is obtained from the plurality of communication modes in response to the first communication request, so as to obtain the target timbre corresponding to the voice mode. The audio after voice conversion in the voice mode with different timbres has different timbres. For example, the voice mode of a loli corresponds to a loli timbre, and the voice mode of an uncle corresponds to an uncle timbre. As an example of obtaining the target timbre corresponding to the voice mode, the user can select the voice mode from the plurality of communication modes displayed in the encoding terminal, and select the target timbre from the plurality of timbres corresponding to the voice mode, so as to determine the voice mode with the target timbre selected by the user.

[0156] In some embodiments, the voice transformation mode has multiple candidate timbres, and the target timbre corresponding to the voice transformation mode in step 103 can be determined from the multiple candidate timbres. In this way, the user can select a target timbre that meets his or her needs from the multiple candidate timbres, meeting the personalized needs, enhancing the user's sense of participation and immersion through diversified timbre selection, automatically recommending or selecting a target timbre according to the application scenario, and improving the applicability of timbre conversion.

[0157] As an example of determining a target timbre from multiple candidate timbres, multiple candidate timbres are displayed in the encoding terminal, and a selected candidate timbre is taken as the target timbre in response to a selection operation on the candidate timbres. For example, multiple candidate timbres (such as loli voice, uncle voice, child voice, etc.) are displayed in the display interface of the encoding terminal, and when the user selects loli voice, the selected loli voice is taken as the target timbre.

[0158] As an example of determining a target timbre from multiple candidate timbres, after multiple candidate timbres are displayed in the encoding terminal, the encoding terminal takes a default candidate timbre as the target timbre if no selection operation on the candidate timbres is received within a set time, where the default candidate timbre is a pre-set default timbre. For example, multiple candidate timbres (such as loli voice, uncle voice, child voice, etc.) are displayed in the display interface of the encoding terminal, and when the user does not select a candidate timbre within a set time, a default candidate timbre (such as loli voice) is taken as the target timbre.

[0159] As an example of determining a target timbre from multiple candidate timbres, multiple candidate timbres are displayed in the encoding terminal, and the encoding terminal automatically determines a target timbre that matches the audio signal from the multiple candidate timbres. For example, multiple candidate timbres (such as loli voice, uncle voice, child voice, female voice, etc.) are displayed in the display interface of the encoding terminal, and when the source timbre of the audio signal is detected to be female voice, the loli voice that matches the audio signal is taken as the target timbre; the similarity between the source timbre of the audio signal and each candidate timbre is detected, and the candidate timbre with the highest similarity is taken as the target timbre.

[0160] In step 104, the encoding features are timbre-converted based on the timbre features to obtain target encoding features.

[0161] The feature dimension of the target encoding features is the same as the feature dimension of the encoding features of the audio signal. The target encoding features are encoding features obtained after timbre conversion, and the timbre of the target encoding features is consistent with the target timbre and retains the content and semantic information of the audio signal.

[0162] Here, the timbre conversion in step 104 is used to convert the encoding feature of the audio signal by using the timbre feature (i.e., the feature related to the timbre), so that the converted encoding feature has the characteristics of the target timbre, thereby obtaining the encoding feature conforming to the target timbre (i.e., the target encoding feature), that is, the source timbre in the encoding feature is converted to the target timbre while the content and semantic information of the audio signal are preserved. For example, the process of the timbre conversion can be: first, separating the timbre-related feature and the content-related feature from the encoding feature; then, converting the timbre part in the encoding feature by using the timbre feature of the target timbre; and finally, recombining the converted timbre feature in the encoding feature and the content-related feature to generate the target encoding feature.

[0163] In some embodiments, referring to FIG. 4D, which is a flow diagram of an audio communication method provided by the embodiments of the present application, FIG. 4D shows that step 104 in FIG. 4A can be implemented by steps 1041-1042.

[0164] In step 1041, the timbre feature and the encoding feature are merged to obtain a merged feature.

[0165] In the embodiments of the present application, the manner of merging the timbre feature and the encoding feature is not limited, for example, the timbre feature e (ID) (n) is 88-dimensional, and the encoding feature F(n) of the audio signal is 56-dimensional, F(n) and e (ID) (n) are merged into a 144x1 merged feature, where the first 88 dimensions of the 144x1 merged feature are the timbre feature, and the last 56 dimensions are the encoding feature; or the first 56 dimensions of the 144x1 merged feature are the encoding feature, and the last 88 dimensions are the timbre feature; or the encoding feature and the timbre feature in the 144x1 merged feature are alternated, for example, the first 44 dimensions of the 144x1 merged feature are 44-dimensional timbre features, then the 44-dimensional timbre features are followed by 28-dimensional encoding features, then the 28-dimensional encoding features are followed by the remaining 44-dimensional timbre features, and finally the remaining 44-dimensional timbre features are followed by the remaining 28-dimensional encoding features. It can be understood that the dimensions of each feature in the embodiments of the present application are not limited to 56, 144 or 88, but can be any number of dimensions.

[0166] After step 1041, in step 1042, the feature conversion is performed on the merged feature to obtain the target encoding feature.

[0167] The target coding feature is consistent with the target timbre and the same as the audio content of the audio signal, that is, the target coding feature and the coding feature are different in timbre. In this application embodiment, the conversion network is used to convert the merged feature to obtain the target coding feature. In this application embodiment, the network structure of the conversion network is not limited. Here, the feature conversion in step 1042 is used to convert the feature of the source timbre part in the merged feature by using the timbre feature in the merged feature, so that the converted coding feature has the characteristics of the target timbre, so as to obtain the coding feature (i.e., the target coding feature) consistent with the target timbre, that is, the source timbre in the coding feature is converted into the target timbre, while the content and semantic information of the audio signal are preserved.

[0168] In this way, in step 1041, the coding feature and the timbre feature are explicitly merged, and after the merged feature, these information are preserved in the feature conversion process, ensuring that the timbre information is fully preserved and utilized in the feature conversion process, and further ensuring that the generated audio is consistent with the source audio signal after the timbre conversion; in step 1042, the mapping relationship between the source timbre and the target timbre in the merged feature can be more accurately learned, thereby improving the accuracy of the timbre conversion.

[0169] In some embodiments, referring to FIG. 4E, which is a flow diagram of an audio communication method provided by an embodiment of the present application, the conversion network includes a plurality of first convolutional layers and a plurality of conversion encoding blocks, and the first convolutional layers and the conversion encoding blocks are alternately connected (as shown in FIG. 13, the first convolutional layer 311 is followed by the first conversion encoding block 321, the first conversion encoding block 321 is followed by the second first convolutional layer 312, the second first convolutional layer 312 is followed by the second conversion encoding block 322, and so on). The first convolutional layer is used to change the number of channels of the feature input to the first convolutional layer, and the conversion encoding block is used to maintain the number of channels of the feature input to the conversion encoding block. FIG. 4E shows that step 1042 in FIG. 4D can be implemented by steps 10421-10423.

[0170] In step 10421, the merged feature is convolved by the first first convolutional layer, and the convolution result of the first first convolutional layer is converted and encoded by the first conversion encoding block alternately connected with the first first convolutional layer to obtain the encoding result of the first conversion encoding block.

[0171] In some embodiments, the first convolutional layer can be a causal convolutional layer, and the merged feature is convolved by the first first convolutional layer to obtain the convolution result of the first first convolutional layer. In this application embodiment, the form of the first convolutional layer is not limited to a causal convolutional layer, but can also be a hollow convolution, a one-dimensional convolution, etc.

[0172] In some embodiments, referring to FIG. 4F, which is a flowchart of an audio communication method according to an embodiment of the present application, each conversion encoding block includes a second convolution layer and at least one residual unit; and step 10421 in FIG. 4E can be implemented by steps 104211-104212.

[0173] In step 104211, the convolution result of the obtained first convolution layer is subjected to residual processing by the at least one residual unit in the first conversion encoding block, to obtain residual features of the first conversion encoding block.

[0174] Here, the processing procedure of the residual unit in the conversion encoding block is similar to the processing procedure of the residual unit in the feature encoding described above.

[0175] Here, based on the characteristics of the residual unit, the residual processing in step 104211 is used to calculate the residual of the convolution result of the first convolution layer at the encoding side, and determine the residual of the convolution result of the first convolution layer as the residual features of the first conversion encoding block, for subsequent processing. For example, the residual of the convolution result of the first convolution layer is obtained by adding the convolution result of the first convolution layer and the output of the at least one residual unit in the first conversion encoding block, that is, the convolution result of the first convolution layer is taken as the input of the residual unit, the convolution result of the first convolution layer is processed by the residual unit to obtain the output of the residual unit, and the input of the residual unit and the output of the residual unit are added through the characteristics of the skip connection of the residual unit, to obtain the residual of the convolution result of the first convolution layer.

[0176] In some embodiments, when the number of the at least one residual unit in the first conversion encoding block is one, the convolution result of the obtained first convolution layer is subjected to residual processing once by the residual unit in the first conversion encoding block, to obtain the residual features of the first conversion encoding block.

[0177] In some embodiments, when the at least one residual unit is a plurality of cascaded residual units, step 104221 can be implemented by the following manner: the convolution result of the obtained first convolution layer is subjected to residual processing by a first residual unit of the plurality of cascaded residual units; the residual result output by the first residual unit is output to the subsequent cascaded residual units, and the residual processing and the output of the residual result are continued by the subsequent cascaded residual units; and the residual result output by the last residual unit is taken as the residual features of the first conversion encoding block.

[0178] Thus, the embodiments of the present application can construct a deeper network structure through the cascaded residual units, gradually extract and convert features, capture more complex patterns and relationships, and in the timbre conversion task, the structure can more accurately learn the mapping relationship between the source timbre and the target timbre. Moreover, the cascaded residual units can realize efficient feature conversion at a lower computational cost, improve the efficiency of timbre conversion, and are suitable for real-time timbre conversion tasks.

[0179] In some embodiments, the residual processing of the obtained convolution result of the first convolution layer by the first residual unit of the plurality of cascaded residual units can be implemented in the following manner: the first residual unit of the plurality of cascaded residual units performs the following processing: performing convolution processing on the obtained convolution result of the first convolution layer to obtain a convolution result of the first residual unit; and performing addition processing on the convolution result of the first residual unit and the convolution result of the first convolution layer to obtain a residual result output by the first residual unit. The residual processing and the output of the residual result by the subsequent cascaded residual units can be implemented in the following manner: the jth residual unit of the plurality of cascaded residual units performs the following processing: performing convolution processing on the residual result output by the (j-1)th residual unit to obtain a convolution result of the jth residual unit; performing addition processing on the convolution result of the jth residual unit and the residual result output by the (j-1)th residual unit to obtain a residual result output by the jth residual unit; and outputting the residual result output by the jth residual unit to the (j+1)th residual unit, where j is a positive integer that increases sequentially, 1

[0180] Thus, the embodiments of the present application ensure that the detailed information of the features input to the residual unit is retained at each layer through the skip connection of the residual unit, avoiding the loss of important information in the depth conversion process. In the timbre conversion task, the structure can ensure the integrity and naturalness of the speech content.

[0181] wherein each residual unit comprises a dilated convolution operator and a causal convolution operator; the convolution processing of the obtained convolution result of the first convolution layer to obtain a convolution result of the first residual unit can be implemented in the following manner: performing dilated convolution processing on the obtained convolution result of the first convolution layer by the dilated convolution operator included in the first residual unit; and performing causal convolution processing on the obtained dilated convolution result by at least one causal convolution operator included in the first residual unit, and taking the obtained causal convolution result as the convolution result of the first residual unit. The processing manner of the jth residual unit is similar to that of the first residual unit, and only the processing object is different. The processing object of the first residual unit is the convolution result of the first convolution layer, and the processing object of the jth residual unit is the residual result output by the (j-1)th residual unit.

[0182] Of course, the residual unit in the conversion encoding block can also apply grouped convolution, similar to the grouped convolution in step 102, which will not be repeated here.

[0183] After step 104211, in step 104212, the residual features of the first conversion encoding block are convoluted by the second convolution layer in the first conversion encoding block to obtain the encoding result of the first conversion encoding block.

[0184] In some embodiments, the second convolution layer is used to maintain the number of channels of the residual features unchanged, that is, the encoding result of the first conversion encoding block generated after the residual features of the first conversion encoding block pass through the second convolution layer has the same number of channels as the residual features of the first conversion encoding block.

[0185] In some embodiments, the second convolution layer can be a causal convolution layer, and the residual features of the first conversion encoding block are convoluted by the second convolution layer in the first conversion encoding block to obtain the encoding result of the first conversion encoding block. The embodiments of the present application are not limited to the form of the second convolution layer, which can also be a causal convolution layer, a hollow convolution, a one-dimensional convolution, etc.

[0186] In this way, the embodiments of the present application realize the feature residual function through the residual unit in step 104211, retain the detailed information of the features input to the residual unit, and in the timbre conversion task, this structure can ensure the integrity and naturalness of the voice content; the second convolution layer in step 104212 is used to convolute the input residual features, further extract and convert the features, and obtain the encoding result of the conversion encoding block. The local features are captured through the convolution operation of the second convolution layer, the expression ability of the features is enhanced, and thus more complex timbre features can be captured, and the conversion accuracy is improved.

[0187] In step 10422, the encoding result of the (i-1)th conversion encoding block is convoluted by the ith first convolution layer connected alternately with the (i-1)th conversion encoding block, and the convolution result of the ith first convolution layer is converted and encoded by the ith conversion encoding block connected alternately with the ith first convolution layer to obtain the encoding result of the ith conversion encoding block, where i is a positive integer that increases sequentially, 1

[0188] The processing procedure of step 10422 is similar to that of step 10421, and only the processing objects are different.

[0189] After step 10422, in step 10423, the convolution result output by the last first convolution layer is taken as the target encoding feature, or the encoding result of the last conversion encoding block is taken as the target encoding feature.

[0190] Here, when the number of the first convolutional layers is the same as the number of the conversion encoding blocks, the encoding result of the last conversion encoding block is taken as the target encoding feature; when the number of the first convolutional layers is greater than the number of the conversion encoding blocks, the convolution result output by the last first convolutional layer is taken as the target encoding feature.

[0191] In this way, the embodiment of the present application realizes the alternative connection of the first convolutional layers and the conversion encoding blocks through steps 10421-10423, forming a multi-level feature processing structure. This connection mode can make full use of feature information at different levels and enhance the expression ability of the conversion network. The first convolutional layers extract local features to provide a basis for subsequent processing, and the conversion encoding blocks gradually perform deep conversion on the features to capture higher-level abstract features. This multi-level feature processing structure gradually refines the features to realize accurate mapping from the source timbre to the target timbre, can effectively process complex timbre conversion tasks, and improve the naturalness and realism of the conversion.

[0192] In some embodiments, as shown in FIG. 14, the conversion network includes a plurality of first convolutional layers and a plurality of conversion encoding blocks, the first convolutional layers and the conversion encoding blocks are alternatively connected, the first conversion encoding block 421 is connected after the first first convolutional layer 411, the second first convolutional layer 412 is connected after the first conversion encoding block 421, the second conversion encoding block 422 is connected after the second first convolutional layer 412, and so on. Each first convolutional layer includes one or more cascaded convolution operators, and each conversion encoding block includes a plurality of cascaded conversion encoding sub-blocks. The processing mode of the conversion encoding sub-blocks is similar to that of the conversion encoding blocks in FIG. 13. Step 10423 can be realized through the following steps: performing convolution processing on the merged features through the first first convolutional layer, performing conversion encoding processing on the convolution result of the first first convolutional layer through the first conversion encoding block which is alternatively connected with the first first convolutional layer, to obtain the encoding result of the first conversion encoding block; performing convolution processing on the encoding result of the i-1th conversion encoding block through the i-th first convolutional layer which is alternatively connected with the i-1th conversion encoding block, and performing conversion encoding processing on the convolution result of the i-th first convolutional layer through the i-th conversion encoding block which is alternatively connected with the i-th first convolutional layer, to obtain the encoding result of the i-th conversion encoding block, where i is a positive integer that increases sequentially, 1 < i ≤ I, and I is the number of the plurality of first convolutional layers; the convolution result output by the last first convolutional layer is taken as the target encoding feature, or the encoding result of the last conversion encoding block is taken as the target encoding feature.

[0193] The convolution operator in the first convolution layer shown in FIG. 14 can be a causal convolution operator. Taking the first convolution layer as an example, the merged features are subjected to cascaded causal convolution by the cascaded causal convolution operators in the first convolution layer, to obtain the convolution result of the first convolution layer.

[0194] Taking the first conversion coding block including multiple cascaded conversion coding sub-blocks as an example, the convolution result of the first convolution layer is subjected to cascaded conversion coding processing by the multiple cascaded conversion coding sub-blocks, to obtain the coding result of the first conversion coding block. As shown in FIG. 14, taking the number of conversion coding sub-blocks in each conversion coding block as 2 as an example, the convolution result of the first convolution layer is subjected to conversion coding processing by the first conversion coding sub-block 4211 in the first conversion coding block 421, to obtain the coding result of the first conversion coding sub-block, the coding result of the first conversion coding sub-block is subjected to conversion coding processing by the second conversion coding sub-block 4212 in the first conversion coding block 421, to obtain the coding result of the second conversion coding sub-block, the coding result of the second conversion coding sub-block is taken as the coding result of the first conversion coding block, and is output to the second convolution layer.

[0195] As shown in FIG. 14, each conversion coding sub-block includes a second convolution layer and at least one residual unit. Taking the first conversion coding sub-block as an example, the first conversion coding sub-block 4211 includes a second convolution layer and at least one residual unit. The convolution result of the first convolution layer is subjected to conversion coding processing by the first conversion coding sub-block 4211 in the first conversion coding block 421, to obtain the coding result of the first conversion coding sub-block, which can be achieved by the following manner: the obtained convolution result of the first convolution layer is subjected to residual processing by the at least one residual unit in the first conversion coding sub-block 4211, to obtain the residual features of the first conversion coding sub-block, wherein the processing process of the residual unit in the conversion coding sub-block is similar to that of the residual unit in the feature coding; the residual features of the first conversion coding sub-block are subjected to convolution processing by the second convolution layer in the first conversion coding sub-block, to obtain the coding result of the first conversion coding sub-block, wherein the processing process of the second convolution layer in the conversion coding sub-block is similar to that of the second convolution layer in the feature coding.

[0196] After the receiving step 104, the target coding features are subjected to signal coding in step 105, to obtain the target audio code stream conforming to the target timbre, and the target audio code stream is transmitted to the decoding terminal.

[0197] Here, the signal coding in step 105 is to subject the target coding features to signal coding by a conventional signal coding manner in the field of digital signal processing, to obtain the target audio code stream conforming to the target timbre.

[0198] In some embodiments, step 105 can be implemented by: quantizing the target coding feature to obtain an index value of the target coding feature; and performing entropy coding on the index value of the target coding feature to obtain the target audio bitstream that conforms to the target timbre and is identical to the audio content of the audio signal. The entropy coding is a lossless compression technique for further reducing the data volume of the index value, and the embodiments of the present application are not limited to the form, for example, Huffman Coding, Arithmetic Coding, Shannon-Fano Coding, etc.

[0199] As an example, the target coding feature can be subjected to scalar quantization (each component is quantized separately) and entropy coding. In addition, the embodiments of the present application are not limited to the combination of vector quantization (adjacent multiple components are combined into a vector for joint quantization) and entropy coding.

[0200] In some embodiments, the audio communication method is implemented through a second encoding network in an audio codec system, the audio codec system further comprising a first encoding network and a first decoding network, the first encoding network and the first decoding network being trained through first audio samples; the second encoding network is trained through the following manner: in response to a configuration request for the first encoding network, determining a second encoding network to be trained corresponding to the first encoding network, wherein the second encoding network to be trained comprises a conversion network to be trained; performing audio encoding on second audio samples based on the conversion network to be trained to obtain audio bitstream samples of the second audio samples, and performing audio decoding on the audio bitstream samples based on the first decoding network to obtain reconstructed audio samples of the second audio samples; updating parameters of the conversion network to be trained based on the reconstructed audio samples to obtain a trained conversion network.

[0201] In this way, since the parameters of the first decoding network are unchanged and only the parameters of the conversion network to be trained in the second encoding network to be trained are updated when training the audio codec system, the training process for the first decoding network is saved, and compared with the related art of simultaneously training the first decoding network and the second encoding network to be trained, the training period of the audio codec system is shortened, thereby reducing the upgrade cost of the conversion network in the audio codec system to meet the actual application requirements of users.

[0202] As one example, the first encoding network includes a first neural network, and the second encoding network to be trained includes a transformation network to be trained and the first neural network; as another example, the first encoding network includes a first neural network, and the second encoding network to be trained includes a transformation network to be trained and a neural network similar to the first neural network; as yet another example, the first encoding network includes a first neural network and a pre-trained old transformation network, and the second encoding network to be trained includes a transformation network to be trained and the first neural network.

[0203] In some embodiments, the second audio sample is encoded based on the second encoding network to be trained to obtain an audio bitstream sample of the second audio sample. This can be achieved by: encoding the second audio sample using a first neural network included in the second encoding network to be trained to obtain the encoding features of the second audio sample; obtaining the timbre sample corresponding to the trained conversion network and determining the timbre features of the timbre sample; performing timbre conversion on the encoding features of the second audio sample based on the timbre features of the timbre sample to obtain an encoding feature sample; and performing signal encoding on the target encoding feature sample to obtain an audio bitstream sample that conforms to the timbre sample.

[0204] The process of encoding the second audio sample is similar to the process of encoding the audio signal, and will not be described in detail here.

[0205] It should be noted that before applying the second encoding network, the second encoding network to be trained needs to be trained, and then the trained second encoding network is put into application. For example, the parameters of the transformation network to be trained are updated based on the reconstructed audio samples to obtain the trained transformation network. This can be achieved in the following way: After determining the value of the loss function of the transformation network based on the reconstructed audio samples and the second audio samples, it can be determined whether the value of the loss function exceeds a preset threshold. When the value of the loss function exceeds the preset threshold, the error signal of the transformation network is determined based on the loss function, and the error information is backpropagated in the transformation network, updating the model parameters of each layer during the propagation process. The embodiments of this application are not limited to the form of the loss function; for example, it can be a cross-entropy loss function, an L2 loss function, etc.

[0206] Here, the back propagation is explained. The training sample data is input to the input layer of the neural network model, passes through the hidden layer, and finally reaches the output layer and outputs the result, which is the forward propagation process of the neural network model. Since there is an error between the output result of the neural network model and the actual result, the error between the output result and the actual value is calculated, and the error is back propagated from the output layer to the hidden layer until it propagates to the input layer. In the process of back propagation, the value of the model parameter is adjusted according to the error, that is, the loss function is constructed according to the error between the output result and the actual value, and the partial derivative of the loss function with respect to the model parameter is calculated layer by layer to generate the gradient of the loss function with respect to each layer model parameter. Since the direction of the gradient indicates the direction of the error expansion, the gradient of the model parameter is taken as the opposite, and the sum of the original parameters of each layer model is taken as the updated parameters of each layer model, so as to reduce the error caused by the model parameters. The above process is iterated until convergence. The conversion network is a kind of neural network model.

[0207] As described above, the audio communication method provided by the embodiments of the present application can be implemented by various types of electronic devices, such as terminals, servers, or a combination of the two. Therefore, the execution subject of each step will not be repeated in the following description. Referring to FIG. 5A, FIG. 5A is a flow diagram of an audio communication method provided by an embodiment of the present application. Taking the decoding terminal as an example, the execution subject of the audio communication method shown in FIG. 5A is described in conjunction with the steps shown in FIG. 5A.

[0208] Before describing the following steps, two decoding methods involved in the audio communication method provided by the embodiments of the present application are explained (i.e., the signal decoding method in signal processing technology and the feature decoding (i.e., the reconstruction method) in artificial intelligence technology). The signal decoding method in signal processing technology is described in detail in step 202 below, and the feature decoding in artificial intelligence technology is described in detail in step 203 below.

[0209] In step 201, a target audio bitstream transmitted by an encoding terminal is received.

[0210] The target audio bitstream is obtained by performing timbre conversion on the target voice mode corresponding to the target timbre.

[0211] In step 202, the target audio bitstream is signal decoded to obtain a target encoding feature estimation value corresponding to the target audio bitstream.

[0212] It should be noted that signal decoding is the inverse process of signal encoding. Therefore, the value generated in the decoding process is an estimated value relative to the value in the encoding process. For example, the target encoding feature generated in the decoding process is an estimated value relative to the encoding feature in the encoding process.

[0213] For example, the signal decoding of the target audio bitstream can be implemented by: performing entropy decoding on the target audio bitstream to obtain an index value corresponding to the target audio bitstream; and performing inverse quantization on the index value corresponding to the target audio bitstream to obtain a target coding feature estimation value corresponding to the target audio bitstream.

[0214] The inverse quantization is implemented by querying a quantization table, which is a mapping table generated in the coding process. As an example, for a received bitstream, entropy decoding is first performed, and the estimation value of the feature vector, i.e., the coding feature estimation value corresponding to the audio bitstream, is obtained by querying the quantization table (i.e., inverse quantization, the quantization table is a mapping table generated in the coding process).

[0215] It should be noted that signal decoding is the inverse process of signal coding. Since the decoding process of the received bitstream at the decoding end is the inverse process of the coding process at the encoding end, the value generated in the decoding process is an estimation value relative to the value in the coding process, for example, the data value representing the target coding feature generated in the decoding process is an estimation value relative to the target coding feature in the coding process. That is, the former is not necessarily equal to the data value of the original target coding feature. For example, there may be a slight difference between the two due to the coding and decoding operation process, and there may be a situation where the original target coding feature data value cannot be completely restored by decoding. Therefore, the former is also referred to as a "target coding feature estimation value".

[0216] In step 203, the target coding feature estimation value is feature decoded to obtain a reconstructed audio signal conforming to the target tone.

[0217] It should be noted that the feature decoding in step 203 is the inverse process of the feature encoding in step 102. Therefore, the value generated in the feature decoding process is an estimation value relative to the value in the feature encoding process, that is, the reconstructed audio signal is not necessarily equal to the original audio signal, and there may be a slight difference between the two due to the coding and decoding operation process, and there may be a situation where the original audio signal cannot be completely restored by decoding.

[0218] In some embodiments, referring to FIG. 5B, which is a flow diagram of an audio communication method provided by an embodiment of the present application, FIG. 5B shows that step 203 in FIG. 5A can be implemented by steps 2031-2032.

[0219] In step 2031, at least one residual unit is used to perform residual processing on the target coding feature estimation value corresponding to the target audio bitstream to obtain an audio feature estimation value corresponding to the target audio bitstream. The number of residual units is not limited in the embodiments of the present application.

[0220] In some embodiments, referring to FIG. 5C, which is a flowchart of an audio communication method according to an embodiment of the present application, FIG. 5C shows that step 2031 in FIG. 5B can be implemented by steps 20311-20312.

[0221] In step 20311, the target coding feature estimate corresponding to the target audio bitstream is subjected to feature decoding processing to obtain a residual feature corresponding to the target audio bitstream.

[0222] For example, feature decoding is the inverse process of feature encoding. The target coding feature estimate is subjected to feature decoding processing to obtain a residual feature (a kind of estimate) corresponding to the target audio bitstream.

[0223] In some embodiments, step 20311 can be implemented by: subjecting the target coding feature estimate corresponding to the target audio bitstream to convolution processing to obtain a convolution feature, wherein the number of channels of the convolution feature is less than the number of channels of the target coding feature estimate corresponding to the target audio bitstream; and subjecting the convolution feature to up-sampling processing to obtain the residual feature corresponding to the target audio bitstream.

[0224] In the field of audio codec, up-sampling operation is used to increase the resolution of feature maps (i.e., convolution features) to more accurately reconstruct audio signals. Up-sampling involves interpolation or other forms of up-sampling techniques to generate higher-precision feature maps, which helps better recover the original details and characteristics of audio signals in the decoding process. By using neural network techniques such as convolution, pooling, and up-sampling in audio decoding, useful features can be effectively extracted, computational complexity can be reduced, and the original content of audio signals can be more accurately reconstructed. These techniques are of great significance to improving the performance and efficiency of audio decoding, and help to promote the development and application of audio codec technology.

[0225] Of course, before step 20311, the target coding feature estimate corresponding to the target audio bitstream can also be subjected to causal convolution to obtain a causal convolution target coding feature estimate, and step 20311 is performed based on the causal convolution target coding feature estimate, i.e., the causal convolution target coding feature estimate is subjected to feature decoding processing to obtain a residual feature corresponding to the target audio bitstream.

[0226] In step 20312, the residual feature corresponding to the target audio bitstream is subjected to feature residual processing by at least one residual unit to obtain an audio feature estimate corresponding to the target audio bitstream.

[0227] In some embodiments, when the at least one residual unit is a plurality of cascaded residual units, step 20312 can be implemented by: subjecting the residual feature to residual processing by one residual unit to obtain an audio feature estimate corresponding to the target audio bitstream.

[0228] In some embodiments, when the at least one residual unit is a plurality of cascaded residual units, step 20312 can be implemented by: performing residual processing on the residual feature by a first residual unit of the plurality of cascaded residual units; outputting the residual result output by the first residual unit to a subsequent cascaded residual unit, and continuing to perform residual processing and output residual results by the subsequent cascaded residual unit; and outputting the residual result output by the last residual unit as the audio feature estimation value corresponding to the target audio bitstream.

[0229] In some embodiments, the second neural network for audio decoding includes a plurality of cascaded decoding blocks, each decoding block including a feature decoding block and at least one residual unit; and step 2031 can be implemented by: performing feature decoding processing on the target encoded feature estimation value corresponding to the target audio bitstream by the feature decoding block in the plurality of cascaded decoding blocks to obtain a residual feature corresponding to the target audio bitstream; and correspondingly, performing residual processing on the residual feature corresponding to the target audio bitstream by the at least one residual unit in the plurality of cascaded decoding blocks to obtain the audio feature estimation value corresponding to the target audio bitstream.

[0230] In some embodiments, the feature decoding block in the first decoding block of the plurality of cascaded decoding blocks performs feature decoding processing on the target encoded feature estimation value corresponding to the target audio bitstream, and outputs the decoding result of the feature decoding block in the first decoding block to at least one residual unit in the first decoding block; the feature decoding block in the i th decoding block of the plurality of cascaded decoding blocks performs feature decoding processing on the residual result output by the at least one residual unit in the i-1 th decoding block, and outputs the decoding result of the feature decoding block in the i th decoding block to at least one residual unit in the i th decoding block; and the decoding result output by the feature decoding block in the last decoding block is taken as the residual feature corresponding to the target audio bitstream; wherein i is a positive integer that increases sequentially, 1 < i ≤ I, and I is the number of decoding blocks. The decoding result output by the feature decoding block in the first decoding block is obtained by performing the following processing on the target encoded feature estimation value corresponding to the target audio bitstream by the feature decoding block in the first decoding block of the plurality of cascaded decoding blocks: performing convolution processing on the target encoded feature estimation value to obtain convolution features, wherein the number of channels of the convolution features is less than the number of channels of the target encoded feature estimation value; and performing up-sampling processing on the convolution features to obtain the decoding result output by the feature decoding block in the first decoding block. The decoding result output by the feature decoding block in the i th decoding block is obtained by performing the following processing on the residual result output by the at least one residual unit in the i-1 th decoding block by the feature decoding block in the i th decoding block: performing convolution processing on the residual result output by the at least one residual unit to obtain convolution features, wherein the number of channels of the convolution features is less than the number of channels of the residual result output by the at least one residual unit; and performing up-sampling processing on the convolution features to obtain the decoding result output by the feature decoding block in the i th decoding block.

[0231] In some embodiments, the residual processing on the residual feature corresponding to the target audio bitstream by the at least one residual unit in the last decoding block of the plurality of cascaded decoding blocks obtains the audio feature estimation value corresponding to the target audio bitstream.

[0232] After step 2031, in step 2032, the audio feature estimation value is subjected to feature reconstruction to obtain a reconstructed audio signal.

[0233] Here, feature reconstruction is the inverse process of feature extraction, and the feature reconstruction processing upgrades the audio feature estimation value to realize the function of data decompression.

[0234] In some embodiments, step 2032 can be implemented by upsampling the audio feature estimation value corresponding to the target audio code stream to obtain an upsampled feature, and performing causal convolution on the upsampled feature to obtain the reconstructed audio signal.

[0235] In summary, steps 101-105 and steps 201-203 implement an audio communication method including a natural sound mode and a transformed sound mode. For the transformed sound mode, the embodiments of the present application implement a streaming transformed sound effect of tens of milliseconds based on the first neural network of the encoding terminal, the conversion network of the encoding terminal, and the second neural network of the decoding terminal, to implement a real-time communication application mode.

[0236] As described above, the audio conversion method provided by the embodiments of the present application can be implemented by various types of electronic devices, such as terminals and servers. Therefore, the execution subject of each step will not be repeated below. Referring to FIG. 15, FIG. 15 is a flowchart of an audio conversion method provided by an embodiment of the present application, and the audio conversion method is applied to a terminal. The steps shown in FIG. 15 will be described in combination.

[0237] In step 301, the audio signal is feature encoded to obtain encoded features of the audio signal.

[0238] Step 301 is similar to step 102 described above, and will not be repeated here.

[0239] In step 302, a target timbre corresponding to the transformed sound mode is obtained, and a timbre feature of the target timbre is determined.

[0240] Step 302 is similar to step 103 described above, and will not be repeated here.

[0241] In step 303, the encoded features are timbre converted based on the timbre feature of the target timbre to obtain target encoded features.

[0242] Step 303 is similar to step 104 described above, and will not be repeated here.

[0243] In step 304, the target encoded features are signal encoded to obtain a target audio code stream conforming to the target timbre.

[0244] Step 304 is similar to step 105 described above, and will not be repeated here.

[0245] In step 305, the target audio code stream is signal decoded to obtain a target encoded feature estimation value corresponding to the target audio code stream.

[0246] Step 305 is similar to step 202 described above, and will not be repeated here.

[0247] In step 306, the target coding feature estimation value is feature decoded to obtain a reconstructed audio signal conforming to the target voice timbre.

[0248] In step 306, the target coding feature estimation value is feature decoded to obtain a reconstructed audio signal conforming to the target voice timbre.

[0249] In summary, steps 301-306 implement an audio conversion method. For the voice conversion mode, the embodiments of the present application implement a streaming voice conversion effect of tens of milliseconds based on the first neural network, the conversion network, and the second neural network deployed locally on the electronic device, to implement a local end, non-real-time communication application mode, which does not require the encoding end to encode the voice signal and send it to the decoding end for decoding to achieve voice conversion, but deploys the encoding network and the decoding network locally, thereby achieving voice conversion of the audio on a single local end. It should be noted that the local audio conversion method of the embodiments of the present application is not limited to a certain application scenario, which is illustrated below.

[0250] As shown in FIG. 6A, if the user (i.e., the source speaker) selects the voice conversion mode, the voice signal of the source speaker is first feature encoded by the first neural network deployed locally on the electronic device to obtain the feature vector (i.e., the coding feature) of the source speaker. Then, a conversion network (such as the conversion network 1 and the conversion network 2 shown in FIG. 6A) is added after the first neural network deployed locally on the electronic device, which converts the feature vector (i.e., the coding feature) related to the source speaker into a feature vector (i.e., the target coding feature) related to the target speaker. Then, the target coding feature estimation value (i.e., the feature vector input to the second neural network) is obtained using the quantization encoding and the quantization decoding locally on the electronic device. Finally, the input feature vector is feature decoded by the second neural network deployed locally on the electronic device to obtain the voice signal converted to the target speaker (such as the voice signal converted to the target speaker 1 and the voice signal converted to the target speaker 2 shown in FIG. 6A).

[0251] Taking the application scenario of voice expression and voice animation as an example, a user can select to enter the expression sending function in the instant messaging software, and through the audio conversion method of the embodiment of the present application, the instant messaging software realizes voice expression and voice animation. For example, when the user enters the conversation interface of instant messaging, the user can use the related functions of the conversation, including but not limited to: recording, playing back, sharing, canceling, exiting, etc., which are not limited herein. For example, when the user selects a certain expression template, the background will call the voice changing mode and correspondingly load the voice changing related configuration file corresponding to the expression template, including but not limited to the model parameters related to the conversion network of the embodiment of the present application, etc., to convert the source timbre in the expression template into the target timbre. Among them, the expression template can also be a voice expression and voice animation recorded locally by the user. In another embodiment, when the user records a voice message, the user can select the voice changing mode to convert the voice of the user into the voice of another person and generate a converted voice message, and then send it to the other party. In this way, the diversity and entertainment of voice messages and voice expression packs or animations can be improved.

[0252] Taking the application scenario of karaoke as an example, the karaoke software is mainly used to mix the user's original voice into the accompaniment or mix the accompaniment with the original voice. Through the audio conversion method of the embodiment of the present application, the karaoke realizes the voice changing function, and the user can select to enter the voice changing function in the karaoke software, and can convert the user's singing voice in real time (such as converting into a timbre close to the original singer's timbre), to achieve better results. For example, when the user enters the karaoke interface, the user can use the related functions, including but not limited to: recording, playing back, sharing, canceling, exiting, etc., which are not limited herein. When the user selects a certain song, the background will call the voice changing mode, and the background will correspondingly load the voice changing related configuration file corresponding to the song, including but not limited to the model parameters related to the conversion network of the embodiment of the present application, etc., to convert the singing voice of the user when singing in karaoke into the original singer's voice, and improve the user's singing experience.

[0253] Taking the application scenario of sound amplification as an example, the sound amplification software is mainly used to amplify the sound so that the sound can be transmitted farther and clearly, and at the same time can reduce sound distortion, making the sound more natural and pleasant. It is usually used in occasions such as speeches, teaching, guiding, music performances, and celebration activities to help improve the clarity and volume of sound transmission. Through the audio conversion method of the embodiment of the present application, the sound amplification realizes the voice changing function, and the user can select to enter the voice changing function in the sound amplification software, and can convert the user's voice in real time (such as converting into a timbre close to a pleasant timbre) in real time, to achieve better results. For example, when the user enters the sound amplification interface, the user can use the voice changing function. When the user selects a certain target timbre, the sound amplification software will call the voice changing mode to convert the original voice input by the user into the target timbre, so that the sound is more in line with the needs of the scene.

[0254] Taking the application scenario of translation as an example, a user needs to translate the dialogue of a character in a foreign language movie or a foreign language video, and the audio conversion method of the embodiment of the application converts the translated dialogue of the original sound output by the user into dialogue of the target voice color of the corresponding character. Because the dialogue of the character after conversion still has the voice color of the role, the translated dialogue audio after voice color conversion is more like the audio directly output by the character, improving the overallity and fusion of the translated dialogue of the character and the original movie.

[0255] In the following, an exemplary application of the embodiment of the application in an actual application scenario will be described.

[0256] The embodiment of the application can be applied to various audio scenarios, such as voice calls, instant messaging, etc. The following takes voice calls as an example for description:

[0257] The embodiment of the application provides a multi-task coding and decoding method (i.e. an audio communication method). For an input signal, a low-dimensional feature vector F(n) is extracted based on neural network (NN) technology, the feature vector is quantized and encoded with fewer bits, and the optimal quality of the restored decoding end is ensured. In the encoding end part, after the input signal is processed by the encoding network, a feature vector with a lower dimension than the input signal is obtained, the feature vector is compressed and encoded, and a similar "blocking" operation is adopted in the neural network to reduce the algorithm complexity. In the decoding end part, the corresponding decoding network is called for the received code stream to complete the reconstruction of the signal, and a similar "blocking" operation is adopted in the neural network to reduce the algorithm complexity.

[0258] It should be noted that the embodiment of the application completes the extension function based on the voice compression (task 1) and the related feature vector involved in the above-mentioned encoding end and decoding end. For example, after the encoding feature is extracted in the encoding end, a conversion network is used to complete the streaming voice conversion (task 2). The conversion network is a multi-layer convolutional network, the input is the feature vector F(n) output by the encoding network and the feature vector e (ID) (n) reflecting the target speaker voice color information of the conversion (i.e. the voice color feature of the target voice color), the feature vector X and the feature vector I are spliced as {F(n), e (ID)(n)}, the output of the conversion network is converted into a feature vector F vc(n) consistent with the dimension of the feature vector output by the encoding network. The user can choose whether to start the conversion network to determine whether the object of the encoding end quantization is F(n) or F vc(n). Correspondingly, is the reconstructed speech at the decoding end the original speaker's voice or the voice of the specified target speaker's voice. Among them, the encoding network can contain one or more conversion networks, each conversion network corresponds to a specific feature vector of the target speaker's voice. Among them, the structure of one or more conversion networks is the same, only the network parameters are different, the dimensions of the feature vectors of one or more target speaker voices are consistent, only the values are different, the values of the feature vectors of one or more target speaker voices are constants, the feature vectors of one or more target speaker voices can be extracted for the processing of streaming voice conversion.

[0259] To solve the problems brought by FIG. 1, the embodiment of the present application proposes to combine voice conversion and voice compression, and make the codec system become a "multi-task codec system". The multi-task voice communication system (implemented by the multi-task codec system) provided by the embodiment of the present application is described below in conjunction with FIG. 6B.

[0260] The above multi-task codec system includes an original sound mode and a changed sound mode. The original sound mode and the changed sound mode are described below respectively.

[0261] If the user selects the original sound mode, the processing flow is consistent with the end-to-end neural network codec flow. Therefore, in the case of low code rate, the content of the source speaker can be transmitted to the decoding end.

[0262] If the user selects the changed sound mode, a conversion network (such as conversion network 1 and conversion network 2 shown in FIG. 6B) is added after the encoding network, which converts the feature vector related to the source speaker into a feature vector related to the target speaker. Then, referring to the original sound mode, the same quantization encoding and decoding end flow is used. Therefore, in the case of low code rate, the timbre of the source speaker can be modified to the timbre of the target speaker, and the content of the source speaker can be transmitted to the decoding end. Since the delay is only tens of milliseconds, a true "streaming voice conversion" is realized.

[0263] Among them, each target speaker can correspond to a specific conversion network. Therefore, the above multi-task codec system can contain one or more conversion networks. In actual use, the corresponding conversion network is selected for processing according to the target speaker selected by the sending end user. For a real-time communication system, the frame length is 20ms. Therefore, the above system can realize the two tasks of compression and voice conversion under tens of milliseconds of delay.

[0264] The embodiment of the present application can be applied to a voice communication link. Taking a voice over Internet Protocol (VoIP) conference system based on an Internet protocol as an example, the multi-task voice communication system can deploy the voice coding and decoding technology involved in the embodiment of the present application in the coding and decoding part, and deploy it in a client or a cloud server according to the computing requirement. For the voice compression function, the coding and decoding part is installed in the client of the sending end and the receiving end respectively. For the voice conversion function, the conversion network is deployed in the client of the sending end.

[0265] The multi-task coding method provided by the embodiment of the present application is described below in combination with FIG. 8.

[0266] The following processing is performed for the encoding end:

[0267] The input audio signal x(n) of the nth frame is called to generate a lower-dimensional feature vector F(n) by using a first neural network. The embodiment of the present application does not limit other NN structures, such as an autoencoder, a full-connection (FC) network, a long short-term memory (LSTM) network, a convolutional neural network (CNN) + LSTM, and the like. In the neural network, a block-like operation is used, which can reduce the algorithm complexity and improve the coding effect.

[0268] If the user uses a non-variable voice mode (i.e., an original voice mode), the feature vector F(n) is subjected to vector or scalar quantization, and the quantized index value is subjected to entropy coding. The obtained code stream is transmitted to the decoding end.

[0269] If the user uses a variable voice mode, the feature vector F(n) is processed by using a conversion network to obtain a new feature vector F_vc(n) with the same dimension as F(n). According to the target speaker identification (ID) selected by the user, the corresponding parameters in the conversion network are configured. The feature vector F_vc(n) processed by the conversion network is subjected to vector or scalar quantization, and the quantized index value is subjected to entropy coding. The obtained code stream is transmitted to the decoding end.

[0270] The following processing is performed for the decoding end:

[0271] The code stream received by the decoding end is decoded to obtain an estimated value F'(n) of the feature vector. A second NN is called to obtain an estimated value x'(n) of the voice signal. In the neural network, a block-like operation is used to reduce the algorithm complexity.

[0272] The decoding end does not directly control the operation of the encoding end, that is, the processing process of the decoding end in the voice changing mode is the same as that in the original voice mode. When the encoding end selects the voice changing mode, the decoding end still decodes according to the decoding method in the original voice mode to output the reconstructed speech signal which is the specific target person timbre selected by the user of the sending end. When the encoding end selects the original voice mode, the processing flow of the decoding end is consistent with the end-to-end neural network coding and decoding flow, and the reconstructed speech signal is the source speaker timbre.

[0273] The multi-task coding and decoding method provided by the embodiment of the application will be described in detail below.

[0274] In some embodiments, a speech signal with a sampling rate Fs=16000Hz is taken as an example (it should be noted that the method provided by the embodiment of the application is also applicable to other sampling rate scenarios, including but not limited to: 8000Hz, 32000Hz, 48000Hz). At the same time, it is assumed that the frame length is set to 20ms, so for Fs=16000Hz, it is equivalent to containing 320 sample points per frame.

[0275] The encoding end and the decoding end will be described in detail below with reference to the flowchart shown in FIG. 8.

[0276] The flow of the encoding end is as follows:

[0277] For an audio signal with a sampling rate Fs=16000Hz, the input signal of the nth frame includes 320 sample points, denoted as input signal x(n).

[0278] Step 11, calling the first NN.

[0279] Based on the input signal x(n), the first NN is called to generate a feature vector F(n) with a lower dimension. It should be noted that the dimension of x(n) is 320 and the dimension of F(n) is 56. From the data amount, the first NN plays a role of "dimension reduction" and realizes the function of data compression. It should be noted that the embodiment of the application is not limited to the dimension of F(n), but can also be other dimensions smaller than that of x(n).

[0280] Referring to the network structure diagram of the first NN shown in FIG. 9, the flow of data compression of the first NN will be described in detail below:

[0281] First, a 16-channel causal convolution is called, which can expand the input tensor (i.e., vector) into a 16x320 tensor.

[0282] Then, the 16x320 tensor is preprocessed. For example, after one convolution operation on the 16x320 tensor, a pooling operation with a factor of 2 is performed and the activation function can be PReLU, to generate a 16x160 tensor.

[0283] Next, four encoding blocks with different down-sampling factors (Down_factor) are cascaded. Each encoding block contains a residual block, a convolution layer and a pooling layer. Each residual block includes five residual units based on a dilated convolution (the feature dimension of the input and output of the residual unit does not change); the convolution layer is used to double the number of input channels, and the activation function can be PReLU, thereby ensuring the amount of data and avoiding data loss; the pooling layer is a pooling operation containing Down_factor to complete the down-sampling and realize data compression. Here, the Down_factor of the four encoding blocks is set to 2, 4, 4 and 5 respectively. Therefore, the output channel numbers of the four encoding blocks are set to 32, 64, 128 and 256 respectively. After processing by the four encoding blocks, the input 16x160 tensor is converted into a 32x80, 64x20, 128x5 and 256x1 tensor respectively. The number of encoding blocks in the present application is not limited to four, but can be any positive integer such as two, three, four or five. In addition, the number of residual units in the encoding block is not limited to five, but can be any positive integer such as two, three, four, five or six. The number of residual units in the multiple encoding blocks can be the same or different, for example, one encoding block contains four residual units and another encoding block contains five residual units.

[0284] The residual unit is further introduced here. The residual unit is a module in a deep neural network. By introducing cross-layer connections in the neural network, the neural network is easier to optimize during training, avoiding problems such as gradient vanishing or gradient explosion. The core idea is to learn the residual of the input within the module, that is, to pass the input information directly to the output by a direct path, bypassing a part of the layers, so that the network can better utilize the shallow feature information during learning. FIG. 10A is a schematic diagram of the residual block structure used in the encoding block in the first NN. The residual block includes five residual units based on a dilated convolution. Each residual unit includes a dilated convolution block with a specified dilation rate, that is, each dilated convolution block includes a convolution operator with a specified dilation rate (such as Dilation rate = 3). In the present application, five dilated convolution blocks with progressive dilation rates are used, which is equivalent to using different receptive fields to extract features of the input at different resolutions, which can better analyze the data relatively comprehensively. After residual processing by the five dilated convolution blocks with specified dilation rates, the input from the skip connection is added to obtain the output result of the residual block, and the output result is output to the convolution layer connected to the residual block.

[0285] Here, further description is made to any one residual unit of FIG. 10A, as shown in FIG. 10B. For any one residual unit, the internal contains a dilated convolution (for expanding the receptive field) with a specified dilation rate, which can use PReLU as the activation function; in addition, one or more causal convolutions (for extracting local information) can be cascaded, which can use PReLU as the activation function. The convolution kernel size of the above-mentioned dilated convolution with a specified dilation rate can be 3, 5, 7, 9, etc., and the convolution kernel size of the above-mentioned causal convolution can be 1, 3, etc. The present embodiment of the present application does not limit the convolution kernel size of the above-mentioned dilated convolution with a specified dilation rate or the causal convolution. In addition, the causal convolution or dilated convolution in the present embodiment of the present application can also be implemented by other convolution units with similar or equivalent functions.

[0286] In addition, for the residual unit, in order to reduce the algorithm complexity, the algorithm of grouped convolution is introduced. The grouped convolution is to divide the input channels into multiple groups for convolution operation, and only the input channels and output channels in each group are associated. Here, it is assumed that the input channel is 16 and the output channel is 32. If the number of groups is 1, each input channel is associated with 32 output channels. If the number of groups is 2, the 16 input channels are first divided into two groups of 0-7 and 8-15, and in the two groups, the input channels are associated with the output channels in the group, for example, the input channels 0-7 in the first group are associated with the output channels 0-15, and the input channels 8-15 in the second group are associated with the output channels 16-31, for example, the 0th output channel is only associated with the 0th-7th input channel, and is not associated with the 8th-15th input channel, and the 25th output channel is only associated with the 8th-15th input channel, and is not associated with the 0th-7th input channel. By comparison, it can be seen that the introduction of grouped convolution can avoid the association of any input channel with all output channels, reduce the number of connections, and reduce the complexity. Of course, since the larger the number of groups is, the smaller the association degree of the input channels and the output channels is, the coding effect will also be affected, so it is not necessarily the larger the number of groups is, the better. In the present embodiment of the present application, the dilated convolutions contained in the four residual blocks corresponding to the four coding blocks can use different group number configurations, and the specific group number configurations are shown in Table 1.

[0287] Table 1. Group number configuration used by residual unit in different coding blocks

[0288] Finally, the 256x1 tensor is further subjected to the causal convolution similar to the preprocessing, and a 56-dimensional feature vector F(n) can be output.

[0289] Step 12, calling the conversion network.

[0290] The multi-task codec system performs step 12 only when the user at the encoding end selects the voice changing mode; otherwise, it directly jumps to step 13.

[0291] When the user uses the voice changing mode, the output F(n) of the first NN is processed by the conversion network to regenerate a feature vector F_vc(n). The dimension of F_vc(n) is the same as that of F(n), that is, 56, and the values are all between -1 and 1. The processing procedure of the conversion network is as follows:

[0292] First, the input of the conversion network includes, in addition to the 56-dimensional F(n), an additional embedding feature e (ID) (n) reflecting the timbre feature of the target speaker (i.e., the timbre feature of the target voice). In the embodiment of the present application, the dimension of the embedding feature can be 88. F(n) and e (ID) (n) are combined into a 144x1 vector. As for the generation procedure of the embedding feature, there are many feature extraction methods in conventional speech signal processing to extract the timbre information of a user. The above information includes but is not limited to F0 fundamental frequency information related features, loudness features, mel-frequency cepstral coefficient (MFCC) spectrum information, etc. The feature information in the embedding feature can be extracted frame by frame, and then the mean value, variance, etc. of several seconds of data are calculated, so that the embedding feature is expressed in multiple dimensions. In the training data, for any target speaker, multiple files can be extracted, and the 88-dimensional features corresponding to each file are calculated. Then, the features are averaged in each dimension to obtain e (ID) (n). Each target speaker corresponds to an 88-dimensional feature, which is identified by an ID index.

[0293] Then, the 144x1 vector is processed by the conversion network to regenerate a 56x1 vector F_vc(n). Because e (ID) (n) provides additional information, the generated F_vc(n) contains not only the content information related to F(n), but also the speaker timbre information of the selected target speaker (identified by the ID).

[0294] The conversion network will be described below in combination with the structure of the conversion network shown in FIG. 11.

[0295] First, a 256-channel causal convolution is called to output a 256x1 tensor from the input 56x1 tensor.

[0296] Then, a 256-channel transposed encoding block is called. Each transposed encoding block contains a residual block and a convolutional layer, and each residual block contains multiple (e.g., 5) residual units based on a dilated convolution. The convolutional layer is used to maintain the number of channels unchanged, and the activation function can be PReLU. It should be noted that, compared with the encoding block of the first NN, the transposed encoding block is less in the number of channel increases and pooling operations, and other configurations can be referred to the implementation of the encoding block in the first NN. Therefore, the input of the transposed encoding block is a 256x1 tensor, and the output is still a 256x1 tensor.

[0297] Next, a 128-channel causal convolution is called, which can output a 128x1 tensor from the input 256x1 tensor.

[0298] Then, a 128-channel transposed encoding block is called. Each transposed encoding block contains a residual block and a convolutional layer, and each residual block contains at least one (e.g., 5) residual unit based on a dilated convolution. The convolutional layer is used to maintain the number of channels unchanged, and the activation function can be PReLU. It should be noted that, compared with the encoding block of the first NN, the transposed encoding block is less in the number of channel increases and pooling operations, and other configurations can be referred to the implementation of the encoding block in the first NN. Therefore, the input of the transposed encoding block is a 128x1 tensor, and the output is still a 128x1 tensor.

[0299] Finally, a 56-channel causal convolution is called, which can output a 56x1 tensor from the input 128x1 tensor. A tahn operation is called to control each element in the 56x1 tensor to be between [-1, 1], and a new feature vector F_vc(n) is obtained.

[0300] In the embodiments of the present application, the number of causal convolutions and transposed encoding blocks in the transposed network is not limited to two, and can be any positive integer such as two, three, four, five, etc. In addition, the number of residual units in the transposed encoding block is not limited to two, and can be any positive integer such as two, three, four, five, six, etc. The number of residual units in multiple transposed encoding blocks can be the same or different, for example, one transposed encoding block contains four residual units, and another transposed encoding block contains five residual units.

[0301] The embodiments of the present application support deploying one or more target speaker streaming conversion functions at the encoding end. For example, the encoding end determines the corresponding e (ID) (n) and the parameters of the transposed network, the conversion of the feature vector from the source speaker voice color to the target speaker voice color can be completed.

[0302] Step 13, quantization encoding.

[0303] For the feature vector F(n) in the original sound mode or the feature vector F_vc(n) in the changed sound mode, both can be subjected to the method of scalar quantization (each component is quantized separately) and entropy encoding. In addition, the embodiments of the present application also do not limit the technical combination of vector quantization (adjacent multiple components are combined into a vector for joint quantization) and entropy encoding.

[0304] According to the above description, a 56-dimensional feature vector is obtained by the first NN, such as the feature vector F(n) in the original sound mode or the feature vector F_vc(n) in the changed sound mode. The embodiments of the present application provide a method based on scalar quantization and entropy encoding, including: 1) for each dimension of the 56-dimensional feature vector, the interval of [-1, 1] is evenly divided into 11 parts to form a codebook containing 11 elements, and each dimension value is quantized into one of the 11 elements; 2) according to the Shannon entropy theorem, for the codebook containing 11 elements and uniformly distributed, the entropy (average bits) is Therefore, the average bits corresponding to each frame of low-frequency sub-band signal is 193.76 bits; 3) for every 20ms frame mode, there are 50 frames in 1 second, therefore, the average code rate is 9.69kbps. According to the entropy encoding theory, the probability distribution statistics can be performed on each dimension to generate 56 codebooks. Generally, each dimension is non-uniformly distributed, therefore, the actual code rate is near 9.69kbps or less than 9.69kbps.

[0305] In addition, in order to achieve the goal of multi-rate coding and decoding of the low-frequency feature vector F LB (n), the following implementation forms can be used:

[0306] For the codebook of the above implementation, i.e. the codebook containing 11 elements for each dimension, it is named codebook-1, and the reference code rate is 9.69kbps.

[0307] Similarly, for each dimension, the interval of [-1, 1] is evenly divided into 9 parts, the entropy is 3.17, and the average code rate is 3.17*56*50 / 1000=8.88kbps. It is named codebook-2, and the reference code rate is 8.88kbps. In this way, there are at least two code rate modes, different code rates correspond to different qualities; multi-rate encoding is achieved.

[0308] By analogy, for each dimension, the interval of [-1, 1] is evenly divided into 7 parts, the average code rate is 7.86kbps, and it is named codebook-3; the interval of [-1, 1] is evenly divided into 5 parts, the average code rate is 6.50kbps, and it is named codebook-4.

[0309] Thus, for each frame of data, the original 56-dimensional feature vector obtained is quantized to achieve at least four coding modes of features at different code rates according to the configuration of quantization accuracy. It should be noted that the embodiments of the present application do not limit other multi-code rate construction methods and the number of codebooks (also referred to as code tables).

[0310] After quantization coding, a code stream can be generated. According to experiments, in the range of 5-10 kbps, high-quality compression of a 16 kHz wideband signal can be achieved. For the above communication system, compression can be performed at only one specific code rate or at multiple code rates. If multiple code rates need to be supported, one or more bits are used in the header of each frame to represent the coding code rate. For example, 9.69 kbps, 8.88 kbps, 7.86 kbps, and 6.50 kbps are supported, and at least four codebooks (also referred to as code tables) of the above code rates are stored in the system.

[0311] It should be noted that for the original voice mode, quantization coding is the encoding of the source speaker content; for the transformed voice mode, quantization coding is the conversion of the content into the timbre of the target speaker and then the encoding of the content.

[0312] The flow at the decoding end is as follows:

[0313] Step 21, quantization decoding.

[0314] Quantization decoding is the inverse process of quantization coding. For the received code stream, entropy decoding is first performed, and the estimated value F'(n) of the feature vector of the code stream is obtained by searching the quantization table.

[0315] For each frame of code stream, the corresponding rate of the encoding end and the corresponding codebook are determined. Through entropy coding and decoding technology, the code stream is analyzed, and according to the corresponding codebook and rate, the estimated value F'(n) of the 56-dimensional feature vector can be obtained.

[0316] Step 22, calling the second NN based on the estimated value F'(n) of the feature vector.

[0317] First, the second NN shown in FIG. 12 is called based on the estimated value F'(n) of the feature vector of the code stream to generate the estimated value x'(n) of the audio signal. The second NN is similar to the first NN, such as causal convolution, and the post-processing structure is similar to the pre-processing structure in the first NN. The specific flow of the second NN is as follows:

[0318] First, a causal convolution is called to expand the input tensor F'(n) from 56x1 to a 256x1 tensor.

[0319] Next, four decoding blocks with different up-sampling factors (Up_factor) are cascaded. Each decoding block contains a convolution layer, an up-sampling module, and a residual block. The convolution layer is used to halve the number of input channels. The up-sampling module contains a specific Up_factor to complete the up-sampling. The residual block includes five residual units based on a dilated convolution. The Up_factor of the four decoding blocks is set to 5, 4, 4, and 2, respectively. Therefore, the output channel number of the four decoding blocks is set to 128, 64, 32, and 16, respectively. After the four decoding blocks, the 256x1 tensor is converted into 128x5, 64x20, 32x80, and 16x160 tensors, respectively. The number of decoding blocks in the embodiments of the present application is not limited to four, but can be any positive integer such as two, three, four, or five. In addition, the number of residual units in the decoding blocks in the embodiments of the present application is not limited to five, but can be any positive integer such as two, three, four, five, or six. The number of residual units in the multiple decoding blocks can be the same or different. For example, one decoding block contains four residual units, and another decoding block contains five residual units.

[0320] Here, for the up-sampling module containing a specific Up_factor, a Repeat operation can be used to complete the up-sampling operation by repeating padding. In this way, the complexity can be saved.

[0321] Here, the configuration of the five residual units based on a dilated convolution at the decoding end is similar to that of the residual units at the encoding end, including but not limited to the internal structure of the residual units, the size of the convolution kernel, the dilation rate, etc. The configuration of the number of groups used by the dilated convolution in the decoding block is shown in Table 2. In the decoding block, a larger number of groups of 2 is used to associate more input channels and output channels and improve the quality of speech reconstruction.

[0322] Table 2. Configuration of the number of groups used by the residual units in different decoding blocks

[0323] Then, post-processing is performed on the 16x160 tensor output by the cascaded decoding blocks. For example, a Repeat operation with a factor of 2 is performed on the 16x160 tensor output by the cascaded decoding blocks to complete the up-sampling, and then a convolution operation is performed and an activation function can be used PReLU operation to generate a 16x320 tensor.

[0324] Finally, a causal convolution is called to convert the input 16x320 tensor into a 1x320 tensor to reconstruct the audio signal.

[0325] It should be noted that, for the original sound mode, the decoding end is the decoding and restoration of the source speaker content; for the voice changing mode, it is converted into the timbre of the target speaker and then the content is decoded and restored.

[0326] The embodiment of the application can obtain optimal parameters by collecting data and jointly training the related networks of the encoding end and the decoding end. The user only needs to prepare the data and set the corresponding network structure, and after the training is completed in the background, the trained model can be put into use.

[0327] It should be noted that, in the end-to-end neural network coding and decoding system, the training mechanism of adversarial learning can be used. The training principle of adversarial learning is to improve the performance of the generation model by letting the generation model and the discrimination model oppose each other. Specifically, the generation model tries to generate realistic samples to fool the discrimination model, while the discrimination model tries to identify the difference between the real samples and the generated samples. This adversarial process is iterated until the quality of the samples generated by the generation model is high enough. The user only needs to prepare the data and set the corresponding network structure, and after the training is completed in the background, the trained model can be put into use.

[0328] For the original sound mode, the encoding network (including the first NN) and the decoding network (including the second NN) are trained based on supervised learning. The input of the encoding network is the source speaker signal, and the output is also the source speaker signal. The purpose of training is to obtain a deep neural network system that ensures that the predicted signal is infinitely close to the source speaker signal.

[0329] For the voice changing mode, the encoding network (including the first NN and the conversion network) and the decoding network (including the second NN) are trained based on supervised learning. The input of the encoding network is the source speaker signal, and the output is the target speaker signal. The purpose of training is to obtain a deep neural network system (including the conversion network corresponding to the ID of the target speaker) that ensures that the predicted signal is infinitely close to the signal of the target speaker.

[0330] Considering that the original sound mode is the basis and the voice changing mode is an extended function in the multi-task voice communication system, and that the voice communication system should support streaming voice changing of one or more target speakers, the following is the training process of the conversion network combined with the trained first NN and second NN:

[0331] 1) Load the first NN and the model parameters of the first NN that have been trained.

[0332] 2) In the training platform, load the embedding and network configuration related to the target speaker corresponding to the ID. The network configuration includes but is not limited to e (ID), model structure, loss function, optimizer, etc.

[0333] It should be noted that the embodiments of the present application need to set the gradient update in the first NN and the second NN to False. By setting the gradient update in the first NN and the second NN to False, when training the conversion network in an end-to-end manner, only the parameters of the first NN and the second NN can be fixed, so the first NN and the second NN will not participate in gradient calculation. In this way, when retraining the conversion network, any parameter update is only performed in the conversion network. If more target speakers need to be supported, the above process can be referred to to obtain a training data set, obtain the corresponding e (ID) , and perform model configuration to obtain the conversion network parameters corresponding to the target speaker. Therefore, in the above multi-task speech communication system, one or more conversion networks are included, and each conversion network supports stream speech conversion of a corresponding target speaker.

[0334] In summary, the multi-task coding and decoding method provided by the embodiments of the present application combines signal processing technology with deep neural networks, significantly improves coding efficiency compared to signal processing solutions while ensuring audio quality and acceptable complexity, and realizes stream speech conversion.

[0335] The audio communication method provided by the embodiments of the present application has been described in combination with the exemplary application and implementation of the terminal device provided by the embodiments of the present application. The audio communication method provided by the embodiments of the present application also provides an audio communication apparatus. In actual application, each functional module in the audio communication apparatus can be cooperatively implemented by hardware resources of an electronic device (such as a terminal device, a server, or a server cluster), such as computing resources (such as a processor), communication resources (such as those used to support various communication modes such as optical cable and cellular), and a memory. FIG. 3A shows an audio communication apparatus 555 stored in a memory 550, and FIG. 3B shows an audio communication apparatus 655 stored in a memory 650. The audio communication apparatus 555 and the audio communication apparatus 655 can be software in the form of programs and plug-ins, for example, software modules designed by software programming languages such as C / C++, Java, special software modules in application software or large software systems designed by software programming languages such as C / C++, Java, application program interfaces, plug-ins, cloud services, and the like. Different implementation manners are exemplarily described below.

[0336] The audio communication apparatus 555 includes a series of modules, including an acquisition module 5551, a feature encoding module 5552, a determination module 5553, a conversion module 5554, and an information encoding module 5555. The scheme for implementing audio encoding by each module in the audio communication apparatus 555 provided by the embodiments of the present application is described below.

[0337] The acquisition module 5551 is configured to acquire a voice changing mode for the first audio signal from a plurality of communication modes in response to a communication request for the first audio signal; the feature encoding module 5552 is configured to perform feature encoding on the audio signal to obtain encoded features of the audio signal; the determination module 5553 is configured to acquire a target timbre corresponding to the voice changing mode and determine timbre features of the target timbre; the conversion module 5554 is configured to perform timbre conversion on the encoded features based on the timbre features to obtain target encoded features; and the information encoding module 5555 is configured to perform signal encoding on the target encoded features to obtain a target audio bitstream conforming to the target timbre, and transmit the target audio bitstream to a decoding terminal.

[0338] In some embodiments, the acquisition module 5551 is further configured to acquire a voice original mode for the audio signal from the plurality of communication modes in response to a second communication request for the audio signal; and perform signal encoding processing on the encoded features based on the voice original mode to obtain an audio bitstream of the audio signal, and transmit the audio bitstream to the decoding terminal.

[0339] In some embodiments, the conversion module 5554 is further configured to merge the timbre features and the encoded features to obtain merged features; and perform feature conversion on the merged features to obtain the target encoded features.

[0340] In some embodiments, the feature conversion is implemented through a conversion network, the conversion network comprising a plurality of first convolutional layers and a plurality of conversion encoding blocks, the first convolutional layers and the conversion encoding blocks being connected alternately; the conversion module 5554 is further configured to perform convolutional processing on the merged features through a first first convolutional layer, and perform conversion encoding on a convolutional result of the first first convolutional layer through a first conversion encoding block connected alternately with the first first convolutional layer to obtain an encoding result of the first conversion encoding block; perform convolutional processing on the encoding result of the i-1th conversion encoding block through an i-th first convolutional layer connected alternately with the i-1th conversion encoding block, and perform conversion encoding on a convolutional result of the i-th first convolutional layer through an i-th conversion encoding block connected alternately with the i-th first convolutional layer to obtain an encoding result of the i-th conversion encoding block, where i is a positive integer that increases sequentially, 1 < i ≤ I, and I is the number of the plurality of first convolutional layers; and the convolutional result output by a last first convolutional layer is taken as the target encoded features, or the encoding result of a last conversion encoding block is taken as the target encoded features.

[0341] In some embodiments, each of the conversion encoding blocks comprises a second convolution layer and at least one residual unit; the conversion module 5554 is further configured to perform residual processing on the convolution result of the first convolution layer by at least one residual unit in the first conversion encoding block to obtain residual features of the first conversion encoding block; and perform convolution processing on the residual features of the first conversion encoding block by the second convolution layer in the first conversion encoding block to obtain an encoding result of the first conversion encoding block.

[0342] In some embodiments, when the at least one residual unit is a plurality of cascaded residual units, the conversion module 5554 is further configured to perform residual processing on the convolution result of the first convolution layer by a first residual unit of the plurality of cascaded residual units; output the residual result output by the first residual unit to a subsequent cascaded residual unit, and continue to perform residual processing and output of the residual result by the subsequent cascaded residual unit; and output the residual result output by a last residual unit as the residual features of the first conversion encoding block.

[0343] In some embodiments, the conversion module 5554 is further configured to perform the following processing by the first residual unit of the plurality of cascaded residual units: perform convolution processing on the convolution result of the first convolution layer to obtain a convolution result of the first residual unit; perform addition processing on the convolution result of the first residual unit and the convolution result of the first convolution layer to obtain a residual result output by the first residual unit; perform the following processing by a jth residual unit of the plurality of cascaded residual units: perform convolution processing on the residual result output by a (j-1)th residual unit to obtain a convolution result of the jth residual unit; perform addition processing on the convolution result of the jth residual unit and the residual result output by the (j-1)th residual unit to obtain a residual result output by the jth residual unit; and output the residual result output by the jth residual unit to a (j+1)th residual unit, where j is a positive integer that increases sequentially, 1

[0344] In some embodiments, each of the residual units comprises a dilated convolution operator and a causal convolution operator; the conversion module 5554 is further configured to perform dilated convolution processing on the convolution result of the first convolution layer by the dilated convolution operator included in the first residual unit; and perform causal convolution processing on the dilated convolution result obtained by at least one causal convolution operator included in the first residual unit, and take the causal convolution result obtained as the convolution result of the first residual unit.

[0345] In some embodiments, the feature encoding module 5552 is further configured to perform feature extraction on the audio signal to obtain audio features of the audio signal, and perform residual processing on the audio features by using at least one residual unit to obtain the encoded features of the audio signal.

[0346] In some embodiments, the audio communication method is implemented by a second encoding network in an audio codec system, the audio codec system further comprising a first encoding network and a first decoding network, the first encoding network and the first decoding network being trained by using first audio samples; and the second encoding network is trained by: in response to a configuration request for the first encoding network, determining a to-be-trained second encoding network corresponding to the first encoding network, wherein the to-be-trained second encoding network comprises a to-be-trained conversion network; performing audio encoding on second audio samples based on the to-be-trained conversion network to obtain audio bitstream samples of the second audio samples, performing audio decoding on the audio bitstream samples based on the first decoding network to obtain reconstructed audio samples of the second audio samples; and updating parameters of the to-be-trained conversion network based on the reconstructed audio samples to obtain a trained conversion network.

[0347] The audio communication apparatus 655 comprises a series of modules, including a receiving module 6551, a signal decoding module 6552, and a feature decoding module 6553. The following continues to describe a scheme for implementing audio decoding by the modules in the audio communication apparatus 655 according to the embodiments of the present application.

[0348] The receiving module 6551 is configured to receive a target audio bitstream transmitted by an encoding terminal, wherein the target audio bitstream is obtained by performing timbre conversion on an audio signal by using a target timbre corresponding to a voice mode; the signal decoding module 6552 is configured to perform signal decoding on the target audio bitstream to obtain a target encoded feature estimation value corresponding to the target audio bitstream; and the feature decoding module 6553 is configured to perform feature decoding on the target encoded feature estimation value to obtain a reconstructed audio signal conforming to the target timbre.

[0349] It should be noted that the audio conversion apparatus according to the embodiments of the present application can reuse the feature encoding module, the determining module, the conversion module, the information encoding module, the signal decoding module, and the feature decoding module in the audio communication apparatus. The following continues to describe a scheme for implementing audio conversion by the modules in the audio conversion apparatus 755 according to the embodiments of the present application.

[0350] The feature encoding module 7551 is configured to encode features of an audio signal to obtain encoded features of the audio signal; the determination module 7552 is configured to obtain a target timbre corresponding to a voice mode, and determine a timbre feature of the target timbre; the conversion module 7553 is configured to perform timbre conversion on the encoded features based on the timbre feature to obtain target encoded features; the information encoding module 7554 is configured to perform signal encoding on the target encoded features to obtain a target audio bitstream conforming to the target timbre; the information decoding module 7555 is configured to perform signal decoding on the target audio bitstream to obtain a target encoded feature estimation value corresponding to the target audio bitstream; and the feature decoding module 7556 is configured to perform feature decoding on the target encoded feature estimation value corresponding to the target audio bitstream to obtain a reconstructed audio signal conforming to the target timbre.

[0351] The computer program product includes a computer program or computer executable instructions stored in a computer readable storage medium. The processor of the electronic device reads the computer program or computer executable instructions from the computer readable storage medium. The processor executes the computer program or computer executable instructions, so that the electronic device executes the audio communication method or the audio conversion method provided in the embodiments of the present application.

[0352] The computer readable storage medium stores computer executable instructions or computer programs. When the computer executable instructions or computer programs are executed by the processor, the processor executes the audio communication method or the audio conversion method provided in the embodiments of the present application, for example, the audio communication method shown in FIG. 4A.

[0353] In some embodiments, the computer readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disc, or CD-ROM; or various devices including one or any combination of the above memories.

[0354] In some embodiments, the computer executable instructions can be in the form of programs, software, software modules, scripts or codes, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and can be deployed in any form, including being deployed as independent programs or being deployed as modules, components, subroutines or other units suitable for use in a computing environment.

[0355] By way of example, computer-executable instructions can include but are not limited to: a) program modules, b) program data, and c) program libraries.

[0356] By way of example, computer-executable instructions can be deployed to be executed on one electronic device or on multiple electronic devices that are located at one site or that are distributed across multiple sites and that are interconnected by a communication network.

[0357] It can be understood that, in the embodiments of the present application, the data related to user information and the like needs to be obtained with the permission or consent of the user, and the collection, use and processing of the related data need to comply with relevant laws, regulations and standards when the embodiments of the present application are applied to specific products or technologies.

[0358] The above merely illustrates the embodiments of the present application, and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement and improvement made within the spirit and scope of the present application shall be included in the protection scope of the present application.

Claims

1. An audio communication method applied to an encoding terminal, the method comprising: obtaining a voice transformation mode for an audio signal from a plurality of communication modes in response to a first communication request for the audio signal; performing feature encoding on the audio signal to obtain encoded features of the audio signal; obtaining a target timbre corresponding to the voice transformation mode and determining timbre features of the target timbre; performing timbre conversion on the encoded features based on the timbre features to obtain target encoded features; performing signal encoding on the target encoded features to obtain a target audio bitstream conforming to the target timbre, and transmitting the target audio bitstream to a decoding terminal. The method further comprises: obtaining an original voice mode for the audio signal from the plurality of communication modes in response to a second communication request for the audio signal; performing signal encoding processing on the encoded features based on the original voice mode to obtain an audio bitstream of the audio signal, and transmitting the audio bitstream to the decoding terminal. The performing timbre conversion on the encoded features based on the timbre features to obtain target encoded features comprises: merging the timbre features and the encoded features to obtain merged features; and performing feature conversion on the merged features to obtain the target encoded features.

4. The method of claim 3, wherein: the feature conversion is implemented by a conversion network, the conversion network comprising a plurality of first convolutional layers and a plurality of conversion encoding blocks, the first convolutional layers and the conversion encoding blocks being connected alternately; the performing feature conversion on the merged features to obtain the target encoded features comprises: performing convolution processing on the merged features by a first first convolutional layer, and performing conversion encoding on a convolution result of the first first convolutional layer obtained by a first conversion encoding block connected alternately with the first first convolutional layer to obtain an encoding result of the first conversion encoding block; performing convolution processing on an encoding result of an (i-1)th conversion encoding block by an i-th first convolutional layer connected alternately with the (i-1)th conversion encoding block, and performing conversion encoding on a convolution result of the i-th first convolutional layer obtained by an i-th conversion encoding block connected alternately with the i-th first convolutional layer to obtain an encoding result of the i-th conversion encoding block, where i is a positive integer that increases sequentially, 1 < i ≤ I, and I is a number of the plurality of first convolutional layers; and the convolution result output by a last first convolutional layer is taken as the target encoded features, or the encoding result of the last conversion encoding block is taken as the target encoded features.

5. The method of claim 4, wherein: each of the conversion encoding blocks comprises a second convolutional layer and at least one residual unit; and the performing conversion encoding on the convolution result of the first first convolutional layer obtained by the first conversion encoding block connected alternately with the first first convolutional layer to obtain the encoding result of the first conversion encoding block comprises: ​ 2. The method of claim 1, wherein, ​ ​ ​ 3. The method according to any one of claims 1-2, wherein, ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ The residual processing is performed on the convolution result of the first convolution layer by at least one residual unit in the first conversion coding block, to obtain residual features of the first conversion coding block. The residual features of the first conversion coding block are convoluted by a second convolution layer in the first conversion coding block, to obtain coding results of the first conversion coding block.

6. The method of claim 5, wherein, When the at least one residual unit is a plurality of cascaded residual units, the residual processing on the convolution result of the first convolution layer by at least one residual unit in the first conversion coding block, to obtain residual features of the first conversion coding block, includes: The residual processing is performed on the convolution result of the first convolution layer by a first residual unit of the plurality of cascaded residual units; The residual result output by the first residual unit is output to a subsequent cascaded residual unit, and the residual processing and the output of the residual result are continued by the subsequent cascaded residual unit; The residual result output by a last residual unit is taken as the residual features of the first conversion coding block.

7. The method of claim 6, wherein, The residual processing on the convolution result of the first convolution layer by a first residual unit of the plurality of cascaded residual units includes: The following processing is performed by the first residual unit of the plurality of cascaded residual units: The convolution processing is performed on the convolution result of the first convolution layer, to obtain a convolution result of the first residual unit; The convolution result of the first residual unit is added to the convolution result of the first convolution layer, to obtain the residual result output by the first residual unit; and / or, The residual processing and the output of the residual result are continued by the subsequent cascaded residual unit, including: The following processing is performed by a jth residual unit of the plurality of cascaded residual units: The convolution processing is performed on the residual result output by a (j-1)th residual unit, to obtain a convolution result of the jth residual unit; The convolution result of the jth residual unit is added to the residual result output by the (j-1)th residual unit, to obtain the residual result output by the jth residual unit; The residual result output by the jth residual unit is output to a (j+1)th residual unit, where j is a positive integer that increases sequentially, 1 8. The method of claim 7, wherein, Each residual unit includes a dilated convolution operator and a causal convolution operator; The convolution processing on the convolution result of the first convolution layer to obtain the convolution result of the first residual unit includes: The dilated convolution processing is performed on the convolution result of the first convolution layer by the dilated convolution operator included in the first residual unit; The causal convolution processing is performed on the dilated convolution result by at least one causal convolution operator included in the first residual unit, and the causal convolution result obtained is taken as the convolution result of the first residual unit.

9. The method of any one of claims 1-8, wherein, The feature encoding of the audio signal obtains the encoded features of the audio signal, and the feature encoding comprises: extracting features of the audio signal to obtain audio features of the audio signal; using at least one residual unit to perform residual processing on the audio features to obtain the encoded features of the audio signal.

10. The method of any one of claims 1-9, wherein, The voice mode has a plurality of candidate timbres, and the target timbre corresponding to the voice mode is obtained by: determining the target timbre from the plurality of candidate timbres.

11. The method of any one of claims 1-10, wherein The audio communication method is implemented by a second encoding network in an audio codec system, the audio codec system further comprising a first encoding network and a first decoding network, the first encoding network and the first decoding network being trained by first audio samples; The second encoding network is trained by: in response to a configuration request for the first encoding network, determining a to-be-trained second encoding network corresponding to the first encoding network, wherein the to-be-trained second encoding network comprises a to-be-trained conversion network; performing audio encoding on second audio samples based on the to-be-trained conversion network to obtain audio bitstream samples of the second audio samples, and performing audio decoding on the audio bitstream samples based on the first decoding network to obtain reconstructed audio samples of the second audio samples; updating parameters of the to-be-trained conversion network based on the reconstructed audio samples to obtain a trained conversion network.

12. An audio communication method applied to a decoding terminal, the method comprising: receiving a target audio bitstream transmitted by an encoding terminal, wherein the target audio bitstream is obtained by performing timbre conversion on an audio signal by a target timbre corresponding to a voice mode; performing signal decoding on the target audio bitstream to obtain a target encoded feature estimate corresponding to the target audio bitstream; performing feature decoding on the target encoded feature estimate to obtain a reconstructed audio signal conforming to the target timbre.

13. An audio conversion method, the method being applied to a terminal, the method comprising: performing feature encoding on an audio signal to obtain encoded features of the audio signal; obtaining a target timbre corresponding to a voice mode and determining timbre features of the target timbre; performing timbre conversion on the encoded features based on the timbre features to obtain target encoded features; performing signal encoding on the target encoded features to obtain a target audio bitstream conforming to the target timbre; performing signal decoding on the target audio bitstream to obtain a target encoded feature estimate corresponding to the target audio bitstream; performing feature decoding on the target encoded feature estimate to obtain a reconstructed audio signal conforming to the target timbre.

14. A method of processing a bitstream, the bitstream being generated according to the audio communication method of any one of claims 1 to 12 or the audio conversion method of claim 13, or being decoded based on the audio communication method of claim 12 or the audio conversion method of claim 13.

15. An audio communication device, the device comprising: The acquisition module is configured to acquire a voice transformation mode for the audio signal from a plurality of communication modes in response to a first communication request for the audio signal; The feature encoding module is configured to perform feature encoding on the audio signal to obtain encoded features of the audio signal; The determination module is configured to obtain a target timbre corresponding to the voice transformation mode, and determine timbre features of the target timbre; The conversion module is configured to perform timbre conversion on the encoded features based on the timbre features to obtain target encoded features; The information encoding module is configured to perform signal encoding on the target encoded features to obtain a target audio bitstream conforming to the target timbre, and transmit the target audio bitstream to a decoding terminal.

16. An audio communication device, the device comprising: The receiving module is configured to receive a target audio bitstream transmitted by an encoding terminal, wherein the target audio bitstream is obtained by performing timbre conversion on an audio signal based on a target timbre corresponding to a voice transformation mode; The information decoding module is configured to perform signal decoding on the target audio bitstream to obtain target encoded feature estimates corresponding to the target audio bitstream; The feature decoding module is configured to perform feature decoding on the target encoded feature estimates to obtain a reconstructed audio signal conforming to the target timbre.

17. An audio conversion device, the device comprising: The feature encoding module is configured to perform feature encoding on an audio signal to obtain encoded features of the audio signal; The determination module is configured to obtain a target timbre corresponding to a voice transformation mode, and determine timbre features of the target timbre; The conversion module is configured to perform timbre conversion on the encoded features based on the timbre features to obtain target encoded features; The information encoding module is configured to perform signal encoding on the target encoded features to obtain a target audio bitstream conforming to the target timbre; The information decoding module is configured to perform signal decoding on the target audio bitstream to obtain target encoded feature estimates corresponding to the target audio bitstream; The feature decoding module is configured to perform feature decoding on the target encoded feature estimates to obtain a reconstructed audio signal conforming to the target timbre.

18. An electronic device, the electronic device comprising: A memory configured to store computer programs or computer executable instructions; A processor configured to execute the computer programs or computer executable instructions stored in the memory to implement the audio communication method of any one of claims 1 to 12, or the audio conversion method of claim 13.

19. A computer readable storage medium storing computer programs or computer executable instructions, the computer programs or computer executable instructions being executed by a processor to implement the audio communication method of any one of claims 1 to 12, or the audio conversion method of claim 13.

20. A computer readable storage medium storing a bitstream formed by computer programs or computer executable instructions, the computer programs or computer executable instructions being executed by a processor to implement the audio communication method of any one of claims 1 to 12, or the audio conversion method of claim 13.

21. A computer program product comprising computer programs or computer executable instructions which, when executed by a processor, implement the audio communication method of any one of claims 1 to 12, or the audio conversion method of claim 13.

Images