Audio processing method and device and storage medium

Abstract

The invention relates to an audio processing method and device and a storage medium, and the method comprises the steps: determining a first historical information length of a first model, the first historical information length comprises a plurality of pieces of first information, the first information is used for the first model to compress second information, and the first information is earlier than the second information; in the above embodiment, the problem that the information used for compression cannot be determined is solved, the information used for compression within a certain time length is determined, the accuracy of compression by adopting the information is ensured, the reliability of information compression is further ensured, and the reliability of communication is further ensured.

Description

Technical Field

[0001] The present disclosure relates to the technical field of audio processing, and in particular to an audio processing method, device and storage medium. Background Art

[0002] At present, speech recognition technology is developing rapidly. After obtaining audio data, speech recognition can be performed on the audio data to facilitate the understanding of the instructions of the audio data and ensure the accuracy of recognition. Furthermore, speech recognition technology can be applied in the fields of medicine, vehicles, terminals, etc. Summary of the invention

[0003] The solution provided by the present disclosure solves the problem of data loss caused by the compression of audio data by the encoder. The embodiment of the present disclosure extracts features from the audio data and then compresses the features, and then reuses the compressed features to obtain mixed features. This not only reduces the amount of data, but also ensures a low degree of data loss through feature extraction, compression and mixing, thereby ensuring the accuracy of the encoder in encoding the data.

[0004] The embodiments of the present disclosure provide an audio processing method, an audio processing device, and a storage medium.

[0005] According to a first aspect of an embodiment of the present disclosure, an audio processing method is proposed. The method is performed by an encoder, and the method includes:

[0006] Performing feature extraction on multiple frames of audio data to obtain multiple audio features of each frame of the audio data;

[0007] Compressing each of the multiple audio features respectively to obtain multiple compressed features;

[0008] The multiple compressed features are multiplexed to obtain mixed features.

[0009] According to a second aspect of an embodiment of the present disclosure, an audio processing method is proposed, the method being executed by a decoder, the method comprising:

[0010] Demultiplexing the mixed features to obtain a plurality of compressed features, wherein the mixed features are obtained based on audio features of multiple frames of audio data;

[0011] Decompressing each compressed feature of the plurality of compressed features to obtain a plurality of decompressed features;

[0012] Speech recognition is performed based on the multiple decompressed features.

[0013] According to a third aspect of an embodiment of the present disclosure, an audio processing method is proposed, the method comprising:

[0014] The encoder performs feature extraction on multiple frames of audio data to obtain multiple audio features of each frame of audio data;

[0015] The encoder compresses each of the multiple audio features respectively to obtain multiple compressed features;

[0016] The encoder multiplexes the multiple compressed features to obtain mixed features;

[0017] The decoder demultiplexes the mixed features to obtain a plurality of compressed features, wherein the mixed features are obtained based on the audio features of the multiple frames of audio data;

[0018] The decoder decompresses each compressed feature of the plurality of compressed features to obtain a plurality of decompressed features;

[0019] The decoder performs speech recognition based on the multiple decompressed features.

[0020] According to a fourth aspect of an embodiment of the present disclosure, an audio processing device is provided, comprising:

[0021] A processing module, used for extracting features from multiple frames of audio data to obtain multiple audio features of each frame of audio data;

[0022] The processing module is further used to compress each of the multiple audio features respectively to obtain multiple compressed features;

[0023] The processing module is also used to multiplex the multiple compressed features to obtain mixed features.

[0024] According to a fifth aspect of an embodiment of the present disclosure, an audio processing device is provided, including:

[0025] A processing module, used for demultiplexing the mixed features to obtain a plurality of compressed features, wherein the mixed features are obtained based on the audio features of the multiple frames of audio data;

[0026] The processing module is further used to decompress each compressed feature of the multiple compressed features to obtain multiple decompressed features;

[0027] The processing module is further used to perform speech recognition based on the multiple decompressed features.

[0028] According to a sixth aspect of an embodiment of the present disclosure, an encoder is provided, comprising:

[0029] one or more processors;

[0030] The encoder is used to execute any method described in the first aspect.

[0031] According to a seventh aspect of the embodiments of the present disclosure, a decoder is provided, including:

[0032] one or more processors;

[0033] The decoder is used to execute any method described in the second aspect.

[0034] According to an eighth aspect of an embodiment of the present disclosure, a communication system is provided, including:

[0035] An encoder and a decoder, wherein the encoder is configured to implement the audio processing method described in the first aspect, and the decoder is configured to implement the audio processing method described in the second aspect.

[0036] According to a ninth aspect of an embodiment of the present disclosure, a storage medium is proposed, wherein the storage medium stores instructions, and when the instructions are executed on a communication device, the communication device executes a method as described in any one of the first aspect or the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings described herein are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the present disclosure. The illustrative embodiments of the embodiments of the present disclosure and their descriptions are used to explain the embodiments of the present disclosure and do not constitute an improper limitation on the embodiments of the present disclosure. In the drawings:

[0038] Figure 1 is a schematic diagram of the architecture of a communication system according to an embodiment of the present disclosure;

[0039] Figure 2A is an interactive schematic diagram of an audio processing method according to an embodiment of the present disclosure;

[0040] Figure 2B is a schematic structural diagram of a hybrid feature according to an embodiment of the present disclosure;

[0041] Figure 2C is a schematic structural diagram of a hybrid feature according to an embodiment of the present disclosure;

[0042] Figure 2D is a schematic structural diagram of a hybrid feature according to an embodiment of the present disclosure;

[0043] Figure 2E is a schematic diagram of a coding process according to an embodiment of the present disclosure;

[0044] Figure 2F is a schematic diagram of a decoding process according to an embodiment of the present disclosure;

[0045] Figure 3A is a flowchart of an audio processing method according to an embodiment of the present disclosure;

[0046] Figure 3B is a flowchart of an audio processing method according to an embodiment of the present disclosure;

[0047] Figure 4 is a flowchart of an audio processing method according to an embodiment of the present disclosure;

[0048] Figure 5 is a flowchart of an audio processing method according to an embodiment of the present disclosure;

[0049] Figure 6 is a flowchart of an audio processing method according to an embodiment of the present disclosure;

[0050] Fig. 7A is a structural schematic diagram of an audio processing device proposed in an embodiment of the present disclosure;

[0051] Figure 7B is a structural schematic diagram of an audio processing device proposed in an embodiment of the present disclosure;

[0052] Fig. 8A is a schematic diagram of the structure of a communication device proposed in an embodiment of the present disclosure;

[0053] Figure 8B It is a schematic diagram of the structure of the chip proposed in the embodiment of the present disclosure. DETAILED DESCRIPTION

[0054] The present disclosure provides an audio processing method, an audio processing device, and a storage medium.

[0055] According to a first aspect of an embodiment of the present disclosure, an audio processing method is proposed. The method is performed by an encoder, and the method includes:

[0056] Performing feature extraction on multiple frames of audio data to obtain multiple audio features of each frame of the audio data;

[0057] Compressing each of the multiple audio features respectively to obtain multiple compressed features;

[0058] The multiple compressed features are multiplexed to obtain mixed features.

[0059] In the above embodiment, the problem of data loss caused by the encoder compressing the audio data is solved. The embodiment of the present disclosure extracts features from the audio data and then compresses the features, and then reuses the compressed features to obtain mixed features. This not only reduces the amount of data, but also ensures a low degree of data loss through feature extraction, compression and mixing, thereby ensuring the accuracy of the encoder in encoding the data.

[0060] In combination with some embodiments of the first aspect, in some embodiments, the method further includes:

[0061] The mixed features are sent to a decoder, and are used for decoding and performing audio recognition.

[0062] In combination with some embodiments of the first aspect, in some embodiments, respectively compressing each of the multiple audio features to obtain multiple compressed features includes:

[0063] Each audio data is compressed using the compression algorithm of each audio feature to obtain the multiple compression features.

[0064] In the above embodiment, different audio features may correspond to different compression algorithms. Compression is performed using the compression algorithm corresponding to the audio feature to obtain the compressed feature, thereby ensuring the accuracy of the obtained compressed feature.

[0065] In combination with some embodiments of the first aspect, in some embodiments, the types of the audio features correspond one-to-one to the compression algorithms of the audio features.

[0066] In the above embodiment, the types of audio features correspond to compression algorithms one by one, which ensures the accuracy of the compression algorithm corresponding to the audio feature determined by the encoder, thereby ensuring the accuracy of compressing the audio feature.

[0067] In combination with some embodiments of the first aspect, in some embodiments, the method further includes:

[0068] Based on the similarity between each audio feature and the audio features of the audio data of the adjacent frames, a compression algorithm corresponding to each audio feature is determined.

[0069] In combination with some embodiments of the first aspect, in some embodiments, the similarity range to which the similarity belongs has a corresponding relationship with the compression algorithm.

[0070] In the above embodiment, the corresponding compression algorithm is determined according to the similarity between the audio feature and the audio feature of the adjacent audio data, so as to ensure the accuracy of the determined compression algorithm.

[0071] In combination with some embodiments of the first aspect, in some embodiments, the method further includes:

[0072] Sending indication information to the decoder, where the indication information is used to indicate a compression algorithm corresponding to each audio feature.

[0073] In the above embodiment, the decoder is informed of the compression algorithm used for each audio feature through indication information, so that both parties can determine the compression algorithm or decompression algorithm, thereby ensuring the accuracy of processing the audio feature.

[0074] In combination with some embodiments of the first aspect, in some embodiments, the audio feature is associated with a frequency, and a compression algorithm of the audio feature is nonlinear quantization.

[0075] In conjunction with some embodiments of the first aspect, in some embodiments, the compression algorithm includes at least one of the following:

[0076] PCA (Principal Components Analysis);

[0077] Inter-frame prediction;

[0078] Lossy compression;

[0079] Lossless compression;

[0080] Quantization compression.

[0081] In conjunction with some embodiments of the first aspect, in some embodiments, the quantization compression includes at least one of the following:

[0082] Delta modulation;

[0083] Adaptive Differential Pulse Code Modulation;

[0084] SDM (Sigma Modulator) modulation;

[0085] Vector quantization;

[0086] Linear quantization;

[0087] Nonlinear quantization.

[0088] In the above embodiment, the types of compression algorithms are expanded to ensure the comprehensiveness of the compression of audio features, thereby ensuring the accuracy of the compression.

[0089] In combination with some embodiments of the first aspect, in some embodiments, extracting features from multiple frames of audio data to obtain multiple audio features of each frame of audio data includes:

[0090] Determining the types of the multiple audio features obtained by extracting features from the multiple frames of audio data based on first information; the first information is used to indicate the type of audio features; or,

[0091] The types of the multiple audio features obtained by performing feature extraction on the multiple frames of audio data are determined based on the preconfiguration.

[0092] In the above embodiments, different schemes for determining the types of extracted audio features are provided to ensure the accuracy of the extracted audio features.

[0093] In combination with some embodiments of the first aspect, in some embodiments, the audio feature includes at least one of the following:

[0094] Acoustic characteristics;

[0095] Voice features;

[0096] Prosodic features;

[0097] Spectral characteristics;

[0098] Timing characteristics;

[0099] Resonant peak characteristics;

[0100] Noise characteristics;

[0101] User characteristics;

[0102] Voice features;

[0103] Contextual features.

[0104] In conjunction with some embodiments of the first aspect, in some embodiments, the mixing feature includes at least one of the following:

[0105] A synchronization field, wherein the synchronization field is used to indicate a start frame of the audio data;

[0106] A flag field, the flag field is used to indicate whether a corresponding audio feature exists in the data field;

[0107] The length field is used to indicate the length of the data field;

[0108] Data fields;

[0109] A check field is used to check whether the hybrid feature is erroneous when being sent.

[0110] In combination with some embodiments of the first aspect, in some embodiments, the flag field includes a first-level flag, and each of the audio features has one first-level flag; or,

[0111] The flag field includes a secondary flag, and the secondary flag includes a first flag and a second flag; each group has a first flag, and the audio features included in each group have a second flag, and each group includes the same type of audio features.

[0112] In the above embodiment, the audio features are indicated by different signs to ensure the accuracy of the indicated audio features.

[0113] In a second aspect, an embodiment of the present disclosure provides an audio processing method, the method being executed by a decoder, the method comprising:

[0114] Demultiplexing the mixed features to obtain a plurality of compressed features, wherein the mixed features are obtained based on audio features of multiple frames of audio data;

[0115] Decompressing each compressed feature of the plurality of compressed features to obtain a plurality of decompressed features;

[0116] Speech recognition is performed based on the multiple decompressed features.

[0117] In conjunction with some embodiments of the second aspect, in some embodiments, the method further includes:

[0118] The mixed feature sent by the encoder is received.

[0119] In combination with some embodiments of the second aspect, in some embodiments, decompressing each compressed feature of the multiple compressed features to obtain multiple decompressed features includes:

[0120] Decompress each compressed feature using the decompression algorithm of each compressed feature to obtain the multiple decompressed features.

[0121] In combination with some embodiments of the second aspect, in some embodiments, the type of the compression feature corresponds one-to-one to the decompression algorithm of the compression feature.

[0122] In conjunction with some embodiments of the second aspect, in some embodiments, the method further includes:

[0123] Indication information sent by the encoder is received, where the indication information is used to indicate a decompression algorithm corresponding to each compression feature.

[0124] In combination with some embodiments of the second aspect, in some embodiments, the compression feature is associated with frequency, and the decompression algorithm of the compression feature is nonlinear decompression.

[0125] In conjunction with some embodiments of the second aspect, in some embodiments, the decompression algorithm includes at least one of the following:

[0126] PCA;

[0127] Inter-frame prediction decompression;

[0128] Lossy decompression;

[0129] Lossless decompression;

[0130] Inverse quantization.

[0131] In conjunction with some embodiments of the second aspect, in some embodiments, the audio feature includes at least one of the following:

[0132] Acoustic characteristics;

[0133] Voice features;

[0134] Prosodic features;

[0135] Spectral characteristics;

[0136] Timing characteristics;

[0137] Resonant peak characteristics;

[0138] Noise characteristics;

[0139] User characteristics;

[0140] Voice features;

[0141] Contextual features.

[0142] In conjunction with some embodiments of the second aspect, in some embodiments, the mixing feature includes at least one of the following:

[0143] A synchronization field, wherein the synchronization field is used to indicate a start frame of the audio data;

[0144] A flag field, the flag field is used to indicate whether a corresponding audio feature exists in the data field;

[0145] The length field is used to indicate the length of the data field;

[0146] Data fields;

[0147] A check field is used to check whether the hybrid feature is erroneous when being sent.

[0148] In combination with some embodiments of the second aspect, in some embodiments, the flag field includes a first-level flag, and each of the audio features has one first-level flag; or,

[0149] The flag field includes a secondary flag, and the secondary flag includes a first flag and a second flag; each group has a first flag, and the audio features included in each group have a second flag, and each group includes the same type of audio features.

[0150] In a third aspect, an embodiment of the present disclosure provides an audio processing method, the method comprising:

[0151] The encoder performs feature extraction on multiple frames of audio data to obtain multiple audio features of each frame of audio data;

[0152] The encoder compresses each of the multiple audio features respectively to obtain multiple compressed features;

[0153] The encoder multiplexes the multiple compressed features to obtain mixed features;

[0154] The decoder demultiplexes the mixed features to obtain a plurality of compressed features, wherein the mixed features are obtained based on the audio features of the multiple frames of audio data;

[0155] The decoder decompresses each compressed feature of the plurality of compressed features to obtain a plurality of decompressed features;

[0156] The decoder performs speech recognition based on the multiple decompressed features.

[0157] In a fourth aspect, an embodiment of the present disclosure provides an audio processing device, which includes at least one of a transceiver module and a processing module; wherein the terminal is used to execute the optional implementation method of the first aspect.

[0158] In a fifth aspect, an embodiment of the present disclosure provides an audio processing device, which includes at least one of a transceiver module and a processing module; wherein the terminal is used to execute the optional implementation method of the second aspect.

[0159] In a sixth aspect, an embodiment of the present disclosure provides an encoder, including:

[0160] one or more processors;

[0161] The encoder is used to execute the method described in any one of the first aspects.

[0162] In a seventh aspect, an embodiment of the present disclosure provides a decoder, including:

[0163] one or more processors;

[0164] The decoder is used to execute any method described in the second aspect.

[0165] In an eighth aspect, an embodiment of the present disclosure provides a storage medium, wherein the storage medium stores first information, and when the first information is run on a communication device, the communication device executes a method as described in any one of the first aspect or the second aspect.

[0166] In a ninth aspect, an embodiment of the present disclosure proposes a program product. When the program product is executed by a communication device, the communication device executes any one of the methods described in the first aspect or the second aspect.

[0167] In a tenth aspect, an embodiment of the present disclosure proposes a computer program, which, when executed on a communication device, enables the communication device to execute any method described in the first aspect or the second aspect.

[0168] In an eleventh aspect, an embodiment of the present disclosure provides a chip or a chip system, wherein the chip or the chip system comprises a processing circuit configured to execute any one of the methods described in the first aspect or the second aspect.

[0169] It is understandable that the above-mentioned terminal, storage medium, program product, computer program, chip or chip system are all used to execute the method proposed in the embodiment of the present disclosure. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method, which will not be repeated here.

[0170] The embodiments of the present disclosure provide an audio processing method, device, and storage medium. In some embodiments, the terms audio processing method, information processing method, data processing method, etc. can be interchangeable, the terms audio processing device, information processing device, data processing device, etc. can be interchangeable, and the terms information processing system, communication system, etc. can be interchangeable.

[0171] The embodiments of the present disclosure are not exhaustive, but are only illustrative of some embodiments, and are not intended to be a specific limitation on the scope of protection of the present disclosure. In the absence of contradiction, each step in a certain embodiment can be implemented as an independent embodiment, and the steps can be arbitrarily combined. For example, a solution after removing some steps in a certain embodiment can also be implemented as an independent embodiment, and the order of the steps in a certain embodiment can be arbitrarily exchanged. In addition, the optional implementation methods in a certain embodiment can be arbitrarily combined; in addition, the embodiments can be arbitrarily combined, for example, some or all of the steps of different embodiments can be arbitrarily combined, and a certain embodiment can be arbitrarily combined with the optional implementation methods of other embodiments.

[0172] In each embodiment of the present disclosure, unless otherwise specified or there is a logical conflict, the terms and / or descriptions between the embodiments are consistent and can be referenced to each other, and the technical features in different embodiments can be combined to form a new embodiment based on their internal logical relationships.

[0173] The terms used in the embodiments of the present disclosure are only for the purpose of describing specific embodiments and are not intended to limit the present disclosure.

[0174] In the embodiments of the present disclosure, unless otherwise specified, elements expressed in the singular form, such as "a", "an", "the", "above", "said", "aforementioned", "this", etc., may mean "one and only one", or "one or more", "at least one", etc. For example, when using articles such as "a", "an", "the" in English in translation, the noun after the article may be understood as a singular expression or a plural expression.

[0175] In the embodiments of the present disclosure, “plurality” refers to two or more.

[0176] In some embodiments, the terms “at least one,” “one or more,” “a plurality of,” “multiple,” etc. may be used interchangeably.

[0177] In some embodiments, "at least one of A and B", "A and / or B", "A in one case, B in another case", "in response to one case A, in response to another case B", etc., may include the following technical solutions according to the situation: in some embodiments, A (A is executed independently of B); in some embodiments, B (B is executed independently of A); in some embodiments, execution is selected from A and B (A and B are selectively executed); in some embodiments, A and B (both A and B are executed). When there are more branches such as A, B, C, etc., the above is also similar.

[0178] In some embodiments, the recording method of "A or B" may include the following technical solutions according to the situation: in some embodiments, A (A is executed independently of B); in some embodiments, B (B is executed independently of A); in some embodiments, execution is selected from A and B (A and B are selectively executed). When there are more branches such as A, B, C, etc., the above is also similar.

[0179] The prefixes such as "first" and "second" in the embodiments of the present disclosure are only used to distinguish different description objects, and do not constitute restrictions on the position, order, priority, quantity or content of the description objects. The statement of the description object refers to the description in the context of the claims or embodiments, and should not constitute unnecessary restrictions due to the use of prefixes. For example, if the description object is a "field", the ordinal number before the "field" in the "first field" and the "second field" does not limit the position or order between the "fields", and the "first" and "second" do not limit whether the "fields" they modify are in the same message, nor do they limit the order of the "first field" and the "second field". For another example, if the description object is a "level", the ordinal number before the "level" in the "first level" and the "second level" does not limit the priority between the "levels". For another example, the number of description objects is not limited by the ordinal number, and can be one or more. Taking the "first device" as an example, the number of "devices" can be one or more. In addition, the objects modified by different prefixes may be the same or different. For example, if the description object is "device", then the "first device" and the "second device" may be the same device or different devices, and their types may be the same or different. For another example, if the description object is "information", then the "first information" and the "second information" may be the same information or different information, and their contents may be the same or different.

[0180] In some embodiments, “including A”, “comprising A”, “used to indicate A”, and “carrying A” can be interpreted as directly carrying A or indirectly indicating A.

[0181] In some embodiments, terms such as "time / frequency", "time / frequency domain", etc. refer to the time domain and / or the frequency domain.

[0182] In some embodiments, terms such as "in response to ...", "in response to determining ...", "in the case of ...", "at the time of ...", "when ...", "if ...", "if ...", etc. can be used interchangeably.

[0183] In some embodiments, terms such as "greater than", "greater than or equal to", "not less than", "more than", "more than or equal to", "not less than", "higher than", "higher than or equal to", "not lower than", and "above" can be replaced with each other, and terms such as "less than", "less than or equal to", "not greater than", "less than", "less than or equal to", "no more than", "lower than", "lower than or equal to", "not higher than", and "below" can be replaced with each other.

[0184] In some embodiments, devices and equipment may be interpreted as physical or virtual, and their names are not limited to the names recorded in the embodiments. In some cases, they may also be understood as "equipment", "device", "circuit", "network element", "node", "function", "unit", "section", "system", "network", "chip", "chip system", "entity", "subject", etc.

[0185] In some embodiments, "network" can be interpreted as devices included in the network, such as access network equipment, core network equipment, etc.

[0186] In some embodiments, "access network device (AN device)" may also be referred to as "radio access network device (RAN device)", "base station (BS)", "radio base station (radio basestation)", "fixed station (fixed station)", and in some embodiments may also be understood as "node (node)", "access point (access point)", "transmission point (TP)", "reception point (reception point, RP)", "transmission and / or reception point (transmission / reception point, TRP)" "panel (panel)", "antenna panel (antenna panel)", "antenna array (antenna array)" "cell (cell)", "macro cell (macro cell)", "small cell (small cell)", "femto cell (femto cell)", "pico cell (pico cell)", "sector (sector)", "cell group (cell group)", "serving cell (serving cell)", "carrier (carrier)", "component carrier (component carrier)", "bandwidth part (bandwidth part, BWP)" and the like.

[0187] In some embodiments, "terminal" or "terminal device" can be referred to as "user equipment (terminal)", "user terminal" "mobile station (MS)", "mobile terminal (MT)", subscriber station (subscriber station), mobile unit (mobile unit), subscriber unit (subscriber unit), wireless unit (wireless unit), remote unit (remote unit), mobile device (mobile device), wireless device (wireless device), wireless communication device (wireless communication device), remote device (remote device), mobile subscriber station (mobile subscriber station), access terminal (access terminal), mobile terminal (mobile terminal), wireless terminal (wireless terminal), remote terminal (remote terminal), handset (handset), user agent (user agent), mobile client (mobile client), client (client), etc.

[0188] In some embodiments, acquisition of data, information, etc. may comply with the laws and regulations of the country where the data is obtained.

[0189] In some embodiments, data, information, etc. may be obtained with the user's consent.

[0190] In addition, each element, each row, or each column in the table of the embodiments of the present disclosure may be implemented as an independent embodiment, and the combination of any elements, any rows, and any columns may also be implemented as an independent embodiment.

[0191] Figure 1 is a schematic diagram of the architecture of a communication system according to an embodiment of the present disclosure, such as Figure 1 As shown, the method provided in the embodiment of the present disclosure can be applied to a communication system 100, which may include an encoder 101 and a decoder 102. It should be noted that the communication system 100 may also include other devices, and the present disclosure does not limit the devices included in the communication system 100.

[0192] In some embodiments, the encoder and decoder are both terminals. The terminal includes, for example, a mobile phone, a wearable device, an Internet of Things device, a car with communication function, a smart car, a tablet computer, a computer with wireless transceiver function, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal device in industrial control, a wireless terminal device in self-driving, a wireless terminal device in remote medical surgery, a wireless terminal device in smart grid, a wireless terminal device in transportation safety, a wireless terminal device in smart city, and at least one of a wireless terminal device in smart home, but is not limited thereto.

[0193] In some embodiments, the communication system provided by the embodiments of the present disclosure is applied to the field of speech recognition. For example, the encoder collects audio data and the decoder recognizes the audio data.

[0194] Alternatively, in the medical field, speech recognition technology is used to transcribe medical records, thereby reducing the workload of healthcare professionals and improving efficiency.

[0195] Optionally, Automotive: In the automotive industry, the technology is used in navigation systems and hands-free mobile connectivity solutions to enhance driver safety and convenience.

[0196] Optionally, consumer electronics: devices such as smartphones, smart TVs, and home automation systems use voice recognition for voice commands, improving the user experience.

[0197] Optionally, call centers: Speech recognition technology is used to record and analyze customer calls, helping to improve customer service and efficiency.

[0198] Optionally, assistive technology: For people with disabilities, speech recognition can provide an alternative way to interact with technology.

[0199] It can be understood that the communication system described in the embodiment of the present disclosure is for the purpose of more clearly illustrating the technical solution of the embodiment of the present disclosure, and does not constitute a limitation on the technical solution proposed in the embodiment of the present disclosure. A person of ordinary skill in the art can know that with the evolution of the system architecture and the emergence of new business scenarios, the technical solution proposed in the embodiment of the present disclosure is also applicable to similar technical problems.

[0200] The following embodiments of the present disclosure can be applied to Figure 1 The communication system 100, or a portion thereof, is shown but is not limited thereto. Figure 1 The various entities shown are examples, and the communication system may include Figure 1 All or part of the subject, and may also include Figure 1 The number and form of other entities are arbitrary, each entity can be physical or virtual, the connection relationship between the entities is illustrative, the entities can be connected or disconnected, the connection can be in any way, it can be direct or indirect, and it can be wired or wireless.

[0201] Figure 2A FIG. 1 is an interactive schematic diagram of an audio processing method according to an embodiment of the present disclosure. Figure 2A As shown, the embodiment of the present disclosure relates to an audio processing method, and the above method includes:

[0202] Step S2101: the encoder obtains multiple frames of audio data.

[0203] In some embodiments, the audio data is collected by an encoder. In some embodiments, the audio data is collected by a collection device and then sent to the encoder.

[0204] In some embodiments, the encoder acquires a frame of audio data at a certain time interval. Optionally, the certain time interval is set by the encoder or agreed upon by a protocol, which is not limited in the embodiments of the present disclosure.

[0205] In some embodiments, the audio data is voice instructions, voice information, interactive information, etc., which is not limited in the embodiments of the present disclosure.

[0206] Step S2102: the encoder extracts features from multiple frames of audio data to obtain multiple audio features for each frame of audio data.

[0207] In the disclosed embodiment, the encoder may extract features of the audio data to obtain audio features of the audio data.

[0208] In some embodiments, since there are multiple types of features in the audio data, different types of audio features can be extracted from the audio data. Optionally, the audio feature includes at least one of the following:

[0209] (1) Acoustic characteristics.

[0210] In some embodiments, the acoustic features include parameters such as Mel-Frequency Cepstral Coefficients (MFCC), Linear Predictive Coding (LPC), Perceptual Linear Prediction (PLP), etc.

[0211] (2) Voice features.

[0212] In some embodiments, the speech features are related to the speech content of the audio data, including parameters such as phonemes, syllables, and words.

[0213] (3) Rhythmic features.

[0214] In some embodiments, the prosodic features are related to the rhythm, stress and intonation of speech. The prosodic features include parameters such as pitch, loudness, duration, etc.

[0215] (4) Spectral characteristics.

[0216] In some embodiments, the spectral features are related to the frequency content of the speech signal, and the spectral features include parameters such as spectral centroid, spectral flux, spectral roll-off, etc.

[0217] (5) Timing characteristics.

[0218] In some embodiments, the timing features are related to the timing information in the speech signal, including parameters such as zero-crossing rate and energy.

[0219] (6) Resonance peak characteristics.

[0220] In some embodiments, the formant characteristics are related to the resonant frequency of the vocal tract, including parameters such as formant frequency and bandwidth.

[0221] (7) Noise characteristics.

[0222] In some embodiments, the noise characteristics are related to the background noise in the speech signal, including parameters such as signal-to-noise ratio and noise energy.

[0223] (8)User characteristics.

[0224] In some embodiments, the user characteristics are related to personal characteristics of the user, including parameters such as the user's identity, gender, and age.

[0225] (9) Voice features.

[0226] In some embodiments, the speech characteristics are related to the language used, including parameters such as language characteristics, dialects, accents, etc.

[0227] (10)Contextual characteristics.

[0228] In some embodiments, the contextual features are related to the context in which the speech is generated, including parameters such as ambient noise, microphone type, and distance from the microphone.

[0229] In some embodiments, the encoder determines the types of the multiple audio features obtained by extracting features from the multiple frames of audio data based on the first information. Optionally, the first information is used to indicate the types of audio features. In the disclosed embodiment, the decoder sends the first information to the encoder, and the first information indicates the types of audio features required by the encoder for feature extraction. The subsequent encoder can extract features from each frame of audio data according to the indication of the first information to obtain audio features of the corresponding types of audio features. For example, if the first information indicates that the types of audio features include the above (1), (2), (6) and (9), the encoder extracts four types of audio features (1), (2), (6) and (9) when extracting features from the audio data. For another example, if the first information indicates that the types of audio features include the above (2), (5), (6) and (10), the encoder extracts four types of audio features (2), (5), (6) and (10) when extracting features from the audio data.

[0230] In some embodiments, the first information is sent by a cloud device. Optionally, the cloud device includes a decoder or a backend device. In some embodiments, the first information is sent by a decoder, or the first information is sent by a backend device. Optionally, the backend device is connected to the decoder and is used to perform speech recognition based on the features decoded by the decoder, and then perform operations corresponding to the content obtained by the voice device. In some embodiments, the first information may also be referred to as feedback information, indication information, etc., which is not limited in the embodiments of the present disclosure.

[0231] In some embodiments, the encoder determines the types of the multiple audio features obtained by extracting features from the multiple frames of audio data based on preconfiguration. In the embodiments of the present disclosure, the encoder preconfigures the types of audio features used when extracting features from audio data in advance. When the encoder needs to extract features from audio data, it extracts features from the audio data according to the preconfigured types of audio features to obtain audio features of the corresponding types. For example, if the encoder preconfigures the types of audio features to include the above (1), (2), (6) and (9), the encoder extracts four types of audio features (1), (2), (6) and (9) when extracting features from audio data.

[0232] Step S2103: The encoder compresses each of the multiple audio features respectively to obtain multiple compressed features.

[0233] In the embodiment of the present disclosure, after extracting the audio features of the audio data, each audio feature may be compressed to obtain a compressed feature. In this way, the amount of data may be reduced and the transmission efficiency may be improved.

[0234] In some embodiments, the encoder compresses each of the multiple audio features included in each frame of audio data to obtain multiple compressed features.

[0235] In some embodiments, the encoder compresses each audio data using a compression algorithm for each audio feature to obtain the multiple compressed features. Optionally, the compression algorithm is used to compress the audio features. In some embodiments, the compression algorithm includes at least one of the following:

[0236] (1) PCA.

[0237] In some embodiments, PCA is a statistical procedure that uses an orthogonal transformation to transform the observed values ​​of a set of possibly correlated variables into a set of values ​​of linearly uncorrelated variables called principal components. Alternatively, it can be understood that the first principal component has the largest possible variance (i.e., explains as much of the variability in the data as possible), and each subsequent component has the highest possible variance under the constraint of being orthogonal to the previous component.

[0238] (2) Inter-frame prediction.

[0239] In some embodiments, in audio coding, inter-frame prediction can predict the spectral content of a frame based on the spectral content of the previous frame. This is based on the observation that the spectral content of the audio signal changes slowly over time, so the spectral content of a frame is likely to be similar to the spectral content of the previous frame. The prediction can be performed in a variety of ways. Optionally, one method is linear prediction, in which the spectral content of a frame is predicted as a linear combination of the spectral content of the previous frame. The coefficients of the linear combination can be estimated based on the correlation between the spectral content of different frames. Once the prediction is made, the difference between the actual spectral content and the prediction (the prediction error) is quantized and transmitted.

[0240] (3) Lossy compression.

[0241] In some embodiments, lossy compression works by identifying and removing patterns that occur in audio files. These patterns can be simple, like repeated sounds or silence, or more complex, like repeated melodies or rhythms. Once these patterns are identified, an algorithm replaces them with a shorter code that represents the pattern. This reduces the size of the file while still perfectly reconstructing the original audio when the file is decompressed.

[0242] (4) Lossless compression.

[0243] In some embodiments, some lossless audio compression algorithms also use a technique called "entropy coding" which further reduces file size by encoding the data in a way that reflects the probability of each sound or pattern occurring.

[0244] (5)Quantization compression.

[0245] Optionally, the quantization compression includes at least one of the following:

[0246] 1. Incremental modulation.

[0247] In some embodiments, delta modulation quantizes only the difference between consecutive samples, rather than the absolute value of each sample. This may result in a lower bit rate, but may also introduce more noise.

[0248] 2. Adaptive differential pulse code modulation.

[0249] In some embodiments, the quantization level is adjusted according to the characteristics of the signal. This can produce a higher quality audio signal at a lower bit rate.

[0250] 3. SDM modulation.

[0251] In some embodiments, SDM modulation uses a feedback loop to continuously adjust the quantization level based on the error of the previous sample. This can produce very high quality audio signals, but requires more processing power.

[0252] 4. Vector quantization.

[0253] In some embodiments, multiple samples are grouped together and quantized into a single unit. This may result in a lower bit rate, but may also introduce more noise.

[0254] 5. Linear quantization.

[0255] In some embodiments, each sample is rounded to the nearest value within a set range. This may result in a high bit rate and may introduce quantization noise.

[0256] 6. Nonlinear quantization.

[0257] In some embodiments, the quantization level is adjusted according to the amplitude of the signal. This can result in lower bit rates and less quantization noise, but may also introduce distortion.

[0258] Optionally, the type of the audio feature corresponds to the compression algorithm of the audio feature. In the embodiment of the present disclosure, the type of audio feature corresponds to the compression algorithm of the audio feature, and each audio feature corresponds to a compression algorithm. After determining what kind of audio feature needs to be compressed, the audio feature can be compressed based on the compression algorithm corresponding to the type of audio feature. For example, acoustic features correspond to linear quantization, speech features correspond to nonlinear quantization, rhythmic features, spectral features and timing features correspond to vector quantization, resonance peak features, noise characteristics, user features and speech features correspond to SDM modulation, and context features correspond to incremental modulation.

[0259] Optionally, the encoder determines the corresponding compression algorithm based on the precision corresponding to the audio feature. Optionally, if the precision corresponding to the audio feature is less than the preset precision, the compression algorithm of the audio feature is determined to be lossy compression, and if the precision corresponding to the audio feature is greater than or equal to the preset precision, the compression algorithm of the audio feature is determined to be lossless compression. The preset precision is set by the encoder, or by the protocol agreement, or by the decoder, and is not limited in the embodiments of the present disclosure.

[0260] Optionally, the encoder determines the compression algorithm corresponding to each audio feature based on the similarity between each audio feature and the audio features of the audio data of the adjacent frame. In the disclosed embodiment, for multiple audio features of the audio data of the current frame, the similarity between the corresponding audio features of the audio data of the adjacent previous frame can be obtained, and then the corresponding compression algorithm is determined based on the determined similarity.

[0261] Optionally, the similarity range to which the similarity belongs has a corresponding relationship with the compression algorithm. Optionally, different similarity ranges correspond to different compression algorithms. For example, if the similarity is low, linear quantization is used. For another example, similarity range 1 corresponds to SDM modulation, similarity range 2 corresponds to inter-frame prediction, similarity range 3 corresponds to PCA, similarity range 4 corresponds to inter-frame prediction, similarity range 5 corresponds to lossy compression, and similarity range 6 corresponds to lossless compression.

[0262] It should be noted that after the encoder determines the compression algorithm corresponding to the audio feature based on the similarity, it is also necessary to indicate the compression algorithm corresponding to each audio feature through indication information. In some embodiments, the encoder sends indication information to the decoder, and the indication information is used to indicate the compression algorithm corresponding to each audio feature.

[0263] Optionally, the audio feature is associated with a frequency, and a compression algorithm of the audio feature is nonlinear quantization.

[0264] In step S2104, the encoder multiplexes the multiple compressed features to obtain mixed features.

[0265] In some embodiments, the mixing feature includes at least one of the following:

[0266] (1) a synchronization field, wherein the synchronization field is used to indicate a start frame of the audio data;

[0267] (2) a flag field, the flag field being used to indicate whether a corresponding audio feature exists in the data field;

[0268] (3) Length field, used to indicate the length of the data field;

[0269] (4) Data field;

[0270] (5) A check field, which is used to check whether the mixed feature is erroneous when it is sent.

[0271] Optionally, the flag field includes a first-level flag, and each of the audio features has one first-level flag.

[0272] Optionally, the flag field includes a secondary flag, the secondary flag includes a first flag and a second flag; each group has a first flag, the audio features included in each group have a second flag, and each group includes the same type of audio features.

[0273] In some embodiments, all features are embedded into the bitstream in the specified order and all features are transmitted.

[0274] In some embodiments, only a subset of all features are transmitted in the bitstream.

[0275] Optionally, the subset is determined by the value of FLAGS:

[0276] a) The values ​​of FLAGS are fixed. These values ​​are set according to the computing power of speech recognition.

[0277] b) The value of FLAGS is time-varying. The value is set according to the real-time feedback control signal of speech recognition.

[0278] In some embodiments, see Figure 2B The format of the mixed feature includes SYNC (synchronization field), FLAGS, BYTES (bytes), DATA (data) and CRC (Cyclic Redundancy Check), where SYNC corresponds to the synchronization field, FLAGS corresponds to the flag field, BYTES corresponds to the length field, DATA corresponds to the data field and CRC corresponds to the check field.

[0279] In some embodiments, Figure 2C For example, see Figure 2C If the flag field is a first-level flag, the first-level flag includes the following formats: MFCC (Mel Frequency Cepstrum Coefficient), LPC, PITCH (pitch), ENERGY (energy) and SNR (Signal to Interference plus Noise Ratio).

[0280] In some embodiments, Figure 2D For example, see Figure 2DIf the flag field includes the first flag and the second flag, the first flag of the flag field includes Acoustic, Temporal, and Formant. Acoustic includes the second flags MFCC and LPC, Temporal includes ZERO-CROSSING RATE and ENERGY, and Formant includes FREQUENCIES and BANDWIDTHS.

[0281] In some embodiments, the execution process of the above steps S2101-2105 can refer to Figure 2E As shown, after obtaining the audio data, MFCC feature extraction, energy feature extraction and resonance peak feature extraction are performed on the audio data respectively, and then the MFCC feature is compressed by inter-frame prediction, the energy feature is compressed by linear quantization, and the resonance peak feature is compressed by nonlinear quantization, and then the compressed features are reused to obtain mixed features.

[0282] Step S2105: the encoder sends the mixed features to the decoder.

[0283] In some embodiments, the decoder receives the mixed features sent by the encoder.

[0284] Step S2106: The decoder demultiplexes the mixed features to obtain multiple compressed features.

[0285] In some embodiments, the compression feature obtained in the above step S2106 may be different from the compression feature obtained in step S2103 of the above embodiment due to the compression and decompression process, but the features themselves are the same.

[0286] Below, we will use an example to illustrate how to demultiplex mixed features. For example, 1. Search for the synchronization word to find the start of the frame. For example, a 4-byte synchronization word can be set to 0xfffffffb. 2. After parsing the synchronization word, parse Flags according to the Feature location configuration table. It includes level 1 flags and level 2 flags. Flag 1 indicates that the Data field contains the corresponding Feature group. Flag 2 indicates that the Data field contains the Feature in the corresponding Feature group. As shown in the figure, there are 10 level 1 flags for speech recognition. First parse the level 1 flag bits. As shown in Table 1, the first bit represents the acoustic feature group. The first bit is 1, indicating that the Feature group, in which the Data field contains Acoustic. Continue to analyze the three secondary flags of Acoustic, all of which are 1, indicating that the Data field contains three features of the Acoustic group. 3. Get the length of the Data field according to the Bytes field. 4. Find the field of the feature group where the Data field is compressed according to the level 1Flag. 5. Parse the feature group according to the level 2Flag and find the field of a single compressed feature. 6. For a single compressed feature field, first analyze the decompression method of the compressed feature field, and then obtain the decompressed features according to the syntax of the corresponding decompression method.

[0287] Table 1

[0288]

[0289]

[0290] Step S2107: the decoder decompresses each of the multiple compressed features to obtain multiple decompressed features.

[0291] In some embodiments, the decompression algorithm of each compressed feature is used to decompress each compressed feature to obtain the multiple decompressed features. In the disclosed embodiment, step S2107 is the inverse process of the compression process of step S2103 in the above embodiment, and will not be described in detail in the disclosed embodiment.

[0292] In some embodiments, the types of the compression features correspond to the decompression algorithms of the compression features one by one. The corresponding relationship in the embodiment of the present disclosure is the same as the corresponding relationship in step S2103 in the above embodiment, and will not be repeated here.

[0293] In some embodiments, the decoder receives indication information sent by the encoder, where the indication information is used to indicate a decompression algorithm corresponding to each compression feature.

[0294] In some embodiments, the compression feature is associated with frequency, and the decompression algorithm of the compression feature is non-linear decompression.

[0295] Optionally, the decompression algorithm includes at least one of the following:

[0296] PCA;

[0297] Inter-frame prediction decompression;

[0298] Lossy decompression;

[0299] Lossless decompression;

[0300] Inverse quantization.

[0301] Step S2108: The decoder performs speech recognition based on multiple decompressed features.

[0302] In some embodiments, the execution process of the above steps S2106-S2108 can refer to Figure 2F As shown, after obtaining the mixed features, the mixed features are first demultiplexed to obtain compressed MFCC features, compressed energy features and compressed resonance peak features, the compressed MFCC features are inter-frame predictive decoding, the compressed energy features are linearly inverse quantized, and the compressed resonance peak features are nonlinearly inverse quantized to obtain uncompressed MFCC features, uncompressed energy features and uncompressed resonance peak features, and then speech recognition is performed based on the uncompressed MFCC features, non-compressed energy features and non-compressed resonance peak features.

[0303] The audio processing method involved in the embodiment of the present disclosure may include at least one of steps S2101 to S2108. For example, step S2101 may be implemented as an independent embodiment, step S2102 may be implemented as an independent embodiment, step S2103 may be implemented as an independent embodiment, step S2104 may be implemented as an independent embodiment, step S2105 may be implemented as an independent embodiment, step S2106 may be implemented as an independent embodiment, step S2107 may be implemented as an independent embodiment, step S2108 may be implemented as an independent embodiment, steps S2101 and S2102 may be implemented as independent embodiments, and step S2109 may be implemented as independent embodiments. 01. Step S2103 can be implemented as an independent embodiment, step S2101 and step S2104 can be implemented as independent embodiments, step S2102 and step S2103 can be implemented as independent embodiments, step S2102 and step S2104 can be implemented as independent embodiments, step S2103 and step S2104 can be implemented as independent embodiments, step S2105 and step S2106 can be implemented as independent embodiments, step S2107 and step S2108 can be implemented as independent embodiments, but are not limited thereto.

[0304] In some embodiments, step S2101 is optional, and one or more of these steps may be omitted or replaced in different embodiments.

[0305] In some embodiments, step S2102 is optional, and one or more of these steps may be omitted or replaced in different embodiments.

[0306] In some embodiments, step S2103 is optional, and one or more of these steps may be omitted or replaced in different embodiments.

[0307] In some embodiments, step S2104 is optional, and one or more of these steps may be omitted or replaced in different embodiments.

[0308] In some embodiments, step S2105 is optional, and one or more of these steps may be omitted or replaced in different embodiments.

[0309] In some embodiments, step S2106 is optional, and one or more of these steps may be omitted or replaced in different embodiments.

[0310] In some embodiments, step S2107 is optional, and one or more of these steps may be omitted or replaced in different embodiments.

[0311] In some embodiments, step S2108 is optional, and one or more of these steps may be omitted or replaced in different embodiments.

[0312] In some embodiments, see Figure 2A Other optional implementations recorded before or after the corresponding description.

[0313] In some embodiments, the names of information, etc. are not limited to the names recorded in the embodiments, and terms such as "information", "message", "signal", "signaling", "report", "configuration", "indication", "instruction", "command", "channel", "parameter", "domain", "field", "symbol", "symbol", "code element", "codebook", "codeword", "codepoint", "bit", "data", "program", and "chip" can be used interchangeably.

[0314] In some embodiments, terms such as "uplink", "uplink", "physical uplink" can be interchangeable, and terms such as "downlink", "downlink", "physical downlink" can be interchangeable, and terms such as "side", "sidelink", "side communication", "sidelink communication", "direct connection", "direct link", "direct communication", "direct link communication" can be interchangeable.

[0315] In some embodiments, "obtain", "obtain", "get", "receive", "transmit", "bidirectional transmission", "send and / or receive" can be interchangeable, and can be interpreted as receiving from other entities, obtaining from protocols, obtaining from high levels, obtaining by self-processing, autonomous implementation, etc.

[0316] In some embodiments, terms such as "send", "transmit", "report", "send", "transmit", "bidirectional transmission", "send and / or receive" can be used interchangeably.

[0317] In some embodiments, terms such as "moment", "time point", "time", and "time position" can be interchangeable, and terms such as "duration", "period", "time window", "window", and "time" can be interchangeable.

[0318] In some embodiments, terms such as "certain", "preset", "preset", "set", "indicated", "some", "any", and "first" can be interchangeable, and "specific A", "preset A", "preset A", "set A", "indicated A", "some A", "any A", and "first A" can be interpreted as A pre-defined in a protocol, etc., or as A obtained through setting, configuration, or indication, etc., and can also be interpreted as specific A, some A, any A, or first A, etc., but is not limited to this.

[0319] Figure 3A FIG. 1 is a flow chart of an audio processing method according to an embodiment of the present disclosure, which is applied to an encoder. Figure 3A As shown, the embodiment of the present disclosure relates to an audio processing method, and the above method includes:

[0320] Step S3101: the encoder obtains multiple frames of audio data.

[0321] The optional implementation of step S3101 can be found in Figure 2A Optional implementations of step S2101, and Figure 2A Other related parts in the embodiments involved will not be described in detail here.

[0322] Step S3102: the encoder performs feature extraction on multiple frames of audio data to obtain multiple audio features of each frame of audio data.

[0323] The optional implementation of step S3102 can be found in Figure 2A Optional implementations of step S2102, and Figure 2A Other related parts in the embodiments involved will not be described in detail here.

[0324] Step S3103: The encoder compresses each of the multiple audio features respectively to obtain multiple compressed features.

[0325] The optional implementation of step S3103 can be found in Figure 2A Optional implementations of step S2103, and Figure 2A Other related parts in the embodiments involved will not be described in detail here.

[0326] In step S3104, the encoder multiplexes multiple compressed features to obtain mixed features.

[0327] The optional implementation of step S3104 can be found in Figure 2A Optional implementations of step S2104, and Figure 2A Other related parts in the embodiments involved will not be described in detail here.

[0328] Step S3105: the encoder sends the mixed features to the decoder.

[0329] The optional implementation of step S3105 can be found in Figure 2A Optional implementations of step S2105, and Figure 2A Other related parts in the embodiments involved will not be described in detail here.

[0330] The audio processing method involved in the embodiment of the present disclosure may include at least one of steps S3101 to S3104. For example, step S3101 may be implemented as an independent embodiment, step S3102 may be implemented as an independent embodiment, step S3103 may be implemented as an independent embodiment, step S3104 may be implemented as an independent embodiment, and step S3105 may be implemented as an independent embodiment.

[0331] Figure 3B FIG. 1 is a flow chart of an audio processing method according to an embodiment of the present disclosure, which is applied to a terminal. Figure 3B As shown, the embodiment of the present disclosure relates to an audio processing method, and the above method includes:

[0332] Step S3201: the encoder performs feature extraction on multiple frames of audio data to obtain multiple audio features of each frame of audio data.

[0333] The optional implementation of step S3201 can be found in Figure 2A Optional implementations of step S2102, and Figure 2A Other related parts in the embodiments involved will not be described in detail here.

[0334] Step S3202: The encoder compresses each of the multiple audio features respectively to obtain multiple compressed features.

[0335] The optional implementation of step S3202 can be found in Figure 2A Optional implementations of step S2103, and Figure 2A Other related parts in the embodiments involved will not be described in detail here.

[0336] In step S3203, the encoder multiplexes multiple compressed features to obtain mixed features.

[0337] The optional implementation of step S3203 can be found in Figure 2A Optional implementations of step S2104, and Figure 2A Other related parts in the embodiments involved will not be described in detail here.

[0338] The audio processing method involved in the embodiment of the present disclosure may include at least one of steps S3201 to S3203. For example, step S3201 may be implemented as an independent embodiment, step S3202 may be implemented as an independent embodiment, and step S3203 may be implemented as an independent embodiment.

[0339] Figure 4 is a flowchart of an audio processing method according to an embodiment of the present disclosure, which is applied to a network device such as Figure 4 As shown, the embodiment of the present disclosure relates to an audio processing method, and the above method includes:

[0340] Step S4101: The decoder demultiplexes the mixed features to obtain multiple compressed features.

[0341] The optional implementation of step S4101 can be found in Figure 2A Step S2106 and Figure 2A Other related parts in the embodiments involved will not be described in detail here.

[0342] Step S4102: The decoder decompresses each compressed feature of the multiple compressed features to obtain multiple decompressed features.

[0343] The optional implementation of step S4102 can be found in Figure 2A Step S2107 and Figure 2A Other related parts in the embodiments involved will not be described in detail here.

[0344] Step S4103: the decoder performs speech recognition based on multiple decompressed features.

[0345] The optional implementation of step S4103 can be found in Figure 2A Step S2108 and Figure 2A Other related parts in the embodiments involved will not be described in detail here.

[0346] The audio processing method involved in the embodiment of the present disclosure may include at least one of steps S4101 to S4103. For example, step S4101 may be implemented as an independent embodiment, step S4102 may be implemented as an independent embodiment, and step S4103 may be implemented as an independent embodiment.

[0347] In some embodiments, the method further comprises:

[0348] The mixed feature sent by the encoder is received.

[0349] In some embodiments, decompressing each compressed feature of the plurality of compressed features to obtain a plurality of decompressed features comprises:

[0350] Decompress each compressed feature using the decompression algorithm of each compressed feature to obtain the multiple decompressed features.

[0351] In some embodiments, the type of the compression feature corresponds one-to-one to the decompression algorithm of the compression feature.

[0352] In some embodiments, the method further comprises:

[0353] Indication information sent by the encoder is received, where the indication information is used to indicate a decompression algorithm corresponding to each compression feature.

[0354] In some embodiments, the compression feature is associated with frequency, and the decompression algorithm of the compression feature is non-linear decompression.

[0355] In some embodiments, the decompression algorithm includes at least one of the following:

[0356] Principal component analysis PCA;

[0357] Inter-frame prediction decompression;

[0358] Lossy decompression;

[0359] Lossless decompression;

[0360] Inverse quantization.

[0361] In some embodiments, the audio feature includes at least one of the following:

[0362] Acoustic characteristics;

[0363] Voice features;

[0364] Prosodic features;

[0365] Spectral characteristics;

[0366] Timing characteristics;

[0367] Resonant peak characteristics;

[0368] Noise characteristics;

[0369] User characteristics;

[0370] Voice features;

[0371] Contextual features.

[0372] In some embodiments, the mixing feature includes at least one of the following:

[0373] A synchronization field, wherein the synchronization field is used to indicate a start frame of the audio data;

[0374] A flag field, the flag field is used to indicate whether a corresponding audio feature exists in the data field;

[0375] The length field is used to indicate the length of the data field;

[0376] Data fields;

[0377] A check field is used to check whether the hybrid feature is erroneous when being sent.

[0378] In some embodiments, the flag field includes a first-level flag, each of the audio features has one first-level flag; or,

[0379] The flag field includes a secondary flag, and the secondary flag includes a first flag and a second flag; each group has a first flag, and the audio features included in each group have a second flag, and each group includes the same type of audio features.

[0380] Figure 5 is a flowchart of an audio processing method according to an embodiment of the present disclosure, such as Figure 5 As shown, the embodiment of the present disclosure relates to an audio processing method, and the above method includes:

[0381] Step S5101: the encoder extracts features from multiple frames of audio data to obtain multiple audio features for each frame of audio data.

[0382] The optional implementation of step S5101 can be found in Figure 2A Optional implementations of step S2102, and Figure 2A Other related parts in the embodiments involved will not be described in detail here.

[0383] Step S5102: The encoder compresses each of the multiple audio features respectively to obtain multiple compressed features.

[0384] The optional implementation of step S5102 can be found in Figure 2A Optional implementations of step S2103, and Figure 2A Other related parts in the embodiments involved will not be described in detail here.

[0385] In step S5103, the encoder multiplexes multiple compressed features to obtain mixed features.

[0386] The optional implementation of step S5103 can be found in Figure 2A Optional implementations of step S2104, and Figure 2A Other related parts in the embodiments involved will not be described in detail here.

[0387] Step S5104: the decoder demultiplexes the mixed features to obtain multiple compressed features.

[0388] The optional implementation of step S5104 can be found in Figure 2A Step S2106 and Figure 2A Other related parts in the embodiments involved will not be described in detail here.

[0389] Step S5105: The decoder decompresses each compressed feature of the multiple compressed features to obtain multiple decompressed features.

[0390] The optional implementation of step S5105 can be found in Figure 2A Step S2107 and Figure 2A Other related parts in the embodiments involved will not be described in detail here.

[0391] Step S5106: The decoder performs speech recognition based on multiple decompressed features.

[0392] The optional implementation of step S5106 can be found in Figure 2A Step S2108 and Figure 2A Other related parts in the embodiments involved will not be described in detail here.

[0393] In some embodiments, the above method may include the method of the above-mentioned communication system side, encoder side, decoder side, etc., which will not be repeated here.

[0394] Figure 6 is a flowchart of an audio processing method according to an embodiment of the present disclosure, such as Figure 6 As shown, the embodiment of the present disclosure relates to an audio processing method, and the above method includes:

[0395] Step S6101: the encoder performs audio compression on the audio data, wherein the audio compression includes feature extraction, feature compression, and MUX.

[0396] In some embodiments, feature extraction includes:

[0397] 1) Acoustic features: These are the main features extracted from the audio signal. They include parameters such as Mel-Frequency Cepstral Coefficients (MFCC), Linear Prediction Coding (LPC), Perceptual Linear Prediction (PLP), etc.

[0398] 2) Speech features: related to the phonetic content of speech, including parameters such as phonemes, syllables, and words.

[0399] Prosodic features: These features are related to the rhythm, stress, and intonation of speech. They include parameters such as pitch, loudness, duration, etc.

[0400] 4) Spectral characteristics: related to the frequency content of the speech signal, including parameters such as spectral centroid, spectral flux, and spectral roll-off.

[0401] 5) Timing features: related to the timing information in the speech signal, including parameters such as zero-crossing rate and energy.

[0402] 6) Formant characteristics: These characteristics are related to the resonant frequency of the vocal tract, including parameters such as formant frequency and bandwidth.

[0403] 7) Noise characteristics: related to the background noise in the speech signal, including parameters such as signal-to-noise ratio and noise energy.

[0404] 8) Speaker characteristics: These characteristics are related to the personal characteristics of the speaker and include parameters such as the speaker’s identity, gender, and age.

[0405] 9) Language features: These features are related to the language used and include parameters such as language features, dialects, accents, etc.

[0406] 10) Contextual features: These features are related to the context in which the speech is produced and include parameters such as ambient noise, microphone type, and distance to the microphone.

[0407] In some embodiments, feature compression includes:

[0408] 1) Principal Component Analysis (PCA): PCA is a statistical procedure that uses an orthogonal transformation to transform the observed values ​​of a set of possibly correlated variables into a set of values ​​of linearly uncorrelated variables called principal components. This transformation is defined so that the first principal component has the largest possible variance (i.e., explains as much of the variability in the data as possible), and each subsequent component has the highest possible variance under the constraint of being orthogonal to the previous component.

[0409] 2) Inter-frame prediction: In audio coding, inter-frame prediction can predict the spectral content of a frame based on the spectral content of the previous frame. This is based on the observation that the spectral content of an audio signal changes slowly over time, so the spectral content of a frame is likely to be similar to the spectral content of the previous frame. The prediction can be done in a variety of ways. A common method is linear prediction, where the spectral content of a frame is predicted as a linear combination of the spectral content of the previous frame. The coefficients of the linear combination can be estimated based on the correlation between the spectral content of different frames. Once the prediction is made, the difference between the actual spectral content and the prediction (the prediction error) is quantized and transmitted. The receiver then reconstructs the original spectral content by adding the prediction error to the prediction.

[0410] 3) Lossless Compression: This algorithm works by identifying and removing patterns that occur in an audio file. These patterns can be simple, like repeated sounds or silence, or more complex, like a repeating melody or rhythm. Once these patterns are identified, the algorithm replaces them with a shorter code that represents the pattern. This reduces the size of the file while still being able to perfectly reconstruct the original audio when the file is decompressed. In addition, some lossless audio compression algorithms also use a technique called "entropy coding," which further reduces file size by encoding the data in a way that reflects the probability of each sound or pattern occurring.

[0411] 4) Quantification:

[0412] a. Delta Modulation (DM): This method quantizes only the difference between consecutive samples, rather than the absolute value of each sample. This can result in a lower bit rate, but may also introduce more noise.

[0413] b. Adaptive Differential Pulse Code Modulation (ADPCM): This is a variation of PCM that adjusts the quantization level based on the characteristics of the signal. This can produce higher quality audio signals at lower bit rates.

[0414] c. Sigma-delta modulation (SDM): This method uses a feedback loop to continuously adjust the quantization level based on the error of the previous sample. This can produce very high-quality audio signals, but requires more processing power.

[0415] d. Vector Quantization (VQ): This method combines multiple samples together and quantizes them into a single unit. This may result in a lower bit rate, but may also introduce more noise.

[0416] Linear Quantization: This is the simplest form of quantization, where each sample is rounded to the nearest value within a set range. This can result in high bit rates and may introduce quantization noise.

[0417] f. Non-linear quantization: This method adjusts the quantization level based on the amplitude of the signal. This can result in lower bit rates and less quantization noise, but may also introduce distortion.

[0418] In some embodiments, MUX: multiplexes multiple compressed audio features together for packet transmission.

[0419] In some embodiments, audio decompression is the inverse process of encoding, including DEMUX and feature decompression. DEMUX: Separates multiple compressed audio features from the bitstream. Feature decompression: It is the inverse process of feature compression.

[0420] In the embodiments of the present disclosure, part or all of the steps and their optional implementations may be arbitrarily combined with part or all of the steps in other embodiments, or may be arbitrarily combined with optional implementations of other embodiments.

[0421] The embodiments of the present disclosure also propose a device for implementing any of the above methods, for example, a device is proposed, the above device includes a unit or module for implementing each step performed by the terminal in any of the above methods. For another example, another device is also proposed, including a unit or module for implementing each step performed by a network device (such as an access network device, a core network function node, a core network device, etc.) in any of the above methods.

[0422] It should be understood that the division of the units or modules in the above device is only a division of logical functions, and in actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. In addition, the units or modules in the device can be implemented in the form of a processor calling software: for example, the device includes a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or implement the functions of the units or modules of the above device, wherein the processor is, for example, a general-purpose processor, such as a central processing unit (CPU) or a microprocessor, and the memory is a memory inside the device or a memory outside the device. Alternatively, the units or modules in the device can be implemented in the form of hardware circuits, and the functions of some or all of the units or modules can be realized by designing the hardware circuits. The above hardware circuits can be understood as one or more processors; for example, in one implementation, the above hardware circuit is an application-specific integrated circuit (ASIC), and the functions of some or all of the above units or modules are realized by designing the logical relationship of the components in the circuit; for example, in another implementation, the above hardware circuit can be realized by a programmable logic device (PLD), taking a field programmable gate array (FPGA) as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by configuring the configuration file, so as to realize the functions of some or all of the above units or modules. All units or modules of the above devices can be realized in the form of software called by the processor, or in the form of hardware circuits, or in part by the form of software called by the processor, and the rest by the form of hardware circuits.

[0423] In the disclosed embodiments, the processor is a circuit with signal processing capability. In one implementation, the processor may be a circuit with instruction reading and running capability, such as a central processing unit (CPU), a microprocessor, a graphics processing unit (GPU) (which may be understood as a microprocessor), or a digital signal processor (DSP); in another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the above hardware circuit may be fixed or reconfigurable, such as a hardware circuit implemented by a processor as an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration may be understood as the process of the processor loading instructions to implement the functions of some or all of the above units or modules. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as ASIC, such as a neural network processing unit (NPU), a tensor processing unit (TPU), a deep learning processing unit (DPU), etc.

[0424] Fig. 7A Schematic diagram of the structure of the audio processing device proposed in the embodiment of the present disclosure. Fig. 7A As shown, the audio processing device 7100 may include: at least one of a transceiver module 7101, a processing module 7102, etc. In some embodiments, the processing module 7102 performs feature extraction on multiple frames of audio data to obtain multiple audio features of each frame of the audio data; compresses each of the multiple audio features respectively to obtain multiple compressed features; and multiplexes the multiple compressed features to obtain mixed features. Optionally, the above-mentioned transceiver module 7101 is used to execute at least one of the communication steps such as sending and / or receiving performed by the encoder in any of the above methods (such as step S2101 but not limited to this), which will not be repeated here. Optionally, the above-mentioned processing module is used to execute at least one of the other steps performed by the encoder in any of the above methods, which will not be repeated here.

[0425] Optionally, the processing module 7102 is used to execute at least one of the communication steps such as processing performed by the encoder in any of the above methods, which will not be repeated here.

[0426] Figure 7B Schematic diagram of the structure of the audio processing device proposed in the embodiment of the present disclosure. Figure 7B As shown, the audio processing device 7200 may include: at least one of a transceiver module 7201, a processing module 7202, etc. In some embodiments, the processing module 7202 is used to demultiplex the mixed features to obtain multiple compressed features, and the mixed features are obtained based on the audio features of multiple frames of audio data; decompress each of the multiple compressed features to obtain multiple decompressed features; and perform speech recognition based on the multiple decompressed features. Optionally, the above-mentioned transceiver module is used to perform at least one of the communication steps such as sending and / or receiving performed by the decoder in any of the above methods, which will not be repeated here.

[0427] Optionally, the processing module 7202 is used to execute at least one of the communication steps such as processing performed by the decoder in any of the above methods, which will not be repeated here.

[0428] In some embodiments, the transceiver module may include a sending module and / or a receiving module, and the sending module and the receiving module may be separate or integrated. Optionally, the transceiver module may be interchangeable with the transceiver.

[0429] In some embodiments, the processing module can be a module or include multiple submodules. Optionally, the multiple submodules respectively execute all or part of the steps required to be executed by the processing module. Optionally, the processing module can be replaced with the processor.

[0430] Fig. 8A 8100 is a schematic diagram of the structure of the communication device 8100 proposed in the embodiment of the present disclosure. The communication device 8100 can be a network device (such as an access network device, a core network device, etc.), or a terminal, or a chip, a chip system, or a processor that supports the network device to implement any of the above methods, or a chip, a chip system, or a processor that supports the terminal to implement any of the above methods. The communication device 8100 can be used to implement the method described in the above method embodiment, and the details can be referred to the description in the above method embodiment.

[0431] like Fig. 8A As shown, the communication device 8100 includes one or more processors 8101. The processor 8101 can be a general-purpose processor or a dedicated processor, for example, a baseband processor or a central processing unit. The baseband processor can be used to process the communication protocol and communication data, and the central processing unit can be used to control the audio processing device (such as a base station, a baseband chip, a terminal, a terminal chip, a DU or a CU, etc.), execute a program, and process program data. The communication device 8100 is used to perform any of the above methods.

[0432] In some embodiments, the communication device 8100 further includes one or more memories 8102 for storing instructions. Optionally, all or part of the memory 8102 may also be outside the communication device 8100.

[0433] In some embodiments, the communication device 8100 further includes one or more transceivers 8103. When the communication device 8100 includes one or more transceivers 8103, the transceiver 8103 performs at least one of the communication steps such as sending and / or receiving in the above method (for example, step S2101, step S2102, step S2103, step S2104, but not limited thereto).

[0434] In some embodiments, the transceiver may include a receiver and / or a transmitter, and the receiver and the transmitter may be separate or integrated. Optionally, the terms such as transceiver, transceiver unit, transceiver, transceiver circuit, etc. may be replaced with each other, the terms such as transmitter, transmission unit, transmitter, transmission circuit, etc. may be replaced with each other, and the terms such as receiver, receiving unit, receiver, receiving circuit, etc. may be replaced with each other.

[0435] In some embodiments, the communication device 8100 may include one or more interface circuits 8104. Optionally, the interface circuit 8104 is connected to the memory 8102, and the interface circuit 8104 may be used to receive signals from the memory 8102 or other devices, and may be used to send signals to the memory 8102 or other devices. For example, the interface circuit 8104 may read instructions stored in the memory 8102 and send the instructions to the processor 8101.

[0436] The communication device 8100 described in the above embodiments may be a network device or a terminal, but the scope of the communication device 8100 described in the present disclosure is not limited thereto, and the structure of the communication device 8100 may not be limited thereto. Fig. 8A The communication device may be an independent device or may be part of a larger device. For example, the communication device may be: 1) an independent integrated circuit IC, or a chip, or a chip system or subsystem; (2) a collection of one or more ICs, optionally, the above IC collection may also include a storage component for storing data and programs; (3) an ASIC, such as a modem; (4) a module that can be embedded in other devices; (5) a receiver, a terminal, an intelligent terminal, a cellular phone, a wireless device, a handheld device, a mobile unit, a vehicle-mounted device, a network device, a cloud device, an artificial intelligence device, etc.; (6) others, etc.

[0437] Figure 8B 8200 is a schematic diagram of the structure of the chip 8200 proposed in the embodiment of the present disclosure. For the case where the communication device 8100 can be a chip or a chip system, please refer to Figure 8BThe structure diagram of the chip 8200 is shown, but is not limited to this.

[0438] The chip 8200 includes one or more processors 8201, and the chip 8200 is used to execute any of the above methods.

[0439] In some embodiments, the chip 8200 further includes one or more interface circuits 8202. Optionally, the interface circuit 8202 is connected to the memory 8203. The interface circuit 8202 can be used to receive signals from the memory 8203 or other devices, and the interface circuit 8202 can be used to send signals to the memory 8203 or other devices. For example, the interface circuit 8202 can read instructions stored in the memory 8203 and send the instructions to the processor 8201.

[0440] In some embodiments, the interface circuit 8202 performs at least one of the communication steps such as sending and / or receiving in the above method, and the processor 8201 performs at least one of the other steps.

[0441] In some embodiments, terms such as interface circuit, interface, transceiver pin, and transceiver may be used interchangeably.

[0442] In some embodiments, the chip 8200 further includes one or more memories 8203 for storing instructions. Optionally, all or part of the memory 8203 may be outside the chip 8200.

[0443] The present disclosure also proposes a storage medium, on which instructions are stored, and when the instructions are executed on the communication device 8100, the communication device 8100 executes any of the above methods. Optionally, the storage medium is an electronic storage medium. Optionally, the storage medium is a computer-readable storage medium, but is not limited to this, and it can also be a storage medium readable by other devices. Optionally, the storage medium can be a non-transitory storage medium, but is not limited to this, and it can also be a temporary storage medium.

[0444] The present disclosure also proposes a program product, which, when executed by the communication device 8100, enables the communication device 8100 to execute any of the above methods. Optionally, the program product is a computer program product.

[0445] The present disclosure also proposes a computer program, which, when executed on a computer, causes the computer to execute any one of the above methods.

Claims

1. An audio processing method, characterized in that: The method is performed by an encoder; the method comprises: Performing feature extraction on multiple frames of audio data to obtain multiple audio features of each frame of the audio data; Compressing each of the multiple audio features respectively to obtain multiple compressed features; The multiple compressed features are multiplexed to obtain mixed features.

2. The method according to claim 1, characterized in that The method further comprises: The mixed features are sent to a decoder, and are used for decoding and performing audio recognition.

3. The method according to claim 1 or 2, characterized in that: The step of compressing each of the plurality of audio features to obtain a plurality of compressed features comprises: Each audio data is compressed using the compression algorithm of each audio feature to obtain the multiple compression features.

4. The method according to claim 3, characterized in that: The types of the audio features correspond one-to-one to the compression algorithms of the audio features.

5. The method according to claim 3, characterized in that: The method further comprises: Based on the similarity between each audio feature and the audio features of the audio data of the adjacent frames, a compression algorithm corresponding to each audio feature is determined.

6. The method according to claim 5, characterized in that The similarity range to which the similarity belongs has a corresponding relationship with the compression algorithm.

7. The method according to claim 5 or 6, characterized in that: The method further comprises: Sending indication information to the decoder, where the indication information is used to indicate a compression algorithm corresponding to each audio feature.

8. The method according to claim 3, characterized in that The audio feature is associated with frequency, and the compression algorithm of the audio feature is nonlinear quantization.

9. The method according to any one of claims 3 to 8, characterized in that: The compression algorithm includes at least one of the following: Principal component analysis PCA; Inter-frame prediction; Lossy compression; Lossless compression; Quantization compression.

10. The method according to claim 9, characterized in that The quantization compression includes at least one of the following: Delta modulation; Adaptive Differential Pulse Code Modulation; SDM modulation; Vector quantization; Linear quantization; Nonlinear quantization.

11. The method according to any one of claims 1 to 10, characterized in that: The extracting features of the multiple frames of audio data to obtain multiple audio features of each frame of the audio data includes: Determining the types of the multiple audio features obtained by extracting features from the multiple frames of audio data based on first information; the first information is used to indicate the type of audio features; or, The types of the multiple audio features obtained by performing feature extraction on the multiple frames of audio data are determined based on the preconfiguration.

12. The method according to any one of claims 1 to 11, characterized in that: The audio feature includes at least one of the following: Acoustic characteristics; Voice features; Prosodic features; Spectral characteristics; Timing characteristics; Resonant peak characteristics; Noise characteristics; User characteristics; Voice features; Contextual features.

13. The method according to any one of claims 1 to 12, characterized in that: The mixing characteristics include at least one of the following: A synchronization field, wherein the synchronization field is used to indicate a start frame of the audio data; A flag field, the flag field is used to indicate whether a corresponding audio feature exists in the data field; The length field is used to indicate the length of the data field; Data fields; A check field is used to check whether the hybrid feature is erroneous when being sent.

14. The method according to claim 13, characterized in that The flag field includes a first-level flag, and each of the audio features has one first-level flag; or, The flag field includes a secondary flag, and the secondary flag includes a first flag and a second flag; each group has a first flag, and the audio features included in each group have a second flag, and each group includes the same type of audio features.

15. An audio processing method, characterized in that: The method is performed by a decoder, and comprises: Demultiplexing the mixed features to obtain a plurality of compressed features, wherein the mixed features are obtained based on audio features of multiple frames of audio data; Decompressing each compressed feature of the plurality of compressed features to obtain a plurality of decompressed features; Speech recognition is performed based on the multiple decompressed features.

16. The method according to claim 15, characterized in that The method further comprises: The mixed feature sent by the encoder is received.

17. The method according to claim 16, characterized in that Decompressing each compressed feature of the multiple compressed features to obtain multiple decompressed features includes: Decompress each compressed feature using the decompression algorithm of each compressed feature to obtain the multiple decompressed features.

18. The method according to claim 17, characterized in that The types of the compression features correspond one-to-one to the decompression algorithms of the compression features.

19. The method according to claim 17, characterized in that The method further comprises: Indication information sent by the encoder is received, where the indication information is used to indicate a decompression algorithm corresponding to each compression feature.

20. The method according to claim 17, characterized in that The compression feature is associated with frequency, and the decompression algorithm of the compression feature is nonlinear decompression.

21. The method according to any one of claims 3 to 8, characterized in that: The decompression algorithm includes at least one of the following: Principal component analysis PCA; Inter-frame prediction decompression; Lossy decompression; Lossless decompression; Inverse quantization.

22. The method according to any one of claims 15 to 21, characterized in that: The audio feature includes at least one of the following: Acoustic characteristics; Voice features; Prosodic features; Spectral characteristics; Timing characteristics; Resonant peak characteristics; Noise characteristics; User characteristics; Voice features; Contextual features.

23. The method according to any one of claims 1 to 22, characterized in that: The mixing characteristics include at least one of the following: A synchronization field, wherein the synchronization field is used to indicate a start frame of the audio data; A flag field, the flag field is used to indicate whether a corresponding audio feature exists in the data field; The length field is used to indicate the length of the data field; Data fields; A check field is used to check whether the hybrid feature is erroneous when being sent.

24. The method according to claim 23, characterized in that The flag field includes a first-level flag, and each of the audio features has one first-level flag; or, The flag field includes a secondary flag, and the secondary flag includes a first flag and a second flag; each group has a first flag, and the audio features included in each group have a second flag, and each group includes the same type of audio features.

25. An audio processing device, characterized in that: The audio processing device comprises: A processing module, used for extracting features from multiple frames of audio data to obtain multiple audio features of each frame of audio data; The processing module is further used to compress each of the multiple audio features respectively to obtain multiple compressed features; The processing module is also used to multiplex the multiple compressed features to obtain mixed features.

26. An audio processing device, characterized in that: The audio processing device comprises: A processing module, used for demultiplexing the mixed features to obtain a plurality of compressed features, wherein the mixed features are obtained based on the audio features of the multiple frames of audio data; The processing module is further used to decompress each compressed feature of the multiple compressed features to obtain multiple decompressed features; The processing module is further used to perform speech recognition based on the multiple decompressed features.

27. An encoder, characterized in that The encoder comprises: one or more processors; Wherein, the processor is used to execute the audio processing method according to any one of claims 1 to 14.

28. A decoder, characterized in that: The decoder comprises: one or more processors; Wherein, the processor is used to execute the audio processing method described in any one of claims 15 to 24.

29. A storage medium storing instructions, characterized in that: When the instruction is executed on a communication device, the communication device is caused to execute the audio processing method according to any one of claims 1 to 24.

30. A computer program product, characterized in that When the computer program product is executed on a communication device, the communication device executes the audio processing method according to any one of claims 1 to 24.

Images