Data processing method and device, terminal equipment and readable storage medium

Abstract

The invention provides a data processing method and device, terminal equipment and a readable storage medium, and relates to the technical field of communication. The method comprises the following steps: receiving a first signal sent by second equipment; decoding the first signal to obtain a first semantic feature; and outputting first voice information of the first sound characteristic according to the first sound characteristic of the user on the second equipment side and the first semantic characteristic. According to the scheme provided by the invention, when the voice of the user at the second equipment side is transmitted, only the character features of the voice are transmitted, so that the transmission data volume is reduced, and communication resources are saved; according to the first sound characteristic of the user on the second equipment side, the first voice information similar to the sound tone of the user on the second equipment side can be output, and effective transmission of the voice information is ensured. The problems that the information format specifically needed by a receiving end user is not considered, the whole voice signal is transmitted, communication resources are wasted, and data congestion is easily caused are solved.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a data processing method, device, terminal equipment and readable storage medium. Background Art

[0002] As an important part of the communication system, voice transmission converts the voice of the sending user into a digital signal based on floating point numbers through sampling, quantization and encoding, and transmits it to the terminal device through the existing communication network to serve application scenarios such as telephone network and cloud conference. In order to realize voice intelligent application, the current method is to first restore the voice signal as accurately as possible at the receiving end through the traditional voice transmission system, and then combine the restored voice signal with the intelligent application model according to the user-defined intelligent application to realize various intelligent applications to serve users.

[0003] Due to the complexity of human voice, voice signals often contain various semantic information, such as text content, sound timbre, emotional expression, etc. During the transmission process, the existing system architecture serving voice intelligence applications does not consider the specific information format required by the receiving end user, and transmits the entire voice signal, which wastes communication resources and easily causes data congestion. Summary of the invention

[0004] The purpose of the present invention is to provide a data processing method, apparatus, terminal device and readable storage medium to solve the problem in the prior art that the information format specifically required by the receiving user is not considered, the overall voice signal is transmitted, communication resources are wasted and data congestion is easily caused.

[0005] To achieve the above object, an embodiment of the present invention provides a data processing method, which is applied to a first device and includes:

[0006] receiving a first signal sent by a second device, wherein the first signal carries only a text feature;

[0007] Decoding the first signal to obtain a first semantic feature;

[0008] According to the first sound characteristic of the user on the second device side and the first semantic feature, first voice information of the first sound characteristic is output.

[0009] Furthermore, the language type of the text corresponding to the first semantic feature is the same as the language type corresponding to the voice information;

[0010] Outputting first voice information of the first voice characteristic according to the first voice characteristic of the user on the second device side and the first semantic feature includes:

[0011] Decoding the first semantic feature to obtain first text information;

[0012] synthesizing the first voice information according to the first text information and the first sound characteristic;

[0013] The first voice information is output.

[0014] Furthermore, before outputting the first voice information of the first voice characteristic according to the first voice characteristic of the user on the second device side and the first semantic feature, the method further includes:

[0015] The first voice characteristic of the user on the second device side is obtained according to the identification information of the second device.

[0016] Furthermore, before outputting the first voice information of the first voice characteristic according to the first voice characteristic of the user on the second device side and the first semantic feature, the method further includes:

[0017] receiving the first sound characteristic sent by the second device;

[0018] Acquire the first sound characteristic from the cloud according to the received identification information of the second device;

[0019] The first sound characteristic is selected from the sound data stored in the first device according to the identification information of the second device.

[0020] Furthermore, the method further comprises:

[0021] Output text information and / or second voice information according to the first semantic feature.

[0022] Furthermore, the language type of the text corresponding to the text semantic feature is a first language type, and outputting text information according to the first semantic feature includes:

[0023] Decoding the first semantic feature to obtain text information of a first language type and outputting the text information of the first language type; and / or,

[0024] Converting the first semantic feature to obtain a second semantic feature;

[0025] Decoding the second semantic feature to obtain and output the text information in a second language type;

[0026] The first language type is different from the second language type.

[0027] Furthermore, the language type of the text corresponding to the first semantic feature is a first language type, and outputting the second voice information according to the first semantic feature includes:

[0028] Converting the first semantic feature to obtain a second semantic feature;

[0029] Decoding the second semantic feature to obtain second text information;

[0030] According to the second text information, second voice information in a second language type is synthesized and output.

[0031] To achieve the above object, an embodiment of the present invention provides a data processing method, which is applied to a second device and includes:

[0032] Encoding the received speech signal to obtain a third semantic feature corresponding to the speech signal;

[0033] Mapping the third semantic feature to obtain a first signal carrying only text features;

[0034] The first signal is sent to the first device.

[0035] Furthermore, encoding the received speech signal to obtain a third semantic feature corresponding to the speech signal includes:

[0036] Performing spectrum extraction on the received speech signal to obtain a spectrum matrix;

[0037] Performing semantic extraction on the frequency spectrum matrix to obtain the third semantic feature.

[0038] Furthermore, the step of performing spectrum extraction on the received speech signal to obtain a spectrum matrix includes:

[0039] The spectrum matrix is ​​obtained by performing spectrum extraction on the speech signal through Hamming window, fast Fourier transform, logarithmic operation and normalization operation.

[0040] Furthermore, encoding the received speech signal to obtain a third semantic feature corresponding to the speech signal includes:

[0041] Performing semantic compression on the spectrum matrix by a first encoder to obtain first semantic representation information;

[0042] Performing feature extraction on the first semantic representation information by a second encoder to obtain the third semantic feature;

[0043] Wherein, the first encoder comprises: a multi-layer cascaded convolutional neural network;

[0044] The second encoder includes: a multi-layer convolutional neural network based on a gated recurrent unit, a fully connected layer, and a normalized exponential layer.

[0045] To achieve the above object, an embodiment of the present invention provides a data processing apparatus, applied to a first device, comprising:

[0046] A receiving module, configured to receive a first signal sent by a second device, wherein the first signal carries only a text feature;

[0047] A decoding module, used for decoding the first signal to obtain a first semantic feature;

[0048] An output module is used to output first voice information of the first sound characteristic according to the first sound characteristic of the user on the second device side and the first semantic feature.

[0049] To achieve the above object, an embodiment of the present invention provides a data processing apparatus, applied to a second device, comprising:

[0050] An encoding module, used for encoding the received speech signal to obtain a third semantic feature corresponding to the speech signal;

[0051] A mapping module, used for mapping the third semantic feature to obtain a first signal carrying only text features;

[0052] A sending module is used to send the first signal to a first device.

[0053] To achieve the above object, an embodiment of the present invention provides a terminal device, including: a transceiver and a processor;

[0054] The transceiver is used to receive a first signal sent by a second device, wherein the first signal carries only text features;

[0055] The processor is used to decode the first signal to obtain a first semantic feature;

[0056] According to the first sound characteristic of the user on the second device side and the first semantic feature, first voice information of the first sound characteristic is output.

[0057] To achieve the above object, an embodiment of the present invention provides a terminal device, including a processor and a transceiver;

[0058] The processor is used to encode the received voice signal to obtain a third semantic feature corresponding to the voice signal;

[0059] Mapping the third semantic feature to obtain a first signal carrying only text features;

[0060] The transceiver is used to send the first signal to the first device.

[0061] To achieve the above-mentioned purpose, an embodiment of the present invention provides a terminal device, including a transceiver, a processor, a memory, and a program or instruction stored in the memory and executable on the processor; when the processor executes the program or instruction, the data processing method as described above is implemented.

[0062] To achieve the above objective, an embodiment of the present invention provides a readable storage medium on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps in the data processing method described above are implemented.

[0063] The beneficial effects of the above technical solution of the present invention are as follows:

[0064] The data processing method of the embodiment of the present invention obtains the first semantic feature by decoding the first signal that carries only text features and is sent by the second device, and outputs the first voice information that is similar to the timbre of the voice of the user on the second device side according to the first voice feature of the user on the second device side and the first semantic feature. The scheme of the present invention, when transmitting the voice of the user on the second device side, only the text features of the voice are transmitted, which reduces the amount of transmitted data and saves communication resources; according to the first voice feature of the user on the second device side, the first voice information that is similar to the timbre of the voice of the user on the second device side can be output, ensuring the effective transmission of voice information. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 A schematic diagram of a flow chart of a data processing method applied to a first device according to an embodiment of the present invention;

[0066] Figure 2 is a flow chart of a data processing method applied to a second device according to an embodiment of the present invention;

[0067] Figure 3 A schematic diagram of the communication system structure of the data transmission method according to an embodiment of the present invention;

[0068] Figure 4 Schematic diagram of the application process of the data transmission method according to an embodiment of the present invention;

[0069] Figure 5 A schematic diagram of the structure of a data processing device applied to a first device according to an embodiment of the present invention;

[0070] Figure 6 A schematic diagram of the structure of a data processing device applied to a second device according to an embodiment of the present invention;

[0071] Figure 7A schematic diagram of the structure of a terminal device according to an embodiment of the present invention;

[0072] Figure 8 A schematic diagram of the structure of a terminal device according to another embodiment of the present invention;

[0073] Fig. 9 This is a schematic diagram of the structure of a terminal device according to another embodiment of the present invention. DETAILED DESCRIPTION

[0074] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0075] It should be understood that the references to "one embodiment" or "an embodiment" throughout the specification mean that the specific features, structures, or characteristics associated with the embodiment are included in at least one embodiment of the present invention. Therefore, the references to "in one embodiment" or "in an embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0076] In various embodiments of the present invention, it should be understood that the size of the serial numbers of the following processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0077] Additionally, the terms "system" and "network" are often used interchangeably herein.

[0078] In the embodiments provided in the present application, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined according to A. However, it should also be understood that determining B according to A does not mean determining B only according to A, and B can also be determined according to A and / or other information.

[0079] like Figure 1 As shown, a data processing method according to an embodiment of the present invention is applied to a first device, comprising the following steps:

[0080] Step 101, receiving a first signal sent by a second device, wherein the first signal carries only text features;

[0081] Step 102: decode the first signal to obtain a first semantic feature;

[0082] Step 103: output first voice information of the first voice characteristic according to the first voice characteristic of the user on the second device side and the first semantic feature.

[0083] In one embodiment of the present invention, based on the received first signal y, a channel decoder consisting of three cascaded fully connected layers and one softmax layer is used to restore it into the N×E first semantic features Where E is the dimension of the character vector. The trainable neural network parameters of the channel decoder are represented as γ, then It can be expressed as:

[0084]

[0085] in, and Represents a channel decoder.

[0086] The data processing method of the embodiment of the present invention obtains the first semantic feature by decoding the first signal that carries only text features and is sent by the second device, and outputs the first voice information that is similar to the timbre of the voice of the user on the second device side according to the first voice feature of the user on the second device side and the first semantic feature. The scheme of the present invention, when transmitting the voice of the user on the second device side, only the text features of the voice are transmitted, which reduces the amount of transmitted data and saves communication resources; according to the first voice feature of the user on the second device side, the first voice information that is similar to the timbre of the voice of the user on the second device side can be output, ensuring the effective transmission of voice information.

[0087] Optionally, the language type of the text corresponding to the first semantic feature is the same as the language type of the voice information;

[0088] Outputting first voice information of the first voice characteristic according to the first voice characteristic of the user on the second device side and the first semantic feature includes:

[0089] Decoding the first semantic feature to obtain first text information;

[0090] synthesizing the first voice information according to the first text information and the first sound characteristic;

[0091] The first voice information is output.

[0092] In an embodiment of the present invention, if the language type of the text corresponding to the first semantic feature is Chinese, then the first text information is Chinese text, and the first voice information is Chinese voice.

[0093] In one embodiment of the present invention, if the language type of the text corresponding to the first semantic feature is a foreign language (such as English), the first text information is a foreign text, and the first voice information is a foreign voice.

[0094] The data processing method of the embodiment of the present invention can synthesize the first voice information similar to the timbre of the voice of the user on the second device side according to the first voice characteristics of the user on the second device side, thereby saving transmission resources and ensuring the effective transmission of voice information.

[0095] Optionally, before outputting the first voice information of the first voice characteristic according to the first voice characteristic of the user on the second device side and the first semantic feature, the method further includes:

[0096] The first voice characteristic of the user on the second device side is obtained according to the identification information of the second device.

[0097] Optionally, acquiring the first voice characteristic of the user on the second device side according to the identification information of the second device includes:

[0098] acquiring the first sound characteristic from the cloud according to the identification information of the second device; or,

[0099] The first sound characteristic is selected from the sound data stored in the first device according to the identification information of the second device.

[0100] Optionally, before outputting the first voice information of the first voice characteristic according to the first voice characteristic of the user on the second device side and the first semantic feature, the method further includes:

[0101] receiving the first sound characteristic sent by the second device;

[0102] Acquire the first sound characteristic from the cloud according to the received identification information of the second device;

[0103] The first sound characteristic is selected from the sound data stored in the first device according to the identification information of the second device.

[0104] In an embodiment of the present invention, the first sound characteristic is sent to the first device via the second device.

[0105] In another embodiment of the present invention, sound characteristics of a plurality of terminal devices are stored in the cloud, and the first sound characteristic is obtained from a sound characteristic database in the cloud according to identification information of the second device.

[0106] In yet another embodiment of the present invention, the first device stores sound characteristics of a plurality of terminal devices, and the first sound characteristic is acquired from a sound characteristic database of the first device according to identification information of the second device.

[0107] The data processing method of the embodiment of the present invention can obtain the first sound characteristics of the second device and combine it with the first text information obtained by decoding to synthesize the first voice information that is similar to the timbre of the voice of the user on the second device side.

[0108] Optionally, the method further comprises:

[0109] Output text information and / or second voice information according to the first semantic feature.

[0110] Optionally, the language type of the text corresponding to the text semantic feature is a first language type, and outputting text information according to the first semantic feature includes:

[0111] Decoding the first semantic feature to obtain text information of a first language type and outputting the text information of the first language type; and / or,

[0112] Converting the first semantic feature to obtain a second semantic feature;

[0113] Decoding the second semantic feature to obtain and output the text information in a second language type;

[0114] The first language type is different from the second language type.

[0115] Optionally, the text information is a text sequence.

[0116] Optionally, the first language type is Chinese, and the second language type is a foreign language; or, the first language type is a foreign language, and the second language type is Chinese.

[0117] The data processing method of the embodiment of the present invention only transmits the semantic features of the voice information on the second device side, and outputs text information and / or voice information of different language types according to the needs of the user on the first device side. The solution of the present invention saves data transmission resources, promotes the effective transmission of semantic information, and improves the utilization rate of semantic information.

[0118] Optionally, the language type of the text corresponding to the first semantic feature is a first language type, and outputting the second voice information according to the first semantic feature includes:

[0119] Converting the first semantic feature to obtain a second semantic feature;

[0120] Decoding the second semantic feature to obtain second text information;

[0121] According to the second text information, second voice information in a second language type is synthesized and output.

[0122] In an embodiment of the present invention, the first language type is Chinese, the second text information is English text information corresponding to the first semantic feature, and the second voice information is English voice information.

[0123] The data processing method of the embodiment of the present invention can convert the language type of the transmitted semantic feature by converting the first semantic feature, so that text information and / or voice information of different language types can be output according to the needs of the user on the first device side, thereby improving the utilization rate of the semantic information.

[0124] Optionally, decoding the received first signal to obtain the first semantic feature includes:

[0125] Decoding the first signal by a first decoder to obtain the first semantic feature;

[0126] The first decoder includes: three cascaded fully connected layers and one normalized exponential layer.

[0127] Specifically, the first device has the following functions:

[0128] Chinese speech recognition: In Chinese speech recognition applications, semantic features Contains the semantic information of Chinese characters, so the first semantic feature can be directly restored through the feature decoder (i.e., beam search decoder) Decode to get Chinese text To minimize the sequence of words The error in is equivalent to maximizing the posterior probability According to the connectionist temporal classification (CTC), the posterior probability It can be expressed as:

[0129]

[0130] in, represents the set of all possible valid alignments of text t to spectrum s;

[0131] Therefore, in order to maximize the posterior probability We introduce a loss function to update the trainable neural network parameters of the semantic channel joint encoder and channel decoder, namely θ 1 =(α, β, γ), the loss function can be expressed as:

[0132]

[0133] Assuming that accurate channel state information is known, then θ 1It can be updated using the stochastic gradient descent (SGD) algorithm:

[0134]

[0135] Among them, η>0 represents the learning rate, represents a differential operator;

[0136] Chinese speech synthesis: For Chinese speech synthesis applications, the semantic features are first Mapping into Chinese characters Afterwards, Input into the speech synthesis module Tacotron 2 to synthesize a Chinese speech sequence as close to the input speech sequence as possible It is worth mentioning that the user ID is input into Tractron 2 here, because some voice characteristics of the sending user are needed to synthesize a speech sequence similar to the sending user; the trainable neural network parameters of Tacotron 2 are represented as θ 2 , the mean-square error (MSE) loss is used to calculate the network loss, expressed as:

[0137]

[0138] Among them, m l and Represent the input speech sequence m and the synthesized speech sequence respectively The lth speech sampling point in , L represents the number of speech sampling points in the two speech sequences;

[0139] Similarly, according to the mean square error loss function, the stochastic gradient descent algorithm is used to update θ 2 ;

[0140] Chinese speech to foreign text translation: For Chinese speech to foreign text translation, a transformer-based machine translation module is first designed at the receiving end to convert the semantic features of Chinese text into Convert the foreign text semantic features into foreign text semantic features, and map the foreign text semantic features into foreign text sequences through feature decoders Technically speaking, we first embed the Map each word one by one, then symbolize it, and map it to get the word vector of each word based on probability, and mark the start and end of the sentence with fixed special symbols. The purpose is to generate a trainable symbol sequence. According to the large number of processed word vectors, a transformer module based on the attention mechanism is used to build a neural network model, and the probabilistic representation of the translated text is output word by word. The trainable neural network parameters of the machine translation module are represented as θ 3 , based on the probability representation q of the correct English text and the probability representation of the damaged foreign text predicted by the neural network The cross entropy (CE) loss function is used to calculate the network loss, which is expressed as:

[0141]

[0142] Similarly, according to the cross entropy loss function, the stochastic gradient descent algorithm is used to update θ 3 ;

[0143] Chinese speech to foreign speech translation: In order to enable the user at the receiving end to obtain a clear foreign speech sequence, the above machine translation module is combined with the foreign speech synthesis module; first, the semantic features of Chinese characters are transformed into Mapping into foreign text sequences Then, the foreign language speech synthesis module Synthesize and output foreign language speech sequences The trainable neural network parameters of the foreign language speech synthesis module are represented as θ 4 , the mean square error loss function is used to calculate the network loss, expressed as:

[0144]

[0145] in, and Represents the correct foreign language speech sequence and synthesizing foreign language speech sequences The kth speech sampling point in , K represents the number of speech sampling points in the two speech sequences;

[0146] Similarly, according to the mean square error loss function, the stochastic gradient descent algorithm is used to update θ 4 .

[0147] The data processing method of the embodiment of the present invention is based on the restored first semantic feature According to different needs of users, the system can serve one or more intelligent applications such as Chinese speech recognition, Chinese speech synthesis, Chinese speech to foreign text translation, Chinese speech to foreign speech translation, etc. It is worth noting that when executing a certain intelligent application, the modules used for other intelligent applications can be ignored. For example, when executing the Chinese speech synthesis application, the calculation of the machine translation module and the foreign speech synthesis module can be terminated. Such a dynamic output mechanism can focus computing resources on user-defined intelligent applications, thereby maximizing the utilization of computing resources.

[0148] like Figure 2 As shown, an embodiment of the present invention provides a data processing method, which is applied to a second device and includes the following steps:

[0149] Step 201, encoding the received speech signal to obtain a third semantic feature corresponding to the speech signal;

[0150] Step 202, mapping the third semantic feature to obtain a first signal carrying only text features;

[0151] Step 203: Send the first signal to the first device.

[0152] The data processing method of the embodiment of the present invention can obtain the third semantic feature by encoding the voice signal, and obtain the first signal carrying only text features by mapping the third semantic feature, thereby realizing the transmission of only the semantic features of the voice signal and saving data transmission resources.

[0153] Optionally, the speech signal is a speech sequence.

[0154] Optionally, the first signal is sent to the first device through a channel.

[0155] In one embodiment of the present invention, the signal x after semantic channel joint coding is normalized and transmitted to the wireless channel, and the receiving end receives the signal y. Assuming that the wireless channel is h, the process from the transmitting end to the receiving end can be expressed as:

[0156] y=hx+w

[0157] Where w represents additive white Gaussian noise.

[0158] Optionally, encoding the received speech signal to obtain a third semantic feature corresponding to the speech signal includes:

[0159] Performing spectrum extraction on the received speech signal to obtain a spectrum matrix;

[0160] Performing semantic extraction on the frequency spectrum matrix to obtain the third semantic feature.

[0161] Optionally, performing spectrum extraction on the received speech signal to obtain a spectrum matrix includes:

[0162] The spectrum matrix is ​​obtained by performing spectrum extraction on the speech signal through Hamming window, fast Fourier transform, logarithmic operation and normalization operation.

[0163] Optionally, encoding the received speech signal to obtain a third semantic feature corresponding to the speech signal includes:

[0164] Performing semantic compression on the spectrum matrix by a first encoder to obtain first semantic representation information;

[0165] Performing feature extraction on the first semantic representation information by a second encoder to obtain the third semantic feature;

[0166] Wherein, the first encoder comprises: a multi-layer cascaded convolutional neural network;

[0167] The second encoder includes: a multi-layer convolutional neural network based on a gated recurrent unit, a fully connected layer, and a normalized exponential layer.

[0168] In one embodiment of the present invention, the first encoder comprises: a three-layer cascaded convolutional neural network;

[0169] The second encoder includes: six layers of convolutional neural networks based on gated recurrent units, one fully connected layer, and one normalized exponential layer.

[0170] Optionally, mapping the third semantic feature to obtain a first signal carrying only text semantic features includes:

[0171] Mapping the third semantic feature through a third encoder to obtain the first signal;

[0172] The third encoder includes: two cascaded fully connected layers and a conversion layer.

[0173] The data processing method of the embodiment of the present invention semantically compresses the input speech and then extracts features based on the compressed semantic information, fully mining the semantic information to obtain text-related semantic features, and combines it with actual intelligent applications to promote the effective transmission of semantic information.

[0174] In one embodiment of the present invention, the purpose of the transmitting device is to map the input speech into a transmittable signal, and to effectively extract text-related semantic features and minimize the impact of the physical channel on signal transmission by designing a semantic channel joint encoder.

[0175] Specifically, the sending end device performs the following steps:

[0176] For any input speech sequence m, the M×161 spectrum matrix s is obtained through Hamming window, fast Fourier transform (FFT), logarithmic operation, and normalization operation. Since the duration of the input speech varies, the first dimension M of the output spectrum matrix is ​​not fixed.

[0177] Semantic-channel joint encoder. In the semantic encoder, based on the spectrum matrix, a two-layer cascaded convolutional neural network is first used for semantic compression to obtain an N×81×32 semantic representation. Then, it passes through six layers of convolutional neural networks based on gated recurrent units, a fully connected layer, and a softmax layer for feature extraction, and outputs N×E text-related semantic features p. In the channel encoder, the input semantic features are mapped into an N×20 complex signal x using two cascaded fully connected layers and a conversion layer. The trainable neural network parameters of the semantic encoder and the channel encoder are represented as α and β respectively, then the signal x sent to the physical channel can be expressed as:

[0178]

[0179] in, and denote semantic encoder and channel encoder respectively;

[0180] After the semantic channel joint coding, the output signal x only carries the semantic features related to the text, while some sound characteristics in the source input speech, such as voice timbre, speaking speed, emotion, etc., will be ignored and discarded. Compared with the traditional system that transmits the entire voice information, the low-dimensional semantic features obtained through such a compression mechanism can greatly reduce the data volume of the transmitted signal, thereby saving communication bandwidth resources, alleviating network data congestion, and improving the user experience quality.

[0181] like Figure 3 As shown, in one embodiment of the present invention, the communication system architecture for implementing the data transmission method is as follows:

[0182] The transmitting end device includes: a semantic channel joint encoder;

[0183] The semantic channel joint encoder is used to:

[0184] Semantically compress the voice input by the sending user and extract low-dimensional text-related semantic features;

[0185] The semantic features are encoded to obtain a transmittable first signal including only text information.

[0186] The first signal is transmitted through a physical channel and sent to a receiving device.

[0187] The receiving end device includes: a channel semantic joint decoder;

[0188] The channel semantic joint decoder is used to:

[0189] After receiving the first signal, performing channel decoding on the first signal to obtain text-related semantic features;

[0190] Decoding the semantic features to obtain text information;

[0191] When the text information is Chinese text information, at least one of the following is performed:

[0192] Outputting the Chinese text information;

[0193] Combining the Chinese text information with the voice characteristics of the sending end user to synthesize voice information similar to the sending end user, and outputting the voice information;

[0194] When the text information is in a foreign language, at least one of the following is performed:

[0195] Outputting the foreign language text information;

[0196] Foreign language voice information is synthesized based on the foreign language text information, and the foreign language voice information is output.

[0197] In one embodiment of the present invention, for the voice signal input by the user, the transmitter adopts a semantic channel joint encoder based on a neural network, firstly generates text-related semantic features from the input voice, and this process uses the feature learning ability of the neural network to learn the semantic information in the voice signal and compress and extract it, and finally outputs semantic features related only to the text, which are expressed in the form of a probability feature matrix based on floating point numbers. The semantic features related to the text are defined as the semantic representation of the text information, and the semantic features related to the text can be mapped into a text sequence through a feature decoder. Subsequently, the text-related semantic features based on the probability matrix are input into the channel encoder constructed by the neural network, mapped into a complex vector based on floating point numbers, and represented as a transmittable signal. Since a large number of deep learning algorithms are used in the semantic channel joint encoder to construct a complex neural network model with strong fitting ability, it is necessary to integrate multiple high-performance graphics processors (GPUs) in the distribution unit (DU) of the physical layer of the transmitter and implement parallel computing.

[0198] In one embodiment of the present invention, the transmittable signal carrying the semantic features related to the text passes through the active antenna unit (AAU) of the physical layer, and then the wireless signal is sent to the physical channel. This process can take into account channel fading and physical noise.

[0199] In one embodiment of the present invention, after the receiving end receives the physical signal, the signal is first restored to text-related semantic features through a channel decoder based on a neural network, and then these semantic features are decoded through a feature decoder according to the needs of the user, thereby obtaining information such as Chinese text, foreign text, Chinese speech, or foreign speech, so as to achieve a variety of downstream tasks, such as video subtitle generation and multi-national conference real-time translation, so as to meet the needs of the receiving end user. Since the channel decoder involves the calculation of the neural network model, the terminal device of the receiving end needs to have a certain performance computing capability to reduce the calculation delay of the neural network and thus reduce the overall delay of the end-to-end communication.

[0200] like Figure 4 As shown, in one embodiment of the present invention, the data transmission method is as follows:

[0201] The speech input by the sending end user is first uniformly spectrally extracted, then input into the semantic channel joint coding, low-dimensional text-related semantic features are learned, and mapped into a numerical matrix based on floating-point numbers, and finally a transmittable signal is generated through the conversion layer;

[0202] After the signal passes through the noisy physical channel, the text-related semantic features are restored through the conversion layer and channel decoder;

[0203] According to different needs of users, the beam search decoder, speech synthesis module, and machine translation module are combined to output four different types of information, namely Chinese text, Chinese speech, foreign text, or foreign speech, to realize intelligent applications of speech recognition, speech synthesis, Chinese speech to foreign text translation, and Chinese speech to foreign speech translation.

[0204] The communication system architecture of the embodiment of the present invention can adaptively serve one or more intelligent applications, and in order to achieve a voice signal close to that of the sending user, the ID of the sending user needs to be sent to the receiving end through a reliable wireless channel, and the receiving end can obtain the user's pre-registered information from the cloud to synthesize a voice signal with a timbre close to that of the sending user.

[0205] like Figure 5 As shown, the embodiment of the present invention further provides a data processing device 500, which is applied to a first device and includes:

[0206] The receiving module 501 is used to receive a first signal sent by a second device, where the first signal only carries a text feature;

[0207] A decoding module 502, configured to decode the first signal to obtain a first semantic feature;

[0208] The output module 503 is used to output the first voice information of the first voice characteristic according to the first voice characteristic of the user on the second device side and the first semantic feature.

[0209] The data processing device of the embodiment of the present invention obtains the first semantic feature by decoding the first signal that carries only text features and is sent by the received second device, and outputs the first voice information that is similar to the timbre of the voice of the user on the second device side according to the first voice feature of the user on the second device side and the first semantic feature. The scheme of the present invention, when transmitting the voice of the user on the second device side, only the text features of the voice are transmitted, which reduces the amount of transmitted data and saves communication resources; according to the first voice feature of the user on the second device side, the first voice information that is similar to the timbre of the voice of the user on the second device side can be output, ensuring the effective transmission of voice information.

[0210] like Figure 6 As shown, the embodiment of the present invention further provides a data processing device 600, which is applied to a second device, including:

[0211] The encoding module 601 is used to encode the received speech signal to obtain a third semantic feature corresponding to the speech signal;

[0212] A mapping module 602, configured to map the third semantic feature to obtain a first signal carrying only text features;

[0213] The sending module 603 is configured to send the first signal to the first device.

[0214] The data processing device of the embodiment of the present invention can obtain the third semantic feature by encoding the voice signal, and obtain the first signal carrying only text features by mapping the third semantic feature, thereby realizing the transmission of only the semantic features of the voice signal and saving data transmission resources.

[0215] like Figure 7 As shown, a terminal device 700 according to an embodiment of the present invention includes a processor 710 and a transceiver 720, wherein:

[0216] The transceiver is used to receive a first signal sent by a second device, wherein the first signal carries only text features;

[0217] The processor is used to decode the first signal to obtain a first semantic feature;

[0218] According to the first sound characteristic of the user on the second device side and the first semantic feature, first voice information of the first sound characteristic is output.

[0219] The terminal device of the embodiment of the present invention obtains the first semantic feature by decoding the first signal that carries only text features and is sent by the second device, and outputs the first voice information that is similar to the timbre of the voice of the user on the second device side according to the first voice feature of the user on the second device side and the first semantic feature. The scheme of the present invention, when transmitting the voice of the user on the second device side, only the text features of the voice are transmitted, which reduces the amount of transmitted data and saves communication resources; according to the first voice feature of the user on the second device side, the first voice information that is similar to the timbre of the voice of the user on the second device side can be output, ensuring the effective transmission of voice information.

[0220] like Figure 8 As shown, a terminal device 800 according to an embodiment of the present invention includes a processor 810 and a transceiver 820, wherein:

[0221] The processor is used to encode the received voice signal to obtain a third semantic feature corresponding to the voice signal;

[0222] Mapping the third semantic feature to obtain a first signal carrying only text features;

[0223] The transceiver is used to send the first signal to the first device.

[0224] The terminal device of the embodiment of the present invention can obtain the third semantic feature by encoding the voice signal, and obtain the first signal carrying only text features by mapping the third semantic feature, thereby realizing the transmission of only the semantic features of the voice signal and saving data transmission resources.

[0225] A terminal device according to another embodiment of the present invention, such as Fig. 9 As shown, it includes a transceiver 910, a processor 900, a memory 920, and a program or instruction stored in the memory 920 and executable on the processor 900; when the processor 900 executes the program or instruction, the above-mentioned data processing method is implemented.

[0226] The transceiver 910 is used to receive and send data under the control of the processor 900 .

[0227] Among them, Fig. 9In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically one or more processors represented by processor 900 and various circuits of memory represented by memory 920 are linked together. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface. The transceiver 910 may be a plurality of components, namely, a transmitter and a receiver, providing a unit for communicating with various other devices on a transmission medium. For different user devices, the user interface 930 may also be an interface capable of externally and internally connecting required devices, and the connected devices include but are not limited to a keypad, a display, a speaker, a microphone, a joystick, and the like.

[0228] The processor 900 is responsible for managing the bus architecture and general processing, and the memory 920 can store data used by the processor 900 when performing operations.

[0229] A readable storage medium according to an embodiment of the present invention stores a program or instruction thereon. When the program or instruction is executed by a processor, the steps in the data processing method described above are implemented and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0230] The processor is the processor in the terminal device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0231] It should be further explained that the terminals described in this specification include but are not limited to smart phones, tablet computers, etc., and many of the functional components described are called modules in order to more particularly emphasize the independence of their implementation methods.

[0232] In the embodiment of the present invention, module can be implemented with software so that it can be executed by various types of processors. For example, an executable code module of an identification can include one or more physical or logical blocks of computer instructions, for example, it can be constructed as an object, process or function. Nevertheless, the executable code of the identified module does not need to be physically located together, but can include different instructions stored in different positions, and when these instructions are logically combined together, it constitutes a module and realizes the specified purpose of the module.

[0233] In fact, executable code module can be a single instruction or many instructions, and can even be distributed on a plurality of different code segments, distributed among different programs, and distributed across a plurality of memory devices. Similarly, operating data can be identified in the module, and can be implemented and organized in the data structure of any appropriate type according to any appropriate form. The operating data can be collected as a single data set, or can be distributed in different locations (including on different storage devices), and can only be present on a system or network as an electronic signal at least in part.

[0234] When a module can be implemented by software, considering the level of existing hardware technology, a person skilled in the art can build a corresponding hardware circuit to implement the corresponding function of the module that can be implemented by software without considering the cost. The hardware circuit includes a conventional very large scale integration (VLSI) circuit or gate array and existing semiconductors such as logic chips, transistors, or other discrete components. The module can also be implemented by a programmable hardware device, such as a field programmable gate array, a programmable array logic, a programmable logic device, etc.

[0235] The above exemplary embodiments are described with reference to the accompanying drawings, and many different forms and embodiments are feasible without departing from the spirit and teachings of the present invention. Therefore, the present invention should not be constructed as a limitation of the exemplary embodiments proposed herein. More specifically, these exemplary embodiments are provided so that the present invention will be perfect and complete, and the scope of the present invention will be conveyed to those who are familiar with the technology. In these figures, the component sizes and relative sizes may be exaggerated for clarity. The terms used here are only based on the purpose of describing specific exemplary embodiments and are not intended to be limiting. As used herein, unless the text clearly indicates otherwise, the singular forms "one", "an" and "the" are intended to include these multiple forms. It will be further understood that the terms "including" and / or "comprising" when used in this specification indicate the presence of the features, integers, steps, operations, components and / or components, but do not exclude the presence or increase of one or more other features, integers, steps, operations, components, components and / or their groups. Unless otherwise indicated, when stated, a range of values ​​includes the upper and lower limits of that range and any subranges therebetween.

[0236] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A data processing method, applied to a first device, characterized in that: include: receiving a first signal sent by a second device, wherein the first signal carries only a text feature; Decoding the first signal to obtain a first semantic feature; According to the first sound characteristic of the user on the second device side and the first semantic feature, first voice information of the first sound characteristic is output.

2. The data processing method according to claim 1, characterized in that: The language type of the text corresponding to the first semantic feature is the same as the language type corresponding to the voice information; Outputting first voice information of the first voice characteristic according to the first voice characteristic of the user on the second device side and the first semantic feature includes: Decoding the first semantic feature to obtain first text information; synthesizing the first voice information according to the first text information and the first sound characteristic; The first voice information is output.

3. The data processing method according to claim 1, characterized in that: Before outputting the first voice information of the first voice characteristic according to the first voice characteristic of the user on the second device side and the first semantic feature, the method further includes: The first voice characteristic of the user on the second device side is obtained according to the identification information of the second device.

4. The data processing method according to claim 1 or 3, characterized in that: Before outputting the first voice information of the first voice characteristic according to the first voice characteristic of the user on the second device side and the first semantic feature, the method further includes: receiving the first sound characteristic sent by the second device; Acquire the first sound characteristic from the cloud according to the received identification information of the second device; The first sound characteristic is selected from the sound data stored in the first device according to the identification information of the second device.

5. The data processing method according to claim 1, characterized in that: The method further comprises: Output text information and / or second voice information according to the first semantic feature.

6. The data processing method according to claim 5, characterized in that: The language type of the text corresponding to the text semantic feature is a first language type, and outputting text information according to the first semantic feature includes: Decoding the first semantic feature to obtain text information of a first language type and outputting the text information of the first language type; and / or, Converting the first semantic feature to obtain a second semantic feature; Decoding the second semantic feature to obtain and output the text information in a second language type; The first language type is different from the second language type.

7. The data processing method according to claim 5, characterized in that: The language type of the text corresponding to the first semantic feature is a first language type, and outputting the second voice information according to the first semantic feature includes: Converting the first semantic feature to obtain a second semantic feature; Decoding the second semantic feature to obtain second text information; According to the second text information, second voice information in a second language type is synthesized and output.

8. A data processing method, applied to a second device, characterized in that: include: Encoding the received speech signal to obtain a third semantic feature corresponding to the speech signal; Mapping the third semantic feature to obtain a first signal carrying only text features; The first signal is sent to the first device.

9. The data processing method according to claim 8, characterized in that: The step of encoding the received speech signal to obtain a third semantic feature corresponding to the speech signal includes: Performing spectrum extraction on the received speech signal to obtain a spectrum matrix; Performing semantic extraction on the frequency spectrum matrix to obtain the third semantic feature.

10. The data processing method according to claim 9, characterized in that: The step of performing spectrum extraction on the received speech signal to obtain a spectrum matrix includes: The spectrum matrix is ​​obtained by performing spectrum extraction on the speech signal through Hamming window, fast Fourier transform, logarithmic operation and normalization operation.

11. The data processing method according to claim 9 or 10, characterized in that: The step of encoding the received speech signal to obtain a third semantic feature corresponding to the speech signal includes: Performing semantic compression on the spectrum matrix by a first encoder to obtain first semantic representation information; Performing feature extraction on the first semantic representation information by a second encoder to obtain the third semantic feature; Wherein, the first encoder comprises: a multi-layer cascaded convolutional neural network; The second encoder includes: a multi-layer convolutional neural network based on a gated recurrent unit, a fully connected layer, and a normalized exponential layer.

12. A data processing device, applied to a first device, characterized in that: include: A receiving module, configured to receive a first signal sent by a second device, wherein the first signal carries only a text feature; A decoding module, used for decoding the first signal to obtain a first semantic feature; An output module is used to output first voice information of the first sound characteristic according to the first sound characteristic of the user on the second device side and the first semantic feature.

13. A data processing device, applied to a second device, characterized in that: include: An encoding module, used for encoding the received speech signal to obtain a third semantic feature corresponding to the speech signal; A mapping module, used for mapping the third semantic feature to obtain a first signal carrying only text features; A sending module is used to send the first signal to a first device.

14. A terminal device, characterized in that: include: transceivers and processors; The transceiver is used to receive a first signal sent by a second device, wherein the first signal carries only text features; The processor is used to decode the first signal to obtain a first semantic feature; According to the first sound characteristic of the user on the second device side and the first semantic feature, first voice information of the first sound characteristic is output.

15. A terminal device, characterized in that: include: transceivers and processors; The processor is used to encode the received voice signal to obtain a third semantic feature corresponding to the voice signal; Mapping the third semantic feature to obtain a first signal carrying only text features; The transceiver is used to send the first signal to the first device.

16. A terminal device, comprising: A transceiver, a processor, a memory, and a program or instruction stored in the memory and executable on the processor; wherein the processor implements the data processing method as described in any one of claims 1 to 7 when executing the program or instruction; or, the processor implements the data processing method as described in any one of claims 8 to 11 when executing the program or instruction.

17. A readable storage medium having a program or instruction stored thereon, characterized in that: When the program or instruction is executed by the processor, the steps in the data processing method as described in any one of claims 1 to 7 are implemented; or, when the program or instruction is executed by the processor, the steps in the data processing method as described in any one of claims 8 to 11 are implemented.

Images