Data conversion method and device and electronic equipment

By encoding and decoding text, audio and video data, and using voice generators to generate podcast audio, the problems of high cost and insufficient quality of podcast audio generation are solved, and efficient and personalized podcast audio generation is achieved.

CN118918877AActive Publication Date: 2024-11-08NETEASE (HANGZHOU) NETWORK CO LTD

Patent Information

Application Number
CN202410969484.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-18
Publication Date
2024-11-08
Estimated Expiration
2044-07-18

AI Technical Summary

Technical Problem

Podcast audio generation is expensive and cannot achieve large-scale automated production. The existing speech synthesis technology has shortcomings in emotional expression, sound quality and personalization, making it difficult to generate natural, smooth and emotional high-quality podcast content.

Method used

By obtaining text, audio and video data, encode and map it to the vector space of preset dimensions, decoding processing to obtain semantic information, and using a speech generator to generate continuous audio signals, realizing the end of multiple data types to audio signals To-end generation, simplify operational processes and improve production efficiency.

Benefits of technology

It realizes end-to-end generation from multiple data types to audio signals, simplifies the voice generation process, improves the production efficiency and quality of audio signals, and meets the high-quality and personalized needs of podcast audio.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918877A_ABST
    Figure CN118918877A_ABST
Patent Text Reader

Abstract

The invention provides a data conversion method and device and electronic equipment, and the method comprises the steps: obtaining input data, carrying out the coding processing of the input data, and obtaining a coding vector; wherein the input data comprises at least one of the following data: text data, audio data and video data; mapping the coding vector to a vector space of a preset dimension to obtain a first vector corresponding to the coding vector; performing decoding processing on the first vector to obtain decoding information; wherein the decoding information is used for indicating semantic information corresponding to the input data; converting the decoding information into a continuous audio signal through a voice generator; wherein the audio signal is matched with semantic information corresponding to the input data. According to the mode, end-to-end generation from the input data of various data types to the audio signals is realized, intermediate links and additional processes are not needed, the operation process of voice generation is simplified, and meanwhile, the mode improves the production efficiency of the audio signals while ensuring the audio conversion quality.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a data conversion method, device and electronic device. Background Art

[0002] Podcast is a form of publishing and disseminating audio content, which is shared and subscribed through the Internet platform. Podcast audio in related technologies requires the participation of hosts, guests and technicians, especially experts in related fields to record, which is costly. At the same time, high-quality recording equipment and professional recording studio environment increase the hardware cost of audio content generation, resulting in the inability to achieve large-scale automated production of podcast audio. Summary of the invention

[0003] The purpose of the present disclosure is to provide a data conversion method, device and electronic device to convert any video data, audio data and text data into podcast audio, so as to ensure the quality of audio conversion and enable podcast audio to be mass-produced.

[0004] In a first aspect, the present disclosure provides a data conversion method, which includes: obtaining input data, encoding the input data, and obtaining a coding vector; wherein the input data includes at least one of the following data: text data, audio data, and video data; mapping the coding vector to a vector space of a preset dimension to obtain a first vector corresponding to the coding vector; decoding the first vector to obtain decoding information; wherein the decoding information is used to indicate semantic information corresponding to the input data; through a speech generator, converting the decoding information into a continuous audio signal; wherein the audio signal matches the semantic information corresponding to the input data.

[0005] In a second aspect, the present disclosure provides a data conversion device, which includes: a data acquisition module, used to acquire input data, encode the input data, and obtain a coding vector; wherein the input data includes at least one of the following data: text data, audio data, and video data; a vector mapping module, used to map the coding vector to a vector space of a preset dimension, and obtain a first vector corresponding to the coding vector; a vector decoding module, used to decode the first vector and obtain decoding information; wherein the decoding information is used to indicate semantic information corresponding to the input data; a speech generation module, used to convert the decoding information into a continuous audio signal through a speech generator; wherein the audio signal matches the semantic information corresponding to the input data.

[0006] In a third aspect, the present disclosure provides an electronic device, which includes a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the above-mentioned data conversion method.

[0007] In a fourth aspect, the present disclosure provides a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the above-mentioned data conversion method.

[0008] The embodiments of the present disclosure bring the following beneficial effects:

[0009] The present disclosure provides a data conversion method, device and electronic device, which first obtain input data, encode the input data and obtain a coding vector; wherein the input data includes at least one of the following data: text data, audio data and video data; then the coding vector is mapped to a vector space of a preset dimension to obtain a first vector corresponding to the coding vector; then the first vector is decoded to obtain decoding information; wherein the decoding information is used to indicate the semantic information corresponding to the input data; then the decoding information is converted into a continuous audio signal through a speech generator; wherein the audio signal matches the semantic information corresponding to the input data. This method realizes end-to-end generation of audio signals from input data of multiple data types, does not require intermediate links and additional processes, simplifies the operational process of speech generation, and at the same time, improves the production efficiency of audio signals while ensuring the quality of audio conversion.

[0010] Other features and advantages of the present disclosure will be set forth in the following description, or some features and advantages may be inferred or unambiguously determined from the description, or may be learned by implementing the above-mentioned technology of the present disclosure.

[0011] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, the following specifically cites preferred implementation modes and describes them in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0013] Figure 1 A flowchart of a data conversion method provided by an embodiment of the present disclosure;

[0014] Figure 2 A schematic diagram of the structure of a data conversion device provided in an embodiment of the present disclosure;

[0015] Figure 3A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. The components of the embodiments of the present disclosure described and shown in the drawings here can be arranged and designed in various different configurations.

[0017] Therefore, the following detailed description of the embodiments of the present disclosure provided in the accompanying drawings is not intended to limit the scope of the present disclosure claimed for protection, but merely represents selected embodiments of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present disclosure.

[0018] Podcast is a form of publishing and disseminating audio content, which is shared and subscribed through the Internet platform. Users can listen to podcast programs through mobile devices or computers, and obtain information and entertainment anytime and anywhere. Podcast content is usually presented in the form of dialogue, interviews, storytelling and special reports, covering a wide range of topics, including education, technology, news, entertainment, culture and health. The characteristics of the podcast format include the following:

[0019] 1. The characteristic of knowledge dissemination is that users can listen to podcasts at any time and any place, which is suitable for listening in fragmented time such as driving, fitness, housework, etc.; podcast programs usually delve into specific topics and provide more detailed and in-depth content than other media; covering various topics to meet the needs and interests of different users.

[0020] 2. Interestingness. Many podcast programs adopt the interview format, with the host interacting with the guests and the discussions being lively and interesting. Podcast hosts can create programs based on their personal style and expression to form a unique listening experience. They can convey emotions through sound and establish an emotional connection with the audience.

[0021] However, podcast audio needs to be recorded manually, which means that the participation of the host, guests, and technicians is required, especially experts in related fields are needed for recording, which is costly. At the same time, high-quality recording equipment and professional studio environment increase the hardware cost. In addition, the recording cycle of podcast audio is relatively long, and it requires pre-processing and post-processing. Pre-processing includes topic selection, script writing, inviting guests and other preparations, which takes a long time. The recording process requires multiple attempts and adjustments to ensure the quality and effect of the content. At the same time, post-processing includes editing, mixing, sound quality optimization, etc., which require professional operation. Moreover, podcast audio processing and editing are difficult. First of all, the recorded audio needs to be edited to remove redundant content and defects to ensure the continuity and smoothness of the program. The audio also needs to be processed by noise reduction, equalization and dynamic range compression to ensure clarity and sound quality.

[0022] Therefore, most podcast content generation relies on manual editing, which is inefficient and cannot achieve large-scale automated production. In addition, existing speech synthesis technology has deficiencies in emotional expression, sound quality, and personalization. For example, Text-to-Speech systems usually lack natural emotional expression and high-fidelity sound quality in the generated speech, making it difficult to generate natural, fluent, and emotional podcast content. Moreover, the generated content lacks coherence and logical structure, making it difficult to maintain a high-quality user experience.

[0023] Based on the above problems, the embodiments of the present invention provide a data conversion method, device and electronic device. The technology can be applied to the scenario of converting video data, audio data and text data into audio signals in a unified format.

[0024] In order to facilitate understanding of the embodiment of the present invention, a data conversion method disclosed in the embodiment of the present invention is first described in detail. Figure 1 As shown, the method comprises the following specific steps:

[0025] Step S102, obtaining input data, encoding the input data, and obtaining a coding vector; wherein the input data includes at least one of the following data: text data, audio data, and video data.

[0026] In specific implementation, the input data can be text data, audio data or video data, or text data and audio data, or audio data and video data, or text data and video data, or text data, audio data and video data. In order to uniformly convert input data of multiple data types into podcast audio content, it is first necessary to preprocess and encode the input data of various data types. This step is crucial because it converts various forms of input into a unified representation for subsequent processing and generation in the same framework.

[0027] Since the characteristics of the input data corresponding to different data types are different, the preprocessing rules and encoding rules corresponding to different data types are different, but the formats of the encoding vectors corresponding to different data types are the same. Among them, the preprocessing rules and encoding rules corresponding to each data type are pre-set according to research and development requirements, and are not specifically limited here. The subsequent embodiments will describe in detail the preprocessing and encoding methods for input data.

[0028] Specifically, the above input data may include casually recorded videos, news hot spots, chat records of friends' conversations, phone recordings, TV programs and other content, which can be arbitrarily combined to generate audio signals. This flexibility allows users to make full use of data from various sources and improve the content richness of audio signals. Compared with traditional podcast recording, which mainly relies on a single modality (such as text or audio) and requires manual integration of data of different data types, this method can achieve unified processing and generation of data of multiple data types through automated means.

[0029] Step S104, mapping the encoding vector to a vector space of a preset dimension to obtain a first vector corresponding to the encoding vector.

[0030] In a specific implementation, in order to uniformly represent the coded vector, it is necessary to map the coded vector to a vector space of a preset dimension to obtain a coded vector mapped to a vector space of a preset dimension, that is, to obtain a first vector. The above preset dimension can be determined according to R&D requirements. This method can map the coded vectors corresponding to input data of different data types to a vector control of a uniform dimension, thereby facilitating subsequent generation for decoding and speech conversion processing.

[0031] Step S106, decoding the first vector to obtain decoding information; wherein the decoding information is used to indicate semantic information corresponding to the input data.

[0032] In a specific implementation, the above-mentioned first vector is a vector of preset dimension. The first vector is input into a pre-trained decoder. The decoder can analyze the contextual relationship in the first vector to obtain semantic information corresponding to the input data, so that the decoder outputs decoding information containing voice information corresponding to the input data.

[0033] Step S108, converting the decoded information into a continuous audio signal through a speech generator; wherein the audio signal matches the semantic information corresponding to the input data.

[0034] Specifically, the decoded information output by the decoder needs to be passed to the speech generator, which uses speech synthesis technology to convert the decoded information into a natural and smooth speech signal. In the process of generating the speech signal, the tone and rhythm of the audio output can be dynamically adjusted according to the input characteristics to ensure that the generated audio is natural and smooth. For example, if the output generated by the decoder is "Welcome to our podcast program", the speech generator converts this information into a natural and smooth speech signal.

[0035] The above data conversion method realizes the end-to-end generation of audio signals from input data of various data types without the need for intermediate links and additional processes, thus simplifying the operational flow of speech generation. At the same time, this method improves the production efficiency of audio signals while ensuring the quality of audio conversion.

[0036] The following embodiments are used to describe the manner of preprocessing and encoding input data.

[0037] Specifically, the specific process of encoding the input data to obtain the encoding vector can be implemented through the following steps 10-12:

[0038] Step 10, standardize the input data according to preset rules to obtain standardized input data; wherein the preset rules include standardized processing rules corresponding to multiple preset data types, and the multiple preset data types include text type, audio type and video type.

[0039] In specific implementation, the above-mentioned preset rules include standardized processing rules corresponding to each preset data type in multiple preset data types. Different preset data types correspond to different standardized processing rules. The standardized processing rules corresponding to each preset data type can be determined according to R&D needs.

[0040] For example, the standardization processing rules corresponding to the text type include: removing extra spaces, special characters, unifying uppercase and lowercase characters, etc. The standardization processing rules corresponding to the text type can be used to standardize the input data of the text type, that is, to remove or replace special characters, including but not limited to HTML (HyperText Markup Language) tags and URL (Uniform Resource Locator), and then convert the uppercase letters in the input data of the text type into lowercase letters for subsequent processing, and then use regular expressions to clean up extra spaces and punctuation marks. The standardization processing rules corresponding to the audio type include: denoising and volume standardization. Since the audio data is a continuous audio signal, it is necessary to standardize the input audio data through the standardization processing rules corresponding to the audio type, that is, to remove the background noise in the audio data through a filter, and then adjust the volume so that different audio data have a consistent loudness level. The standardization processing rules corresponding to the video data include: frame rate adjustment, resolution standardization, etc. Video data can be standardized through the standardized processing rules corresponding to video data, that is, the frame rate of the video data is adjusted so that different video clips have a consistent frame rate (such as 30 frames per second), and then the video resolution is adjusted to have a consistent size (such as 224x224 pixels).

[0041] Step 11, dividing the input data after the standardization process to obtain a plurality of divided sub-data.

[0042] In specific implementation, since the input data after standardization processing varies in length, it is necessary to divide the input data after standardization processing into multiple sub-data of equal length so as to uniformly encode the sub-data. Among them, the division methods corresponding to input data of different preset data types are different. For example, text data needs to be segmented into words, video data needs to be segmented into frames, and audio data needs to be segmented into frames.

[0043] Text segmentation is to divide a continuous text data into meaningful words or subword units. For example, the Byte-Pair Encoding (BPE) algorithm can be used, which gradually merges the most frequently occurring characters or subword pairs to decompose text data into smaller units while maintaining good semantic representation. For example, "hello world" can be decomposed into "he", "ll", "o", "wor", and "ld". Among them, subwords refer to text units between words and characters. Subwords divide a continuous text into smaller and meaningful units. These units can be either whole words or parts of words. The subword segmentation method can find a balance between words and characters. It can process word-level information and effectively process new or rare words by dividing them into subword units.

[0044] Audio framing is to divide continuous audio data into multiple small segments for further processing. For example, the audio data is divided into audio frames of 20-50 milliseconds in length, and there may be some overlap between each frame (such as 10 milliseconds). 10 seconds of audio data can be divided into 500 20 millisecond audio frames. Specifically, audio data can be divided into multiple short time frames using, including but not limited to, audio processing tools. For example, the audio data can be framed using the functions in the Librosa library.

[0045] Video framing is the process of dividing video data into multiple independent video frames, each of which is an image. To divide video data into independent video frames, it is necessary to ensure that each second of video contains the same number of frames. For example, a 10-second video data with a frame rate of 30 frames per second will be divided into 300 frames.

[0046] Step 12: For multiple sub-data, encode the sub-data to obtain an encoding vector corresponding to each sub-data.

[0047] In specific implementation, the sub-data of different preset data types have different corresponding encoding rules, and the encoding rules corresponding to each preset data type are pre-set. According to the preset encoding rules, each sub-data can be encoded into a string (equivalent to the above encoding vector) so that the string can be subsequently decoded uniformly.

[0048] The encoding rule for text data is to convert the text after word segmentation (that is, the above sub-data) into a vector representation so that it can be processed by the machine learning model. Specifically, you can use an embedding layer including but not limited to a variety of pre-trained language models (such as BERT, GPT, Transformer, etc.) to convert each text word segmentation into a vector representation of fixed dimension. For example, "hello" may be converted into a vector of length 768: [0.1, 0.2, 0.3, ..., 0.768].

[0049] In the encoding process of audio data, it is first necessary to generate a spectrogram corresponding to each sub-data, and then convert the spectrogram into a vector representation of fixed dimension. Specifically, the spectrogram is a representation of audio data in the frequency domain, which is used to describe the frequency change of the audio signal over time. The time domain signal of each frame in the sub-data can be converted into a frequency domain representation using a short-time Fourier transform to obtain a spectrogram. For example, the audio data is first divided into several short time periods (frames), each frame is usually 20-50 milliseconds, and a window function (such as a Hamming window) is applied to each frame to reduce the edge effect. Then, a fast Fourier transform is performed on each frame to convert the time domain signal into a frequency domain signal, and then the amplitude square of each frequency domain signal is taken to obtain a power spectrum. The power spectrum of each frame is arranged together to form a two-dimensional image, with the horizontal axis being time, the vertical axis being frequency, and the color or brightness representing power. After obtaining the spectrogram, the spectrogram needs to be converted into a vector representation so that it can be processed by a machine learning model. Specifically, a pre-trained audio model (such as Audio Spectrogram Transformer, AST) can be used to encode the spectrogram to generate a vector representation of fixed dimension. For example, a spectrum graph is encoded as a vector of length 512: [0.1, 0.2, 0.3, ..., 0.512].

[0050] In the encoding process of video data, it is first necessary to extract the feature vector of the sub-data, and then convert the feature vector of the word count into a unified vector representation to align it with the vector representation of text data and voice data. Specifically, for each video frame corresponding to the video data, it is necessary to use a pre-trained visual encoder (ViT for short) to extract feature vectors from the image corresponding to each video frame; wherein ViT is a model that uses the Transformer architecture to process image data and can extract global visual features. The working principle is: first, the image is divided into blocks of fixed size, such as 16x16 pixels, and then each block is flattened into a one-dimensional vector, and mapped to a high-dimensional space through a linear transformation, and a position code is added to each block to retain spatial information; then the block embedding and position code are input into the Transformer, and the relationship between each block is calculated using the self-attention mechanism to generate a global representation of the image. After obtaining the feature vector of the video frame, it is necessary to linearly transform the feature vector of the video frame so that it is in the same vector space as the vector representation of text and audio. For example, the feature vector of the video frame is mapped to a vector space of length 768 to align with the text representation.

[0051] The above method can convert input data of different modalities into a unified vector representation by preprocessing and encoding text, audio, and video data. These vector representations will be passed to the multimodal model for further processing and generation, and ultimately achieve end-to-end podcast audio content output. These steps ensure that data of various input modalities can be processed uniformly in the same framework, laying a solid foundation for subsequent multimodal generation.

[0052] The following embodiments are used to describe the method of uniformly representing vectors.

[0053] Specifically, the specific process of mapping the coding vector to a vector space of preset dimension to obtain a first vector corresponding to the coding vector may include: mapping the coding vector to a preset vector space to obtain a mapped coding vector; and mapping the mapped coding vector to a vector space of preset dimension by multiplying the mapped coding vector with a target matrix of preset dimension to obtain a first vector.

[0054] In the specific implementation, in order to uniformly process different preset data types, it is necessary to map the encoding vectors corresponding to different preset data types to the same representation space, which is also the vector space of the above preset dimensions. Specifically, a cross-modal attention mechanism can be used to map data of different modalities (where the modality is equivalent to the preset data type) to the same representation space. For example, the cross-modal attention network (MCAN) can be used to align text, audio, and video features.

[0055] In a specific implementation, the above-mentioned process of mapping the coding vector to a preset vector space to obtain the mapped coding vector may include: generating a position code and a modal tag corresponding to the coding vector; wherein the position code is used to indicate the order relationship of each element in the coding vector, and the modal tag is used to indicate the data type of the input data corresponding to the coding vector; and embedding the position code and the modal tag into the coding vector to obtain the mapped coding vector.

[0056] Specifically, the above-mentioned position coding is used to add position information to each coding vector to preserve the order relationship of the elements in the coding vector. This position information is required in the subsequent decoding process because they do not have order information in essence. In practical applications, the process of generating position coding can include: generating corresponding position information for each input element (such as words in text, frames in audio, frames in video), and the position coding is usually generated using sine and cosine functions to ensure that the coding of different positions is unique, and then the generated position information is added to each coding vector to form a feature representation with position information. For example, for text input, the generated position coding is added to the feature vector of the text, that is, for each position, the generated sine and cosine values ​​are added to the corresponding dimensions of the feature vector to obtain a coding vector with position coding.

[0057] The above modal labeling is to add labeling information to the encoding vector of each modality to distinguish data of different modalities. Among them, modality is also a data type, which includes text type, audio type and video type. The process of generating modal labels includes: generating a fixed embedding vector for each modality as a label of the modality. For example, the labeling vector of the text type can be a specific fixed vector, and the labeling vectors of the audio type and the video type are other fixed vectors respectively; then the modal label is embedded in the encoding vector of the corresponding modality, so that the model can recognize and distinguish data of different modalities. For example, the text encoding vector is added with EtextE_{text}Etext (equivalent to the modal label), and the audio encoding vector is added with EaudioE_{audio}Eaudio (equivalent to the modal label).

[0058] After obtaining the mapped coding vector, it is also necessary to map the mapped coding vector to a uniform vector space of preset dimensions through a target matrix of preset dimensions to obtain a first vector, so as to subsequently uniformly process the coding vectors of different modalities. Here, the feature vectors of different modalities are mapped to the vector space of the same dimension by means of linear transformation. During linear transformation, it is necessary to define a linear transformation layer (fully connected layer) for each modality to map the input feature vector to a preset dimension. For example, the coding vectors of audio, video, and text are all mapped to a 768-dimensional vector space. Among them, the target matrix corresponding to the linear transformation layer is learned during the training process.

[0059] In the above method, the cross-modal attention mechanism and linear transformation technology are used to achieve the alignment and fusion of different modal data, ensuring the coherence and consistency of the generated content. This technology allows multimodal data to be processed in the same representation space, which is rare in the existing technology.

[0060] The following embodiments are used to describe the method of feature fusion.

[0061] Specifically, the above-mentioned input data includes data of at least two data types, that is, the input data is multimodal data; based on this, after mapping the encoding vector to a vector space of a preset dimension to obtain a first vector corresponding to the encoding vector, it is also necessary to perform feature fusion processing on the first vector corresponding to the input data of at least two data types to obtain the first vector after feature fusion, so that the first vector after feature fusion can be subsequently decoded to obtain decoding information.

[0062] For example, the input data includes text data, audio data and video data, which requires feature fusion processing of the first vector corresponding to the text, the first vector corresponding to the video and the first vector corresponding to the audio to generate a unified encoding vector to obtain a unified vector representation.

[0063] In specific real time, the specific process of performing feature fusion processing on the first vectors corresponding to the input data of at least two data types to obtain the first vector after feature fusion may include: determining the fusion weight corresponding to the first vector of each data type of at least two data types; multiplying the first vector of each data type with the corresponding fusion weight to obtain the multiplication result corresponding to each data type, and adding the multiplication results corresponding to each data type to obtain the first vector after feature fusion.

[0064] Specifically, an attention layer can be designed to calculate the fusion weights between the first vectors corresponding to different data types, and then use the attention mechanism to the first vectors corresponding to different data types and fuse them into a unified feature vector representation. For example, the fusion weights between the first vectors corresponding to text and audio are calculated, and the first vectors corresponding to the two are fused into a unified 768-dimensional vector. Among them, the attention mechanism is a method for calculating the relationship between vectors, especially for processing the relationship between data of different modalities (such as text, audio, and video). It highlights important information and reduces the impact of irrelevant information on the model by calculating the fusion weights. Through the calculation of fusion weights, vectors of different data types can be aligned and fused in the same representation space, ensuring that the information between vectors corresponding to multiple data types can complement each other and improve the performance of the model.

[0065] The purpose of calculating the fusion weight is to highlight the important information in the vectors of different data models, to align the vectors corresponding to different data types, and to ensure that they are represented and processed in the same vector space; through the attention mechanism, the model can dynamically focus on the most important part of the input data for the generation task, and improve the generation effect and quality. In a specific embodiment, the step of calculating the fusion weight includes: the input data includes text data, audio data and video data, and the first vector includes a text feature vector, an audio feature vector and a video feature vector; for the feature vector corresponding to each data type, based on the linear transformation of the characteristic vector, a weight matrix Q and a weight matrix K are obtained, and the dot product of the weight matrix Q and the weight matrix K is calculated, and then divided by a scaling factor to obtain a fusion weight matrix, and then the fusion matrix is ​​normalized to obtain a normalized fusion weight matrix.

[0066] It should be noted that in the encoding stage, the encoding vectors corresponding to each data type have been mapped to a unified vector space of preset dimensions, which only converts data of different data types into vector representations of the same dimension for subsequent processing. However, even if the vectors are unified in dimension, the information and semantics they contain may not be completely aligned, so feature fusion is needed to ensure that the information between different data types is consistently represented and combined in the semantic space.

[0067] In a specific embodiment, the feature fusion method may include: using a cross-modal attention mechanism so that the information between vectors of different data types can refer to and complement each other, and the vectors are aligned, and the alignment process ensures the semantic consistency of text, audio and video features, and the aligned vectors are weighted summed to generate a unified feature representation, that is, a fused vector is obtained, which contains information of all data types and improves the performance of subsequent generation tasks.

[0068] In the above feature fusion method, the fused vector can better represent the overall picture of the input data and improve the model's ability to understand the input data. At the same time, through alignment and fusion, the model can more accurately generate the target output (such as high-quality audio), improving the effect of the generation task.

[0069] The following embodiments are used to describe the decoding method and the method of generating an audio signal.

[0070] Specifically, the above-mentioned decoding process of the first vector to obtain the decoding information may include: inputting the first vector into a pre-trained decoder, obtaining the association relationship between the elements in the first vector through the decoder, and obtaining the decoding information corresponding to the first vector based on the association relationship and the position encoding corresponding to the first vector.

[0071] In specific implementation, the correlation between each element in the first vector and other elements can be calculated by self-attention, that is, multi-head self-attention calculation can be performed on the first vector to obtain the context representation of each time step to determine the association relationship between the elements in the first vector, thereby capturing the global dependency. Since the decoder does not have sequential information, the position encoding is used to provide the position information between the elements in the first vector. The association relationship and position information between the elements in the first vector are combined to obtain the decoding information corresponding to the first vector, which includes the semantic information of the input data corresponding to the first vector.

[0072] In a specific embodiment, the specific process of obtaining the decoding information corresponding to the first vector based on the association relationship and the position coding corresponding to the first vector may include: dividing the first vector into multiple sub-vectors according to the position coding corresponding to the first vector; for multiple sub-vectors, generating the decoding information corresponding to the current sub-vector based on the relationship between the elements in the current sub-vector through a decoder, and generating the decoding information corresponding to the next sub-vector of the current sub-vector based on the association relationship between the decoding information corresponding to the current sub-vector and the elements in the next sub-vector of the current sub-vector; combining the decoding information corresponding to the multiple sub-vectors to obtain the decoding information corresponding to the first vector. In this method, the decoding information generated by each sub-vector will be used as the input of the next sub-vector corresponding to it, so as to gradually obtain the entire decoding information.

[0073] In an optional embodiment, the decoder predicts the audio signal of each time step in the first vector after feature fusion based on an autoregressive generation mechanism, thereby gradually generating semantic and audio signals. For example, assuming that the first vector represents a podcast content introducing a certain topic, the decoder will gradually generate the audio signal of the podcast. Among them, the autoregressive mechanism generates outputs step by step, and the output of each step is used as the input of the next step. This generation method ensures the coherence and consistency of the generated content.

[0074] Furthermore, in order to generate high-quality audio signals, it is necessary to perform denoising and refinement processing on the decoded information to obtain high-fidelity decoded information, so that the speech generator can perform speech generation processing on the high-fidelity decoded information.

[0075] In specific implementation, the latent diffusion model can be used for denoising and refinement, that is, by diffusing the decoded information in the latent space, gradually denoising to generate high-quality target data, which is the high-fidelity decoded information mentioned above. Specifically, in audio generation, the latent diffusion model first randomly samples the decoded information in the latent space to generate an initial low-resolution audio signal, and then gradually improves the quality and details of the audio through the diffusion process to generate a high-resolution audio signal, that is, through a series of diffusion steps, each step is based on the output of the previous step, and gradually improves the quality and details of the audio.

[0076] The diffusion process can also be implemented through the following steps: the potential diffusion model learns the distribution of data by gradually adding noise and then denoising, that is, adding noise in the initial stage to generate a low-resolution audio signal, and then gradually denoising to generate a high-resolution audio signal. For example, assuming that the initially generated low-resolution audio signal represents "Welcome to our podcast program", the diffusion process gradually improves the quality of the audio and finally generates a high-fidelity audio signal. Then, the high-fidelity audio signal is input into the speech generator to generate an audio signal with a natural flow.

[0077] The above method, based on the autoregressive generation mechanism and the latent diffusion model, realizes the direct generation of high-quality audio content from multimodal input, avoiding intermediate links and additional processes, and ensuring that the generated audio is natural and smooth and the content is coherent.

[0078] The following examples are used to provide a detailed introduction to the training of the model used in the present invention.

[0079] We first introduce the data set preparation and preprocessing.

[0080] The dataset includes a large amount of text data, audio data, and video data. Among them, text data includes news releases, speech manuscripts, podcast scripts, etc., covering a wide range of topics; audio data includes podcast recordings, speech recordings, conversation recordings, etc.; video data includes lecture videos, interview videos, news videos, etc. Specifically, web crawlers can be used to collect data from public podcast platforms, news websites, video sharing platforms, etc., and audio and video recording devices can also be used to collect high-quality original content.

[0081] After obtaining the data in the dataset, the data needs to be annotated: annotate the text data with themes, key points, and emotions; annotate the voice, music, and noise parts in the audio data; annotate the important frames, characters, and scenes in the video data. After annotating the data, the data needs to be cleaned and preprocessed. Cleaning is to remove noise and irrelevant information in the data, ensure data quality, and ensure the time synchronization of text data, audio data, and video data by aligning the data of each data type through timestamps.

[0082] In order to ensure that data of multiple data types can be effectively processed and trained by the multimodal language model, the training data preprocessing step is crucial. Data preprocessing includes data alignment and data standardization. Data alignment is achieved by:

[0083] 1. Text-Audio Alignment

[0084] Add timestamps to text data and audio data to mark the start and end time of each sentence. For example, in a lecture recording, mark the start and end time of each sentence, and the text of the lecture recording is annotated as: "Hello everyone, welcome to the lecture. [00:00:05-00:00:10]". According to the timestamp, divide the audio data into corresponding sentence segments and align them with the text sentences. For example, a certain audio segment in a lecture recording is synchronized with the corresponding text sentence. In the specific implementation, you can use tools including but not limited to audio processing tools (such as Praat) for timestamp annotation. Taking Praat as an example, here you can manually or automatically annotate timestamps, and then segment the audio according to the annotation results to obtain audio segments synchronized with the text, thereby achieving the data alignment goal.

[0085] 2. Text-Video Alignment

[0086] First, extract key frames from the video data. These frames usually correspond to important events or scenes in the video data. For example, extract the start and end frames of each speech from the news video. Then add a timestamp to each key frame to mark the specific time when the frame appears. For example, mark the timestamp on the key frame of each speech. Finally, align the video frame with the text content based on the timestamp. For example, a text description in a news video is synchronized with the corresponding video frame. In the specific implementation, key frame extraction and time labeling can be used, including but not limited to video processing tools (such as FFmpeg). Taking FFmpeg as an example, we can extract key frames in the video and add timestamps to each frame, and then align these frames with the text content to achieve our data alignment goal.

[0087] When standardizing data, first calculate the mean and standard deviation of the feature vectors of each data type. For example, calculate the mean and standard deviation of each frequency component in the audio spectrogram; then standardize the feature vectors so that their mean is 0 and their standard deviation is 1, for example, subtract the mean from each frequency component in the audio spectrogram and divide it by the standard deviation. In specific implementations, feature standardization can be performed using tools including but not limited to data processing tools (such as NumPy). Taking NumPy as an example, here we can calculate the mean and standard deviation of the feature vectors, and then standardize them to achieve our feature standardization goal.

[0088] After completing the dataset preparation, the next step is to use the dataset to train a multimodal language model. The following are detailed technical steps, describing the operation process of each step from a technical perspective:

[0089] 1. Data loading and batch processing:

[0090] Data loading refers to loading preprocessed datasets, including feature vectors of text data, audio data, and video data. Specifically, the preprocessed dataset is read from the storage system, and then the data in the dataset is organized into an easy-to-process format, such as a tensor or array, to ensure that the data maintains the consistency of its feature vectors and annotations during the loading process. In order to facilitate model training, a data loader is used to load the data in the dataset into memory in batches.

[0091] Divide the data in the loaded dataset into several batches to reduce memory usage and improve computing efficiency. First, divide the data in the dataset into batches of fixed size (such as 32 data samples per batch), and then ensure that the data in each batch is consistent in data type and time. Specifically, batch processing technology is used to divide the data into small blocks suitable for parallel computing to improve training efficiency and speed.

[0092] 2. Model Initialization

[0093] The multimodal language model includes an encoder, a decoder, a fusion layer, and a generation layer. According to the predefined model architecture, the encoder (such as a text encoder, an audio encoder, and a video encoder) is initialized, the fusion layer (such as a cross-modal attention mechanism for feature fusion processing of vectors) is initialized, and the decoder and generation layer (the generation layer includes an autoregressive generation machine, a potential diffusion model, and a speech generator) are initialized. Specifically, the network structure of each part of the model can be constructed and initialized using a deep learning framework (such as TensorFlow, PyTorch).

[0094] Then, initialize the model parameters (such as weights and biases) to prepare for the training process. Use the parameters of the pre-trained model for initialization (such as loading pre-trained parameters from models such as BERT, ViT, etc.), and use random initialization methods (such as Xavier initialization) for newly added layers or non-pre-trained parts. In the specific implementation, a combination of pre-training and random initialization is used to ensure that the model has good performance and convergence in the early stages of training.

[0095] 3. Forward propagation step:

[0096] First, the batched data is input into the initialized model to start the forward propagation calculation. Specifically, the feature vectors of text data, audio data, and video data are respectively input into the corresponding encoders to obtain the high-dimensional feature vectors (equivalent to the above-mentioned encoding vectors) output by each encoder. Through forward propagation, the input data is calculated through each layer of the network to obtain the output of each layer. Then, the cross-modal attention mechanism is used to fuse the feature vectors of different data types to generate a unified multimodal feature vector, and then the cross-modal attention layer is used to calculate the fusion weights between the text, audio, and video feature vectors, and these feature vectors are weighted and fused to obtain a unified feature representation. Through the cross-modal attention mechanism, it is ensured that the information of different modalities is comprehensively considered after fusion, thereby improving the effectiveness of feature representation.

[0097] Then, the fused feature vector is converted into the target output (such as an audio signal) through the decoder and the generation layer. Specifically, the autoregressive generation mechanism is used to gradually generate the target audio signal, and then the potential diffusion model is used to further optimize the generated audio signal to improve its quality.

[0098] 4. Calculate the loss function

[0099] First, you need to select a suitable loss function to evaluate the difference between the model output and the true label. For audio generation tasks, you can use mean square error (MSE) or spectrum reconstruction error as a loss function, which is used to quantify the error between the model output and the true target and provide guidance for model parameter optimization. Then compare the audio output generated by the model with the true audio label and calculate the loss value.

[0100] 5. Back propagation and parameter update

[0101] First, the back-propagation algorithm is used to calculate the gradient of the loss function relative to the model parameters. Specifically, starting from the output of the loss function, the gradient is calculated backward layer by layer, and the gradient is passed to the parameters of each layer. Through the chain rule, the influence of each layer parameter on the loss function is calculated to obtain the gradient information. Then, an optimization algorithm (such as Adam, SGD) is used to update the model parameters according to the calculated gradient. Specifically, according to the set learning rate, the model parameters are adjusted to optimize in the direction of minimizing the loss, thereby improving the performance and accuracy of the model.

[0102] 6. Model evaluation and validation

[0103] During the training process, the validation set is used regularly to evaluate the model and check its generalization performance. Specifically, the validation set data is input into the model to calculate its loss value and evaluation indicators (such as accuracy, precision, recall, etc.). The performance of the model on unseen data is evaluated through the validation set to ensure that it has good generalization ability. Then, based on the evaluation results, the model architecture, parameters or training strategy are adjusted to further optimize the model performance. That is, based on the evaluation results of the validation set, the learning rate is adjusted, the number of model layers is increased or decreased, the loss function is modified, etc. Through iterative optimization, the model is continuously adjusted and improved to improve its final generation effect.

[0104] 7. Model storage and deployment

[0105] Save the trained model parameters and architecture to the file system for subsequent use and deployment. Specifically, you can use the save function provided by the deep learning framework to export the model parameters and architecture as a file. Save the trained model to ensure that it can be effectively loaded and used in future reasoning and generation processes. Then deploy the trained model to the production environment to provide real-time audio generation services. You can also deploy the model to a server or cloud platform to build an inference service interface for users. Through deployment, the model can be called in actual applications to achieve end-to-end generation from multimodal input to audio output.

[0106] Through the above steps, the training process of the multimodal language model can be completed, from data loading, model initialization, forward propagation, loss calculation, back propagation, parameter update to the final model evaluation, storage and deployment, to achieve the automatic generation of high-quality podcast audio content.

[0107] The following embodiments are used to describe the method of adding personalized audio style.

[0108] Specifically, based on the preset input features, the audio signal is adjusted to obtain the final audio signal; wherein the input features include at least one of the following: emotional characteristics, style characteristics, opening characteristics, ending characteristics and sound quality characteristics. The input features need to be processed together with the input data to obtain an audio signal matching the input features.

[0109] In the specific implementation, when fine-tuning based on the pre-trained multimodal large language model to achieve high-quality audio generation, targeted processing is required in the preparation and preprocessing of training data. The following optimizations are made around five key aspects: naturalness and emotional expression of speech, clarity and sound quality of speech, speech style and personalization, content coherence and logical structure, and distinctive opening and ending.

[0110] 1. Voice naturalness and emotional expression

[0111] Collect audio materials containing different emotional expressions, such as happiness, sadness, anger, etc., to ensure that these materials contain common emotional expressions in podcast content. Then perform emotion annotation on the training data, marking the emotion type and intensity of each audio or text content. Then, use the emotion annotation information as an additional input feature and input it into the model for training. Emotion embedding can be performed using technologies including but not limited to emotion recognition.

[0112] During the training process, the emotion transfer mechanism is added so that the model can generate audio content with corresponding emotions based on the input features. For example, if the input emotion is marked as "happy", the audio generated by the model should show a happy tone and emotion.

[0113] 2. Voice clarity and sound quality

[0114] Collect high-quality audio material and ensure high standards for recording equipment and environment. For example, clear audio recorded in a studio environment. Perform noise reduction on the audio material to remove background noise and interference to ensure clear speech. Use spectrum enhancement technology to enhance the spectral details of the audio signal. For example, you can use spectrum smoothing and enhancement algorithms to make the audio signal clearer. Then, apply dynamic range compression technology to balance the loudness of the audio signal to ensure the clarity and consistency of the speech.

[0115] 3. Voice style and personalization

[0116] Collect audio materials from different announcers, hosts or characters, covering various voice styles and personalities, and then perform style annotation on the audio data, marking the voice style and personality characteristics of each audio segment. Use the style annotation information as input features and input it into the model for training. Use style transfer technology to enable the model to generate audio content with the corresponding style based on the input features.

[0117] In the model generation process, a personalized tuning mechanism is added to make the generated audio content have a specific voice style and personality. For example, by adjusting the model parameters, the generated audio can simulate the voice style of a specific host.

[0118] 4. Content coherence and logical structure

[0119] Collect text data with good coherence and logical structure to ensure the logical coherence and consistency of the content. Specifically, collect conversational audio materials to ensure the fluency and coherence of the conversation, then segment and annotate the text data with logical structure to ensure the logical coherence of the input content, and use semantic alignment technology to semantically align the input text and generated audio to ensure the logical coherence and consistency of the generated content.

[0120] 5. Featured podcast opening and closing

[0121] Collect and organize common podcast opening and ending templates to ensure that the generated content has podcast characteristics. Standardize the opening and ending to form template data in a fixed format. In the specific implementation, embed the opening and ending templates into the input features of the model so that the podcast content generated by the model has a standardized opening and ending. During the generation process, the model automatically generates the opening and ending based on the input features, making the podcast content more professional and complete.

[0122] For example, the emotion type and intensity are annotated for each audio or text, such as happy (high), sad (medium), angry (low), and training data with emotion annotations is generated. The emotion annotation information is embedded into the model as an additional input feature, and the audio generated by the model can express the corresponding emotion. The voice style is annotated for each audio, such as formal, informal, humorous, etc., and training data with style annotations is generated. The style annotation information is embedded into the model as an input feature, and the audio generated by the model has a specific voice style and personality.

[0123] The above method, through emotion and style embedding, makes the generated audio personalized and emotional, meeting the needs of different users. Traditional methods are difficult to achieve personalization and emotionalization of audio content. This solution achieves personalized and emotionally rich audio generation through model tuning and data annotation.

[0124] This solution has taken a variety of optimization measures in the data preprocessing and annotation stages, including sentiment annotation, style annotation, semantic alignment, etc., to ensure the high quality and consistency of the input data, laying a solid foundation for subsequent generation. This solution not only realizes automated podcast generation, but also uses personalized tuning technology to make the generated audio content personalized and emotionally expressive. This technical path that combines automation and personalization is innovative and meets users' needs for high-quality, personalized content.

[0125] By utilizing pre-trained multimodal large models, this solution realizes automated podcast generation, significantly reducing manual involvement and time costs, and users can generate high-quality podcast content without complex manual editing. Traditional methods rely on manual recording and editing, which are inefficient and costly, while this solution improves production efficiency through automation and meets the needs of large-scale content generation.

[0126] Corresponding to the above method embodiment, the embodiment of the present invention also provides a data conversion device, such as Figure 2 As shown, the device comprises:

[0127] The data acquisition module 30 is used to acquire input data, encode the input data, and obtain an encoding vector; wherein the input data includes at least one of the following data: text data, audio data, and video data.

[0128] The vector mapping module 31 is used to map the encoding vector to a vector space of a preset dimension to obtain a first vector corresponding to the encoding vector.

[0129] The vector decoding module 32 is used to decode the first vector to obtain decoding information; wherein the decoding information is used to indicate semantic information corresponding to the input data.

[0130] The speech generation module 33 is used to convert the decoded information into a continuous audio signal through a speech generator; wherein the audio signal matches the semantic information corresponding to the input data.

[0131] The above-mentioned data conversion device realizes end-to-end generation of audio signals from input data of various data types without the need for intermediate links and additional processes, thereby simplifying the operational flow of speech generation. At the same time, this method improves the production efficiency of audio signals while ensuring the quality of audio conversion.

[0132] Specifically, the above-mentioned data acquisition module 30 is used to: standardize the input data according to preset rules to obtain standardized input data; wherein the preset rules include standardized processing rules corresponding to multiple preset data types, and the multiple preset data types include text type, audio type and video type; divide the input data after standardization to obtain multiple sub-data after division; for multiple sub-data, encode the sub-data to obtain the encoding vector corresponding to each sub-data.

[0133] In an optional embodiment, the above-mentioned vector mapping module 31 is used to: map the coding vector to a preset vector space to obtain a mapped coding vector; and map the mapped coding vector to a vector space of preset dimensions by multiplying the mapped coding vector with a target matrix of preset dimensions to obtain a first vector.

[0134] Furthermore, the above-mentioned vector mapping module 31 is also used to: generate position code and modal mark corresponding to the encoding vector; wherein the position code is used to indicate the order relationship of each element in the encoding vector, and the modal mark is used to indicate the data type of the input data corresponding to the encoding vector; embed the position code and modal mark into the encoding vector to obtain the mapped encoding vector.

[0135] In a specific implementation, the input data includes data of at least two data types; based on this, the device also includes a feature fusion module, which is used to: after mapping the coding vector to a vector space of a preset dimension to obtain a first vector corresponding to the coding vector, perform feature fusion processing on the first vector corresponding to the input data of at least two data types to obtain a first vector after feature fusion, so as to subsequently perform decoding processing on the first vector after feature fusion to obtain decoding information.

[0136] In an optional embodiment, the feature fusion module is also used to: determine the fusion weight corresponding to the first vector of each data type of at least two data types; multiply the first vector of each data type by the corresponding fusion weight to obtain the multiplication result corresponding to each data type, and add the multiplication results corresponding to each data type to obtain the first vector after feature fusion.

[0137] In an optional embodiment, the above-mentioned vector decoding module 32 is used to: input the first vector into a pre-trained decoder, obtain the association relationship between the elements in the first vector through the decoder, and obtain the decoding information corresponding to the first vector based on the association relationship and the position code corresponding to the first vector.

[0138] Furthermore, the above-mentioned vector decoding module 32 is also used to: divide the first vector into multiple sub-vectors according to the position code corresponding to the first vector; for the multiple sub-vectors, through the decoder, generate decoding information corresponding to the current sub-vector based on the relationship between the elements in the current sub-vector, and generate decoding information corresponding to the next sub-vector of the current sub-vector based on the association relationship between the decoding information corresponding to the current sub-vector and the elements in the next sub-vector of the current sub-vector; combine the decoding information corresponding to multiple sub-vectors to obtain the decoding information corresponding to the first vector.

[0139] Furthermore, the above-mentioned device also includes an information optimization module, which is used to: before converting the decoded information into a continuous audio signal through a speech generator, denoise and refine the decoded information to obtain high-fidelity decoded information, so that the speech generator can perform speech generation processing on the high-fidelity decoded information.

[0140] In a specific implementation, the above-mentioned device also includes an audio adjustment module, which is used to: adjust the audio signal based on preset input features to obtain a final audio signal; wherein the input features include at least one of the following: emotional characteristics, style characteristics, opening characteristics, ending characteristics and sound quality characteristics.

[0141] The data conversion device provided in the embodiment of the present disclosure has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0142] The embodiment of the present invention further provides an electronic device, such as Figure 3 As shown, the electronic device includes a processor and a memory, the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the above data conversion method.

[0143] Specifically, the above-mentioned data conversion method includes: obtaining input data, encoding the input data, and obtaining a coding vector; wherein the input data includes at least one of the following data: text data, audio data, and video data; mapping the coding vector to a vector space of a preset dimension to obtain a first vector corresponding to the coding vector; decoding the first vector to obtain decoding information; wherein the decoding information is used to indicate semantic information corresponding to the input data; through a speech generator, converting the decoding information into a continuous audio signal; wherein the audio signal matches the semantic information corresponding to the input data.

[0144] The above data conversion method realizes the end-to-end generation of audio signals from input data of various data types without the need for intermediate links and additional processes, thus simplifying the operational flow of speech generation. At the same time, this method improves the production efficiency of audio signals while ensuring the quality of audio conversion.

[0145] In an optional embodiment, the above-mentioned step of encoding the input data to obtain the encoding vector includes: standardizing the input data according to preset rules to obtain standardized input data; wherein the preset rules include standardized processing rules corresponding to multiple preset data types, and the multiple preset data types include text type, audio type and video type; dividing the standardized input data to obtain multiple sub-data after division; for multiple sub-data, encoding the sub-data to obtain the encoding vector corresponding to each sub-data.

[0146] In an optional embodiment, the above-mentioned step of mapping the coding vector to a vector space of a preset dimension to obtain a first vector corresponding to the coding vector includes: mapping the coding vector to a preset vector space to obtain a mapped coding vector; and mapping the mapped coding vector to a vector space of a preset dimension by multiplying the mapped coding vector with a target matrix of a preset dimension to obtain a first vector.

[0147] In an optional embodiment, the above-mentioned step of mapping the coding vector to a preset vector space to obtain the mapped coding vector includes: generating a position code and a modal tag corresponding to the coding vector; wherein the position code is used to indicate the order relationship of each element in the coding vector, and the modal tag is used to indicate the data type of the input data corresponding to the coding vector; and embedding the position code and the modal tag into the coding vector to obtain the mapped coding vector.

[0148] In an optional embodiment, the input data includes data of at least two data types; after the step of mapping the encoding vector to a vector space of a preset dimension to obtain a first vector corresponding to the encoding vector, the method further includes: performing feature fusion processing on the first vector corresponding to the input data of at least two data types to obtain a first vector after feature fusion, so as to subsequently perform decoding processing on the first vector after feature fusion to obtain decoding information.

[0149] In an optional embodiment, the above-mentioned step of performing feature fusion processing on the first vector corresponding to the input data of at least two data types to obtain the first vector after feature fusion includes: determining the fusion weight corresponding to the first vector of each data type of at least two data types; multiplying the first vector of each data type with the corresponding fusion weight to obtain the multiplication result corresponding to each data type, and adding the multiplication results corresponding to each data type to obtain the first vector after feature fusion.

[0150] In an optional embodiment, the above-mentioned step of decoding the first vector to obtain decoding information includes: inputting the first vector into a pre-trained decoder, obtaining the association relationship between elements in the first vector through the decoder, and obtaining decoding information corresponding to the first vector based on the association relationship and the position encoding corresponding to the first vector.

[0151] In an optional embodiment, the step of obtaining decoding information corresponding to the first vector based on the association relationship and the position coding corresponding to the first vector includes: dividing the first vector into multiple sub-vectors according to the position coding corresponding to the first vector; for the multiple sub-vectors, generating decoding information corresponding to the current sub-vector based on the relationship between elements in the current sub-vector through a decoder, and generating decoding information corresponding to the next sub-vector of the current sub-vector based on the association relationship between the decoding information corresponding to the current sub-vector and the elements in the next sub-vector of the current sub-vector; combining the decoding information corresponding to multiple sub-vectors to obtain decoding information corresponding to the first vector.

[0152] In an optional embodiment, before the step of converting the decoded information into a continuous audio signal through a speech generator, the method further includes: denoising and refining the decoded information to obtain high-fidelity decoded information, so that the speech generator performs speech generation processing on the high-fidelity decoded information.

[0153] In an optional embodiment, the above method also includes: adjusting the audio signal based on preset input features to obtain a final audio signal; wherein the input features include at least one of the following: emotional characteristics, style characteristics, opening characteristics, ending characteristics and sound quality characteristics.

[0154] Further, Figure 3 The electronic device shown further includes a bus 102 and a communication interface 103 , and the processor 101 , the communication interface 103 and the memory 100 are connected via the bus 102 .

[0155] The memory 100 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 103 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used. The bus 102 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0156] The processor 101 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 101 or the instruction in the form of software. The above processor 101 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present invention can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 100, and the processor 101 reads the information in the memory 100 and completes the steps of the method of the above embodiment in combination with its hardware.

[0157] An embodiment of the present invention also provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the above-mentioned data conversion method. The specific implementation can be found in the method embodiment, which will not be repeated here.

[0158] Specifically, the above-mentioned data conversion method includes: obtaining input data, encoding the input data, and obtaining a coding vector; wherein the input data includes at least one of the following data: text data, audio data, and video data; mapping the coding vector to a vector space of a preset dimension to obtain a first vector corresponding to the coding vector; decoding the first vector to obtain decoding information; wherein the decoding information is used to indicate semantic information corresponding to the input data; through a speech generator, converting the decoding information into a continuous audio signal; wherein the audio signal matches the semantic information corresponding to the input data.

[0159] The above data conversion method realizes the end-to-end generation of audio signals from input data of various data types without the need for intermediate links and additional processes, thus simplifying the operational flow of speech generation. At the same time, this method improves the production efficiency of audio signals while ensuring the quality of audio conversion.

[0160] In an optional embodiment, the above-mentioned step of encoding the input data to obtain the encoding vector includes: standardizing the input data according to preset rules to obtain standardized input data; wherein the preset rules include standardized processing rules corresponding to multiple preset data types, and the multiple preset data types include text type, audio type and video type; dividing the standardized input data to obtain multiple sub-data after division; for multiple sub-data, encoding the sub-data to obtain the encoding vector corresponding to each sub-data.

[0161] In an optional embodiment, the above-mentioned step of mapping the coding vector to a vector space of a preset dimension to obtain a first vector corresponding to the coding vector includes: mapping the coding vector to a preset vector space to obtain a mapped coding vector; and mapping the mapped coding vector to a vector space of a preset dimension by multiplying the mapped coding vector with a target matrix of a preset dimension to obtain a first vector.

[0162] In an optional embodiment, the above-mentioned step of mapping the coding vector to a preset vector space to obtain the mapped coding vector includes: generating a position code and a modal tag corresponding to the coding vector; wherein the position code is used to indicate the order relationship of each element in the coding vector, and the modal tag is used to indicate the data type of the input data corresponding to the coding vector; and embedding the position code and the modal tag into the coding vector to obtain the mapped coding vector.

[0163] In an optional embodiment, the input data includes data of at least two data types; after the step of mapping the encoding vector to a vector space of a preset dimension to obtain a first vector corresponding to the encoding vector, the method further includes: performing feature fusion processing on the first vector corresponding to the input data of at least two data types to obtain a first vector after feature fusion, so as to subsequently perform decoding processing on the first vector after feature fusion to obtain decoding information.

[0164] In an optional embodiment, the above-mentioned step of performing feature fusion processing on the first vector corresponding to the input data of at least two data types to obtain the first vector after feature fusion includes: determining the fusion weight corresponding to the first vector of each data type of at least two data types; multiplying the first vector of each data type with the corresponding fusion weight to obtain the multiplication result corresponding to each data type, and adding the multiplication results corresponding to each data type to obtain the first vector after feature fusion.

[0165] In an optional embodiment, the above-mentioned step of decoding the first vector to obtain decoding information includes: inputting the first vector into a pre-trained decoder, obtaining the association relationship between elements in the first vector through the decoder, and obtaining decoding information corresponding to the first vector based on the association relationship and the position encoding corresponding to the first vector.

[0166] In an optional embodiment, the step of obtaining decoding information corresponding to the first vector based on the association relationship and the position coding corresponding to the first vector includes: dividing the first vector into multiple sub-vectors according to the position coding corresponding to the first vector; for the multiple sub-vectors, generating decoding information corresponding to the current sub-vector based on the relationship between elements in the current sub-vector through a decoder, and generating decoding information corresponding to the next sub-vector of the current sub-vector based on the association relationship between the decoding information corresponding to the current sub-vector and the elements in the next sub-vector of the current sub-vector; combining the decoding information corresponding to multiple sub-vectors to obtain decoding information corresponding to the first vector.

[0167] In an optional embodiment, before the step of converting the decoded information into a continuous audio signal through a speech generator, the method further includes: denoising and refining the decoded information to obtain high-fidelity decoded information, so that the speech generator performs speech generation processing on the high-fidelity decoded information.

[0168] In an optional embodiment, the above method also includes: adjusting the audio signal based on preset input features to obtain a final audio signal; wherein the input features include at least one of the following: emotional characteristics, style characteristics, opening characteristics, ending characteristics and sound quality characteristics.

[0169] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a terminal device, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0170] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0171] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A data conversion method, characterized in that: The method comprises: Acquire input data, and perform encoding processing on the input data to obtain an encoding vector; wherein the input data includes at least one of the following data: text data, audio data, and video data; Mapping the encoding vector to a vector space of a preset dimension to obtain a first vector corresponding to the encoding vector; Decoding the first vector to obtain decoding information; wherein the decoding information is used to indicate semantic information corresponding to the input data; The decoded information is converted into a continuous audio signal through a speech generator; wherein the audio signal matches the semantic information corresponding to the input data.

2. The method according to claim 1, characterized in that The step of encoding the input data to obtain the encoding vector comprises: According to preset rules, the input data is standardized to obtain standardized input data; wherein the preset rules include standardized processing rules corresponding to multiple preset data types, and the multiple preset data types include text type, audio type and video type; Performing division processing on the input data after the standardization processing to obtain a plurality of divided sub-data; For the multiple sub-data, the sub-data are encoded to obtain an encoding vector corresponding to each of the sub-data.

3. The method according to claim 1, characterized in that The step of mapping the encoding vector to a vector space of a preset dimension to obtain a first vector corresponding to the encoding vector includes: Mapping the encoding vector to a preset vector space to obtain a mapped encoding vector; The mapped encoding vector is mapped to the vector space of the preset dimension by multiplying the mapped encoding vector with a target matrix of the preset dimension to obtain the first vector.

4. The method according to claim 3, characterized in that: The step of mapping the encoding vector to a preset vector space to obtain the mapped encoding vector comprises: Generate a position code and a modal tag corresponding to the encoding vector; wherein the position code is used to indicate the order relationship of each element in the encoding vector, and the modal tag is used to indicate the data type of the input data corresponding to the encoding vector; The position code and the modality tag are embedded into the code vector to obtain the mapped code vector.

5. The method according to claim 1, characterized in that The input data includes data of at least two data types; After the step of mapping the encoding vector to a vector space of a preset dimension to obtain a first vector corresponding to the encoding vector, the method further includes: A feature fusion process is performed on a first vector corresponding to the input data of the at least two data types to obtain a feature-fused first vector, so that a decoding process is subsequently performed on the feature-fused first vector to obtain decoding information.

6. The method according to claim 5, characterized in that The step of performing feature fusion processing on the first vector corresponding to the input data of the at least two data types to obtain the first vector after feature fusion includes: Determine a fusion weight corresponding to a first vector of each data type of the at least two data types; The first vector of each data type is multiplied by the corresponding fusion weight to obtain the multiplication result corresponding to each data type, and the multiplication results corresponding to each data type are added to obtain the first vector after feature fusion.

7. The method according to claim 1, characterized in that The step of decoding the first vector to obtain decoding information includes: The first vector is input into a pre-trained decoder, the association relationship between elements in the first vector is obtained through the decoder, and decoding information corresponding to the first vector is obtained based on the association relationship and the position code corresponding to the first vector.

8. The method according to claim 7, characterized in that The step of obtaining decoding information corresponding to the first vector based on the association relationship and the position code corresponding to the first vector includes: Dividing the first vector into a plurality of sub-vectors according to a position code corresponding to the first vector; For the multiple sub-vectors, generating, by the decoder, decoding information corresponding to the current sub-vector based on the relationship between elements in the current sub-vector, and generating decoding information corresponding to the next sub-vector of the current sub-vector based on the association relationship between the decoding information corresponding to the current sub-vector and the elements in the next sub-vector of the current sub-vector; The decoded information corresponding to the multiple sub-vectors is combined to obtain the decoded information corresponding to the first vector.

9. The method according to claim 1, characterized in that: Before the step of converting the decoded information into a continuous audio signal by a speech generator, the method further comprises: The decoded information is subjected to denoising and refinement processing to obtain high-fidelity decoded information, so that the speech generator performs speech generation processing on the high-fidelity decoded information.

10. The method according to claim 1, characterized in that The method further comprises: Based on preset input features, the audio signal is adjusted to obtain a final audio signal; wherein the input features include at least one of the following: emotional characteristics, style characteristics, opening characteristics, ending characteristics and sound quality characteristics.

11. A data conversion device, characterized in that: The device comprises: A data acquisition module, used to acquire input data, encode the input data, and obtain an encoding vector; wherein the input data includes at least one of the following data: text data, audio data, and video data; A vector mapping module, used to map the encoding vector to a vector space of a preset dimension to obtain a first vector corresponding to the encoding vector; A vector decoding module, used for decoding the first vector to obtain decoding information; wherein the decoding information is used to indicate semantic information corresponding to the input data; The speech generation module is used to convert the decoded information into a continuous audio signal through a speech generator; wherein the audio signal matches the semantic information corresponding to the input data.

12. An electronic device, characterized in that: The electronic device includes a processor and a memory, the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the data conversion method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the data conversion method according to any one of claims 1 to 10.

Citation Information

Patent Citations

  • Matrix combination for matrix-weighted intra prediction in video coding

    CA3150554A1

  • A multi-input and multi-output recoding processing method of multi-subchannel

    CN101217304A

  • Voice conversion method and device, equipment and storage medium

    CN113470664A

  • Voice recognition method and system based on audio and video dual modes

    CN114974215A

  • Speech recognition method and device, medium and computing equipment

    CN115064153A

Cited By

  • Audio and video joint coding and decoding method and system based on generative artificial intelligence

    CN119583873A

  • Voice generation method based on multi-modal input and related equipment

    CN121053996A

  • Speech generation method based on multi-modal input and related device

    CN121053996B

  • Controllable text-to-voice conversion method based on decoupling multi-mode prompt and chained guidance

    CN121281490A