Method and apparatus for generating audio data, storage medium, and electronic device

Abstract

The embodiment of the present disclosure relates to a kind of audio data generation method and device, storage medium, electronic equipment, involve big data processing technical field, the method includes: based on the outline extraction model of pre-set, extract the original text outline included in original text content, and according to the original text outline, the original text content is segmented and handled, to obtain original text paragraph;The original text paragraph and original text outline are input into the content generation big model of pre-set, to obtain the paragraph interpretation information corresponding to the original text paragraph;Based on the audio conversion model of pre-set, the paragraph interpretation information is audio converted, to obtain paragraph audio data, and the paragraph audio data is integrated, to obtain the audio data corresponding to the original text content.The present disclosure improves the generation efficiency of audio data.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the technical field of big data processing, and more particularly, to a method for generating audio data, an apparatus for generating audio data, a computer readable storage medium, and an electronic device. BACKGROUND

[0002] This section is intended to provide background information to facilitate a better understanding of embodiments of the disclosure recited in the claims. The description herein does not constitute admission that the information provided herein is prior art to the present disclosure.

[0003] At present, the speed reading podcast programs made by the industry are made and uploaded by artificial. However, this way has the problems of low efficiency and inability to quickly follow hot spots. SUMMARY

[0004] However, in the related technical solutions, on the one hand, the corresponding audio data generated by the artificial production method has the problem of low generation efficiency; on the other hand, the artificial production method cannot quickly follow hot information, thereby making the access volume of the obtained audio data low.

[0005] Therefore, there is a great need for an improved method for generating audio data to extract an original text outline included in original text content based on a preset outline extraction model, and segment the original text content according to the original text outline to obtain original text paragraphs; and input the original text paragraphs and the original text outline into a preset content generation large model to obtain paragraph interpretation information corresponding to the original text paragraphs; based on a preset audio conversion model, the paragraph interpretation information is audio converted to obtain paragraph audio data, and the paragraph audio data is integrated to obtain audio data corresponding to the original text content, so that the obtained audio data can follow hot information on the basis of improving the generation efficiency of the audio data, thereby achieving the purpose of improving the access volume of the audio data.

[0006] In this context, embodiments of the present disclosure aim to provide a method for generating audio data, an apparatus for generating audio data, a computer readable storage medium, and an electronic device.

[0007] According to one aspect of the present disclosure, a method for generating audio data is provided, comprising:

[0008] extracting an original text outline included in original text content based on a preset outline extraction model, and segmenting the original text content according to the original text outline to obtain original text paragraphs;

[0009] input the original text paragraph and the original text outline into a preset content generation large model to obtain paragraph interpretation information corresponding to the original text paragraph;

[0010] perform audio conversion on the paragraph interpretation information based on a preset audio conversion model to obtain paragraph audio data, and integrate the paragraph audio data to obtain audio data corresponding to the original text content.

[0011] In an exemplary embodiment of the present disclosure, the preset outline extraction model includes a paragraph extraction layer, a paragraph encoding layer, and a topic extraction layer.

[0012] The extracting of the original text outline included in the original text content based on the preset outline extraction model includes:

[0013] segmenting the original text content based on the paragraph extraction layer to obtain a plurality of content segmentation intervals;

[0014] encoding the content segmentation intervals based on the paragraph encoding layer to obtain sub-paragraph identifiers, and extracting sub-interval topic words included in sub-interval text content in the content segmentation intervals based on the topic extraction layer;

[0015] generating a sub-paragraph outline according to the content segmentation intervals, the sub-paragraph identifiers, and the sub-interval topic words, and generating the original text outline according to the sub-paragraph outline.

[0016] In an exemplary embodiment of the present disclosure, the topic extraction layer includes a first dynamic weight fusion Bert layer and a first full connection layer.

[0017] The extracting of the sub-interval topic words included in the sub-interval text content in the content segmentation intervals based on the topic extraction layer includes:

[0018] calculating semantic text vectors of the sub-interval text content and an importance of each semantic text vector to the sub-interval text content based on the first dynamic weight fusion Bert layer, and obtaining a first encoding vector according to the semantic text vectors and the importance;

[0019] calculating the first encoding vector based on the first full connection layer to obtain a confidence that each character included in the sub-interval text content belongs to a sub-interval topic word, and determining a starting position and an ending position of the sub-interval topic word according to whether the confidence is greater than a first preset threshold;

[0020] The starting position and the ending position of the sub-interval topic word are marked by using a first preset identifier, and characters corresponding to the positions marked by the first preset identifier are intercepted to obtain a sub-interval topic word included in the sub-interval text content.

[0021] In an example embodiment of the present disclosure, the first dynamic weight fusion Bert layer includes a plurality of Transformer models.

[0022] The semantic text vectors of the sub-interval text content and the importance of each semantic text vector to the sub-interval text content are calculated based on the first dynamic weight fusion Bert layer, and a first encoding vector is obtained according to the semantic text vectors and the importance, including:

[0023] The sub-interval text content is word embedded to obtain word embedding vectors, word embedding matrices and position embedding matrices of the sub-interval text content.

[0024] The sub-interval embedding vectors are generated according to the word embedding vectors, the word embedding matrices and the position embedding matrices, and the sub-interval embedding vectors are input into the first Transformer model to obtain first text semantic vectors.

[0025] The first text semantic vectors are input into other Transformer models to obtain text semantic vectors corresponding to the other Transformer models; wherein the output of a previous Transformer model in the other Transformer models is the input of a next Transformer model corresponding thereto.

[0026] The importance of each Transformer model to the sub-interval text content is calculated, and the first encoding vector is obtained according to each importance and the sub-interval embedding vectors and each text semantic vector.

[0027] In an example embodiment of the present disclosure, the first encoding vector is obtained according to each importance and the sub-interval embedding vectors and each text semantic vector, including:

[0028] The importance degrees are spliced to obtain an importance degree splicing result, and the importance degree splicing result is normalized to obtain a normalized importance degree.

[0029] The normalized importance degree and the sub-interval embedding vectors and each text semantic vector are weighted and summed to obtain the first encoding vector.

[0030] In an example embodiment of the present disclosure, the original text content is segmented according to the original text outline to obtain original text paragraphs, including:

[0031] The original text outline is parsed to obtain a plurality of sub-paragraph outlines, and the starting character position and the ending character position of a content segmentation interval included in each of the sub-paragraph outlines are determined to obtain the interval position of the content segmentation interval in the original text content;

[0032] Based on the interval position, a sub-interval text content corresponding to the content segmentation interval is extracted from the original text content to obtain a sub-text paragraph, and the sub-text paragraph and the sub-paragraph identifier included in the sub-paragraph outline are used to generate the paragraph interpretation information corresponding to the original text paragraph.

[0033] In an example embodiment of the present disclosure, the preset content generation large model includes an embedding mapping layer, an encoding layer, and a hybrid expert model layer;

[0034] The original text paragraph and the original text outline are input into the preset content generation large model to obtain the paragraph interpretation information corresponding to the original text paragraph, including:

[0035] The sub-paragraph outline corresponding to the text paragraph is determined from the original text outline according to the sub-paragraph identifier of the sub-text paragraph included in the original text paragraph, and the sub-paragraph context information is generated according to the sub-paragraph outline and the preset content parameter prompt information corresponding to the sub-text paragraph;

[0036] The sub-text paragraph is subjected to word embedding processing based on the embedding mapping layer to obtain a sub-text feature, and the sub-paragraph context information is subjected to embedding mapping processing to obtain a context marker sequence of the sub-text paragraph;

[0037] The sub-text feature and the context marker sequence of the sub-text paragraph are subjected to encoding processing based on the encoding layer to obtain a context overall representation, and the context marker sequence of the sub-text paragraph and the context overall representation are subjected to prediction based on the hybrid expert model layer to obtain the sub-paragraph generation content corresponding to the sub-text paragraph;

[0038] The sub-paragraph generation content is sorted according to the sub-paragraph identifier, and the sorted sub-paragraph generation content is spliced to obtain the paragraph interpretation information corresponding to the original text paragraph.

[0039] In an example embodiment of the present disclosure, the hybrid expert model layer includes a gating network model and an expert neural network model;

[0040] The context token sequence and the context overall representation of the subtext passage are input into the target neural network model to obtain the sub-passage generation content corresponding to the subtext passage.

[0041] The context token sequence of the subtext passage is input into the gating network model to determine the sub-passage model weight of the expert neural network model, and the target neural network model required to perform the prediction task of the sub-passage generation content is determined from the expert neural network according to the sub-passage model weight.

[0042] The context token sequence of the subtext passage is input into the gating network model to determine the sub-passage model weight of the expert neural network model, and the target neural network model required to perform the prediction task of the sub-passage generation content is determined from the expert neural network according to the sub-passage model weight.

[0043] In an exemplary embodiment of the present disclosure, the gating network model includes a first gating network model and a second gating network model.

[0044] The context token sequence of the subtext passage is input into the gating network model to determine the sub-passage model weight of the expert neural network model, and the target neural network model required to perform the prediction task of the sub-passage generation content is determined from the expert neural network according to the sub-passage model weight.

[0045] The context token sequence of the subtext passage is input into the first gating network model to determine the first sub-passage model weight of the expert neural network model in the content fit degree dimension, and the first target neural network model required to perform the prediction task of the sub-passage generation content in the content fit degree dimension is determined from the expert neural network according to the first sub-passage model weight.

[0046] The context token sequence of the subtext passage is input into the second gating network model to determine the second sub-passage model weight of the expert neural network model in the content fluency dimension, and the second target neural network model required to perform the prediction task of the sub-passage generation content in the content fluency dimension is determined from the expert neural network according to the second sub-passage model weight.

[0047] In an exemplary embodiment of the present disclosure, the preset audio conversion model includes a text preprocessing layer, a speech synthesis layer, and a speech adjustment layer; wherein the passage interpretation information is audio converted based on the preset audio conversion model to obtain passage audio data, including:

[0048] text preprocessing based on the text preprocessing layer on the paragraph interpretation information, to obtain standard text information, and based on the speech synthesis layer on the standard text information, speech synthesis is performed to obtain an original speech file;

[0049] Based on the speech adjustment layer, the waveform of the original speech file is cropped to obtain an audio cropping result, and the speech volume in the audio cropping result is adjusted to obtain the paragraph audio data.

[0050] In an exemplary embodiment of the present disclosure, based on the speech synthesis layer, the standard text information is synthesized to obtain an original speech file, comprising:

[0051] Based on the speech feature extraction model in the speech synthesis layer, the original speech features of the standard text information are extracted, and an original audio spectrum graph is generated according to the original speech features, so as to generate an original speech file based on the original audio spectrum graph.

[0052] In an exemplary embodiment of the present disclosure, the method for generating audio data further comprises:

[0053] According to the sub-interval topic words included in the original text outline, the recommended words are configured for the audio data corresponding to the original text content;

[0054] The audio data corresponding to the original text content is published, and the published audio data is identified based on the recommended words.

[0055] According to an aspect of the present disclosure, an audio data generation device is provided, comprising:

[0056] A text outline extraction module is configured to extract an original text outline included in original text content based on a preset outline extraction model, and segment the original text content based on the original text outline to obtain original text paragraphs;

[0057] A paragraph interpretation information generation module is configured to input the original text paragraphs and the original text outline into a preset content generation large model to obtain paragraph interpretation information corresponding to the original text paragraphs;

[0058] An audio data generation module is configured to perform audio conversion on the paragraph interpretation information based on a preset audio conversion model to obtain paragraph audio data, and integrate the paragraph audio data to obtain audio data corresponding to the original text content.

[0059] In an exemplary embodiment of the present disclosure, the preset outline extraction model comprises a paragraph extraction layer, a paragraph encoding layer and a topic extraction layer;

[0060] The original text outline included in the original text content is extracted based on a preset outline extraction model, including:

[0061] The original text content is segmented based on a paragraph extraction layer to obtain a plurality of content segmentation intervals;

[0062] The content segmentation intervals are encoded based on a paragraph encoding layer to obtain sub-paragraph identifiers, and sub-interval topic words included in sub-interval text content in the content segmentation intervals are extracted based on the topic extraction layer;

[0063] A sub-paragraph outline is generated according to the content segmentation intervals, the sub-paragraph identifiers, and the sub-interval topic words, and the original text outline is generated according to the sub-paragraph outline.

[0064] In an exemplary embodiment of the present disclosure, the topic extraction layer includes a first dynamic weight fusion Bert layer and a first full connection layer;

[0065] The sub-interval topic words included in the sub-interval text content in the content segmentation intervals are extracted based on the topic extraction layer, including:

[0066] The semantic text vectors of the sub-interval text content and the importance of each semantic text vector to the sub-interval text content are calculated based on the first dynamic weight fusion Bert layer, and a first encoding vector is obtained according to the semantic text vectors and the importance;

[0067] The first full connection layer is used to calculate the first encoding vector to obtain the confidence of each character included in the sub-interval text content belonging to a sub-interval topic word, and the starting position and the ending position of the sub-interval topic word are determined according to whether the confidence is greater than a first preset threshold;

[0068] The starting position and the ending position of the sub-interval topic word are marked with a first preset identifier, and the characters corresponding to the positions marked with the first preset identifier are intercepted to obtain the sub-interval topic words included in the sub-interval text content.

[0069] In an exemplary embodiment of the present disclosure, the first dynamic weight fusion Bert layer includes a plurality of Transformer models;

[0070] The semantic text vectors of the sub-interval text content and the importance of each semantic text vector to the sub-interval text content are calculated based on the first dynamic weight fusion Bert layer, and a first encoding vector is obtained according to the semantic text vectors and the importance, including:

[0071] perform word embedding on the sub-interval text content to obtain a word embedding vector, a word embedding matrix and a position embedding matrix of the sub-interval text content;

[0072] generate a sub-interval embedding vector according to the word embedding vector, the word embedding matrix and the position embedding matrix, and input the sub-interval embedding vector into a first Transformer model to obtain a first text semantic vector;

[0073] input the first text semantic vector into other Transformer models to obtain a text semantic vector corresponding to each of the other Transformer models, wherein the output of a previous Transformer model in the other Transformer models is the input of a next Transformer model corresponding to the previous Transformer model;

[0074] calculate an importance degree of each of the Transformer models for the sub-interval text content, and obtain the first encoding vector according to the importance degrees and the sub-interval embedding vector and the text semantic vectors.

[0075] In an exemplary embodiment of the present disclosure, obtaining the first encoding vector according to the importance degrees and the sub-interval embedding vector and the text semantic vectors includes:

[0076] concatenate the importance degrees to obtain an importance degree concatenation result, and perform normalization processing on the importance degree concatenation result to obtain normalized importance degrees;

[0077] perform weighted summation on the normalized importance degrees and the sub-interval embedding vector and the text semantic vectors to obtain the first encoding vector.

[0078] In an exemplary embodiment of the present disclosure, segmenting the original text content according to the original text outline to obtain original text paragraphs includes:

[0079] analyze the original text outline to obtain a plurality of sub-paragraph outlines, and determine an interval position of a content segmentation interval included in each of the sub-paragraph outlines in the original text content according to a start character position and an end character position of the content segmentation interval;

[0080] extract a sub-interval text content corresponding to the content segmentation interval from the original text content based on the interval position to obtain a sub-text paragraph, and generate the original text paragraph according to the sub-text paragraph and a sub-paragraph identifier included in the sub-paragraph outline.

[0081] In an example embodiment of the present disclosure, the preset content generation large model comprises an embedding mapping layer, an encoding layer, and a hybrid expert model layer.

[0082] The original text paragraph and the original text outline are input into the preset content generation large model to obtain paragraph interpretation information corresponding to the original text paragraph, which comprises:

[0083] According to the sub-paragraph identifier of the sub-text paragraph included in the original text paragraph, a sub-paragraph outline corresponding to the text paragraph is determined from the original text outline, and sub-paragraph context information is generated according to the sub-paragraph outline and the preset content parameter prompt information corresponding to the sub-text paragraph.

[0084] The sub-text paragraph is subjected to word embedding processing based on the embedding mapping layer to obtain sub-text features, and the sub-paragraph context information is subjected to embedding mapping processing to obtain a context mark sequence of the sub-text paragraph.

[0085] The sub-text features and the context mark sequence of the sub-text paragraph are subjected to encoding processing based on the encoding layer to obtain a context overall representation, and the context mark sequence of the sub-text paragraph and the context overall representation are predicted based on the hybrid expert model layer to obtain sub-paragraph generation content corresponding to the sub-text paragraph.

[0086] The sub-paragraph generation content is sorted according to the sub-paragraph identifier, and the sorted sub-paragraph generation content is spliced to obtain paragraph interpretation information corresponding to the original text paragraph.

[0087] In an example embodiment of the present disclosure, the hybrid expert model layer comprises a gating network model and an expert neural network model.

[0088] The context mark sequence of the sub-text paragraph and the context overall representation are predicted based on the hybrid expert model layer to obtain sub-paragraph generation content corresponding to the sub-text paragraph, which comprises:

[0089] The sub-paragraph model weight of the expert neural network model is determined according to the context mark sequence of the sub-text paragraph based on the gating network model, and a target neural network model required for performing a prediction task of sub-paragraph generation content is determined from the expert neural network according to the sub-paragraph model weight.

[0090] The context mark sequence of the sub-text paragraph and the context overall representation are input into the target neural network model to obtain sub-paragraph generation content corresponding to the sub-text paragraph.

[0091] In an example embodiment of the present disclosure, the gating network model comprises a first gating network model and a second gating network model;

[0092] The first gating network model comprises a first gating network model and a second gating network model.

[0093] The first gating network model comprises a first gating network model and a second gating network model.

[0094] The first gating network model comprises a first gating network model and a second gating network model.

[0095] In an example embodiment of the present disclosure, the preset audio conversion model comprises a text preprocessing layer, a speech synthesis layer, and a speech adjustment layer; wherein the audio conversion of the paragraph interpretation information based on the preset audio conversion model obtains paragraph audio data, comprising:

[0096] The text preprocessing layer preprocesses the paragraph interpretation information to obtain standard text information, and the speech synthesis layer synthesizes the standard text information to obtain an original speech file.

[0097] The speech adjustment layer trims the waveform of the original speech file to obtain an audio trimming result, and adjusts the speech volume in the audio trimming result to obtain the paragraph audio data.

[0098] In an example embodiment of the present disclosure, the speech synthesis of the standard text information based on the speech synthesis layer obtains an original speech file, comprising:

[0099] The speech feature extraction model in the speech synthesis layer extracts the original speech features of the standard text information, and generates an original audio spectrum graph according to the original speech features, so as to generate an original speech file based on the original audio spectrum graph.

[0100] In an example embodiment of the present disclosure, the audio data generation apparatus further comprises:

[0101] a recommended word configuration module configured to configure recommended words for audio data corresponding to the original text content according to topic words included in the sub-interval in the original text outline;

[0102] an audio data identification module configured to publish the audio data corresponding to the original text content and identify the published audio data based on the recommended words.

[0103] According to an aspect of the present disclosure, there is provided a computer readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the audio data generation method of any of the above.

[0104] According to an aspect of the present disclosure, there is provided an electronic device comprising:

[0105] a processor; and

[0106] a memory configured to store executable instructions of the processor;

[0107] wherein the processor is configured to execute the audio data generation method of any of the above via execution of the executable instructions.

[0108] The audio data generation method and the audio data generation apparatus according to the embodiments of the present disclosure can extract an original text outline included in original text content based on a preset outline extraction model, segment the original text content according to the original text outline to obtain original text paragraphs, input the original text paragraphs and the original text outline into a preset content generation large model to obtain paragraph interpretation information corresponding to the original text paragraphs, and finally perform audio conversion on the paragraph interpretation information based on a preset audio conversion model to obtain paragraph audio data, and integrate the paragraph audio data to obtain audio data corresponding to the original text content, without manually producing the audio data by artificial means, thereby significantly reducing the problem of low audio generation efficiency caused by manually producing the audio data, and reducing the problem that audio data produced by artificial means cannot quickly follow hot information, thereby making the obtained audio data have a low access volume, and providing a better experience for users. BRIEF DESCRIPTION OF DRAWINGS

[0109] The above and other objects, features and advantages of the example embodiments of the present disclosure will be more apparent from the following detailed description taken in conjunction with the accompanying drawings, which show, by way of example, several embodiments of the present disclosure. In the drawings:

[0110] Figure 1 A flowchart of a method for generating audio data is schematically shown according to an example embodiment of the present disclosure;

[0111] Figure 2 A structural example diagram of a schema extraction model is schematically shown according to an example embodiment of the present disclosure;

[0112] Figure 3 A structural example diagram of a topic extraction layer is schematically shown according to an example embodiment of the present disclosure;

[0113] Figure 4 A structural example diagram of a content generation large model is schematically shown according to an example embodiment of the present disclosure;

[0114] Figure 5 A structural example diagram of an audio conversion model is schematically shown according to an example embodiment of the present disclosure;

[0115] Figure 6 A scene example diagram of a specific generation process of sub-paragraph content is schematically shown according to an example embodiment of the present disclosure;

[0116] Figure 7 An example diagram of obtained paragraph interpretation information is schematically shown according to an example embodiment of the present disclosure;

[0117] Figure 8 A scene example diagram of content review of paragraph interpretation information is schematically shown according to an example embodiment of the present disclosure;

[0118] Figure 9 A scene example diagram of a specific generation process of paragraph audio data is schematically shown according to an example embodiment of the present disclosure;

[0119] Figure 10 A scene example diagram of a specific publication process of audio data is schematically shown according to an example embodiment of the present disclosure;

[0120] Figure 11 A scene example diagram of a display interface obtained after publication of audio data is schematically shown according to an example embodiment of the present disclosure;

[0121] Figure 12 A process scene example diagram of generation and publication of audio data is schematically shown according to an example embodiment of the present disclosure;

[0122] Figure 13 A block diagram of a device for generating audio data is schematically shown according to an example embodiment of the present disclosure;

[0123] Figure 14A computer-readable storage medium for storing a generation method of audio data according to an example embodiment of the present disclosure is schematically shown.

[0124] Figure 15 A structural example diagram of an electronic device for implementing a generation method of audio data according to an example embodiment of the present disclosure is schematically shown.

[0125] In the drawings, identical or corresponding reference signs indicate identical or corresponding parts. DETAILED DESCRIPTION

[0126] The principles and spirits of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are given only to enable those skilled in the art to better understand and implement the present disclosure, and in no way limit the scope of the present disclosure. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0127] Those skilled in the art know that the embodiments of the present disclosure can be implemented as a system, a device, an apparatus, a method or a computer program product. Therefore, the present disclosure can be embodied in the form of entire hardware, entire software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0128] According to embodiments of the present disclosure, a generation method of audio data, a generation apparatus of audio data, a computer-readable storage medium and an electronic device are proposed.

[0129] In this document, any number of elements in the drawings are used for illustration and not limitation, and any naming is only for distinction and does not have any limiting meaning.

[0130] The principles and spirits of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are given only to enable those skilled in the art to better understand and implement the present disclosure, and in no way limit the scope of the present disclosure. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art. SUMMARY

[0132] The present applicant finds that at present, most of the speed reading podcast programs made by the industry are realized by artificial production; however, this way not only has low efficiency, but also has the problem of being unable to quickly follow hot spots; in some other schemes, podcast programs can also be made based on artificial intelligence; however, the podcast programs obtained based on this way have problems such as short duration and incoherent content; that is, the accuracy of the obtained audio data is low.

[0133] Based on this, the example embodiments of the present disclosure provide an audio data generation method, which converts the content such as novels, books, and hot news into a speed reading podcast program through a large language model and an audio conversion technology. In the actual application process, first, a detailed interpretation outline can be produced according to the content, including the interval and structure of each content, then the content is segmented according to the outline, and the rewritten interpretation script is output in turn, and finally the corresponding audio data is generated through the speech conversion technology; the audio data generation method recorded in the example embodiments of the present disclosure can support the production of programs with a duration of more than 30 minutes, greatly increase the diversity of podcast content and quickly produce hot content to attract user consumption, thereby achieving the purpose of improving the access of audio data.

[0134] After introducing the basic principles of the present disclosure, the various non-limiting embodiments of the present disclosure will be specifically introduced below.

[0135] Exemplary methods

[0136] The example embodiments of the present disclosure first provide an audio data processing method, which can run on a terminal device, a server, a server cluster, or a cloud server, etc. Of course, those skilled in the art can also run the method of the present disclosure on other platforms according to the needs, which is not specially limited in the present example embodiment. Specifically, referring to Figure 1 The audio data generation method can include the following steps:

[0137] Step S110. Extracting an original text outline included in original text content based on a preset outline extraction model, and segmenting the original text content according to the original text outline to obtain original text paragraphs;

[0138] Step S120. Inputting the original text paragraphs and the original text outline into a preset content generation large model to obtain paragraph interpretation information corresponding to the original text paragraphs;

[0139] Step S130. Audio conversion of the paragraph interpretation information based on a preset audio conversion model to obtain paragraph audio data, and integration of the paragraph audio data to obtain audio data corresponding to the original text content.

[0140] In the method for processing audio data, the original text outline included in the original text content can be extracted based on a preset outline extraction model, and the original text content can be segmented based on the original text outline to obtain original text paragraphs. Then, the original text paragraphs and the original text outline are input into a preset content generation large model to obtain paragraph interpretation information corresponding to the original text paragraphs. Finally, the paragraph interpretation information is converted into audio based on a preset audio conversion model to obtain paragraph audio data, and the paragraph audio data is integrated to obtain audio data corresponding to the original text content. Thus, the audio data can be obtained without manual production, thereby significantly reducing the problem of low audio generation efficiency caused by manual production of audio data, and reducing the problem that audio data produced by manual production cannot quickly follow hot information, thereby reducing the access of the obtained audio data, thereby providing a better experience for users.

[0141] In the following, the method for generating audio data according to the example embodiments of the present disclosure will be explained and described in detail with reference to the accompanying drawings.

[0142] First, the terms related to the example embodiments of the present disclosure will be explained and described.

[0143] LLM: LLM generally refers to "Large Language Model", which is a kind of artificial intelligence model trained using deep learning technology, which can understand and generate natural language text; in the process of practical application, large language models can usually be trained based on a large amount of text data to learn the patterns and structures of language, so as to perform various language-related tasks such as text generation, translation, text summarization, sentiment analysis, etc.

[0144] TTS: TTS (Text-to-Speech) technology, i.e. text-to-speech conversion technology, is a technology that converts text information into spoken language output.

[0145] Token: In the field of natural language processing, Token can be understood as the smallest unit in text. In English, a Token can be a word or a punctuation mark; in Chinese, a Token is usually a character or a word; further, in the process of specific language tasks, a large language model can split the input text into Tokens, so that the model can process and understand them; at the same time, each Token is associated with a unique numerical identifier, and the model distinguishes different Tokens through these identifier information.

[0146] Secondly, the preset outline extraction model related to the example embodiments of the present disclosure will be explained and described. Specifically, refer to Figure 2As shown, the preset outline extraction model includes a first input layer 210, a paragraph extraction layer 220, a paragraph encoding layer 230, a topic extraction layer 240, and a first output layer 250; further, referring to Figure 3 As shown, the topic extraction layer described herein includes a first dynamic weight fusion Bert layer and a first full connection layer; the first dynamic weight fusion Bert layer includes a plurality of Transformer models. The specific functions of the model layer will be described one by one in the following, and will not be further described here.

[0147] In the following, the encoding principle of the dynamic weight fusion Bert layer will be explained and described. Specifically, the Bert described in the example embodiments of the present disclosure can be used as an encoder to extract semantic features of text. In detail: at the data input end, first encode the input sentence X into a word embedding matrix W t and a position embedding matrix W p ; wherein X = (x1, x2,..., x n ). Then, add the word embedding matrix and the position embedding matrix two vectors as the total input embedding representation h0, and then input the vector representation h0 through an N-layer Transformer network to obtain the text semantic representation vector h l , which can be specifically shown as follows:

[0148] h0 = XW t +W p ; Equation (1)

[0149] h l = Transformer(h l-1 ), l ∈ [1, N]; Equation (2)

[0150] Where h l is a hidden layer vector, that is, the output of the lth layer Transformer network.

[0151] At the same time, in order to effectively utilize the information of each layer in Bert, first, the importance μ of each Transformer can be learned by a dense unit, and then the results of all Transformer layers are weighted and stacked to obtain the final semantic representation, which can be specifically shown as follows:

[0152] μ l = σ(Dense unit＝1 (h l )); Equation (3)

[0153] Among them, σ is the ReLu activation function; secondly, the weights of each layer are spliced ​​and normalized with the softmax function to obtain a 1*L weight vector μ L , which can be specifically expressed as the following formula (4):

[0154] μ L =softmax(concatenate([μ1,μ2,...,μ l ])); Formula (4)

[0155] Then, all the Transformer results in BERT are concatenated to obtain a fused representation, which can be specifically expressed as follows:

[0156] h L =concatenate([h1,h2,...,h l ]); Formula (5)

[0157] Finally, the weighted fusion obtained by formula (4) can be used to obtain the final fusion semantic representation h DWT , which is the final encoding vector, can be specifically expressed as follows:

[0158] h DWF =matmul([μ L ,h L ]); Formula (6)

[0159] It should be further explained here that since the weight μ is automatically updated according to the importance of each layer during network training, the Bert layer can be called adaptive weight fusion BERT.

[0160] Furthermore, the content generation model involved in the exemplary embodiment of the present disclosure is explained and illustrated. Figure 4 As shown, the content generation model described herein may include a second input layer 410, an embedding mapping layer 420, a coding layer 430, a hybrid expert model layer 440, and a second output layer 450; at the same time, the hybrid expert model layer described herein may include a gated network model and an expert neural network model; the gated network model described herein may include a first gated network model and a second gated network model. The specific functions of the model layer will be described in detail later, and will not be further elaborated here. The embedding mapping layer described herein may include an Embbeding embedding mapping layer and / or a Bert embedding mapping layer; the coding layer described herein may be a coding layer composed of multiple Transformers, and so on.

[0161] Further, the audio conversion model involved in the example embodiments of the present disclosure is explained and described. Specifically, referring to FIG. 5, the audio conversion model described herein can include a third input layer 510, a text preprocessing layer 520, a speech synthesis layer 530, a speech adjustment layer 540, and a third output layer 550. The specific functions of the model layers will be described in detail later, and will not be further described here. Figure 5

[0162] In the following, the processing method of the audio data shown in FIG. 1 will be further explained and described. Specifically: Figures 2-5 The processing method of the audio data shown in FIG. 1 will be further explained and described. Specifically: Figure 1

[0163] In step S110, the original text outline included in the original text content is extracted based on a preset outline extraction model, and the original text content is segmented based on the original text outline to obtain original text paragraphs.

[0164] In the example embodiments, first, the original text outline is extracted. Specifically, the original text content can be segmented based on a paragraph extraction layer to obtain a plurality of content segmentation intervals, the content segmentation intervals are encoded based on a paragraph encoding layer to obtain sub-paragraph identifiers, and sub-interval topic words included in the sub-interval text content in the content segmentation intervals are extracted based on a topic extraction layer. Then, a sub-paragraph outline is generated according to the content segmentation intervals, the sub-paragraph identifiers, and the sub-interval topic words, and the original text outline is generated according to the sub-paragraph outline. Specifically, the content segmentation processing described herein can be implemented based on a preset segmentation rule. For example, a content segmentation interval can be set to include a specific number of characters (such as 4000 characters, or other number of characters, which is not specially limited in the example). Then, in the actual segmentation process, the original text content can be traversed based on the paragraph extraction layer, and then every 4000 characters is divided into a content segmentation interval, and then encoded based on the paragraph encoding layer to obtain the sub-paragraph identifier.

[0165] ​​In an example embodiment, based on the topic extraction layer extracting the sub-interval topic words included in the sub-interval text content in the content segmentation interval, the extraction can be achieved in the following way: first, based on the first dynamic weight fusion Bert layer, the semantic text vector of the sub-interval text content and the importance of each semantic text vector to the sub-interval text content are calculated, and a first encoding vector is obtained according to the semantic text vector and the importance; second, based on the first full connection layer, the first encoding vector is calculated to obtain the confidence that each character included in the sub-interval text content belongs to the sub-interval topic word, and the starting position and the ending position of the sub-interval topic word are determined according to whether the confidence is greater than a first preset threshold; then, the starting position and the ending position of the sub-interval topic word are marked with a first preset identifier, and the characters corresponding to the positions marked with the first preset identifier are intercepted to obtain the sub-interval topic words included in the sub-interval text content.

[0166] In an example embodiment, based on the first dynamic weight fusion Bert layer, the semantic text vector of the sub-interval text content and the importance of each semantic text vector to the sub-interval text content are calculated, and a first encoding vector is obtained according to the semantic text vector and the importance, which can be achieved in the following way: word embedding is performed on the sub-interval text content to obtain word embedding vectors, word embedding matrices and position embedding matrices of the sub-interval text content; the sub-interval embedding vectors are generated according to the word embedding vectors, word embedding matrices and position embedding matrices, and the sub-interval embedding vectors are input into the first Transformer model to obtain the first text semantic vector; the first text semantic vector is input into other Transformer models to obtain text semantic vectors corresponding to the other Transformer models; wherein, in the other Transformer models, the output of the previous Transformer model is the input of the corresponding next Transformer model; the importance of each Transformer model to the sub-interval text content is calculated, and the first encoding vector is obtained according to each importance and the sub-interval embedding vectors, the text semantic vectors. Further, the specific determination process of the first encoding vector described herein can be achieved in the following way: the importance degrees are spliced to obtain an importance degree splicing result, and the importance degree splicing result is normalized to obtain a normalized importance degree; the normalized importance degree and the sub-interval embedding vectors, the text semantic vectors are weighted and summed to obtain the first encoding vector.

[0167] The following will further explain and illustrate the specific generation process of the original text outline. Specifically, in the actual application process, the content to be produced as a speed reading program can be given to the outline extraction model, and then the overall interpretation outline (including paragraph numbers (i.e., sub-paragraph identifiers), paragraph corresponding original content intervals, and paragraph topics (i.e., sub-interval topic words)) can be output. Furthermore, in the actual application process, the output original text outline can be specifically as follows:

[0168] Paragraph:1

[0169] Interval:[0,4000]

[0170] Topic: 1. Create a humorous opening; 2. Outline the theme of the program;…

[0171] Paragraph:2

[0172] Interval:[4001,8000]

[0173] Topic: 1. Topic 1; 2. Topic 2; ...

[0174] Topic: 8. Design a humorous opening; 2. Outline the theme of the program;…

[0175] Paragraph:8

[0176] Interval:[16000,20000]

[0177] Topic: 1. Topic 1; 2. Topic 2; 3. Design a closing statement.

[0178] Secondly, after obtaining the original text outline, the original text content can be segmented according to the original text outline to obtain original text paragraphs; specifically, this can be achieved in the following way: first, the original text outline is parsed to obtain multiple sub-paragraph outlines, and the interval position of the content segment interval included in each sub-paragraph outline is determined; secondly, based on the interval position, the sub-interval text content corresponding to the content segment interval is extracted from the original text content to obtain a sub-text paragraph, and the original text paragraph is generated according to the sub-text paragraph and the sub-paragraph identifier included in the sub-paragraph outline. Among them, the specific division process of the original text paragraph can be achieved in the following way:

[0179] / / Define a function to segment the article

[0180] def segment(String text,int offset,int length){

[0181] …

[0182] return segments

[0183] }

[0184] Also, the returned segment is the original text paragraph.

[0185] In step S120, the original text paragraph and the original text outline are input into a preset content generation large model to obtain paragraph interpretation information corresponding to the original text paragraph.

[0186] Specifically, the specific generation process of the paragraph interpretation information can be realized in the following manner: according to a sub-paragraph identifier of a sub-text paragraph included in the original text paragraph, a sub-paragraph outline corresponding to the text paragraph is determined from the original text outline, and sub-paragraph context information is generated according to the sub-paragraph outline and a preset content parameter prompt information corresponding to the sub-text paragraph; the sub-text paragraph is subjected to word embedding processing based on the embedding mapping layer to obtain sub-text features, and the sub-paragraph context information is subjected to embedding mapping processing to obtain a context marker sequence of the sub-text paragraph; the sub-text features and the context marker sequence of the sub-text paragraph are subjected to encoding processing based on the encoding layer to obtain a context overall representation, and the context marker sequence of the sub-text paragraph and the context overall representation are subjected to prediction based on the hybrid expert model layer to obtain sub-paragraph generated content corresponding to the sub-text paragraph; the sub-paragraph generated content is sorted according to the sub-paragraph identifier, and the sorted sub-paragraph generated content is spliced to obtain paragraph interpretation information corresponding to the original text paragraph.

[0187] In an example embodiment, the specific generation process of the sub-paragraph generated content can be implemented in the following manner: determining the sub-paragraph model weight of the expert neural network model based on the context token sequence of the sub-text paragraph according to the gating network model, and determining the target neural network model required for performing the prediction task of the sub-paragraph generated content from the expert neural network according to the sub-paragraph model weight; inputting the context token sequence of the sub-text paragraph and the overall context representation into the target neural network model to obtain the sub-paragraph generated content corresponding to the sub-text paragraph. The specific determination process of the target neural network model recorded herein is: determining the first sub-paragraph model weight of the expert neural network model in the content fit degree dimension based on the context token sequence of the sub-text paragraph according to the first gating network model, and determining the first target neural network model required for performing the prediction task of the sub-paragraph generated content in the content fit degree dimension from the expert neural network according to the first sub-paragraph model weight; determining the second sub-paragraph model weight of the expert neural network model in the content fluency dimension based on the context token sequence of the sub-text paragraph according to the second gating network model, and determining the second target neural network model required for performing the prediction task of the sub-paragraph generated content in the content fluency dimension from the expert neural network according to the second sub-paragraph model weight.

[0188] In an example embodiment, after obtaining the first target neural network and the second target neural network, the context token sequence of the sub-text paragraph and the overall context representation can be input into the first target neural network to obtain the first prediction result in the content fit dimension; then, the context token sequence of the sub-text paragraph and the overall context representation are input into the second target neural network to obtain the second prediction result in the content fluency dimension; finally, the first prediction result and the second prediction result can be obtained to obtain the sub-paragraph generated content corresponding to the sub-text paragraph. Meanwhile, it should be further pointed out that in the actual application process, the first prediction result and the second prediction result can also be configured with corresponding weight values according to the actual situation; then, the first prediction result and the second prediction result are weighted and summed to obtain the sub-paragraph generated content. The scene diagram of the specific generation process of the sub-paragraph content can be referred to in Figure 6 Meanwhile, the obtained paragraph interpretation information can be referred to in Figure 7 .

[0189] Further, after obtaining the paragraph interpretation information, the paragraph interpretation information needs to be audited; the audited paragraph interpretation information can be referred to in Figure 8 . Based on this way, the accuracy of the obtained paragraph interpretation information can be further improved.

[0190] In step S130, the audio conversion is performed on the paragraph interpretation information based on a preset audio conversion model to obtain paragraph audio data, and the paragraph audio data is integrated to obtain audio data corresponding to the original text content.

[0191] In the example embodiment, first, the paragraph audio data is generated. Specifically, the paragraph audio data can be generated in the following manner: first, the text preprocessing layer is used to preprocess the paragraph interpretation information to obtain standard text information, and the speech synthesis layer is used to synthesize speech based on the standard text information to obtain an original speech file; second, the waveform of the original speech file is cropped based on the speech adjustment layer to obtain an audio cropping result, and the volume of the speech in the audio cropping result is adjusted to obtain the paragraph audio data. The original speech file described herein can be generated in the following manner: the original speech features of the standard text information are extracted based on the speech feature extraction model in the speech synthesis layer, and an original audio spectrum graph is generated based on the original speech features, so that the original speech file is generated based on the original audio spectrum graph. Meanwhile, a scene example diagram of the generation process of the specific paragraph audio data can be referred to as shown in FIG. 6. Figure 9

[0192] It should be noted that the text preprocessing process described herein can include but is not limited to removing irrelevant characters (such as extra spaces, special symbols, etc.), text normalization (such as standardized expression of numbers and dates), analysis and segmentation of words, etc. Of course, text preprocessing can also include part-of-speech tagging, syntax analysis, and semantic understanding, etc. Meanwhile, part-of-speech tagging, syntax analysis, and semantic understanding can help the model fully understand the intent and context of the text, and lay a foundation for generating natural and fluent speech. Further, the speech feature extraction model described herein can include but is not limited to a large language model or a deep neural network model, etc. The example does not make special limitations on this.

[0193] Secondly, after obtaining the paragraph audio data, the paragraph audio data can be integrated to obtain the audio data. Specifically, the specific integration process of the audio data can be implemented in the following manner:

[0194] def merge_audio_files(audio_files,output_file):

[0195] audio_buffers = []

[0196] for file_path in audio_files:

[0197] ​audio_buffers.append(AudioSegment.from_file(file_path))

[0198] merged_audio = AudioSegment.concatenate(audio_buffers)

[0199] merged_audio.export(output_file, format="mp3")

[0200] At this point, the specific generation process of the audio data has been fully implemented. Finally, after obtaining the audio data, the generation method of the audio data further includes: configuring a recommended word for the audio data corresponding to the original text content according to the sub-interval topic word included in the original text outline; publishing the audio data corresponding to the original text content, and identifying the published audio data based on the recommended word. The scene diagram of the specific publishing process of the audio data can be referred to as shown in Figure 10 The interface example diagram after publishing can be referred to as shown in Figure 11 .

[0201] In the following, the generation and publishing process of the audio data disclosed in the example embodiments of the present disclosure will be further explained and described in combination with Figure 12 The generation and publishing process of the audio data can include the following steps as shown in Figure 12 .

[0202] Step 1: generating an outline according to the text content;

[0203] Step 2: segmenting according to the outline content;

[0204] Step 3: generating an interpretation according to the segmentation;

[0205] Step 4: generating audio according to the interpretation;

[0206] Step 5: synthesizing all audio segments in order;

[0207] Step 6: publishing the program.

[0208] Further, based on the foregoing content, it can also be known that the generation method of the audio data disclosed in the example embodiments of the present disclosure can be used to make a speed reading type long duration podcast program making method, which is suitable for popular messages, hot news, IP and other content carriers, reduces the labor production cost and threshold, and enriches the podcast content ecology.

[0209] Exemplary apparatus

[0210] Having introduced the method for generating audio data of the exemplary embodiments of the present disclosure, next, with reference to Figure 13 An apparatus for generating audio data of the exemplary embodiments of the present disclosure is explained and described. Specifically, referring to FIG. 13, the apparatus for generating audio data can include a text outline extraction module 1310, a paragraph interpretation information generation module 1320, and an audio data generation module 1330. Among them: Figure 13

[0211] The text outline extraction module 1310 can be configured to extract an original text outline included in original text content based on a preset outline extraction model, and segment the original text content according to the original text outline to obtain original text paragraphs.

[0212] The paragraph interpretation information generation module 1320 can be configured to input the original text paragraphs and the original text outline into a preset content generation large model to obtain paragraph interpretation information corresponding to the original text paragraphs.

[0213] The audio data generation module 1330 can be configured to perform audio conversion on the paragraph interpretation information based on a preset audio conversion model to obtain paragraph audio data, and integrate the paragraph audio data to obtain audio data corresponding to the original text content.

[0214] In an exemplary embodiment of the present disclosure, the preset outline extraction model includes a paragraph extraction layer, a paragraph encoding layer, and a topic extraction layer; wherein extracting an original text outline included in original text content based on a preset outline extraction model includes: segmenting the original text content based on the paragraph extraction layer to obtain a plurality of content segmentation intervals; encoding the content segmentation intervals based on the paragraph encoding layer to obtain sub-paragraph identifiers, and extracting sub-interval topic words included in sub-interval text content in the content segmentation intervals based on the topic extraction layer; generating a sub-paragraph outline according to the content segmentation intervals, the sub-paragraph identifiers, and the sub-interval topic words, and generating the original text outline according to the sub-paragraph outline.

[0215] ​In an example embodiment of the present disclosure, the topic extraction layer includes a first dynamic weight fusion Bert layer and a first full connection layer; wherein the topic extraction layer extracts a sub-interval topic word included in sub-interval text content in the content segmentation interval, including: based on the first dynamic weight fusion Bert layer, calculating a semantic text vector of the sub-interval text content and an importance of each semantic text vector to the sub-interval text content, and obtaining a first encoding vector according to the semantic text vector and the importance; based on the first full connection layer, calculating the first encoding vector to obtain a confidence that each character in the sub-interval text content belongs to a sub-interval topic word, and determining a starting position and an ending position of the sub-interval topic word according to whether the confidence is greater than a first preset threshold; using a first preset identifier to mark the starting position and the ending position of the sub-interval topic word, and intercepting characters corresponding to positions marked with the first preset identifier to obtain a sub-interval topic word included in the sub-interval text content.

[0216] In an example embodiment of the present disclosure, the first dynamic weight fusion Bert layer includes a plurality of Transformer models; wherein based on the first dynamic weight fusion Bert layer, calculating a semantic text vector of the sub-interval text content and an importance of each semantic text vector to the sub-interval text content, and obtaining a first encoding vector according to the semantic text vector and the importance, includes: performing word embedding on the sub-interval text content to obtain word embedding vectors, word embedding matrices, and position embedding matrices of the sub-interval text content; generating a sub-interval embedding vector according to the word embedding vectors, the word embedding matrices, and the position embedding matrices, and inputting the sub-interval embedding vector into a first Transformer model to obtain a first text semantic vector; inputting the first text semantic vector into other Transformer models to obtain text semantic vectors corresponding to the other Transformer models; wherein in the other Transformer models, the output of a previous Transformer model is the input of a next Transformer model corresponding thereto; calculating the importance of each Transformer model to the sub-interval text content, and obtaining the first encoding vector according to each importance and the sub-interval embedding vector, each text semantic vector.

[0217] In an example embodiment of the present disclosure, the first encoding vector is obtained according to the importance degrees, the sub-interval embedding vectors, and the text semantic vectors, including: splicing the importance degrees to obtain an importance degree splicing result, and performing normalization processing on the importance degree splicing result to obtain normalized importance degrees; and performing weighted summation on the normalized importance degrees, the sub-interval embedding vectors, and the text semantic vectors to obtain the first encoding vector.

[0218] In an example embodiment of the present disclosure, the original text content is segmented according to the original text outline to obtain original text paragraphs, including: parsing the original text outline to obtain a plurality of sub-paragraph outlines, and determining the interval position of a content segmentation interval in the original text content according to the start character position and the end character position of the content segmentation interval included in each of the sub-paragraph outlines; extracting a sub-interval text content corresponding to the content segmentation interval from the original text content based on the interval position to obtain a sub-text paragraph, and generating the original text paragraph according to the sub-text paragraph and a sub-paragraph identifier included in the sub-paragraph outline.

[0219] In an example embodiment of the present disclosure, the preset content generation large model includes an embedding mapping layer, an encoding layer, and a hybrid expert model layer; and the paragraph interpretation information corresponding to the original text paragraph is obtained by inputting the original text paragraph and the original text outline into the preset content generation large model, including: determining a sub-paragraph outline corresponding to the text paragraph from the original text outline according to a sub-paragraph identifier of a sub-text paragraph included in the original text paragraph, and generating sub-paragraph context information according to the sub-paragraph outline and a preset content parameter prompt information corresponding to the sub-text paragraph; performing word embedding processing on the sub-text paragraph based on the embedding mapping layer to obtain sub-text features, and performing embedding mapping processing on the sub-paragraph context information to obtain a context token sequence of the sub-text paragraph; performing encoding processing on the sub-text features and the context token sequence of the sub-text paragraph based on the encoding layer to obtain a context overall representation, and performing prediction on the context token sequence of the sub-text paragraph and the context overall representation based on the hybrid expert model layer to obtain sub-paragraph generated content corresponding to the sub-text paragraph; sorting the sub-paragraph generated content according to the sub-paragraph identifier, and splicing the sorted sub-paragraph generated content to obtain the paragraph interpretation information corresponding to the original text paragraph.

[0220] In an example embodiment of the present disclosure, the mixed expert model layer includes a gating network model and an expert neural network model; wherein the context token sequence and the context overall representation of the subtext passage are input into the mixed expert model layer to obtain the sub-passage generation content corresponding to the subtext passage, including: determining the sub-passage model weight of the expert neural network model based on the context token sequence of the subtext passage by the gating network model, and determining the target neural network model required to perform the prediction task of the sub-passage generation content from the expert neural network according to the sub-passage model weight; inputting the context token sequence and the context overall representation of the subtext passage into the target neural network model to obtain the sub-passage generation content corresponding to the subtext passage.

[0221] In an example embodiment of the present disclosure, the gating network model includes a first gating network model and a second gating network model; wherein the context token sequence of the subtext passage is input into the gating network model to determine the sub-passage model weight of the expert neural network model, and the target neural network model required to perform the prediction task of the sub-passage generation content from the expert neural network according to the sub-passage model weight, including: determining the first sub-passage model weight of the expert neural network model in the content fit degree dimension based on the context token sequence of the subtext passage by the first gating network model, and determining the first target neural network model required to perform the prediction task of the sub-passage generation content in the content fit degree dimension from the expert neural network according to the first sub-passage model weight; determining the second sub-passage model weight of the expert neural network model in the content fluency dimension based on the context token sequence of the subtext passage by the second gating network model, and determining the second target neural network model required to perform the prediction task of the sub-passage generation content in the content fluency dimension from the expert neural network according to the second sub-passage model weight.

[0222] In an example embodiment of the present disclosure, the preset audio conversion model includes a text preprocessing layer, a speech synthesis layer, and a speech adjustment layer; wherein the paragraph interpretation information is input into the preset audio conversion model to obtain the paragraph audio data, including: performing text preprocessing on the paragraph interpretation information based on the text preprocessing layer to obtain standard text information, and performing speech synthesis on the standard text information based on the speech synthesis layer to obtain an original speech file; performing waveform clipping on the original speech file based on the speech adjustment layer to obtain an audio clipping result, and adjusting the speech volume in the audio clipping result to obtain the paragraph audio data.

[0223] In an example embodiment of the present disclosure, the voice synthesis based on the voice synthesis layer on the standard text information is obtained, including: based on the voice feature extraction model in the voice synthesis layer, extracting the original voice features of the standard text information, and generating an original audio spectrum graph according to the original voice features, so as to generate an original voice file based on the original audio spectrum graph.

[0224] In an example embodiment of the present disclosure, the audio data generation device further comprises:

[0225] The recommendation word configuration module is configured to configure a recommendation word for the audio data corresponding to the original text content according to the sub-interval topic word included in the original text outline;

[0226] The audio data identification module is configured to publish the audio data corresponding to the original text content, and identify the published audio data based on the recommendation word.

[0227] Exemplary storage media

[0228] After introducing the audio data generation method and the audio data generation device of the example embodiment of the present disclosure, next, with reference to Figure 14 The storage medium of the example embodiment of the present disclosure is described.

[0229] Referring to Figure 14 As shown in the figure, a program product 1400 for implementing the above method according to the embodiment of the present disclosure is described, which can adopt a portable compact disc read-only memory (CD-ROM) and includes program code, and can run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited to this.

[0230] The program product can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium may, for example, be but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: electrical connections with one or more wires, portable disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disc read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0231] Computer readable signal media may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable program code is carried. The data signal of this propagation may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable signal medium may also be any readable medium other than a readable storage medium. The program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and also including conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN).

[0232] Exemplary electronic devices

[0233] After introducing the storage medium of the exemplary embodiment of the present disclosure, next, reference is made to Figure 15 An electronic device according to an exemplary embodiment of the present disclosure will be described.

[0234] Figure 15 The electronic device 1500 shown is merely an example and should not limit the functionality and scope of use of the embodiments of the present disclosure.

[0235] like Figure 15 As shown, electronic device 1500 is implemented as a general-purpose computing device. Components of electronic device 1500 may include, but are not limited to, the aforementioned at least one processing unit 1510, the aforementioned at least one storage unit 1520, a bus 1530 connecting various system components (including storage unit 1520 and processing unit 1510), and a display unit 1540.

[0236] The storage unit 1520 stores program codes, which can be executed by the processing unit 1510, so that the processing unit 1510 performs the steps described in the "Exemplary Method" section of the present disclosure according to various exemplary embodiments. For example, the processing unit 1510 can perform the following steps: Figure 1 Steps S110-S130 shown in .

[0237] The storage unit 1520 may include a volatile storage unit, such as a random access memory unit (RAM) 15201 and / or a cache memory unit 15202 , and may further include a read-only memory unit (ROM) 15203 .

[0238] The storage unit 1520 can also include a program / utility 15204 having a set of programs / modules 15205, each of which performs one or more of the operations / acts detailed above. These programs / modules 15205 include, but are not limited to, one or more of: an operating system, one or more application programs, other program modules, and program data, each of which or a combination of which can include implementation of a networking environment.

[0239] The bus 1530 can include a data bus, an address bus, and a control bus.

[0240] The electronic device 1500 can also communicate with one or more external devices 1600 such as a keyboard or pointing device, by way of Input / Output (I / O) interface 1550. Additionally, the electronic device 1500 can communicate with one or more networks, such as a Local Area Network (LAN), a Wide Area Network (WAN), and / or a public network, such as the Internet, by way of the network adapter 1560. As depicted, the network adapter 1560 communicates with the other components of the electronic device 1500 by way of the bus 1530. It should be appreciated that the

[0241] It should be noted that, although several modules or sub-modules of the pop-up processing apparatus are mentioned in the above detailed description, such a division is merely exemplary and not mandatory. Indeed, according to embodiments of the present disclosure, the features and functionalities of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functionalities of one unit / module described above can be further divided into several units / modules.

[0242] Moreover, while operations of the methods of the present disclosure are described in a particular order in the figures, this is not required or implied in any particular order or that all of the illustrated operations be performed to achieve desirable results. Additionally or alternatively, certain steps can be omitted, combined, performed in a different order, and / or performed in parallel.

[0243] While the spirit and principles of the present disclosure have been described with reference to several specific embodiments, it is to be understood that the present disclosure is not limited to the precise embodiments disclosed and that various modifications can be made in aspects without departing from the scope of the present disclosure. The present disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A method of generating audio data, characterized by, The method comprises the following steps: extracting an original text outline included in original text content based on a preset outline extraction model, and segmenting the original text content according to the original text outline to obtain original text paragraphs; wherein the original text outline comprises a sub-paragraph outline, and the sub-paragraph outline includes a sub-interval topic word; the sub-interval topic word is obtained by the following method: based on a first dynamic weight fusion Bert layer, calculating a semantic text vector of the sub-interval text content and the importance of each semantic text vector to the sub-interval text content, and obtaining a first encoding vector according to the semantic text vector and the importance; based on a first full connection layer, calculating the first encoding vector to obtain the confidence of each character included in the sub-interval text content belonging to the sub-interval topic word, and judging the starting position and the ending position of the sub-interval topic word according to whether the confidence is greater than a first preset threshold; using a first preset identifier to mark the starting position and the ending position of the sub-interval topic word, and intercepting the characters corresponding to the positions marked with the first preset identifier to obtain the sub-interval topic word included in the sub-interval text content; inputting the original text paragraphs and the original text outline into a preset content generation large model to obtain paragraph interpretation information corresponding to the original text paragraphs; based on a preset audio conversion model, converting the paragraph interpretation information into audio data to obtain paragraph audio data, and integrating the paragraph audio data to obtain audio data corresponding to the original text content.

2. The method of generating audio data according to claim 1, wherein, The preset outline extraction model comprises a paragraph extraction layer, a paragraph encoding layer and a topic extraction layer. The method for extracting the original text outline included in the original text content based on the preset outline extraction model comprises the following steps: segmenting the original text content based on the paragraph extraction layer to obtain a plurality of content segmentation intervals; encoding the content segmentation intervals based on the paragraph encoding layer to obtain sub-paragraph identifiers, and extracting sub-interval topic words included in the sub-interval text content in the content segmentation intervals based on the topic extraction layer; generating a sub-paragraph outline according to the content segmentation intervals, the sub-paragraph identifiers and the sub-interval topic words, and generating the original text outline according to the sub-paragraph outline.

3. The method of generating audio data according to claim 2, wherein, The topic extraction layer comprises a first dynamic weight fusion Bert layer and a first full connection layer.

4. The method of generating audio data according to claim 3, wherein, The first dynamic weight fusion Bert layer comprises a plurality of Transformer models. The method for calculating the semantic text vector of the sub-interval text content and the importance of each semantic text vector to the sub-interval text content based on the first dynamic weight fusion Bert layer, and obtaining a first encoding vector according to the semantic text vector and the importance, comprises the following steps: performing word embedding on the sub-interval text content to obtain word embedding vectors, word embedding matrices and position embedding matrices of the sub-interval text content; The sub-interval embedding vector is generated according to the word embedding vector, the word embedding matrix and the position embedding matrix, and the sub-interval embedding vector is input into a first Transformer model to obtain a first text semantic vector; The first text semantic vector is input into other Transformer models to obtain a text semantic vector corresponding to each of the other Transformer models; wherein the output of a previous Transformer model in the other Transformer models is the input of a next Transformer model corresponding to the previous Transformer model; The importance of each Transformer model to the sub-interval text content is calculated, and the first encoding vector is obtained according to each importance and the sub-interval embedding vector and each text semantic vector.

5. The method of generating audio data according to claim 4, wherein, The first encoding vector is obtained according to each importance and the sub-interval embedding vector and each text semantic vector, including: The importance is spliced to obtain an importance splicing result, and the importance splicing result is normalized to obtain normalized importance; The normalized importance, the sub-interval embedding vector and each text semantic vector are weighted and summed to obtain the first encoding vector.

6. The method of generating audio data according to claim 1, wherein, The original text content is segmented according to the original text outline to obtain an original text paragraph, including: The original text outline is parsed to obtain a plurality of sub-paragraph outlines, and the starting character position and the ending character position of a content segmentation interval included in each sub-paragraph outline are determined to determine the interval position of the content segmentation interval in the original text content; Based on the interval position, the sub-interval text content corresponding to the content segmentation interval is extracted from the original text content to obtain a sub-text paragraph, and the sub-text paragraph and the sub-paragraph identifier included in the sub-paragraph outline are used to generate the original text paragraph.

7. The method of generating audio data according to claim 1, wherein, The preset content generation large model includes an embedding mapping layer, an encoding layer and a hybrid expert model layer; The original text paragraph and the original text outline are input into the preset content generation large model to obtain paragraph interpretation information corresponding to the original text paragraph, including: The sub-paragraph outline corresponding to the text paragraph is determined from the original text outline according to the sub-paragraph identifier of the sub-text paragraph included in the original text paragraph, and the sub-paragraph context information is generated according to the sub-paragraph outline and the preset content parameter prompt information corresponding to the sub-text paragraph; The sub-text feature is obtained by performing word embedding processing on the sub-text paragraph based on the embedding mapping layer, and the context mark sequence of the sub-text paragraph is obtained by performing embedding mapping processing on the sub-paragraph context information; Encoding the sub-text features and the contextual marker sequence of the sub-text paragraph based on the encoding layer to obtain a contextual overall representation, and predicting the contextual marker sequence and the contextual overall representation of the sub-text paragraph based on the hybrid expert model layer to obtain sub-paragraph generated content corresponding to the sub-text paragraph; The sub-paragraph generated contents are sorted according to the sub-paragraph identifiers, and the sorted sub-paragraph generated contents are pieced together to obtain paragraph interpretation information corresponding to the original text paragraph.

8. The method of generating audio data according to claim 7, wherein, The hybrid expert model layer includes a gated network model and an expert neural network model; The method of predicting the contextual marker sequence and the overall context representation of the sub-text paragraph based on the hybrid expert model layer to obtain the sub-paragraph generated content corresponding to the sub-text paragraph includes: Determining a sub-paragraph model weight of the expert neural network model based on the contextual tag sequence of the sub-text paragraph based on the gating network model, and determining a target neural network model required for performing a task of predicting sub-paragraph generated content from the expert neural network based on the sub-paragraph model weight; The context marker sequence and the overall context representation of the sub-text paragraph are input into the target neural network model to obtain the sub-paragraph generated content corresponding to the sub-text paragraph.

9. The method of generating audio data according to claim 8, wherein, The gated network model includes a first gated network model and a second gated network model; The method includes determining the sub-paragraph model weights of the expert neural network model based on the contextual tag sequence of the sub-text paragraph based on the gating network model, and determining the target neural network model required for performing the task of predicting the sub-paragraph generated content from the expert neural network based on the sub-paragraph model weights. Determining, based on the first gating network model and the contextual marker sequence of the sub-text paragraph, a first sub-paragraph model weight of the expert neural network model in a dimension of content fit with the sub-text paragraph, and determining, from the expert neural network, a first target neural network model required for performing a task of predicting sub-paragraph generated content in a dimension of content fit based on the first sub-paragraph model weight; Based on the second gating network model, the second sub-paragraph model weight of the expert neural network model in the content fluency dimension is determined according to the contextual marker sequence of the sub-text paragraph, and based on the second sub-paragraph model weight, the second target neural network model required to perform the prediction task of the sub-paragraph generated content in the content fluency dimension is determined from the expert neural network.

10. The method of generating audio data according to claim 1, wherein, The preset audio conversion model includes a text preprocessing layer, a speech synthesis layer, and a speech adjustment layer; wherein, the paragraph interpretation information is converted into audio based on the preset audio conversion model to obtain paragraph audio data, including: Performing text preprocessing on the paragraph interpretation information based on the text preprocessing layer to obtain standard text information, and performing speech synthesis on the standard text information based on the speech synthesis layer to obtain an original speech file; The waveform of the original voice file is clipped based on the voice adjustment layer to obtain an audio clipping result, and the voice volume in the audio clipping result is adjusted to obtain the paragraph audio data.

11. The method of generating audio data according to claim 10, wherein, The standard text information is synthesized into a voice based on the voice synthesis layer to obtain an original voice file, including: Original voice features of the standard text information are extracted based on a voice feature extraction model in the voice synthesis layer, and an original audio spectrum graph is generated according to the original voice features, so as to generate an original voice file based on the original audio spectrum graph.

12. The method of generating audio data according to claim 1, wherein, The method for generating the audio data further includes: A recommendation word is configured for the audio data corresponding to the original text content according to the sub-interval topic word included in the original text outline; The audio data corresponding to the original text content is published, and the published audio data is identified based on the recommendation word.

13. An apparatus for generating audio data, characterized by It includes: A text outline extraction module is configured to extract an original text outline included in original text content based on a preset outline extraction model, and segment the original text content according to the original text outline to obtain original text paragraphs; wherein the original text outline includes a sub-paragraph outline, and the sub-paragraph outline includes a sub-interval topic word, which is obtained by: calculating semantic text vectors of sub-interval text content and the importance of each semantic text vector to the sub-interval text content based on a first dynamic weight fusion Bert layer, and obtaining a first encoding vector according to the semantic text vectors and the importance; calculating the first encoding vector based on a first full connection layer to obtain the confidence that each character included in the sub-interval text content belongs to a sub-interval topic word, and determining the starting position and the ending position of the sub-interval topic word according to whether the confidence is greater than a first preset threshold; marking the starting position and the ending position of the sub-interval topic word using a first preset identifier, and cutting the characters corresponding to the positions marked with the first preset identifier to obtain the sub-interval topic word included in the sub-interval text content; A paragraph interpretation information generation module is configured to input the original text paragraphs and the original text outline into a preset content generation large model to obtain paragraph interpretation information corresponding to the original text paragraphs; An audio data generation module is configured to perform audio conversion on the paragraph interpretation information based on a preset audio conversion model to obtain paragraph audio data, and integrate the paragraph audio data to obtain audio data corresponding to the original text content.

14. The apparatus of claim 13, wherein The preset outline extraction model includes a paragraph extraction layer, a paragraph encoding layer, and a topic extraction layer; wherein, based on the preset outline extraction model, the original text outline included in the original text content includes: The original text content is segmented based on the paragraph extraction layer to obtain a plurality of content segmentation intervals; Encode the content segmentation interval based on the paragraph encoding layer to obtain a sub-paragraph identifier, and extract a sub-interval topic word included in sub-interval text content in the content segmentation interval based on the topic extraction layer; Generate a sub-paragraph outline according to the content segmentation interval, the sub-paragraph identifier, and the sub-interval topic word, and generate the original text outline according to the sub-paragraph outline.

15. The apparatus of claim 14, wherein, The topic extraction layer comprises a first dynamic weight fusion Bert layer and a first full connection layer.

16. The apparatus of claim 15, wherein, The first dynamic weight fusion Bert layer comprises a plurality of Transformer models. The first dynamic weight fusion Bert layer is used to calculate semantic text vectors of the sub-interval text content and importance of each semantic text vector to the sub-interval text content, and a first encoding vector is obtained according to the semantic text vectors and the importance, comprising: The word embedding of the sub-interval text content is performed to obtain a word embedding vector, a word embedding matrix, and a position embedding matrix of the sub-interval text content; The sub-interval embedding vector is generated according to the word embedding vector, the word embedding matrix, and the position embedding matrix, and the sub-interval embedding vector is input into the first Transformer model to obtain a first text semantic vector; The first text semantic vector is input into other Transformer models to obtain text semantic vectors corresponding to the other Transformer models; wherein the output of a previous Transformer model in the other Transformer models is the input of a next Transformer model corresponding thereto; The importance of each Transformer model to the sub-interval text content is calculated, and the first encoding vector is obtained according to each importance and the sub-interval embedding vector and each text semantic vector.

17. The apparatus of claim 16, wherein, The first encoding vector is obtained according to each importance and the sub-interval embedding vector and each text semantic vector, comprising: The importance is spliced to obtain an importance splicing result, and the importance splicing result is normalized to obtain a normalized importance; The normalized importance, the sub-interval embedding vector, and each text semantic vector are weighted and summed to obtain the first encoding vector.

18. The apparatus of claim 13, wherein, The original text content is segmented according to the original text outline to obtain an original text paragraph, comprising: The original text outline is parsed to obtain a plurality of sub-paragraph outlines, and the interval position of the content segmentation interval in the original text content is determined according to the start character position and the end character position of the content segmentation interval included in each sub-paragraph outline; The sub-interval text content corresponding to the content segmentation interval is extracted from the original text content based on the interval position to obtain a sub-text paragraph, and the original text paragraph is generated according to the sub-text paragraph and the sub-paragraph identifier included in the sub-paragraph outline.

19. The apparatus of claim 13, wherein, The preset content generation model includes an embedding mapping layer, a coding layer and a hybrid expert model layer; The original text paragraph and the original text outline are input into a preset content generation model to obtain paragraph interpretation information corresponding to the original text paragraph, including: Determining, from the original text outline, a subparagraph outline corresponding to the text paragraph based on a subparagraph identifier of a subtext paragraph included in the original text paragraph, and generating subparagraph context information based on the subparagraph outline and preset content parameter prompt information corresponding to the subtext paragraph; Performing word embedding processing on the sub-text paragraph based on the embedding mapping layer to obtain sub-text features, and performing embedding mapping processing on the sub-paragraph context information to obtain a context marker sequence of the sub-text paragraph; Encoding the sub-text features and the contextual marker sequence of the sub-text paragraph based on the encoding layer to obtain a contextual overall representation, and predicting the contextual marker sequence and the contextual overall representation of the sub-text paragraph based on the hybrid expert model layer to obtain sub-paragraph generated content corresponding to the sub-text paragraph; The sub-paragraph generated contents are sorted according to the sub-paragraph identifiers, and the sorted sub-paragraph generated contents are pieced together to obtain paragraph interpretation information corresponding to the original text paragraph.

20. The apparatus of claim 19, wherein, The hybrid expert model layer includes a gated network model and an expert neural network model; The method of predicting the contextual marker sequence and the overall context representation of the sub-text paragraph based on the hybrid expert model layer to obtain the sub-paragraph generated content corresponding to the sub-text paragraph includes: Determining a sub-paragraph model weight of the expert neural network model based on the contextual tag sequence of the sub-text paragraph based on the gating network model, and determining a target neural network model required for performing a task of predicting sub-paragraph generated content from the expert neural network based on the sub-paragraph model weight; The context marker sequence and the overall context representation of the sub-text paragraph are input into the target neural network model to obtain the sub-paragraph generated content corresponding to the sub-text paragraph.

21. The apparatus of claim 20, wherein, The gated network model includes a first gated network model and a second gated network model; The method includes determining the sub-paragraph model weights of the expert neural network model based on the contextual tag sequence of the sub-text paragraph based on the gating network model, and determining the target neural network model required for performing the prediction task of the sub-paragraph generated content from the expert neural network based on the sub-paragraph model weights. Determining, based on the first gating network model and the contextual marker sequence of the sub-text paragraph, a first sub-paragraph model weight of the expert neural network model in a dimension of content fit with the sub-text paragraph, and determining, from the expert neural network, a first target neural network model required for performing a task of predicting sub-paragraph generated content in a dimension of content fit based on the first sub-paragraph model weight; determine, based on the context token sequence of the subtext passage, a second subpassage model weight of the expert neural network model in the content fluency dimension, and determine, based on the second subpassage model weight, a second target neural network model required for performing a prediction task of generating content of a subpassage in the content fluency dimension from the expert neural network.

22. The apparatus of claim 13, wherein, The preset audio conversion model includes a text preprocessing layer, a speech synthesis layer, and a speech adjustment layer; wherein the paragraph interpretation information is converted into paragraph audio data based on the preset audio conversion model, including: based on the text preprocessing layer, the paragraph interpretation information is text preprocessed to obtain standard text information, and based on the speech synthesis layer, the standard text information is speech synthesized to obtain an original speech file; based on the speech adjustment layer, the waveform of the original speech file is cropped to obtain an audio cropping result, and the speech volume in the audio cropping result is adjusted to obtain the paragraph audio data.

23. The apparatus of claim 22, wherein, based on the speech synthesis layer, the standard text information is speech synthesized to obtain an original speech file, including: based on the speech feature extraction model in the speech synthesis layer, the original speech features of the standard text information are extracted, and based on the original speech features, an original audio spectrum graph is generated to generate an original speech file based on the original audio spectrum graph.

24. The apparatus of claim 13, wherein, The audio data generation apparatus further includes: a recommended word configuration module configured to configure recommended words for audio data corresponding to the original text content based on subinterval topic words included in the original text outline; an audio data identification module configured to publish audio data corresponding to the original text content and identify the published audio data based on the recommended words.

25. A computer readable storage medium, characterized in that, A computer program is stored thereon, and the computer program is executed by a processor to implement the audio data generation method of any one of claims 1-12.

26. An electronic device, comprising: including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the audio data generation method of any one of claims 1-12 by executing the executable instructions.

Images