Station group article intelligent broadcasting system and method based on multi-modal emotional speech synthesis

Abstract

The application discloses a station group article intelligent broadcasting system and method based on a multi-modal emotional speech synthesis, relates to the technical field of speech synthesis, and comprises the following steps: extracting paragraph-level semantic and emotional features, performing dimension mapping on the semantic and emotional features, and generating a set of emotional compensation parameters; storing hidden layer state features generated in a previous text paragraph prosody prediction stage according to paragraph granularity based on the set of emotional compensation parameters; when performing prosody prediction on a current text paragraph, tracing back and introducing corresponding previous paragraph hidden layer state features to participate in prosody prediction of the current paragraph, and generating cross-paragraph consistent prosody prediction signals; performing cross-modal alignment and fusion on the prosody prediction signals and the set of emotional compensation parameters, generating acoustic feature prediction results, converting the acoustic feature prediction results into mel spectrum features, and generating corresponding digital audio signal streams; and the application improves speech naturalness and emotional consistency of long text automatic broadcasting.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech synthesis, in particular to a station group article intelligent broadcasting system and method based on multi-modal emotional speech synthesis. BACKGROUND

[0002] In recent years, neural text-to-speech (TTS) technology has made great progress driven by deep learning algorithms, especially end-to-end synthesis models based on the Transformer architecture, which can achieve highly realistic speech generation. In the distributed station group management scenario, the existing technology generally adopts a scheme combining a streaming inference framework and a pre-trained acoustic model, which realizes automatic speech content production by extracting semantic representations from large-scale text corpora and cooperating with a vocoder. This kind of technology greatly improves the digital dissemination efficiency of station group information by integrating attention mechanisms and multi-scale frequency prediction, and provides a basic technical guarantee for the content distribution of large-scale vertical sites.

[0003] However, when facing the continuous broadcasting needs of long articles in the station group, the existing speech synthesis technology still has obvious limitations in modeling the continuity of cross-paragraph speech; specifically, the existing method cannot effectively utilize the speech state formed by the previous paragraph to constrain the prosody generation of the current paragraph when generating speech paragraph by paragraph, resulting in sudden changes in emotional expression and intonation rhythm during paragraph switching; for example, when continuously broadcasting multiple paragraphs of news text, the speech of the next paragraph cannot continue the speech rate, pitch trend and stress distribution formed by the previous paragraph at the starting position, but starts again with the default broadcasting state, causing the speech rhythm to be fast and slow, the emotional ups and downs to be incoherent, and the overall broadcasting effect to lack coherence, making it difficult to meet the requirements of natural flow and listening experience. SUMMARY

[0004] In view of the above existing problems, the present application is proposed.

[0005] Therefore, the present application provides a station group article intelligent broadcasting method based on multi-modal emotional speech synthesis to solve the problem of incoherent emotional prosody between paragraphs in station group article broadcasting.

[0006] To solve the above technical problems, the present application provides the following technical solutions:

[0007] In a first aspect, the present application provides a station group article intelligent broadcasting method based on multi-modal emotional speech synthesis, which comprises:

[0008] Obtaining multi-source text content in a distributed station group, stripping non-text labels, and performing semantic enhancement on the multi-source text content to generate a serialized text stream with broadcasting guidance;

[0009] Based on the serialized text stream, the paragraph-level semantic and emotional features are extracted, and the semantic and emotional features are dimensionally mapped to generate a set of sentiment compensation parameters;

[0010] Based on the set of sentiment compensation parameters, the hidden layer state features generated in the previous text paragraph prosody prediction stage are stored by paragraph granularity. When performing prosody prediction on the current text paragraph, the corresponding previous paragraph hidden layer state features are introduced and used to participate in the prosody prediction of the current paragraph, to generate a cross-paragraph consistent prosody prediction signal;

[0011] The prosody prediction signal and the set of sentiment compensation parameters are cross-modally aligned and fused to generate an acoustic feature prediction result, and the acoustic feature prediction result is converted into a mel-spectrum feature to generate a corresponding digital audio signal stream;

[0012] According to the site attribute metadata, the digital audio signal stream is timbre matched and encapsulated, and is pushed to the playing entrance of the corresponding site cluster for intelligent broadcasting.

[0013] Preferably, the method for generating a serialized text stream with broadcasting guidance comprises:

[0014] Obtain multi-source text content from a distributed site cluster, and remove scripts and format tags in the multi-source text content to extract pure text information;

[0015] Perform industry term and numerical annotation on the pure text information, and perform syntax analysis to determine pause points and stress points; insert corresponding broadcasting control symbols in the text sequence, and format and encapsulate according to the broadcasting guidance protocol to generate a serialized text stream.

[0016] Preferably, the method for generating a set of sentiment compensation parameters comprises:

[0017] According to the paragraph boundary information in the text, the serialized text stream is split to obtain multiple text paragraphs and extract the corresponding paragraph-level semantics of each text paragraph;

[0018] Based on the paragraph-level semantics, the semantic content and emotional expression features of each text paragraph are analyzed, the emotional tendency of the text paragraph is identified, and the corresponding emotional category and emotional intensity parameter of the emotional expression strength are determined based on the emotional tendency, to form the paragraph-level emotional features corresponding to each text paragraph;

[0019] Based on the paragraph-level semantic and emotional features, the mapping relationship between them and the acoustic performance of voice synthesis is constructed, and the corresponding acoustic control features are generated accordingly;

[0020] Based on the acoustic control features, the corresponding prosody control elements are extracted, and the prosody control elements are vector combined to generate a set of sentiment compensation parameters.

[0021] Preferably, the hidden layer state features generated by the pre-paragraph prosody prediction stage are stored in paragraph granularity, including:

[0022] Based on the set of emotional compensation parameters, in the order of the text paragraphs in the serialized text stream, combined with the corresponding broadcast guide of the text paragraphs, the prosody control elements of the text paragraphs are predicted to obtain the prosody prediction results corresponding to each text paragraph;

[0023] In the process of generating prosody prediction results, hidden layer state features related to the corresponding text paragraphs are extracted to form paragraph-level hidden layer state features;

[0024] The paragraph-level hidden layer state features are associated with the paragraph position of the corresponding text paragraph in the serialized text stream, and a paragraph-level hidden layer state index relationship is constructed. The hidden layer state features corresponding to the pre-paragraph text paragraphs are sequentially stored in paragraph granularity to form a paragraph-level hidden layer state set that can be called by the subsequent prosody prediction process.

[0025] Preferably, the method for generating cross-paragraph consistent prosody prediction signals includes:

[0026] Before performing prosody prediction on the current text paragraph, based on the paragraph-level hidden layer state index relationship, the backtracking range is limited to the pre-paragraph text paragraphs adjacent to the current text paragraph position in the serialized text stream, and the corresponding paragraph-level hidden layer state features are located;

[0027] The paragraph-level hidden layer state features corresponding to the pre-paragraph text paragraphs are used as context constraint information in prosody prediction, and are used together with the set of emotional compensation parameters to participate in the prosody control element prediction of the current text paragraph;

[0028] Based on the context constraint information and the set of emotional compensation parameters, the prosody control elements of the current text paragraph are continuously predicted to generate prosody prediction signals that maintain the consistency of emotional and prosodic evolution between adjacent paragraphs.

[0029] Preferably, the method for generating acoustic feature prediction results includes:

[0030] Based on the structural position of the text paragraph in the serialized text stream, the prosody prediction signal and the set of emotional compensation parameters are synchronized and aligned in time and paragraph dimensions;

[0031] The prosody control elements in the aligned prosody prediction signal and the emotional intensity parameters and acoustic control features in the set of emotional compensation parameters are fused at the feature level to construct a joint acoustic control feature representation;

[0032] Based on the joint acoustic control feature representation, the basic acoustic parameters in the speech synthesis process are predicted to generate corresponding acoustic feature prediction results.

[0033] Preferably, the method of generating a corresponding digital audio signal stream comprises:

[0034] Based on the acoustic feature prediction result, the multi-dimensional acoustic control information required in the speech synthesis process is time-sequentially organized to form an acoustic feature sequence matching the text paragraph structure;

[0035] According to the acoustic feature sequence, mapping from the time-sequential acoustic feature space to the spectral representation space is performed to generate mel-spectral features; based on the mel-spectral features, acoustic reconstruction is performed to convert the spectral representation into a time-domain continuous waveform to generate a continuous digital audio signal stream.

[0036] Preferably, the intelligent broadcasting comprises:

[0037] Through the site management interface, the site attribute metadata of the target site in the site cluster is collected; based on the site attribute metadata, timbre matching processing is performed on the digital audio signal stream, and the digital audio signal stream after timbre matching is format-encapsulated according to the playing interface specification corresponding to the target site to generate site-adapted broadcast audio data;

[0038] The broadcast audio data is pushed to the playing entrance of the corresponding site cluster to trigger the automatic broadcast process on the site side and perform intelligent broadcasting of the site cluster article.

[0039] Preferably, the method of continuously constraining and predicting the prosody control elements of the current text paragraph based on the context constraint information and the set of emotion compensation parameters comprises:

[0040] When performing prosody prediction on the current text paragraph, the paragraph-level hidden state features corresponding to the previous text paragraph obtained by backtracking are introduced into the prosody prediction process of the current text paragraph as context constraint information and participate in recursive calculation as the initial state of the current text paragraph;

[0041] In the recursive calculation process, the set of emotion compensation parameters corresponding to the current text paragraph and the broadcast guide information are introduced step by step according to the time step evolution order to impose continuity constraints on the prosody control elements, generate a continuously changing prosody prediction value sequence step by step, and obtain a prosody prediction signal that maintains the continuity of emotion evolution and the consistency of prosody transition between adjacent text paragraphs.

[0042] In the second aspect, the present application provides a site cluster article intelligent broadcasting system based on multi-modal emotion speech synthesis, comprising:

[0043] A text processing module is configured to acquire multi-source text content in a distributed site cluster, strip non-text labels, and perform semantic enhancement on the multi-source text content to generate a serialized text stream with broadcast guidance;

[0044] An emotion mapping module is configured to extract paragraph-level semantic and emotional features based on the serialized text stream, and perform dimension mapping on the semantic and emotional features to generate a set of emotion compensation parameters;

[0045] A prosody scheduling module is configured to store the hidden layer state features generated in the prosody prediction phase of the previous text paragraph based on the set of emotion compensation parameters, and when performing prosody prediction on the current text paragraph, backtrack and introduce the corresponding hidden layer state features of the previous paragraph to participate in the prosody prediction of the current paragraph to generate a cross-paragraph consistent prosody prediction signal.

[0046] An acoustic synthesis module is configured to perform cross-modal alignment and fusion of the prosody prediction signal and the set of emotion compensation parameters to generate an acoustic feature prediction result, and convert the acoustic feature prediction result into a mel-spectrum feature to generate a corresponding digital audio signal stream.

[0047] A distribution module is configured to perform timbre matching and packaging on the digital audio signal stream according to the site attribute metadata, and push the digital audio signal stream to the playback entrance of the corresponding site cluster for intelligent broadcasting.

[0048] The present application has the following advantages: by performing semantic enhancement and emotion dimension mapping on multi-source text content, a more delicate and controllable voice expression basis is constructed for station group article broadcasting; and by introducing a backtracking storage mechanism of the hidden layer state of the previous paragraph, the emotion evolution trend and prosody rhythm information are continuously modeled and transmitted in the cross-paragraph range, so that the consistency and smoothness of voice performance are maintained at the paragraph junction. The participation of this long-range context information enables the entire article to maintain stable speech rate changes, tone trends and emotional evolution characteristics during the broadcasting process, reducing the incoherent listening phenomenon that may occur during paragraph switching, making the synthesized voice closer to natural broadcasting effect in terms of overall rhythm organization and narrative coherence. Therefore, the present application effectively improves the voice naturalness and emotional consistency in the automatic broadcasting scenario of large-scale long text, and enhances the auditory experience and user immersion in the station group information distribution process. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0050] Figure 1 The flowchart of the station group article intelligent broadcasting method based on multi-modal emotion speech synthesis in the present application;

[0051] Figure 2A schematic diagram of a station group article intelligent broadcasting system based on multi-modal emotional speech synthesis in the present application;

[0052] Figure 3 A flowchart of generating a set of emotional compensation parameters in the present application;

[0053] Figure 4 A flowchart of generating a cross-paragraph consistent prosody prediction signal in the present application. DETAILED DESCRIPTION

[0054] In order to make the above objectives, features and advantages of the present application more apparent, specific embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0055] In the following description, a large number of specific details are set forth in order to facilitate a thorough understanding of the present application, but the present application can also be implemented in other ways different from those described herein, and those skilled in the art can make similar generalizations without departing from the spirit of the present application, therefore the present application is not limited to the specific embodiments disclosed below.

[0056] Secondly, the "one embodiment" or "embodiment" referred to herein means that the specific features, structures or characteristics can be included in at least one implementation of the present application. "In one embodiment" appearing in different places in the specification does not mean the same embodiment, nor is it an independent or alternative embodiment that excludes other embodiments.

[0057] Reference Figure 1 , Figure 2 , Figure 3 and Figure 4 In one embodiment of the present application, the embodiment provides a station group article intelligent broadcasting method based on multi-modal emotional speech synthesis, comprising the following steps:

[0058] The method for generating a serialized text stream with broadcasting guidance includes:

[0059] Obtain multi-source text content from a distributed station cluster, and remove scripts and format tags in the multi-source text content to extract pure text information.

[0060] Specifically, a hypertext transfer protocol request is initiated to a distributed site cluster through a network communication interface, a source data storage location of each site in the distributed site cluster is located, and multi-source text content containing hypertext markup language tags, style sheet scripts, and web page logic scripts is obtained. The multi-source text content is scanned line by line and characterized by using a character pattern matching rule containing a specific escape sequence combined with a wildcard, specific start symbols, end symbols, and script logic code clamped between the symbols such as "<script.?>.?" and "<style.?>.?" in the multi-source text content are located, the identified script and format tags are removed through character offset positioning and string replacement operations, only the original character sequence carrying the actual semantics in the multi-source text content is retained, and the extraction of pure text information is completed.

[0061] The pure text information is subjected to industry term and numerical value phonetic notation, and syntax analysis is performed to determine pause points and stress points; corresponding broadcast control symbols are inserted in the text sequence, and are formatted and packaged according to a broadcast guide protocol to generate a serialized text stream.

[0062] Specifically, the pure text information is normalized sentence by sentence, industry-specific words and numerical sequences in the text are mapped to corresponding phonetic notation representations through pattern matching and context feature recognition; syntax analysis is then performed on the phonetic notated text content, pause point positions within the sentence are determined through punctuation symbol position distribution and semantic boundary rules, and main sentences and modifying components are distinguished to determine stress point positions for voice emphasis through word arrangement order, dependency relationship, and part of speech annotation results; according to the character offset of the pause point positions and the stress point positions in the text, corresponding broadcast control symbols are inserted in the text sequence, and the text characters, phonetic notation representations, and corresponding broadcast control symbols are fielded according to the temporal topology structure to package into structured data packets containing header identifiers, semantic payloads, and control cells, generating a serialized text stream with clear broadcast structure guidance.

[0063] The method for generating a set of sentiment compensation parameters includes:

[0064] According to the paragraph boundary information in the text, the serialized text stream is split to obtain multiple text paragraphs and extract the paragraph-level semantics corresponding to each text paragraph.

[0065] It should be noted that the binary stream is parsed, the character data and the broadcast control symbol in the serialized text stream are read, the specific control code marked as a line feed control symbol or a paragraph end marker is identified, the position of the specific control code in the serialized text stream is confirmed as paragraph boundary information; based on the byte offset index of the paragraph boundary information in the serialized text stream, the data slicing operation is performed on the serialized text stream, the data segment containing the header identifier, semantic load and control information element between two adjacent paragraph boundary information is extracted, and the independent text paragraph is constructed;

[0066] The semantic load content in the text paragraph is extracted, the semantic load content is converted into a dense vector sequence by using a word embedding query table; the dense vector sequence is respectively multiplied by a query linear transformation matrix and a key linear transformation matrix to obtain a query matrix and a key matrix The correlation weight of the dense vector sequence in the global context is calculated by a multi-head self-attention mechanism, and a corresponding context hidden layer state sequence is generated based on the correlation weight; the expression is: wherein, is the correlation weight matrix, is a normalized exponential function, is a scaling factor, which is derived from the dimension value of a single key vector in the key matrix, is a matrix transpose operator, is the index of the key matrix dimension.

[0067] The average pooling operation is performed on the context hidden layer state sequence in the time step dimension, the variable-length context hidden layer state sequence is aggregated into a fixed-dimension numerical feature vector, and the numerical feature vector is confirmed as the paragraph-level semantic corresponding to the text paragraph.

[0068] Based on the paragraph-level semantic, the semantic content and the emotional expression feature of each text paragraph are analyzed, the sentiment tendency of the text paragraph is identified, and the corresponding emotional category and emotional intensity parameter of the emotional expression strength degree are determined based on the sentiment tendency, forming the paragraph-level emotional feature corresponding to each text paragraph.

[0069] It should be noted that the corresponding paragraph-level semantic of each text paragraph is multiplied by the matrix and the bias vector is accumulated to realize linear mapping from the semantic feature space to the emotion label space, and then the mapping result is executed by the normalization exponential function operation to generate the probability distribution score of multiple emotion dimensions. Based on the probability distribution score, the emotion dimension corresponding to the highest score is confirmed as the emotion category, and the highest score original value is mapped to the intensity interval of 0 to 1 to obtain the emotion intensity parameter. Finally, the emotion category and the emotion intensity parameter are field combined to generate the paragraph-level emotion features corresponding to each text paragraph. Example: the preset emotion label space includes calm, happy, angry, sad and fear dimensions;

[0070] It should be noted that the preset process of the emotion weight matrix is: obtaining the feature dimension value of the numerical feature vector corresponding to the paragraph-level semantic, setting it as the row dimension of the emotion weight matrix, and obtaining the total number of classification labels in the preset emotion label space, setting it as the column dimension of the emotion weight matrix, constructing a two-dimensional matrix structure based on the row dimension and the column dimension, and determining the weight value inside the matrix structure as the linear transformation coefficient for projecting the semantic feature space to the emotion label space.

[0071] Based on the paragraph-level semantic and the paragraph-level emotion feature, the mapping relationship between them and the acoustic performance of the speech synthesis is constructed, and the corresponding acoustic control feature is generated accordingly.

[0072] Specifically, the paragraph-level semantic and the paragraph-level emotion feature are executed in the feature dimension to perform vector splicing to construct a joint feature vector that integrates semantic background and emotional attributes; the joint feature vector is input into the acoustic mapping linear transformation matrix to perform matrix multiplication operation, and the mapping relationship between the joint feature vector and the physical parameters such as fundamental frequency, time length and energy is established to generate acoustic control features representing the basic pronunciation characteristics in speech synthesis;

[0073] It should be noted that the preset process of the acoustic mapping linear transformation matrix is: obtaining the splicing dimension value of the joint feature vector as the row dimension, obtaining the total amount of physical parameters contained in the acoustic control feature as the column dimension, constructing the corresponding conversion weight matrix, which is used to convert the semantic emotion features in the text dimension to the parameter expression in the acoustic dimension.

[0074] Based on the acoustic control feature, the corresponding prosody control elements are extracted, and the prosody control elements are combined to generate a set of emotion compensation parameters.

[0075] It should be noted that the numerical components that can reflect the speed rhythm, pitch fluctuation and pause mode are extracted from the acoustic control feature, which are confirmed as the prosody control elements; the prosody control elements are executed by the vector combination operation to encapsulate the emotion compensation parameter set containing multi-dimensional pronunciation adjustment information.

[0076] Storing the hidden layer state features generated in the prosody prediction stage of the preceding text paragraph by paragraph granularity comprises:

[0077] Based on the set of emotional compensation parameters, in the order of the text paragraphs in the serialized text stream, combined with the corresponding broadcast guide of the text paragraphs, the prosody control elements of the text paragraphs are predicted to obtain the prosody prediction result corresponding to each text paragraph.

[0078] Specifically, the arrangement order of each text paragraph in the serialized text stream is obtained, the set of emotional compensation parameters corresponding to each text paragraph is aligned with the broadcast control symbol in the broadcast guide in the time dimension to form a data sequence arranged by time steps; according to the time sequence of the data sequence, the data of each time step is executed to perform numerical mapping calculation containing historical state feedback; the expression is: , wherein is the state information used to represent the prosody evolution state at the current time step, is the recursive calculation rule of numerical mapping and nonlinear transformation, is the input data composed of the emotional compensation parameters at the current time step and the broadcast guide information, is the state information reserved at the last time step;

[0079] The recursive calculation rule specifically embodies that a recursive relationship is established between the input data at the current time step and the state information reserved at the last time step, the numerical sequence reflecting the prosody length change, the fundamental frequency evolution track and the energy envelope change is generated step by step through continuous accumulation and nonlinear transformation, and the numerical sequence is confirmed as the prosody prediction result of the corresponding text paragraph;

[0080] It should be noted that the recursive calculation process introduces the state information of the previous time step at each time step, so that the numerical calculation result of the current time step is simultaneously constrained by the current set of emotion compensation parameters and the historical speech state, so as to depict the continuous change of the prosody control element over time; before determining the recursive calculation relationship, a sample data set composed of multiple sets of text paragraphs and their corresponding standard speech broadcast data is constructed, wherein the standard speech broadcast data is extracted by acoustic analysis processing to obtain real duration parameters, fundamental frequency trajectory parameters and energy envelope parameters; taking the set of emotion compensation parameters corresponding to the text paragraph and the broadcast guide as input, and taking the extracted real speech prosody parameters as reference, the mean square deviation between the generated prosody prediction result and the real speech prosody parameters is gradually reduced through multiple rounds of iterative correction of the numerical mapping coefficients in the recursive calculation, until the prediction error converges to a preset threshold, for example, not greater than 0.001; after the numerical mapping relationship is determined, the speech speed, fundamental frequency trajectory and energy distribution in the prosody prediction result are weighted and adjusted according to the emotion category and emotion intensity parameters in the paragraph-level emotion features, so that the generated prosody prediction result reflects the emotional expression logic of the text at the acoustic level.

[0081] In the process of generating the prosody prediction result, the hidden layer state features related to the corresponding text paragraph are extracted to form the paragraph-level hidden layer state features.

[0082] It should be noted that in the process of performing recursive calculation step by step, when the data processing of the last time step of the current text paragraph is completed, the state information used for participating in the calculation of the next time step is saved, and the state information is confirmed as the paragraph-level hidden layer state feature corresponding to the current text paragraph; for example, the paragraph-level hidden layer state feature is a feature vector containing 256 numerical values.

[0083] The paragraph-level hidden layer state features are associated with the paragraph positions of the corresponding text paragraphs in the serialized text stream to construct a paragraph-level hidden layer state index relationship; and the hidden layer state features corresponding to the previous text paragraphs are sequentially stored in paragraph granularity to form a paragraph-level hidden layer state set that can be called by the subsequent prosody prediction process.

[0084] Specifically, the physical paragraph number of the current text paragraph in the serialized text stream is obtained, and the physical paragraph number and the corresponding paragraph-level hidden layer state feature are mapped and associated in the key-value pair storage space to form a paragraph-level hidden layer state index relationship; according to the increasing order of the physical paragraph number, the paragraph-level hidden layer state features obtained after the processing of each text paragraph are written into the circular cache queue in turn to construct a paragraph-level hidden layer state set for saving the speech state information of the previous text paragraphs, which is called by the prosody prediction process of the subsequent text paragraph.

[0085] The method for generating a prosody prediction signal consistent across paragraphs comprises:

[0086] Before performing prosody prediction on the current text paragraph, based on the paragraph-level hidden state index relationship, the backtracking range is limited to the previous text paragraph adjacent to the current text paragraph position in the serialized text stream, and the corresponding paragraph-level hidden state feature is located.

[0087] Specifically, before performing prosody prediction on the current text paragraph, one or more text paragraphs adjacent to and before the current text paragraph in the serialized text stream are determined by searching the paragraph-level hidden state index relationship according to the physical paragraph number; according to the determined physical paragraph number, the corresponding paragraph-level hidden state feature is extracted from the circular buffer queue, and the extracted paragraph-level hidden state feature is confirmed as context constraint information for constraining the prosody evolution of the current paragraph.

[0088] The paragraph-level hidden state feature corresponding to the previous text paragraph is used as context constraint information in prosody prediction, and participates in prosody control element prediction of the current text paragraph together with the set of emotional compensation parameters.

[0089] It should be noted that when performing prosody control element prediction on the current text paragraph, the paragraph-level hidden state feature is used as context constraint information for representing the prosody evolution state of the previous speech, which participates in the initial state setting of the prosody prediction of the current text paragraph, so that the semantic and emotional evolution trend formed in the previous text paragraph can be continuously transmitted in the paragraph switching process; the set of emotional compensation parameters corresponding to the current text paragraph is introduced into the prosody prediction process according to the time step order, as an adjustment factor for driving the prosody change amplitude and change trend within the current paragraph, so that the prosody control element is simultaneously constrained by the historical prosody state and the current paragraph emotional expression demand in the time evolution process.

[0090] Based on the context constraint information and the set of emotional compensation parameters, the prosody control element of the current text paragraph is continuously constrained and predicted to generate a prosody prediction signal that maintains the consistency of emotional and prosodic evolution between adjacent paragraphs.

[0091] The method for generating acoustic feature prediction results includes:

[0092] Based on the structural position of the text paragraph in the serialized text stream, the prosody prediction signal and the set of emotional compensation parameters are synchronized and aligned in the time dimension and the paragraph dimension.

[0093] Specifically, the duration prediction value contained in the prosody prediction signal is read to determine the total length of the acoustic frame sequence corresponding to the current text paragraph in the speech synthesis process. According to the structural position number of the text paragraph in the serialized text stream, the set of emotion compensation parameters corresponding to the text paragraph is retrieved from the cache. A vector expansion operation is performed to time-series copy and expand the set of emotion compensation parameters at the paragraph level according to the total length of the acoustic frame sequence, so that it is strictly consistent with the number of frames contained in the prosody prediction signal in the time dimension, and the synchronization alignment of the prosody prediction signal and the set of emotion compensation parameters in the time dimension and the paragraph dimension is completed. Example: The total length of the acoustic frame sequence is 500 frames, and 1 paragraph-level vector is copied 500 times to form a sequence.

[0094] The feature-level fusion of the prosody control elements in the aligned prosody prediction signal and the emotion intensity parameters and acoustic control features in the filled set of emotion compensation parameters is performed to construct a joint acoustic control feature representation.

[0095] It should be noted that the prosody control elements (such as fundamental frequency, duration, and energy values) contained in the aligned prosody prediction signal and the filled set of emotion compensation parameters (containing emotion intensity and acoustic control values) are spliced in the dimension layer to construct a composite numerical vector containing multiple acoustic information. The composite numerical vector is subjected to weight weighting mapping, the dimension differences between the features are eliminated and the internal correlation information is fused by weighting and summing the values from different sources and nonlinear conversion, and a joint acoustic control feature representation with global acoustic guiding function is constructed.

[0096] Based on the joint acoustic control feature representation, the basic acoustic parameters in the speech synthesis process are predicted to generate corresponding acoustic feature prediction results.

[0097] It should be noted that according to the time sequence of the acoustic frame sequence, the composite numerical vector corresponding to the current acoustic frame in the joint acoustic control feature representation is read frame by frame, and linear mapping and nonlinear transformation calculation are performed on the composite numerical vector to gradually convert the prosody control elements, emotion intensity parameters, and acoustic control features contained therein into numerical expressions consistent with the physical parameters of speech synthesis; during the conversion process, dimension rearrangement and weighted sum operation are performed on the composite numerical vector to generate an intermediate acoustic mapping vector, and then nonlinear compression and numerical smoothing processing are performed on the numerical values in the intermediate acoustic mapping vector to depict the corresponding relationship between the joint acoustic control feature representation and the fundamental frequency variation, spectral envelope shape, and non-periodic component distribution; by repeating the above mapping calculation for each acoustic frame, the fundamental frequency value, spectral envelope parameter, and non-periodic component parameter corresponding to each acoustic frame are output frame by frame, and the basic acoustic parameters of each acoustic frame are combined in time sequence to generate acoustic feature prediction results consistent with the text paragraph structure, prosody prediction signal, and emotion evolution process in the time dimension.

[0098] The method for generating a corresponding digital audio signal stream comprises:

[0099] Based on the acoustic feature prediction result, the multi-dimensional acoustic control information required in the speech synthesis process is time-sequentially organized to form an acoustic feature sequence matching the text paragraph structure.

[0100] It should be noted that the corresponding acoustic feature prediction result is read by paragraph according to the arrangement order of the text paragraphs in the serialized text stream; the acoustic feature prediction result contains the fundamental frequency value, the spectral envelope parameter and the non-periodic component parameter arranged in the order of acoustic frames;

[0101] The fundamental frequency value, the spectral envelope parameter and the non-periodic component parameter at the same acoustic frame time point are field-combined to form an acoustic feature unit corresponding to the acoustic frame; according to the time order of the acoustic frames, the acoustic feature units are sequentially arranged to construct a complete acoustic feature sequence.

[0102] According to the acoustic feature sequence, the mapping from the time-sequential acoustic feature space to the spectral representation space is performed to generate a mel-spectral feature; based on the mel-spectral feature, the acoustic reconstruction is performed to convert the spectral representation into a time-domain continuous waveform to generate a continuous digital audio signal stream.

[0103] It should be noted that the acoustic feature units in the acoustic feature sequence are read frame by frame according to the time order of the acoustic frames, the fundamental frequency value is used to determine the reference position of the spectral energy distribution, the spectral envelope parameter is used to describe the energy attenuation form of each frequency component, and the non-periodic component parameter is used to compensate for the random disturbance of the spectral energy, so as to convert the fundamental frequency value, the spectral envelope parameter and the non-periodic component parameter into a corresponding short-time spectral energy distribution; then the short-time spectral energy distribution is mapped to the mel frequency scale, the spectral energy in the mel frequency interval is weighted and aggregated to generate a mel-spectral feature sequence;

[0104] According to the time order of the mel-spectral feature sequence, the spectral inverse transformation is performed frame by frame to reconstruct a short-time amplitude spectrum, and the phase estimation and random disturbance compensation are performed in combination with the fundamental frequency value and the non-periodic component parameter of the corresponding acoustic frame to generate an acoustic frame-level time-domain waveform segment; the overlap-add and smooth splicing are performed on the time-domain waveform segments in time order to form a continuous digital audio signal stream segment, and the sequential splicing is performed according to the arrangement order of the text paragraphs in the serialized text stream, and the corresponding duration of the silence interval is inserted between adjacent paragraphs according to the pause control symbol in the broadcast guide to obtain a digital audio signal stream.

[0105] The intelligent broadcasting comprises:

[0106] The station attribute metadata of the target station in the station group is collected through the station management interface; based on the station attribute metadata, timbre matching processing is performed on the digital audio signal stream, and the digital audio signal stream after the timbre matching is completed is format-encapsulated according to the playing interface specification corresponding to the target station to generate station-adapted broadcast audio data.

[0107] It should be noted that when collecting the station attribute metadata, the station identification information, terminal type identification information, playing device type information and playing interface specification information corresponding to the target station are read respectively; according to the playing interface specification information, the sampling rate, the number of channels and the encoding format required by the audio data are determined; according to the terminal type identification information and the playing device type information, the spectral distribution, the amplitude range and the overall volume of the digital audio signal stream are parameter-adjusted to complete the timbre matching processing; after the timbre matching processing is completed, the digital audio signal stream is format-encapsulated according to the determined sampling rate, the number of channels and the encoding format to obtain the station-adapted broadcast audio data.

[0108] The broadcast audio data is pushed to the playing entrance of the corresponding station cluster to trigger the automatic broadcast process on the station side and perform intelligent broadcast of the station group article.

[0109] It should be noted that according to the station identification information of the target station in the station group, the station-adapted broadcast audio data is sent to the playing entrance address corresponding to the target station through the network communication interface; after the broadcast audio data is successfully written into the playing entrance, a playing trigger instruction is sent to the target station, so that the target station performs automatic playing operation according to the time structure and broadcast guide information in the broadcast audio data, thereby completing the intelligent broadcast of the station group article on the target station side.

[0110] The method for continuously constraining and predicting the prosody control elements of the current text paragraph based on the context constraint information and the set of emotional compensation parameters comprises:

[0111] When performing prosody prediction on the current text paragraph, the paragraph-level hidden layer state feature corresponding to the previous text paragraph obtained by backtracking is taken as the context constraint information, introduced into the prosody prediction process of the current text paragraph, and taken as the initial state of the current text paragraph to participate in recursive calculation.

[0112] It should be noted that by taking the paragraph-level hidden layer state feature reserved when the previous text paragraph completes prosody prediction as the starting constraint of the recursive calculation of the current text paragraph, the semantic and emotional state information contained in the previous text paragraph is continuously transmitted in the paragraph switching process, avoiding the interruption of state information in the prosody prediction process, thereby providing a stable context constraint basis for the prosody prediction of the current text paragraph.

[0113] In the recursive calculation process, the emotion compensation parameter set corresponding to the current text paragraph and the broadcasting guide information are introduced in the time step evolution order, the continuity constraint is applied to the prosody control element, the continuously changing prosody prediction value sequence is generated in the time step, and the prosody prediction signal maintaining the continuity of emotion evolution and the consistency of prosody transition between adjacent text paragraphs is obtained.

[0114] Specifically, in the time step by step recursive process, the constraint of the context constraint information on the time evolution direction of the prosody prediction is combined with the driving effect of the emotion compensation parameter set on the prosody adjustment amplitude and the change trend, so that the generated prosody prediction value presents the smooth transition characteristics in the dimensions of tone connection, speech speed rhythm and stress distribution, and the synthesized speech maintains the natural and coherent listening effect in the long text continuous broadcasting scene.

[0115] The embodiment also provides a station group article intelligent broadcasting system based on multi-modal emotion speech synthesis, which comprises:

[0116] A text processing module is configured to acquire multi-source text content in a distributed station cluster, strip non-text labels, and perform semantic enhancement on the multi-source text content to generate a serialized text stream with broadcasting guide;

[0117] An emotion mapping module is configured to extract paragraph-level semantics and emotional features based on the serialized text stream, and perform dimension mapping on the semantics and emotional features to generate an emotion compensation parameter set;

[0118] A prosody scheduling module is configured to store hidden layer state features generated in a prosody prediction phase of a previous text paragraph according to a paragraph granularity based on the emotion compensation parameter set; and when performing prosody prediction on a current text paragraph, the corresponding hidden layer state features of the previous paragraph are introduced back and used to participate in the prosody prediction of the current paragraph to generate a cross-paragraph consistent prosody prediction signal.

[0119] An acoustic synthesis module is configured to perform cross-modal alignment and fusion of the prosody prediction signal and the emotion compensation parameter set to generate an acoustic feature prediction result, and convert the acoustic feature prediction result into a mel-frequency spectrum feature to generate a corresponding digital audio signal stream.

[0120] A distribution module is configured to perform timbre matching and packaging on the digital audio signal stream according to station attribute metadata, and push the digital audio signal stream to a playback entrance of a corresponding station cluster for intelligent broadcasting.

[0121] The embodiment also provides a computer device suitable for the station group article intelligent broadcasting method based on multi-modal emotion speech synthesis, which comprises a memory and a processor; the memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions to implement the station group article intelligent broadcasting method based on multi-modal emotion speech synthesis proposed in the above embodiment.

[0122] The computer device can be a terminal, which includes a processor, a memory, a communication interface, a display screen and an input device connected by a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is configured to perform wired or wireless communication with an external terminal. The wireless communication can be achieved by WIFI, an operator network, NFC (Near Field Communication) or other technologies. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0123] The embodiment also provides a storage medium having a computer program stored thereon, the program being executed by a processor to implement the method for intelligently broadcasting a station group article based on multi-modal sentiment speech synthesis according to the above embodiment. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk or an optical disk.

[0124] To sum up, the application builds a more delicate and controllable voice expression basis for station group article broadcasting by performing semantic enhancement and emotion dimension mapping on multi-source text content. And by introducing a backtracking storage mechanism of the hidden layer state of the pre-paragraph, the emotion evolution trend and the rhythm rhythm information are continuously modeled and transmitted across paragraphs, so as to maintain the consistency and smoothness of voice performance at the paragraph junction. The participation of this long-range context information enables the entire article to maintain stable speech rate changes, tone trends and emotional evolution characteristics during the broadcasting process, reducing the possible incoherent phenomenon of hearing at the paragraph switching, and making the synthesized voice closer to the natural broadcasting effect in terms of overall rhythm organization and narrative coherence. Therefore, the application effectively improves the voice naturalness and emotional consistency in the automatic broadcasting scene of large-scale long text, and enhances the auditory experience and user immersion in the station group information distribution process.

[0125] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit it. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the present application, which should be covered by the scope of the claims of the present application.

Claims

1. A station group article intelligent broadcasting method based on multi-modal emotional speech synthesis, characterized in that, The method comprises the following steps: obtaining multi-source text content in a distributed site cluster, stripping non-text labels, and performing semantic enhancement on the multi-source text content to generate a serialized text stream with broadcasting guidance; based on the serialized text stream, extracting paragraph-level semantic and emotional features, and performing dimension mapping on the semantic and emotional features to generate a set of emotional compensation parameters; based on the set of emotional compensation parameters, the hidden layer state features generated in the previous text paragraph prosody prediction stage are stored according to the paragraph granularity; when performing prosody prediction on the current text paragraph, the corresponding hidden layer state features of the previous paragraph are backtracked and introduced to participate in the prosody prediction of the current paragraph, generating a cross-paragraph consistent prosody prediction signal, which specifically includes: Before performing prosody prediction on the current text paragraph, based on the paragraph-level hidden layer state index relationship, the backtracking range is limited to the previous text paragraph adjacent to the current text paragraph position in the serialized text stream, and the corresponding paragraph-level hidden layer state features are located; the corresponding paragraph-level hidden layer state features of the previous text paragraph are used as context constraint information in prosody prediction, and are used together with the set of emotional compensation parameters to participate in the prosody control element prediction of the current text paragraph; based on the context constraint information and the set of emotional compensation parameters, the prosody control elements of the current text paragraph are continuously constrained and predicted to generate a prosody prediction signal that maintains the consistency of emotional and prosodic evolution between adjacent paragraphs; align and fuse the prosody prediction signal and the set of emotional compensation parameters across modalities to generate an acoustic feature prediction result, and convert the acoustic feature prediction result to a mel spectrum feature to generate a corresponding digital audio signal stream; According to the site attribute metadata, the digital audio signal stream is matched and packaged in terms of tone, and is pushed to the corresponding site cluster to perform intelligent broadcasting.

2. The station group article intelligent broadcasting method based on multi-modal emotional speech synthesis according to claim 1, characterized in that, The method for generating a serialized text stream with broadcasting guidance comprises: obtaining multi-source text content from a distributed site cluster, and removing scripts and format tags in the multi-source text content to extract pure text information; performing industry terminology and numerical annotation on the pure text information, and performing syntax analysis to determine pause points and stress points; insert the corresponding broadcasting control symbols in the text sequence, and format and package according to the broadcasting guidance protocol to generate a serialized text stream. 3.The station group article intelligent broadcasting method based on multi-modal emotional speech synthesis of claim 2, wherein, The method for generating a set of emotional compensation parameters comprises: According to the paragraph boundary information in the text, the serialized text stream is split to obtain a plurality of text paragraphs and extract the paragraph-level semantics corresponding to each text paragraph; based on the paragraph-level semantics, analyze the semantic content and emotional expression features of each text paragraph, identify the emotional tendency of the text paragraph, and determine the emotional category and emotional intensity parameter of the emotional expression strength based on the emotional tendency, forming the paragraph-level emotional features corresponding to each text paragraph; based on the paragraph-level semantic and emotional features, construct the mapping relationship between them and the acoustic performance of voice synthesis, and generate the corresponding acoustic control features accordingly; based on the acoustic control features, extract the corresponding prosody control elements, and generate a set of emotional compensation parameters by combining the prosody control elements.

4. The station group article intelligent broadcasting method based on multi-modal emotional speech synthesis of claim 3, wherein, The method for storing the hidden layer state features generated in the previous text paragraph prosody prediction stage according to the paragraph granularity comprises: Based on the emotion compensation parameter set, the prosody control elements of the text paragraphs are predicted according to the order of the text paragraphs in the serialized text stream and in combination with the corresponding broadcasting guide of the text paragraphs, and the prosody prediction result corresponding to each text paragraph is obtained. In the process of generating the prosody prediction result, the hidden layer state features related to the corresponding text paragraph are extracted to form paragraph-level hidden layer state features. The paragraph-level hidden layer state features are associated with the paragraph position of the corresponding text paragraph in the serialized text stream, and a paragraph-level hidden layer state index relationship is constructed. The hidden layer state features corresponding to the previous text paragraph are sequentially stored in paragraph granularity to form a paragraph-level hidden layer state set that can be called by the subsequent prosody prediction process. 5.The station group article intelligent broadcasting method based on multi-modal emotional speech synthesis of claim 1, wherein, The method for generating the acoustic feature prediction result includes: Based on the structural position of the text paragraph in the serialized text stream, the prosody prediction signal and the emotion compensation parameter set are synchronized and aligned in the time dimension and the paragraph dimension; The emotion intensity parameters and the acoustic control features in the emotion compensation parameter set are fused with the prosody control elements in the aligned prosody prediction signal at the feature level to construct a joint acoustic control feature representation; Based on the joint acoustic control feature representation, the basic acoustic parameters in the speech synthesis process are predicted to generate the corresponding acoustic feature prediction result.

6. The station group article intelligent broadcasting method based on multi-modal emotional speech synthesis of claim 5, wherein, The method for generating the corresponding digital audio signal stream includes: Based on the acoustic feature prediction result, the multi-dimensional acoustic control information required in the speech synthesis process is time-sequentially organized to form an acoustic feature sequence matching the text paragraph structure; According to the acoustic feature sequence, the mapping from the time-sequential acoustic feature space to the spectral representation space is performed to generate the mel spectrum feature; based on the mel spectrum feature, the acoustic reconstruction is performed to convert the spectral representation into a time-domain continuous waveform to generate a continuous digital audio signal stream.

7. The station group article intelligent broadcasting method based on multi-modal emotional speech synthesis of claim 6, wherein, The intelligent broadcasting includes: Through the site management interface, the site attribute metadata of the target site in the site cluster is collected; based on the site attribute metadata, the timbre matching processing is performed on the digital audio signal stream, and the digital audio signal stream after the timbre matching is format-encapsulated according to the playing interface specification corresponding to the target site to generate the broadcasting audio data adapted to the site; The broadcasting audio data is pushed to the playing entrance of the corresponding site cluster to trigger the automatic broadcasting process on the site side and perform the intelligent broadcasting of the article in the site cluster. 8.The station group article intelligent broadcasting method based on multi-modal emotional speech synthesis of claim 1, wherein, The method for continuously predicting the prosody control elements of the current text paragraph based on the context constraint information and the emotion compensation parameter set includes: When the prosody prediction of the current text paragraph is performed, the paragraph-level hidden layer state features corresponding to the previous text paragraph obtained by backtracking are introduced into the prosody prediction process of the current text paragraph as the context constraint information and participate in the recursive calculation as the initial state of the current text paragraph; In the recursive calculation process, the emotion compensation parameter set corresponding to the current text paragraph and the broadcasting guide information are introduced step by step according to the time step evolution order to impose a continuity constraint on the prosody control elements, generate a continuously changing prosody prediction value sequence step by step, and obtain a prosody prediction signal that maintains the continuity of the emotional evolution and the consistency of the prosody transition between adjacent text paragraphs.

9. The station group article intelligent broadcasting system based on multi-modal emotional speech synthesis, according to any one of claims 1-8, wherein, It includes: The text processing module is configured to acquire multi-source text content in the distributed site cluster, strip non-text labels, and perform semantic enhancement on the multi-source text content to generate a serialized text stream with broadcasting guidance; The sentiment mapping module is configured to extract paragraph-level semantic and emotional features based on the serialized text stream, and perform dimension mapping on the semantic and emotional features to generate a set of sentiment compensation parameters; The prosody scheduling module is configured to store hidden layer state features generated in the prosody prediction stage of a previous text paragraph according to a paragraph granularity based on the set of sentiment compensation parameters; when performing prosody prediction on a current text paragraph, the corresponding hidden layer state features of the previous paragraph are recalled and introduced to participate in the prosody prediction of the current paragraph, and a cross-paragraph consistent prosody prediction signal is generated; The acoustic synthesis module is configured to perform cross-modal alignment and fusion of the prosody prediction signal and the set of sentiment compensation parameters to generate an acoustic feature prediction result, convert the acoustic feature prediction result into a mel-frequency spectrum feature, and generate a corresponding digital audio signal stream; The distribution module is configured to perform timbre matching and packaging on the digital audio signal stream according to site attribute metadata, and push the digital audio signal stream to a corresponding site cluster for intelligent broadcasting.

Images