Station group article intelligent broadcasting system and method based on multi-mode emotional speech synthesis

Abstract

The invention discloses a station group article intelligent broadcasting system and method based on multi-mode emotional speech synthesis, and relates to the technical field of speech synthesis, and the method comprises the steps: extracting paragraph-level semantic and emotional characteristics, carrying out the dimension mapping of the semantic and emotional characteristics, and generating an emotional compensation parameter set; based on the emotion compensation parameter set, hidden layer state features generated in the preorder text paragraph rhythm prediction stage are stored according to paragraph granularity; when rhythm prediction is executed on a current text paragraph, backtracking and introducing corresponding preorder paragraph hidden layer state features, participating in rhythm prediction of the current paragraph, and generating a cross-paragraph consistent rhythm prediction signal; and performing cross-modal alignment fusion on the rhythm prediction signal and the emotion compensation parameter set to generate an acoustic feature prediction result, converting the acoustic feature prediction result into a Mel spectrum feature, and generating a corresponding digital audio signal stream, thereby improving the voice naturalness and emotion consistency of long text automatic broadcast.

Description

Technical Field

[0001] This invention relates to the field of speech synthesis technology, and in particular to an intelligent broadcasting system and method for articles on a website cluster based on multimodal emotional speech synthesis. Background Technology

[0002] In recent years, neural text-to-speech (TTS) technology has made significant progress driven by deep learning algorithms, especially end-to-end synthesis models based on the Transformer architecture, which can now achieve highly realistic speech generation. In distributed website cluster management scenarios, existing technologies generally employ a combination of streaming inference frameworks and pre-trained acoustic models. This involves extracting semantic representations from large-scale text corpora and using vocoders to achieve automated speech content production. By integrating attention mechanisms and multi-scale frequency prediction, this type of technology greatly improves the efficiency of digital dissemination of information across website clusters, providing a fundamental technical guarantee for content distribution on large-scale vertical websites.

[0003] However, when faced with the need for continuous broadcasting of long articles in a network of websites, existing speech synthesis technologies still have significant limitations in modeling the continuity of speech across paragraphs. Specifically, when generating speech segment by segment, existing methods fail to effectively utilize the speech state already formed in the previous segment to constrain the prosodic generation of the current segment, causing abrupt changes in emotional expression and intonation rhythm during segment switching. For example, when broadcasting multiple news texts continuously, the speech of the next segment does not continue the speech rate, pitch direction, and stress distribution already formed in the previous segment at the beginning position, but starts again with the default broadcast state, resulting in fluctuating speech rhythm and discontinuous emotional fluctuations. The overall broadcast effect lacks continuity and fails to meet the requirements for a natural and flowing listening experience. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a method for intelligent broadcasting of articles in a website group based on multimodal emotional speech synthesis to solve the problem of inconsistent emotional rhythm between paragraphs in the broadcasting of articles in a website group.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides a method for intelligent broadcasting of articles from website clusters based on multimodal emotional speech synthesis, comprising: The system acquires multi-source text content from a distributed site cluster, removes non-text tags, performs semantic enhancement on the multi-source text content, and generates a serialized text stream with broadcast instructions. Based on the serialized text stream, paragraph-level semantic and emotional features are extracted, and the semantic and emotional features are dimensionally mapped to generate a set of emotional compensation parameters. Based on the set of sentiment compensation parameters, the hidden state features generated in the prosody prediction stage of the preceding text paragraphs are stored at the paragraph level. When performing prosody prediction on the current text paragraph, the corresponding hidden state features of the preceding paragraphs are backtracked and introduced to participate in the prosody prediction of the current paragraph, generating a consistent prosody prediction signal across paragraphs. The prosody prediction signal and the emotion compensation parameter set are fused across modal modes to generate acoustic feature prediction results, which are then converted into Mel-spectral features to generate the corresponding digital audio signal stream. Based on the site attribute metadata, the digital audio signal stream is matched and encapsulated for timbre matching, and then pushed to the playback entry of the corresponding site cluster for intelligent playback.

[0007] Preferably, the method for generating a serialized text stream with broadcast instructions includes: Obtain multi-source text content from a distributed site cluster, and remove scripts and formatting tags from the multi-source text content to extract plain text information; The system adds industry-specific terminology and numerical phonetic transcription to plain text information and performs syntactic analysis to determine pause points and stress points. It then inserts corresponding broadcast control characters into the text sequence and formats and encapsulates the data according to the broadcast guide protocol to generate a serialized text stream.

[0008] Preferably, the method for generating the set of emotion compensation parameters includes: The serialized text stream is split based on the paragraph boundary information in the text to obtain multiple text paragraphs and extract the paragraph-level semantics corresponding to each text paragraph. Based on paragraph-level semantics, the semantic content and emotional expression features of each text paragraph are analyzed to identify the emotional tendency of the text paragraph. Based on the emotional tendency, the corresponding emotional category and emotional intensity parameters of emotional expression strength are determined to form paragraph-level emotional features corresponding to each text paragraph. Based on paragraph-level semantic and paragraph-level emotional features, a mapping relationship between them and the acoustic performance of speech synthesis is constructed, and corresponding acoustic control features are generated accordingly. Based on acoustic control features, corresponding prosodic control elements are extracted, and vector combinations of prosodic control elements are performed to generate a set of emotion compensation parameters.

[0009] Preferably, storing the hidden state features generated in the preceding text paragraph prosody prediction stage at the paragraph granularity includes: Based on the set of emotion compensation parameters, according to the order of text segments in the serialized text stream, and combined with the corresponding broadcast guidance of the text segments, the prosodic control elements of the text segments are predicted to obtain the prosodic prediction results for each text segment. In the process of generating prosody prediction results, hidden state features related to the corresponding text paragraphs are extracted to form paragraph-level hidden state features. The paragraph-level hidden state features are associated with the corresponding text paragraphs in the serialized text stream to construct a paragraph-level hidden state index relationship; and the hidden state features corresponding to the preceding text paragraphs are stored sequentially according to paragraph granularity to form a set of paragraph-level hidden states that can be called by the subsequent prosodic prediction process.

[0010] Preferably, the method for generating a consistent prosodic prediction signal across paragraphs includes: Before performing prosodic prediction on the current text paragraph, based on the paragraph-level hidden state index relationship, the backtracking range is limited to the preceding text paragraphs that are adjacent to the current text paragraph in the serialized text stream, and the corresponding paragraph-level hidden state features are located. The paragraph-level hidden state features corresponding to the preceding text paragraph are used as contextual constraint information in prosodic prediction, and together with the sentiment compensation parameter set, they participate in the prediction of the prosodic control elements of the current text paragraph. Based on contextual constraint information and sentiment compensation parameter set, the prosodic control elements of the current text paragraph are predicted with continuity constraints, generating a prosodic prediction signal that maintains the consistency of sentiment and prosodic evolution between adjacent paragraphs.

[0011] Preferably, the method for generating acoustic feature prediction results includes: Based on the structural position of text paragraphs in the serialized text stream, the prosodic prediction signal and the sentiment compensation parameter set are synchronously aligned in the time dimension and paragraph dimension. Feature-level fusion is performed on the prosodic control elements in the aligned prosodic prediction signal and the emotional intensity parameters and acoustic control features in the emotional compensation parameter set to construct a joint acoustic control feature representation; Based on joint acoustic control feature representation, the basic acoustic parameters in the speech synthesis process are predicted, and the corresponding acoustic feature prediction results are generated.

[0012] Preferably, the method for generating the corresponding digital audio signal stream includes: Based on the acoustic feature prediction results, the multi-dimensional acoustic control information required in the speech synthesis process is organized temporally to form an acoustic feature sequence that matches the structure of the text paragraph. Based on the acoustic feature sequence, a mapping from the temporal acoustic feature space to the spectral representation space is performed to generate Mel spectral features; based on the Mel spectral features, acoustic reconstruction is performed to convert the spectral representation into a continuous waveform in the time domain, generating a continuous digital audio signal stream.

[0013] Preferably, the intelligent broadcasting includes: Collect site attribute metadata of the target site in the site group through the site management interface; perform timbre matching processing on the digital audio signal stream based on the site attribute metadata, and encapsulate the digital audio signal stream with completed timbre matching according to the playback interface specification of the target site to generate broadcast audio data adapted to the site. The audio data is pushed to the playback entry of the corresponding site cluster, triggering the automatic broadcast process on the site side to intelligently broadcast articles from the site cluster.

[0014] Preferably, the method for predicting the continuity constraints of prosodic control elements of the current text paragraph based on contextual constraint information and a set of sentiment compensation parameters includes: When performing prosodic prediction on the current text paragraph, the paragraph-level hidden state features corresponding to the previous text paragraph obtained by backtracking are used as contextual constraint information, introduced into the prosodic prediction process of the current text paragraph, and used as the initial state of the current text paragraph in the recursive calculation. During the recursive calculation process, the set of emotional compensation parameters and broadcast guidance information corresponding to the current text segment are introduced step by step according to the time step evolution order. Continuity constraints are imposed on the prosodic control elements, and a continuously changing prosodic prediction numerical sequence is generated step by step to obtain a prosodic prediction signal that maintains the continuity of emotional evolution and the consistency of prosodic transition between adjacent text segments.

[0015] Secondly, this invention provides an intelligent article broadcasting system for website clusters based on multimodal emotional speech synthesis, including: The text processing module is used to acquire multi-source text content from a distributed site cluster, remove non-text tags, and perform semantic enhancement on the multi-source text content to generate a serialized text stream with broadcast instructions. The sentiment mapping module is used to extract paragraph-level semantic and sentiment features based on serialized text streams, and to perform dimensional mapping on semantic and sentiment features to generate a set of sentiment compensation parameters. The prosody scheduling module is used to store the hidden state features generated in the prosody prediction stage of the preceding text paragraphs at the paragraph level based on the sentiment compensation parameter set; when performing prosody prediction on the current text paragraph, it backtracks and introduces the corresponding hidden state features of the preceding paragraphs to participate in the prosody prediction of the current paragraph, generating a consistent prosody prediction signal across paragraphs. The acoustic synthesis module is used to perform cross-modal alignment and fusion of the prosody prediction signal and the emotion compensation parameter set to generate acoustic feature prediction results, and convert the acoustic feature prediction results into Mel spectrum features to generate the corresponding digital audio signal stream; The distribution module is used to perform timbre matching and encapsulation of digital audio signal streams based on site attribute metadata, and push them to the playback entry of the corresponding site cluster for intelligent playback.

[0016] The beneficial effects of this invention are as follows: By semantically enhancing multi-source text content and performing sentiment dimension mapping, a more delicate and controllable foundation for speech expression is constructed for the broadcasting of articles across website clusters. Furthermore, by introducing a retrospective storage mechanism for the hidden states of preceding paragraphs, the trend of emotional evolution and prosodic rhythm information are continuously modeled and transmitted across paragraphs, thus maintaining consistency and smoothness in speech performance at paragraph transitions. The participation of this long-range contextual information enables the entire article to maintain stable changes in speech rate, intonation, and emotional evolution characteristics during broadcasting, reducing potential auditory discontinuities during paragraph transitions and making the synthesized speech more closely resemble natural broadcasting in terms of overall rhythmic organization and narrative coherence. Therefore, this invention effectively improves the naturalness and emotional consistency of speech in large-scale long text automatic broadcasting scenarios, enhancing the auditory experience and user immersion during the distribution of information across website clusters. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of the intelligent broadcasting method for website group articles based on multimodal emotional speech synthesis in this invention; Figure 2 This is a schematic diagram of the intelligent article broadcasting system for website groups based on multimodal emotional speech synthesis in this invention; Figure 3 This is a flowchart illustrating the generation of the emotional compensation parameter set in this invention; Figure 4 This is a flowchart illustrating the generation of a consistent prosodic prediction signal across paragraphs in this invention. Detailed Implementation

[0019] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0020] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0021] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0022] Reference Figure 1 , Figure 2 , Figure 3 and Figure 4 This is one embodiment of the present invention, which provides a method for intelligent broadcasting of articles in a website group based on multimodal emotional speech synthesis, including the following steps: Methods for generating serialized text streams with broadcast instructions include: The system retrieves multi-source text content from a distributed site cluster and removes scripts and formatting tags from the multi-source text content to extract plain text information.

[0023] Specifically, it initiates a Hypertext Transfer Protocol (HTTP) request to a distributed site cluster via a network communication interface to locate the source data storage location of each site in the distributed site cluster, and obtains multi-source text content containing Hypertext Markup Language (HTML) tags, stylesheet scripts, and web page logic scripts. Then, it uses character pattern matching rules containing specific escape sequences and wildcard combinations to perform line-by-line scanning and feature recognition of the multi-source text content, locating characters in the multi-source text content that resemble "<script.?> .?"as well as"<style.?> The specific start and end symbols of ".?", as well as the script logic code sandwiched between the symbols, are used to remove the identified script and format tags through character offset positioning and string replacement operations, retaining only the original character sequences that carry the actual semantics in the multi-source text content, thus completing the extraction of plain text information.

[0024] The system adds industry-specific terminology and numerical phonetic transcription to plain text information and performs syntactic analysis to determine pause points and stress points. It then inserts corresponding broadcast control characters into the text sequence and formats and encapsulates the data according to the broadcast guide protocol to generate a serialized text stream.

[0025] Specifically, the plain text information is standardized sentence by sentence. By performing pattern matching and context feature recognition on industry-specific vocabulary and numerical sequences in the text, it is mapped to the corresponding phonetic representation. Then, syntactic analysis is performed on the phonetic text content. The position of pauses within sentences is determined by the distribution of punctuation marks and semantic boundary rules. Combined with word order, dependency relationships, and part-of-speech tagging results, the main sentences and modifiers are distinguished to determine the stress points for speech emphasis. According to the character offset of the pause and stress points in the text, the corresponding broadcast control characters are inserted into the text sequence. By arranging the text characters, phonetic representations, and corresponding broadcast control characters according to the temporal topology, they are encapsulated into a structured data message containing header identifiers, semantic payloads, and control information elements, generating a serialized text stream with clear broadcast structure guidance.

[0026] Methods for generating a set of emotion compensation parameters include: The serialized text stream is split based on paragraph boundary information in the text to obtain multiple text paragraphs, and the paragraph-level semantics corresponding to each text paragraph are extracted.

[0027] It should be noted that binary streaming parsing is performed on the serialized text stream, traversing and reading the character data and broadcast control characters contained in the serialized text stream, identifying specific control codes marked as newline control characters or paragraph end marks, and confirming the position of the specific control codes in the serialized text stream as paragraph boundary information; based on the byte offset index of the paragraph boundary information in the serialized text stream, data slicing operation is performed on the serialized text stream to extract the data segments containing header identifiers, semantic payloads and control cells between two adjacent paragraph boundary information, and construct mutually independent text segments; The semantic payload content is extracted from the text paragraphs and transformed into a dense vector sequence using a word embedding lookup table. The dense vector sequence is then multiplied by both the query linear transformation matrix and the key linear transformation matrix to obtain the query matrix. AND key matrix The association weights of dense vector sequences in the global context are calculated using a multi-head self-attention mechanism, and the corresponding hidden state sequences of the context are generated based on these association weights; the expression is: ,in, It is a correlation weight matrix. It is a normalized exponential function. It is a scaling factor, derived from the dimension value of a single key vector in the key matrix. It is the matrix transpose operator. It is an index of the key matrix dimension.

[0028] A mean pooling operation is performed on the context hidden state sequence along the time step dimension to aggregate the variable-length context hidden state sequence into a fixed-dimensional numerical feature vector, and the numerical feature vector is identified as the paragraph-level semantics corresponding to the text paragraph.

[0029] Based on paragraph-level semantics, the semantic content and emotional expression features of each text paragraph are analyzed to identify the emotional tendency of the text paragraph. Based on the emotional tendency, the corresponding emotional category and emotional intensity parameters of emotional expression strength are determined to form paragraph-level emotional features corresponding to each text paragraph.

[0030] It should be noted that the paragraph-level semantics corresponding to each text paragraph are multiplied with a preset emotion weight matrix, and the bias vector is accumulated to achieve a linear mapping from the semantic feature space to the emotion label space. Then, a normalized exponential function is applied to the mapping result to generate probability distribution scores for multiple emotion dimensions. Based on the probability distribution scores, the emotion dimension corresponding to the highest score is identified as the emotion category, and the original value of this highest score is extracted and mapped to an intensity range of 0 to 1 to obtain the emotion intensity parameter. Finally, the emotion category and the emotion intensity parameter are combined in a field-based manner to generate the paragraph-level emotion features corresponding to each text paragraph. Example: The preset emotion label space includes dimensions of calm, happiness, anger, sadness, and fear. It should be noted that the preset process for the emotion weight matrix is ​​as follows: obtain the feature dimension values ​​of the numerical feature vectors corresponding to paragraph-level semantics, set them as the row dimension of the emotion weight matrix, obtain the total number of classification labels in the preset emotion label space, set them as the column dimension of the emotion weight matrix, construct a two-dimensional matrix structure based on the row and column dimensions, and establish the weight values ​​inside the matrix structure as linear transformation coefficients used to project the semantic feature space onto the emotion label space.

[0031] Based on paragraph-level semantic and paragraph-level emotional features, a mapping relationship between them and the acoustic performance of speech synthesis is constructed, and corresponding acoustic control features are generated accordingly.

[0032] Specifically, paragraph-level semantics and paragraph-level emotional features are concatenated along the feature dimension to construct a joint feature vector that integrates semantic background and emotional attributes. The joint feature vector is then input into the acoustic mapping linear transformation matrix to perform matrix multiplication. By establishing a mapping relationship between the joint feature vector and physical parameters such as fundamental frequency, duration, and energy, acoustic control features that characterize the basic pronunciation characteristics in speech synthesis are generated. It should be noted that the preset process for the acoustic mapping linear transformation matrix is ​​as follows: obtain the concatenation dimension value of the joint feature vector as the row dimension, obtain the total amount of physical parameters contained in the acoustic control features as the column dimension, and construct the corresponding transformation weight matrix, which is used to transform the semantic emotion features of the text dimension into the parameter expression of the acoustic dimension.

[0033] Based on acoustic control features, corresponding prosodic control elements are extracted, and vector combinations of prosodic control elements are performed to generate a set of emotion compensation parameters.

[0034] It should be noted that numerical components that can reflect speech rate rhythm, pitch fluctuations and pause patterns are extracted from acoustic control features and identified as prosodic control elements; vector combination operations are performed on prosodic control elements to encapsulate and form a set of emotion compensation parameters containing multi-dimensional pronunciation adjustment information.

[0035] The hidden state features generated during the prosodic prediction stage of the preceding text paragraphs are stored at the paragraph level, including: Based on the set of sentiment compensation parameters, the prosodic control elements of the text segments are predicted according to their order in the serialized text stream and in conjunction with the corresponding broadcast guidelines, so as to obtain the prosodic prediction results for each text segment.

[0036] Specifically, the process involves obtaining the order of text segments in the serialized text stream, aligning the set of sentiment compensation parameters corresponding to each text segment with the broadcast control characters in the broadcast guidelines along the time dimension to form a data sequence arranged by time steps; and then performing a numerical mapping calculation, including historical state feedback, on the data at each time step according to the time order of the data sequence. The expression is: ,in It is the state information used to characterize the evolution of speech prosody at the current time step. It is the recursive calculation rule for performing numerical mapping and nonlinear transformation. The input data consists of the emotion compensation parameters at the current time step and the broadcast guidance information. It is the state information retained from the previous time step; The recursive calculation rule is specifically reflected in the following: a recursive relationship is established between the input data of the current time step and the state information retained in the previous time step. Through continuous accumulation and nonlinear transformation, a numerical sequence reflecting the changes in speech duration, fundamental frequency evolution trajectory and energy envelope is generated step by step, and the numerical sequence is confirmed as the prosodic prediction result of the corresponding text segment. It should be noted that the recursive calculation process incorporates the state information of the previous time step at each time step, so that the numerical calculation result of the current time step is simultaneously constrained by the current set of emotion compensation parameters and the historical speech state, in order to characterize the continuous change of prosodic control elements over time. Before determining this recursive calculation relationship, a sample dataset consisting of multiple sets of text segments and their corresponding standard speech broadcast data is constructed. Among them, the standard speech broadcast data is processed by acoustic analysis to extract the true duration parameters, fundamental frequency trajectory parameters, and energy envelope parameters. The set of emotion compensation parameters corresponding to the text segments and the broadcast index are used to determine the recursive calculation relationship. Using the extracted real speech prosodic parameters as input and the numerical mapping coefficients in the recursive calculation as reference, the mean square deviation between the generated prosodic prediction result and the real speech prosodic parameters is gradually reduced through multiple rounds of iterative correction, until the prediction error converges to a preset threshold, for example, no greater than 0.001. After the numerical mapping relationship is determined, the speech rate, fundamental frequency trajectory and energy distribution in the prosodic prediction result are weighted and adjusted according to the emotion category and emotion intensity parameters in the paragraph-level emotion features, so that the generated prosodic prediction result reflects the emotional expression logic of the text at the acoustic level.

[0037] In the process of generating prosodic prediction results, hidden state features related to the corresponding text paragraphs are extracted to form paragraph-level hidden state features.

[0038] It should be noted that during the recursive calculation step by step, after the data processing of the last time step of the current text paragraph is completed, the state information used to participate in the calculation of the next time step is saved, and this state information is identified as the paragraph-level hidden state feature corresponding to the current text paragraph; Example: The paragraph-level hidden state feature is a feature vector containing 256-dimensional values.

[0039] The paragraph-level hidden state features are associated with the corresponding text paragraphs in the serialized text stream to construct a paragraph-level hidden state index relationship; and the hidden state features corresponding to the preceding text paragraphs are stored sequentially according to paragraph granularity to form a set of paragraph-level hidden states that can be called by the subsequent prosodic prediction process.

[0040] Specifically, the physical segment number of the current text segment in the serialized text stream is obtained, and the physical segment number is mapped and associated with the corresponding segment-level hidden state features in the key-value pair storage space to form a segment-level hidden state index relationship. According to the ascending order of the physical segment number, the segment-level hidden state features obtained after processing each text segment are sequentially written into the circular cache queue to construct a segment-level hidden state set for storing the speech state information of the preceding text segment, which can be called by the prosodic prediction process of the subsequent text segment.

[0041] Methods for generating consistent prosodic prediction signals across paragraphs include: Before performing prosodic prediction on the current text paragraph, the backtracking range is limited to the preceding text paragraphs adjacent to the current text paragraph in the serialized text stream, based on the paragraph-level hidden state index relationship, and the corresponding paragraph-level hidden state features are located.

[0042] Specifically, before performing prosody prediction on the current text segment, the paragraph-level hidden state index relationship is retrieved to determine one or more text segments in the serialized text stream whose physical segment numbers are adjacent to and precede the current text segment; based on the determined physical segment numbers, the corresponding paragraph-level hidden state features are extracted from the circular cache queue, and the extracted paragraph-level hidden state features are confirmed as contextual constraint information used to constrain the prosodic evolution of the current segment.

[0043] The paragraph-level hidden state features corresponding to the preceding text paragraph are used as contextual constraint information in prosodic prediction, and together with the sentiment compensation parameter set, they participate in the prediction of the prosodic control elements of the current text paragraph.

[0044] It should be noted that when performing prosodic control element prediction on the current text paragraph, the paragraph-level hidden state features are used as contextual constraint information to characterize the prosodic evolution state of the preceding speech, participating in the initial state setting of the current text paragraph prosodic prediction, so that the semantic and emotional evolution trends formed in the preceding text paragraph can be continuously transmitted during the paragraph switching process; the set of emotional compensation parameters corresponding to the current text paragraph is introduced into the prosodic prediction process in the order of time steps, as a regulating factor to drive the prosodic change amplitude and trend within the current paragraph, so that the prosodic control elements are simultaneously constrained by the historical prosodic state and the emotional expression needs of the current paragraph during the time evolution process.

[0045] Based on contextual constraint information and sentiment compensation parameter set, the prosodic control elements of the current text paragraph are predicted with continuity constraints, generating a prosodic prediction signal that maintains the consistency of sentiment and prosodic evolution between adjacent paragraphs.

[0046] Methods for generating acoustic feature prediction results include: Based on the structural position of text paragraphs in the serialized text stream, the prosodic prediction signal and the sentiment compensation parameter set are synchronized and aligned in the time dimension and paragraph dimension.

[0047] Specifically, the process involves reading the duration prediction values ​​contained in the prosody prediction signal to determine the total length of the acoustic frame sequence corresponding to the current text segment during speech synthesis. Based on the structural position number of the text segment in the serialized text stream, the sentiment compensation parameter set corresponding to that text segment is retrieved from the cache. A vector expansion operation is performed, temporally copying and expanding the segment-level sentiment compensation parameter set according to the total length of the acoustic frame sequence, ensuring that it strictly matches the number of frames contained in the prosody prediction signal in the time dimension. This completes the synchronization and alignment of the prosody prediction signal and the sentiment compensation parameter set in both the time and segment dimensions. Example: If the total length of the acoustic frame sequence is 500 frames, a segment-level vector is copied 500 times to form the sequence.

[0048] We perform feature-level fusion of prosodic control elements in the aligned prosodic prediction signal with emotional intensity parameters and acoustic control features in the emotional compensation parameter set to construct a joint acoustic control feature representation.

[0049] It should be noted that the prosodic control elements (such as fundamental frequency, duration, and energy values) contained in the aligned prosodic prediction signal are concatenated at the dimensional level with the padded set of emotion compensation parameters (including emotion intensity and acoustic control values) to construct a composite numerical vector containing multiple acoustic information. A weighted mapping is then performed on this composite numerical vector. By weighted summation and nonlinear transformation of values ​​from different sources, the dimensional differences between features are eliminated and their intrinsic correlation information is fused, constructing a joint acoustic control feature representation with global acoustic guidance.

[0050] Based on joint acoustic control feature representation, the basic acoustic parameters in the speech synthesis process are predicted, and the corresponding acoustic feature prediction results are generated.

[0051] It should be noted that, following the temporal order of the acoustic frame sequence, the composite numerical vector corresponding to the current acoustic frame in the joint acoustic control feature representation is read frame by frame. Linear mapping and nonlinear transformation calculations are then performed on the composite numerical vector to progressively convert the prosodic control elements, emotional intensity parameters, and acoustic control features contained therein into numerical expressions consistent with the physical parameters of speech synthesis. During the conversion process, dimensional rearrangement and weighted summation operations are performed on the composite numerical vector to generate an intermediate acoustic mapping vector. Subsequently, nonlinear compression and numerical smoothing are performed on the numerical values ​​of each dimension in the intermediate acoustic mapping vector to characterize the correspondence between the joint acoustic control feature representation and fundamental frequency changes, spectral envelope morphology, and aperiodic component distribution. By repeating the above mapping calculation for each acoustic frame, the fundamental frequency value, spectral envelope parameter, and aperiodic component parameter corresponding to each acoustic frame are output frame by frame. The basic acoustic parameters of each acoustic frame are then combined in temporal order to generate acoustic feature prediction results that are consistent with the text paragraph structure, prosodic prediction signal, and emotional evolution process in the temporal dimension.

[0052] Methods for generating the corresponding digital audio signal stream include: Based on the acoustic feature prediction results, the multi-dimensional acoustic control information required in the speech synthesis process is organized temporally to form an acoustic feature sequence that matches the structure of the text paragraph.

[0053] It should be noted that the acoustic feature prediction results are read segment by segment according to the order of the text segments in the serialized text stream; the acoustic feature prediction results include the fundamental frequency value, spectral envelope parameter and non-periodic component parameter arranged in the order of acoustic frames. The fundamental frequency value, spectral envelope parameter, and aperiodic component parameter at the same acoustic frame time point are combined in a field-based manner to form the acoustic feature unit of the corresponding acoustic frame; according to the time order of the acoustic frames, the acoustic feature units are arranged in sequence to construct a complete acoustic feature sequence.

[0054] Based on the acoustic feature sequence, a mapping from the temporal acoustic feature space to the spectral representation space is performed to generate Mel spectral features; based on the Mel spectral features, acoustic reconstruction is performed to convert the spectral representation into a continuous waveform in the time domain, generating a continuous digital audio signal stream.

[0055] It should be noted that the acoustic feature units in the acoustic feature sequence are read frame by frame in chronological order of the acoustic frames. The fundamental frequency value is used to determine the reference position of the spectral energy distribution, the spectral envelope parameter is used to describe the energy attenuation pattern of each frequency component, and the spectral energy is compensated for random perturbation by combining the aperiodic component parameter. The fundamental frequency value, spectral envelope parameter, and aperiodic component parameter are converted into the corresponding short-time spectral energy distribution. Then, the short-time spectral energy distribution is mapped to the Mel frequency scale, and the spectral energy in the Mel frequency range is weighted and aggregated to generate the Mel spectral feature sequence. Perform inverse spectrum transformation frame by frame according to the time sequence of the Mel spectrum feature sequence to reconstruct the short-time amplitude spectrum. Combine the fundamental frequency value and non-periodic component parameters of the corresponding acoustic frame to perform phase estimation and random disturbance compensation to generate acoustic frame-level time-domain waveform segments. Perform overlapping, addition and smooth splicing on each time-domain waveform segment in time sequence to form a continuous digital audio signal stream segment. Then, splice the segments sequentially according to the arrangement order of each text segment in the serialized text stream. Insert silence intervals of corresponding duration between adjacent segments according to the pause control characters in the broadcast instructions to obtain the digital audio signal stream.

[0056] Intelligent broadcasting includes: The system collects site attribute metadata of the target site in the site group through the site management interface; based on the site attribute metadata, it performs timbre matching processing on the digital audio signal stream, and encapsulates the timbre-matched digital audio signal stream according to the playback interface specification corresponding to the target site to generate site-adapted broadcast audio data.

[0057] It should be noted that when collecting site attribute metadata, the site identifier information, terminal type identifier information, playback device type information, and playback interface specification information corresponding to the target site are read respectively. Based on the playback interface specification information, the required sampling rate, number of channels, and encoding format of the audio data are determined. According to the terminal type identifier information and playback device type information, the spectral distribution, amplitude range, and overall volume of the digital audio signal stream are adjusted to complete the timbre matching process. After the timbre matching process is completed, the digital audio signal stream is format-encapsulated according to the determined sampling rate, number of channels, and encoding format to obtain the broadcast audio data adapted to the site.

[0058] The audio data is pushed to the playback entry of the corresponding site cluster, triggering the automatic broadcast process on the site side to intelligently broadcast articles from the site cluster.

[0059] It should be noted that, according to the site identification information of the target site in the site group, the site-adapted broadcast audio data is sent to the corresponding playback entry address of the target site through the network communication interface; after the broadcast audio data is successfully written to the playback entry, a playback trigger command is sent to the target site, so that the target site can perform automatic playback operation according to the time structure and broadcast guidance information in the broadcast audio data, thereby completing the intelligent broadcast of the articles in the site group on the target site side.

[0060] Methods for predicting the continuity constraints of prosodic control elements in the current text paragraph based on contextual constraint information and sentiment compensation parameter sets include: When performing prosodic prediction on the current text paragraph, the paragraph-level hidden state features corresponding to the previous text paragraph obtained by backtracking are used as contextual constraint information, introduced into the prosodic prediction process of the current text paragraph, and used as the initial state of the current text paragraph in the recursive calculation.

[0061] It should be noted that by using the paragraph-level hidden state features retained by the preceding text paragraph when completing prosodic prediction as the starting constraint for the recursive calculation of the current text paragraph, the semantic and emotional state information contained in the preceding text paragraph can be continuously transmitted during paragraph switching, avoiding the interruption of state information during prosodic prediction, thus providing a stable contextual constraint basis for the prosodic prediction of the current text paragraph.

[0062] During the recursive calculation process, the set of emotional compensation parameters and broadcast guidance information corresponding to the current text segment are introduced step by step according to the time step evolution order. Continuity constraints are imposed on the prosodic control elements, and a continuously changing prosodic prediction numerical sequence is generated step by step to obtain a prosodic prediction signal that maintains the continuity of emotional evolution and the consistency of prosodic transition between adjacent text segments.

[0063] Specifically, during the time-step recursion process, the constraint effect of contextual constraint information on the temporal evolution direction of prosodic prediction, combined with the driving effect of the emotion compensation parameter set on the prosodic adjustment amplitude and change trend, enables the generated prosodic prediction values ​​to exhibit smooth transition characteristics in dimensions such as pitch connection, speech rate rhythm and stress distribution, so that the synthesized speech maintains a natural and coherent listening effect in long text continuous broadcast scenarios.

[0064] This embodiment also provides an intelligent article broadcasting system for website clusters based on multimodal emotional speech synthesis, including: The text processing module is used to acquire multi-source text content from a distributed site cluster, remove non-text tags, and perform semantic enhancement on the multi-source text content to generate a serialized text stream with broadcast instructions. The sentiment mapping module is used to extract paragraph-level semantic and sentiment features based on serialized text streams, and to perform dimensional mapping on semantic and sentiment features to generate a set of sentiment compensation parameters. The prosody scheduling module is used to store the hidden state features generated in the prosody prediction stage of the preceding text paragraphs at the paragraph level based on the sentiment compensation parameter set; when performing prosody prediction on the current text paragraph, it backtracks and introduces the corresponding hidden state features of the preceding paragraphs to participate in the prosody prediction of the current paragraph, generating a consistent prosody prediction signal across paragraphs. The acoustic synthesis module is used to perform cross-modal alignment and fusion of the prosody prediction signal and the emotion compensation parameter set to generate acoustic feature prediction results, and convert the acoustic feature prediction results into Mel spectrum features to generate the corresponding digital audio signal stream; The distribution module is used to perform timbre matching and encapsulation of digital audio signal streams based on site attribute metadata, and push them to the playback entry of the corresponding site cluster for intelligent playback.

[0065] This embodiment also provides a computer device applicable to the intelligent broadcasting method for website group articles based on multimodal emotional speech synthesis, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the intelligent broadcasting method for website group articles based on multimodal emotional speech synthesis as proposed in the above embodiment.

[0066] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0067] This embodiment also provides a storage medium storing a computer program. When executed by a processor, the program implements the intelligent broadcasting method for website group articles based on multimodal emotional speech synthesis as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0068] In summary, this invention provides a more nuanced and controllable foundation for speech expression in website group article broadcasting by: semantically enhancing multi-source text content and performing sentiment dimension mapping; and by introducing a retrospective storage mechanism for the hidden state of preceding paragraphs, enabling continuous modeling and transmission of emotional evolution trends and prosodic rhythm information across paragraphs, thus maintaining consistency and smoothness in speech performance at paragraph transitions. The involvement of this long-range contextual information allows the entire article to maintain stable speech rate variations, intonation, and emotional evolution characteristics during broadcasting, reducing potential auditory discontinuities during paragraph transitions and making the synthesized speech more closely resemble natural broadcasting in terms of overall rhythmic organization and narrative coherence. Therefore, this invention effectively improves the naturalness and emotional consistency of speech in large-scale long text automatic broadcasting scenarios, enhancing the auditory experience and user immersion during website group information distribution.

[0069] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A station group article intelligent broadcasting method based on multi-modal emotional speech synthesis, characterized in that, The method comprises the following steps: obtaining multi-source text content in a distributed site cluster, stripping non-text labels, and performing semantic enhancement on the multi-source text content to generate a serialized text stream with broadcasting guidance; based on the serialized text stream, extracting paragraph-level semantics and emotional features, and mapping the semantics and emotional features to generate a set of emotional compensation parameters; based on the set of emotional compensation parameters, storing the hidden layer state features generated in the previous text paragraph rhythm prediction stage according to the paragraph granularity; when performing rhythm prediction on the current text paragraph, the corresponding previous paragraph hidden layer state features are introduced to participate in the rhythm prediction of the current paragraph, and a cross-paragraph consistent rhythm prediction signal is generated; aligning and fusing the rhythm prediction signal and the set of emotional compensation parameters in the cross-modal mode to generate an acoustic feature prediction result, and converting the acoustic feature prediction result into a mel spectrum feature to generate a corresponding digital audio signal stream; According to the site attribute metadata, the digital audio signal stream is matched and packaged in terms of tone, and is pushed to the corresponding site cluster to perform intelligent broadcasting. 2.The station group article intelligent broadcasting method based on multi-modal emotional speech synthesis of claim 1, wherein, The method for generating a serialized text stream with broadcasting guidance comprises: obtaining multi-source text content from a distributed site cluster, and removing scripts and format tags in the multi-source text content to extract pure text information; performing industry terminology and numerical annotation on the pure text information, and performing syntax analysis to determine the pause points and stress points; insert the corresponding broadcasting control symbols in the text sequence, and format and package according to the broadcasting guidance protocol to generate a serialized text stream. 3.The station group article intelligent broadcasting method based on multi-modal emotional speech synthesis of claim 2, wherein, The method for generating a set of emotional compensation parameters comprises: According to the paragraph boundary information in the text, the serialized text stream is split to obtain a plurality of text paragraphs and extract the corresponding paragraph-level semantics of each text paragraph; Based on the paragraph-level semantics, the semantic content and emotional expression features of each text paragraph are analyzed, the emotional tendency of the text paragraph is identified, and the corresponding emotional category and emotional intensity parameter of the emotional expression strength are determined based on the emotional tendency, forming the paragraph-level emotional features corresponding to each text paragraph; Based on the paragraph-level semantics and paragraph-level emotional features, the mapping relationship between them and the acoustic performance of voice synthesis is constructed, and the corresponding acoustic control features are generated accordingly; Based on the acoustic control features, the corresponding rhythm control elements are extracted, and the rhythm control elements are combined to generate a set of emotional compensation parameters.

4. The station group article intelligent broadcasting method based on multi-modal emotional speech synthesis of claim 3, wherein, The method for storing the hidden layer state features generated in the previous text paragraph rhythm prediction stage according to the paragraph granularity comprises: Based on the set of emotional compensation parameters, the rhythm control elements of the text paragraph are predicted according to the order of the text paragraph in the serialized text stream, combined with the broadcasting guidance corresponding to the text paragraph, to obtain the rhythm prediction result corresponding to each text paragraph; In the process of generating the rhythm prediction result, the hidden layer state features related to the corresponding text paragraph are extracted to form the paragraph-level hidden layer state features; The paragraph-level hidden layer state features are associated with the paragraph positions of the corresponding text paragraphs in the serialized text stream, and an index relationship of the paragraph-level hidden layer states is constructed. The hidden layer state features corresponding to the previous text paragraphs are sequentially stored in the paragraph granularity, and a paragraph-level hidden layer state set that can be called by a subsequent prosody prediction process is formed.

5. The station group article intelligent broadcasting method based on multi-modal emotional speech synthesis according to claim 4, characterized in that, The method for generating the prosody prediction signal consistent across paragraphs includes: Before performing prosody prediction on the current text paragraph, based on the paragraph-level hidden layer state index relationship, the backtracking range is limited to the previous text paragraphs adjacent to the position of the current text paragraph in the serialized text stream, and the corresponding paragraph-level hidden layer state features are located; The paragraph-level hidden layer state features corresponding to the previous text paragraphs are used as context constraint information in prosody prediction, and are used together with the set of emotional compensation parameters to predict the prosody control elements of the current text paragraph; Based on the context constraint information and the set of emotional compensation parameters, the prosody control elements of the current text paragraph are continuously predicted to generate a prosody prediction signal that maintains the consistency of the evolution of emotion and prosody between adjacent paragraphs. 6.The station group article intelligent broadcasting method based on multi-modal emotional speech synthesis of claim 5, wherein, The method for generating the acoustic feature prediction result includes: Based on the structural position of the text paragraph in the serialized text stream, the prosody prediction signal and the set of emotional compensation parameters are synchronized and aligned in the time dimension and the paragraph dimension; The prosody control elements in the aligned prosody prediction signal and the emotional intensity parameters and acoustic control features in the set of emotional compensation parameters are fused at the feature level to construct a joint acoustic control feature representation; Based on the joint acoustic control feature representation, the basic acoustic parameters in the speech synthesis process are predicted to generate the corresponding acoustic feature prediction result.

7. The station group article intelligent broadcasting method based on multi-modal emotional speech synthesis of claim 6, wherein, The method for generating the corresponding digital audio signal stream includes: Based on the acoustic feature prediction result, the multi-dimensional acoustic control information required in the speech synthesis process is time-sequentially organized to form an acoustic feature sequence matching the text paragraph structure; According to the acoustic feature sequence, mapping from the time-sequential acoustic feature space to the spectral representation space is performed to generate a mel-frequency spectrum feature; based on the mel-frequency spectrum feature, acoustic reconstruction is performed to convert the spectral representation into a time-domain continuous waveform to generate a continuous digital audio signal stream. 8.The station group article intelligent broadcasting method based on multi-modal emotional speech synthesis of claim 7, wherein, The method for performing intelligent broadcasting includes: Through the site management interface, the site attribute metadata of the target site in the site cluster is collected; based on the site attribute metadata, timbre matching processing is performed on the digital audio signal stream, and the digital audio signal stream after timbre matching is format-encapsulated according to the playback interface specification corresponding to the target site to generate site-adapted broadcast audio data; The broadcast audio data is pushed to the playback entrance of the corresponding site cluster to trigger the automatic broadcast process on the site side and perform intelligent broadcasting of the site cluster article. 9.The station group article intelligent broadcasting method based on multi-modal emotional speech synthesis of claim 5, wherein, The method for performing continuous constraint prediction on the prosody control elements of the current text paragraph based on the context constraint information and the set of emotional compensation parameters includes: When performing prosody prediction on the current text paragraph, the paragraph-level hidden layer state features corresponding to the previous text paragraphs obtained by backtracking are used as context constraint information, introduced into the prosody prediction process of the current text paragraph, and used as the initial state of the current text paragraph to participate in recursive calculation; In the recursive calculation process, the emotion compensation parameter set corresponding to the current text paragraph and the broadcasting guide information are introduced step by step according to the evolution order of time steps, the continuity constraint is applied to the rhythm control elements, the continuously changing rhythm prediction value sequence is generated step by step, and the rhythm prediction signal maintaining the continuity of emotion evolution and the consistency of rhythm transition between adjacent text paragraphs is obtained.

10. A station group article intelligent broadcasting system based on multi-modal emotional speech synthesis, the station group article intelligent broadcasting method based on multi-modal emotional speech synthesis according to any one of claims 1-9, characterized in that, Comprise: A text processing module for obtaining multi-source text content in a distributed site cluster, stripping non-text labels, and performing semantic enhancement on the multi-source text content to generate a serialized text stream with broadcasting guidance; An emotion mapping module for extracting paragraph-level semantics and emotional features based on the serialized text stream, and performing dimension mapping on the semantics and emotional features to generate an emotion compensation parameter set; A rhythm scheduling module for storing hidden layer state features generated in the previous text paragraph rhythm prediction phase based on the emotion compensation parameter set according to paragraph granularity; when performing rhythm prediction on the current text paragraph, the corresponding previous paragraph hidden layer state features are introduced back to participate in the rhythm prediction of the current paragraph, and a cross-paragraph consistent rhythm prediction signal is generated; An acoustic synthesis module for cross-modal alignment and fusion of the rhythm prediction signal and the emotion compensation parameter set to generate acoustic feature prediction results, and for converting the acoustic feature prediction results into mel-spectrum features to generate a corresponding digital audio signal stream; A distribution module for matching and packaging the digital audio signal stream according to the site attribute metadata, and pushing it to the corresponding site cluster for intelligent broadcasting.

Images