A discrete speech representation system and method oriented to symbolic expression

Abstract

This invention relates to the field of discrete speech representation, and discloses a discrete speech representation system and method oriented towards symbolic expression. The system includes: a symbolic normalization module, a speech synthesis module, a speech discretization module, a repetition suppression module, a boundary anchoring module, a cross-modal generation module, and a controlled decoding module. This invention not only rapidly constructs large-scale, high-quality symbolic speech training corpora, but also significantly improves the reading accuracy and robustness of end-to-end speech systems in symbolic tasks through the trained model, providing directly deployable technical support for educational, scientific research, and engineering applications.

Description

Technical Field

[0001] This invention relates to the field of discrete speech representation, and more specifically to a discrete speech representation system and method oriented towards symbolic representation. Background Technology

[0002] In recent years, end-to-end large-scale speech models have achieved significant breakthroughs in general speech tasks. Compared with traditional cascaded speech systems, end-to-end models eliminate multiple stages such as automatic speech recognition, text processing, and speech synthesis, and have the advantages of low latency and ease of industrial deployment. They have been widely used in application scenarios such as intelligent assistants, real-time translation, and voice dialogue.

[0003] However, existing end-to-end speech models have significant limitations when dealing with symbolic representations (such as mathematical formulas, physical symbols, and chemical molecular formulas). These representations not only contain hierarchical structures such as subscripts, superscripts, fractions, and square roots, but also involve complex computational relationships and specific pronunciation rules, far exceeding the linear sequence modeling capabilities of natural language. Furthermore, the lack of large-scale symbolic speech training data makes models prone to problems such as missing pronunciations, errors, or incomplete structural representations, making it difficult to meet the high-precision requirements for symbolic reading in education and research.

[0004] Furthermore, training end-to-end speech models typically relies on large-scale discrete speech representation corpora. While abundant general-purpose speech data exists, discrete speech representation data specifically for symbolic representations remains severely lacking, and existing publicly available resources are insufficient to support the model's full learning of symbols. This data scarcity is a major bottleneck restricting the effectiveness of end-to-end speech models in symbolic representation scenarios.

[0005] Current mainstream methods typically employ a cascaded process, which involves first converting symbolic representations into textual descriptions, then calling a speech synthesis model to generate speech, and finally extracting representations through a speech discretization model. However, this process has several shortcomings: First, the generation process is lengthy, requiring multiple conversion steps for each sample, resulting in low efficiency and difficulty in supporting the construction of millions of data points. Second, the symbolic sequences and speech sequences lack explicit boundary markers, leading to unstable alignment and potential errors in cross-modal correspondences. Third, discrete speech representations often contain a large number of redundant and repetitive units, wasting storage resources and slowing down model convergence. Fourth, most existing solutions are optimized for natural language and lack specific designs for symbolic representations, thus limiting their performance in application scenarios such as mathematics, physics, and chemistry. Summary of the Invention

[0006] To address the technical problems mentioned above, this invention provides a discrete speech representation system oriented towards symbolic representation, comprising: a symbolic normalization module, a speech synthesis module, a speech discretization module, a repetition suppression module, a boundary anchoring module, a cross-modal generation module, and a controlled decoding module;

[0007] The symbol standardization module is used to standardize the input symbol sequence and generate a standardized symbol sequence.

[0008] The speech synthesis module is used to synthesize the standardized symbol sequence into a continuous speech signal;

[0009] The speech discretization module is used to discretize the continuous speech signal into a sequence of original discrete units;

[0010] The repetition suppression module is used to perform repetition suppression processing on the original discrete unit sequence to obtain a suppressed discrete unit sequence.

[0011] The boundary anchoring module is used to perform boundary anchoring processing on the suppressed discrete unit sequence to generate a discrete unit sequence with boundary markers and its alignment information with the normalized symbol sequence.

[0012] The cross-modal generation module is used to train a parameterized generation model based on the standardized symbol sequence and the discrete unit sequence with boundary markers, so as to learn the mapping relationship from the symbol sequence to the discrete unit sequence and output the predicted discrete unit sequence.

[0013] The controlled decoding module is used to perform controlled decoding on the predicted discrete unit sequence and generate the final discrete speech representation by constraining the output unit set.

[0014] Preferably, the workflow of the symbol standardization module includes: converting the input symbol sequence into a standardized symbol sequence with unified identification and hierarchical information through normalization, structural expansion and labeling.

[0015] Preferably, the workflow of the speech synthesis module includes: mapping a standardized symbol sequence to a time-domain speech waveform using a speech synthesis model.

[0016] ;

[0017] in, Indicates continuous speech signals; Indicates model parameters; τ represents the speech synthesis model; τ represents the normalized symbol sequence mapped to the prompt.

[0018] Preferably, the workflow of the speech discretization module includes: compressing a continuous speech signal into a discrete unit sequence using a quantization operator.

[0019] ;

[0020] in, Represents a sequence of discrete units; Represents the quantization operator; For quantization dictionary or encoder parameters; Let i represent discrete speech units, i=1,2,3...M.

[0021] Preferably, the workflow of the repetition suppression module includes: using a window-based repetition reduction operator to detect and merge consecutive identical segments with a length not less than a preset minimum repetition length.

[0022] ;

[0023] in, It is the suppressed discrete unit sequence; This represents a repetition suppression operator based on window size or minimum repetition length n.

[0024] Preferably, the workflow of the boundary anchoring module includes: adding explicit boundary symbols to both ends of the discrete unit sequence, and calculating the correspondence between the symbol sequence and the discrete unit sequence as alignment information.

[0025] ;

[0026] ;

[0027] in, Represents a sequence of discrete units with boundary markers; Explicit anchoring symbols indicating the start and end of a sequence; Represents the standardized symbol sequence; α represents the cross-modal alignment operator; α represents the alignment information.

[0028] Preferably, the workflow of the cross-modal generation module includes: defining a parameterized generation model. It directly learns the conditional distribution from standardized symbol sequences to discrete speech representations; its optimization objective is:

[0029] ;

[0030] in, Indicates training loss; Represents the learnable parameters of a cross-modal generative model; This represents the summation over all pairs of samples in the training dataset; The time step index k of the target sequence ranges from 1 to the length. The gradual accumulation; for Length; This represents the target discrete speech unit at time step index k; This is the prefix preceding time step index k; The parameter represents the input of a given symbol and the history prefix. Defined conditional probability.

[0031] Preferably, the workflow of the controlled decoding module includes: imposing constraints on the decoding space and setting the allowed set of speech units as... The decoding constraint is:

[0032] ;

[0033] in, This represents the final generated sequence of discrete units; This indicates selecting the sequence from the candidate set that maximizes the objective function value; A sequence of candidate discrete units; This represents a series of product steps over time step index k; Let be the discrete speech unit at the k-th position of the candidate sequence; Indicates that in a given input sequence of symbols and prefix Under the condition of model parameters Defined generation probability distribution; This represents the set of legal speech units that are allowed to be output. This indicates a constraint.

[0034] This invention also provides a discrete speech representation method oriented towards symbolic representation, which is applied to the above-mentioned system and includes the following steps:

[0035] The input symbol sequence is standardized to generate a standardized symbol sequence;

[0036] The standardized symbol sequence is synthesized into a continuous speech signal;

[0037] The continuous speech signal is discretized into a sequence of original discrete units;

[0038] The original discrete unit sequence is subjected to repetition suppression processing to obtain the suppressed discrete unit sequence;

[0039] The suppressed discrete unit sequence is subjected to boundary anchoring processing to generate a discrete unit sequence with boundary markers and its alignment information with the normalized symbol sequence;

[0040] Based on the standardized symbol sequence and the discrete unit sequence with boundary markers, a parameterized generation model is trained to learn the mapping relationship from the symbol sequence to the discrete unit sequence and output the predicted discrete unit sequence.

[0041] The predicted discrete unit sequence is decoded in a controlled manner, and the final discrete speech representation is generated by constraining the output unit set.

[0042] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0043] This invention not only rapidly constructs a large-scale, high-quality symbolic speech training corpus, but also significantly improves the reading accuracy and robustness of end-to-end speech systems in symbolization tasks through the trained model, providing directly deployable technical support for education, scientific research, and engineering applications. Attached Figure Description

[0044] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a schematic diagram of the system structure according to an embodiment of the present invention. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0048] Example 1:

[0049] like Figure 1 The diagram shown is a flowchart of the method in this embodiment, including: a symbol normalization module, a speech synthesis module, a speech discretization module, a repetition suppression module, a boundary anchoring module, a cross-modal generation module, and a controlled decoding module; each module is relatively independent yet closely coupled, working together to realize the complete process from input symbol sequence to discrete speech representation generation and quality control.

[0050] First, the input symbol sequence is standardized using the symbol normalization module to generate a normalized symbol sequence. In the symbol normalization module, the input symbol sequence can be represented as:

[0051] ;

[0052] in, (i=1,2,3...N) represents a basic unit in the symbolic representation, which may be a mathematical symbol, a physical quantity symbol, a chemical element, or other label. To reduce representational differences, this embodiment defines a normalized operator:

[0053] ;

[0054] in, Including normalization, structure expansion, and labeling processing, the output... This module provides a standardized symbol sequence with unified identifiers and hierarchical information. Its purpose is to provide stable and consistent symbol input for subsequent speech synthesis and discretization.

[0055] Secondly, the continuous speech signal is discretized into a sequence of original discrete units using a speech discretization module. In the speech synthesis module, a standardized symbol sequence is used. It is mapped to a cue τ and input into the speech synthesis model. Generate continuous speech signals:

[0056] ;

[0057] in, Indicates continuous speech signals; This module represents the model parameters. Its function is to convert symbolic representations into high-fidelity, semantically consistent speech signals, supporting batch generation to meet the needs of large-scale training data.

[0058] Then, the speech discretization module discretizes the continuous speech signal into a sequence of original discrete units. In the speech discretization module, the continuous speech signal... Converted into a sequence of discrete units:

[0059] ;

[0060] in, Represents a sequence of discrete units; Represents the quantization operator; For quantization dictionary or encoder parameters; This represents a discrete speech unit, i=1,2,3...M. The function of this module is to compress continuous signals into discrete symbols, facilitating storage and downstream modeling.

[0061] Subsequently, the original discrete unit sequence is subjected to repetition suppression processing using a repetition suppression module to obtain the suppressed discrete unit sequence. In the repetition suppression module, for long repetitive segments in the discrete unit sequence Y, this embodiment proposes a window-based repetition reduction operator:

[0062] ;

[0063] in, The original discrete unit sequence is i = 1, 2, 3...M, where M is the sequence length; This represents a repetition suppression operator based on window size / minimum repetition length n, used to detect and merge (or fold and replace) consecutive identical segments of length not less than n. It is a suppressed discrete unit sequence, and The purpose of this module is to reduce redundancy, increase information density, and accelerate model convergence.

[0064] The suppressed discrete unit sequence is subjected to boundary anchoring processing using a boundary anchoring module, generating a discrete unit sequence with boundary markers and its alignment information with the normalized symbol sequence. In the boundary anchoring module, to enhance the correspondence between symbols and speech, explicit boundary symbols are added to both ends of the discrete unit sequence, and an alignment mapping is generated.

[0065] ;

[0066] ;

[0067] in, Represents a sequence of discrete units with boundary markers; Explicit anchoring symbols indicating the start and end of a sequence; Represents the standardized symbol sequence; α represents the cross-modal alignment operator, used to calculate the correspondence between symbol sequences and discrete unit sequences; α represents the alignment information, which can be specifically implemented as a set of index pairs or an alignment matrix, used to indicate the correspondence between symbol segments and speech units.

[0068] A cross-modal generation module is used, based on standardized symbol sequences and discrete unit sequences with boundary markers. A parametric generation model is trained to learn the mapping relationship from symbol sequences to discrete unit sequences and outputs predicted discrete unit sequences. In this embodiment, a parametric generation model is defined within the cross-modal generation module. It directly learns the conditional distribution from standardized symbol sequences to discrete speech representations; its optimization objective is:

[0069] ;

[0070] in, Indicates training loss; Represents the learnable parameters of a cross-modal generative model; This represents the summation over all pairs of samples in the training dataset; The time step index k of the target sequence ranges from 1 to the length. The gradual accumulation; for Length; This represents the target discrete speech unit at time step index k; Prefix before time step index k ; The parameter represents the input of a given symbol and the history prefix. Defined conditional probability.

[0071] Finally, the controlled decoding module performs controlled decoding on the predicted discrete unit sequence, generating the final discrete speech representation by constraining the output unit set. In this embodiment, the controlled decoding module imposes constraints on the decoding space to ensure the stability and validity of the generated result. Let the allowed set of speech units be... The decoding constraint is:

[0072] ;

[0073] in, This represents the final generated sequence of discrete units; This indicates selecting the sequence from the candidate set that maximizes the objective function value; A sequence of candidate discrete units; This represents a series of product steps over time step index k; Let be the discrete speech unit at the k-th position of the candidate sequence; Indicates that in a given input sequence of symbols and prefix Under the condition of model parameters Defined generation probability distribution; This represents the set of legal speech units that are allowed to be output. This indicates a constraint.

[0074] The purpose of this module is to improve the quality and practicality of discrete speech representation by constraining the set of legal units and termination conditions to avoid generation drift or redundancy.

[0075] In summary, this invention, through the synergistic effect of modules such as symbol standardization, speech synthesis, speech discretization, repetition suppression, boundary anchoring, cross-modal generation, and controlled decoding, constitutes a complete discrete speech representation method and system oriented towards symbolic expression. It can achieve the construction of large-scale, high-quality symbolic speech data while ensuring efficiency and stability.

[0076] After training, the model is able to solve existing technical problems in the following scenarios:

[0077] (1) Inputting mathematical formulas in educational settings The model can directly generate the precise reading of the symbol "b / a plus c", solving the problem of missing pronunciation and incorrect order in the existing model when reading formulas.

[0078] (2) In the scientific research literature scenario, input the physical expression. The model can output "force equals mass multiplied by acceleration plus the change in momentum", ensuring structural integrity and consistent symbol pronunciation.

[0079] (3) In engineering applications, input chemical reaction formulas The model can generate "two hydrogen molecules plus one oxygen molecule to produce two water molecules", solving the technical problem that traditional speech systems cannot process chemical formulas.

[0080] Therefore, this invention not only rapidly constructs a large-scale, high-quality symbolic speech training corpus, but also significantly improves the reading accuracy and robustness of end-to-end speech systems in symbolization tasks through the trained model, providing directly deployable technical support for education, scientific research and engineering applications.

[0081] Example 2:

[0082] This embodiment also provides a discrete speech representation method oriented towards symbolic representation, comprising the following steps: standardizing the input symbolic sequence to generate a standardized symbolic sequence; synthesizing the standardized symbolic sequence into a continuous speech signal; discretizing the continuous speech signal into a sequence of original discrete units; performing repetition suppression on the original discrete unit sequence to obtain a suppressed discrete unit sequence; performing boundary anchoring on the suppressed discrete unit sequence to generate a discrete unit sequence with boundary markers and its alignment information with the standardized symbolic sequence; training a parameterized generation model based on the standardized symbolic sequence and the discrete unit sequence with boundary markers to learn the mapping relationship from the symbolic sequence to the discrete unit sequence, and outputting a predicted discrete unit sequence; performing controlled decoding on the predicted discrete unit sequence, and generating the final discrete speech representation by constraining the output unit set.

[0083] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A discrete phonetic representation system oriented towards symbolic expression, characterized by, Comprising: The symbol normalization module, the speech synthesis module, the speech discretization module, the repetition suppression module, the boundary anchoring module, the cross-modal generation module and the controlled decoding module; The symbol normalization module is used for standardizing the input symbol sequence to generate a standardized symbol sequence; The speech synthesis module is used for synthesizing the standardized symbol sequence into a continuous speech signal; The speech discretization module is used for discretizing the continuous speech signal into an original discrete unit sequence; The repetition suppression module is used for suppressing the repetition of the original discrete unit sequence to obtain a suppressed discrete unit sequence; The boundary anchoring module is used for anchoring the boundaries of the suppressed discrete unit sequence to generate a discrete unit sequence with boundary identification and alignment information between the discrete unit sequence and the standardized symbol sequence; The cross-modal generation module is used for training a parametric generation model based on the standardized symbol sequence and the discrete unit sequence with boundary identification to learn the mapping relationship from the symbol sequence to the discrete unit sequence, and output a predicted discrete unit sequence; The controlled decoding module is used for controlled decoding of the predicted discrete unit sequence to generate a final discrete speech representation by constraining the output unit set.

2. The symbol-oriented expression discrete speech representation system of claim 1, wherein, The workflow of the symbol normalization module includes: converting the input symbol sequence into a standardized symbol sequence with uniform identification and hierarchical information through normalization, structure expansion and labeling.

3. The symbol-oriented expression discrete speech representation system of claim 1, wherein, The workflow of the speech synthesis module includes: using a speech synthesis model to map the standardized symbol sequence to a time-domain speech waveform: ； wherein, represents a continuous speech signal; represents a model parameter; represents a speech synthesis model; τ represents a sequence of normalized symbols mapped to a prompt.

4. The symbol-oriented expression discrete speech representation system of claim 3, wherein, The workflow of the speech discretization module includes: using a quantization operator to compress the continuous speech signal into a discrete unit sequence: ； wherein, represents a sequence of discrete units; represents a quantization operator; is a quantization dictionary or encoder parameter; represents a discrete speech unit, i = 1, 2, 3... M.

5. The symbol-oriented expression discrete speech representation system of claim 4, wherein, The workflow of the repetition suppression module includes: using a window-based repetition reduction operator to detect and merge continuous identical segments with a length not less than a preset minimum repetition length: ； wherein, is the sequence of discrete units after suppression; denotes a repetition suppression operator based on a window size or a minimum repetition length n.

6. The symbol-oriented expression discrete speech representation system of claim 5, wherein, The workflow of the boundary anchoring module includes: adding explicit boundary symbols at both ends of the discrete unit sequence, and calculating the corresponding relationship between the symbol sequence and the discrete unit sequence as alignment information: ； ； wherein, represents a sequence of discrete units with boundary markers; represents explicit anchor symbols for sequence start and end; represents a normalized sequence of symbols; represents a cross-modal alignment operator; a represents alignment information.

7. The symbol-oriented expression discrete speech representation system of claim 6, wherein, The workflow of the cross-modal generation module includes: defining a parameterized generation model directly learning a conditional distribution from normalized symbolic sequences to discrete speech representations; and an optimization objective is: ； wherein, denotes the training loss; denotes the learnable parameters of the cross-modal generative model; denotes the summation over all sample pairs in the training dataset; denotes the step-wise accumulation of the target sequence's time step index k from 1 to length of the target sequence; is the length of ; denotes the target discrete speech unit at time step index k; is the prefix up to time step index k; denotes the conditional probability defined by parameters given the symbol input and the history prefix.

8. The symbol-oriented expression discrete speech representation system of claim 7, wherein, The workflow of the controlled decoding module includes: applying constraints to the decoding space, setting the allowed set of phonetic units as , and the decoding constraint is: ； wherein, denotes the final generated discrete unit sequence; denotes selecting the sequence that maximizes the objective function value among the candidate set; is a candidate discrete unit sequence; denotes step-wise multiplication over time step index k; is the discrete speech unit at the k-th position of the candidate sequence; denotes the generation probability distribution defined by the model parameters under the condition of the given input normalized symbol sequence and the prefix ; denotes the set of legal speech units allowed for output; denotes the constraints.

9. A method for discrete phonetic representation oriented to symbolic expression, the method being applied to the system of any of claims 1-8, characterized by the steps of Comprising: standardizing the input symbol sequence to generate a standardized symbol sequence; synthesizing the standardized symbol sequence into a continuous speech signal; discretizing the continuous speech signal into an original discrete unit sequence; suppressing the repetition of the original discrete unit sequence to obtain a suppressed discrete unit sequence; anchoring the boundaries of the suppressed discrete unit sequence to generate a discrete unit sequence with boundary identification and alignment information between the discrete unit sequence and the standardized symbol sequence; training a parametric generation model based on the standardized symbol sequence and the discrete unit sequence with boundary identification to learn the mapping relationship from the symbol sequence to the discrete unit sequence, and output a predicted discrete unit sequence; controlling the decoding of the predicted discrete unit sequence to generate a final discrete speech representation by constraining the output unit set.

Images