Speech understanding method and system based on text alignment and electronic equipment

Abstract

The embodiment of the invention provides a speech understanding method and system based on text alignment and electronic equipment. The method comprises the steps that training data are input into a voice understanding model, the training data comprise a training text, and the voice understanding model comprises a CTC posterior simulation module; in a CTC posteriori simulation module, the training text is converted into a pseudo CTC posteriori simulating real audio distribution characteristics; and a pseudo CTC posteriori is utilized to generate a pseudo posteriori supervision projection module, the pseudo posteriori supervision projection module is used for performing projection reasoning on the input voice to obtain a structured semantic alignment posteriori representation, and a large language model is utilized to determine a voice understanding result of the semantic alignment posteriori representation. According to the embodiment of the invention, clean text symbol labels are converted into noise multi-frame false posteriors, the false posteriors are close to the distribution characteristics of real voice, and the real voice is kept compressed efficiently, so that the over-fitting is relieved and the training / reasoning is accelerated. The multi-task expandability and maintainability are improved, and multi-task zero sample generalization is supported.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of intelligent speech, in particular to a speech understanding method and system based on text alignment and an electronic device. BACKGROUND

[0002] With the development of Speech LLMs, a unified architecture for cross-lingual speech understanding tasks is paved. Unlike the traditional cascaded speech understanding that provides text input for automatic speech recognition, LLMs directly align speech and text modalities through mechanisms such as continuous feature projection or discrete token enhancement to obtain speech understanding results. In the prior art, the following techniques are usually used for cross-lingual speech understanding tasks: 1, alignment paradigm based on hidden state projection; 2, more effective downsampling using CTC signals; 3, weighting LLM word vectors with CTC posteriors for reconstruction; 4, large speech LLMs, etc.

[0003] The above methods complete speech-text alignment with "continuous acoustic hidden representation + heuristic downsampling" or use discrete token / CTC clues for alignment enhancement. Through matching speech-text data and a large amount of training, semantic understanding for the training data domain can be achieved.

[0004] In the process of implementing the present application, the inventors have found at least the following problems in the related art: The existing alignment paradigm has two main limitations. First, continuous feature projection, although it can preserve detailed audio information, usually introduces a lot of redundancy. This redundancy not only increases the computational cost in the training and inference process, but also increases the risk of overfitting. Second, to alleviate these problems, a large amount of paired audio-text data and a complex training pipeline are usually required to achieve competitive multi-task performance. That is, the above methods usually require a large amount of paired speech-text data and are accompanied by complex training pipelines, and the generalization ability in the multi-task small sample scene is limited. Specifically, redundancy: there are a large number of blank frames and repeated frames in continuous acoustic representation, resulting in increased computational overhead and overfitting risk; dependence: highly dependent on large-scale audio-text alignment corpus; weak generalization: in cross-domain / multi-task scenarios, if there is a lack of sufficient paired data, the effect will be significantly degraded. That is, the acoustic hidden state and the text embedding are mismatched in terms of time structure and information granularity; the existing downsampling strategies are mostly heuristic, and it is difficult to balance compression and semantic preservation; the training target and interface are not unified, resulting in mismatch between the training / inference two stages and insufficient knowledge transfer. SUMMARY

[0005] To at least solve the problems of large computational overhead, weak generalization, and mismatch between training / inference two stages in speech understanding in the prior art.

[0006] In a first aspect, an embodiment of the present application provides a text alignment based speech understanding method, comprising: inputting training data into the speech understanding model, wherein the training data comprises training text, and the speech understanding model comprises a CTC posterior simulation module; in the CTC posterior simulation module, converting the training text into pseudo CTC posterior simulating real audio distribution characteristics; generating a pseudo posterior supervised projection module using the pseudo CTC posterior, the pseudo posterior supervised projection module being used for projecting inference on input speech to obtain a structured semantic alignment posterior representation, and determining a speech understanding result of the semantic alignment posterior representation using a large language model.

[0007] In a second aspect, an embodiment of the present application provides a text alignment based speech understanding system, comprising: an input module configured to input training data into the speech understanding model, wherein the training data comprises training text, and the speech understanding model comprises a CTC posterior simulation module; a simulation module configured to convert, in the CTC posterior simulation module, the training text into pseudo CTC posterior simulating real audio distribution characteristics; a speech understanding module configured to generate a pseudo posterior supervised projection module using the pseudo CTC posterior, the pseudo posterior supervised projection module being used for projecting inference on input speech to obtain a structured semantic alignment posterior representation, and determining a speech understanding result of the semantic alignment posterior representation using a large language model.

[0008] In a third aspect, an electronic device is provided, comprising at least one processor, and a memory connected to the at least one processor in communication, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform steps of the text alignment based speech understanding method of any embodiment of the present application.

[0009] In a fourth aspect, an embodiment of the present application provides a storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements steps of the text alignment based speech understanding method of any embodiment of the present application.

[0010] In a fifth aspect, an embodiment of the present application provides a computer program product comprising computer programs / instructions, wherein the computer programs / instructions, when executed by a processor, implement steps of the text alignment based speech understanding method of any embodiment of the present application.

[0011] The beneficial effects of the embodiment of the present application are that the CPS of the method converts clean text symbol labels into noise multi-frame pseudo posteriors, which are very close to the distribution characteristics of real audio, and the real speech can be efficiently compressed by using the LSD, which helps to alleviate overfitting and accelerate training / inference. In this way, the pseudo CTC posterior and the compact posterior sequence are used to supervise the projection module, the pseudo posterior sequence is used to supervise the Projector training, the LLM is kept frozen, and it is trained as a unified shared projection module for text-speech isomorphism. In this way, the unified shared projection module interface allows different tasks to share the Projector and the LLM, improves the multi-task scalability and maintainability, and supports multi-task zero-shot generalization. BRIEF DESCRIPTION OF DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0013] Figure 1 is a flowchart of a text alignment-based speech understanding method provided by an embodiment of the present application; Figure 2 is a whole training flow structure diagram of a text alignment-based speech understanding method provided by an embodiment of the present application; Figure 3 is a comparison diagram of different alignment strategies of a multi-modal speech understanding model of a text alignment-based speech understanding method provided by an embodiment of the present application; Figure 4 is a multi-task generalization diagram of a speech understanding method using TASU provided by an embodiment of the present application; Figure 5 is a comparison diagram of different alignment paradigms of a text alignment-based speech understanding method provided by an embodiment of the present application; Figure 6 is an LSD ablation research diagram of a text alignment-based speech understanding method provided by an embodiment of the present application; Figure 7 is a structure diagram of a text alignment-based speech understanding system provided by an embodiment of the present application; Figure 8 is a structure diagram of an electronic device for a text alignment-based speech understanding provided by an embodiment of the present application. DETAILED DESCRIPTION

[0014] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0015] As Figure 1 shown is a flowchart of a text alignment-based speech understanding method provided by an embodiment of the present application, comprising the following steps: S11: inputting training data to the speech understanding model, wherein the training data comprises training text, and the speech understanding model comprises a CTC posterior simulation module; S12: in the CTC posterior simulation module, converting the training text into pseudo CTC posterior simulating real audio distribution characteristics; S13: generating a pseudo posterior supervision projection module by using the pseudo CTC posterior, the pseudo posterior supervision projection module being used for projecting and reasoning input speech to obtain a structured semantic alignment posterior representation, and determining a speech understanding result of the semantic alignment posterior representation by using a large language model.

[0016] The present application finds that speech LLM (Large Language Model) directly aligns speech and text modalities to obtain a speech understanding result through continuous feature projection or discrete label enhancement mechanisms, which either produces redundant and noisy representations or has the risk of discarding key information. In addition, acoustic features exhibit high temporal variability, which does not match the structured nature of text embeddings, making it difficult to implement cross-modal processing. In order to solve the above problems, the present method proposes TASU (text-only alignment for speech understanding), as Figure 2 shown (wherein, in the training process, only text input is used: the transcription is labeled as a one-hot vector and converted into a pseudo CTC posterior by simulation. In the reasoning process, the speech is encoded to generate real CTC posteriors, which are refined by label-synchronous decoding. Both the pseudo CTC and the real CTC posteriors are mapped into a frozen LLM by a trainable projector, producing outputs such as transcriptions or other speech understanding tasks), the present application directly aligns speech and text after CTC (Connectionist Temporal Classification, connectionist temporal classification (time alignment framework containing blank)), and in general, the key idea of the present method is to train and reason to establish a unified posterior interface: Training: Text transcripts are labeled as one-hot vectors and converted to pseudo posteriors by a CPS module, which supervises the trainable projection.

[0017] Inference: Raw speech is encoded into real CTC posteriors, refined by LSD, and mapped into the frozen LLM by the pretrained projection. CTC (Connectionist Temporal Classification) posterior probability refers to the probability distribution of all possible output sequences given the input sequence.

[0018] In this way, the TASU of the present method realizes pure text training while ensuring that the two modes share compact, structured, and semantically aligned posterior representations.

[0019] For step S11, the speech understanding model of the present application is as shown in the "training" part of Figure 2 The purpose is to generate pseudo CTC posteriors by CPS (CTC posterior simulation) during training using only text to supervise the training of the projection module. Therefore, for training data, the present method only needs "text" to realize the training of speech understanding. The training text is simpler to obtain, and can be obtained by crawling forum dialogue text or collecting user text interaction records. A large amount of dialogue text can be obtained through the above-mentioned ways, of course, it can also be obtained through other ways, which will not be described here. When only "text" is used, the speech understanding model performs processing in the tokenization-encoder, CTC posterior simulation part of the "training" in the diagram.

[0020] For step S12, in order to be able to use pure text data for training, the present method proposes a CPS (CTC posterior simulation module) that converts each token extracted from text into a pseudo posterior sequence. CPS consists of three random stages to mimic the variability of real CTC outputs.

[0021] As an implementation, converting the training text into pseudo CTC posteriors that simulate the characteristics of real audio distribution includes: Performing one-hot random smoothing processing on the training text to generate a smoothed posterior that simulates real acoustics; Performing random deletion processing on part of the smoothed posterior to simulate alignment error scenarios of frame loss; After the random deletion processing, random insertion is performed to introduce alignment jitter to obtain pseudo CTC posteriors that simulate the characteristics of real audio distribution.

[0022] The random insertion after the random deletion processing includes simulating repeated copies of CTC and blank labels.

[0023] In the embodiment, the text is segmented, and after segmentation, One-Hot encoding is performed to obtain token labels, wherein One-Hot processing classifies features, and converts discrete category variables into binary vector representation. The core idea is to map each category value to an integer with a unique index, and the binary form of the integer is used as a feature vector.

[0024] In random label smoothing, for a label y, it is represented as one-hot The method uses uniform distribution to interpolate it to obtain the smoothed posterior:

[0025] The above steps generate an initial sequence The random factor α ensures that the generated distribution covers a wide range of confidence levels to simulate the uncertainty similar to the real acoustic posterior.

[0026] Then random deletion is performed, Each element of is independently removed with a probability p del Random frame deletion simulates the label loss commonly seen in CTC alignment. This operation simulates the fact that non-empty labels occasionally disappear due to alignment errors, forcing the system to be robust to missing content.

[0027] Finally, random insertion processing is performed, which can be performed:

[0028] Wherein, is used to control the insertion rate.

[0029] For each insertion, the position is sampled, and inserted with equal probability: (1) A copy of is inserted, that is, adjacent frames are simulated to repeat; (2) Insert a blank one-hot vector e blank This step introduces alignment jitter to capture CTC-specific repetition and blank separation, mitigating the inaccuracy of CTC and enhancing robustness, otherwise the performance will decrease significantly according to the experiment.

[0030] In the above manner, the training text is converted into a pseudo-CTC posterior that simulates the characteristics of the real audio distribution.

[0031] For step S13, the CPS (CTC posterior simulation) in step S12 converts the clean symbol label into a noise multi-frame pseudo posterior, which is very close to the distribution characteristics of real audio. In this way, the pseudo CTC posterior close to the distribution characteristics of real audio can be directly used to generate a pseudo posterior supervised projection module. Through the pseudo posterior supervised projection module, a unified interface is established at the "posterior distribution level" by the large language model to realize projection reasoning on the input speech, obtain a structured semantic alignment posterior representation, and finally determine the speech understanding result of the semantic alignment posterior representation by using the LLM word segmentation and other processing.

[0032] As can be seen from this embodiment, the method converts large-scale text resources into "pseudo posterior" to improve the cross-domain robustness of the speech understanding model trained with zero speech samples.

[0033] To further improve the training effect, the method further trains with small-sample text-speech. As an implementation manner, the training data includes training speech corresponding to the training text, and the speech understanding model includes a label synchronous decoding module. The training method of the speech understanding model includes: In the CTC posterior simulation module, the training text is converted into a pseudo CTC posterior simulating the distribution characteristics of real audio. In the label synchronous decoding module, the semantic content is retained and the timing redundancy of the training speech is compressed to obtain a compact posterior sequence. The pseudo posterior supervised projection module is supervised trained by using the pseudo CTC posterior and the compact posterior sequence, and is trained into a text-speech isomorphic unified shared projection module. The unified shared projection module is used to project and reason on the input speech to obtain a cross-modal bridged semantic alignment posterior representation, and determine the speech understanding result of the semantic alignment posterior representation by using the large language model.

[0034] In this embodiment, to further perform two-stage training, the method prepares small-sample text-speech training data. Since the quantity requirement is relatively low, the acquisition is relatively simple.

[0035] In the further two-stage training, in addition to the "training" part in the figure, the method also performs "reasoning".

[0036] The text is input into the word segmentation-encoder for CTC posterior simulation. This part has been described in the foregoing and will not be repeated here.

[0037] The speech is input to an audio encoder to obtain a real CTC posterior, and the real CTC posterior is input to a label-synchronous decoding (LSD) module of the method.

[0038] Since CTC decoding usually involves a large number of blank labels and consecutive repetitions of the same label, directly inputting these back into the speech LLM propagates redundant and ambiguous semantics. Therefore, LSD compresses the sequence by two operations while maintaining semantic fidelity.

[0039] As an implementation, in the label-synchronous decoding module, the semantic content is preserved and the timing redundancy of the training speech is compressed to obtain a compact posterior sequence, including: determining a real CTC posterior of the training speech using an encoder; removing blank frames and adjacent frames in the real CTC posterior to obtain a real compact posterior sequence.

[0040] In this implementation, deleting blank frames can be given a posterior sequence with T frames and a vocabulary V , and frames dominated by blank probability are discarded with an adjustable threshold τ:

[0041] And for consecutive frame merging, we can set denotes the top symbol at frame t. For each maximum number of consecutive identical frames Sj of the same yt, the vectors are averaged:

[0042] where J is the number of frames retained after the above formula Pt'. This process eliminates blank-dominated frames and folds redundant repetitions, resulting in a time-compact posterior wall that preserves the basic information of alignment. The method achieves significant compression of the acoustic feature sequence without sacrificing semantic integrity.

[0043] As can be seen, through the above operations, CPS converts clean text symbol labels into noisy multi-frame pseudo-posteriors that are very close to the distribution characteristics of real audio, and using LSD can keep real speech efficiently compressed, which helps to alleviate overfitting and speed up training / inference. In this way, the pseudo CTC posterior and the compact posterior sequence are used to supervise the projection of the pseudo posterior, the LLM is kept frozen, and the training is performed as a text-speech isomorphic unified shared projection module. In this way, the unified shared projection module interface allows different tasks to share the projector and LLM, improving multi-task scalability and maintainability, and supporting multi-task zero-shot generalization.

[0044] To more intuitively understand the differences between this method, TASU, and other alignment paradigms, such as... Figure 3 A concise comparison of speech LLM alignment paradigms is provided (wherein, a concise comparison of different alignment strategies in multimodal speech understanding models is provided. Training part: E = encoder, P = projection module, L = LLM (parentheses indicate optional components)). In particular, only the TASU model of this method, trained solely on text, achieves zero-shot performance across multiple tasks. It is noteworthy that LSD achieves an average downsampling rate of nearly 6 on experimental data, significantly improving the speed of training and inference.

[0045] Specific experiments are provided to illustrate this method, demonstrating that its TASU implementation can achieve zero-shot conversion from text training to speech inference. To verify its rationality and effectiveness, a control experiment was conducted.

[0046] Regarding the model architecture, since this TASU method relies on reliable CTC posterior probabilities, it employs SenseVoice Small as the speech encoder and Qwen2.5-1.5B as the language model backbone. The projection is instantiated as a Linear-SiLU-Linar module, with only its parameters being trainable. The bottleneck is typically set to 1024. For broader speech understanding tasks, it is set to 2048, as... Figure 4 As shown (speech understanding multi-task generalization using TASU. The models in the upper-level modules are built on the same components and training settings and share the same multi-task data, while TASU only uses text. Results are reported as WER%, BLEU, and accuracy).

[0047] Regarding the training data, for ASR, the datasets include LibriSpeech, SlideSpeech, and CommonVoice4. For speech-to-text translation (ST), this method uses CoVoST2 En→Zh, and for spoken language teaching comprehension, it uses SLURP[21,22,23,24,25].

[0048] Regarding training settings, for LSD, the parameter τ is set to 0.9. For CPS, this method sets the label smoothing range (λlow, λhigh) to (0.8, 1.0), and the deletion and copying probabilities p... del and p dup All values ​​are set to 0.05. The learning rate is fixed at 5×10⁻⁵. -5 There are 5 training cycles. A checkpoint is selected when the assessment loss stops decreasing.

[0049] On the evaluation dataset and setup, the TASU model of the present method is evaluated on ASR and speech understanding tasks. For ASR, the word error rate (WER) is reported on the standard in-domain test set. To further evaluate the generalization capability, TED-LIUM3 is adopted to test the robustness on different topics and acoustic domains (talks). For speech understanding tasks, the performance is evaluated on the MMSU benchmark. WER is computed using the official Wenet toolkit.

[0050] The experimental results are presented in two parts. First, the present method demonstrates that TASU can achieve zero-shot speech recognition and, when used as a curriculum pre-training stage, allows a model fine-tuned on source domain audio data to effectively generalize to new domains. Second, TASU is evaluated on multi-task speech understanding, which achieves zero-shot generalization from limited text and provides strong performance on the MMSU benchmark.

[0051] To evaluate the effectiveness of TASU in speech recognition, the present method conducts a series of experiments, as shown in Figure 5 Comparison of different alignment paradigms. All systems share the same components and training setup, only the projection is trainable. Libri = LibriSpeech, Ted-3 = TedLium-3, Slide = SlideSpeech. Results are WER %. TASU (+SFT) denotes a two-stage curriculum learning procedure. To enable a controlled comparison, the SLAM alignment strategy proposed in SLAM-LLM is implemented without downsampling to avoid potential performance degradation. In LibriSpeech testing, TASU shows less than 1.5% WER gap compared to the baseline, which indicates that it can achieve reasonable semantic alignment without paired audio-text training. Moreover, when SlideSpeech transcripts are incorporated into TASU training, a consistent improvement on SlideSpeech itself can be observed, even outperforming the TedLium-3 baseline on the new domain.

[0052] To further explore the scalability, the present method extends TASU as a pre-training stage of curriculum learning. In this stage, SlideSpeech and LibriSpeech text transcripts are used to train a CTC posterior simulator, which is then fine-tuned using LibriSpeed audio-text pairs. The results show that TASU not only maintains performance on Librispeech but also makes substantial progress on both TedLium-3 and SlideSpeech. These findings highlight the scalability of TASU in leveraging large-scale pure text resources for domain generalization.

[0053] To further demonstrate the validity of the baselines shown above, ablation studies were conducted to compare recognition performance under current alignment paradigms. For example... Figure 6 The model architecture and training settings remain unchanged (all models are trained only on Librispeech with the same structure. CTC refers to the CTC tail. TASU without LSD cannot work, resulting in unusable WER scores). SLAM refers to the alignment paradigm used in SLAM-LLM. It can be observed that LSD can almost completely preserve the semantic information of speech, alleviate model overfitting, and play an indispensable role in TASU.

[0054] To further investigate the performance of the TASU method in multi-task speech and semantic understanding, the following steps were performed: Figure 4 The experiments summarized in [the document] remain the baseline for the SLAM method: using hidden states as projective features without downsampling, reflecting the alignment paradigm prevalent in most existing speech LLMs. Given that SLAM architectures cannot develop multi-task capabilities when trained on limited task-specific data, this method extends the training data to ensure fair comparisons: LibriSpeech and CommonVoice4 for ASR, CoVoST2 En→Zh for ST, and SLURP for taught understanding. TASU uses only text, while TASU(+SFT) uses half of the audio-text pairs in the second-stage SFT. Furthermore, to more intuitively evaluate TASU, it is further compared with results from other speech LLMs in the MMSU benchmark.

[0055] TASU demonstrates strong zero-shot multi-task generalization in speech understanding: it outperforms SLAM on MMSU without any audio-text pairs. When half of the audio-text data is combined for SFT, the model shows rapid improvements on ASR and ST tasks. Notably, TASU even surpasses several large-scale speech LLMs, highlighting its efficiency as a lightweight yet effective speech understanding paradigm.

[0056] In summary, our method, TASU, represents a novel alignment paradigm for speech LLM trained solely on text data. On one hand, TASU achieves zero-shot speech recognition with only a slight decrease in accuracy. It can also serve as the first stage of ASR curriculum learning, improving performance in new target domains while maintaining accuracy in source domain recognition. On the other hand, TASU provides robust zero-shot multi-task speech understanding on limited text data, highlighting its potential as a simple yet effective paradigm for scalable and generalizable speech LLM.

[0057] like Figure 7The diagram shown is a structural schematic of a text-aligned speech understanding system according to an embodiment of the present invention. The system can execute the text-aligned speech understanding method described in any of the above embodiments and is configured in a terminal.

[0058] This embodiment provides a text-aligned speech understanding system 10, which includes an input module 11, a simulation module 12, and a speech understanding module 13.

[0059] The input module 11 is used to input training data into the speech understanding model. The training data includes training text. The speech understanding model includes: a CTC posterior simulation module, which is an application used to obtain the device screen configuration of the Android system; a simulation module 12, which is used in the CTC posterior simulation module to convert the training text into a pseudo-CTC posterior that simulates the distribution characteristics of real audio; and a speech understanding module 13, which is used to generate a pseudo-posterior supervised projection module using the pseudo-CTC posterior. The pseudo-posterior supervised projection module is used to perform projection inference on the input speech to obtain a structured semantically aligned posterior representation, and uses a large language model to determine the speech understanding result of the semantically aligned posterior representation.

[0060] This invention also provides a non-volatile computer storage medium storing computer-executable instructions that can execute the text-alignment-based speech understanding method in any of the above method embodiments. In one embodiment, the non-volatile computer storage medium of the present invention stores computer-executable instructions, which are configured as follows: The training data is input into the speech understanding model, wherein the training data includes training text, and the speech understanding model includes: a CTC posterior simulation module; In the CTC posterior simulation module, the training text is converted into a pseudo-CTC posterior that simulates the characteristics of real audio distribution. The pseudo-CTC posterior is used to generate a pseudo-posterior supervised projection module, which is used to perform projection reasoning on the input speech to obtain a structured semantically aligned posterior representation. The speech understanding result of the semantically aligned posterior representation is determined using a large language model.

[0061] As a non-volatile computer-readable storage medium, it can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the methods in the embodiments of the present invention. One or more program instructions are stored in the non-volatile computer-readable storage medium, and when executed by a processor, the text-aligned speech understanding method in any of the above method embodiments is executed.

[0062] Figure 8 This is a schematic diagram of the hardware structure of an electronic device using a text-aligned speech understanding method according to another embodiment of this application, as shown below. Figure 8 As shown, the device includes: One or more processors 810 and memory 820, Figure 8 Taking a processor 810 as an example, the device based on the text alignment-based speech understanding method may also include an input device 830 and an output device 840.

[0063] The processor 810, memory 820, input device 830, and output device 840 can be connected via a bus or other means. Figure 8 Taking the example of a connection between China and Israel via a bus.

[0064] The memory 820, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the text-aligned speech understanding method in the embodiments of this application. The processor 810 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 820, thereby implementing the text-aligned speech understanding method in the above-described embodiments.

[0065] The memory 820 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store data, etc. Furthermore, the memory 820 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 820 may optionally include memory remotely located relative to the processor 810, and these remote memories can be connected to the mobile device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0066] Input device 830 can receive input numerical or character information. Output device 840 may include display devices such as a display screen.

[0067] The one or more modules are stored in the memory 820, and when executed by the one or more processors 810, they execute the text-aligned speech understanding method in any of the above method embodiments.

[0068] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0069] Non-volatile computer-readable storage media may include a stored program area and a stored data area, wherein the stored program area may store an operating system and an application program required for at least one function; the stored data area may store data created based on the use of the device, etc. Furthermore, the non-volatile computer-readable storage medium may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include memory remotely located relative to the processor, and these remote memories may be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0070] This invention also provides an electronic device comprising: at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the text-aligned speech understanding method of any embodiment of this invention.

[0071] The electronic devices described in this application exist in various forms, including but not limited to: (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include smartphones, multimedia phones, feature phones, and low-end phones.

[0072] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, such as tablet computers.

[0073] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players, handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0074] (4) Other electronic devices with data processing functions.

[0075] In this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising" or "including" include not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0076] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0077] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech understanding method based on text alignment, comprising: The training data is input into the speech understanding model, wherein the training data includes training text, and the speech understanding model includes: a CTC posterior simulation module; In the CTC posterior simulation module, the training text is converted into a pseudo-CTC posterior that simulates the characteristics of real audio distribution. The pseudo-CTC posterior is used to generate a pseudo-posterior supervised projection module, which is used to perform projection reasoning on the input speech to obtain a structured semantically aligned posterior representation. The speech understanding result of the semantically aligned posterior representation is determined using a large language model.

2. The method according to claim 1, wherein, The training data includes: training speech corresponding to the training text; the speech understanding model includes: a label synchronous decoding module. The training method for the speech understanding model includes: In the CTC posterior simulation module, the training text is converted into a pseudo-CTC posterior that simulates the characteristics of real audio distribution. In the tag synchronization decoding module, semantic content is preserved and temporal redundancy of the training speech is compressed to obtain a compact posterior sequence; The pseudo-posterior supervised projection module is trained using the pseudo-CTC posterior and the compact posterior sequence, and trained into a unified shared projection module with text-speech isomorphism. The unified shared projection module is used to perform projection inference on the input speech to obtain a cross-modal bridged semantically aligned posterior representation. The speech understanding result of the semantically aligned posterior representation is determined by a large language model.

3. The method according to claim 1, wherein, In the CTC posterior simulation module, converting the training text into a pseudo-CTC posterior that simulates the characteristics of real audio distribution includes: The training text is subjected to one-hot random smoothing to generate a smoothed posterior that simulates real acoustics; The smoothed posterior portion is randomly deleted to simulate an alignment error scenario of frame loss. Random insertion is performed after the random deletion process to introduce alignment jitter and obtain a pseudo-CTC posterior that simulates the characteristics of real audio distribution.

4. The method according to claim 2, wherein, In the label synchronization decoding module, semantic content is preserved and temporal redundancy of the training speech is compressed to obtain a compact posterior sequence, including: The encoder is used to determine the true CTC posterior of the training speech; The blank frames and adjacent frames in the true CTC posterior are removed to obtain the true compact posterior sequence.

5. The method according to claim 3, wherein, Random insertion following the random deletion process includes: simulating duplicate copies of CTC and blank labels.

6. A text-alignment-based speech understanding system, comprising: An input module is used to input training data into a speech understanding model, wherein the training data includes training text, and the speech understanding model includes a CTC posterior simulation module. The simulation module is used in the CTC posterior simulation module to convert the training text into a pseudo-CTC posterior that simulates the characteristics of real audio distribution. The speech understanding module is used to generate a pseudo-posterior supervised projection module using the pseudo-CTC posterior. The pseudo-posterior supervised projection module is used to perform projection reasoning on the input speech to obtain a structured semantically aligned posterior representation. The speech understanding result of the semantically aligned posterior representation is determined using a large language model.

7. The system according to claim 6, wherein, The training data includes: training speech corresponding to the training text; the speech understanding model includes: a label synchronous decoding module. The training method for the speech understanding model includes: In the CTC posterior simulation module, the training text is converted into a pseudo-CTC posterior that simulates the characteristics of real audio distribution. In the tag synchronization decoding module, semantic content is preserved and temporal redundancy of the training speech is compressed to obtain a compact posterior sequence; The pseudo-posterior supervised projection module is trained using the pseudo-CTC posterior and the compact posterior sequence, and trained into a unified shared projection module with text-speech isomorphism. The unified shared projection module is used to perform projection inference on the input speech to obtain a cross-modal bridged semantically aligned posterior representation. The speech understanding result of the semantically aligned posterior representation is determined by a large language model.

8. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1-5.

9. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1-5.

10. A storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-5.

Images