A speech-text joint pre-training method and system

Through multiple iterative training and closed-loop speech chain mechanism, combined with the continuous cumulative distribution mechanism, the loss function is calculated using non-paired and paired data, the problem of insufficient data utilization and unmined alignment relationship in speech text joint pre-training is solved, and the problem of insufficient data utilization and unmined alignment relationship in speech text joint pre-training is improved. Model performance.

CN114842834BActive Publication Date: 2024-11-05INST OF AUTOMATION CHINESE ACAD OF SCI
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210346308.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-31
Publication Date
2024-11-05
Estimated Expiration
2042-03-31

AI Technical Summary

Technical Problem

The problems of insufficient utilization of non-paired data and insufficient mining of paired data alignment relationships in existing speech and text joint pre-training techniques.

Method used

The speech text joint pre-training method with multiple iteration training is adopted, combined with the closed-loop speech chain mechanism and the continuous accumulation distribution mechanism, and the initial, update and final loss totals are calculated through non-paired speech data, non-paired text data and paired speech text data. Function, train joint models to improve model performance.

Benefits of technology

It effectively solves the problems of insufficient utilization of non-paired data and insufficient learning of paired data alignment relationships, and improves the performance of speech text joint pre-trained models, especially in speech recognition and synthesis capabilities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114842834B_ABST
    Figure CN114842834B_ABST
Patent Text Reader

Abstract

The present invention provides a speech-text joint pre-training method and system, comprising: inputting unpaired speech data and unpaired text data into a pre-training joint model to obtain an initial total loss function and an initial training joint model; inputting unpaired speech data, unpaired text data and paired speech-text data into the initial training joint model to obtain an updated total loss function and an updated training joint model; inputting unpaired speech data, unpaired text data and paired speech-text data into the updated training joint model to obtain a final total loss function and a final training joint model. The present invention utilizes unpaired speech data, unpaired text data and paired speech-text data to perform multiple iterative training on the joint model, adopts a closed-loop speech chain mechanism and a continuous cumulative release mechanism, and effectively solves the problem of insufficient utilization of unpaired data and insufficient learning of paired data alignment relationships.
Need to check novelty before this filing date? Find Prior Art