基于预训练模型的中文跨语言知识增强方法
By constructing a bilingual dictionary to acquire initial consonant and translation knowledge, and combining a tree-structured attention mechanism and BERT to generate embeddings, the shortcomings of Chinese pre-trained models in the fusion of polyphonic characters and cross-linguistic knowledge are addressed, thereby improving the performance of Chinese sentiment analysis and other tasks.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUILIN UNIV OF ELECTRONIC TECH
- Filing Date
- 2023-10-26
- Publication Date
- 2026-07-17
AI Technical Summary
Existing Chinese pre-trained models have shortcomings in understanding text semantics and Chinese character features, especially in the problem of polyphonic characters. At the same time, they ignore the characteristics of Chinese itself when integrating cross-language knowledge, which limits the effectiveness of sentiment analysis and other natural language processing tasks.
By constructing a bilingual dictionary to acquire initial consonant and translation knowledge, we calculate attention scores using tree-based attention and variable attention mechanisms, and combine BERT to generate embeddings. We then use singular value decomposition and focus loss function for fine-tuning to enhance the semantic information of the Chinese distributed representation.
It significantly improves the model's performance in sentiment analysis, named entity recognition, natural language inference, and domain question answering tasks, reaching state-of-the-art levels. It also solves the problem of polyphonic characters and enhances the semantic representation of Chinese word embeddings.
Smart Images
Figure CN117648935B_ABST