Machine learning natural language processing and generation

Abstract

A machine learning natural language generation computing system utilizes single computing hardware device (GPU) to train or pretrain MLNLG models from scratch, allowing pretraining of MLNLG generative language models with extended sequence length context size in both human language and computer code. The training method employs a scaled version of RoPE to compress token position indices, to manage longer sequences beyond the computing hardware devices memory capacity. The system also features a novel tokenization technique, combining greedy subword BPE and Unigram algorithms for enhanced grammatically semantic and syntactic accurate language generation, particularly for mid and low-resource languages. By implementing weight tying and sensitivity pruning, the system reduces model parameters compared to LLMs, eliminating the need for multiple GPUs. This system preserves performance without weight quantization, allowing low-latency inference even on CPUs, resulting in a lightweight and efficient training and generative inference process on a single computing hardware device.

Images