Language model training method and prediction method

A language model and training method technology, applied in the computer field, can solve problems such as poor effect and low discrimination of confusing words, and achieve the effect of easy distinction

Active Publication Date: 2019-09-03
BEIJING BAIDU NETCOM SCI & TECH CO LTD
View PDF6 Cites 50 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] This application proposes a language model training method and prediction method, which is used to solve the problems in related technologies that the

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Language model training method and prediction method
  • Language model training method and prediction method
  • Language model training method and prediction method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0049] Embodiments of the present application are described in detail below, examples of which are shown in the drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary, and are intended to explain the present application, and should not be construed as limiting the present application.

[0050] The language model training method and prediction method of the embodiments of the present application are described below with reference to the accompanying drawings.

[0051]figure 1 It is a schematic flowchart of a language model training method provided in the embodiment of the present application.

[0052] Such as figure 1 As shown, the training method of the language model includes:

[0053] Step 101, obtain a training text sequence, and randomly generate a word or a target position of a word in the training t...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a language model training method and a prediction method. The training method comprises the following steps of: obtaining training text sequences, randomly generating a target position of a character or a word which needs to be modeled and analyzed through a language model in the training text sequence; carrying out mask replacement on characters or words at a target position; generating a text sequence subjected to mask replacement, generating a limited word list of replaced characters or words, generating probability distribution on a limited word list space accordingto the text sequence subjected to mask replacement and the limited word list, calculating a cross entropy function according to the replaced characters or words and the probability distribution on thelimited word list space, and carrying out iterative optimization. According to the method, the limited word list is introduced to the decoding end of the model, and the original word information is fully utilized during model training, so that confusable words are easier to distinguish by the language model, and the effect of the language model on tasks such as error detection or error correctionis improved.

Description

technical field [0001] The present application relates to the field of computer technology, in particular to a language model training method and prediction method. Background technique [0002] At present, the decoding candidate set of the decoding end of the language model based on Recurrent Neural Network (RNN) is the entire vocabulary space (for example, based on the Chinese character dictionary / word segmentation dictionary), the decoding space of the language model is too large, resulting in model complexity High and difficult to converge, which reduces the degree of discrimination of confusing words and reduces the effect of language models on tasks such as error detection or error correction. Contents of the invention [0003] This application proposes a language model training method and prediction method, which is used to solve the problems in the related art that the RNN-based language model has a reduced degree of discrimination for confusing words and poor resu...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F16/33G06N3/08
CPCG06N3/084G06F16/3344
Inventor 罗希意邓卓彬赖佳伟付志宏何径舟
Owner BEIJING BAIDU NETCOM SCI & TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap