A method and device for optimizing target word vector model

By constructing a loss function containing font information and iterating the training of word vector model parameters, the training efficiency and performance of word vector model is solved in the existing technology caused by the unreasonable introduction of font information in the existing technology is improved.

CN114528834BActive Publication Date: 2024-11-01NORTH CHINA ELECTRICAL POWER RES INST +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210142079.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-02-16
Publication Date
2024-11-01
Estimated Expiration
2042-02-16

AI Technical Summary

Technical Problem

In the prior art, the algorithm selection of the word vector model for introducing font information is unreasonable, resulting in low training efficiency and lack of appropriate training methods.

Method used

By pre-initializing the parameter matrix, converting the single-hot encoding in the dictionary into an M-dimensional column vector, building a loss function containing font information, and combining the original loss function to build an optimized loss function, iteratively train the target word vector model parameters to improve training efficiency .

Benefits of technology

Effectively introduce font information into the word vector model, improving the training efficiency of the target word vector model and optimizing the model performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114528834B_ABST
    Figure CN114528834B_ABST
Patent Text Reader

Abstract

The present invention provides a method and device for optimizing a target word vector model, the method comprising: converting the one-hot codes corresponding to each word in a dictionary into an M-dimensional column vector according to a pre-initialized parameter matrix, determining a loss function containing font information according to the one-hot codes, the parameter matrix, the M-dimensional column vector, and the prediction results of each word output by the target word vector model, and constructing an optimized loss function according to the loss function and the original loss function corresponding to the target word vector model; according to a method for iteratively training the parameters of the target word vector model according to the original loss function, iteratively training the parameters of the target word vector model and the parameters of the parameter matrix according to the optimized loss function, and obtaining an optimized target word vector model after the training is completed. The device is used to execute the above method. The method and device provided by the embodiments of the present invention improve the training efficiency of the optimized target word vector model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of word vector technology, and in particular to a method and device for optimizing a target word vector model. Background Art

[0002] Word embedding is a general term for a set of language modeling and feature learning techniques in natural language processing (NLP), in which words or phrases from a vocabulary are mapped to vectors of real numbers. Conceptually, it involves mathematical embedding from a one-dimensional space per word to a continuous vector space with lower dimensions. When used as the underlying input representation, word and phrase embeddings have been shown to improve the performance of NLP tasks such as grammatical parsing and sentiment analysis.

[0003] In short, word vector technology converts words into dense vectors, and for similar words, the corresponding word vectors are also similar. In natural language processing tasks, we first need to consider how words are represented in computers. Generally, there are two representation methods: one-hot representation and distribution representation. Traditional rule-based or statistical-based natural semantic processing methods treat words as atomic symbols, which is called one-hot representation. One-hot representation represents each word as a long vector. The dimension of this vector is the size of the vocabulary. Only one dimension in the vector has a value of 1, and the rest are 0. This dimension represents the current word. It is equivalent to assigning an id to each word, which means that this representation method cannot show the relationship between words. In addition, it will cause the feature space to be very large, but it also brings a benefit, that is, in high-dimensional space, many application tasks are linearly separable. Distributed representation refers to converting words into a distributed representation, also known as word vector. Distributed representation represents words as a fixed-length continuous dense vector. The advantages of distributed representation are as follows:

[0004] (1) There is a similarity between words: There is a concept of "distance" between words, which is very helpful for many natural language processing tasks. (2) Contains more information: Word vectors can contain more information, and each dimension has a specific meaning. When using one-hot features, feature vectors can be pruned, but word vectors cannot.

[0005] The language model generates word vectors by training the neural network language model, and the word vectors are the incidental output of the language model. The basic idea behind it is to predict the words that appear in the context. This prediction of the context is essentially a learning of co-occurrence statistical features.

[0006] Font information plays a key role in word recognition. The existing technology introduces font information into word vectors, but the algorithm selection as the algorithm skeleton is unreasonable and there is no suitable training method, which leads to low algorithm training efficiency. Summary of the invention

[0007] In response to the problems in the prior art, an embodiment of the present invention provides a target word vector model optimization method and device, which can at least partially solve the problems existing in the prior art.

[0008] On the one hand, the present invention proposes a target word vector model optimization method, comprising:

[0009] According to a pre-initialized parameter matrix, the one-hot codes corresponding to each word in the dictionary are converted into M-dimensional column vectors, where M is the number of columns of the parameter matrix; the dimension of the one-hot code is determined according to a preset code value range corresponding to the font input method, and the number of the one-hot codes is determined according to a preset number of characters included in the preset word and a preset font code bit number corresponding to the font input method;

[0010] Determine a loss function including font information according to the one-hot encoding, the parameter matrix, the M-dimensional column vector, and the prediction result of each word output by the target word vector model, and construct an optimized loss function according to the loss function and the original loss function corresponding to the target word vector model;

[0011] According to the method of iteratively training the target word vector model parameters according to the original loss function, the target word vector model parameters and the parameters of the parameter matrix are iteratively trained according to the optimized loss function, and the optimized target word vector model is obtained after the training is completed.

[0012] Wherein, determining a loss function containing font information according to the one-hot encoding, the parameter matrix, the M-dimensional column vector, and the prediction result of each word output by the target word vector model includes:

[0013] Obtain a prediction result vector according to the M-dimensional column vector and the prediction result of each word output by the target word vector model;

[0014] Obtaining a correct result vector according to the one-hot encoding corresponding to the words of the predetermined correct result and the parameter matrix;

[0015] A loss function including font information is determined according to the predicted result vector and the correct result vector.

[0016] The step of constructing an optimized loss function according to the loss function and the original loss function corresponding to the target word vector model includes:

[0017] An optimized loss function is constructed according to the loss function, the original loss function corresponding to the target word vector model and the loss function weight.

[0018] Wherein, the target word vector model includes a Skip-gram model or a CBOW model; accordingly, the optimized target word vector model obtained after the training is completed includes:

[0019] After the training is completed, an optimized Skip-gram model is obtained; or after the training is completed, an optimized CBOW model is obtained.

[0020] Wherein, after obtaining the optimized Skip-gram model; or obtaining the optimized CBOW model, the target word vector model optimization method further includes:

[0021] Inputting a word to be predicted into the optimized Skip-gram model, and using an output result of the optimized Skip-gram model as a context prediction result corresponding to the word to be predicted;

[0022] Or input the context to be predicted into the optimized CBOW model, and use the output result of the optimized CBOW model as the word prediction result corresponding to the context to be predicted.

[0023] Among them, if the number of the font input methods is greater than 1, an optimized loss function is constructed according to the number of the font input methods, the loss function, the original loss function, and the loss function weight.

[0024] On the one hand, the present invention provides a target word vector model optimization device, comprising:

[0025] An acquisition unit, used for converting the one-hot codes corresponding to each word in the dictionary into an M-dimensional column vector according to a pre-initialized parameter matrix, where M is the number of columns of the parameter matrix; the dimension of the one-hot code is determined according to a preset code value range corresponding to the font input method, and the number of the one-hot codes is determined according to a preset number of characters included in the preset word and a preset font code bit number corresponding to the font input method;

[0026] A construction unit, configured to determine a loss function including font information according to the one-hot encoding, the parameter matrix, the M-dimensional column vector, and the prediction result of each word output by the target word vector model, and to construct an optimized loss function according to the loss function and an original loss function corresponding to the target word vector model;

[0027] The optimization unit is used to iteratively train the target word vector model parameters and the parameters of the parameter matrix according to the optimized loss function in accordance with the method of iteratively training the target word vector model parameters according to the original loss function, and obtain the optimized target word vector model after the training is completed.

[0028] In another aspect, an embodiment of the present invention provides an electronic device, including: a processor, a memory, and a bus, wherein:

[0029] The processor and the memory communicate with each other via the bus;

[0030] The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the following method:

[0031] According to a pre-initialized parameter matrix, the one-hot codes corresponding to each word in the dictionary are converted into M-dimensional column vectors, where M is the number of columns of the parameter matrix; the dimension of the one-hot code is determined according to a preset code value range corresponding to the font input method, and the number of the one-hot codes is determined according to a preset number of characters included in the preset word and a preset font code bit number corresponding to the font input method;

[0032] Determine a loss function including font information according to the one-hot encoding, the parameter matrix, the M-dimensional column vector, and the prediction result of each word output by the target word vector model, and construct an optimized loss function according to the loss function and the original loss function corresponding to the target word vector model;

[0033] According to the method of iteratively training the target word vector model parameters according to the original loss function, the target word vector model parameters and the parameters of the parameter matrix are iteratively trained according to the optimized loss function, and the optimized target word vector model is obtained after the training is completed.

[0034] An embodiment of the present invention provides a non-transitory computer-readable storage medium, including:

[0035] The non-transitory computer-readable storage medium stores computer instructions, which cause the computer to execute the following method:

[0036] According to a pre-initialized parameter matrix, the one-hot codes corresponding to each word in the dictionary are converted into M-dimensional column vectors, where M is the number of columns of the parameter matrix; the dimension of the one-hot code is determined according to a preset code value range corresponding to the font input method, and the number of the one-hot codes is determined according to a preset number of characters included in the preset word and a preset font code bit number corresponding to the font input method;

[0037] Determine a loss function including font information according to the one-hot encoding, the parameter matrix, the M-dimensional column vector, and the prediction result of each word output by the target word vector model, and construct an optimized loss function according to the loss function and the original loss function corresponding to the target word vector model;

[0038] According to the method of iteratively training the target word vector model parameters according to the original loss function, the target word vector model parameters and the parameters of the parameter matrix are iteratively trained according to the optimized loss function, and the optimized target word vector model is obtained after the training is completed.

[0039] The target word vector model optimization method and device provided by the embodiment of the present invention converts the one-hot codes corresponding to each word in the dictionary into an M-dimensional column vector according to a pre-initialized parameter matrix, determines the loss function containing font information according to the one-hot code, the parameter matrix, the M-dimensional column vector, and the prediction results of each word output by the target word vector model, and constructs an optimized loss function according to the loss function and the original loss function corresponding to the target word vector model; according to the method of iteratively training the target word vector model parameters according to the original loss function, iteratively trains the target word vector model parameters and the parameters of the parameter matrix according to the optimized loss function, and obtains the optimized target word vector model after the training is completed. The font information is introduced into the target word vector model, and the training efficiency of the optimized target word vector model is improved through an adaptive training method. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0041] Figure 1 It is a flowchart of a target word vector model optimization method provided by an embodiment of the present invention.

[0042] Figure 2 It is a structural diagram of a target word vector model optimization device provided by one embodiment of the present invention.

[0043] Figure 3 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical scheme and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention. It should be noted that, in the absence of conflict, the embodiments in this application and the features in the embodiments can be combined with each other arbitrarily.

[0045] Figure 1FIG. 1 is a flow chart of a target word vector model optimization method provided by an embodiment of the present invention. Figure 1 As shown, the target word vector model optimization method provided by the embodiment of the present invention includes:

[0046] Step S1: Obtain the one-hot encoding corresponding to each word in the dictionary according to a pre-initialized parameter matrix and convert it into an M-dimensional column vector, where M is the number of columns of the parameter matrix; the dimension of the one-hot encoding is determined according to a preset encoding value range corresponding to the font input method, and the number of the one-hot encoding is determined according to a preset number of characters contained in a preset word and a preset number of font encoding bits corresponding to the font input method.

[0047] Step S2: Determine a loss function containing font information based on the one-hot encoding, the parameter matrix, the M-dimensional column vector, and the prediction results of each word output by the target word vector model, and construct an optimized loss function based on the loss function and the original loss function corresponding to the target word vector model.

[0048] Step S3: According to the method of iteratively training the target word vector model parameters according to the original loss function, the target word vector model parameters and the parameters of the parameter matrix are iteratively trained according to the optimized loss function, and the optimized target word vector model is obtained after the training is completed.

[0049] In the above step S1, the device converts the one-hot encoding corresponding to each word in the dictionary into an M-dimensional column vector according to a pre-initialized parameter matrix, where M is the number of columns of the parameter matrix; the dimension of the one-hot encoding is determined according to a preset encoding value range corresponding to the font input method, and the number of the one-hot encoding is determined according to the preset number of characters contained in the preset word and the preset font encoding bit number corresponding to the font input method. The device may include a computer device, etc., for executing the method.

[0050] The font information involved in the embodiment of the present invention can be obtained from an existing font input method. These font input methods include Wubi font input method, Sijiao number input method and stroke input method, etc., and one of them can be selected, or several of them can be selected to combine. Since some font input methods can encode a single Chinese character or a word, for the sake of uniformity, the encoding information of a single Chinese character is used in the embodiment of the present invention. The font information used in the embodiment of the present invention does not need to introduce additional training corpus, but comes directly from the existing font input method, and has strong applicability.

[0051] When constructing a word vector for Chinese, a word vector can be constructed for a single Chinese character, or a word vector can be constructed for a word after the word segmentation algorithm is used to segment the corpus. Since the encoding information of a single Chinese character in the input method is used in the embodiment of the present invention, the word vector constructed for a single Chinese character is the basis, and the word vector constructed for a word is its extension and expansion. The extension and expansion method adopted in the embodiment of the present invention is to determine the maximum length of the word, and to combine the font encoding sequence of each character constituting the word to form the font encoding of the word. If the number of characters in the word is greater than the maximum length, a truncation method is adopted. If the number of characters in the word is less than the maximum length, a padding method is adopted, and the padding code is a character other than the font encoding.

[0052] The following uses the four-corner number input method as an example to introduce a method for obtaining a word vector that contains font information.

[0053] The four-corner number input method uses four or five digits to encode each font, and each digit is one of the ten numbers from 0 to 9. For the sake of uniformity, the encoding of each font is set to five digits, and the font with a four-digit encoding has an additional code added at the end. In this way, the preset number of characters contained in the preset word is N, and the four-corner number encoding can obtain a unique hot encoding with a dimension of 12 consisting of 5N. The "unique hot encoding with a dimension of 12" includes 10 dimensions formed by 0 to 9, one dimension of the aforementioned padding code, and one dimension of the aforementioned additional code. When N is 1, it is converted into a word vector constructed for a single font.

[0054] That is, before obtaining the word vector, a 60N(5N×12)×M matrix is ​​randomly initialized in advance. The function of this matrix is ​​to convert the 5N sequentially arranged one-hot encodings with a dimension of 12 into a column vector with a dimension of M. The font input method reflects the font information through its encoding, which has the characteristic of one-hot encoding. In order to be able to convert it into a continuous encoding, an additional embedding layer is introduced by randomly initializing a 60N×M matrix. This layer adopts the same method in the entire word vector training process and is trained synchronously to improve the training efficiency.

[0055] In the above step S2, the device determines a loss function containing font information according to the one-hot encoding, the parameter matrix, the M-dimensional column vector, and the prediction results of each word output by the target word vector model, and constructs an optimized loss function according to the loss function and the original loss function corresponding to the target word vector model. The target word vector model can be understood as a model suitable for identifying word vectors, which can include a Skip-gram model or a CBOW model.

[0056] Furthermore, the prediction result of each word outputted according to the one-hot encoding, the parameter matrix, the M-dimensional column vector, and the target word vector model includes:

[0057] According to the M-dimensional column vector and the prediction results of each word output by the target word vector model, a prediction result vector is obtained; let the total number of words in the dictionary in the Skip-gram model or CBOW model (if the number of characters of the limited word is 1, the total number of words) be V. The unique hot code corresponding to each word can be determined when performing one-hot representation.

[0058] The prediction results output by the Skip-gram model or the CBOW model can be further normalized, for example, y1, y2, ...y k …y V , respectively represent the probability that the predicted word is the kth word in the total words of the dictionary.

[0059] Since each word corresponds to a 60N (5N × 12) unique hot code, and after multiplication with the aforementioned 60N × M matrix, it is converted into an M-dimensional vector, all the words in the dictionary form a total of V M-dimensional vectors. k …y V The V M-dimensional vectors are weighted and summed, and then the results are normalized to obtain the prediction result vector p.

[0060] The correct result vector is obtained based on the one-hot encoding corresponding to the words of the predetermined correct result and the parameter matrix; the 60N (5N×12) one-hot encoding corresponding to the words of the correct result is multiplied with the aforementioned 60N×M dimensional matrix, converted into an M dimensional vector, and then normalized to obtain the correct result vector q.

[0061] A loss function including font information is determined according to the predicted result vector and the correct result vector.

[0062] Furthermore, determining a loss function including font information according to the prediction result vector and the correct result vector includes:

[0063] The loss function containing font information is determined according to the following formula:

[0064] The loss function = 1 - the predicted result vector × the correct result vector.

[0065] Furthermore, constructing an optimized loss function according to the loss function and an original loss function corresponding to the target word vector model includes:

[0066] According to the loss function, the original loss function corresponding to the target word vector model and the loss function weight, an optimized loss function is constructed, specifically including:

[0067] The optimization loss function is constructed according to the following formula:

[0068] The optimized loss function = a × the loss function + (1-a) × the original loss function;

[0069] Among them, a is the weight of the loss function, which can be selected as any value between 0 and 1, and the expression of the original loss function can be selected as a loss function expression known in the art, which will not be repeated in the embodiments of the present invention.

[0070] The weight of the loss function of font information can be adjusted according to actual needs. For example, in some professional fields, such as medicine and chemical engineering, the Chinese characters of professional vocabulary in these fields often have the same radicals or fonts. In this case, the weight can be increased appropriately. In some general fields, a smaller weight can be selected.

[0071] In the above step S3, the device iteratively trains the target word vector model parameters and the parameters of the parameter matrix according to the optimized loss function in accordance with the method of iteratively training the target word vector model parameters according to the original loss function, and obtains the optimized target word vector model after the training is completed.

[0072] The method of iteratively training the target word vector model parameters according to the original loss function is a known method in the art and will not be described in detail in the embodiments of the present invention.

[0073] It should be noted that the parameters of the parameter matrix are iteratively trained during the first training, and then the parameter matrix is ​​updated to obtain a 60N×M dimensional matrix, which is updated once for each iterative training. The iterative training of the target word vector model parameters can refer to the above description of iterative training of the parameters of the parameter matrix, which will not be repeated here. After the final training is completed, the optimized target model formed includes both the context information of the corpus and part of the font information introduced by the four-corner number encoding.

[0074] When applying other font encodings, it is only necessary to modify the above 5N×12 unique hot encoding. 5 corresponds to the preset font encoding bit number, and when using Wubi encoding, it is modified to 4. 12 corresponds to the preset encoding value range, and when using Wubi encoding, it is modified to 27 (A to Y is 25 bits, 1 bit of padding encoding, and 1 bit of additional encoding).

[0075] When multiple font encodings are used, each font encoding corresponds to a loss function. The weight of each loss function is a / K, where K is the number of font encodings used.

[0076] Furthermore, if the number of the font input methods is greater than 1, an optimized loss function is constructed according to the number of the font input methods, the loss function, the original loss function, and the loss function weight.

[0077] Specifically include:

[0078] The optimization loss function is constructed according to the following formula:

[0079] The optimized loss function = (a / K) × ∑ the i-th loss function + (1-a) × the original loss function;

[0080] Among them, K is the number of font input methods, i is the i-th font input method, and each font input method corresponds to a loss function.

[0081] The target word vector model optimization method provided by the embodiment of the present invention converts the one-hot codes corresponding to each word in the dictionary into an M-dimensional column vector according to a pre-initialized parameter matrix, determines the loss function containing font information according to the one-hot code, the parameter matrix, the M-dimensional column vector, and the prediction results of each word output by the target word vector model, and constructs an optimized loss function according to the loss function and the original loss function corresponding to the target word vector model; according to the method of iteratively training the parameters of the target word vector model according to the original loss function, iteratively trains the parameters of the target word vector model and the parameters of the parameter matrix according to the optimized loss function, and obtains the optimized target word vector model after the training is completed. The font information is introduced into the target word vector model, and the training efficiency of the optimized target word vector model is improved through an adaptive training method.

[0082] Figure 2 is a schematic diagram of the structure of a target word vector model optimization device provided by an embodiment of the present invention. Figure 2 As shown, the target word vector model optimization device provided by the embodiment of the present invention includes an acquisition unit 201, a construction unit 202 and an optimization unit 203, wherein:

[0083] The acquisition unit 201 is used to convert the one-hot codes corresponding to each word in the dictionary into an M-dimensional column vector according to a pre-initialized parameter matrix, where M is the number of columns of the parameter matrix; the dimension of the one-hot code is determined according to the preset code value range corresponding to the font input method, and the number of the one-hot codes is determined according to the preset number of characters contained in the preset word and the preset font code bit number corresponding to the font input method; the construction unit 202 is used to determine the loss function containing font information according to the one-hot code, the parameter matrix, the M-dimensional column vector, and the prediction results of each word output by the target word vector model, and to construct an optimized loss function according to the loss function and the original loss function corresponding to the target word vector model; the optimization unit 203 is used to iteratively train the target word vector model parameters and the parameters of the parameter matrix according to the optimized loss function according to the method of iteratively training the target word vector model parameters according to the original loss function, and obtain the optimized target word vector model after the training is completed.

[0084] Specifically, the acquisition unit 201 in the target word vector model optimization device is used to convert the one-hot encoding corresponding to each word in the dictionary into an M-dimensional column vector according to a pre-initialized parameter matrix, where M is the number of columns of the parameter matrix; the dimension of the one-hot encoding is determined according to the preset encoding value range corresponding to the font input method, and the number of the one-hot encodings is determined according to the preset number of characters contained in the preset words and the preset font encoding bit number corresponding to the font input method; the construction unit 202 is used to determine the loss function containing font information according to the one-hot encoding, the parameter matrix, the M-dimensional column vector, and the prediction results of each word output by the target word vector model, and to construct an optimized loss function according to the loss function and the original loss function corresponding to the target word vector model; the optimization unit 203 is used to iteratively train the target word vector model parameters and the parameters of the parameter matrix according to the optimized loss function according to the method of iteratively training the target word vector model parameters according to the original loss function, and obtain the optimized target word vector model after the training is completed.

[0085] The target word vector model optimization device provided by the embodiment of the present invention converts the one-hot codes corresponding to each word in the dictionary into an M-dimensional column vector according to a pre-initialized parameter matrix, determines the loss function containing font information according to the one-hot code, the parameter matrix, the M-dimensional column vector, and the prediction results of each word output by the target word vector model, and constructs an optimized loss function according to the loss function and the original loss function corresponding to the target word vector model; according to the method of iteratively training the parameters of the target word vector model according to the original loss function, iteratively trains the parameters of the target word vector model and the parameters of the parameter matrix according to the optimized loss function, and obtains the optimized target word vector model after the training is completed. The font information is introduced into the target word vector model, and the training efficiency of the optimized target word vector model is improved through an adaptive training method.

[0086] The embodiment of the target word vector model optimization device provided by the embodiment of the present invention can be specifically used to execute the processing flow of the above-mentioned method embodiments. Its functions will not be repeated here, and reference can be made to the detailed description of the above-mentioned method embodiments.

[0087] Figure 3 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention, such as Figure 3 As shown, the electronic device includes: a processor (processor) 301, a memory (memory) 302 and a bus 303;

[0088] The processor 301 and the memory 302 communicate with each other via a bus 303;

[0089] The processor 301 is used to call the program instructions in the memory 302 to execute the methods provided by the above method embodiments, for example, including:

[0090] According to a pre-initialized parameter matrix, the one-hot codes corresponding to each word in the dictionary are converted into M-dimensional column vectors, where M is the number of columns of the parameter matrix; the dimension of the one-hot code is determined according to a preset code value range corresponding to the font input method, and the number of the one-hot codes is determined according to a preset number of characters included in the preset word and a preset font code bit number corresponding to the font input method;

[0091] Determine a loss function including font information according to the one-hot encoding, the parameter matrix, the M-dimensional column vector, and the prediction result of each word output by the target word vector model, and construct an optimized loss function according to the loss function and the original loss function corresponding to the target word vector model;

[0092] According to the method of iteratively training the target word vector model parameters according to the original loss function, the target word vector model parameters and the parameters of the parameter matrix are iteratively trained according to the optimized loss function, and the optimized target word vector model is obtained after the training is completed.

[0093] This embodiment discloses a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer can perform the methods provided by the above method embodiments, for example, including:

[0094] According to a pre-initialized parameter matrix, the one-hot codes corresponding to each word in the dictionary are converted into M-dimensional column vectors, where M is the number of columns of the parameter matrix; the dimension of the one-hot code is determined according to a preset code value range corresponding to the font input method, and the number of the one-hot codes is determined according to a preset number of characters included in the preset word and a preset font code bit number corresponding to the font input method;

[0095] Determine a loss function including font information according to the one-hot encoding, the parameter matrix, the M-dimensional column vector, and the prediction result of each word output by the target word vector model, and construct an optimized loss function according to the loss function and the original loss function corresponding to the target word vector model;

[0096] According to the method of iteratively training the target word vector model parameters according to the original loss function, the target word vector model parameters and the parameters of the parameter matrix are iteratively trained according to the optimized loss function, and the optimized target word vector model is obtained after the training is completed.

[0097] This embodiment provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program enables the computer to execute the methods provided by the above method embodiments, for example, including:

[0098] According to a pre-initialized parameter matrix, the one-hot codes corresponding to each word in the dictionary are converted into M-dimensional column vectors, where M is the number of columns of the parameter matrix; the dimension of the one-hot code is determined according to a preset code value range corresponding to the font input method, and the number of the one-hot codes is determined according to a preset number of characters included in the preset word and a preset font code bit number corresponding to the font input method;

[0099] Determine a loss function including font information according to the one-hot encoding, the parameter matrix, the M-dimensional column vector, and the prediction result of each word output by the target word vector model, and construct an optimized loss function according to the loss function and the original loss function corresponding to the target word vector model;

[0100] According to the method of iteratively training the target word vector model parameters according to the original loss function, the target word vector model parameters and the parameters of the parameter matrix are iteratively trained according to the optimized loss function, and the optimized target word vector model is obtained after the training is completed.

[0101] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0102] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0103] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0104] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0105] In the description of this specification, the description with reference to the terms "one embodiment", "a specific embodiment", "some embodiments", "for example", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0106] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A target word vector model optimization method, characterized in that: include: According to the pre-initialized parameter matrix, the one-hot encoding corresponding to each word in the dictionary is converted into an M-dimensional column vector, where M is the number of columns of the parameter matrix; The dimension of the one-hot encoding is determined according to a preset encoding value range corresponding to the font input method, and the number of the one-hot encoding is determined according to a preset number of characters included in a preset word and a preset number of font encoding bits corresponding to the font input method; Determine a loss function including font information according to the one-hot encoding, the parameter matrix, the M-dimensional column vector, and the prediction result of each word output by the target word vector model, and construct an optimized loss function according to the loss function and the original loss function corresponding to the target word vector model; According to the method of iteratively training the target word vector model parameters according to the original loss function, the target word vector model parameters and the parameters of the parameter matrix are iteratively trained according to the optimized loss function, and the optimized target word vector model is obtained after the training is completed; Determining a loss function including font information according to the one-hot encoding, the parameter matrix, the M-dimensional column vector, and the prediction result of each word output by the target word vector model includes: Obtain a prediction result vector according to the M-dimensional column vector and the prediction result of each word output by the target word vector model; Obtaining a correct result vector according to the one-hot encoding corresponding to the words of the predetermined correct result and the parameter matrix; Determining a loss function including font information according to the prediction result vector and the correct result vector; The step of determining a loss function including font information according to the prediction result vector and the correct result vector comprises: The loss function containing font information is determined according to the following formula: The loss function=1-the predicted result vector×the correct result vector; The step of constructing an optimized loss function according to the loss function and the original loss function corresponding to the target word vector model includes: According to the loss function, the original loss function corresponding to the target word vector model and the loss function weight, an optimized loss function is constructed, specifically including: The optimization loss function is constructed according to the following formula: The optimized loss function = a × the loss function + (1-a) × the original loss function; Among them, a is the weight of the loss function.

2. The target word vector model optimization method according to claim 1, characterized in that: The target word vector model includes a Skip-gram model or a CBOW model; accordingly, the optimized target word vector model obtained after the training is completed includes: After the training is completed, an optimized Skip-gram model is obtained; or after the training is completed, an optimized CBOW model is obtained.

3. The target word vector model optimization method according to claim 2, characterized in that: After obtaining the optimized Skip-gram model; or obtaining the optimized CBOW model, the target word vector model optimization method further includes: Inputting a word to be predicted into the optimized Skip-gram model, and using an output result of the optimized Skip-gram model as a context prediction result corresponding to the word to be predicted; Or input the context to be predicted into the optimized CBOW model, and use the output result of the optimized CBOW model as the word prediction result corresponding to the context to be predicted.

4. The target word vector model optimization method according to claim 1, characterized in that: If the number of the font input methods is greater than 1, an optimized loss function is constructed according to the number of the font input methods, the loss function, the original loss function, and the loss function weight.

5. A target word vector model optimization device, characterized in that: include: An acquisition unit, used for converting the one-hot codes corresponding to each word in the dictionary into an M-dimensional column vector according to a pre-initialized parameter matrix, where M is the number of columns of the parameter matrix; the dimension of the one-hot code is determined according to a preset code value range corresponding to the font input method, and the number of the one-hot codes is determined according to a preset number of characters included in the preset word and a preset font code bit number corresponding to the font input method; A construction unit, configured to determine a loss function including font information according to the one-hot encoding, the parameter matrix, the M-dimensional column vector, and the prediction result of each word output by the target word vector model, and to construct an optimized loss function according to the loss function and an original loss function corresponding to the target word vector model; An optimization unit, configured to iteratively train the target word vector model parameters and the parameters of the parameter matrix according to the optimized loss function in accordance with the method of iteratively training the target word vector model parameters according to the original loss function, and obtain an optimized target word vector model after the training is completed; Determining a loss function including font information according to the one-hot encoding, the parameter matrix, the M-dimensional column vector, and the prediction result of each word output by the target word vector model includes: Obtain a prediction result vector according to the M-dimensional column vector and the prediction result of each word output by the target word vector model; Obtaining a correct result vector according to the one-hot encoding corresponding to the words of the predetermined correct result and the parameter matrix; Determining a loss function including font information according to the prediction result vector and the correct result vector; The step of determining a loss function including font information according to the prediction result vector and the correct result vector comprises: The loss function containing font information is determined according to the following formula: The loss function=1-the predicted result vector×the correct result vector; The step of constructing an optimized loss function according to the loss function and the original loss function corresponding to the target word vector model includes: According to the loss function, the original loss function corresponding to the target word vector model and the loss function weight, an optimized loss function is constructed, specifically including: The optimization loss function is constructed according to the following formula: The optimized loss function = a × the loss function + (1-a) × the original loss function; Among them, a is the weight of the loss function.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.