A model acceleration training method and device based on prior knowledge word vectorization

Abstract

The present invention provides a model acceleration training method and device based on prior knowledge word vectorization. When a corpus is vectorized, a priori part and a knowledge vector to be learned are constructed. The priori part determines the vector dimension based on a statistical method, and the prior knowledge is integrated to reduce the model learning burden and improve efficiency. During the training process, the priori part is fixed first to update the model and the knowledge vector to be learned, and then the priori part is fine-tuned. The knowledge vector to be learned is used to synchronously update the model during the training process, so as to capture implicit information, improve the understanding of the context, enhance the generalization ability to handle ambiguity and accelerate the convergence of the model.

Description

Technical Field

[0001] The present invention relates to the field of machine learning technology, and in particular to a model acceleration training method and device based on prior knowledge word vectorization. Background Art

[0002] With the development of the Internet and information technology, natural language processing (NLP) has become an important branch of artificial intelligence, playing a vital role in search engines, machine translation, sentiment analysis, intelligent customer service, and many other scenarios. As a core technology in NLP, word vectors effectively capture the semantic information of words by mapping them to vectors in a high-dimensional space, providing strong support for various NLP tasks.

[0003] Although existing word vectorization methods, such as Word2Vec and GloVe, have achieved remarkable results in many NLP tasks, they still have some limitations. They are mainly reflected in the following aspects: 1) Low training efficiency. Traditional word vector training methods require a large amount of corpus for iterative learning, and the training process is time-consuming, especially on large-scale corpora. 2) Convergence difficulties. During the training process, it takes a long time for word vectors to reach a convergence state, and even the gradient vanishing problem occurs, which directly affects the training efficiency of natural language models. 3) Generalization ability. The model needs to learn text knowledge from scratch. The number of required parameters is large and the model complexity is high. It is easy to learn overly specific patterns for training samples. Therefore, there is an urgent need for a natural language model acceleration training method combined with a new word vectorization scheme to improve the performance of model training. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a model acceleration training method and device based on prior knowledge word vectorization to eliminate or improve one or more defects existing in the prior art and solve the problems of low efficiency, difficult convergence and weak generalization ability of the natural language model training process in the prior art.

[0005] One aspect of the present invention provides a model acceleration training method based on prior knowledge word vectorization, the method comprising the following steps:

[0006] Performing statistical analysis on the part of speech, word frequency and position of words in each sentence in a preset corpus according to multiple preset dimensions, wherein the part of speech is divided into multiple dimensions according to the part of speech;

[0007] Standardizing the statistical analysis results of the dimensions related to the part of speech, and determining the reserved dimensions for the part of speech by performing dimensionality reduction based on principal component analysis, and combining the reserved dimensions, the word frequency, and the preset dimensions related to the position as a priori dimensions;

[0008] Constructing a part-of-speech vector, a word frequency vector and a position vector for the words in the preset corpus according to the prior dimension;

[0009] Based on the data set with parts of speech marked in the preset corpus, the probability of the part of speech belonging to each position of the sentence is counted, the part of speech vector of the sentence is constructed for the sentence using the part of speech distribution probability, and the part of speech vector is mapped and replaced based on the mapping relationship obtained by dimensionality reduction analysis of the part of speech to obtain a grammatical vector;

[0010] A knowledge vector to be learned of a set dimension is configured for each word and initialized, the part-of-speech vector, the word frequency vector, the position vector and the grammatical vector corresponding to each word are used as a priori parts, and the priori parts corresponding to each word and the knowledge vector to be learned are jointly constructed into a word vector;

[0011] A plurality of sentences in the preset corpus after word vectorization are constructed as a training sample set, and a target natural language model is trained; for each batch of training data, the prior part of each word vector in the training sample set is locked in the first training stage, and the parameters of the target natural language model and the knowledge vector to be learned corresponding to each word vector are updated using a gradient descent algorithm until the preset performance index of the target neural network model on the test set converges and the second training stage is executed, and the prior part corresponding to each word vector is fine-tuned in the second training stage.

[0012] In some embodiments, the parts of speech include content words, function words, interjections and onomatopeia; content words are divided into 8 small categories including nouns, verbs, adjectives, numerals, quantifiers, pronouns, adverbs and onomatopeia; function words are divided into 5 small categories including prepositions, conjunctions, auxiliary words, interjections and modal particles, and are marked with a 13-dimensional vector; the word frequency is marked with a 1-dimensional vector; the position is marked according to the dimension established according to the length of the sentence in the preset corpus.

[0013] In some embodiments, the statistical analysis results of the dimensions related to the part of speech are standardized, and the dimension reduction is performed based on the principal component analysis to determine the retained dimensions for the part of speech, including:

[0014] Counting the part-of-speech distribution of each word in the preset corpus and expressing it as a matrix, wherein the rows of the matrix correspond to words, the columns correspond to the classification dimensions of the part-of-speech, and the value of each cell is the number of times the word appears in the preset corpus with the corresponding part-of-speech;

[0015] Standardize the data of each part-of-speech column so that its mean is 0 and its variance is 1;

[0016] Calculating the covariance matrix of the matrix after data standardization;

[0017] For the eigenvalues ​​of the covariance matrix solution and their corresponding contribution rates, a set number of principal components whose contribution rates reach 90% are retained as retained dimensions.

[0018] In some embodiments, the method further includes: setting the dimension of the knowledge vector to be learned to account for 50% to 70% of the dimension of the word vector, and initializing the value range of the knowledge vector to be learned to be [-0.1, 0.1].

[0019] In some embodiments, based on a data set labeled with parts of speech in the preset corpus, the probability of the part of speech belonging to each position of the sentence is counted, and a sentence part-of-speech vector is constructed for the sentence using the part-of-speech distribution probability, and the part-of-speech vector is mapped and replaced based on a mapping relationship obtained by performing a dimensionality reduction analysis on the part of speech to obtain a grammatical vector, including:

[0020] For sentences in the preset corpus, the words themselves are replaced with part-of-speech values ​​to obtain a part-of-speech dataset that only retains the part-of-speech;

[0021] The probability of different parts of speech at each position of the sentence in the part-of-speech data set is counted, the part-of-speech vector of the sentence is defined by using the part-of-speech distribution probability, and the grammatical vector is obtained by performing dimensionality reduction according to the retained dimension determined by principal component analysis.

[0022] In some embodiments, the method further includes: using accuracy and recall as preset performance indicators to evaluate the processing performance of the target neural network model on the test set.

[0023] In some embodiments, the method further comprises:

[0024] Introducing an attention mechanism into the target natural language model to perform feature fusion on the prior part in the word vector and the knowledge vector to be learned;

[0025] In the second training stage, the prior part corresponding to each word vector is fine-tuned by controlling the learning rate.

[0026] On the other hand, the present invention also provides a model acceleration training device based on prior knowledge word vectorization, including a processor, a memory and a computer program / instructions stored in the memory, wherein the processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the device implements the steps of the above method.

[0027] On the other hand, the present invention further provides a computer-readable storage medium having a computer program / instruction stored thereon, which implements the steps of the above method when the computer program / instruction is executed by a processor.

[0028] On the other hand, the present invention also provides a computer program product, comprising a computer program / instruction, which implements the steps of the above method when executed by a processor.

[0029] The beneficial effects of the present invention are at least:

[0030] The model acceleration training method and device based on prior knowledge word vectorization described in the present invention constructs a priori part and a knowledge vector to be learned when vectorizing a corpus. The prior part determines the vector dimension based on a statistical method, and integrates the prior knowledge to reduce the model learning burden and improve efficiency. During the training process, the prior part is fixed first to update the model and the knowledge vector to be learned, and then the prior part is fine-tuned. The knowledge vector to be learned is used to synchronously update during the model training process, so as to capture implicit information, improve the understanding of the context, enhance the generalization ability to handle ambiguity and accelerate the convergence of the model.

[0031] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention may be achieved and obtained by the structures specifically indicated in the specification and the accompanying drawings.

[0032] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of the present application, and do not constitute a limitation of the present invention. In the drawings:

[0034] Figure 1 This is a flowchart of a model acceleration training method based on prior knowledge word vectorization described in one embodiment of the present invention.

[0035] Figure 2 This is a logical schematic diagram of a model acceleration training method based on prior knowledge word vectorization according to an embodiment of the present invention.

[0036] Figure 3 This is a schematic diagram of the feature splicing structure in the model acceleration training method based on prior knowledge word vectorization described in one embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0038] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0039] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0040] Traditional natural language processing models require word vectorization of input data. During the training process, word vectorization schemes such as Word2Vec and GloVe have limitations such as low training efficiency, difficulty in convergence, and weak generalization ability. The present invention provides a new word vectorization scheme to accelerate and optimize the training of natural language models, aiming to reduce model complexity and improve the convergence speed of model training by optimizing the dimensional definition of word vectors and the initialization method of dimensional values, thereby accelerating the training of natural language models and improving generalization ability. It provides a new technical path for the development of the field of natural language processing.

[0041] Specifically, the present invention provides a model acceleration training method based on prior knowledge word vectorization, such as Figure 1 and Figure 2 As shown, the method includes the following steps S101 to S106:

[0042] Step S101: statistically analyzing the part of speech, word frequency and position of words in a preset corpus in each sentence according to multiple preset dimensions, and dividing the part of speech into multiple dimensions according to the part of speech.

[0043] Step S102: Standardize the statistical analysis results of the dimensions related to parts of speech, and determine the retained dimensions for parts of speech based on dimensionality reduction based on principal component analysis, and combine the retained dimensions, word frequency, and preset dimensions related to position as prior dimensions.

[0044] Step S103: constructing a part-of-speech vector, a word frequency vector and a position vector for the words in the preset corpus according to a priori dimensions.

[0045] Step S104: Based on a data set with parts of speech annotated in a preset corpus, count the probability of the part of speech belonging to each position in the sentence, use the part of speech distribution probability to construct a sentence part of speech vector for the sentence, and map and replace the part of speech vector based on the mapping relationship obtained by dimensionality reduction analysis of the part of speech to obtain a grammatical vector.

[0046] Step S105: configure and initialize a knowledge vector to be learned of a set dimension for each word, use the part-of-speech vector, word frequency vector, position vector and grammatical vector corresponding to each word as a priori parts, and jointly construct the priori parts and the knowledge vector to be learned corresponding to each sentence into a word vector.

[0047] Step S106: construct multiple sentences in the preset corpus after word vectorization as a training sample set, and train the target natural language model; for each batch of training data, lock the prior part of each word vector in the training sample set in the first training stage, and use the gradient descent algorithm to update the parameters of the target natural language model and the knowledge vector to be learned corresponding to each word vector until the preset performance indicators of the target neural network model on the test set converge and execute the second training stage, and fine-tune the prior part corresponding to each word vector in the second training stage.

[0048] In step S101, firstly, statistics are performed on the words in all sentences in the preset corpus, and a statistical vector is established through a preset dimension.

[0049] In some embodiments, for example, the parts of speech include content words, function words, interjections, and onomatopoeia; content words are divided into 8 subcategories including nouns, verbs, adjectives, numerals, quantifiers, pronouns, adverbs, and onomatopoeia; function words are divided into 5 subcategories including prepositions, conjunctions, auxiliary words, interjections, and modal particles, and are marked with a 13-dimensional vector; word frequency is marked with a 1-dimensional vector; and the position is marked by establishing a dimension based on the length of the sentence in the preset corpus. The position dimension takes the average or maximum value of the number of words contained in the sentence after the segmentation length, and fills in 0 when it is insufficient.

[0050] The original part-of-speech dimension of each word has 13 dimensions. A part-of-speech distribution matrix is ​​constructed, where rows represent different words and columns represent the 13 part-of-speech dimensions of each word. The value of each dimension is the number of times the word appears as that part of speech in the dataset. If the number of words in the dataset is n, then the dataset is 13-dimensional n groups.

[0051] The average length or maximum length of all sentences in the corpus is counted (for example, assuming L=10). If the actual length of a sentence is less than this length, zero padding can be performed to unify the length.

[0052] The word positions are counted sentence by sentence. In each sentence, each word is recorded according to its position (numbered from left to right: 1, 2, 3, ...). For example, in the sentence "The cat jumped on the chair", the position of "cat" is 1, the position of "on" is 2, and the position of "jump" is 5. For sentences that are not long enough, they are subsequently padded with 0.

[0053] The unified encoding rule is that the position of each word is represented by a one-hot vector: for a sentence of length L, the position vector is an L-dimensional vector, the position of the i-th word is the i-th dimension marked as 1, and the other dimensions are 0. For example, for a sentence of length 5, "cat is jumping", the position vector of "cat": 1,0,0,0,0; the position vector of "in": 0,1,0,0,0; the position vector of "jump": 0,0,1,0,0.

[0054] In order to systematically store the position information of words in all sentences, a matrix can be used to represent the position information of each sentence in the entire corpus. Assuming there are N sentences in the corpus, the length of each sentence is fixed to L, and is achieved by zero padding. A three-dimensional matrix of N×L×L is constructed, and the matrix of the i-th sentence is an L×L matrix, which represents the unique hot position encoding of each word in the sentence.

[0055] In step S102, data standardization is first performed to eliminate the order of magnitude differences in the frequencies of occurrence of different parts of speech so that each dimension of the data has the same scale, and then PCA principal component analysis is performed on the dimensions related to the parts of speech to reduce the dimensionality.

[0056] In some embodiments, the statistical analysis results of the dimensions related to the part of speech are standardized, and the dimension reduction is performed based on the principal component analysis to determine the retained dimensions for the part of speech, including steps S201 to S204:

[0057] Step S201: Count the part-of-speech distribution of each word in the preset corpus and express it as a matrix, where the rows of the matrix correspond to words, the columns correspond to the classification dimensions of the part-of-speech, and the value of each cell is the number of times the word appears in the preset corpus with the corresponding part-of-speech.

[0058] Step S202: Standardize the data of each part-of-speech column so that its mean is 0 and its variance is 1. The calculation formula is:

[0059]

[0060] Here, x is the original value, μ is the mean of the column, and σ is the standard deviation of the column.

[0061] Step S203: Calculate the covariance matrix of the matrix after data standardization. The calculation formula is:

[0062]

[0063] Among them, X is the standardized data.

[0064] Step S204: Solve the eigenvalues ​​and their corresponding contribution rates for the covariance matrix, and retain a set number of principal components with a contribution rate of 90% as the retained dimensions. The eigenvalue equation is:

[0065] |∑-γI|=0;

[0066] Where λ is the eigenvalue of the covariance matrix, I is the identity matrix, and the corresponding eigenvectors v are calculated. These eigenvectors are the directions of the information principal components. The contribution value is obtained by calculating the proportion of each eigenvalue to the total eigenvalues.

[0067] The eigenvectors whose sum of eigenvalues ​​accounts for 90% of the sum of all eigenvalues ​​are selected to determine the retained dimensions.

[0068] Project the original data onto the selected principal components to obtain the reduced-dimensional part-of-speech data. If the first k principal components are selected, the reduced-dimensional data Y can be calculated using the following formula:

[0069] Y=Xv k ;

[0070] Among them, v k is the matrix consisting of the first k eigenvectors.

[0071] In step S103, the parts of speech are marked in the data set based on the part-of-speech classification system, the number of occurrences of each word in the data set in different parts of speech is counted, and the statistical results are normalized to obtain the initial value of the part-of-speech vector of each word, and then this initial value is mapped to a reduced-dimensional part-of-speech vector.

[0072] The number of occurrences of each word in the statistical data set is counted, and then the statistical results are normalized to obtain the word frequency value after normalization as the initial value of the word frequency vector of this word.

[0073] The position distribution of words in the sentence is counted and then the statistical results are normalized. Considering the variability of sentence length, the normalized results are encoded using the cosine function so that they can be expanded to accommodate longer sentences. The word position vector is initialized using cosine encoding.

[0074] In step S104, based on a data set with parts of speech annotated in a preset corpus, the probability of the part of speech belonging to each position of the sentence is counted, and the part of speech vector of the sentence is constructed using the part of speech distribution probability. The part of speech vector is mapped and replaced based on the mapping relationship obtained by dimensionality reduction analysis of the part of speech to obtain a grammatical vector, including steps S301 to S302:

[0075] Step S301: For sentences in a preset corpus, the word itself is replaced with the part-of-speech value to obtain a part-of-speech dataset that only retains the part-of-speech.

[0076] Step S302: Count the probabilities of different parts of speech at each position of the sentence in the part-of-speech data set, define the sentence part-of-speech vector using the part-of-speech distribution probability, and perform dimensionality reduction according to the retained dimensions determined by principal component analysis to obtain a grammatical vector.

[0077] For example, there are the following three sentences in the data set, sentence 1: a cat walking slowly; sentence 2: a big dog barking; sentence 3: there is a tall tree.

[0078] The part of speech (such as adjective, verb, noun, etc.) of each word in each sentence is extracted through syntactic analysis to form a part of speech tag sequence:

[0079] Sentence 1: The cat (noun) walks slowly (adjective).

[0080] Sentence 2: Big (adjective) dog (noun) barks (verb).

[0081] Sentence 3: There is (verb) on the tall (adjective) tree (noun) (preposition).

[0082] All parts of speech in the dataset include the following five categories: adjective, verb, auxiliary word, noun, and preposition, so the dimension of the part of speech probability vector is 5.

[0083] Calculate the probability distribution of sentence parts of speech. For each sentence, count the proportion of each part of speech in the sentence to form a part of speech probability vector. For example, sentence 1 (slow walking cat): adjective proportion = 1 / 4 = 0.25, verb proportion = 1 / 4 = 0.25, auxiliary word proportion = 1 / 4 = 0.25, noun proportion = 1 / 4 = 0.25, preposition proportion = 0 / 4 = 0, part of speech probability vector: [0.25, 0.25, 0.25, 0.25, 0].

[0084] Sentence 2 (big dog barking): adjective proportion = 1 / 3≈0.33, verb proportion = 1 / 3≈0.33, noun proportion = 1 / 3≈0.33, auxiliary word proportion = 0, preposition proportion = 0, part-of-speech probability vector: [0.33, 0.33, 0, 0.33, 0].

[0085] Sentence 3 (on the high tree): adjective proportion = 1 / 4 = 0.25, verb proportion = 1 / 4 = 0.25, noun proportion = 1 / 4 = 0.25, preposition proportion = 1 / 4 = 0.25, auxiliary word proportion = 0, part-of-speech probability vector: [0.25, 0.25, 0, 0.25, 0.25].

[0086] The part-of-speech probability vectors of each sentence are formed into a matrix:

[0087]

[0088] Through principal component analysis (PCA), the part-of-speech probability vector is reduced from the original 5 dimensions to 3 dimensions to extract the main features. The covariance matrix is ​​calculated to analyze the linear correlation between the part-of-speech dimensions. The eigenvalues ​​and eigenvectors of the covariance matrix are solved, and the eigenvectors corresponding to the first 3 main eigenvalues ​​are selected. The original vector is re-expressed using the eigenvector matrix after dimensionality reduction. The sentence part-of-speech feature vector matrix after dimensionality reduction is obtained as follows:

[0089]

[0090] In step S105, the knowledge vector to be learned is used to learn the implicit semantic relations and patterns in the corpus, which is the main goal of traditional word vectors (such as Word2Vec, GloVe). Unlike prior knowledge vectors (explicit features such as part of speech, word frequency, position, etc.), the dimension of the knowledge vector to be learned is not restricted by prior statistics, but provides space for exploratory learning for the model. If the dimension is too small, the vector expression ability is insufficient, and the model may not be able to learn complex semantic relationships. If the dimension is too large: it increases the computational cost and may cause overfitting problems, especially when the corpus is small.

[0091] In some embodiments, the dimension of the knowledge vector to be learned is set to 50% to 70% of the dimension of the word vector, and the value range of the knowledge vector to be learned is initialized to [-0.1, 0.1].

[0092] In step S106, if Figure 3 As shown in the figure, feature concatenation is performed to concatenate the prior knowledge vector with the unknown knowledge vector of the word to form a complete word vector representation. The attention mechanism is introduced into the model to allow interaction between different feature vectors and enhance the expressiveness of the model.

[0093] In the early stage of training, the prior knowledge vector is locked, and only the model is allowed to adjust the knowledge vector to be learned. The performance of the model on the validation set is monitored in real time, and the adjustment mechanism is triggered when the performance indicators (such as accuracy and recall) stagnate. The value of the unknown knowledge vector is dynamically adjusted through the gradient descent algorithm. When the performance bottleneck of the model is detected, a fine-tuning strategy is adopted to allow a small range of adjustments to the prior knowledge vector.

[0094] In this process, the knowledge vectors to be learned can be used to capture implicit information. Unknown knowledge dimensions can help word vectors express vocabulary usage and semantic information that are not easily defined by explicit rules. For example, the usage of certain words in a specific context may be very special, and these usages may not be fully described by traditional part-of-speech tagging. Enhance generalization ability. By introducing unknown knowledge dimensions, the knowledge represented by word vectors can be better generalized to unseen data. These dimensions may contain complex patterns and associations in the data, allowing word vectors to express more accurate predictions when faced with new or diverse texts. Dealing with ambiguity. Ambiguity in language is common. Unknown knowledge dimensions may help distinguish between cases with the same part of speech but different usages. For example, the same word may have different meanings and functions in different contexts. Discovering new rules: When processing large amounts of text data, unknown knowledge dimensions may reveal new language rules or trends that linguists may not have discovered yet. Improve context understanding. When processing natural language, context is very important. Unknown knowledge dimensions may contain context information to help word vectors better express the overall meaning of the sentence.

[0095] The introduction of the unknown knowledge dimension is to enhance the word vector's ability to represent language complexity, especially when faced with language properties that are difficult to describe using traditional methods.

[0096] In some embodiments, accuracy and recall are used as preset performance indicators to evaluate the processing performance of the target neural network model on the test set.

[0097] In some embodiments, the method further includes: introducing an attention mechanism in the target natural language model to perform feature fusion on the prior part in the word vector and the knowledge vector to be learned; and fine-tuning the prior part corresponding to each word vector by controlling the learning rate in the second training stage.

[0098] On the other hand, the present invention also provides a model acceleration training device based on prior knowledge word vectorization, including a processor, a memory and a computer program / instructions stored in the memory, wherein the processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the device implements the steps of the above method.

[0099] Corresponding to the above method, the present invention also provides an apparatus / system, which includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, the processor is used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the apparatus / system implements the steps of the method described above.

[0100] The embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the aforementioned edge computing server deployment method are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0101] The above-mentioned solution has the following advantages over the existing word vectorization solution:

[0102] 1. Separation of prior knowledge and unknown knowledge: By dividing word vectors into prior knowledge dimensions and unknown knowledge dimensions, this solution can more clearly distinguish and process known and unknown information, helping the model to focus more on learning potential knowledge that is difficult to obtain directly from data.

[0103] 2. Optimize dimension definition: Using statistical methods combined with principal component dimensionality reduction technology (PCA) to determine the main influencing factors of prior knowledge dimensions can more scientifically define the dimensionality of word vectors instead of relying on empirical values, thereby improving the information content and representation efficiency of the vectors.

[0104] 3. Effective use of prior knowledge: By integrating prior knowledge such as grammar, part of speech, word frequency and position, the model’s learning burden of such knowledge is reduced, the model parameters are reduced, and the training efficiency and generalization ability of the model are improved.

[0105] 4. Initialization of prior knowledge vector: Using mathematical statistics methods to initialize the prior knowledge vector instead of random initialization provides more reasonable initial parameters and helps to speed up the convergence of the model.

[0106] 5. Reduce training time and computing resources: By locking the prior knowledge vector in the early stage of training, the number of parameters that need to be adjusted in the model is reduced, thereby reducing the time and computing resources required for training.

[0107] 6. Adaptive fine-tuning of performance bottlenecks: When encountering a performance bottleneck, the adaptive prior knowledge vector fine-tuning mechanism is activated, allowing the model to slightly adjust the value of the prior knowledge vector, which helps to compensate for possible deviations in the statistical method and further improve model performance.

[0108] 7. Multi-dimensional feature fusion: Different language features (such as part of speech, word frequency, position, and grammar) are integrated into word vectors to form a richer and more accurate word feature representation, enhancing the expressive power of the model.

[0109] 8. Improve semantic understanding and text analysis capabilities: Through the above optimizations, the semantic understanding and text analysis capabilities of the model have been improved.

[0110] 9. Reduce overfitting: By integrating prior knowledge, the model does not need to learn this knowledge from scratch, thus reducing the number of parameters that need to be learned. This helps reduce the complexity of the model and reduces the risk of overfitting. When the model is not too complex, it is less likely to learn overly specific patterns on the training data, which may not be applicable on the test data. Prior knowledge (such as part of speech, word frequency, position, and grammar) usually has stable language characteristics and does not change drastically with context. Integrating these stable features into word vectors can provide more consistent and reliable input, allowing the model to better recognize and utilize these common language patterns when faced with different texts. When part of the model's knowledge is replaced by prior knowledge, the model can focus more on learning potential relationships and patterns that are difficult to obtain directly from statistical data. This helps the model capture more essential language laws and perform better on new data. The use of prior knowledge reduces the model's need for large-scale training data. This means that even in the absence of a large amount of labeled data, the model can obtain certain performance through prior knowledge, which is especially important for areas where data is scarce. Because the model has integrated multiple language features during training, it can better adapt and generalize when faced with texts of different styles, domains, or contexts. For example, a model that has learned parts of speech and grammatical structures during training is more likely to correctly parse and understand new sentence structures. The adaptive fine-tuning mechanism allows the model to be optimized based on performance feedback during training, which helps the model adjust when encountering new data distributions, thereby improving generalization capabilities.

[0111] In summary, the model acceleration training method and device based on prior knowledge word vectorization described in the present invention, when word vectorization is performed on a corpus, a priori part and a knowledge vector to be learned are constructed, the priori part determines the vector dimension based on a statistical method, and the prior knowledge is integrated to reduce the model learning burden and improve efficiency; during the training process, the priori part is fixed first to update the model and the knowledge vector to be learned, and then the priori part is fine-tuned, and the knowledge vector to be learned is used to synchronously update during the model training process, so as to capture implicit information, improve the understanding of the context, enhance the generalization ability to handle ambiguity and accelerate the convergence of the model.

[0112] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0113] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0114] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0115] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A model acceleration training method based on prior knowledge word vectorization, characterized in that: The method comprises the following steps: Performing statistical analysis on the part of speech, word frequency and position of words in each sentence in a preset corpus according to multiple preset dimensions, wherein the part of speech is divided into multiple dimensions according to the part of speech; Standardizing the statistical analysis results of the dimensions related to the part of speech, and determining the reserved dimensions for the part of speech by performing dimensionality reduction based on principal component analysis, and combining the reserved dimensions, the word frequency, and the preset dimensions related to the position as a priori dimensions; Constructing a part-of-speech vector, a word frequency vector and a position vector for the words in the preset corpus according to the prior dimension; Based on the data set with parts of speech marked in the preset corpus, the probability of the part of speech belonging to each position of the sentence is counted, the part of speech vector of the sentence is constructed for the sentence using the part of speech distribution probability, and the part of speech vector is mapped and replaced based on the mapping relationship obtained by dimensionality reduction analysis of the part of speech to obtain a grammatical vector; A knowledge vector to be learned of a set dimension is configured for each word and initialized, the part-of-speech vector, the word frequency vector, the position vector and the grammatical vector corresponding to each word are used as a priori parts, and the priori parts corresponding to each word and the knowledge vector to be learned are jointly constructed into a word vector; A plurality of sentences in the preset corpus after word vectorization are constructed as a training sample set, and a target natural language model is trained; for each batch of training data, the prior part of each word vector in the training sample set is locked in the first training stage, and the parameters of the target natural language model and the knowledge vector to be learned corresponding to each word vector are updated using a gradient descent algorithm until the preset performance index of the target neural network model on the test set converges and the second training stage is executed, and the prior part corresponding to each word vector is fine-tuned in the second training stage.

2. The model acceleration training method based on prior knowledge word vectorization according to claim 1 is characterized in that: The parts of speech include content words, function words, interjections and onomatopeia; content words are divided into 8 small categories including nouns, verbs, adjectives, numerals, quantifiers, pronouns, adverbs and onomatopeia; function words are divided into 5 small categories including prepositions, conjunctions, auxiliary words, interjections and modal particles, and are marked with a 13-dimensional vector; the word frequency is marked with a 1-dimensional vector; the position is marked by establishing a dimension based on the length of the sentence in the preset corpus.

3. The model acceleration training method based on prior knowledge word vectorization according to claim 2 is characterized in that: The statistical analysis results of the dimensions related to the part of speech are standardized, and the dimension reduction is performed based on the principal component analysis to determine the retained dimensions for the part of speech, including: Counting the part-of-speech distribution of each word in the preset corpus and expressing it as a matrix, wherein the rows of the matrix correspond to words, the columns correspond to the classification dimensions of the part-of-speech, and the value of each cell is the number of times the word appears in the preset corpus with the corresponding part-of-speech; Standardize the data of each part-of-speech column so that its mean is 0 and its variance is 1; Calculating the covariance matrix of the matrix after data standardization; For the eigenvalues ​​of the covariance matrix solution and their corresponding contribution rates, a set number of principal components whose contribution rates reach 90% are retained as retained dimensions.

4. The model acceleration training method based on prior knowledge word vectorization according to claim 1 is characterized in that: The method also includes: setting the dimension of the knowledge vector to be learned to account for 50% to 70% of the word vector dimension, and initializing the value range of the knowledge vector to be learned to [-0.1, 0.1].

5. The model acceleration training method based on prior knowledge word vectorization according to claim 1 is characterized in that: Based on the data set with parts of speech marked in the preset corpus, the probability of the part of speech belonging to each position of the sentence is counted, the part of speech vector of the sentence is constructed for the sentence using the part of speech distribution probability, and the part of speech vector is mapped and replaced based on the mapping relationship obtained by dimensionality reduction analysis of the part of speech to obtain a grammatical vector, including: For sentences in the preset corpus, the words themselves are replaced with part-of-speech values ​​to obtain a part-of-speech dataset that only retains the part-of-speech; The probability of different parts of speech at each position of the sentence in the part-of-speech data set is counted, the part-of-speech vector of the sentence is defined by using the part-of-speech distribution probability, and the grammatical vector is obtained by performing dimensionality reduction according to the retained dimension determined by principal component analysis.

6. The model acceleration training method based on prior knowledge word vectorization according to claim 1 is characterized in that: The method also includes: using accuracy and recall as preset performance indicators to evaluate the processing performance of the target neural network model on the test set.

7. The model acceleration training method based on prior knowledge word vectorization according to claim 1 is characterized in that: The method further comprises: Introducing an attention mechanism into the target natural language model to perform feature fusion on the prior part in the word vector and the knowledge vector to be learned; In the second training stage, the prior part corresponding to each word vector is fine-tuned by controlling the learning rate.

8. A model acceleration training device based on prior knowledge word vectorization, comprising a processor, a memory, and a computer program / instruction stored in the memory, characterized in that: The processor is used to execute the computer program / instructions. When the computer program / instructions are executed, the device implements the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method as claimed in any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images