Model training method, Chinese spelling correction method and retrieval method

Abstract

A model training method and a Chinese spelling error correction method are provided. The model training method comprises: inputting a sentence sample into an embedding layer to obtain a word vector of each Chinese character in the sentence sample; obtaining multiple word vectors containing sentence information based on the word vector of each Chinese character in the sentence sample; inputting multiple word vectors containing sentence information into a classification layer, and the classification layer multiplies each word vector containing sentence information with each one-dimensional vector in a weight parameter matrix to obtain multiple scores; for each Chinese character in the sentence sample, using the Chinese character corresponding to the maximum score among the multiple scores obtained as the error correction Chinese character; adjusting the weight parameter matrix using the loss between the correct error correction Chinese character and the wrong error correction Chinese character, and then using the back propagation algorithm to adjust the weight parameters of the corresponding layer of the specified model that generates the weight parameter matrix. Through this method, the difference in the degree of similarity between characters is further distinguished using the dynamically generated confusion set.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, specifically to the field of artificial intelligence technology such as natural language processing and deep learning, and more specifically to a model training method, a Chinese spelling correction method, and a retrieval method. Background Art

[0002] Text error correction is an important application in natural language processing (NLP). It is widely used in many natural language processing systems such as search engines, machine translation, and question-answering systems. These systems will correct the user's input text to obtain the correct representation without changing the semantics of the original text. Chinese text error correction can be mainly divided into two categories: Chinese spelling correction and Chinese grammar correction. Chinese spelling correction uses confusion sets to correct typos in input text, so confusion sets are crucial for Chinese spelling correction.

[0003] In the related art, a large amount of error correction corpus is often collected manually (generally on learning websites for non-Chinese native language learners), and confusion sets are generated based on the error correction corpus to correct Chinese text. So far, the confusion set has reached about 5,000 Chinese characters. In addition, considering that errors generally occur in the case of similar pronunciation / similar shape, most traditional Chinese error correction systems limit the search space to the similar pronunciation / similar shape confusion set to reduce the search space. However, this retrieval method does not distinguish between spelling errors that are neither similar in pronunciation nor similar in shape. Summary of the invention

[0004] In view of this, the present disclosure aims to provide a model training method, a Chinese spelling correction method and a retrieval method to solve the problems existing in the prior art.

[0005] According to a first aspect of the present disclosure, a model training method is provided, comprising:

[0006] Input the sentence sample to the embedding layer to obtain a character vector for each Chinese character in the sentence sample;

[0007] Obtaining a plurality of word vectors containing sentence information based on the word vector of each Chinese character in the sentence sample;

[0008] Inputting the plurality of word vectors containing sentence information to a classification layer, the classification layer multiplies each word vector containing sentence information by each one-dimensional vector in a weight parameter matrix to obtain a plurality of scores, wherein each one-dimensional vector in the weight parameter matrix is ​​used to represent a word vector of a Chinese character in a set of commonly used Chinese characters;

[0009] For each Chinese character in the sentence sample, the Chinese character corresponding to the maximum score among the multiple scores obtained is used as the error correction Chinese character;

[0010] The weight parameter matrix is ​​adjusted using the loss between the correct error-corrected Chinese characters and the incorrect error-corrected Chinese characters, and the weight parameters of the corresponding layers of the specified model that generates the weight parameter matrix are further adjusted using a back-propagation algorithm.

[0011] The specified model includes the embedding layer, at least one graph neural network layer coupled to the embedding layer, and the classification layer coupled to the at least one graph neural network layer, and the graph neural network layer is used to perform graph calculations based on the input word vectors and confusion sets and generate new word vectors.

[0012] In some embodiments, the at least one graph neural network layer includes a first graph neural network layer and a second graph neural network layer, and the specified model further includes: an attention layer and an encoding layer,

[0013] The embedding layer is further used to encode each Chinese character in the commonly used Chinese character set into a word vector, and provide a plurality of first word vectors corresponding to a plurality of Chinese characters belonging to a first confusion set to the first graph neural network layer, and provide a plurality of second word vectors corresponding to a plurality of Chinese characters not belonging to the first confusion set to the classification layer, wherein the first confusion set indicates easily confused Chinese characters in the commonly used Chinese character set;

[0014] The first graph neural network layer is used to perform graph calculation according to the plurality of first word vectors and the graph data corresponding to the first confusion set to generate a plurality of new graph data, obtain a plurality of third word vectors from the new graph data, and provide the plurality of third word vectors to the attention layer and the second graph neural network layer;

[0015] The attention layer is used to calculate the confusable values ​​between the plurality of third character vectors, and thereby obtain a second confusion set, wherein the second confusion set indicates not only the confusable Chinese characters in the commonly used Chinese character set, but also the confusable values;

[0016] The second graph neural network layer is used to perform graph calculation according to the plurality of third word vectors and the graph data corresponding to the second confusion set to generate new graph data, obtain a plurality of fourth word vectors from the new graph data, and provide the plurality of fourth word vectors to the classification layer, wherein the plurality of fourth word vectors and the plurality of second word vectors are organized into a weight parameter matrix of the classification layer;

[0017] The encoding layer is used to execute the step of obtaining a plurality of word vectors containing sentence information based on the word vector of each Chinese character in the sentence sample;

[0018] The method of using the back-propagation algorithm to adjust the weight parameters of the corresponding layers of the specified model that generates the weight parameter matrix includes: adjusting the weight parameters of the second graph neural network, the first graph neural network layer, the attention layer and the embedding layer in sequence through the back-propagation algorithm.

[0019] In some embodiments, obtaining a plurality of word vectors containing sentence information based on the word vector of each Chinese character in the sentence sample comprises:

[0020] Obtaining position information and context information of each Chinese character in the sentence sample; and

[0021] A word vector containing sentence information is obtained based on the word vector of each Chinese character and the position information and context information of each Chinese character in the sentence sample.

[0022] In some embodiments, the calculating of the easily confused values ​​between the plurality of third word vectors includes: calculating the easily confused values ​​between the plurality of third word vectors using an attention mechanism, the attention layer further including: uniformly processing the confused values ​​that are less than or equal to a set threshold as a set negative value; and processing the easily confused values ​​using a normalization method.

[0023] In some embodiments, the embedding layer and the encoding layer adopt corresponding layers in the BERT model.

[0024] According to a second aspect of the present disclosure, a model for training a Chinese spelling error correction model is provided, comprising: an embedding layer, an encoding layer, a first graph neural network layer, a second graph neural network layer, an attention layer and a classification layer, wherein the Chinese spelling error correction model comprises the embedding layer, the encoding layer and the classification layer.

[0025] The embedding layer is used to encode each Chinese character in the input sentence into a word vector, and to encode each Chinese character in the commonly used Chinese character set into a word vector, to provide a plurality of first word vectors corresponding to a plurality of Chinese characters belonging to a first confusion set to the first graph neural network layer, and to provide a plurality of second word vectors corresponding to a plurality of Chinese characters not belonging to the first confusion set to the classification layer, wherein the first confusion set indicates easily confused Chinese characters in the commonly used Chinese character set;

[0026] The encoding layer is used to encode the word vector of each Chinese character in the input sentence into a word vector with sentence information;

[0027] The first graph neural network layer is used to perform graph calculation according to the plurality of first word vectors and the graph data corresponding to the first confusion set to generate a plurality of new graph data, obtain a plurality of third word vectors from the new graph data, and provide the plurality of third word vectors to the attention layer and the second graph neural network layer;

[0028] The attention layer is used to calculate the confusable values ​​between the plurality of third character vectors, and thereby obtain a second confusion set, wherein the second confusion set indicates not only the confusable Chinese characters in the commonly used Chinese character set, but also the confusable values;

[0029] The second graph neural network layer is used to perform graph calculation according to the plurality of third word vectors and the graph data corresponding to the second confusion set to generate new graph data, obtain a plurality of fourth word vectors from the new graph data, and provide the plurality of fourth word vectors to the classification layer, wherein the plurality of fourth word vectors and the plurality of second word vectors are organized into a weight parameter matrix of the classification layer;

[0030] The classification layer is used to multiply each word vector containing sentence information by each one-dimensional vector in the weight parameter matrix to obtain multiple scores, wherein the Chinese character corresponding to the maximum score among the multiple scores is used as the error correction Chinese character of the corresponding Chinese character, and each one-dimensional vector in the weight parameter matrix is ​​used to represent the word vector of a Chinese character in a commonly used Chinese character set.

[0031] According to a third aspect of the present disclosure, a Chinese spelling error correction method is provided, which is applied to a server side and includes:

[0032] receiving an input sentence; and

[0033] Providing the input sentence to a Chinese spelling error correction model to obtain a score representing the degree of confusion between each Chinese character in a commonly used Chinese character set and each Chinese character in the input sentence;

[0034] For each Chinese character in the input sentence, the Chinese character corresponding to the maximum score is used as the error correction Chinese character.

[0035] Among them, the Chinese spelling error correction model includes an embedding layer, an encoding layer and a classification layer, the embedding layer is used to encode each Chinese character in the input sentence into a word vector, the encoding layer is used to encode the word vector of each Chinese character in the input sentence into a word vector with sentence information, and the classification layer is used to matrix multiply each word vector containing sentence information with the word vector corresponding to each Chinese character in the commonly used Chinese character set to obtain a score representing the degree of confusion between the corresponding Chinese character of the input sentence and each Chinese character in the commonly used Chinese character set.

[0036] According to a fourth aspect of the present disclosure, a Chinese error correction method is provided, which is applied to a terminal and includes:

[0037] An input sentence is received and sent to a server, the server uses a Chinese spelling correction model to obtain a score representing the degree of confusion between each Chinese character in a commonly used Chinese character set and each Chinese character in the input sentence, and performs Chinese spelling correction on the input sentence according to the score to obtain a corrected sentence;

[0038] The corrected sentence is received from the server and displayed.

[0039] According to a fifth aspect of the present disclosure, a retrieval method is provided, comprising:

[0040] receiving a search term;

[0041] Providing the search term to a Chinese spelling error correction model to obtain a score representing the degree of confusion between each Chinese character in a commonly used Chinese character set and each Chinese character in the search term;

[0042] For each Chinese character in the search term, the Chinese character corresponding to the maximum score is used as the error correction Chinese character to obtain the corrected search term;

[0043] The corrected search term and / or the search term are used to perform a database search.

[0044] According to a sixth aspect of the present disclosure, an electronic device is provided, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the model training method described in any one of the above items is implemented, or when the processor executes the program, the Chinese spelling correction method described in any one of the above items is implemented, or when the processor executes the program, the retrieval method described in the above items is implemented.

[0045] According to a seventh aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the model training method described in any one of the above items is implemented, or when the computer program is executed by a processor, the Chinese spelling correction method described in any one of the above items is implemented, or when the computer program is executed by a processor, the retrieval method described in the above items is implemented.

[0046] In summary, the embodiments of the present disclosure provide a model training method and a Chinese spelling correction method. In the model training method, an embedding layer and a classification layer for implementing Chinese spelling correction are obtained by training a specific model, and the specific model generates a new confusion set using a graph neural network and an attention mechanism, and the new confusion set adds a confusion value that characterizes the degree of confusion of Chinese characters, so that the difference in the degree of similarity between the sound and shape of characters can be further distinguished in Chinese spelling correction. Experiments show that this method can effectively improve the accuracy of Chinese spelling correction. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The above and other objects, features and advantages of the present disclosure will become more apparent through the description of the embodiments of the present disclosure with reference to the following drawings, in which:

[0048] Figure 1A structural diagram for illustrating the model training and application scenarios of the present disclosure;

[0049] Figure 2 is a schematic diagram of an exemplary electronic device;

[0050] Figure 3 A schematic diagram of a network structure used to illustrate a model used in an embodiment of the present disclosure;

[0051] Figure 4a is a schematic diagram for exemplarily representing graph data corresponding to a confusion set;

[0052] Figure 4b yes Figure 4a The graph data corresponding to the confusion set shown is used to output new graph data through graph calculation;

[0053] Figure 4c yes Figure 4b A schematic diagram of the graph data of the confusion set with the confusion values ​​shown;

[0054] Figure 5 It is a schematic diagram for illustrating the extraction of corresponding layers from the trained model as a Chinese spelling correction system;

[0055] Figure 6 is a flow chart of a model training method provided by an embodiment of the present disclosure;

[0056] Figure 7 It is a flow chart of a Chinese spelling error correction method provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0057] The present disclosure is described below based on embodiments, but the present disclosure is not limited to these embodiments. In the detailed description of the present disclosure below, some specific details are described in detail. It is possible for a person skilled in the art to fully understand the present disclosure without the description of these details. In order to avoid confusing the essence of the present disclosure, well-known methods, processes, and procedures are not described in detail. In addition, the drawings are not necessarily drawn to scale.

[0058] Application scenarios and system architecture

[0059] Figure 1A structural diagram for illustrating the model training and application scenarios of the present invention. The schematic diagram includes a training device 110 and an execution device 120. The training device 110 and the execution device 120 communicate with each other through a network 101. The network 101 is a combination of one or more of various communication technologies based on the exchange of signals, including but not limited to wired technology using electrical and / or optical transmission cables, and wireless technology using infrared, radio frequency and / or other forms. In different application scenarios, the network 101 can be the Internet, a wide area network, or a local area network, such as a company's proprietary network. The network 101 can also be a wired network or a wireless network. The training device 110 and the execution device 120 can have the following: Figure 2 The hardware structure shown.

[0060] like Figure 1 As shown, the training device 110 stores a model 111 to be trained, and the training is performed through a model training device 112. The training process can be described as follows: a large number of training samples 114 are provided to the model 111 to be trained and the model 111 to be trained is repeatedly executed, so that the model 111 to be trained continuously generates result data, and the weight parameters of the model 111 to be trained are continuously adjusted according to the loss between the expected result data and the actual result data until the loss between the expected result data and the actual result data meets the requirements. The model training device 112 here is a virtual device, which can be understood as a script program for executing the model 111 to be trained and providing training samples to the model 111 to be trained. The model obtained after stopping the training is called the trained model 113. Typically, an application system uses a trained model 113 to perform some processing, such as providing a trained machine translation model to a machine translation system on the Internet. However, in the application scenario of the present disclosure, the training device 110 provides a partial layer 1211 extracted from the trained model 113 to the model application system 121 on the execution device 120, that is, the model application system 121 only needs the functions provided by the partial layer 1211 in the trained model 113.

[0061] Figure 2 2 is a schematic diagram of an exemplary structure of an electronic device 200. As shown in the figure, the server 200 includes a scheduler 201, a storage unit 203, an I / O interface 204, and a plurality of model acceleration units 202 coupled via a bus 205.

[0062] The storage unit 203 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) and / or a cache memory unit. The storage unit 203 may also include a readable medium in the form of a non-volatile storage unit, such as a read-only memory unit (ROM), a flash memory, and various disk memories.

[0063] The storage unit 203 can store various program modules and data, including operating systems, applications that provide functions such as text processing, video playback, software editing and compilation, etc. The executable codes of these applications are read out and executed by the scheduler 201 from the storage unit 203 to implement the predetermined operations of these program modules. The scheduler 201 is generally a processor (CPU).

[0064] Bus 205 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0065] The server 200 may communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the server 200, and / or any device that enables the server 200 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via input / output (I / O) interface 204. In addition, the server 200 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter (not shown). For example, via a network adapter, Figure 1 The terminal 103 in the figure can access the server 200. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used based on the server 200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0066] As shown in the figure, the server 200 includes multiple model acceleration units 202. The traditional processor architecture design is very effective in logic control, but not efficient enough in large-scale parallel computing, so it is not efficient for model computing. For this reason, a model acceleration unit is developed, and different models can be adapted to different model acceleration units. The model acceleration unit is, for example, a neural network acceleration unit (NPU). The NPU adopts a data-driven parallel computing architecture, and is a processing unit for processing a large number of operations (such as convolution, pooling, etc.) of each neural network node. Or it is a graphics processing unit (GPU), which is used to perform image and graphics related computing work. Since the graphics processing unit uses a large number of computing units specifically used for graphics computing, the graphics card reduces its dependence on the CPU and undertakes some of the computationally intensive graphics and image processing work that the CPU originally undertook, so the processing efficiency of image data is greatly improved. Multiple model acceleration units 202 are controlled by the scheduler 201, and through the control of the scheduler 201, multiple model acceleration units 202 can work together.

[0067] Combination Figure 1 As shown, the training device 110 can be used as Figure 2 The hardware structure of the present invention is as follows: the scheduler 201 controls the training process, puts the execution of the model 111 on multiple model acceleration units 202 and summarizes the execution results of multiple model acceleration units 202. The data exchange between multiple model acceleration units 202 can be realized through the scheduler 201, or the multiple model acceleration units 202 can directly exchange data. Similarly, the execution device 120 can also adopt the following method: Figure 2 In the hardware structure, except for some modules related to the model, such as part of the layer 1211, which can be executed by multiple model acceleration units 202, the execution of other modules is implemented by the scheduler 201.

[0068] Model training method and Chinese spelling error correction method disclosed in the present invention

[0069] Figure 3 A schematic diagram of a network structure used to illustrate a model used in an embodiment of the present disclosure. Figure 3 and Figure 1 As shown, Figure 1 The model to be trained 111 in the figure corresponds to an embedding layer 311, a graph neural network layer 312, a graph neural network layer 313, an encoding layer 315 and a classification layer 314.

[0070] Before introducing the training process using the model, the functions of each layer are introduced below.

[0071] The classification layer 314 is used to perform a matrix operation W*V, where W refers to a weight parameter matrix and V refers to a word vector of an input sentence. Each one-dimensional vector in the weight parameter matrix W is used to represent a word vector of a Chinese character in a commonly used Chinese character set.

[0072] The embedding layer 311 is used to encode each Chinese character in the conventional Chinese character set into a character vector. The conventional Chinese character set summarizes about 7,000 Chinese characters that are frequently used by Chinese people in their daily work and life. The conventional Chinese character set can be divided into Chinese characters belonging to the confusion set and Chinese characters that do not belong to the confusion set. The confusion set summarizes the easily confused Chinese characters in the conventional Chinese character set. There are currently about 5,000 Chinese characters in the confusion set. The embedding layer 311 provides the character vector Hg0 corresponding to the Chinese characters belonging to the confusion set to the graph neural network layer 312, and provides the character vector H'g1 corresponding to the Chinese characters that do not belong to the confusion set to the classification layer 314. C1 is used to identify the confusion set in the figure. The confusion set C1 indicates the easily confused Chinese characters in the conventional Chinese character set. The embedding layer 311 is also used to encode each Chinese character in the input sentence into a character vector.

[0073] The graph neural network layer 312 performs graph calculations based on the word vector Hg0 belonging to the confusion set C1 and the confusion set C1 to generate new graph data, obtains the word vector Hg1 belonging to the confusion set C1 from the new graph data, and provides the word vector Hg1 belonging to the confusion set C1 to the graph neural network layer 313 and the attention layer 316.

[0074] The attention layer 316 calculates the confusing values between the word vectors Hg1 belonging to the confusion set C1, and thus obtains the confusion set C2 with confusing values. The confusion set C2 with confusing values adds an indication of the confusing values between the confusing Chinese characters on the basis of the confusion set C1. Note that the attention layer 316 can calculate the confusing values between any two word vectors Hg1, or only calculate the confusing values between the word vectors Hg1 corresponding to two related Chinese characters based on the confusion set C1.

[0075] The graph neural network layer 313 performs graph calculations based on the word vector Hg1 and the confusion set C2 with confusing values to generate new graph data, obtains the word vector Hg2 containing confusing value information from the new graph data, and provides the word vector Hg2 belonging to the confusion set to the classification layer 314.

[0076] Figure 4a It is a schematic diagram for exemplarily characterizing the graph data corresponding to the confusion set. Figure 4b is Figure 4a The graph data corresponding to the shown confusion set outputs new graph data through graph neural network calculations. Figure 4c is Figure 4b A schematic diagram of the graph data of the shown confusion set with confusing values. Refer to Figure 4a , the solid line without a cross indicates that the Chinese characters belong to confusing Chinese characters. The confusing Chinese characters of the Chinese character "竟" on the graph are: "竟", "净", "经", "京", "晶", "境", "井" and "静", see Figure 4b , where the dotted line indicates the newly added confusing Chinese characters, and "境" and "井" are also classified as confusing Chinese characters, while Figure 4c not only indicates the confusing Chinese characters, but also uses confusing values to indicate the confusing degree between the confusing Chinese characters. These confusing values can be calculated by the attention layer 316.

[0077] For the graph calculations in the graph neural network layer 312, it can be understood through the Figure 4a and Figure 4b shown graph data. The graph neural network structure usually can include multiple feature calculation layers. Each feature calculation layer receives the graph data and generates new graph data. Each feature calculation layer is for example as Figure 4aFor each Chinese character node in the shown graph data, operations are performed on the word vectors corresponding to its neighboring Chinese character nodes around it and its own word vector, and the operation result is used as the feature representation of this Chinese character node. And so on. In this way, new feature representations of each Chinese character node can be obtained, and new connections can be established based on the new feature representations of all Chinese character nodes, thus finally obtaining the graph data as shown in Figure 4b shown.

[0078] Then for the graph calculation in the graph neural network layer 313, since the input graph data is as shown in Figure 4c and contains confusing values, for each Chinese character node in the graph data as shown in Figure 4c in each feature calculation layer, when performing operations on the word vectors of its neighboring Chinese character nodes around it and its own word vector, the confusing values between the neighboring Chinese character nodes and itself can be combined for the operations. For example, if the operation is a weighted sum calculation, the weight values can be determined according to the confusing values.

[0079] Generally speaking, the word vector Hg1 generated by the graph neural network layer 312 carries the information of the confusion set C1. However, since the confusion set C1 only indicates the confusing Chinese characters and does not indicate the degree of confusion (i.e., the confusing values), the word vector Hg1 does not implicitly contain the information of the degree of confusion. However, the graph neural network layer 312 can add new connections between Chinese characters that are not connected in the confusion set C1 (such as the connection between "境" and "井" in Figure 4b ), that is, the graph neural network layer 312 adds new confusing Chinese characters. The word vector Hg2 generated by the graph neural network layer 312 carries the information of the confusion set C2. Since the confusion set C2 not only indicates the confusing Chinese characters but also indicates the degree of confusion, the word vector Hg2 implicitly contains the information of the degree of confusion between the confusing Chinese characters. Thus, the weight parameter matrix constructed based on the word vector Hg2 implicitly contains the information of the newly added confusing Chinese characters and the degree of confusion between the confusing Chinese characters, and thus is more helpful for distinguishing confusing Chinese characters.

[0080] According to the above summary, we can further realize that using two graph neural networks has the following advantages: First, the first graph neural network collects the neighbor information of the node to itself. Although the second graph neural network also collects the neighbor information of each node to its own representation, due to the existence of the first graph neural network, each node already contains the information of the neighbor node, so after passing through the second graph neural network, its own representation will also contain the representation of the neighbor node of the neighbor node. For example, A and B are similar, B and C are similar, but A and C are not told to be similar. After the first graph neural network, B's information will contain C's information. After the second graph neural network, A's information will contain the latest representation of B. At this time, C's information is transmitted to A through B. At the same time, the information on the degree of confusion between each node and its neighbor nodes is also collected into the node representation through the two graph neural networks.

[0081] Understandably, based on Figure 3 As shown, the embodiment of the present disclosure can be implemented using a model containing only one graph neural network layer, that is, the embedding layer 311 provides Hg0 to the graph neural network layer 312 and the classification layer 314, the graph neural network layer 312 is directly coupled to the classification layer 314, and provides Hg1 to the classification layer 314. However, accordingly, the above advantages generated by using two graph neural networks no longer exist.

[0082] Continue to refer Figure 3 As shown, the word vector H'g1 output by the embedding layer 311 that does not belong to the confusion set and the Hg2 output by the graph neural network layer 313 are combined together to obtain a word vector matrix corresponding to the regular Chinese character set. Conversely, each Chinese character in the regular Chinese character set can find a one-dimensional vector corresponding to it in the word vector matrix. The word vector matrix is ​​provided to the classification layer 314 as a weight parameter matrix.

[0083] As can be seen from the figure, the weight parameter matrix of the classification layer 314 is composed of word vectors H'g1 and word vectors Hg1. Therefore, after the training process is started, the word vectors H'g1 and word vectors Hg1 can be output as the weight parameter matrix of the classification layer 314 through the embedding layer 311, the graph neural network layer 312, the attention layer 316 and the graph neural network layer 313.

[0084] Then, the sample sentence is provided to the embedding layer 311 to obtain the word vectors of each Chinese character in the sample sentence. Then, the word vectors are provided to the encoding layer 315, and the encoding layer 315 outputs multiple word vectors containing sentence information. Each word vector with sentence information is operated on with the current weight parameter matrix in the classification layer 314 to calculate multiple scores, and the Chinese character with the highest score is used as the corrected Chinese character for the corresponding Chinese character. For example, for the character "竟" in the sample sentence shown in the figure, its corresponding word vector Vi(1*D) containing sentence information is multiplied by the weight parameter matrix W(S*D) of the classification layer. S is the number of Chinese characters in the regular Chinese character set, and D represents the column dimension corresponding to the word vector. The corresponding formula can be expressed by formula (1):

[0085] Pi(1*S) = Vi(1*D)*T(W(S*D)) i=0, ……,N-1 Formula (1)

[0086] Among them, T(x) represents the transpose operation on the matrix x, i ranges from 0 to N-1, and N represents the sentence length.

[0087] Finally, S scores are obtained, and each score corresponds to a Chinese character. The Chinese character with the maximum score is used as the corrected Chinese character for "竟". As shown in the figure, the correct corrected Chinese character for "竟" in the current sample sentence is "境". Then, if the corrected Chinese character with the maximum score obtained is not "境", but for example "井", then the loss between "境" and "井" needs to be calculated, and based on the loss, the weight parameter matrix in the classification layer 314 is adjusted one by one using the backpropagation algorithm, and further, the weight parameters of a certain layer or some layers in the graph neural network layer 313, the graph neural network layer 312, the embedding layer 311, and the attention layer 316 are adjusted.

[0088] For each sample sentence, the above training process is repeated until the loss calculated by the loss function can meet the requirements.

[0089] In some embodiments,[[]] Figure 3 The embedding layers 311 and 315 shown adopt the corresponding layers in the BERT model. The BERT (Bidirectional Encoder Representations from Transformers) model is a pre-trained model that converts the input sentence or paragraph into a vector. The pre-trained model can provide a set of good weight parameter values. R & D personnel can directly apply it to the actual scenario, or only need to perform some incremental training and fine-tune the weight parameter values before applying it to the actual scenario.

[0090] In some embodiments,[[]] Figure 3The attention layer 316 in calculates the confusion values between multiple third word vectors using the attention mechanism, but may not calculate the confusion values between two Chinese characters indicating no relationship in the original confusion set C1. For example, Figure 4c In the example of , the calculated confusion value between "Jing" and "A" is 0.001. However, these two Chinese characters indicate no relationship in the original confusion set C1 (the cross on the line indicates irrelevance). Then, in another example, the confusion value between "Jing" and "A" may not be calculated and can be directly marked as '0'.

[0091] In some embodiments, Figure 3 The attention layer 316 in uniformly processes the confusion values less than or equal to a set threshold into a set negative value, and then uses a normalization method to process all the confusion values. It can be understood that whether or not the attention mechanism is used to calculate the confusion values between multiple third word vectors, some very small confusion values will be generated. Processing these confusion values into zero helps to improve the computational performance of the subsequent graph calculation.

[0092] Figure 5 is a schematic diagram for schematically extracting the corresponding layer from a trained model as a Chinese spelling correction system. Combining Figure 3 As shown, after the training process ends, the embedding layer 311, the encoding layer 315, and the classification layer 314 are extracted and applied to the Chinese spelling correction system 500. In this way, the Chinese spelling correction system 500 uses these layers to implement the corresponding functions to achieve the purpose of Chinese spelling correction for the input sentence. Specifically, the embedding layer 311 generates a word vector for each Chinese character in the input sentence, the encoding layer 315 generates multiple word vectors containing sentence information, and the classification layer 314 multiplies the multiple word vectors containing sentence information by the weight parameter matrix stored internally. This weight parameter matrix represents the word vector matrix of the regular Chinese character set. Therefore, for each Chinese character in the input sentence, the number of Chinese characters in the regular Chinese character set of fractional values will be obtained, and the Chinese character corresponding to the maximum score is used as the corrected Chinese character. For example, for the input sentence "Encountered reverse Jing" in the figure, the output sentence is "Encountered adversity", indicating that the first three Chinese characters in the input sentence are correct, and only the last "Jing" needs to be corrected to "Jing".

[0093] Referring to the above, an embodiment of the present disclosure provides a model training method, and its flowchart is as Figure 6 shown.

[0094] In step S600, the sentence sample is input to the embedding layer to obtain the word vector of each Chinese character in the sentence sample.

[0095] In step S601, multiple word vectors containing sentence information are obtained based on the word vector of each Chinese character in the sentence sample.

[0096] In step S602, multiple word vectors containing sentence information are input to the classification layer, and the classification layer multiplies each word vector containing sentence information with each one-dimensional vector in the weight parameter matrix to obtain multiple scores, wherein each one-dimensional vector in the weight parameter matrix is ​​used to represent the word vector of a Chinese character in the commonly used Chinese character set.

[0097] In step S603, for each Chinese character in the sentence sample, the Chinese character corresponding to the maximum score among the multiple scores obtained is used as the error correction Chinese character.

[0098] In step S604, the weight parameter matrix is ​​adjusted using the loss between the correctly corrected Chinese characters and the incorrectly corrected Chinese characters, and then the weight parameters of the corresponding layers of the specified model that generates the weight parameter matrix are adjusted using the back propagation algorithm.

[0099] In step S605, the above steps are continuously performed until a weight parameter matrix that meets the requirements is obtained.

[0100] In some embodiments, Figure 6 The model training method shown is based on the following model. The model includes an embedding layer, a first graph neural network layer, an attention layer, an encoding layer, and a second graph neural network layer.

[0101] The embedding layer is used to encode each Chinese character in the commonly used Chinese character set into a word vector, and provide a plurality of first word vectors corresponding to a plurality of Chinese characters belonging to a first confusion set to a first graph neural network layer, and provide a plurality of second word vectors corresponding to a plurality of Chinese characters not belonging to the first confusion set to a classification layer, and encode each Chinese character in a sentence sample into a word vector, wherein the first confusion set indicates easily confused Chinese characters;

[0102] The first graph neural network layer is used to perform graph calculation according to the graph data corresponding to the plurality of first word vectors and the first confusion set to generate a plurality of new graph data, obtain a plurality of third word vectors from the new graph data, and provide the plurality of third word vectors to the attention layer and the second graph neural network layer;

[0103] The attention layer is used to calculate the confusable values ​​between the plurality of third character vectors, and thereby obtain a second confusion set, where the second confusion set indicates not only the confusable Chinese characters but also the confusable values;

[0104] The second graph neural network layer is used to perform graph calculation according to the graph data corresponding to the plurality of third word vectors and the second confusion set to generate new graph data, obtain a plurality of fourth word vectors from the new graph data, and provide the plurality of fourth word vectors to the classification layer, wherein the plurality of fourth word vectors and the plurality of second word vectors are organized into a weight parameter matrix of the classification layer;

[0105] The encoding layer is used to generate multiple word vectors containing sentence information based on the word vector of each Chinese character in the sentence sample output by the embedding layer.

[0106] Referring to the above, an embodiment of the present disclosure provides a Chinese spelling correction method, which is applied to the server side, and its flow chart is as follows: Figure 7 shown.

[0107] In step S701, an input sentence is received.

[0108] In step S702, the input sentence is provided to a Chinese spelling error correction model to obtain a score representing the degree of confusion between each Chinese character in the commonly used Chinese character set and each Chinese character in the input sentence.

[0109] In this step, the Chinese spelling error correction model includes an embedding layer, an encoding layer and a classification layer. The embedding layer is used to encode each Chinese character in the input sentence into a word vector, the encoding layer is used to encode the word vector of each Chinese character in the input sentence into a word vector with sentence information, and the classification layer is used to perform matrix multiplication of each word vector containing sentence information with the word vector corresponding to each Chinese character in the commonly used Chinese character set to obtain a score representing the degree of confusion between the corresponding Chinese character of the input sentence and each Chinese character in the commonly used Chinese character set.

[0110] In step S703, for each Chinese character in the input sentence, the Chinese character corresponding to the maximum score is used as the error correction Chinese character.

[0111] According to this embodiment, the Chinese character is spelled corrected by calculating a score representing the degree of confusion between each Chinese character in the input sentence and each Chinese character in the conventional Chinese character set, thereby solving spelling errors that are neither similar in pronunciation nor in shape.

[0112] In some embodiments, the Figure 3 The model shown in FIG. 1 is trained to obtain the Chinese spelling error correction model in this embodiment, and the Chinese spelling error correction model includes the following information obtained from the trained model: Figure 3 An embedding layer 311, an encoding layer 315 and a classification layer 314 are shown.

[0113] Accordingly, the present disclosure also provides a Chinese error correction method corresponding to the above Chinese error correction method and executed in a terminal, comprising: receiving an input sentence and sending it to a server, and receiving and displaying the corrected sentence from the server, wherein the server uses a Chinese spelling correction model to obtain a score representing the degree of confusion between each Chinese character in a commonly used Chinese character set and each Chinese character in the input sentence, and performs Chinese spelling correction on the input sentence according to the score to obtain the corrected sentence. In some embodiments, for example, Figure 3The model shown in FIG. 1 is trained to obtain the Chinese spelling error correction model in this embodiment, and the Chinese spelling error correction model includes the following information obtained from the trained model: Figure 3 An embedding layer 311, an encoding layer 315 and a classification layer 314 are shown.

[0114] Accordingly, the present disclosure also provides a search method, including: receiving a search term; providing the search term to a Chinese spelling correction model to obtain a score representing the degree of confusion between each Chinese character in a commonly used Chinese character set and each Chinese character in the search term; for each Chinese character in the search term, using the Chinese character corresponding to the maximum score as its correction Chinese character to obtain a corrected search term; and using the corrected search term and / or the search term to perform a database search. In some embodiments, the following may be used for the following: Figure 3 The model shown in FIG. 1 is trained to obtain the Chinese spelling error correction model in this embodiment, and the Chinese spelling error correction model includes the following information obtained from the trained model: Figure 3 An embedding layer 311, an encoding layer 315 and a classification layer 314 are shown.

[0115] In summary, the embodiments of the present disclosure provide a model training method, a Chinese spelling error correction method, and a retrieval method. In these methods, a Chinese spelling error correction model is obtained by training a specific model, and the specific model dynamically generates a confusion set using a graph neural network and an attention mechanism, and the confusion set adds a confusion value that characterizes the degree of confusion of Chinese characters, so that the difference in the degree of similarity between the sound and shape of characters can be further distinguished in Chinese error correction. Experiments show that the Chinese error correction model obtained thereby can effectively improve the accuracy and performance of Chinese error correction.

[0116] In addition, the embodiment of the present disclosure also provides a computer-readable medium for storing computer-readable instructions for implementing the above-mentioned model training method, Chinese error correction method and retrieval method.

[0117] Commercial value of this disclosure

[0118] The disclosed embodiments provide a model training method, a Chinese error correction method, and a retrieval method. Compared with the prior art, this method can distinguish the differences between characters in terms of the similarity of sound and form, and experiments show that this method can effectively improve the accuracy and performance of Chinese error correction. Therefore, the above method should have practical application value and economic value.

[0119] It should be understood that the above is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, there are many variations of the embodiments of this specification. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

[0120] It should be understood that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0121] It should be understood that the above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0122] It should be understood that an element described in singular form herein or shown in the drawings as only one does not limit the number of the element to one. In addition, modules or elements described or shown as separate herein may be combined into a single module or element, and modules or elements described or shown as single herein may be split into multiple modules or elements.

[0123] It should also be understood that the terms and expressions used herein are for description only, and one or more embodiments of this specification should not be limited to these terms and expressions. The use of these terms and expressions does not mean to exclude any equivalent features of the illustrations and descriptions (or parts thereof), and it should be recognized that various modifications that may exist should also be included in the scope of the claims. Other modifications, changes and substitutions may also exist. Accordingly, the claims should be deemed to cover all such equivalents.

Claims

1. A model training method, comprising: Input the sentence sample to the embedding layer to obtain the character vector of each Chinese character in the sentence sample; Obtaining a plurality of word vectors containing sentence information based on the word vector of each Chinese character in the sentence sample; Inputting the plurality of word vectors containing sentence information to a classification layer, the classification layer multiplies each word vector containing sentence information by each one-dimensional vector in a weight parameter matrix to obtain a plurality of scores, wherein each one-dimensional vector in the weight parameter matrix is ​​used to represent a word vector of a Chinese character in a commonly used Chinese character set, and the plurality of scores are used to represent a degree of confusion between each Chinese character in the sentence sample and each Chinese character in the commonly used Chinese character set; For each Chinese character in the sentence sample, the Chinese character corresponding to the maximum score among the multiple scores obtained is used as the error correction Chinese character; The weight parameter matrix is ​​adjusted using the loss between the correct error-corrected Chinese characters and the incorrect error-corrected Chinese characters, and the weight parameters of the corresponding layers of the specified model that generates the weight parameter matrix are further adjusted using a back-propagation algorithm. The specified model includes the embedding layer, at least one graph neural network layer coupled to the embedding layer, and the classification layer coupled to the at least one graph neural network layer, and the graph neural network layer is used to perform graph calculation according to the input word vector and the graph data corresponding to the confusion set, and generate a new word vector; Wherein, the specified model also includes: an attention layer, which is used to calculate the easily confused values ​​between multiple third word vectors to obtain a second confusion set to determine the weight parameter matrix, wherein the multiple third word vectors are word vectors obtained by performing graph calculations based on multiple first word vectors and graph data corresponding to the first confusion set, the multiple first word vectors are used to represent word vectors corresponding to multiple Chinese characters in the first confusion set, and the first confusion set is used to indicate easily confused Chinese characters in the commonly used Chinese character set.

2. The model training method according to claim 1, wherein: The at least one graph neural network layer includes a first graph neural network layer and a second graph neural network layer, and the specified model further includes: an encoding layer, The embedding layer is further used to encode each Chinese character in the commonly used Chinese character set into a word vector, and provide the plurality of first word vectors corresponding to the plurality of Chinese characters belonging to the first confusion set to the first graph neural network layer, and provide the plurality of second word vectors corresponding to the plurality of Chinese characters not belonging to the first confusion set to the classification layer; The first graph neural network layer is used to perform graph calculation according to the plurality of first word vectors and the graph data corresponding to the first confusion set to generate a plurality of new graph data, obtain the plurality of third word vectors from the new graph data, and provide the plurality of third word vectors to the attention layer and the second graph neural network layer; The second graph neural network layer is used to perform graph calculation according to the plurality of third word vectors and the graph data corresponding to the second confusion set to generate new graph data, obtain a plurality of fourth word vectors from the new graph data, and provide the plurality of fourth word vectors to the classification layer, wherein the second confusion set is used to indicate the easily confused Chinese characters in the commonly used Chinese character set and indicates the easily confused numerical value, and the plurality of fourth word vectors and the plurality of second word vectors are organized into a weight parameter matrix of the classification layer; The encoding layer is used to execute the step of obtaining a plurality of word vectors containing sentence information based on the word vector of each Chinese character in the sentence sample; The method of using the back-propagation algorithm to adjust the weight parameters of the corresponding layers of the specified model that generates the weight parameter matrix includes: adjusting the weight parameters of the second graph neural network, the first graph neural network layer, the attention layer and the embedding layer in sequence through the back-propagation algorithm.

3. The model training method according to any one of claims 1 to 2, wherein: The step of obtaining a plurality of word vectors containing sentence information based on the word vector of each Chinese character in the sentence sample comprises: Obtaining position information and context information of each Chinese character in the sentence sample; and A word vector containing sentence information is obtained based on the word vector of each Chinese character and the position information and context information of each Chinese character in the sentence sample.

4. The model training method according to claim 2, wherein: The calculating of the easily confused values ​​between the plurality of third word vectors includes: calculating the easily confused values ​​between the plurality of third word vectors using an attention mechanism, the attention layer further including: uniformly processing the confused values ​​that are less than or equal to a set threshold as a set negative value; and processing the easily confused values ​​using a normalization method.

5. The model training method according to claim 2, wherein: The embedding layer and the encoding layer adopt corresponding layers in the BERT model.

6. A system deployed with a Chinese spelling error correction model, characterized in that: The Chinese spelling error correction model includes: an embedding layer, an encoding layer, a first graph neural network layer, a second graph neural network layer, an attention layer and a classification layer; The embedding layer is used to encode each Chinese character in the sample sentence into a word vector, and to encode each Chinese character in the commonly used Chinese character set into a word vector, to provide a plurality of first word vectors corresponding to a plurality of Chinese characters belonging to a first confusion set to the first graph neural network layer, and to provide a plurality of second word vectors corresponding to a plurality of Chinese characters not belonging to the first confusion set to the classification layer, wherein the first confusion set indicates easily confused Chinese characters in the commonly used Chinese character set; The encoding layer is used to encode the word vector of each Chinese character in the sample sentence into a word vector with sentence information; The first graph neural network layer is used to perform graph calculation according to the plurality of first word vectors and the graph data corresponding to the first confusion set to generate a plurality of new graph data, obtain a plurality of third word vectors from the new graph data, and provide the plurality of third word vectors to the attention layer and the second graph neural network layer; The attention layer is used to calculate the confusable values ​​between the plurality of third character vectors, and thereby obtain a second confusion set, wherein the second confusion set indicates not only the confusable Chinese characters in the commonly used Chinese character set, but also the confusable values; The second graph neural network layer is used to perform graph calculation according to the plurality of third word vectors and the graph data corresponding to the second confusion set to generate new graph data, obtain a plurality of fourth word vectors from the new graph data, and provide the plurality of fourth word vectors to the classification layer, wherein the plurality of fourth word vectors and the plurality of second word vectors are organized into a weight parameter matrix of the classification layer; The classification layer is used to multiply each word vector containing sentence information with each one-dimensional vector in the weight parameter matrix to obtain multiple scores, wherein the Chinese character corresponding to the maximum score among the multiple scores is used as the error correction Chinese character of the corresponding Chinese character, each one-dimensional vector in the weight parameter matrix is ​​used to represent the word vector of a Chinese character in a commonly used Chinese character set, and the multiple scores are used to represent the degree of confusion between each Chinese character in the sentence sample and each Chinese character in the commonly used Chinese character set.

7. A Chinese spelling correction method, applied to a server, comprising: receiving an input sentence; as well as Providing the input sentence to a Chinese spelling error correction model to obtain a score representing the degree of confusion between each Chinese character in a commonly used Chinese character set and each Chinese character in the input sentence; For each Chinese character in the input sentence, the Chinese character corresponding to the maximum score is used as the error correction Chinese character. The Chinese spelling error correction model includes an embedding layer, an encoding layer and a classification layer, wherein the embedding layer is used to encode each Chinese character in the input sentence into a word vector, the encoding layer is used to encode the word vector of each Chinese character in the input sentence into a word vector with sentence information, and the classification layer is used to perform matrix multiplication of each word vector containing sentence information with the word vector corresponding to each Chinese character in the commonly used Chinese character set to obtain a score representing the degree of confusion between the corresponding Chinese character of the input sentence and each Chinese character in the commonly used Chinese character set; Among them, the Chinese spelling error correction model also includes: an attention layer, which is used to calculate the easily confused values ​​between multiple third word vectors to obtain a second confusion set to determine a weight parameter matrix, wherein the multiple third word vectors are word vectors obtained by performing graph calculations based on multiple first word vectors and graph data corresponding to the first confusion set, the multiple first word vectors are used to represent word vectors corresponding to multiple Chinese characters in the first confusion set, and the first confusion set is used to indicate easily confused Chinese characters in the commonly used Chinese character set.

8. A Chinese error correction method, applied to a terminal, comprising: An input sentence is received and sent to a server, the server uses a Chinese spelling correction model to obtain a score representing the degree of confusion between each Chinese character in a commonly used Chinese character set and each Chinese character in the input sentence, and performs Chinese spelling correction on the input sentence according to the score to obtain a corrected sentence; Receive and display the corrected sentence from the server; Among them, the Chinese spelling error correction model also includes: an attention layer, which is used to calculate the easily confused values ​​between multiple third word vectors to obtain a second confusion set to determine a weight parameter matrix, wherein the multiple third word vectors are word vectors obtained by performing graph calculations based on multiple first word vectors and graph data corresponding to the first confusion set, the multiple first word vectors are used to represent word vectors corresponding to multiple Chinese characters in the first confusion set, and the first confusion set is used to indicate easily confused Chinese characters in the commonly used Chinese character set.

9. A retrieval method comprising: receiving a search term; Providing the search term to a Chinese spelling error correction model to obtain a score representing the degree of confusion between each Chinese character in a commonly used Chinese character set and each Chinese character in the search term; For each Chinese character in the search term, the Chinese character corresponding to the maximum score is used as the error correction Chinese character to obtain the corrected search term; Performing a database search using the corrected search term and / or the search term; Among them, the Chinese spelling error correction model also includes: an attention layer, which is used to calculate the easily confused values ​​between multiple third word vectors to obtain a second confusion set to determine a weight parameter matrix, wherein the multiple third word vectors are word vectors obtained by performing graph calculations based on multiple first word vectors and graph data corresponding to the first confusion set, the multiple first word vectors are used to represent word vectors corresponding to multiple Chinese characters in the first confusion set, and the first confusion set is used to indicate easily confused Chinese characters in the commonly used Chinese character set.

10. An electronic device, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the model training method as described in any one of claims 1 to 5 is implemented, or when the processor executes the program, the Chinese spelling correction method as described in any one of claims 7 to 8 is implemented, or when the processor executes the program, the retrieval method as described in claim 9 is implemented.

11. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the model training method as described in any one of claims 1 to 5 is implemented, or when the computer program is executed by a processor, the Chinese spelling correction method as described in any one of claims 7 to 8 is implemented, or when the computer program is executed by a processor, the retrieval method as described in claim 9 is implemented.

Images