Method, device, equipment and computer-readable storage medium for identifying misspelled characters

Abstract

The present application provides a method, apparatus, device, computer-readable storage medium, and model training method for identifying misspelled words; wherein, the method for identifying misspelled words includes: obtaining a sentence to be identified, and encoding features of the words to be identified in the sentence; utilizing a trained language model to extract features of the words to be identified in the sentence, and obtaining context features of the words to be identified in the sentence; utilizing a trained deep neural network model to identify the encoding features and context features of the words to be identified in the sentence, and obtaining a recognition result representing whether the words to be identified are incorrect. In the present application, when identifying misspelled words, the strong generalization characteristics of the language model and the high recognition accuracy of the deep neural network model can be utilized to better identify misspelled words in the sentence, and there is no need to manually set and maintain a large number of model fusion rules, which can reduce labor costs.

Description

Technical Field

[0001] This application relates to, but is not limited to, the field of artificial intelligence, and particularly relates to a method, apparatus, device, computer-readable storage medium for identifying misspelled words and a model training method. Background Art

[0002] In today's era of self-media, creators on various content distribution platforms create millions of articles every day for recommendation to hundreds of millions of users. Among the large number of articles generated every day, there are often some misspelled words, which greatly affects the reading experience of users. Compared with English, Chinese expressions are more rich and diverse, and the grammar and syntactic structures are also more flexible and changeable. Therefore, the reasons for errors are even more diverse. For example, there are shallow spelling errors, such as homophone errors, near-homophone errors, and shape-similar errors, etc.; there are also deep grammar errors, such as misuse of prepositions, improper collocations, etc.; and there are even more difficult-to-capture logical errors, such as the misuse of "ba" sentences and "bei" sentences. This greatly increases the difficulty of Chinese error correction and makes the problems faced by Chinese error correction more challenging. Therefore, how to quickly and accurately identify misspelled words in an article is an important but challenging task.

[0003] The misspelled word recognition methods in related technologies mainly include misspelled word recognition based on a language model and misspelled word recognition based on a deep neural network model. The misspelled word recognition method based on a language model has good generalization, but the recognition accuracy is relatively low. Compared with the misspelled word recognition method based on a language model, the misspelled word recognition method based on a deep neural network model has higher recognition accuracy, but relatively poor generalization. In related technologies, usually these two models are respectively used to identify misspelled words in the text, and then the recognition results of the two models are fused based on a series of rules to obtain the final recognition result. However, this fusion of the two models is an artificial shallow fusion method, which requires setting a large number of thresholds and adjusting a large number of rules, making the system difficult to maintain and further optimize. Summary of the Invention

[0004] Embodiments of this application provide a method, apparatus, device, computer-readable storage medium for identifying misspelled words and a model training method, which can utilize the strong generalization characteristics of the language model and the high recognition accuracy characteristics of the deep neural network model when identifying misspelled words, better identify misspelled words in a sentence, and because on the basis of the deep neural network model, the features of the language model are deeply fused to perform end-to-end misspelled word recognition, without the need to manually set and maintain a large number of model fusion rules, thus greatly reducing the labor cost.

[0005] The technical solution of the embodiments of this application is implemented as follows:

[0006] The present invention provides a method for identifying misspelled characters, including:

[0007] Obtaining a sentence to be recognized and encoding features of words to be recognized in the sentence;

[0008] Using the trained language model, extracting features of the words to be recognized in the sentence to obtain context features of the words to be recognized in the sentence;

[0009] The trained deep neural network model is used to identify the encoding features and context features of the words to be recognized in the sentence, and obtain a recognition result indicating whether the words to be recognized are incorrect.

[0010] In some embodiments, the language model includes a bidirectional language model, the contextual features include implicit features, and the use of the trained language model to extract features of the words to be recognized in the sentence to obtain the contextual features of the words to be recognized in the sentence includes: using the trained bidirectional language model to determine a first vector representing the first N characters of the words to be recognized in the case of forward prediction, and a second vector representing the last M characters of the words to be recognized in the case of reverse prediction, wherein N and M are both integers greater than or equal to 0 and less than the length of the sentence; based on the first vector and the second vector, obtaining the implicit features of the words to be recognized in the sentence.

[0011] In some embodiments, the contextual features include explicit features, and the method of using a trained language model to extract features of the words to be recognized in the sentence to obtain contextual features of the words to be recognized in the sentence includes: using a trained language model to predict the words to be recognized in the sentence to obtain a probability parameter representing whether the words to be recognized are wrong; and determining the explicit features of the words to be recognized in the sentence based on the probability parameter.

[0012] In some embodiments, the probability parameters include the current word probability and the maximum candidate word probability, and determining the explicit features of the word to be recognized in the sentence based on the probability parameters includes: discretizing the difference between the current word probability and the maximum candidate word probability to obtain a discretized difference; determining the first target value interval to which the discretized difference belongs from a preset first value interval list; and determining the explicit features of the word to be recognized in the sentence based on the first target value interval.

[0013] In some embodiments, the probability parameter includes at least one perplexity reduction ratio of the word to be recognized, and the method of using the trained language model to predict the word to be recognized in the sentence to obtain the probability parameter characterizing whether the word to be recognized is wrong includes: using the trained language model to determine the current perplexity of the sentence when the position of the word to be recognized is the current word, and the post-replacement perplexity of the sentence when the position is replaced by each candidate typo in the typo candidate set of the word to be recognized; based on the current perplexity of the word to be recognized and at least one post-replacement perplexity, determining at least one perplexity reduction ratio of the word to be recognized.

[0014] In some embodiments, determining the explicit features of the word to be recognized in the sentence based on the probability parameter includes: discretizing at least one perplexity reduction ratio of the word to be recognized to obtain the discretized perplexity reduction ratio; determining the second target value interval to which the discretized perplexity reduction ratio belongs from a preset second value interval list; and determining the explicit features of the word to be recognized in the sentence based on the second target value interval.

[0015] In some embodiments, the method of using a trained deep neural network model to identify the encoding features and context features of the words to be identified in the sentence to obtain a recognition result representing whether the words to be identified are incorrect includes: merging the encoding features and context features of the words to be identified to obtain a fusion vector of the words to be identified; using a trained deep neural network model to identify the fusion vector of the words to be identified in the sentence to obtain a recognition result representing whether the words to be identified are incorrect.

[0016] In some embodiments, before using the trained deep neural network model to identify the encoding features and context features of the words to be recognized in the sentence to obtain a recognition result indicating whether the words to be recognized are incorrect, the method further includes: obtaining each sample sentence in the annotated sample set and the encoding features of the words to be recognized in each of the sample sentences; each sample sentence in the annotated sample set has a label sequence for marking whether the words to be recognized in the sample sentence are typos; for each of the sample sentences, using the trained language model to extract features of the words to be recognized in the sample sentence to obtain the context features of the words to be recognized in the sample sentence, and using the deep neural network model to identify the encoding features and context features of the words to be recognized in the sample sentence to obtain a recognition result indicating whether the words to be recognized in the sample sentence are incorrect; based on the recognition result corresponding to each sample sentence and the label sequence of each sample sentence, using a loss function to determine a loss value; and when it is determined that the loss function has not converged according to the loss value, updating the parameters of the deep neural network model based on a parameter optimization algorithm.

[0017] The present invention provides a model training method, including:

[0018] Obtaining encoding features of each sample sentence in the labeled sample set and the words to be recognized in each of the sample sentences; each sample sentence in the labeled sample set has a label sequence for marking whether the words to be recognized in the sample sentence are typos;

[0019] For each of the sample sentences, using the trained language model, extracting features of the words to be recognized in the sample sentences to obtain context features of the words to be recognized in the sample sentences, and using the deep neural network model to identify the encoding features and context features of the words to be recognized in the sample sentences to obtain a recognition result indicating whether the words to be recognized in the sample sentences are incorrect;

[0020] Based on the recognition result corresponding to each sample sentence and the label sequence of each sample sentence, a loss value is determined using a loss function;

[0021] When it is determined according to the loss value that the loss function has not converged, the parameters of the deep neural network model are updated based on a parameter optimization algorithm.

[0022] The present invention provides a device for identifying misspelled characters and words, including:

[0023] A first acquisition module is used to acquire a sentence to be recognized and encoding features of words to be recognized in the sentence;

[0024] A first extraction module is configured to extract features of the words to be recognized in the sentence using the trained language model to obtain context features of the words to be recognized in the sentence;

[0025] The first recognition module is used to use the trained deep neural network model to identify the encoding features and context features of the words to be recognized in the sentence, and obtain a recognition result indicating whether the words to be recognized are incorrect.

[0026] In some embodiments, the language model includes a bidirectional language model, the contextual features include implicit features, and the first extraction module is further used to: use the trained bidirectional language model to determine a first vector representing the first N characters of the word to be recognized in the case of forward prediction, and a second vector representing the last M characters of the word to be recognized in the case of reverse prediction, wherein N and M are both integers greater than or equal to 0 and less than the length of the sentence; based on the first vector and the second vector, obtain the implicit features of the word to be recognized in the sentence.

[0027] In some embodiments, the contextual features include explicit features, and the first extraction module is further used to: use a trained language model to predict the words to be recognized in the sentence to obtain a probability parameter representing whether the words to be recognized are wrong; based on the probability parameter, determine the explicit features of the words to be recognized in the sentence.

[0028] In some embodiments, the probability parameters include the current word probability and the maximum candidate word probability, and the first extraction module is further used to: discretize the difference between the current word probability and the maximum candidate word probability to obtain a discretized difference; determine the first target value interval to which the discretized difference belongs from a preset first value interval list; and determine the explicit features of the word to be recognized in the sentence based on the first target value interval.

[0029] In some embodiments, the probability parameter includes at least one perplexity reduction ratio of the word to be recognized, and the first extraction module is further used to: use a trained language model to determine the current perplexity of the sentence when the position of the word to be recognized is the current word, and the post-replacement perplexity of the sentence when the position is replaced by each candidate typo in the typo candidate set of the word to be recognized; based on the current perplexity of the word to be recognized and at least one post-replacement perplexity, determine at least one perplexity reduction ratio of the word to be recognized.

[0030] In some embodiments, the first extraction module is further used to: discretize at least one perplexity reduction ratio of the word to be recognized to obtain a discretized perplexity reduction ratio; determine a second target value interval to which the discretized perplexity reduction ratio belongs from a preset second value interval list; and determine the explicit features of the word to be recognized in the sentence based on the second target value interval.

[0031] In some embodiments, the first recognition module is further used to: merge the encoding features and context features of the word to be recognized to obtain a fusion vector of the word to be recognized; use a trained deep neural network model to recognize the fusion vector of the word to be recognized in the sentence to obtain a recognition result representing whether the word to be recognized is wrong.

[0032] In some embodiments, the device further includes: a second acquisition module for acquiring each sample sentence in the annotated sample set and the encoding features of the words to be recognized in each of the sample sentences; each sample sentence in the annotated sample set has a label sequence for marking whether the words to be recognized in the sample sentence are typos; a second recognition module for extracting features of the words to be recognized in the sample sentence using the trained language model for each of the sample sentences, obtaining the context features of the words to be recognized in the sample sentence, and using the deep neural network model to identify the encoding features and context features of the words to be recognized in the sample sentence, obtaining a recognition result representing whether the words to be recognized in the sample sentence are wrong; a first determination module for determining a loss value using a loss function based on the recognition result corresponding to each sample sentence and the label sequence of each sample sentence; a first update module for updating the parameters of the deep neural network model based on a parameter optimization algorithm when it is determined that the loss function has not converged according to the loss value.

[0033] The present invention provides a model training apparatus, including:

[0034] A third acquisition module is configured to acquire encoding features of each sample sentence in the annotated sample set and the words to be recognized in each of the sample sentences; each sample sentence in the annotated sample set has a label sequence for marking whether the words to be recognized in the sample sentence are typos;

[0035] a third recognition module configured to, for each sample sentence, use the trained language model to perform feature extraction on the words to be recognized in the sample sentence, obtain context features of the words to be recognized in the sample sentence, and use the deep neural network model to recognize the encoding features and context features of the words to be recognized in the sample sentence, to obtain a recognition result indicating whether the words to be recognized in the sample sentence are incorrect;

[0036] A second determination module is configured to determine a loss value using a loss function based on a recognition result corresponding to each sample sentence and a label sequence of each sample sentence;

[0037] The second updating module is used to update the parameters of the deep neural network model based on a parameter optimization algorithm when it is determined according to the loss value that the loss function has not converged.

[0038] An embodiment of the present application provides a device for identifying misspelled words, comprising: a memory for storing executable instructions; and a processor for implementing the method for identifying misspelled words provided in the embodiment of the present application when executing the executable instructions stored in the memory.

[0039] An embodiment of the present application provides a model training device, comprising: a memory for storing executable instructions; and a processor for implementing the model training method provided in the embodiment of the present application when executing the executable instructions stored in the memory.

[0040] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute the method provided in the embodiment of the present application.

[0041] The embodiments of the present application have the following beneficial effects:

[0042] First, obtain the sentence to be recognized and the coding features of the words to be recognized in the sentence; then, use the trained language model to extract features of the words to be recognized in the sentence to obtain the context features of the words to be recognized in the sentence; finally, use the trained deep neural network model to identify the coding features and context features of the words to be recognized in the sentence to obtain a recognition result that characterizes whether the words to be recognized are wrong. In this way, when identifying wrong words, we can not only take advantage of the strong generalization of the language model, but also take advantage of the high recognition accuracy of the deep neural network model, so that we can better identify wrong words in the sentence. And because the features of the language model are deeply integrated on the basis of the deep neural network model to perform end-to-end wrong words recognition, there is no need to manually set and maintain a large number of model fusion rules, which can greatly reduce labor costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is an optional architectural diagram of a system for identifying misspelled characters provided in an embodiment of the present application;

[0044] Figure 2A This is an optional structural diagram of a device for identifying misspelled characters provided in an embodiment of the present application;

[0045] Figure 2B This is an optional structural diagram of the model training device provided in the embodiment of the present application;

[0046] Figure 3 This is an optional flowchart of the method for identifying misspelled characters provided in the embodiment of the present application;

[0047] Figure 4 This is an optional flowchart of the method for identifying misspelled characters provided in the embodiment of the present application;

[0048] Figure 5 This is an optional flowchart of the method for identifying misspelled characters provided in the embodiment of the present application;

[0049] Figure 6 This is an optional flowchart of the method for identifying misspelled characters provided in the embodiment of the present application;

[0050] Figure 7 This is an optional flowchart of the method for identifying misspelled characters provided in the embodiment of the present application;

[0051] Figure 8 This is an optional flowchart of the model training method provided in the embodiment of the present application;

[0052] Figure 9A This is a schematic diagram of the overall architecture of a deep neural network-based typo recognition model that integrates a bidirectional neural language model, provided in an embodiment of the present application;

[0053] Figure 9B A schematic diagram of the composition architecture of a bidirectional neural language model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0055] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0056] If similar descriptions of "first / second" appear in the application documents, the following explanation is added. In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific order for the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0058] In order to better understand the method for identifying misspelled words provided in the embodiment of the present application, the misspelled word recognition scheme in the related art is first described below.

[0059] In related technologies, there are two typo recognition schemes:

[0060] 1) Typos recognition method based on language model. This method relies on language models trained on large-scale corpus, and generally includes two language models: one is a statistical language model based on a multi-source language model (N-gram), and the other is a neural network language model based on deep learning. After the language model is trained, the perplexity (PPL) value that represents the fluency of the sentence can be calculated to determine whether the sentence is a reasonable natural language expression. Specifically, a sentence is input, and a variety of candidate words (including homophone candidates, near-phonetic candidates, similar candidates, etc.) are used to replace each word. The PPL score reduction ratio before and after the sentence is replaced is calculated. If it is greater than a certain threshold, it means that the character or word is a typo.

[0061] 2) Typos recognition method based on deep neural network model. This method mainly uses a sequence labeling model to predict whether each word in the input sentence is a typo. Commonly used sequence labeling models based on deep neural networks include the Bi-directional Long Short-Term Memory (Bi-LSTM) model, the Long Short-Term Memory (LSTM) model combined with the conditional random field (CRF), and the Bidirectional Encoder Representation from Transformers (BERT) model. Among them, the BERT-based deep neural network model comprehensively updates the optimal indicators on multiple natural language processing tasks. Therefore, the BERT-based sequence labeling model is widely used in typos recognition tasks.

[0062] Among the typo recognition methods of the above-mentioned related technologies, the advantages of the typo recognition method based on the language model are fast calculation speed, simple implementation, and good generalization because it is trained with a large amount of unsupervised text, but the typo recognition accuracy is not high; the typo recognition method based on the deep neural network model uses targeted annotated training data to train the model, and the recognition accuracy of typos is higher than that of the language model, but because the deep neural network model is only trained on a small amount of annotated training data, the generalization is poorer than that of the language model trained with a large amount of text.

[0063] The embodiments of the present application provide a method, apparatus, device and computer-readable storage medium for identifying misspelled words. When identifying misspelled words, the method can utilize the strong generalization characteristics of the neural language model and the high accuracy characteristics of the deep neural network recognition model to better identify misspelled words in the sentence. Moreover, on the basis of the deep neural network model, the features of the neural language model are deeply fused to perform end-to-end recognition of misspelled words. There is no need to manually set and maintain a large number of model fusion rules, which can greatly reduce labor costs. The following describes an exemplary application of the device for identifying misspelled words provided by the embodiment of the present application. The device for identifying misspelled words and the model training device provided by the embodiment of the present application can be implemented as various types of user terminals such as laptop computers, tablet computers, desktop computers, car navigation systems, set-top boxes, mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), and can also be implemented as servers. Below, an exemplary application of the device for identifying misspelled words when it is implemented as a server will be described.

[0064] See also Figure 1 , Figure 1 This is an optional architectural diagram of the system 100 for identifying misspelled words provided in an embodiment of the present application, which can realize the recognition of misspelled words in the sentence to be recognized. The terminal (terminal 400-1 and terminal 400-2 are shown as examples) is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0065] The terminal is used to: display an interactive interface for users to identify wrong words in sentences on a graphical interface (graphic interface 410-1 and graphical interface 410-2 are shown as examples), receive the user's wrong word recognition operation for the sentence to be recognized, and send the sentence to be recognized to the server 200.

[0066] The server 200 is used to: obtain a sentence to be recognized and the encoding features of the words to be recognized in the sentence; use a trained language model to extract features of the words to be recognized in the sentence to obtain context features of the words to be recognized in the sentence; use a trained deep neural network model to recognize the encoding features and context features of the words to be recognized in the sentence to obtain a recognition result indicating whether the words to be recognized are incorrect.

[0067] In addition, the system for identifying misspelled words involved in the embodiments of the present application may also be a distributed system applied to a blockchain system. The distributed system may be a distributed node formed by multiple nodes and clients. The nodes may be any form of computing device in the access network, such as a server, a user terminal, etc. A peer-to-peer (P2P) network is formed between the nodes, and the misspelled word recognition device implemented as a server may be a node on the blockchain.

[0068] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal 400 may be an automatic map data collection vehicle, a smartphone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to these. The terminal and the server may be connected directly or indirectly via wired or wireless communication, which is not limited in the embodiments of the present invention.

[0069] See also Figure 2A , Figure 2A 2 is a schematic diagram of the structure of the device 200 for identifying misspelled characters provided in an embodiment of the present application. Figure 2AThe device 200 for identifying misspelled words includes: at least one processor 210, a memory 250, at least one network interface 220, and a user interface 230. The various components in the device 200 for identifying misspelled words are coupled together via a bus system 240. It is understood that the bus system 240 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 240 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 240 is not described in detail. Figure 2A Various buses are labeled as bus system 240 .

[0070] The processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0071] The user interface 230 includes one or more output devices 231 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 230 also includes one or more input devices 232, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0072] The memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 250 may optionally include one or more storage devices that are physically remote from the processor 210.

[0073] The memory 250 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 250 described in the embodiments of the present application is intended to include any suitable type of memory.

[0074] In some embodiments, the memory 250 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0075] Operating system 251, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks;

[0076] A network communication module 252 for reaching other computing devices via one or more (wired or wireless) network interfaces 220 , exemplary network interfaces 220 including Bluetooth, WiFi, and USB;

[0077] a presentation module 253 for enabling presentation of information via one or more output devices 231 (e.g., a display screen, a speaker, etc.) associated with the user interface 230 (e.g., a user interface for operating peripheral devices and displaying content and information);

[0078] The input processing module 254 is configured to detect one or more user inputs or interactions from one of the one or more input devices 232 and to translate the detected inputs or interactions.

[0079] In some embodiments, the device for identifying misspelled characters provided in the embodiments of the present application can be implemented in software. Figure 2A A device 255 for identifying misspelled words stored in a memory 250 is shown, which may be software in the form of a program or plug-in, and includes the following software modules: a first acquisition module 2551, a first extraction module 2552, and a first identification module 2553. These modules are logical, and therefore can be arbitrarily combined or further split according to the functions implemented.

[0080] The functions of each module will be described below.

[0081] In other embodiments, the device for recognizing misspelled words provided in the embodiments of the present application may be implemented in hardware. As an example, the device for recognizing misspelled words provided in the embodiments of the present application may be a processor in the form of a hardware decoding processor, which is programmed to execute the method for recognizing misspelled words provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor may be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0082] See also Figure 2B , Figure 2B3 is a schematic diagram of the structure of the model training device 300 provided in an embodiment of the present application. Figure 2B The model training device 300 shown includes: at least one processor 310, a memory 350, at least one network interface 320 and a user interface 330. The various components in the model training device 300 are coupled together via a bus system 340. It is understood that the bus system 340 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 340 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 340 is not described in detail. Figure 2B Various buses are labeled as bus system 340 .

[0083] The processor 310 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0084] The user interface 330 includes one or more output devices 331 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 330 also includes one or more input devices 332, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0085] The memory 350 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 350 may optionally include one or more storage devices that are physically remote from the processor 310.

[0086] The memory 350 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 350 described in the embodiments of the present application is intended to include any suitable type of memory.

[0087] In some embodiments, memory 350 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0088] Operating system 351, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks;

[0089] A network communication module 352 for reaching other computing devices via one or more (wired or wireless) network interfaces 320 , exemplary network interfaces 320 including Bluetooth, WiFi, and USB;

[0090] a presentation module 353 for enabling presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 331 (e.g., a display screen, a speaker, etc.) associated with the user interface 330;

[0091] The input processing module 354 is configured to detect one or more user inputs or interactions from one of the one or more input devices 332 and to translate the detected inputs or interactions.

[0092] In some embodiments, the model training device provided in the embodiments of the present application can be implemented in software. Figure 2B A model training device 355 stored in the memory 350 is shown, which can be software in the form of programs and plug-ins, etc., including the following software modules: a third acquisition module 3551, a third identification module 3552, a second determination module 3553 and a second update module 3554. These modules are logical and can therefore be arbitrarily combined or further split according to the functions implemented.

[0093] The functions of each module will be described below.

[0094] In other embodiments, the model training device provided in the embodiments of the present application can be implemented in hardware. As an example, the model training device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the model training method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more ASICs, DSPs, PLDs, complex CPLDs, FPGAs or other electronic components.

[0095] The following will illustrate the method for identifying misspelled words provided in the embodiment of the present application in combination with the exemplary application and implementation of the terminal or server provided in the embodiment of the present application.

[0096] See also Figure 3 , Figure 3 This is an optional flow chart of the method for identifying misspelled words provided in the embodiment of the present application. Figure 3The following steps are explained with reference to the steps shown in the figure. The execution subject of the following steps can be the terminal or server mentioned above.

[0097] In step S101, a sentence to be recognized and encoding features of words to be recognized in the sentence are obtained.

[0098] Here, the sentence to be recognized is a sentence that needs to be identified as having misspellings, and may include but is not limited to one or more of Chinese sentences, English sentences, etc. The words to be recognized may be words in the sentence that need to be identified as having misspellings, and each word in the sentence may be a single word in the sentence, such as a single Chinese character, a single word, etc., or a phrase in the sentence, such as a Chinese character word, a word phrase, etc. The sentence to be recognized may include at least one word to be recognized. During implementation, the sentence to be recognized may be a sentence input by a user or other system, or may be obtained from a preset text, which is not limited here. The words to be recognized in the sentence may be all the words in the sentence, or may be some words filtered out from the sentence based on a preset word filtering strategy. During implementation, the words to be recognized can be filtered out according to the parts of speech of the words in the sentence. For example, the probability of typos appearing in prepositions or modal particles in a sentence is usually low, so typos can be recognized only for words other than prepositions and modal particles in the sentence. The words to be recognized can also be filtered out according to the position of the words in the sentence. For example, if the probability of typos appearing in words at the beginning and end of a sentence is low, typos can be recognized only for words in the middle position of the sentence.

[0099] The coding features of the words to be recognized are the coding information corresponding to the words to be recognized that can uniquely represent the words, and can be represented by any appropriate means such as vectors, matrices, numerical identifiers, etc. During implementation, the coding features of the words to be recognized in the sentence can be obtained by querying a preset coding feature table, such as a word vector table, a word vector table, etc., or the words to be recognized in the sentence can be encoded or vectorized using preset coding rules, mapping algorithms, etc. to obtain the coding features of the words to be recognized, which are not limited here. In some embodiments, the words to be recognized can also be processed by a preset natural language processing model to obtain the coding features of the words to be recognized. For example, the word vectors of the words to be recognized can be generated based on the BERT model as the coding features of the words to be recognized.

[0100] In step S102, the trained language model is used to extract features of the words to be recognized in the sentence to obtain context features of the words to be recognized in the sentence.

[0101] Here, language model can be any suitable probability distribution that can represent natural language text in corpus, for judging whether sentence is the model of normal description, can adopt suitable training mode to train language model in advance.In implementation, language model can be statistical language model, such as unigram language model (Uni-gram) based on N-gram, bigram language model (Bi-gram), trigram language model (Tri-gram) etc., or it can be neural language model, such as probabilistic feedforward neural network language model, cyclic neural network language model, language model based on converter structure, language model based on LSTM structure, language model based on convolutional neural network etc.In addition, language model can be unidirectional language model, or it can be bidirectional language model. Those skilled in the art can select suitable language model according to actual situation when implementing, and are not limited here.

[0102] In some embodiments, the trained language model can be obtained by training based on an unlabeled corpus. By using a large amount of text in the unlabeled corpus to train the language model, the trained language model has good generalization properties.

[0103] The contextual features of the word to be recognized in the sentence may include any appropriate features related to the context of the word to be recognized in the sentence, and may include but are not limited to one or more of the representation vector of the word to be recognized in the context of the sentence, the probability of the word to be recognized appearing in the context of the sentence, the contextual information corresponding to the sentence to be recognized, and the part-of-speech features of the word to be recognized in the context of the sentence. The language model can extract the contextual features of the word to be recognized in the sentence by processing the representation vector of the context of the word to be recognized in the sentence, or predicting the probability distribution of the current word to be recognized or the perplexity of the entire sentence. During implementation, those skilled in the art can determine the appropriate contextual features and the method of extracting the contextual features according to actual conditions, which is not limited here.

[0104] In step S103, the trained deep neural network model is used to identify the encoding features and context features of the words to be recognized in the sentence to obtain a recognition result indicating whether the words to be recognized are incorrect.

[0105] Here, the deep neural network model can be any suitable classification model, and the deep neural network model can be trained in advance using a suitable training method. During implementation, the deep neural network model can be a sequence labeling model based on a deep neural network, which can include but is not limited to one or more of a Bi-LSTM model, an LSTM-CRF model, a BERT model, etc., or other neural network models, such as a convolutional neural network (CNN) model, a recurrent neural network (RNN) model, etc. One or more of the models. Those skilled in the art can select a suitable language model according to actual conditions during implementation, which is not limited here.

[0106] In some embodiments, the trained deep neural network model can be trained based on a preset annotated sample set. The annotated sample set can include multiple sample sentences, each of which has a label sequence that indicates whether the word to be recognized in the sample sentence is a typo. Using the annotated sample set to train the deep neural network model, the resulting trained language model has a high accuracy rate for identifying typos.

[0107] The deep neural network model can identify whether a word to be recognized is incorrect by classifying its encoding features and contextual features in a sentence. The recognition result may include a probability that the word to be recognized is incorrect and a probability that it is correct. Based on the probability of the word to be recognized being incorrect and the probability of the word to be recognized being correct, it can be determined whether the word to be recognized is incorrect. The recognition result may also include a result indicating that the word to be recognized is incorrect or a result indicating that the word to be recognized is correct.

[0108] In an embodiment of the present application, first, the sentence to be recognized and the coding features of the words to be recognized in the sentence are obtained; then, the trained language model is used to extract features of the words to be recognized in the sentence to obtain the context features of the words to be recognized in the sentence; finally, the trained deep neural network model is used to identify the coding features and context features of the words to be recognized in the sentence to obtain a recognition result that characterizes whether the words to be recognized are wrong. In this way, when identifying wrong words, we can not only take advantage of the strong generalization of the language model, but also take advantage of the high recognition accuracy of the deep neural network model, so that we can better identify wrong words in the sentence. And because the features of the language model are deeply integrated on the basis of the deep neural network model to perform end-to-end wrong words recognition, there is no need to manually set and maintain a large number of model fusion rules, which can greatly reduce labor costs.

[0109] In some embodiments, see Figure 4 , Figure 4 This is an optional flow chart of the method for identifying misspelled words provided in the embodiment of the present application, based on Figure 3 , the language model includes a bidirectional language model, the context features include implicit features, Figure 3 The step S102 shown in the figure can be implemented by the following steps S401 to S402. The following description will be made in combination with each step. The execution subject of the following steps can be the terminal or server mentioned above.

[0110] In step S401, the trained bidirectional language model is used to determine a first vector representing the first N characters of the word to be recognized in the case of forward prediction, and a second vector representing the last M characters of the word to be recognized in the case of reverse prediction, where N and M are both integers greater than or equal to 0 and less than the length of the sentence.

[0111] Here, the bidirectional language model can be any suitable model that can perform probability prediction on the words in a sentence from front to back (i.e., forward prediction) and from back to front (i.e., reverse prediction). When performing forward prediction on the word to be recognized, the bidirectional language model can determine the representation vectors of the N words before the word to be recognized in the sentence to be recognized, that is, the first vector representing the first N words of the word to be recognized. When performing reverse prediction on the word to be recognized, the bidirectional language model can determine the representation vectors of the M words after the word to be recognized in the sentence to be recognized, that is, the second vector representing the last M words of the word to be recognized.

[0112] In some embodiments, a bidirectional language model can be determined by two unidirectional language models, that is, it can include a forward prediction language model and a reverse prediction language model. When the forward prediction language model predicts a word to be recognized, it can determine a first vector representing the first N characters of the word to be recognized, and when the reverse prediction language model predicts a word to be recognized, it can determine a second vector representing the last M characters of the word to be recognized.

[0113] In step S402, based on the first vector and the second vector, the implicit features of the word to be recognized in the sentence are obtained.

[0114] Here, the implicit features of the word to be recognized in the sentence may include a representation vector of the word to be recognized in the context of the sentence. During implementation, the first vector and the second vector may be merged to obtain the implicit features of the word to be recognized in the sentence. During implementation, the method for merging the first vector and the second vector may include, but is not limited to, one or more of addition and concatenation.

[0115] In an embodiment of the present application, a trained bidirectional language model is used to determine a first vector representing the first N characters of a word to be recognized in the case of forward prediction, and a second vector representing the last M characters of the word to be recognized in the case of reverse prediction, wherein N and M are both integers greater than or equal to 0 and less than the length of the sentence, and based on the first vector and the second vector, the implicit features of the word to be recognized in the sentence are obtained. In this way, the implicit features of the word to be recognized in the sentence can simultaneously reflect the contextual information and the contextual information of the word to be recognized in the sentence, so that based on the implicit features, the accuracy of identifying misspelled words can be further improved.

[0116] In some embodiments, see Figure 5 , Figure 5 This is an optional flow chart of the method for identifying misspelled words provided in the embodiment of the present application, based on Figure 3 , the context features include explicit features, Figure 3 The step S102 shown in the figure can be implemented by the following steps S501 to S502, which will be described below in combination with each step. The execution subject of the following steps can be the terminal or server mentioned above.

[0117] In step S501, the trained language model is used to predict the words to be recognized in the sentence to obtain a probability parameter indicating whether the words to be recognized are wrong.

[0118] Here, the language model can predict the probability distribution of the words to be recognized in the sentence to obtain a probability parameter indicating whether the word to be recognized is incorrect. The probability parameter may include, but is not limited to, one or more of the following: the probability that the word to be recognized is at the position where the word to be recognized is located in the sentence, the probability of the possible words and each possible word at that position, and a parameter indicating the fluency of the sentence when the word to be recognized is located at that position. Those skilled in the art can determine appropriate probability parameters based on actual circumstances, and these are not limited here.

[0119] In step S502, based on the probability parameter, the explicit features of the word to be recognized in the sentence are determined.

[0120] Here, the explicit features of the word to be recognized in the sentence may include any suitable features related to the predicted probability distribution of the word to be recognized in the sentence. For example, the explicit features of the word to be recognized in the sentence may include the probability that the position of the word to be recognized in the sentence is the word to be recognized, the difference or ratio between the probability of the word with the highest probability at the position and the probability of the word to be recognized at the position, the PPL value of the sentence when the word to be recognized is at the position, the PPL reduction ratio of the sentence when the position is replaced by other words, etc. It may also include a preset feature vector corresponding to the probability parameter that characterizes whether the word to be recognized is wrong, etc.

[0121] In some embodiments, the probability parameter can be directly determined as an explicit feature of the word to be recognized in the sentence. In some embodiments, the probability parameter can also be mapped to a preset feature vector, and the feature vector is determined as the explicit feature of the word to be recognized in the sentence.

[0122] In some embodiments, the probability parameters include the current word probability and the maximum candidate word probability. The above step S502 can be implemented by the following steps S511 to S513:

[0123] Step S511, discretizing the difference between the current word probability and the maximum candidate word probability to obtain a discretized difference;

[0124] Here, the current word probability is the probability that the current word is the current word at the position in the sentence predicted by the language model. The maximum candidate word probability is the probability corresponding to the word with the highest probability among the possible words at the position in the sentence predicted by the language model.

[0125] During implementation, any suitable discretization method can be used to discretize the difference between the current word probability and the maximum candidate word probability. The difference can be discretized evenly or unevenly, and there is no limitation here. The discretization interval can be determined based on actual conditions, for example, it can be discretized on the interval [0, 1] or the interval [0, 100].

[0126] Step S512, determining a first target value interval to which the discretized difference value belongs from a preset first value interval list;

[0127] Here, the first value interval list may include multiple pre-set value intervals, which may be determined by discretizing the difference between the current word probability and the maximum candidate word probability. The discretized difference obtained after discretizing the difference has and belongs to a unique value interval, and this value interval is the first target value interval.

[0128] Step S513: Determine the explicit features of the word to be recognized in the sentence based on the first target value interval.

[0129] Here, each value interval in the first value interval list may correspond to a preset feature. After determining the first target value interval, the feature corresponding to the first target value interval may be determined as the display feature of the word to be recognized in the sentence.

[0130] In some embodiments, the probability parameter includes at least one perplexity reduction ratio of the word to be recognized. The above step S501 may be implemented by the following steps S521 to S522:

[0131] Step S521, using the trained language model, determining the current perplexity of the sentence when the position of the word to be recognized is the current word, and the post-replacement perplexity of the sentence when the position is replaced by each candidate misspelling in a candidate set of misspellings of the word to be recognized;

[0132] Here, the candidate set of misspellings for the word to be recognized includes words that are easily confused with or incorrectly identified misspellings, such as similar characters or homophones of the word to be recognized. In practice, the candidate set of misspellings for the word to be recognized can be determined based on an open source dictionary library or can be pre-summarized based on manual experience, which is not limited here.

[0133] The set of misspelling candidates for the word to be recognized may include at least one candidate misspelling word. For each candidate misspelling word, the position of the word to be recognized in the sentence may be replaced with the candidate misspelling word, and the perplexity after the replacement is calculated to obtain a post-replacement perplexity.

[0134] Step S522: Determine at least one perplexity reduction ratio of the word to be recognized based on the current perplexity of the word to be recognized and at least one replaced perplexity.

[0135] Here, the perplexity reduction ratio can be the difference between the current perplexity and the perplexity after replacement, or the ratio of the current perplexity to the perplexity after replacement, and is not limited here. The perplexity reduction ratio can reflect the probability that the to-be-recognized word in the sentence is incorrect. The larger the perplexity reduction ratio, the greater the probability that the to-be-recognized word in the sentence is incorrect.

[0136] In some embodiments, the above-mentioned step S502 can be implemented through the following steps S531 to S533: Step S531, discretize at least one perplexity reduction ratio of the word to be recognized to obtain the discretized perplexity reduction ratio; Step S532, determine the second target value interval to which the discretized perplexity reduction ratio belongs from a preset second value interval list; Step S533, based on the second target value interval, determine the explicit features of the word to be recognized in the sentence. Here, the implementation of steps S531 to S533 is similar to that of the aforementioned steps S511 to S513. When implementing, the specific implementation of the aforementioned steps S511 to S513 can be referred to.

[0137] In some embodiments, the context features include explicit features and implicit features, and the above-mentioned step S102 can be implemented through the following steps S541 to S544: Step S541, using the trained bidirectional language model, determine the first vector representing the first N characters of the word to be recognized in the case of forward prediction, and the second vector representing the last M characters of the word to be recognized in the case of reverse prediction, wherein N and M are both integers greater than or equal to 0 and less than the length of the sentence; Step S542, based on the first vector and the second vector, obtain the implicit features of the word to be recognized in the sentence; Step S543, using the trained language model, predict the word to be recognized in the sentence to obtain a probability parameter representing whether the word to be recognized is wrong; Step S544, based on the probability parameter, determine the explicit features of the word to be recognized in the sentence.

[0138] In some embodiments, the probability parameters include the current word probability, the maximum candidate word probability and at least one perplexity reduction ratio of the word to be recognized, and the explicit features include a first explicit feature and a second explicit feature. The above-mentioned step S102 can be implemented by the following steps S551 to S556: Step S551, discretizing the difference between the current word probability and the maximum candidate word probability to obtain a discretized difference; Step S552, determining the first target value interval to which the discretized difference belongs from a preset first value interval list; Step S553, determining the explicit features of the word to be recognized in the sentence based on the first target value interval; Step S554, discretizing at least one perplexity reduction ratio of the word to be recognized to obtain a discretized perplexity reduction ratio; Step S555, determining the second target value interval to which the discretized perplexity reduction ratio belongs from a preset second value interval list; Step S556, determining the explicit features of the word to be recognized in the sentence based on the second target value interval.

[0139] In an embodiment of the present application, a trained language model is used to predict the words to be recognized in a sentence, and a probability parameter representing whether the words to be recognized are wrong is obtained, and based on the probability parameter, the explicit features of the words to be recognized in the sentence are determined. In this way, the display features of the words to be recognized in the sentence can be determined based on the prediction results of the language model for the words to be recognized in the sentence. The display features thus obtained can better reflect the context of the words to be recognized in the sentence, and thus can further improve the accuracy of the recognition of wrong words, and can effectively improve the generalization of the recognition of wrong words. Furthermore, the context features can include both explicit features and implicit features, so as to further improve the expression ability of the context features on the context of the words to be recognized in the sentence.

[0140] In some embodiments, see Figure 6 , Figure 6 This is an optional flow chart of the method for identifying misspelled words provided in the embodiment of the present application, based on Figure 3 The above step S103 can be implemented through the following steps S601 to S602. The following will be explained in combination with each step. The execution subject of the following steps can be the terminal or server mentioned above.

[0141] In step S601, the encoding features and context features of the word to be recognized are merged to obtain a fusion vector of the word to be recognized;

[0142] Here, any suitable method can be used to merge the encoding features and context features of the word to be recognized, and this is not limited here. For example, the encoding features and context features can be added to obtain a fusion vector of the word to be recognized; or the encoding features and context features can be concatenated to obtain a fusion vector of the word to be recognized.

[0143] In step S602, the trained deep neural network model is used to identify the fusion vector of the word to be recognized in the sentence to obtain a recognition result indicating whether the word to be recognized is incorrect.

[0144] Here, the deep neural network model can identify whether the words to be recognized are wrong by classifying the fused features of the words to be recognized in the sentence.

[0145] In an embodiment of the present application, the encoding features and context features of the word to be recognized are combined to obtain a fusion vector of the word to be recognized. The fusion vector of the word to be recognized in the sentence is then recognized using a trained deep neural network model to obtain a recognition result indicating whether the word to be recognized is incorrect. In this way, the encoding features and context features of the word to be recognized can be combined to identify incorrect words, which can further improve the recognition accuracy of incorrect words.

[0146] In some embodiments, see Figure 7 , Figure 7 This is an optional flow chart of the method for identifying misspelled words provided in the embodiment of the present application, based on Figure 3 Before the above step S103, the following steps S701 to S704 may be performed. The following will be described in combination with each step. The execution subject of the following steps may be the terminal or server mentioned above.

[0147] In step S701, each sample sentence in the annotated sample set and the encoding features of the words to be recognized in each sample sentence are obtained; each sample sentence in the annotated sample set has a label sequence for marking whether the words to be recognized in the sample sentence are typos.

[0148] Here, the annotated sample set may be pre-annotated manually or automatically, and may include at least one sample sentence. The sample sentence may include at least one word to be recognized, and the label sequence of the sample sentence may include a label corresponding to each word to be recognized, for marking whether the word to be recognized is a typo.

[0149] In step S702, for each of the sample sentences, the trained language model is used to extract features of the words to be recognized in the sample sentences to obtain context features of the words to be recognized in the sample sentences, and the deep neural network model is used to identify the encoding features and context features of the words to be recognized in the sample sentences to obtain a recognition result representing whether the words to be recognized in the sample sentences are incorrect.

[0150] Here, step S702 corresponds to the aforementioned step S102 and step S103. When implementing, reference may be made to the specific implementations of the aforementioned step S102 and step S103.

[0151] Step S703 : Based on the recognition result corresponding to each sample sentence and the label sequence of each sample sentence, a loss value is determined using a loss function.

[0152] Here, any suitable loss function can be used to determine the loss value, such as absolute value loss function, square loss function, cross entropy loss function, exponential loss function, etc.

[0153] Step S704: When it is determined according to the loss value that the loss function has not converged, the parameters of the deep neural network model are updated based on a parameter optimization algorithm.

[0154] Here, the parameter optimization algorithm may be any suitable algorithm, such as gradient descent method, conjugate gradient method, Newton algorithm, etc.

[0155] It should be noted that the above steps S701 to S704 are not limited to the following when being implemented. Figure 7 The execution order shown, for example, the above steps S701 to S704 may also be executed before step S101.

[0156] In some embodiments, after step S704, the method may further include: step S705, for each of the sample sentences, using the trained language model to extract features of the words to be recognized in the sample sentences, and obtain context features of the words to be recognized in the sample sentences, and using the deep neural network model after parameter update to identify the encoding features and context features of the words to be recognized in the sample sentences, and obtain an updated recognition result representing whether the words to be recognized in the sample sentences are wrong; step S706, based on the updated recognition results corresponding to each sample sentence and the label sequence of each sample sentence, using the loss function to determine the updated loss value; step S707, when it is determined that the loss function has not converged according to the updated loss value, updating the parameters of the deep neural network model based on the parameter optimization algorithm.

[0157] In some embodiments, after step S704, the method may further include: step S708, when it is determined that the loss function converges according to the loss value or when the number of parameter updates reaches a preset threshold, determining the current deep neural network model as a trained deep neural network model.

[0158] In the embodiment of the present application, a preset annotated sample set is used, and the features of the trained language model are integrated to train the deep neural network model to achieve end-to-end learning. In this way, the trained deep neural network model can not only take advantage of the strong generalization characteristics of the language model, but also take advantage of the high recognition accuracy of the deep neural network model, so that it can better identify typos in sentences. And because the features of the language model are deeply integrated on the basis of the deep neural network model, end-to-end learning is performed, there is no need to manually set and maintain a large number of model fusion rules, which can greatly reduce labor costs.

[0159] The model training method provided in the embodiment of the present application will be explained below in combination with the exemplary application and implementation of the terminal or server provided in the embodiment of the present application.

[0160] See also Figure 8 , Figure 8 This is an optional flow chart of the model training method provided in the embodiment of the present application. Figure 8 The following steps are explained with reference to the steps shown in the figure. The execution subject of the following steps can be the terminal or server mentioned above.

[0161] In step S801, encoding features of each sample sentence in the annotated sample set and the words to be recognized in each sample sentence are obtained; each sample sentence in the annotated sample set has a label sequence for marking whether the words to be recognized in the sample sentence are typos;

[0162] In step S802, for each of the sample sentences, the trained language model is used to extract features of the words to be recognized in the sample sentence to obtain context features of the words to be recognized in the sample sentence, and the deep neural network model is used to identify the encoding features and context features of the words to be recognized in the sample sentence to obtain a recognition result indicating whether the words to be recognized in the sample sentence are incorrect.

[0163] In step S803, based on the recognition result corresponding to each sample sentence and the label sequence of each sample sentence, a loss value is determined using a loss function;

[0164] In step S804, when it is determined according to the loss value that the loss function has not converged, the parameters of the deep neural network model are updated based on a parameter optimization algorithm.

[0165] In some embodiments, after step S804, the method may further include: step S805, for each of the sample sentences, using the trained language model to extract features of the words to be recognized in the sample sentences, and obtain context features of the words to be recognized in the sample sentences, and using the deep neural network model after parameter update to identify the encoding features and context features of the words to be recognized in the sample sentences, and obtain an updated recognition result representing whether the words to be recognized in the sample sentences are wrong; step S806, based on the updated recognition results corresponding to each sample sentence and the label sequence of each sample sentence, using the loss function to determine the updated loss value; step S807, when it is determined that the loss function has not converged according to the updated loss value, updating the parameters of the deep neural network model based on the parameter optimization algorithm.

[0166] In some embodiments, after step S804, the method may further include: step S808, when it is determined that the loss function converges according to the loss value or when the number of updates of the parameter reaches a preset threshold number, determining the current deep neural network model as a trained deep neural network model.

[0167] It should be noted that steps S801 to S808 correspond to the aforementioned steps S701 to S708 respectively. When implementing, reference may be made to the specific implementation of the aforementioned steps S701 to S708.

[0168] In the embodiment of the present application, a preset annotated sample set is used, and the features of the trained language model are integrated to train the deep neural network model to achieve end-to-end learning. In this way, the trained deep neural network model can not only take advantage of the strong generalization characteristics of the language model, but also take advantage of the high recognition accuracy of the deep neural network model, so that it can better identify typos in sentences. And because the features of the language model are deeply integrated on the basis of the deep neural network model, end-to-end learning is performed, there is no need to manually set and maintain a large number of model fusion rules, which can greatly reduce labor costs.

[0169] Below, an exemplary application of the embodiment of the present application in an actual application scenario will be described. The method for identifying misspelled words provided by the embodiment of the present application can be applied to the scenario of identifying misspelled words in a single sentence or article. For example, it can automatically identify and prompt misspelled information in articles on information flow products such as Tiantian Kuaibao and QQ Kandian, assist in manual review and identification of misspelled words, and intercept low-quality articles with too many misspelled words or containing bad misspelled words from entering the recommendation pool; it can also assist self-media authors in posting articles in the Penguin posting assistant, help users check suspected misspelled words in the article before posting, and prompt correction information, thereby helping authors reduce or avoid misspelled words in the article.

[0170] In actual application scenarios, there are many types of typos. Taking Chinese typo recognition as an example, see the following Table 1. Table 1 lists common Chinese typo error types and examples, which may include homophone errors, near-phonetic errors, similar-looking errors, and other errors. It also shows examples of typos of each error type and gives the correct word in brackets after the corresponding typo position:

[0171] Table 1 Common types of Chinese typos

[0172]

[0173] In view of this, the embodiment of the present application proposes a typo recognition model based on a deep neural network that integrates a bidirectional neural language model. On the basis of the traditional typo recognition model based on a deep neural network, the characteristics of the neural language model are deeply integrated to perform end-to-end learning. Due to the deep utilization of the neural language model trained on a large amount of text, the typo recognition model based on the deep neural network can better identify typos in articles. Through this deep fusion method, the two models are merged into one model, which not only utilizes the strong generalization characteristics of the neural language model trained on a large amount of text, but also maintains the high accuracy of the deep neural network, so that the model can better identify typos in articles.

[0174] See also Figure 9A , Figure 9AThis is a schematic diagram of the overall architecture of a deep neural network-based typo recognition model that integrates a bidirectional neural language model provided in an embodiment of the present application. Figure 9A As shown, the model includes a converter-based bidirectional language model 910 and a BERT sequence labeling model 920. The input of the model is a sentence, and the output is the sequence labeling result output by the BERT sequence labeling model 920. The output sequence labeling result includes a labeling of whether each word in the input sentence is a typo.

[0175] The process of identifying typos in an input sentence using the transformer-based bidirectional language model 910 and the BERT sequence tagging model 920 includes the following steps S901 to S903:

[0176] Step S901: Input a sentence into a bidirectional language model 910 based on a transformer. For each word in the sentence, two types of feature vectors are obtained: one is an intermediate representation vector obtained during the prediction process of the language model (LM), called an implicit feature vector; the other is a feature vector determined based on the LM prediction result, called an explicit feature vector.

[0177] Here, the input sentence can be spliced ​​into a character sequence according to the following rules and then input into the bidirectional language model:

[0178] [CLS]Char1 Char2...CharL[SEP];

[0179] Among them, [CLS] and [SEP] are predefined marking symbols, Char1, Char2 and CharL are each word that appears in sequence in the input sentence, and L is the total number of words in the sentence.

[0180] Since a unidirectional neural language model can only utilize a unilateral context, a bidirectional neural language model is proposed in the embodiment of the present application. Figure 9B , Figure 9BSchematic diagram of the composition architecture of a bidirectional neural language model provided by an embodiment of the present application. This model consists of two unidirectional LM models, including an LM model 911 that makes predictions from front to back and an LM model 912 that makes predictions from back to front. When predicting the word at the current position in the input sentence, the LM model 911 can determine the representation vector of the previous word of the current word in the process of making predictions from front to back, and the LM model 912 can determine the representation vector of the next word of the current word in the process of making predictions from back to front. By concatenating the representation vector of the previous word of the current word determined by the LM model 911 and the representation vector of the next word of the current word determined by the LM model 912, an implicit feature vector of the current word can be obtained. Based on this implicit feature vector, the word at the current position can be predicted. For example, assuming the input sentence is "Let citizens obtain a sense of happiness", when predicting the word at the position of "辛", it is obtained by concatenating the representation vectors of the previous word "感" and the next word "福".

[0181] The bidirectional neural language model provided by an embodiment of the present application can be trained based on a preset unlabeled corpus. When training the bidirectional neural language model, instead of training the unidirectional model from front to back and the unidirectional model from back to front separately, the two models are trained together. In this way, the bidirectional neural language model can utilize the complete context of both directions simultaneously during the training process, and only need to perform one model prediction on the input sentence to obtain the implicit feature vectors of each word in the sentence.

[0182] The explicit feature vector of each word in the input sentence can include the probability of the top1 word predicted by the bidirectional language model at the current position, the probability of the current word, and the feature vector corresponding to the discretized difference between the probability of the top1 word and the probability of the current word. Here, the probability of the top1 word is the probability of the word with the highest probability predicted by the bidirectional language model at the current position, and the probability of the current word is the probability that the current position is the word predicted by the bidirectional language model. The feature vector corresponding to the discretized difference between the probability of the top1 word and the probability of the current word can be obtained as follows: determine the difference between the probability of the top1 word and the probability of the current word, and then evenly discretize this difference into a preset number of buckets (for example, 20 buckets). Each bucket corresponds to a feature vector, and the feature vector corresponding to the bucket into which the discretized difference falls can be used as the feature vector corresponding to the discretized difference between the probability of the top1 word and the probability of the current word at the current position.

[0183] The explicit feature vector of each word in the input sentence may also include a feature vector corresponding to the PPL reduction ratio predicted by the bidirectional language model at the current position. Here, for each word in the input sentence, each candidate typo in the typo candidate set corresponding to the word can be replaced, and the PPL reduction ratio of the sentence before and after each replacement can be calculated. For each PPL reduction ratio, the PPL reduction ratio can be discretized into a plurality of preset buckets (for example, 20 buckets) on average, each bucket corresponding to a feature vector, and the feature vector corresponding to the bucket into which the PPL reduction ratio falls after discretization can be used as the feature vector corresponding to the PPL reduction ratio predicted at the current position. During implementation, the PPL of a sentence can be calculated using the following formula 1-1:

[0184]

[0185] Among them, s represents the sentence to be calculated, n is the length of the sentence, w1w2…w n Represents the word at each position in the sentence s, P(w1w2…w n ) means the words in sentence s are w1w2…w n The probability when p(w i |w1w2…w i-1 ) indicates that the words at positions 1 to i-1 are w1w2…w i-1 When position i is w i The smaller PPL(s) is, the greater the probability that there is a typo in sentence s.

[0186] Step S902: For each word in the sentence, add the two types of vectors obtained by LM and the original BERT word vector corresponding to the word to obtain a summed vector;

[0187] Here, the original BERT word vector corresponding to each word can be obtained by encoding each word using any appropriate vectorized encoding method, or by querying a specific word vector table.

[0188] In step S903 , the summed vector corresponding to each character is input into the BERT sequence tagging model 920 to obtain a sequence tagging result including a tag indicating whether each character in the input sentence is a typo.

[0189] The above-mentioned converter-based bidirectional language model can be pre-trained using a large amount of unlabeled corpus. The BERT sequence labeling model 920 can be trained based on a pre-labeled training sample set and using the features of the already trained converter-based bidirectional language model. When training the BERT sequence labeling model, the sum of the cross entropy at each position in the input sentence can be used as the loss value. For example, the loss value Loss can be calculated using the following formula 1-2:

[0190]

[0191] Where n is the length of the input sentence, P k is the probability distribution of whether there is a typo at position k in the input sentence.

[0192] In some embodiments, the bidirectional language model can be replaced with a BERT model, and the BERT sequence labeling model can be replaced with other neural network models, such as convolutional neural networks (CNN), recurrent neural networks (RNN), etc.

[0193] The following continues to describe the exemplary structure of the device 255 for identifying misspelled words provided in the embodiment of the present application as a software module. In some embodiments, such as Figure 2A As shown, the software modules stored in the misspelled word recognition device 255 of the memory 250 may include:

[0194] A first acquisition module 2551 is used to acquire a sentence to be recognized and encoding features of words to be recognized in the sentence;

[0195] A first extraction module 2552 is configured to extract features of the words to be recognized in the sentence using the trained language model to obtain context features of the words to be recognized in the sentence;

[0196] The first recognition module 2553 is used to use the trained deep neural network model to identify the encoding features and context features of the words to be recognized in the sentence, and obtain a recognition result indicating whether the words to be recognized are incorrect.

[0197] In some embodiments, the language model includes a bidirectional language model, the contextual features include implicit features, and the first extraction module is further used to: use the trained bidirectional language model to determine a first vector representing the first N characters of the word to be recognized in the case of forward prediction, and a second vector representing the last M characters of the word to be recognized in the case of reverse prediction, wherein N and M are both integers greater than or equal to 0 and less than the length of the sentence; based on the first vector and the second vector, obtain the implicit features of the word to be recognized in the sentence.

[0198] In some embodiments, the contextual features include explicit features, and the first extraction module is further used to: use a trained language model to predict the words to be recognized in the sentence to obtain a probability parameter representing whether the words to be recognized are wrong; based on the probability parameter, determine the explicit features of the words to be recognized in the sentence.

[0199] In some embodiments, the probability parameters include the current word probability and the maximum candidate word probability, and the first extraction module is further used to: discretize the difference between the current word probability and the maximum candidate word probability to obtain a discretized difference; determine the first target value interval to which the discretized difference belongs from a preset first value interval list; and determine the explicit features of the word to be recognized in the sentence based on the first target value interval.

[0200] In some embodiments, the probability parameter includes at least one perplexity reduction ratio of the word to be recognized, and the first extraction module is further used to: use a trained language model to determine the current perplexity of the sentence when the position of the word to be recognized is the current word, and the post-replacement perplexity of the sentence when the position is replaced by each candidate typo in the typo candidate set of the word to be recognized; based on the current perplexity of the word to be recognized and at least one post-replacement perplexity, determine at least one perplexity reduction ratio of the word to be recognized.

[0201] In some embodiments, the first extraction module is further used to: discretize at least one perplexity reduction ratio of the word to be recognized to obtain a discretized perplexity reduction ratio; determine a second target value interval to which the discretized perplexity reduction ratio belongs from a preset second value interval list; and determine the explicit features of the word to be recognized in the sentence based on the second target value interval.

[0202] In some embodiments, the first recognition module is further used to: merge the encoding features and context features of the word to be recognized to obtain a fusion vector of the word to be recognized; use a trained deep neural network model to recognize the fusion vector of the word to be recognized in the sentence to obtain a recognition result representing whether the word to be recognized is wrong.

[0203] In some embodiments, the device further includes: a second acquisition module for acquiring each sample sentence in the annotated sample set and the encoding features of the words to be recognized in each of the sample sentences; each sample sentence in the annotated sample set has a label sequence for marking whether the words to be recognized in the sample sentence are typos; a second recognition module for extracting features of the words to be recognized in the sample sentence using the trained language model for each of the sample sentences, obtaining the context features of the words to be recognized in the sample sentence, and using the deep neural network model to identify the encoding features and context features of the words to be recognized in the sample sentence, obtaining a recognition result representing whether the words to be recognized in the sample sentence are wrong; a first determination module for determining a loss value using a loss function based on the recognition result corresponding to each sample sentence and the label sequence of each sample sentence; a first update module for updating the parameters of the deep neural network model based on a parameter optimization algorithm when it is determined that the loss function has not converged according to the loss value.

[0204] The following continues to describe the exemplary structure of the model training device 555 provided in the embodiment of the present application as a software module. In some embodiments, such as Figure 2B As shown, the software modules stored in the misspelling recognition device 355 of the memory 350 may include:

[0205] The third acquisition module 3551 is used to obtain the encoding features of each sample sentence in the annotated sample set and the words to be recognized in each sample sentence; each sample sentence in the annotated sample set has a label sequence for marking whether the words to be recognized in the sample sentence are typos;

[0206] a third recognition module 3552 configured to, for each sample sentence, extract features of the words to be recognized in the sample sentence using the trained language model, obtain context features of the words to be recognized in the sample sentence, and recognize the encoding features and context features of the words to be recognized in the sample sentence using the deep neural network model to obtain a recognition result indicating whether the words to be recognized in the sample sentence are incorrect;

[0207] A second determination module 3553 is configured to determine a loss value using a loss function based on the recognition result corresponding to each sample sentence and the label sequence of each sample sentence;

[0208] The second updating module 3554 is used to update the parameters of the deep neural network model based on a parameter optimization algorithm when it is determined that the loss function has not converged according to the loss value.

[0209] The present application provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method for identifying misspelled characters or the model training method described in the present application.

[0210] The embodiment of the present application provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the method for identifying misspelled words or the model training method provided by the embodiment of the present application, for example, Figure 3 The method shown.

[0211] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or various devices including one or any combination of the above memories.

[0212] In some embodiments, executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0213] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0214] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0215] In summary, the embodiments of the present application can utilize both the strong generalization of the language model and the high recognition accuracy of the deep neural network model when identifying misspelled words, thereby better identifying misspelled words in sentences. Furthermore, because the features of the language model are deeply integrated on the basis of the deep neural network model, end-to-end misspelled word recognition is performed without the need to manually set and maintain a large number of model fusion rules, which can greatly reduce labor costs.

[0216] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A method for identifying misspelled characters, characterized in that: include: Obtaining a sentence to be recognized and encoding features of words to be recognized in the sentence; Using the trained language model, predicting the words to be recognized in the sentence to obtain a probability parameter representing whether the words to be recognized are incorrect; When the probability parameter includes a current word probability and a maximum candidate word probability, discretizing the difference between the current word probability and the maximum candidate word probability to obtain a discretized difference; and determining a first target value interval to which the discretized difference belongs from a preset first value interval list; Determining, based on the first target value interval, an explicit feature of the word to be recognized in the sentence; When the probability parameter includes at least one perplexity reduction ratio of the word to be recognized, discretizing the at least one perplexity reduction ratio of the word to be recognized to obtain a discretized perplexity reduction ratio; Determine a second target value interval to which the discretized perplexity reduction ratio belongs from a preset second value interval list; Determining, based on the second target value interval, an explicit feature of the word to be recognized in the sentence; Adding or concatenating the encoding features and context features of the word to be recognized in the sentence to obtain a fusion vector of the word to be recognized, wherein the context features include the explicit features; The trained deep neural network model is used to identify the fusion vector of the word to be recognized in the sentence, and a recognition result is obtained to indicate whether the word to be recognized is incorrect.

2. The method according to claim 1, characterized in that The language model includes a bidirectional language model, the context features further include implicit features, and the method further includes: Determining, using the trained bidirectional language model, a first vector representing the first N characters of the word to be recognized in a forward prediction scenario, and a second vector representing the last M characters of the word to be recognized in a backward prediction scenario, where N and M are both integers greater than or equal to 0 and less than the length of the sentence; Based on the first vector and the second vector, implicit features of the word to be recognized in the sentence are obtained.

3. The method according to claim 1, characterized in that When the probability parameter includes at least one perplexity reduction ratio of the word to be recognized, the method further includes: Using the trained language model, determine the current perplexity of the sentence when the position of the word to be recognized is the current word, and the post-replacement perplexity of the sentence when the position is replaced by each candidate misspelling word in a candidate set of misspelling words of the word to be recognized; At least one perplexity reduction ratio of the word to be recognized is determined based on the current perplexity of the word to be recognized and at least one replaced perplexity.

4. The method according to claim 1, wherein The words to be recognized in the sentence are determined by any of the following methods: Filtering the words to be recognized according to the parts of speech of the words in the sentence to be recognized; The words to be recognized are filtered out according to the positions of the words in the sentence to be recognized.

5. A model training method, characterized in that: The method comprises: Obtaining encoding features of each sample sentence in the labeled sample set and the words to be recognized in each of the sample sentences; each sample sentence in the labeled sample set has a label sequence for marking whether the words to be recognized in the sample sentence are typos; For each of the sample sentences, the trained language model is used to predict the words to be recognized in the sample sentence to obtain a probability parameter that characterizes whether the words to be recognized are wrong; wherein, when the probability parameter includes the current word probability and the maximum candidate word probability, the difference between the current word probability and the maximum candidate word probability is discretized to obtain a discretized difference; the first target value interval to which the discretized difference belongs is determined from a preset first value interval list; based on the first target value interval, the explicit features of the words to be recognized in the sentence are determined; when the probability parameter includes at least one perplexity reduction ratio of the words to be recognized, the discretized difference is obtained. Discretizing at least one perplexity reduction ratio of the recognized word to obtain a discretized perplexity reduction ratio; determining a second target value interval to which the discretized perplexity reduction ratio belongs from a preset second value interval list; determining the explicit features of the word to be recognized in the sentence based on the second target value interval; and adding or concatenating the encoding features and context features of the word to be recognized in the sample sentence to obtain a fusion vector of the word to be recognized, wherein the context features include the explicit features; using a deep neural network model, recognizing the fusion vector of the word to be recognized in the sample sentence to obtain a recognition result indicating whether the word to be recognized in the sample sentence is incorrect; Based on the recognition result corresponding to each sample sentence and the label sequence of each sample sentence, a loss function is used to determine the loss value; When it is determined according to the loss value that the loss function has not converged, the parameters of the deep neural network model are updated based on a parameter optimization algorithm.

6. A device for identifying misspelled characters, characterized in that: include: A first acquisition module is used to acquire a sentence to be recognized and encoding features of words to be recognized in the sentence; A first extraction module is configured to predict the words to be recognized in the sentence using the trained language model to obtain a probability parameter representing whether the words to be recognized are incorrect; The first extraction module is further configured to, when the probability parameter includes a current word probability and a maximum candidate word probability, discretize the difference between the current word probability and the maximum candidate word probability to obtain a discretized difference; and determine a first target value interval to which the discretized difference belongs from a preset first value interval list; Determining, based on the first target value interval, an explicit feature of the word to be recognized in the sentence; When the probability parameter includes at least one perplexity reduction ratio of the word to be recognized, discretizing the at least one perplexity reduction ratio of the word to be recognized to obtain a discretized perplexity reduction ratio; Determine a second target value interval to which the discretized perplexity reduction ratio belongs from a preset second value interval list; Determining, based on the second target value interval, an explicit feature of the word to be recognized in the sentence; a first recognition module, configured to add or concatenate encoding features and context features of a word to be recognized in the sentence to obtain a fusion vector of the word to be recognized, wherein the context features include the explicit features; The trained deep neural network model is used to identify the fusion vector of the word to be recognized in the sentence, and a recognition result is obtained to indicate whether the word to be recognized is incorrect.

7. The device according to claim 6, characterized in that The language model includes a bidirectional language model, the context features also include implicit features, and the first extraction module is further configured to: Determining, using the trained bidirectional language model, a first vector representing the first N characters of the word to be recognized in a forward prediction scenario, and a second vector representing the last M characters of the word to be recognized in a backward prediction scenario, where N and M are both integers greater than or equal to 0 and less than the length of the sentence; Based on the first vector and the second vector, implicit features of the word to be recognized in the sentence are obtained.

8. The device according to claim 6, characterized in that When the probability parameter includes at least one perplexity reduction ratio of the word to be recognized, the first extraction module is further configured to: Using the trained language model, determine the current perplexity of the sentence when the position of the word to be recognized is the current word, and the post-replacement perplexity of the sentence when the position is replaced by each candidate misspelling word in the misspelling candidate set of the word to be recognized; At least one perplexity reduction ratio of the word to be recognized is determined based on the current perplexity of the word to be recognized and at least one replaced perplexity.

9. A device for identifying misspelled characters, characterized in that: include: a memory for storing executable instructions; A processor, configured to implement the method according to any one of claims 1 to 5 when executing the executable instructions stored in the memory.

10. A computer-readable storage medium, characterized in that Executable instructions are stored, and when executed by a processor, they are used to implement the method described in any one of claims 1 to 5.

11. A computer program product, characterized in that The computer program product comprises computer instructions for implementing the method according to any one of claims 1 to 5 when executed by a processor.

Images