Proprietary name correction method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a computer-implemented proper noun correction method, which relates to natural language processing, and particularly relates to the identification of proper nouns. The implementation is as follows: obtaining a plurality of candidate words based on first text content; in response to determining that a first candidate word in the plurality of candidate words matches a first proper noun in a preset proper noun library, generating second text content by replacing the first candidate word in the first text content with a specific identifier; determining whether to replace the first candidate word based on the first text content and the second text content; and in response to determining to replace the first candidate word, replacing the first candidate word in the first text content with the first proper noun.

Description

Technical Field

[0001] This disclosure relates to natural language processing, and more particularly to the recognition of proper nouns, specifically to a computer-implemented method, apparatus, electronic device, computer-readable storage medium, and computer program product for correcting proper nouns. Background Technology

[0002] Named Entity (MORE) correction refers to the process of correcting erroneous proper nouns in a text, such as names of people, organizations, and places.

[0003] The methods described in this section are not necessarily methods that had been previously conceived or adopted. Unless otherwise specified, no method described in this section should be assumed to be prior art simply because it is included in this section. Similarly, unless otherwise specified, the issues mentioned in this section should not be considered to be accepted in any prior art. Summary of the Invention

[0004] This disclosure provides a method, apparatus, electronic device, computer-readable storage medium, and computer program product for correcting proper nouns implemented by a computer.

[0005] According to one aspect of this disclosure, a computer-implemented method for correcting proper nouns is provided. The method includes: acquiring a plurality of candidate word segments based on first text content; in response to determining that a first candidate word among the plurality of candidate word segments matches a first proper noun in a preset proper noun library, generating second text content by replacing the first candidate word in the first text content with a specific identifier; determining whether to replace the first candidate word based on the first text content and the second text content; and in response to determining to replace the first candidate word, replacing the first candidate word in the first text content with the first proper noun.

[0006] According to another aspect of this disclosure, a training method for a proper noun correction model is provided. The method includes: acquiring a sample dataset, wherein each sample data in the sample dataset includes first text content, second text content, and a label, and wherein the second text content is generated by replacing a first word segment in the first text content with a specific identifier; initializing multiple parameters of the correction model; and for each sample data, performing the following operations: inputting the first text content and the second text content into the correction model to obtain an output value, the output value representing the probability that the first word segment is a preset proper noun; calculating a loss function based on the output value and the label corresponding to the sample data; and adjusting multiple parameters in the correction model based on the loss function.

[0007] According to another aspect of this disclosure, a proper noun correction device is provided, the device comprising: a candidate word segmentation acquisition unit configured to acquire a plurality of candidate words based on a first text content; a text generation unit configured to generate second text content by replacing the first candidate word in the first text content with a specific identifier in response to determining that a first candidate word among the plurality of candidate words matches a first proper noun in a preset proper noun library; a judgment unit configured to determine whether to replace the first candidate word based on the first text content and the second text content; and a replacement unit configured to replace the first candidate word in the first text content with the first proper noun in response to determining that the first candidate word should be replaced.

[0008] According to another aspect of this disclosure, a training apparatus for a proper noun correction model is provided. The apparatus includes: a sample acquisition unit configured to acquire a sample dataset, wherein each sample data in the sample dataset includes first text content, second text content, and a label, and wherein the second text content is generated by replacing a first word segment in the first text content with a specific identifier; an initialization unit configured to initialize multiple parameters of the correction model; an input unit configured to input the first text content and the second text content into the correction model to obtain an output value, the output value representing the probability that the first word segment is a preset proper noun; a loss function calculation unit configured to calculate a loss function based on the output value and the label corresponding to the sample data; and a parameter update unit configured to adjust multiple parameters in the correction model based on the loss function.

[0009] According to one or more embodiments of this disclosure, the efficiency and accuracy of proper name correction can be improved.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0012] Figure 1 A schematic diagram of an exemplary system in which the various methods described herein may be implemented according to embodiments of the present disclosure is shown;

[0013] Figure 2 A flowchart of a computer-implemented method for correcting proper nouns according to an embodiment of the present disclosure is shown;

[0014] Figure 3 An example of proper name correction of text according to an embodiment of the present disclosure is shown;

[0015] Figure 4 A flowchart illustrating a training method for a proper noun correction model according to an embodiment of the present disclosure is shown;

[0016] Figure 5 A block diagram of a proper noun correction apparatus according to an embodiment of the present disclosure is shown;

[0017] Figure 6 A block diagram of a training apparatus for a proper noun correction model according to an embodiment of the present disclosure is shown;

[0018] Figure 7 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0019] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0020] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0021] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. Furthermore, the term "and / or" as used in this disclosure covers any one of the listed items and all possible combinations thereof.

[0022] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0023] Figure 1A schematic diagram of an exemplary system 100 in which the various methods and apparatus described herein can be implemented according to embodiments of this disclosure is shown. Reference Figure 1 The system 100 includes one or more client devices 101, 102, 103, 104, 105 and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105 and 106 can be configured to execute one or more applications.

[0024] In embodiments of this disclosure, server 120 may run one or more services or software applications that enable the execution of proper noun correction methods.

[0025] In some embodiments, server 120 may also provide other services or software applications that may include non-virtual and virtual environments. In some embodiments, these services may be provided as web-based services or cloud services, such as to users of client devices 101, 102, 103, 104, 105 and / or 106 under a Software as a Service (SaaS) model.

[0026] exist Figure 1 In the configuration shown, server 120 may include one or more components that implement the functions performed by server 120. These components may include software components, hardware components, or combinations thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 can sequentially interact with server 120 using one or more client applications to utilize the services provided by these components. It should be understood that various different system configurations are possible and may differ from system 100. Therefore, Figure 1 This is an example of a system used to implement the various methods described herein, and is not intended to be limiting.

[0027] Users can use client devices 101, 102, 103, 104, 105, and / or 106 to implement proper noun correction functions, such as receiving and / or sending input data that may require proper noun correction processing. The client devices can provide an interface that allows users to interact with the client devices. The client devices can also output information to the user through this interface. Although... Figure 1 Only six client devices are described, but those skilled in the art will understand that this disclosure can support any number of client devices.

[0028] Client devices 101, 102, 103, 104, 105, and / or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptops), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices. These computer devices can run various types and versions of software applications and operating systems, such as Microsoft Windows, Apple iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as Google Chrome OS); or include various mobile operating systems, such as Microsoft Windows Mobile OS, iOS, Windows Phone, and Android. Portable handheld devices may include cellular phones, smartphones, tablets, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Gaming systems may include various handheld gaming devices, internet-enabled gaming devices, etc. Client devices are capable of executing various applications, such as various internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and can use various communication protocols.

[0029] Network 110 can be any type of network well known to those skilled in the art, and can use any of a variety of available protocols (including but not limited to TCP / IP, SNA, IPX, etc.) to support data communication. By way of example only, one or more networks 110 can be a local area network (LAN), an Ethernet-based network, a token ring network, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0030] Server 120 may include one or more general-purpose computers, special-purpose server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for servers). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0031] The computing unit in server 120 can run one or more operating systems, including any of the aforementioned operating systems and any commercially available server operating system. Server 120 can also run any of a variety of additional server applications and / or middleware applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0032] In some implementations, server 120 may include one or more applications to analyze and merge data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105 and / or 106. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105 and / or 106.

[0033] In some implementations, server 120 can be a server for a distributed system or a server integrated with blockchain. Server 120 can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in the cloud computing service system, designed to address the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0034] System 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and video files. Databases 130 may reside in various locations. For example, a database used by server 120 may be local to server 120, or it may be located away from server 120 and may communicate with server 120 via a network-based or dedicated connection. Databases 130 may be of different types. In some embodiments, the database used by server 120 may be, for example, a relational database. One or more of these databases may store, update, and retrieve data from and from the databases in response to commands.

[0035] In some embodiments, one or more of the databases 130 may also be used by an application to store application data. The databases used by the application may be of different types, such as key-value stores, object stores, or regular stores supported by a file system.

[0036] Figure 1 The system 100 can be configured and operated in various ways to enable the application of the various methods and apparatus described in this disclosure.

[0037] Figure 2 This is a flowchart illustrating a proper noun correction method according to an exemplary embodiment of the present disclosure.

[0038] like Figure 2 As shown, the method includes: step S201, obtaining multiple candidate word segments based on first text content; step S202, in response to determining that a first candidate word among the multiple candidate word segments matches a first proper noun in a preset proper noun library, generating second text content by replacing the first candidate word in the first text content with a specific identifier; step S203, determining whether to replace the first candidate word based on the first text content and the second text content; and step S204, in response to determining to replace the first candidate word, replacing the first candidate word in the first text content with the first proper noun.

[0039] The proper noun correction method disclosed herein improves the recognition probability of proper nouns in the proper noun database during the correction process by using a pre-set proper noun database and then correcting errors, thereby improving the efficiency and accuracy of proper noun correction.

[0040] For specific use cases, users can pre-store corresponding specialized terminology databases. For example, in conversational scenarios within the field of machine learning, a specialized terminology database covering various neural networks, deep learning, and other specialized terms can be pre-generated.

[0041] In step S202, the specific identifier is used as an identifier. Furthermore, during each execution of the above method, only one candidate segmentation word among multiple candidate segmentation words is replaced with the specific identifier.

[0042] According to some embodiments, obtaining multiple candidate word segments based on the first text content includes: segmenting the first text content using the N-Gram algorithm based on the length of the first proper noun. N-Gram allows for more efficient and targeted acquisition of candidate word segments.

[0043] In this context, N-Gram refers to processing the text content byte-by-byte using a sliding window of size N, forming a sequence of byte segments of length N. Each byte segment is called a Gram. The frequency of occurrence of all byte segments is statistically analyzed and filtered according to a pre-set threshold to form a key Gram list, which is a list composed of multiple candidate words. It should be noted that the length of the sliding window can be adjusted according to actual needs, and this disclosure does not impose a limit on the length of the sliding window.

[0044] In some examples, the length of the sliding window is adjusted according to the corresponding proper noun. For instance, if the proper noun is two characters long, the length of the sliding window is set to one character, two characters, and three characters respectively. The first text content is then segmented according to these three sliding window lengths, resulting in three Gram lists for each of the three lengths. All three Gram lists correspond to the first proper noun, and the matching between the first proper noun and the three Gram lists is determined.

[0045] In some examples, word segmentation algorithms specifically designed for Chinese characters, such as those based on Hidden Markov Models (HMMs), can be used for word segmentation. Specifically, HMMs are used for Chinese word segmentation because, unlike English which uses spaces as natural separators, this algorithm can combine several characters into words to express their true meaning during Chinese character semantic recognition.

[0046] According to some embodiments, based on the pinyin edit distance between the first proper noun and each of the plurality of candidate word segments, it is determined whether the candidate word segment matches the first proper noun. Based on the pinyin edit distance, the candidate word segment with the closest pronunciation to the proper noun can be determined, thereby improving the accuracy of proper noun correction for properly nouns with similar pronunciations in the same language, such as in scenarios involving speech recognition to text conversion.

[0047] The pinyin edit distance refers to the minimum number of edit operations required to convert one noun into another. Allowed edit operations include adding, deleting, and replacing a phonetic symbol. Here, the pinyin edit distance refers to the minimum number of edit operations required to convert the pinyin of a candidate word into the pinyin of the first proper noun.

[0048] In some examples, the pinyin of the first proper noun and multiple candidate segments is obtained by querying, the pinyin edit distance of each candidate segment relative to the first proper noun is calculated, and the candidate segment with the smallest pinyin edit distance is selected as the first candidate segment.

[0049] According to some embodiments, based on the semantic similarity between the first proper noun and each of the plurality of candidate word segments, it is determined whether the candidate word segment matches the first proper noun. Based on semantic similarity, the candidate word segment most semantically similar to the proper noun can be determined, improving the accuracy of proper noun correction in scenarios such as foreign language translation.

[0050] In some examples, the first proper noun and multiple candidate segments can be embedded to obtain the representation vector of each noun, and the candidate segment whose representation vector is closest to the first proper noun can be selected as the first candidate segment.

[0051] According to some embodiments, determining whether to replace the first candidate word based on the first text content and the second text content includes: inputting the first text content and the second text content as joint input values ​​into a classifier to determine whether to replace the first candidate word.

[0052] Unlike previous methods that used end-to-end models for proper noun correction, the method disclosed in this paper uses a classifier to determine whether to replace the selected candidate words with the corresponding proper nouns, thus transforming the proper noun correction problem into a classification problem.

[0053] According to embodiments of this disclosure, there is no need to limit the length of the first candidate word segment used for replacement. In other words, error correction can be achieved even if the character length of the first candidate word segment differs from that of the proper noun, thereby solving the problems related to variable-length error correction in related technologies. Variable-length error correction refers to the difficulty in matching incorrect proper nouns with correct proper nouns in text if their lengths (number of characters) are inconsistent.

[0054] In some examples, when combining the first and second text content, an interpolation such as "" can be added between them. <sep>The separator symbol "" can further improve the accuracy of terminology correction.

[0055] In response to the input joint text, the classifier determines whether to "replace" or "not replace". If the determination is to "replace", the first candidate word in the first text is replaced with the first proper noun, or a specific identifier in the second text is replaced with the first proper noun. Otherwise, the first text is not changed.

[0056] According to some embodiments, the classifier is trained based on the ERNIE model. The ERNIE model is a model specifically designed for natural language processing, capable of analyzing contextual information relative to text to improve the accuracy of proper name correction. It should be noted that the classifier is not limited to the ERNIE model; other pre-trained language models can also be considered.

[0057] According to some embodiments, determining whether to replace the first candidate word based on the first text content and the second text content includes: inputting the first text content and the second text content into a dual encoder network model to determine whether to replace the first candidate word.

[0058] In the case of using a dual-encoder network model, the first text content can be input into one encoder network, and the second text content into another encoder network. The outputs of the two encoder networks are combined to determine whether to perform a replacement. In response to this determination, the first candidate word segmentation is processed.

[0059] According to some embodiments, the first text content is obtained in a simultaneous interpreting scenario. The proper noun correction method of this disclosure can meet the high requirements for efficiency and accuracy in simultaneous interpreting scenarios. It is understood that the first text content can be obtained through speech-to-text conversion, which will not be elaborated upon here.

[0060] Figure 3 An example of proper name correction of text is shown according to an embodiment of the present disclosure.

[0061] like Figure 3 As shown, the text content of the first text is "Zhang San just introduced the AI ​​development platform to everyone." Here, "Zhang San" corresponds to the first proper noun "Zhang San" in the proper noun database, and is a noun that needs to be corrected. It is understandable that in practical application scenarios, "Zhang San" can be the name of a public figure, and "Zhang San" can be any word that sounds the same as that person's name but has different characters.

[0062] Before step S301, multiple candidate word segments can be obtained based on the first text content. For example, the first text is segmented into multiple candidate word segments such as "Zhang San" and "development platform". Of course, in this process, word segmentation can also be performed based on the length of the first proper noun and by using N-Gram. For example, if the character length of the proper noun "Zhang San" is two characters. In response to this, the length of the N-Gram sliding window is set to two characters, and the obtained candidate word segments become "Zhang San", "development", and "platform", etc.

[0063] Step S301: Determine that the first candidate word segment among the multiple candidate word segments matches the first proper noun in the preset proper noun library.

[0064] Step S302: In response to step S301, generate the second text content by replacing the first candidate word segment in the first text content with a specific identifier.

[0065] As Figure 3 shown, in step S301, based on the phonetic edit distance between the proper noun and each candidate noun among the multiple candidate nouns, determine whether the candidate word segment matches the proper noun. For example, the proper noun "Zhang San" corresponds to the pinyin "zhangsan"; the pinyin of one of the multiple candidate word segments "Zhang San" is also "zhangsan"; the pinyin of one of the multiple candidate word segments "development" is "kaifa". It can be understood that the phonetic edit distance of "Zhang San" is 0, and the phonetic edit distance of "development" is 5. Therefore, "Zhang San" in the first text can be used as the first candidate word segment.

[0066] In step S302, in response to the minimum phonetic edit distance of "Zhang San", replace "Zhang San" in the first text with the specific identifier " <ne>", generating the second text "Just now" <ne>"I introduced the AI ​​development platform to everyone." Among them, there are pre-set specific identifiers. <ne>"This is just an example; other forms of specific identifiers may also be considered."

[0067] Step S303: Based on the first text content and the second text content, determine whether to replace the first candidate word segment.

[0068] like Figure 3 As shown, the first text content and the second text content are input as a joint input to the classifier to determine whether to replace the first candidate word segment. For example, by combining the first and second texts, we get "Just now, Zhang San introduced the AI ​​development platform to everyone." <ne>I introduced the AI ​​development platform to everyone. This text was then input into an ERNIE-based classifier. In response, the classifier output the instruction "Replace".

[0069] Step S304: In response to determining to replace the first candidate word, replace the first candidate word in the first text content with the first proper noun.

[0070] like Figure 3 As shown, the classifier outputs the instruction "replace", replacing "Zhang San" with "Zhang San" in the first text, and outputs the text after proper noun correction: "Just now Zhang San introduced the AI ​​development platform to everyone".

[0071] Step S305: In response to determining that the first candidate word segment will not be replaced, the first text content is retained.

[0072] like Figure 3 As shown, the classifier outputs the instruction "Do not replace", retaining the original first text and outputting it, that is, the output text "Zhang San just introduced the AI ​​development platform to everyone."

[0073] Figure 4 This is a flowchart illustrating a training method for a proper noun correction model according to an exemplary embodiment.

[0074] like Figure 4 As shown, the training method of the proper noun correction model includes: Step 401, obtaining a sample dataset, wherein each sample data in the sample dataset includes first text content, second text content, and a label, and wherein the second text content is generated by replacing the first word segment in the first text content with a specific identifier; Step 402, initializing multiple parameters of the correction model; and for each sample data, performing the following operations: Step 403, inputting the first text content and the second text content into the correction model to obtain an output value, wherein the output value represents the probability that the first word segment is a preset proper noun; Step 404, calculating a loss function based on the output value and the label corresponding to the sample data; and Step 405, adjusting multiple parameters in the correction model based on the loss function.

[0075] The training process is based on the first text with correct content, which is a supervised learning process that is easy to transfer. The sample dataset is easy to construct, and the proper name correction effect is better than that of unsupervised learning.

[0076] According to some embodiments, the sample dataset includes multiple positive sample data, wherein the first word segment of the first text content of each of the multiple positive sample data is a preset proper noun. The label of the positive sample data is set to 1.

[0077] According to some embodiments, the sample dataset includes multiple negative sample data, wherein the first word segment in the first text content of each of the multiple negative sample data is not a preset proper noun. The label of the negative sample data is set to 0.

[0078] Table 1 lists an example of positive and negative sample data. Data numbered 1 represents positive sample data, where the data is identified by a specific identifier. <ne>"The alternative word 'Zhang San' is a pre-defined proper noun; dataset number 2 is negative sample data, in which..." <ne>The alternative word "school gate" is not the preset first segmentation word. In practical application scenarios, not all identified proper nouns need to be corrected. The method described in the embodiments of this disclosure can improve the targeting of the proper noun correction model, thereby improving the efficiency of correction.

[0079]

[0080] Table 1

[0081] According to some embodiments, the preset proper nouns are identified based on proper noun recognition, NER, and models.

[0082] When constructing the sample dataset, a Named Entity Recognition (NER) model was introduced to achieve semi-automatic generation of sample data, simplifying the human resource input. Named Entity Recognition (NER) refers to identifying nouns in text by entity class, time class, and number class, or by person name, organization name, place name, time, date, currency, and percentage. For example, "Zhang San" and "school gate" in Table 1 can both be proper nouns identified using the NER model.

[0083] According to some embodiments, the error correction model is an ERNIE-based classifier, wherein the first and second text contents of each sample data are input into the classifier as joint input values.

[0084] By using the ERNIE model, it is easier to train the classifier and save training costs.

[0085] According to some embodiments, when combining the first text content and the second text content, an interpolation such as "" can be added between the two. <sep>The separator symbol "" is used to further improve the model accuracy.

[0086] According to some embodiments, the error correction model is a dual encoder network model.

[0087] Figure 5 A block diagram of a proper noun correction apparatus according to an embodiment of the present disclosure is shown.

[0088] According to one aspect of this disclosure, a proper noun correction device 500 is also disclosed, the device comprising: a candidate word segmentation acquisition unit 501, configured to acquire a plurality of candidate words based on a first text content; a text generation unit 502, configured to generate second text content by replacing the first candidate word in the first text content with a specific identifier in response to determining that a first candidate word among the plurality of candidate words matches a first proper noun in a preset proper noun library; a judgment unit 503, configured to determine whether to replace the first candidate word based on the first text content and the second text content; and a replacement unit 504, configured to replace the first candidate word in the first text content with the first proper noun in response to determining that the first candidate word should be replaced.

[0089] Figure 6 A block diagram of a training apparatus for a proper noun correction model according to an embodiment of the present disclosure is shown.

[0090] According to one aspect of this disclosure, a training device 600 for a proper noun correction model is also disclosed. The device includes: a sample acquisition unit 601, configured to acquire a sample dataset, wherein each sample data in the sample dataset includes first text content, second text content, and a label, and wherein the second text content is generated by replacing a first word segment in the first text content with a specific identifier; an initialization unit 602, configured to initialize multiple parameters of the correction model; an input unit 603, configured to input the first text content and the second text content into the correction model to obtain an output value, the output value representing the probability that the first word segment is a preset proper noun; a loss function calculation unit 604, configured to calculate a loss function based on the output value and the label corresponding to the sample data; and a parameter update unit 605, configured to adjust multiple parameters in the correction model based on the loss function.

[0091] According to one aspect of this disclosure, an electronic device is also disclosed, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the methods described above.

[0092] According to one aspect of this disclosure, a non-transitory computer-readable storage medium storing computer instructions is also disclosed, wherein the computer instructions are used to cause a computer to perform the above-described method.

[0093] According to one aspect of this disclosure, a computer program product is also disclosed, comprising a computer program that, when executed by a processor, implements the above-described method.

[0094] refer to Figure 7 The present invention describes a structural block diagram of an electronic device 700 that can serve as a server or client of the present disclosure, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0095] like Figure 7 As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded into random access memory (RAM) 703 from storage unit 708. RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0096] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, output unit 707, storage unit 708, and communication unit 709. Input unit 706 can be any type of device capable of inputting information to device 700. Input unit 706 can receive input numerical or character information and generate key signal inputs related to user settings and / or function control of the electronic device, and may include, but is not limited to, a mouse, keyboard, touchscreen, trackpad, trackball, joystick, microphone, and / or remote control. Output unit 707 can be any type of device capable of presenting information, and may include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 708 may include, but is not limited to, a hard disk and an optical disk. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, a modem, network card, infrared communication device, wireless communication transceiver, and / or chipset, such as Bluetooth. TM Devices, 802.11 devices, WiFi devices, WiMax devices, cellular communication devices and / or the like.

[0097] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as the proper noun correction method. For example, in some embodiments, the proper noun correction method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by computing unit 701, one or more steps of the proper noun correction method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the proper noun correction method by any other suitable means (e.g., by means of firmware).

[0098] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0099] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0100] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0101] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0102] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0103] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0104] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0105] While embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of the invention is not limited by these embodiments or examples, but only by the granted claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. Furthermore, the steps may be performed in a different order than that described in this disclosure. Further, various elements in the embodiments or examples may be combined in various ways. Importantly, as the technology evolves, many elements described herein can be replaced by equivalents that appear after this disclosure.< / sep> < / ne> < / ne> < / ne> < / ne> < / ne> < / ne> < / sep>

Claims

1. A computer-implemented proper noun error correction method, the method comprising: obtaining a plurality of candidate segments based on first text content; in response to determining that a first candidate segment in the plurality of candidate segments matches a first proper noun in a pre-set proper noun library, generating second text content by replacing the first candidate segment in the first text content with a specific identifier, wherein the first candidate segment comprises at least two characters that are replaced as a whole by a single specific identifier, and wherein a character length of the first candidate segment is different from a character length of the first proper noun; determining whether to replace the first candidate segment based on the first text content and the second text content, comprising: concatenating the first text content and the second text content as a whole sentence to obtain a joint input value; and inputting the joint input value to a classifier to determine whether to replace the first candidate segment; and in response to determining to replace the first candidate segment, replacing the first candidate segment in the first text content with the first proper noun.

2. The method of claim 1, wherein, The obtaining a plurality of candidate segments based on first text content comprises: performing segmentation on the first text content using an N-Gram algorithm based on a length of the first proper noun.

3. The method of claim 1 or 2, further comprising: determining whether each candidate segment in the plurality of candidate segments matches the first proper noun based on a phonetic edit distance between the first proper noun and the candidate segment.

4. The method of claim 1 or 2, further comprising: determining whether each candidate segment in the plurality of candidate segments matches the first proper noun based on a semantic similarity between the first proper noun and the candidate segment.

5. The method of claim 1 or 2, wherein, The classifier is trained based on an ERNIE model.

6. The method of claim 1 or 2, wherein, The determining whether to replace the first candidate segment based on the first text content and the second text content comprises: inputting the first text content and the second text content to a dual-encoder network model to determine whether to replace the first candidate segment.

7. The method of claim 1 or 2, wherein, The first text content is obtained in a simultaneous interpretation translation scenario.

8. A training method of a proper noun error correction model, the method comprising: obtaining a sample data set, wherein each sample data in the sample data set comprises first text content, second text content, and a label, and wherein the second text content is generated by replacing a first segment in the first text content with a specific identifier, wherein the first segment comprises at least two characters that are replaced as a whole by a single specific identifier, and wherein at least one sample data in the sample data set comprises a first segment in the first text content that has a different character length than a pre-set proper noun corresponding to the first segment; initializing a plurality of parameters of the error correction model; and for each sample data, performing the following operations: inputting the first text content and the second text content into the error correction model to obtain an output value representing a probability that the first segmented word is a preset proper noun; calculating a loss function based on the output value and a label corresponding to the sample data; and adjusting a plurality of parameters in the error correction model based on the loss function, wherein the error correction model is a classifier, and wherein a joint input value obtained by concatenating the first text content and the second text content of each sample data is input into the classifier.

9. The training method of claim 8, wherein, The sample data set includes a plurality of positive sample data, and wherein the first segmented word in the first text content of each positive sample data in the plurality of positive sample data is a preset proper noun.

10. The training method of claim 8 or 9, wherein, The sample data set includes a plurality of negative sample data, and wherein the first segmented word in the first text content of each negative sample data in the plurality of negative sample data is not a preset proper noun.

11. The training method of claim 8 or 9, wherein, The preset proper noun is identified based on a name recognition, NER, model.

12. The training method of claim 8 or 9, wherein, The error correction model is based on an ERNIE model.

13. The training method of claim 8 or 9, wherein, The error correction model is a double-encoder network model.

14. A proper noun error correction apparatus, the apparatus comprising: a candidate segmented word obtaining unit configured to obtain a plurality of candidate segmented words based on first text content; a text generating unit configured to, in response to determining that a first candidate segmented word in the plurality of candidate segmented words matches a first proper noun in a preset proper noun library, generate second text content by replacing the first candidate segmented word in the first text content with a specific identifier, wherein the first candidate segmented word includes at least two characters that are replaced as a whole by a single specific identifier, and the character length of the first candidate segmented word is different from the character length of the first proper noun; a judging unit configured to determine whether to replace the first candidate segmented word based on the first text content and the second text content, including: concatenating the first text content and the second text content to obtain a joint input value; and inputting the joint input value into a classifier to determine whether to replace the first candidate segmented word; and a replacing unit configured to, in response to determining to replace the first candidate segmented word, replace the first candidate segmented word in the first text content with the first proper noun.

15. A training apparatus of a proper noun error correction model, the apparatus comprising: a sample obtaining unit configured to obtain a sample data set, wherein each sample data in the sample data set includes first text content, second text content, and a label, and wherein the second text content is generated by replacing a first segmented word in the first text content with a specific identifier, wherein the first segmented word includes at least two characters that are replaced as a whole by a single specific identifier, and the character length of the first segmented word in the first text content included in at least one sample data in the sample data set is different from the character length of a preset proper noun corresponding to the first segmented word; an initialization unit configured to initialize a plurality of parameters of the error correction model; an input unit configured to input the first text content and the second text content into the error correction model to obtain an output value representing a probability that the first segmentation is a pre-set proper noun; a loss function calculation unit configured to calculate a loss function based on the output value and a label corresponding to the sample data; and a parameter updating unit configured to adjust the plurality of parameters in the error correction model based on the loss function, wherein the error correction model is a classifier, and wherein the joint input value obtained by concatenating the first text content and the second text content of each sample data is input into the classifier.

16. An electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 13.

17. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable a computer to perform the method according to any one of claims 1 to 13.

18. A computer program product comprising a computer program, wherein, The computer program, when executed by a processor, implements the method according to any one of claims 1 to 13. The computer program, when executed by a processor, implements the method according to any one of claims 1 to 13.

Images