Recovery and parsing of zero pronouns

Abstract

A method, computer program, and computer system for recovering omitted pronouns are provided. The method, computer program, and computer system involve receiving data corresponding to one or more input words and determining a contextual representation of the received input word data. The omitted pronouns can be identified based on probability values ​​associated with the contextual representation, and a span associated with one or more of the received input words can be determined. The span can correspond to the input word referred to by the omitted pronoun in the input word.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Patent Application No. 16 / 704,241 filed in the U.S. Patent and Trademark Office on December 5, 2019, the entire disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present invention relates to the field of natural language processing, and more particularly, to a method and device, a computer system and a storage medium for restoring omitted pronouns. Background Art

[0004] Many languages ​​around the world omit subject pronouns from sentences. While subject pronouns are rarely omitted in non-pro-drop languages ​​such as English, pro-drop occurs frequently in other languages ​​such as Chinese, where up to 30% of subject pronouns may be omitted. For these languages, the omitted pronouns or zero pronouns may correspond to the subject of the sentence that can be easily inferred from the context. In these cases, the subject can be omitted for brevity and efficiency when it does not cause ambiguity to human listeners. Zero pronoun omission can result in the loss of information such as the subject in a sentence or the object of a central predicate, which can cause ambiguity in many NLP applications such as machine translation, multi-turn dialogue-based question answering, and dialogue understanding. Summary of the invention

[0005] Embodiments relate to methods, systems, and computer-readable media for recovering omitted pronouns. According to one aspect, a method for recovering omitted pronouns is provided. The method may include receiving, by a computer, data corresponding to one or more input words, and determining a contextual representation of the received input word data. The computer may confirm the omitted pronoun based on a probability value associated with the contextual representation, and may determine a span associated with one or more of the received input words, the span corresponding to the input word referred to by the omitted pronoun in the input word.

[0006] According to another aspect, a computer system is provided. The computer system may include: one or more processors; one or more computer-readable memories; one or more computer-readable tangible storage devices; and program instructions, the program instructions stored on at least one of the one or more storage devices, for execution by at least one of the one or more processors via at least one of the one or more memories, whereby the computer system is capable of executing the method. The method may include receiving data corresponding to one or more input words by a computer, and determining a contextual representation of the received input word data. The computer may confirm the omitted pronoun based on a probability value associated with the contextual representation, and may determine a span associated with one or more received input words, the span corresponding to the input word referred to by the omitted pronoun in the input word.

[0007] According to another aspect, an apparatus for recovering omitted pronouns is provided. The apparatus comprises: a receiving module for receiving data corresponding to one or more input characters; a first determining module for determining a contextual representation of the received input character data; a confirming module for confirming the omitted pronouns based on a probability value associated with the contextual representation; and a second determining module for determining a span associated with one or more of the received input characters, the span corresponding to the input character referred to by the omitted pronoun in the input characters.

[0008] According to another aspect, a computer-readable medium for recovering omitted pronouns is provided. The computer-readable medium may include one or more computer-readable storage devices and program instructions stored on at least one of the one or more tangible storage devices, and the program instructions can be executed by a processor. The program instructions can be executed by a processor for performing the following method, which method can accordingly include receiving data corresponding to one or more input words by a computer and determining the contextual representation of the received input word data. The computer can confirm the omitted pronoun based on the probability value associated with the contextual representation, and can determine the span associated with one or more of the received input words, which span corresponds to the input word referred to by the omitted pronoun in the input word.

[0009] As can be seen from the above description, the present invention provides a method and apparatus, a computer system and a storage medium for recovering omitted pronouns. The method includes: receiving data corresponding to one or more input words by a computer; determining the contextual representation of the received input word data by the computer; confirming the omitted pronoun by the computer based on a probability value associated with the contextual representation; and determining the span associated with one or more of the received input words by the computer, the span corresponding to the input word referred to by the omitted pronoun in the input word. Through the method of the present invention, the omitted pronouns in the sentence can be accurately recovered, and the objects referred to by the omitted pronouns can also be accurately determined, thereby realizing the recovery and parsing of the omitted pronouns, so that ambiguity will not occur in many NLP applications such as machine translation, question answering based on multi-round dialogues, and dialogue understanding. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] These and other objects, features and advantages will become apparent from the following detailed description of illustrative embodiments read in conjunction with the accompanying drawings. Since the illustrations are intended to clearly assist those skilled in the art in understanding the present disclosure in conjunction with the specific embodiments, the various features of the drawings are not to scale. In the drawings:

[0011] Figure 1 illustrates a networked computer environment according to at least one embodiment;

[0012] Figure 2 is a block diagram of a process for detecting and restoring omitted pronouns according to at least one embodiment;

[0013] Figure 3 is an operational flow diagram of steps performed by a program for detecting and restoring omitted pronouns according to at least one embodiment;

[0014] Figure 4 According to at least one embodiment, Figure 1 A block diagram of an illustrative cloud computing environment of a computer system depicted in;

[0015] Figure 5 According to at least one embodiment Figure 4 A block diagram of the functional layers of an illustrative cloud computing environment; and

[0016] Figure 6 According to at least one embodiment Figure 1 A block diagram of the internal and external components of a computer and server is depicted in FIG. DETAILED DESCRIPTION

[0017] Detailed embodiments of the claimed structures and methods are disclosed herein; however, it is understood that the disclosed embodiments are merely illustrative of the claimed structures and methods that may be implemented in various forms. However, various aspects of the present disclosure may be implemented in many different forms and should not be construed as limited to the exemplary embodiments set forth herein. Instead, these exemplary embodiments are provided so that the present disclosure will be thorough and complete and will fully convey the scope of the present disclosure to those skilled in the art. In the specification, details of well-known features and techniques may be omitted to avoid unnecessarily obscuring the embodiments presented.

[0018] Embodiments relate generally to the field of computing, and more particularly to natural language processing. The exemplary embodiments described below provide systems, methods, and program products for predicting whether a pronoun is omitted from a sentence and recovering the omitted pronoun based on context in the sentence, among other things. Thus, some embodiments have the ability to improve the field of computing by allowing a computer to recover an omitted pronoun and determine a predicate associated with the omitted subject pronoun. Thus, the computer-implemented methods, computer systems, and computer-readable media disclosed herein can be used to improve natural language processing applications, among other things, for users who speak a language that omits pronouns.

[0019] As mentioned above, many languages ​​around the world omit subject pronouns. While subject pronouns are rarely omitted in non-pro-drop languages ​​such as English, pro-drop occurs frequently in other languages ​​such as Chinese, where up to 30% of subject pronouns may be omitted. For these languages, the omitted pronouns or zero pronouns may correspond to the subject of the sentence that can be easily inferred from the context. In these cases, the subject can be omitted for simplicity and efficiency when it does not cause ambiguity to human listeners. However, because computers using natural language processing (NLP) may not be able to determine the omitted subject pronouns, pronoun omission may cause problems for these computers. Zero pronoun omission may cause the loss of information such as the subject in a sentence or the object of the central predicate, which may cause ambiguity in many NLP applications such as machine translation, multi-round dialogue-based question answering, and dialogue understanding.

[0020] There are two long-standing tasks for zero pronouns: zero pronoun recovery, which can be used to recover the original pronoun; and zero pronoun resolution, which can be used to determine the word in the sentence that each omitted pronoun may refer to. Because the results of these two tasks highly interact with each other, it may be advantageous to solve these two tasks together. The present method, computer system, and computer-readable medium can jointly solve these two tasks under a multi-task learning framework. For example, a Bidirectional Encoder Representation From Transformers (BERT) model can be used. The model can be trained in a heterogeneous manner on multiple mixed datasets before the model can be updated, based on the task-specific loss values ​​of the batch at each training step.

[0021] Various aspects are described herein with reference to flowchart illustrations and / or block diagrams of methods, devices (systems) and computer-readable media according to certain embodiments. It will be understood that each block of the flowchart illustration and / or block diagram and the combination of blocks in the flowchart illustration and / or block diagram can be implemented by computer-readable program instructions.

[0022] The exemplary embodiments described below provide systems, methods and program products for detecting and recovering omitted pronouns. According to the present embodiments, the detection and classification can be provided by contextual analysis of received input words and calculation of probabilities associated with detection, recovery and resolution of omitted pronouns.

[0023] Now refer to Figure 1 , shows a functional block diagram of a networked computer environment showing a zero pronoun recovery system 100 (hereinafter "system") for improving detection, recovery and resolution of omitted pronouns. It should be understood that Figure 1 This merely provides an illustration of one implementation and does not imply any limitation with respect to the environments in which different embodiments may be implemented. Many modifications to the depicted environment may be made based on design and implementation requirements.

[0024] System 100 may include computer 102 and server computer 114. Computer 102 may communicate with server computer 114 via communication network 110 (hereinafter "network"). Computer 102 may include processor 104 and software program 108, which is stored on data storage device 106 and is capable of interfacing with a user and communicating with server computer 114. As will be described below with reference to Figure 6As discussed, computer 102 may include internal components 800A and external components 900A, respectively, and server computer 114 may include internal components 800B and external components 900B, respectively. Computer 102 may be, for example, a mobile device, a phone, a personal digital assistant, a netbook, a laptop computer, a tablet computer, a desktop computer, or any type of computing device capable of running programs, accessing a network, and accessing a database.

[0025] As follows about Figure 4 and Figure 5 As discussed, the server computer 114 may also operate in a cloud computing service model such as Software as a Service (SaaS), Platform as a Service (PaaS), or Infrastructure as a Service (IaaS). The server computer 114 may also be located in a cloud computing deployment model such as a private cloud, a community cloud, a public cloud, or a hybrid cloud.

[0026] The server computer 114, which can be used to detect, recover and resolve omitted pronouns, can run a zero pronoun recovery program 116 (hereinafter "the program") that can interact with the database 112. Figure 3 The zero pronoun recovery program method is explained in more detail. In one embodiment, computer 102 can operate as an input device including a user interface, and program 116 can be primarily run on server computer 114. In an alternative embodiment, program 116 can be primarily run on one or more computers 102, and server computer 114 can be used to process and store data used by program 116. It should be noted that program 116 can be a stand-alone program, or can be integrated into a larger zero pronoun recovery program.

[0027] However, it should be noted that in some instances, processing for program 116 may be shared in any ratio between computer 102 and server computer 114. In another embodiment, program 116 may operate on more than one computer, server computer, or some combination of computers and server computers, such as, for example, multiple computers 102 in communication with a single server computer 114 across network 110. In another embodiment, for example, program 116 may operate on multiple server computers 114 in communication with multiple client computers across network 110. Alternatively, the program may operate on a network server in communication with the server and multiple client computers across the network.

[0028] The network 110 may include a wired connection, a wireless connection, a fiber optic connection, or some combination thereof. In general, the network 110 may be any combination of connections and protocols that will support communication between the computer 102 and the server computer 114. The network 110 may include various types of networks, such as, for example, a local area network (LAN), a wide area network (WAN) such as the Internet, a telecommunication network such as a public switched telephone network (PSTN), a wireless network, a public switched network, a satellite network, a cellular network (e.g., a fifth generation (5G) network, a long-term evolution (LTE) network, a third generation (3G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a metropolitan area network (MAN), a private network, an ad hoc network, an intranet, a fiber-based network, etc., and / or a combination of these or other types of networks.

[0029] Figure 1 The number and arrangement of devices and networks shown are provided as examples. Figure 1 In addition to the devices and / or networks shown, there may be additional devices and / or networks, fewer devices and / or networks, different devices and / or networks, or differently arranged devices and / or networks. Figure 1 Two or more of the devices shown may be implemented in a single device, or Figure 1 The single device shown may be implemented as multiple distributed devices. Additionally or alternatively, one or more devices of the system 100 may perform one or more functions described as being performed by another set of devices of the system 100.

[0030] Reference Figure 2 , depicts a block diagram of the zero pronoun recovery procedure 116 . Figure 2 Can be helped by Figure 1 According to one or more embodiments, the zero pronoun recovery program 116 may be located on the computer 102 ( Figure 1 ) or server computer 114 ( Figure 1). The zero pronoun recovery program 116 may accordingly include a bidirectional encoder representation quantity (BERT) encoder 202 from a transformer and a zero pronoun (Zero Pronoun, ZP) function module 204, among others. The BERT encoder 202 may receive one or more input words 206A to 206N, and may append a [CLS] class token to the input words 206A to 206N. The BERT encoder 202 may output context representations 208A to 208N that may correspond to each of the input words 206A to 206N. The BERT encoder 202 may also output a context representation 208' that may correspond to the context representation of the [CLS] token. The ZP function module 204 may receive the context representations 208' and 208A to 208N as input to perform one or more functions.

[0031] The task of zero pronoun recovery can be to recover any omitted pronouns of the input text. Since pronouns are enumerable, the zero pronoun recovery program 116 can convert this task into a classification problem for each word space. The ZP function module 204 can be configured to perform the zero pronoun recovery function f using the context representations 208A to 208N output by the BERT encoder 202. rec . Restore the pronoun rec at the space between input word 206(i-1) and input word 206i i The probability can be p(rec i |X,i)=softmax(W rec (h i-1 ,h i ]+b rec ), where W rec and b rec , may be parameters of a linear classifier, and h may correspond to context representations 208 ′ and 208A to 208N.

[0032] The zero pronoun recovery program 116 may perform the zero pronoun resolution function f by predicting the span associated with each omitted pronoun. res. According to one embodiment, zero pronoun recovery can be performed first, and this information can be used in zero pronoun parsing. According to an alternative embodiment, the span "(0, 0)" can be assigned to non-zero pronoun positions (i.e., positions with input words). This can minimize conflicts because the position "0" can correspond to the [CLS] token so that no real span can be "(0, 0)". The zero pronoun recovery program 116 can convert the zero pronoun parsing task for each word space between the input word 206(i-1) and the input word 206i into a machine reading comprehension problem, where the parsing span can correspond to the target answer. The starting position and the ending position can be determined for each span based on the concatenation of all word states and a self-attention module for predicting the starting position and the ending position of each zero pronoun parsing span. Therefore, the ZP function module 204 can be configured to perform the zero pronoun parsing function f res . Predict the entire span res i The probability can be

[0033]

[0034] Where H can be the concatenation of context representations 208' and 208A to 208N, and SelfAttn st and SelfAttn ed They can be self-attention modules at the start and end positions respectively.

[0035] To improve the robustness of heterogeneous training, zero pronoun detection can be used as a subtask of both zero pronoun detection and zero pronoun recovery as well as additional training value. Because zero pronoun detection can be a binary task that determines whether each word space has a zero pronoun, zero pronoun recovery can be performed across multiple types of datasets. As an auxiliary task, the zero pronoun recovery program 116 can detect omitted pronouns for improved training. Since determining whether a sentence is likely to have omitted pronouns can be a binary classification, the ZP function module 204 can be configured to perform a zero pronoun detection function f det The probability of detecting an omitted pronoun can be p(det i |X,i)=softmax(W det (h i-1 ,h i ]+b det ), where W det and b det , may be a model parameter, and h may correspond to context representations 208 ′ and 208A to 208N.

[0036] The ZP function module 204 can be trained using a combination of zero pronoun recovery and zero pronoun resolution tasks and shuffled data to take advantage of more supervision. For example, each data instance can contain only annotations for zero pronoun recovery or zero pronoun resolution. Therefore, the loss value of an example can be defined as loss = -∑ i∈1..N αlogp(rec i |X, i)-βlogp(res i |X, i)-γlogp(det i |X, i), where α, β, and γ can be used as coefficients and indicators for each task. For α and β, the value can be "1" if the corresponding supervision value exists; otherwise, the value can be "0". Because the supervision value associated with zero pronoun detection may exist for all instances, the value of γ can be set to a low value to avoid skewing the loss function.

[0037] Now refer to Figure 3 , depicts an operational flow chart 300 showing steps performed by a program for restoring omitted pronouns. Figure 3 Can be helped by Figure 1 and Figure 2 As described above, the zero pronoun recovery procedure 116 ( Figure 1 ) can quickly and efficiently detect and restore omitted pronouns.

[0038] At 302, data corresponding to one or more input words is received by a computer. The input words may be, for example, sentences in substantially any language. The sentences may have, among other things, omitted subject pronouns. In operation, the zero pronoun recovery program 116 ( Figure 1 ) may be stored in computer 102 ( Figure 1 ) or server computer 114 ( Figure 1 According to one embodiment, the zero pronoun recovery program 116 can retrieve the data from the database 112 ( Figure 1 ) to search for input characters 206A to 206N ( Figure 2 According to an alternative embodiment, the input words 206A to 206N may be stored in the data storage device 106 ( Figure 1 ), and the zero pronoun recovery program 116 can be connected via the communication network 110 ( Figure 1 ) receives input words 206A to 206N from computer 102.

[0039] At 304, a contextual representation of the received input word data is determined by a computer. The contextual representation may be determined by a Bidirectional Encoder Representation from Transformer (BERT) model. The contextual representation may be used to determine how one or more input words are used in the context of their corresponding sentence. In operation, the BERT encoder 202 ( Figure 2 ) may receive as input input words 206A to 206N and may output context representations 208A to 208N that may correspond to the input words 206A to 206N ( Figure 2 ). The BERT encoder 202 may append a [CLS] class token to the beginning of the input words 206A to 206N and may output a contextual representation 208' corresponding to the [CLS] token.

[0040] At 306, the omitted pronoun is confirmed by the computer based on the probability value associated with the contextual representation. Because pronouns are enumerable and there are a finite number of possible pronouns and corresponding types, the recovery of omitted pronouns can be transformed into a classification problem for each word space. Therefore, the computer may be able to determine which pronoun is omitted from the sentence based on the context. In operation, the ZP function module 204 ( Figure 2 ) can represent the context 208' and 208A to 208N ( Figure 2 ) Execute the zero pronoun recovery function f rec The ZP function module 204 can be based on the probability p(rec i |X, i) to determine the input word 206A to 206N ( Figure 2 ) Which pronoun is omitted in the sentence?

[0041] At 308, a span associated with one or more of the received input words is determined by the computer. The span may correspond to one or more of the input words to which the omitted pronoun refers. By determining which words constitute the predicate corresponding to the omitted subject pronoun, the determination of the recovered subject pronoun may be improved. In operation, the ZP function module 204 ( Figure 2 ) can represent the context 208' and 208A to 208N ( Figure 2 ) Execute the zero pronoun resolution function f res , in order to predict the span of the predicate referred to by the omitted pronoun. The ZP function module 204 can be based on the probability p(res i |X, i) to determine the span.

[0042] Understandably, Figure 3 This merely provides an illustration of one implementation and does not imply any limitation on how different embodiments may be implemented.Many modifications may be made to the described environment based on design and implementation requirements.

[0043] It is understood in advance that although the present disclosure includes detailed descriptions about cloud computing, the implementation of the teachings recorded herein is not limited to a cloud computing environment. Instead, some embodiments can be implemented in conjunction with any other type of computing environment now known or later developed.

[0044] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be quickly provisioned and released with minimal management effort or interaction with a service provider. The cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0045] Features are as follows:

[0046] On-demand self-service: Cloud consumers can unilaterally and automatically provision computing capabilities such as server time and network storage as needed, without requiring manual interaction with the service provider.

[0047] Broad network access: Capabilities are available over the network and accessed through standard mechanisms that facilitate the use of heterogeneous thin-client platforms or thick-client platforms (e.g., mobile phones, laptops, and PDAs).

[0048] Resource pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated based on demand. There is a sense of location independence, as consumers typically do not control or know the exact location of the resources provided, but may be able to specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0049] Rapid elasticity: Capacity can be quickly and elastically provisioned (in some cases automatically) to quickly scale out, and quickly released to quickly scale in. To the consumer, the capacity available for provisioning generally appears unlimited and can be purchased in any quantity at any time.

[0050] Metered Services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both the provider and consumer of the utilized service.

[0051] The service model is as follows:

[0052] Software as a Service (SaaS): The capability provided to the consumer is to use the provider's applications running on a cloud infrastructure. The applications can be accessed from a variety of client devices through a thin client interface such as a web browser (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure including networks, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.

[0053] Platform as a Service (PaaS): The capability provided to consumers is to deploy consumer-created or acquired applications created using programming languages ​​and tools supported by the provider onto the cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure including networks, servers, operating systems, or storage, but has control over the deployed applications and possible configuration of the application hosting environment.

[0054] Infrastructure as a Service (IaaS): The capability provided to consumers is the provision of processing, storage, networking, and other basic computing resources on which consumers can deploy and run arbitrary software, which may include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but have control over the operating system, storage, deployed applications, and possibly limited control over selected network components (e.g., host firewalls).

[0055] The deployment model is as follows:

[0056] Private cloud: Cloud infrastructure is operated only for an organization. The cloud infrastructure can be managed by the organization or a third party and can exist locally (on-premises) or externally (off-premises).

[0057] Community Cloud: Cloud infrastructure is shared by a number of organizations and supports a specific community with shared concerns (e.g., mission, security requirements, policies, and compliance considerations). Cloud infrastructure can be managed by the organization or a third party and can exist on-premises or off-premises.

[0058] Public cloud: Cloud infrastructure available to the general public or large industry groups and owned by the organization that sells cloud services.

[0059] Hybrid cloud: A cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain unique entities but are bound together by standardized or proprietary technologies to enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0060] The cloud computing environment is service-oriented, with a focus on statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing is an infrastructure consisting of a network of interconnected nodes.

[0061] Reference Figure 4 , depicts an illustrative cloud computing environment 400. As shown, the cloud computing environment 400 includes one or more cloud computing nodes 10, and local computing devices used by cloud consumers, such as, for example, personal digital assistants (PDAs) or cellular phones 54A, desktop computers 54B, laptop computers 54C, and / or automotive computer systems 54N, can communicate with the cloud computing nodes 10. The cloud computing nodes 10 can communicate with each other. The cloud computing nodes 10 can be physically or virtually grouped (not shown) into one or more networks, such as a private cloud, community cloud, public cloud, or hybrid cloud, or a combination thereof, as described above. This allows the cloud computing environment 400 to provide infrastructure, platforms, and / or software as a service, and cloud consumers do not need to maintain resources on local computing devices for the services. It should be understood that Figure 4 The types of computing devices 54A-54N shown in are intended to be illustrative only, and cloud computing node 10 and cloud computing environment 400 may communicate with any type of computerized device over any type of network and / or network addressable connection (eg, using a web browser).

[0062] Reference Figure 5 , showing a cloud computing environment 400 ( Figure 4 ) provides a set of functional abstraction layers 500. It should be understood in advance that Figure 5 The components, layers, and functions shown in are intended to be illustrative only, and the implementation of the present disclosure is not limited thereto. As depicted in the figure, the following layers and corresponding functions are provided:

[0063] The hardware and software layer 60 includes hardware components and software components. Examples of hardware components include: host 61; server 62 based on RISC (Reduced Instruction Set Computer) architecture; server 63; blade server 64; storage device 65; and network and network components 66. In some embodiments, the software components include network application server software 67 and database software 68.

[0064] Virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual servers 71 ; virtual storage 72 ; virtual networks 73 , which include virtual private networks; virtual applications and operating systems 74 ; and virtual clients 75 .

[0065] In one example, the management layer 80 may provide the following functions. Resource provisioning 81 provides dynamic acquisition of computing resources and other resources for performing tasks in a cloud computing environment. Metering and pricing 82 provides cost tracking when utilizing resources in a cloud computing environment, as well as billing or invoicing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides authentication for cloud consumers and tasks, as well as protection for data and other resources. User portal 83 provides access to the cloud computing environment for consumers and system administrators. Service level management 84 provides cloud computing resource allocation and management so that the required service levels are met. Service Level Agreement (SLA) planning and fulfillment 85 provides pre-scheduling and acquisition of cloud computing resources, anticipating future demand for cloud computing resources based on the SLA.

[0066] Workload layer 90 provides examples of functionality for which a cloud computing environment may be utilized. Examples of workloads and functionality that may be provided from this layer include: mapping and navigation 91; software development and lifecycle management 92; virtual classroom instructional delivery 93; data analytics processing 94; transaction processing 95; and zero pronoun recovery 96. Zero pronoun recovery 96 may detect and recover omitted pronouns for sentences written in a language that may allow the use of zero pronouns.

[0067] Figure 6 According to the illustrative embodiment Figure 1 600 of the internal and external components of a computer depicted in FIG. It should be understood that Figure 6 This merely provides an illustration of one implementation and does not imply any limitation with respect to the environments in which different embodiments may be implemented. Many modifications to the depicted environment may be made based on design and implementation requirements.

[0068] Computer 102 ( Figure 1 ) and server computer 114 ( Figure 1 ) can include Figure 6 Respective sets of internal components 800A, 800B and external components 900A, 900B are shown. Each set of internal components 800 includes: one or more processors 820 on one or more buses 826, one or more computer-readable RAMs 822 and one or more computer-readable ROMs 824; one or more operating systems 828; and one or more computer-readable tangible storage devices 830.

[0069] Processor 820 is implemented in hardware, firmware or a combination of hardware and software. Processor 820 is a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC) or other types of processing components. In some implementations, processor 820 includes one or more processors that can be programmed to perform functions. Bus 826 includes components that allow communication between internal components 800A and 800B.

[0070] One or more operating systems 828, software programs 108 ( Figure 1 ) and server computer 114 ( Figure 1 ) on zero pronoun recovery program 116 ( Figure 1 ) are stored on one or more corresponding computer-readable tangible storage devices 830 for execution by one or more corresponding processors 820 via one or more corresponding RAMs 822 (which typically include cache memory). Figure 6 In the illustrated embodiment, each of the computer-readable tangible storage devices 830 is a disk storage device of an internal hard drive. Alternatively, each of the computer-readable tangible storage devices 830 is a semiconductor storage device such as ROM 824, EPROM, flash memory, optical disk, magneto-optical disk, solid state disk, compact disc (CD), digital versatile disc (DVD), floppy disk, cassette, magnetic tape, and / or another type of non-transitory computer-readable tangible storage device capable of storing computer programs and digital information.

[0071] Each set of internal components 800A, 800B also includes a W / R drive or interface 832 to read from and write to one or more portable computer readable tangible storage devices 936 such as CD-ROM, DVD, memory stick, tape, disk, optical disk, or semiconductor storage device. Figure 1 ) and zero pronoun recovery program 116 ( Figure 1) can be stored on one or more corresponding portable computer-readable tangible storage devices 936, can be read via a corresponding R / W drive or interface 832, and can be loaded into a corresponding hard disk drive 830.

[0072] Each set of internal components 800A, 800B also includes a network adapter or interface 836, such as a TCP / IP adapter card, a wireless Wi-Fi interface card, or a 3G, 4G or 5G wireless interface card or other wired or wireless communication link. Software Program 108 ( Figure 1 ) and server computer 114 ( Figure 1 ) on zero pronoun recovery program 116 ( Figure 1 ) can be downloaded from an external computer to the computer 102 via a network (e.g., the Internet, a local area network, or other wide area network) and a corresponding network adapter or interface 836 ( Figure 1 ) and server computer 114. Software program 108 and zero pronoun recovery program 116 on server computer 114 are loaded from network adapter or interface 836 into respective hard disk drives 830. The network may include copper wire, fiber optics, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers.

[0073] Each set of external components 900A, 900B may include a computer display monitor 920, a keyboard 930, and a computer mouse 934. External components 900A, 900B may also include a touch screen, a virtual keyboard, a touch pad, a pointing device, and other human interface devices. Each set of internal components 800A, 800B also includes a device driver 840 that interfaces with the computer display monitor 920, the keyboard 930, and the computer mouse 934. The device driver 840, the R / W driver or interface 832, and the network adapter or interface 836 include hardware and software (stored in the storage device 830 and / or ROM 824).

[0074] Some embodiments may involve systems, methods and / or computer-readable media at any possible level of integration technology detail.Computer-readable media may include a computer-readable non-transitory storage medium (or multiple media) having computer-readable program instructions thereon for causing a processor to perform operations.

[0075] A computer readable storage medium may be a tangible device capable of retaining and storing instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer readable storage media includes the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanical encoding device such as a punch card or a raised structure in a groove on which instructions are recorded, and any suitable combination of the foregoing. Computer-readable storage media as used herein should not be construed as being themselves transient signals, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted through wires.

[0076] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The network can include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network, and forwards the computer-readable program instructions to be stored in a computer-readable storage medium in the corresponding computing / processing device.

[0077] The computer readable program code / instructions for performing operations may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data of an integrated circuit system, or source code or object code written in any combination of one or more programming languages ​​(including object-oriented programming languages ​​such as Smalltalk, C++, etc. and procedural programming languages ​​such as "C" programming language or similar programming languages). The computer readable program instructions may be executed completely on the user's computer as a stand-alone software package, partially on the user's computer, partially on the user's computer and partially on a remote computer, or completely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, an electronic circuit system including, for example, a programmable logic circuit system, a field-programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to personalize the electronic circuit system so as to perform various aspects or operations.

[0078] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a processor of a special-purpose computer, or a processor of other programmable data processing devices to generate a mechanism, such that the instructions executed by the processor of the computer or other programmable data processing devices create a method for implementing the functions / actions specified in the flowchart and / or block diagram blocks or blocks. These computer-readable program instructions may also be stored in a computer-readable storage medium, which may direct the operation of a computer, a programmable data processing device, and / or other devices in a particular manner, such that the computer-readable storage medium having the instructions stored therein includes an article of manufacture, which includes instructions for implementing various aspects of the functions / actions specified in the flowchart and / or block diagram blocks or blocks.

[0079] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing device, or other apparatus, so that a series of operational steps to be performed on the computer, other programmable device, or other apparatus can produce a computer-implemented process, so that the instructions executed on the computer, other programmable device, or other apparatus implement the functions / actions specified in the flowchart and / or block diagram block or blocks.

[0080] The flow chart and block diagram in the figure show the architecture, function and operation of the possible implementation of the system, method and computer-readable medium according to various embodiments. In this regard, each block in the flow chart or block diagram can represent a part of a module, segment or instruction, and each block includes one or more executable instructions for realizing (one or more) specified logical functions. Compared with the block depicted in the figure, the method, computer system and computer-readable medium can include other blocks, fewer blocks, different blocks or blocks of different arrangements. In some alternative implementations, the function marked in the block may not occur in the order marked in the figure. For example, the two blocks shown in succession can actually be performed simultaneously or substantially simultaneously, or the block can sometimes be performed in reverse order according to the function involved. It will also be noted that each block in the block diagram and / or flow chart illustration and the combination of the blocks in the block diagram and / or flow chart illustration can be realized by a hardware-based dedicated system, which performs a specific function or action or realizes a combination of dedicated hardware and computer instructions.

[0081] It will be apparent that the systems and / or methods described herein can be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods does not limit the implementation. Therefore, the operation and behavior of the systems and / or methods are described herein without reference to specific software code - it should be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0082] Unless explicitly stated, the elements, actions or instructions used herein should not be interpreted as being critical or necessary. In addition, as used herein, the articles "one" and "an" are intended to include one or more items, and can be used interchangeably with "one or more". In addition, as used herein, the term "group" is intended to include one or more items (e.g., related items, unrelated items, combinations of related items and unrelated items, etc.), and can be used interchangeably with "one or more". In the case of meaning only one item, the term "one" or similar language is used. In addition, as used herein, the terms "have", "have", "contain" etc. are intended to be open terms. In addition, unless explicitly stated otherwise, the phrase "based on" is intended to mean "based at least in part on".

[0083] Descriptions of various aspects and embodiments have been presented for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Although combinations of features are described in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features can be combined in a manner that is not specifically described in the claims and / or disclosed in the specification. Although each dependent claim listed below can be directly based on only one claim, the disclosure of possible implementations includes each dependent claim combined with each other claim in the claim group. Without departing from the scope of the described embodiments, many modifications and changes will be obvious to those of ordinary skill in the art. The terms used herein are selected to best illustrate the principles of the embodiments, the practical application of the technology found on the market or the improvement of the technology, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for restoring omitted pronouns, characterized in that: The method comprises: receiving, by a computer, data corresponding to one or more input words; determining, by the computer, a contextual representation of the received input word data; confirming, by the computer, the omitted pronoun based on a probability value associated with the contextual representation; and determining, by the computer, a span associated with one or more of the received input characters, the span corresponding to the ones of the input characters that are referred to by the omitted pronoun; The context representation is determined by a Bidirectional Encoder Representation from Transformer (BERT) encoder, where the probability value is obtained by softmax (W rec [h i-1 ,h i ]+b rec ) determines that the i-1 corresponds to the first input word, the i corresponds to the second input word, the h corresponds to the context representation, and W rec and b rec Corresponds to the parameters of the linear classifier.

2. The method according to claim 1, characterized in that: The method also includes detecting, by the computer, that the omitted pronoun is omitted.

3. The method according to claim 2, characterized in that The omitted pronoun is detected using binary classification based on a second probability value associated with the context representation.

4. The method according to claim 2, characterized in that: The method further includes calculating a loss value based on the detection of the omitted pronoun, the confirmation of the omitted pronoun, and the determination of the span.

5. The method according to claim 1, characterized in that The method also includes appending, by the computer, a class token to the received input word data, wherein the appended class token corresponds to a start position associated with the received input word.

6. The method according to claim 5, characterized in that A span of zero is assigned to the appended class token.

7. The method according to claim 1, characterized in that The method further comprises: determining, by the computer, that data corresponding to the received input word includes an omitted pronoun; and A type associated with the omitted pronoun is determined by the computer.

8. The method according to claim 1, characterized in that: The span is determined based on a received concatenation of the one or more input words and based on a self-attention determination associated with a start position and an end position associated with the span.

9. The method according to claim 1, characterized in that: Confirming the omitted pronoun based on the probability value associated with the context representation includes confirming, based on the probability value, that the omitted pronoun will appear between two of the received input words.

10. A computer system, characterized in that: The computer system comprises: one or more computer-readable non-transitory storage media configured to store computer program code; and One or more computer processors, the one or more computer processors being configured to access the computer program code and operate as instructed by the computer program code to perform the method of any one of claims 1 to 9.

11. A device for restoring omitted pronouns, characterized in that: The device comprises: A receiving module, configured to receive data corresponding to one or more input words; A first determination module, configured to determine a context representation of the received input word data; a confirmation module for confirming the omitted pronoun based on a probability value associated with the context representation; and a second determination module for determining a span associated with one or more of the received input characters, the span corresponding to the input characters of the input characters referred to by the omitted pronoun; The context representation is determined by a Bidirectional Encoder Representation from Transformer (BERT) encoder, where the probability value is obtained by softmax (W rec [h i-1 ,h i ]+b rec ) determines that the i-1 corresponds to the first input word, the i corresponds to the second input word, the h corresponds to the context representation, and W rec and b rec Corresponds to the parameters of the linear classifier.

12. A non-transitory computer readable medium, characterized in that A computer program for restoring omitted pronouns is stored on the non-transitory computer readable medium, the computer program being configured to cause one or more computer processors to execute the method of any one of claims 1 to 9.

Images