Targeted portion re-enrichment of corpus based on NLP model augmentation

Abstract

A computer-implemented method includes determining that at least one natural language processing (NLP) request is associated with at least one surface form, the NLP request directed to a corpus, a database including pre-existing annotations associated with the corpus. An indexed query related to the at least one surface form is executed to generate an indexed query result, the indexed query result including an identification of a portion of the corpus affected by the NLP request. A scope of the NLP request related to the database is determined based on the indexed query result, the scope including an identification of affected candidate annotations of the pre-existing annotations affected by the NLP request. The NLP service is executed on the corpus according to the scope and the portion, resulting in an update. The update is submitted to the database associated with the corpus.

Description

Technical Field

[0001] This invention generally relates to computer systems, and more specifically, to the targeted enrichment of corpora based on natural language processing models enhanced on computer systems. Background Technology

[0002] Natural Language Processing (NLP) involves the interaction between computers and human (natural) language, and more specifically, how to program computers to process and analyze large amounts of natural language data. This natural language data is sometimes called a corpus. In linguistics, a corpus, or text corpus, is a language resource consisting of large and structured collections of text. NLP processing can occur on large corpora, resulting in numerous annotations associated with that corpus. Summary of the Invention

[0003] Embodiments of the present invention target the targeted re-enrichment of corpora augmented by natural language processing models on computer systems. A non-limiting example computer-implemented method includes determining that at least one natural language processing (NLP) request is associated with at least one surface form, the NLP request targeting a corpus comprising pre-existing annotations associated with the corpus. The method includes performing an index query associated with the at least one surface form to generate an index query result including identifiers of portions of the corpus affected by the NLP request. The method includes determining, based on the index query result, the scope of the NLP request associated with the database, the scope including identifiers of affected candidate annotations of the pre-existing annotations affected by the NLP request. Furthermore, the method includes causing an NLP service to perform on the corpus according to the scope and portions, resulting in an update, and submitting the update to the database associated with the corpus.

[0004] In addition to one or more features described above or below, or as an alternative, other embodiments may include displaying a preview of the update before submitting the update to the database.

[0005] In addition to one or more features described above or below, or as an alternative, other embodiments may include a corpus comprising the at least one surface form.

[0006] In addition to one or more features described above or below, or as an alternative, other embodiments may include the identifiers of at least one or more documents from multiple documents in the corpus affected by the NLP request.

[0007] In addition to one or more features described above or below, or as an alternative, other embodiments may include portions of the corpus affected by the NLP request that include identifiers of at least one or more paragraphs in documents within the corpus.

[0008] In addition to one or more of the features described above or below, or as an alternative, other embodiments may include where the update includes new annotations.

[0009] In addition to one or more of the features described above or below, or as an alternative, other embodiments may include an update that includes new annotations and modifications to the affected candidate annotations.

[0010] In addition to one or more of the features described above or below, or as an alternative, other embodiments may include having the NLP service executed on the corpus according to the scope and the portion therein in a shorter amount of time than having the NLP service executed on the entire corpus.

[0011] In addition to one or more of the features described above or below, or as an alternative, other embodiments may include enabling unilateral provisioning of computing power to submit updates to a database associated with the corpus, wherein index queries and range determination are performed.

[0012] Other embodiments of the present invention implement the features of the above-described method in computer systems and computer program products.

[0013] Additional technical features and benefits are achieved through the technology of this invention. Embodiments and aspects of the invention are described in detail herein and are considered part of the claimed subject matter. For a better understanding, refer to the detailed description and accompanying drawings. Attached Figure Description

[0014] The details of the proprietary rights described herein are specifically pointed out and clearly claimed in the claims at the end of the specification. The foregoing and other features and advantages of embodiments of the invention will be apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0015] Figure 1 This is a block diagram of an example computer system used in conjunction with one or more embodiments of the present invention;

[0016] Figure 2 This is a block diagram of a system for targeted partial re-enrichment of a corpus based on natural language processing model enhancement, according to one or more embodiments of the present invention.

[0017] Figure 3 This is a flowchart of a process for targeted re-enrichment of a corpus based on natural language processing model enhancement according to one or more embodiments of the present invention;

[0018] Figure 4 This is a flowchart of a computer-implemented method for targeted portion re-enrichment of a corpus according to one or more embodiments of the present invention;

[0019] Figure 5 A cloud computing environment according to one or more embodiments of the present invention is described; and

[0020] Figure 6 An abstract model layer is described according to one or more embodiments of the present invention. Detailed Implementation

[0021] One or more embodiments of the present invention provide a system and method for targeted partial re-enrichment of a corpus based on one or more enhancements to an NLP model. In one or more embodiments, re-enrichment can occur on a target set of paragraphs rather than the entire corpus, when the NLP model enhancements can be traced back to surface forms in the text of the corpus. The surface form of a word is the form of a word appearing in the text of the corpus. Re-enriching the entire corpus to take advantage of the enhancements to the NLP model is an expensive computational process that can take a significant amount of time, such as several days, if the corpus contains many documents (e.g., hundreds, thousands, or millions of documents). Instead of processing the entire corpus, one or more embodiments facilitate targeted enrichment of a corpus based on NLP edits that can be traced back to surface forms within the corpus, such as adding / removing surface forms from a dictionary and / or modifying annotations built on top of other annotations (attributes). When employing targeted enrichment techniques, one or more embodiments provide the option to preview the effects of cartridge enhancements on a corpus without actually applying the re-enriched data, until a domain expert (e.g., an operator) is satisfied with the NLP edits and expects the corpus to be enriched. A cartridge is a collection of items from the entire set of deployable assets in an NLP pipeline. A cartridge may include and / or refer to the services to be used, the configurations of those services, dictionaries, filters, etc.

[0022] Now go to Figure 1Computer system 100 is generally illustrated according to one or more embodiments of the present invention. Computer system 100 may be an electronic computer framework comprising and / or employing any number and combination of computing devices and networks utilizing various communication technologies, as described herein. Computer system 100 can be easily scaled, extended, and modularized, with the ability to change to different services or reconfigure certain features independently of others. Computer system 100 may be, for example, a server, desktop computer, laptop computer, tablet computer, or smartphone. In some examples, computer system 100 may be a cloud computing node. Computer system 100 can be described in the general context of computer system executable instructions such as program modules executed by the computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc., that perform specific tasks or implement specific abstract data types. Computer system 100 can be practiced in a distributed cloud computing environment, where tasks are performed by remote processing devices linked via a communication network. In a distributed cloud computing environment, program modules may reside in local and remote computer system storage media, including memory storage devices.

[0023] like Figure 1 As shown, computer system 100 has one or more central processing units (CPUs) 101a, 101b, 101c, etc. (collectively or collectively referred to as processor 101). Processor 101 may be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. Processor 101 (also referred to as processing circuitry) is coupled to system memory 103 and various other components via system bus 102. System memory 103 may include read-only memory (ROM) 104 and random access memory (RAM) 105. ROM 104 is coupled to system bus 102 and may include a basic input / output system (BIOS) or its successor, such as a unified extensible firmware interface (UEFI), which controls certain basic functions of computer system 100. RAM is read-write memory coupled to system bus 102 for use by processor 101. System memory 103 provides temporary storage space for the operation of the instructions during operation. System memory 103 may include random access memory (RAM), read-only memory, flash memory, or any other suitable memory system.

[0024] Computer system 100 includes an input / output (I / O) adapter 106 and a communication adapter 107 coupled to a system bus 102. I / O adapter 106 may be a Small Computer System Interface (SCSI) adapter that communicates with a hard disk 108 and / or any other similar component. I / O adapter 106 and hard disk 108 are collectively referred to herein as mass storage device 110.

[0025] Software 111 for execution on computer system 100 may be stored in mass storage device 110. Mass storage device 110 is an example of a tangible storage medium readable by processor 101, wherein software 111 is stored as instructions executed by processor 101 to cause computer system 100 to operate, as described below with reference to the accompanying drawings. Examples of computer program products and the execution of such instructions are discussed in more detail here. Communication adapter 107 interconnects system bus 102 with network 112, which may be an external network, enabling computer system 100 to communicate with other such systems. In one embodiment, a portion of system memory 103 and mass storage device 110 jointly store an operating system, which may be a coordinating... Figure 1 The functions of the various components shown can be applied to any suitable operating system.

[0026] Additional input / output devices are shown connected to system bus 102 via display adapter 115 and interface adapter 116. In one embodiment, adapters 106, 107, 115, and 116 may be connected to one or more I / O buses connected to system bus 102 via an intermediate bus bridge (not shown). Display 119 (e.g., screen or display monitor) is connected to system bus 102 via display adapter 115, which may include a graphics controller and a video controller for improving performance in graphics-intensive applications. Keyboard 121, mouse 122, speakers 123, etc., may be interconnected to system bus 102 via interface adapter 116, which may include, for example, a super I / O chip integrating multiple device adapters into a single integrated circuit. Suitable I / O buses for connecting peripheral devices such as hard disk controllers, network adapters, and graphics adapters typically include common protocols such as Peripheral Component Interconnect (PCI) and Peripheral Component Interconnect Fast (PCIe). Therefore, as Figure 1 The computer system 100 configured includes processing capabilities in the form of a processor 101, storage capabilities including system memory 103 and mass storage device 110, input devices such as a keyboard 121 and a mouse 122, and output capabilities including a speaker 123 and a display 119.

[0027] In some embodiments, the communication adapter 107 can send data using any suitable interface or protocol, such as an Internet Small Computer System Interface (IPS). The network 112 can be a cellular network, radio network, wide area network (WAN), local area network (LAN), or the Internet. External computing devices can connect to the computer system 100 via the network 112. In some examples, the external computing device can be an external network server or a cloud computing node.

[0028] It should be understood that Figure 1 The block diagram is not intended to indicate that the computer system 100 includes Figure 1 All components shown. Conversely, computer system 100 may include... Figure 1 Any suitable fewer or additional components not shown herein (e.g., additional memory components, embedded controllers, modules, additional network interfaces, etc.). Furthermore, the embodiments described herein with respect to computer system 100 can be implemented with any suitable logic, wherein, in various embodiments, the logic referred to herein may include any suitable hardware (e.g., processor, embedded controller, or application-specific integrated circuit, etc.), software (e.g., applications, etc.), firmware, or any suitable combination of hardware, software, and firmware.

[0029] Figure 2 This is a block diagram of a system 200 for targeted partial re-enrichment of a corpus based on natural language processing model enhancement, according to one or more embodiments of the present invention. Figure 3 This is a flowchart of a process 300 for targeted partial re-enrichment of a corpus based on natural language processing model enhancements according to one or more embodiments of the present invention. Re-enriching the entire corpus to take advantage of NLP model enhancements is an expensive proposition in terms of time and computational resources. Given NLP model enhancements that can be traced back to the surface form in the text, according to one or more embodiments, re-enrichment can occur on a target set of paragraphs rather than the entire corpus. Figure 2 One or more computer systems 202 coupled to computer system 220 are depicted. Components of computer system 100 may be used and / or integrated into computer system 202 and computer system 220.

[0030] The following will refer to Figure 2 and Figure 3 describe Figure 3In process 300. In block 302, software application 204 on computer system 202 is configured to receive one or more natural language processing (NLP) requests 230. Software application 204 may be implemented as software 111 executing on one or more processors 101. Software application 204 may be used to receive NLP requests 230 from computer system 220. NLP request 230 includes NLP model enhancements traced back to surface forms in text in corpus 208. NLP model enhancements include updates associated with one or more surface forms of text in corpus 208. The surface form of a word is the form of a word appearing in the text in corpus 208. In one or more embodiments, NLP model enhancements traceable to specific surface forms within the corpus may be identified within a cognitive tool platform (e.g., via software application 204 and / or client application 222 of software application 204) when editing is performed and / or editing is sent to software application 204. Client application 222 may be implemented using software 111 configured to execute on one or more processors 101, such as Figure 1 As discussed in [the document]. Furthermore, software application 204 can determine the traceability of surface forms to text in corpus 208 via indicators or choices made by the creator of NLP request 230, and these indicators / choices are included in NLP request 230. Example NLP model enhancements may include adding and / or removing surface forms from a dictionary, adding and / or removing building block concepts from an attribute set, etc. Additionally, NLP model enhancements include any edits that can be linked back to (i.e., identified as) surface forms within corpus 208.

[0031] In box 304, software application 204 on computer system 202 is configured to perform an indexed corpus query in index 240 based on previous / modified surface form updates from NLP request 230, in order to index text (e.g., words) throughout corpus 208. Text, term, single, etc., are used interchangeably. All text in corpus 208 has previously been indexed in index 240, and the indexed corpus query searches for indexed text in index 240. Index 240 is a list of all text / words (i.e., surface forms) in corpus 208 along with their associated positions within corpus 208. For example, for each text / word in corpus 208, index 240 may include page number, paragraph, document name, line position, number of mentions of the text / word (surface form), etc. Index 240 is a database index or other search index (i.e., Lucene or Elasticsearch index). In this search-type application used for index 240, all relevant entities extracted by the NLP pipeline (or black box) have been indexed to allow search applications (including software application 204) to perform fast lookups. Indexed corpus queries return query results 242, and these query results 242 are stored in memory 206. Memory 206 can be implemented as mass storage device 110 or any other readable storage medium. Query results 242 include which surface forms of NLP model-enhanced text / words are found in corpus 208 and exactly where these surface forms are found (e.g., including page numbers, paragraphs, document names, line positions, etc.). As mentioned above, corpus 208 can include hundreds, thousands, and / or millions of documents, also referred to as “big data.” Query results 242 identify surface forms and specifically identify where they are located in corpus 208. In one or more embodiments, index 240 can be a Lucene-type index searched using Lucene-type corpus queries. Lucene is a reverse full-text index. This means it takes all documents, splits them into words, and then builds an index for each word. Because the index is an exact string match, queries can be very fast.

[0032] In box 306, software application 204 on computer system 202 is configured to use query results 242 to extract relevant ranges 244 of text / words from corpus 208, as determined by upstream / stacks linked to surface-form-modified NLP enhancements. Upstream / stacks refer to applications utilizing index 240 (e.g., software application 204 and / or applications invoked by software application 204). There might be an NLP process that only looks for two things (i.e., terms), disease and drug. Index 240 would include disease and drug annotations along with associated corpus documents. For example, using query results 242 for NLP model-modified text / words, software application 204 analyzes metadata / annotation database 210 to determine which pre-existing candidate annotations in metadata / annotation database 210 are affected by surface-form-modified NLP model-modified text / words linked to corpus 208. Affected candidate annotations are then backed up to surface forms and NLP requests 230 in corpus 208. Furthermore, using the query result 242 of the NLP model-enhanced text / words, the software application 204 determines which pre-existing candidate annotators in the annotator 250 are affected by the NLP model-enhanced text / words whose surface form has been modified and linked to the corpus 208. Affected candidate annotators may include annotators that were initially created in the metadata / annotation database 210, and those affected candidate annotators use the surface form of the text in the NLP request 230. Additionally, affected candidate annotators may be associated with other paragraphs, documents, and / or pages in the corpus 208 that are not directly and / or directly related to the surface form of other text (and / or concepts) in the NLP request 230, and therefore these other texts (and / or concepts) may not have been identified in earlier query results 242, but rather indirectly depend on the surface form of the text in the NLP request 230. Therefore, scope 244 may include: (1) identifiers of affected candidate annotations existing in metadata / annotation database 210 (where the affected candidate annotations may be directly and / or indirectly associated with the surface form of the text in NLP request 230 / query result 242); (2) identifiers in annotator 250 identifying affected candidate annotators (where the affected candidate annotators may be directly and / or indirectly associated with the surface form of the text in NLP request 230 / query result 242); (3) identifiers of query result 242 and / or identifiers of information in query result 242; and identifiers of the precise locations of which paragraphs and documents in corpus 208 need to be reprocessed or enriched (for the surface form of the text in NLP request 230 / query result 242). In other words, the identified candidate annotations and the identified candidate annotators are affected by changes made by NLP model enhancements.

[0033] In box 308, software application 204 on computer system 202 is configured to route the extracted range 244 of text / words augmented by the NLP model to the required NLP service in NLP service 212. Various types of NLP services 212 may be present on one or more computer systems 202. Some NLP services 212 may have different benefits or trade-offs. The NLP service may use annotator 250 to parse the text to find the meaning stored in metadata / annotation database 210. In box 310, software application 204 on computer system 202 is configured to cause NLP service 212 to perform (only) the portion of corpus 208 relevant to range 244. Range 244 details which specific portions or documents of corpus 208 need to be reprocessed without reprocessing the entire corpus 208, the location in the portion / document where reprocessing needs to occur, the specific affected candidate annotations in metadata / annotation database 210, and the affected candidate annotations in affected annotator 250. Therefore, 208 as a whole does not require reprocessing.

[0034] At box 312, software application 204 on computer system 202 is configured to receive output from NLP service 212 for scope 244 and use that output to perform a corpus reconciliation process to generate corpus update 246, resulting in corpus update 246 being temporarily stored in memory 206. Corpus update 246 includes new and / or updated annotations to be made in metadata / annotation database 210 and how these new and / or updated annotations change existing metadata / annotation 210. At box 314, update 246 may be previewed by an operator on a display screen (e.g., display 119) before the changes are applied to metadata / annotation database 210. After operator approval using software application 204 and / or if the preview is skipped, at box 316, software application 204 on computer system 202 is configured to apply / submit update 246 to metadata / annotation database 210 for scope 244 (only). If the operator does not approve update 246, the operator can modify the NLP enhancement (e.g., using software application 204), and the process returns to box 302.

[0035] The techniques and systems discussed herein offer various technical benefits. According to one or more embodiments, when ready to re-enrich the metadata / annotation database 210 with an update 246 corresponding to an enhanced NLP model, the NLP service reprocesses documents / paragraphs identified as affected by the NLP model enhancement and re-enriches those documents / paragraphs with the model update. Any previous annotations on those documents / paragraphs in corpus 208 (in the metadata / annotation database 210) can be truncated and / or replaced with annotations from the update 246 generated by the enhanced NLP model. If paragraph or document-level rules exist in the metadata / annotation database 210, those broader scopes can be re-enriched, not just sentences / paragraphs, or documents can be selectively re-enriched. When previewing the results of the NLP model enhancement associated with corpus 208 before re-enrichment, the re-enriched paragraphs are stored in memory 206 or persistently stored outside the metadata / annotation database 210 for previewing the changes that will occur in the metadata / annotation database 210 when enriching 208 with the enhanced model corpus. If enhancements produce undesirable results, operators can use this preview capability as a means to improve their enhancements. They can do this quickly without having to wait and / or need to re-enrich the entire corpus.

[0036] Figure 4This is a flowchart of a computer-implemented method 400 for re-enriching a target portion of a corpus 208 based on natural language processing model enhancement, according to one or more embodiments. At block 402, a software application 204 on computer system 202 is configured to determine that at least one natural language processing (NLP) request 230 is associated with at least one surface form, the NLP request 230 targeting corpus 208, the database including pre-existing annotations associated with corpus 208 (e.g., metadata / annotation database 210). At block 404, the software application 204 on computer system 202 is configured to perform an index query associated with the at least one surface form to generate an index query result 242, the index query result 242 including identifiers of the portions of corpus 208 affected by the NLP request 230. At box 406, software application 204 on computer system 202 is configured to determine, based on index query results 242, the scope 244 of NLP request 230 associated with a database (e.g., metadata / annotation database 210), including the identifiers of affected candidate annotations of previously existing annotations (e.g., in metadata / annotation database 210) affected by NLP request 230. At box 408, software application 204 on computer system 202 is configured to cause an NLP service (e.g., one of NLP services 212) to execute on corpus 208 according to scope 244 and portions (e.g., identified in index query results 242), resulting in update 246. At box 410, software application 204 on computer system 202 is configured to submit update 246 to the database associated with corpus 208 (e.g., metadata / annotation database 210).

[0037] Software application 204 on computer system 202 is configured to visually display a preview 246 of the update before submitting it to the database. Corpus 208 includes at least one surface form. A portion of corpus 208 affected by an NLP request includes identifiers (e.g., location identifiers) of at least one or more documents among a plurality of documents in corpus 208. A portion of corpus 208 affected by an NLP request includes identifiers (e.g., location identifiers) of at least one or more paragraphs among a document in corpus 208. Update 246 includes new annotations according to NLP request 230. Update 246 includes new annotations according to NLP request 230 and modifications to affected candidate annotations. Enabling an NLP service (e.g., NLP service 212) to execute on corpus 208 according to scope 244 occurs in a shorter amount of time compared to enabling an NLP service to execute on corpus 208 as a whole. Performing index queries and determining scope enables unilateral provisioning of computational power to submit updates to the database associated with the corpus.

[0038] It should be understood that although this disclosure includes a detailed description of cloud computing, the implementation of the teachings cited herein is not limited to cloud computing environments. Rather, embodiments of the invention can be implemented in conjunction with any other type of computing environment now known or developed hereafter.

[0039] Cloud computing is a service delivery model that provides convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing power, storage, applications, virtual machines, and services), which can be rapidly provisioned and released with minimal management effort or interaction with service providers. This cloud model may include at least five features, at least three service models, and at least four deployment models.

[0040] The features are as follows:

[0041] On-demand self-service: Cloud consumers can automatically and unilaterally provide computing power, such as server time and network storage, as needed, without requiring manual interaction with service providers.

[0042] Wide network access: The capability is available on the network and accessed through standard mechanisms that facilitate the use of heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0043] Resource pooling: pooling computing resources from providers to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated based on demand. Location independence is significant because consumers typically do not have control or knowledge of the exact location of the resources provided, but are able to specify a location at a higher level of abstraction (e.g., country, state, or data center).

[0044] Rapid flexibility: In some cases, capacity can be automatically, quickly, and flexibly provided to rapidly shrink and rapidly expand. For consumers, the capacity available for supply often appears unlimited and can be purchased in any quantity at any time.

[0045] Metering services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the service type (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both service providers and consumers.

[0046] The service model is as follows:

[0047] Software as a Service (SaaS): The capability offered to consumers is the ability to use the provider's applications running on cloud infrastructure. Applications can be accessed from various client devices through thin client interfaces such as web browsers (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, storage, or even individual application capabilities, with possible exceptions such as limited user-specific application configuration settings.

[0048] Platform as a Service (PaaS): This provides consumers with the ability to deploy applications created or acquired by the consumer onto cloud infrastructure, using programming languages ​​and tools supported by the provider. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage devices, but they have control over the deployed applications and the configuration of any application hosting environment.

[0049] Infrastructure as a Service (IaaS): The capabilities provided to consumers are processing, storage, networking, and other basic computing resources that enable consumers to deploy and run arbitrary software, which may include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but have control over the operating system, storage, deployed applications, and possibly limited control over selected networking components (e.g., host firewalls).

[0050] The deployment model is as follows:

[0051] Private cloud: A cloud infrastructure that operates for only one organization. It can be managed by the organization or a third party and can exist on-premises or externally.

[0052] Community cloud: Cloud infrastructure shared by several organizations and supporting a specific community with shared concerns (e.g., tasks, security requirements, policies, and compliance considerations). It can be managed by an organization or a third party and can exist on-premises or externally.

[0053] Public cloud: Cloud infrastructure available to the general public or large industrial groups and owned by organizations that sell cloud services.

[0054] Hybrid cloud: A cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain a single entity but are bound together by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursts for load balancing between clouds).

[0055] Cloud computing environments are service-oriented environments focused on statelessness, loose coupling, modularity, and semantic interoperability. At the heart of cloud computing is the infrastructure of a network of interconnected nodes.

[0056] Now for reference Figure 5The figure depicts an illustrative cloud computing environment 50. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10, and local computing devices used by cloud consumers (e.g., personal digital assistants (PDAs) or cellular phones 54A, desktop computers 54B, laptop computers 54C, and / or automotive computer systems 54N) that can communicate with these cloud computing nodes 10. The nodes 10 can communicate with each other. They can be physically or virtually grouped in one or more networks (not shown), such as private clouds, community clouds, public clouds, or hybrid clouds, or combinations thereof, as described above. This allows the cloud computing environment 50 to provide infrastructure, platforms, and / or software as services that cloud consumers do not need to maintain on their local computing devices. It should be understood that... Figure 5 The types of computing devices 54A-N shown are intended to be illustrative only, and computing node 10 and cloud computing environment 50 can communicate with any type of computerized device via any type of network and / or network-addressable connectivity (e.g., using a web browser).

[0057] Now for reference Figure 6 This demonstrates a cloud computing environment of 50 ( Figure 5 This provides a set of functional abstractions. It should be understood beforehand that... Figure 6 The components, layers, and functions shown are for illustrative purposes only, and embodiments of the present invention are not limited thereto. As shown in the figures, the following layers and corresponding functions are provided:

[0058] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include: a mainframe 61; a server 62 based on a RISC (Reduced Instruction Set Computer) architecture; a server 63; a blade server 64; a storage device 65; and network and networking components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0059] The virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual server 71; virtual storage 72; virtual network 73, including virtual private network; virtual application and operating system 74; and virtual client 75.

[0060] In one example, management layer 80 can provide the functions described below. Resource provisioning 81 provides dynamic acquisition of computing resources and other resources used to perform tasks within the cloud computing environment. Metering and pricing 82 provides cost tracking for utilizing resources within the cloud computing environment, as well as billing or invoicing the consumption of these resources. In one example, these resources may include application software licenses. Security provides authentication for cloud consumers and tasks, and protection for data and other resources. User portal 83 provides access to the cloud computing environment for consumers and system administrators. Service level management 84 provides cloud resource allocation and management to meet required service levels. Service level agreement (SLA) planning and fulfillment 85 provides pre-planning and procurement of cloud resources based on anticipated future demand for cloud resources according to the SLA.

[0061] Workload layer 90 provides examples of functionalities that can be utilized in a cloud computing environment. Examples of workloads and functionalities that can be provided from this layer include: mapping and navigation 91; software development and lifecycle management 92; virtual classroom education delivery 93; data analytics and processing 94; transaction processing 95; and software applications 96 (e.g., software application 204) implemented within workloads and functionalities. Furthermore, software applications can run with and / or integrate with resource provisioning 81.

[0062] Various embodiments of the invention are described herein with reference to the accompanying drawings. Alternative embodiments of the invention may be devised without departing from its scope. Various connections and positional relationships (e.g., above, below, adjacent, etc.) between elements are illustrated in the following description and drawings. Unless otherwise specified, these connections and / or positional relationships may be direct or indirect, and the invention is not intended to limit this aspect. Thus, coupling of entities may refer to direct or indirect coupling, and positional relationships between entities may be direct or indirect positional relationships. Furthermore, the various tasks and process steps described herein may be incorporated into a more comprehensive procedure or process with additional steps or functionality not described in detail herein.

[0063] One or more of the methods described herein can be implemented using any of the following techniques or a combination thereof, each of which is well known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs), programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc., with appropriate combinations of logic gates.

[0064] For the sake of brevity, conventional techniques associated with the manufacture and use of various aspects of the present invention may or may not be described in detail herein. In particular, various aspects of computing systems and specific computer programs for implementing the various technical features described herein are well known. Therefore, for the sake of brevity, only brief mentions of details of many conventional embodiments are made herein, or they are omitted entirely without providing well-known system and / or process details.

[0065] In some embodiments, various functions or actions may occur at a given location and / or be related to the operation of one or more devices or systems. In some embodiments, a portion of a given function or action may be performed at a first device or location, and the remainder of the function or action may be performed at one or more additional devices or locations.

[0066] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used herein, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that, when used in this specification, the terms “comprising” and / or “including” specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof.

[0067] All the means or steps plus functional elements in the following claims are intended to include corresponding structures, materials, actions, and equivalents for performing the function in combination with other claimed elements as specifically claimed. This disclosure has been presented for purposes of illustration and description, but it is not intended to be exhaustive or limited to the forms disclosed. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of this disclosure. The embodiments were chosen and described in order to best explain the principles and practical application of this disclosure and to enable others skilled in the art to understand the disclosure of various embodiments with various modifications suitable for the particular intended use.

[0068] The diagrams described herein are illustrative. Many variations may be made to the diagrams or steps (or operations) described herein without departing from the scope of this disclosure. For example, actions may be performed in a different order, or actions may be added, deleted, or modified. Furthermore, the term "coupling" describes the presence of a signal path between two elements and does not imply a direct connection between elements without any inserted elements / connections. All such variations are considered part of this disclosure.

[0069] The following definitions and abbreviations are used to interpret the claims and the specification. As used herein, the terms “comprising,” “including,” “containing,” “having,” “containing,” or any other variations thereof are intended to cover a non-exclusive inclusion. For example, a composition, mixture, process, method, article, or apparatus that comprises a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to such compositions, mixtures, processes, methods, articles, or apparatus.

[0070] Additionally, the term "exemplary" is used herein to mean "serving as an example, illustration, or description." Any embodiment or design described herein as "exemplary" need not be construed as being more preferred or advantageous than other embodiments or designs. The terms "at least one" and "one or more" should be understood to include any integer greater than or equal to 1, i.e., 1, 2, 3, 4, etc. The term "multiple" should be understood to include any integer greater than or equal to 2, i.e., 2, 3, 4, 5, etc. The term "connection" can include both indirect "connection" and direct "connection."

[0071] The terms “about,” “substantially,” “approximately,” and variations thereof are intended to include the degree of error associated with a measurement based on a specific quantity of equipment available at the time of filing of this application. For example, “about” may include a range of ±8%, 5%, or 2% of a given value.

[0072] This invention can be a system, method, and / or computer program product at any possible level of integration technical detail. A computer program product may include a computer-readable storage medium (or medium) having computer-readable program instructions thereon for causing a processor to execute aspects of the invention.

[0073] Computer-readable storage media can be tangible devices capable of retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: a portable computer disk, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory stick, floppy disk, mechanically encoded devices such as punched cards or raised structures in slots on which instructions are recorded, and any suitable combination of the foregoing. Computer-readable storage media as used herein should not be construed as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0074] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device, or downloaded via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network) to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the respective computing / processing device.

[0075] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or connected to an external computer (e.g., via the Internet through an Internet service provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may execute computer-readable program instructions by personalizing the electronic circuitry with state information from the computer-readable program instructions in order to perform aspects of this invention.

[0076] This document describes aspects of the invention with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0077] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more boxes of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can instruct a computer, programmable data processing apparatus, and / or other apparatus to function in a particular manner, such that the computer-readable storage medium in which the instructions are stored includes an article of writing comprising instructions for implementing aspects of the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0078] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device, thereby producing a computer-implemented process, such that the instructions that execute on the computer, other programmable apparatus or other device implement the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0079] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than shown in the figures. For example, depending on the function involved, two consecutively shown blocks may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order. It will also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0080] Various embodiments of the invention have been described for illustrative purposes, but these descriptions are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein is chosen to best explain the principles of the embodiments, practical application, or technical improvements to existing technologies in the market, or to enable those skilled in the art to understand the embodiments described herein.

Claims

1. A computer-implemented method, comprising: Determine at least one natural language processing request associated with at least one surface form, the natural language processing request being directed to a corpus including pre-existing annotations associated with the corpus, the natural language processing request including model enhancements of the at least one surface form traced back to text in the corpus; Execute an index query related to the at least one surface form to generate an index query result, the index query result including an identifier of the portion of the corpus affected by the natural language processing request; Based on the index query results, the scope of the natural language processing request related to the database is determined, and the scope includes the identifiers of the pre-existing annotations affected by the natural language processing request. According to the scope and the portion, the natural language processing service is executed on the portion of the corpus, resulting in an update that modifies the affected candidate annotations by reprocessing the portion of the corpus; as well as Submit the update to the database associated with the corpus.

2. The computer-implemented method according to claim 1 further includes: A preview of the update is displayed before it is submitted to the database.

3. The computer-implemented method according to claim 1, wherein the corpus includes the at least one surface form.

4. The computer-implemented method of claim 1, wherein the portion of the corpus affected by the natural language processing request includes identifiers of at least one or more documents from a plurality of documents in the corpus.

5. The computer-implemented method of claim 1, wherein the portion of the corpus affected by the natural language processing request includes identifiers of at least one or more paragraphs in documents within the corpus.

6. The computer-implemented method of claim 1, wherein executing the natural language processing service on the corpus according to the scope and the portion occurs in a shorter amount of time than executing the natural language processing service on the entire corpus.

7. The computer-implemented method of claim 1, wherein performing the index query and determining the range enables unilateral provisioning of computing power for submitting the update to a database associated with the corpus.

8. A computer system, comprising: Memory, containing computer-readable instructions; as well as One or more processors are configured to execute the computer-readable instructions, the computer-readable instructions controlling the one or more processors to perform operations, the operations including: Determine at least one natural language processing request associated with at least one surface form, the natural language processing request being directed to a corpus including pre-existing annotations associated with the corpus, the natural language processing request including model enhancements of the at least one surface form traced back to text in the corpus; Execute an index query related to the at least one surface form to generate an index query result, the index query result including an identifier of the portion of the corpus affected by the natural language processing request; Based on the index query results, the scope of the natural language processing request related to the database is determined, and the scope includes the identifiers of the pre-existing annotations affected by the natural language processing request. According to the scope and the portion, the natural language processing service is executed on the portion of the corpus, resulting in an update that modifies the affected candidate annotations by reprocessing the portion of the corpus; and Submit the update to the database associated with the corpus.

9. The computer system according to claim 8, further comprising: A preview of the update is displayed before it is submitted to the database.

10. The computer system of claim 8, wherein the corpus includes the at least one surface form.

11. The computer system of claim 8, wherein the portion of the corpus affected by the natural language processing request includes identifiers of at least one or more documents from a plurality of documents in the corpus.

12. The computer system of claim 8, wherein the portion of the corpus affected by the natural language processing request includes identifiers of at least one or more paragraphs in documents within the corpus.

13. The computer system of claim 8, wherein executing the natural language processing service on the corpus according to the scope and the portion occurs in a shorter amount of time than executing the natural language processing service on the entire corpus.

14. A computer program product comprising program instructions executable by a processor to cause the processor to perform operations, the operations including: Determine at least one natural language processing request associated with at least one surface form, the natural language processing request being directed to a corpus including pre-existing annotations associated with the corpus, the natural language processing request including model enhancements of the at least one surface form traced back to text in the corpus; Execute an index query related to the at least one surface form to generate an index query result, the index query result including an identifier of the portion of the corpus affected by the natural language processing request; Based on the index query results, the scope of the natural language processing request related to the database is determined, and the scope includes the identifiers of the pre-existing annotations affected by the natural language processing request. According to the scope and the portion, the natural language processing service is executed on the portion of the corpus, resulting in an update that modifies the affected candidate annotations by reprocessing the portion of the corpus; as well as Submit the update to the database associated with the corpus.

15. The computer program product according to claim 14, further comprising: A preview of the update is displayed before it is submitted to the database.

16. The computer program product of claim 14, wherein the corpus includes the at least one surface form.

17. The computer program product of claim 14, wherein the portion of the corpus affected by the natural language processing request includes identifiers of at least one or more documents from a plurality of documents in the corpus.

Images