Clustering and Dynamic Re-clustering of Similar Text Documents

Abstract

To efficiently update and re-generate a machine-learning (ML) clustering model based on a training set of text records.SOLUTION: A system includes a processor circuit and at least one non-transitory memory device which stores instructions that cause a processor circuit to execute operations including: receiving a first text record; determining a first location of the first text record in a vector space on the basis of a time of generation and a time of resolution of the first text record using an ML clustering model; determining a distance between the first location of the first text record in the vector space and a second location of a specific cluster of a text record in the vector space; and determining whether the first text record is related to the specific cluster on the basis of the determined distance between the first location of the first text record and the second location of the specific cluster.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] [CROSS REFERENCE TO RELATED APPLICATIONS] This application claims the benefit of U.S. Non-Provisional Patent Application No. 62 / 843,007, filed May 3, 2019, the contents of which are incorporated by reference in their entirety. [Background technology]

[0002] Clustering text samples into similar groups is beneficial in a variety of applications. This may be done to facilitate text search by identifying interrelated groups of text samples within a corpus of text samples. Such grouping may facilitate text search or other text-related processing by limiting searches to text samples within a cluster of related text samples (e.g., a cluster of text samples similar to a query text string), avoiding the time and computational expense of searching through text records that are not within the cluster of related text samples. Also, identifying such clusters within text portions of records that contain non-text information (including, for example, geographic information, timestamps, weather-related information, information related to the performance or operation of a technical system) may facilitate operations based on the non-text information.

[0003] Related techniques include, for example, a machine learning classifier that predicts the category to which an observation belongs from a set of i discrete categories based on a training set of j observations whose membership in the correct categories is known (see, for example, Patent Document 1). Another related technique involves training a machine learning model to determine the similarity between text records, and using the trained machine learning model to determine the similarity based on the distance between vectors (see, for example, Patent Document 2). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] US Patent Application Publication No. 2019 / 0102682 [Patent Document 2] US Patent Application Publication No. 2017 / 0243112 Summary of the Invention [Means for solving the problem]

[0005] Natural language processing or other methods may be used to generate clusters of text samples and, for each text sample, to identify one or more clusters to which the individual text sample should be assigned. This may be done based on determined similarities between text samples and / or between the text sample and text samples within the cluster. This clustering may be done to facilitate searching within the text samples (e.g., by limiting a sample-level search to text samples within the identified cluster that are relevant to the search query), to facilitate modeling of managed networks whose operations are associated with the text sample, to identify and / or automatically implement a course of action that would have been effective if implemented in response to other text samples assigned to the cluster, to resolve or otherwise operate as a group on all incident reports associated with text samples within a particular cluster, or to provide other benefits.

[0006] In one possible example, a query may include a sample of text that describes a problem a user is experiencing. A machine learning (ML) model or some other algorithm may then be applied to assign the query to a cluster of other textual records similar to the query (e.g., other queries, incident reports containing the query and / or its solution, knowledge base articles). Information related to the selected cluster may then be provided to the user and / or technician. Such information may include knowledge base articles, similar resolved incident reports, or some other information that may provide a solution and / or answer to the query. Additionally or alternatively, similar queries or other information within the selected cluster may be provided to facilitate determining a solution and / or answering the query. This clustering may improve the quality of results provided to the user and / or reduce the amount of time a user spends before determining and implementing a solution to the problem represented by the query.

[0007] However, implementing such clustering can be computationally expensive, requiring large processor cycles, memory, or other resources to generate an ML clustering model based on a training corpus of text records. Therefore, the ML model may be regenerated based on an updated training set of text records according to a regular schedule. A user may manually designate clusters that have value, and the designated clusters may be retained through regeneration of the ML model to maintain the benefits of the designated clusters. For example, the identified clusters may correspond to ongoing events and / or classes of incidents that the user may continue to experience. Retaining the designated clusters may include defining one of the clusters in the regenerated ML model based on the designated clusters. For example, the clusters in the regenerated ML model may be defined based on identifying training text samples that belong to the designated clusters, based on centroids or other information that define the designated clusters, or based on some other specified feature. Additionally or alternatively, retaining the designated clusters may include using information about the designated clusters as seeds for regeneration of the ML model.

[0008] Between scheduled regenerations of the ML model, additional text records (e.g., newly generated incident reports) may accumulate. These additional records may belong to additional clusters not reflected in the ML model and may therefore be incorrectly assigned to clusters in the ML model and / or assigned to a set of residual text records. To avoid the computational cost of completely regenerating the ML model to include the additional text records, the ML model may be updated via an iterative update algorithm to take into account the information represented by the additional text records.

[0009] This may include attempting to form additional clusters of text records from within the set of residual text records. When a new text record is received, it may be compared to the existing clusters. If the new text record does not fit into any of the existing clusters (e.g., by being less similar to each of the clusters than a specified threshold similarity), the new text record may be assigned to the set of residual text records. Thus, the set of residual text records will include text records determined by the ML model not to belong to any of the clusters defined by the ML model. If one or more additional clusters are identified within the residual, the ML model may be updated to include the additional clusters. Additionally, any text records from the set of residual text records that correspond to the identified additional clusters may be reassigned from the residual set to the additional clusters.

[0010] In some examples, a user may manually specify a priori conditions under which cluster formation should occur. Such manually specified conditions may be specified according to geographic location, tags or other metadata, a personal identifier associated with the text record (e.g., the user who generated the incident report that forms part of the text record), or according to some other consideration. Additional clusters may then be identified within the user-specified conditions. This manual pre-clustering may provide various advantages. For example, a clustering algorithm may perform more efficiently by operating on fewer text records. These methods may also provide advantages by aligning the identified clusters with organizational considerations so that the generated clusters are aligned with those considerations. Additionally, the clusters generated within the user-specified conditions may be improved due to the pre-partitioning enabled by the user-specified conditions.

[0011] Transmitting a representation of one or more text records in a corpus of text records may include transmitting a variety of different information. In some examples, transmitting a representation of a text record includes transmitting an identification number (e.g., a globally unique identifier (GUID)), a date and time stamp, a location in a database, or some other identifying information that may enable an end user instance to identify or otherwise access the represented text record in a database. Additionally or alternatively, transmitting a representation of a text record may include transmitting a copy of the text record itself or a portion thereof. For example, transmitting a representation of a text record may include transmitting a copy of a “problem resolution” field of an incident report.

[0012] Transmitting a representation of one or more text records in a corpus of text records may include transmitting a representation of one or more clusters of text records. This may include transmitting an identification number (e.g., a GUID), a location in a database, or some other identifying information that may enable an end user instance to identify the represented cluster of text records. Additionally or alternatively, a representation of the content of a cluster of text records may be transmitted. This may include transmitting identifying information (e.g., a GUID, a location in a database) for the text records in the cluster, all or part of the content of the text records in the cluster (e.g., text from the “problem resolution” field of an incident report in the cluster), the content of one or more representative text records associated with the cluster (e.g., a knowledge base article associated with the cluster, the “problem resolution” field, or other content of an incident report near the centroid of the cluster), or some other content associated with the cluster.

[0013] It should be noted that the methods described herein for determining similarities between text records and performing clustering based on such determined similarities may be applied to other types of records. For example, textual aspects (e.g., text fields of an incident report describing the problem, a solution to the problem, or some other information) and non-textual aspects (e.g., dates, geographic locations, listed categories, identifiers associated with users or technicians, or other non-textual information in an incident report) may be clustered by an ML model based on both the textual and non-textual information. Indeed, the methods described herein may be applied to determining numerical similarities and / or performing clustering based on such determined numerical similarities for entirely non-textual records (e.g., a combination of genotype / phenotype information for individuals receiving medical treatment, historical usage statistics for users of media or network services, etc.).

[0014] Thus, a first exemplary embodiment includes: (i) receiving a first text record by a prediction computing instance and from an end-user computing instance, the end-user computing instance being dedicated to a managed network and both the prediction computing instance and the end-user computing instance being deployed within a remote network management platform; (ii) determining, by an ML clustering model of the prediction computing instance representing a set of clusters of text records, that the first text record corresponds to a particular cluster within the set of clusters of text records; (iii) transmitting a representation of the particular cluster by the prediction computing instance and to the end-user computing instance; and (iv) receiving a second text record by the prediction computing instance and from the end-user computing instance. (v) determining, by the ML clustering model, that the second text record does not correspond to any cluster of text records of the set of clusters of text records; (vi) in response to determining that the second text record does not correspond to any cluster of text records of the set of clusters of text records, adding, by the prediction computing instance, the second text record to the stored set of residual text records; (vii) identifying, by the prediction computing instance, an additional cluster of text records based on the stored set of residual text records; and (viii) transmitting, by the prediction computing instance and to the end-user computing instance, a representation of the additional cluster of text records.

[0015] In a second exemplary embodiment, a computer-implemented method includes: (i) receiving a first plurality of text records; (ii) determining, based on the plurality of text records, a first ML clustering model representing a first set of clusters of text records, wherein each cluster of the first ML clustering model corresponds to a distinct set of text records in the first plurality of text records; (iii) receiving additional text records; (iv) using the additional text records to update the first ML clustering model via an iterative update process; (v) receiving an indication of a preferred cluster of the first set of clusters of text records represented by the first ML clustering model; (vi) determining that model refresh criteria are met; and (vii) in response to determining that the model refresh criteria are met, determining, based on the additional text records, a second ML clustering model representing a second set of clusters of text records, wherein each cluster of the second ML clustering model corresponds to a distinct set of text records in the additional text records, and wherein a first cluster in the second set of clusters of text records of the second ML clustering model is determined based on the preferred cluster.

[0016] In a third exemplary embodiment, a computer-implemented method includes: (i) obtaining a plurality of text records divided into clusters and a residual set of text records, wherein an ML clustering model has divided the plurality of text records based on a similarity metric; (ii) receiving, from a client device, a particular text record representing a query; (iii) determining, via the ML clustering model and based on the similarity metric, that the particular text record does not fit into any of the clusters; and (iv) in response to determining that the particular text record does not fit into any of the clusters, adding the particular text record to the residual set of text records.

[0017] In a fourth exemplary embodiment, an article of manufacture may include a non-transitory computer-readable medium having stored thereon program instructions that, when executed by a computing system, cause the computing system to perform operations in accordance with the first, second, or third exemplary embodiments.

[0018] In a fifth exemplary embodiment, a computing system may include at least one processor, as well as memory and program instructions that may be stored in the memory and that, when executed by the at least one processor, may cause the computing system to perform operations according to the first, second, or third exemplary embodiments.

[0019] In the sixth exemplary embodiment, the system may include various means for performing each of the operations of the first, second, or third exemplary embodiments.

[0020] These and other embodiments, aspects, advantages, and alternatives will become apparent to those skilled in the art upon reading the following detailed description, with appropriate reference to the accompanying drawings. Moreover, this Summary of the Invention, as well as the other descriptions and figures provided herein, are intended to illustrate embodiments by way of example only, and, as such, many variations are possible. Illustratively, structural elements and process steps may be rearranged, combined, distributed, omitted, or otherwise modified while remaining within the scope of the embodiments as claimed. [Brief explanation of the drawings]

[0021] [Figure 1] 1 illustrates a schematic diagram of a computing device in accordance with an illustrative embodiment. [Figure 2] 1 illustrates a schematic diagram of a server device cluster in accordance with an example embodiment. [Figure 3] 1 depicts a remote network management architecture in accordance with an example embodiment. [Figure 4] 1 depicts a communication environment including a remote network management architecture in accordance with an example embodiment. [Figure 5A] 1 depicts another communication environment including a remote network management architecture in accordance with an example embodiment. [Figure 5B] 1 is a flowchart according to an example embodiment. [Figure 6] 1 depicts an incident report in accordance with an exemplary embodiment; [Figure 7] 1 depicts a database query architecture in accordance with an example embodiment. [Figure 8A] 1 depicts an artificial neural network (ANN) configured to learn the contextual meaning of words, according to an exemplary embodiment. [Figure 8B] 9B depicts a set of training data for the ANN of FIG. 9A in accordance with an exemplary embodiment. [Figure 9A] 1 depicts an artificial neural network (ANN) configured to learn the contextual meaning of words, according to an exemplary embodiment. [Figure 9B] 9B depicts a set of training data for the ANN of FIG. 9A in accordance with an exemplary embodiment. [Figure 10A] 1 depicts training an ANN on paragraph vectors according to an exemplary embodiment. [Figure 10B] 1 depicts training an ANN on paragraph vectors according to an exemplary embodiment. [Figure 10C] 1 depicts training an ANN on paragraph vectors according to an exemplary embodiment. [Figure 10D] 1 depicts using a trained ANN to determine paragraph vectors for previously unseen paragraphs, according to an exemplary embodiment. [Figure 11A] 1 illustrates the location of a record in two-dimensional space according to an example embodiment. [Figure 11B] 11B depicts the records of FIG. 11A grouped into clusters, according to an example embodiment. [Figure 11C]11B depicts the records of FIG. 11A grouped into clusters, according to an example embodiment. [Figure 11D] 11B depicts the records of FIG. 11A grouped into clusters, according to an example embodiment. [Figure 11E] 11B depicts the records of FIG. 11A grouped into clusters, according to an example embodiment. [Figure 11F] 11B depicts the records of FIG. 11A grouped into clusters, according to an example embodiment. [Figure 12A] 1 depicts the location of a record in two-dimensional space, according to an exemplary embodiment. [Figure 12B] 12B depicts the records of FIG. 12A grouped into clusters, where some of the records do not belong to any of the clusters, according to an example embodiment. [Figure 12C] 12B depicts the records of FIG. 12A grouped into clusters, where some of the records do not belong to any of the clusters, and an additional set of records, according to an example embodiment. [Figure 12D] 12D depicts the records of FIG. 12C grouped into clusters, where some of the records do not belong to any of the clusters, according to an example embodiment. [Figure 13] 1 is a flowchart according to an example embodiment. [Figure 14] 1 is a flowchart according to an example embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0022] Exemplary methods, devices, and systems are described herein. It should be understood that, as used herein, the words "example" and "exemplary" are used to mean "serving as an example, instance, or illustration." Any embodiment or feature described herein as "example" or "exemplary" should not necessarily be construed as preferred or advantageous over other embodiments or features, unless so stated. Accordingly, other embodiments may be utilized, and other changes may be made, without departing from the scope of the subject matter presented herein.

[0023] Accordingly, the exemplary embodiments described herein are not intended to be limiting. It will be readily understood that aspects of the present disclosure as generally described herein and illustrated in the Figures can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations. For example, the separation of mechanisms into "client" and "server" components can occur in a variety of ways.

[0024] Furthermore, unless the context suggests otherwise, the features shown in each of the figures may be used in combination with one another. Thus, the figures should generally be viewed as component aspects of one or more overall embodiments, with the understanding that not all illustrated features are required for each embodiment.

[0025] Additionally, any listing of elements, blocks, or steps in this specification or claims is for purposes of clarity and, therefore, should not be construed as requiring or implying that these elements, blocks, or steps conform to a particular arrangement or be performed in a particular order.

[0026] I. Introduction Large enterprises are complex entities with many interrelated operations. Some of these can be found throughout the enterprise, such as human resources (HR), supply chain, information technology (IT), finance, etc. However, each enterprise also has unique operations that provide essential capabilities and / or create competitive advantage.

[0027] To support widely implemented operations, enterprises typically use off-the-shelf software applications, such as customer relationship management (CRM) and human capital management (HCM) packages. However, they may also require custom software applications to meet their unique requirements. Large enterprises often have dozens or even hundreds of these custom software applications. Nevertheless, the advantages provided by embodiments herein are not limited to large enterprises and may be applicable to enterprises of any size or any other type of organization.

[0028] Many such software applications are developed by individual departments within an enterprise. These range from simple spreadsheets to custom-built software tools and databases. However, the proliferation of siloed, custom software applications has many drawbacks. It negatively impacts an enterprise's ability to operate and grow its operations, innovate, and meet regulatory requirements. Enterprises can find it difficult to consolidate, streamline, and enhance their operations due to the lack of a single system that integrates their subsystems and data.

[0029] To efficiently create custom applications, businesses would benefit from a remotely hosted application platform that eliminates unnecessary development complexity. The goal of such a platform would be to reduce time-consuming, repetitive application development tasks so that software engineers and individuals in other roles can focus on developing unique, high-value features.

[0030] To achieve this goal, the concept of Application Platform as a Service (aPaaS) is introduced to intelligently automate workflows across an enterprise. An aPaaS system is hosted remotely from the enterprise but can access data, applications, and services within the enterprise via a secure connection. Such an aPaaS system can have several advantageous capabilities and features that can improve enterprise operations and workflows for IT, HR, CRM, customer service, application development, and security.

[0031] aPaaS systems can support the development and execution of Model-View-Controller (MVC) applications. MVC applications divide their functionality into three interconnected parts (model, view, and controller) to separate the representation of information from the way that information is presented to the user, thereby enabling efficient code reuse and parallel development. These applications can be web-based and can provide create, read, update, and delete (CRUD) functionality. This allows new applications to be built on a common application infrastructure.

[0032] An aPaaS system may support standardized application components, such as a standardized set of widgets for graphical user interface (GUI) development. In this way, applications built using an aPaaS system have a common look and feel. Other software components and modules may also be standardized. In some cases, this look and feel may be branded or skinned with a company's custom logo and / or color scheme.

[0033] aPaaS systems may support the ability to configure application behavior using metadata, allowing application behavior to be quickly adapted to specific needs. Such an approach reduces development time and increases flexibility. Additionally, aPaaS systems may support GUI tools that facilitate the creation and management of metadata, thus reducing metadata errors.

[0034] An aPaaS system may support well-defined interfaces between applications so that software developers can avoid unwanted inter-application dependencies. Accordingly, an aPaaS system may implement a service layer where persistent state information and other data is stored.

[0035] An aPaaS system may support a rich set of integration mechanisms so that applications running on it can interact with traditional and third-party applications. As an illustration, an aPaaS system may support a custom employee onboarding system that integrates with traditional HR, IT, and accounting systems.

[0036] An aPaaS system may support enterprise-grade security. Additionally, because an aPaaS system may be hosted remotely, it should utilize security procedures when interacting with systems within the enterprise or with third-party networks and services hosted outside the enterprise. For example, an aPaaS system may be configured to share data between the enterprise and other parties to detect and identify common security threats.

[0037] There may be other features, functions, and advantages of aPaaS systems; this description is for purposes of example and is not intended to be limiting.

[0038] As an example of an aPaaS development process, a software developer may be tasked with creating a new application using an aPaaS system. First, the developer may define a data model that identifies the types of data the application will use and the relationships between them. Then, through the aPaaS system's GUI, the developer inputs (e.g., uploads) the data model. The aPaaS system automatically creates all of the corresponding database tables, fields, and relationships, which can then be accessed through an object-oriented services layer.

[0039] The aPaaS system may also build a fully functional MVC application with a client-side interface and server-side CRUD logic. This generated application may serve as a basis for further development for the user. Advantageously, developers do not need to spend much time on basic application functionality. Furthermore, because the application may be web-based, it may be accessible from any internet-enabled client device. Alternatively or additionally, it may be possible to access a local copy of the application, for example, if internet service is not available.

[0040] aPaaS systems may also support a rich set of predefined functionality that can be added to applications. These mechanisms include support for search, email, templates, workflow design, reporting, analytics, social media, scripting, mobile-friendly output, and customized GUIs.

[0041] The following embodiments describe architectural and functional aspects of an exemplary aPaaS system, as well as its mechanisms and advantages.

[0042] II. Examples of Computing Devices and Cloud-Based Computing Environments 1 is a simplified block diagram illustrating a computing device 100 and describes some of the components that may be included within a computing device arranged to operate in accordance with embodiments herein. Computing device 100 may be a client device (e.g., a device actively operated by a user), a server device (e.g., a device that provides computing services to client devices), or some other type of computing platform. Some server devices may occasionally act as client devices to perform certain operations, and some client devices may incorporate server functionality.

[0043] In this example, computing device 100 includes a processor 102, memory 104, a network interface 106, and input / output units 108, all of which may be coupled by a system bus 110 or similar mechanism. In some embodiments, computing device 100 may include other components and / or peripheral devices (e.g., removable storage, printers, etc.).

[0044] The processor 102 may be one or more of a central processing unit (CPU), a coprocessor (e.g., a mathematical, graphics, or encryption coprocessor), a digital signal processor (DSP), a network processor, and / or any type of computer processing element, such as an integrated circuit or controller, that performs the operations of a processor. In some cases, the processor 102 may be one or more single-core processors. In other cases, the processor 102 may be one or more multi-core processors with multiple independent processing units. The processor 102 may also include register memory for temporarily storing instructions and associated data being executed, and cache memory for temporarily storing recently used instructions and data.

[0045] Memory 104 may be any form of computer-usable memory, including, but not limited to, random access memory (RAM), read-only memory (ROM), and non-volatile memory (e.g., flash memory, hard disk drives, solid-state drives, compact discs (CDs), digital video discs (DVDs), and / or tape storage). Memory 104 thus represents both a main memory unit and long-term storage. Other types of memory may include bio-memory.

[0046] Memory 104 may store program instructions and / or data on which the program instructions may operate. By way of example, memory 104 may store these program instructions on a non-transitory computer-readable medium such that the instructions are executable by processor 102 to perform any of the methods, processes, or operations disclosed in this specification or the accompanying drawings.

[0047] As shown in FIG. 1 , memory 104 may include firmware 104A, kernel 104B, and / or applications 104C. Firmware 104A may be program code used to boot or otherwise start some or all of computing device 100. Kernel 104B may be an operating system including modules for memory management, process scheduling and management, input / output, and communications. Kernel 104B may also include device drivers that allow the operating system to communicate with hardware modules (e.g., memory units, network interfaces, ports, and buses) of computing device 100. Applications 104C may be one or more user-space software programs, such as a web browser or email client, and any software libraries used by these programs. Memory 104 may also store data used by these and other programs and applications.

[0048] Network interface 106 may take the form of one or more wired interfaces, such as Ethernet (e.g., Fast Ethernet, Gigabit Ethernet, etc.). Network interface 106 may also support communication over one or more non-Ethernet media, such as coaxial cable or power line, or over wide area media, such as Synchronous Optical Network (SONET) or Digital Subscriber Line (DSL) technology. Network interface 106 may also take the form of one or more wireless interfaces, such as IEEE 802.11 (Wi-Fi), BLUETOOTH, Global Positioning System (GPS), or wide area radio interfaces. However, other types of physical layer interfaces and other types of standard or proprietary communication protocols may be used over network interface 106. Furthermore, network interface 106 may include multiple physical interfaces. Illustratively, some embodiments of computing device 100 may include Ethernet, BLUETOOTH, and Wi-Fi interfaces.

[0049] The input / output unit 108 may facilitate user and peripheral device interaction with the computing device 100. The input / output unit 108 may include one or more types of input devices, such as a keyboard, a mouse, and a touchscreen. Similarly, the input / output unit 108 may include one or more types of output devices, such as a screen, a monitor, a printer, and / or one or more light-emitting diodes (LEDs). Additionally or alternatively, the computing device 100 may communicate with other devices using, for example, a Universal Serial Bus (USB) or a High-Definition Multimedia Interface (HDMI) port interface.

[0050] In some embodiments, one or more computing devices, such as computing device 100, may be deployed to support an aPaaS architecture. The exact physical location, connectivity, and configuration of these computing devices may be unknown and / or unimportant to the client device. Thus, the computing devices may be referred to as “cloud-based” devices that may be hosted at various remote data center locations.

[0051] 2 depicts a cloud-based server cluster 200 according to an example embodiment. In FIG. 2, the operation of a computing device (e.g., computing device 100) may be distributed among server devices 202, data storage 204, and routers 206, all of which may be connected by a local cluster network 208. The number of server devices 202, data storage 204, and routers 206 in server cluster 200 may depend on the computing tasks and / or applications assigned to server cluster 200.

[0052] For example, server devices 202 may be configured to perform various computing tasks for computing device 100. Accordingly, computing tasks may be distributed among one or more of server devices 202. To the extent that these computing tasks may be performed in parallel, such distribution of tasks may reduce the total time to complete these tasks and return results. For purposes of simplicity, both server cluster 200 and individual server devices 202 may be referred to as "server devices." This nomenclature should be understood to mean that one or more separate server devices, data storage devices, and cluster routers may be involved in the operation of a server device.

[0053] Data storage 204 may be a data storage array including a drive array controller configured to manage read and write access to a group of hard disk drives and / or solid state drives. The drive array controller, alone or in combination with server devices 202, may also be configured to manage backup or redundant copies of data stored in data storage 204 to protect against drive failures or other types of failures that prevent one or more of server devices 202 from accessing units of data storage 204. Other types of memory besides drives may also be used.

[0054] Router 206 may include network equipment configured to provide internal and external communications to server cluster 200. For example, router 206 may include one or more packet switching and / or routing devices (including switches and / or gateways) configured to provide network communications between (i) server devices 202 and data storage 204 via local cluster network 208, and / or (ii) server cluster 200 and other devices via communication link 210 to network 212.

[0055] The configuration of router 206 may also be based at least in part on the data communication requirements of server device 202 and data storage 204, the latency and throughput of local cluster network 208, the latency, throughput, and cost of communication link 210, and / or other factors that may contribute to the cost, speed, fault tolerance, resilience, efficiency, and / or other design goals of the system architecture.

[0056] As a possible example, data storage 204 may include any type of database, such as a Structured Query Language (SQL) database. Various types of data structures may store information in such a database, including, but not limited to, tables, arrays, lists, trees, tuples, etc. Furthermore, any database in data storage 204 may be monolithic or distributed across multiple physical devices.

[0057] The server device 202 may be configured to send data to and receive data from the data storage 204. This sending and retrieval may take the form of SQL queries or other types of database queries, and the output of such queries, respectively. Additional text, images, video, and / or audio may also be included. Furthermore, the server device 202 may organize the received data into web page representations. Such representations may take the form of a markup language, such as Hypertext Markup Language (HTML), Extensible Markup Language (XML), or some other standardized or proprietary format. Furthermore, the server device 202 may be capable of executing various types of computerized scripting languages, such as, but not limited to, Perl, Python, PHP Hypertext Preprocessor (PHP), Active Server Pages (ASP), and JAVASCRIPT®. Computer program code written in these languages ​​may facilitate the provision of web pages to client devices and their interaction with the web pages.

[0058] III. Remote Network Management Architecture Example 3 depicts a remote network management architecture according to an example embodiment. The architecture includes three main components: a managed network 300, a remote network management platform 320, and a third-party network 340, all connected via the Internet 350.

[0059] Managed network 300 may be, for example, an enterprise network used by an entity for computing and communication tasks, as well as for storing data. Thus, managed network 300 may include client devices 302, server devices 304, routers 306, virtual machines 308, firewalls 310, and / or proxy servers 312. Client devices 302 may be embodied by computing devices 100, server devices 304 may be embodied by computing devices 100 or server clusters 200, and routers 306 may be any type of router, switch, or gateway.

[0060] Virtual machine 308 may be embodied by one or more of computing device 100 or server cluster 200. Generally, a virtual machine is an emulation of a computing system that mimics the functionality (e.g., processor, memory, and communication resources) of a physical computer. A single physical computing system, such as server cluster 200, may support up to thousands of individual virtual machines. In some embodiments, virtual machines 308 may be managed by a centralized server device or application that facilitates allocation of physical computing resources to individual virtual machines, as well as performance and error reporting. Enterprises often use virtual machines to efficiently allocate computing resources as needed. Providers of virtualized computing systems include VMWARE® and MICROSOFT®.

[0061] Firewall 310 may be one or more specialized router or server devices that protect managed network 300 from unauthorized access attempts to devices, applications, and services therein while allowing authorized communications initiated from managed network 300. Firewall 310 provides intrusion detection, web filtering, virus scanning, application layer gateways, and other applications or services. In some embodiments not shown in FIG. 3, managed network 300 may include one or more virtual private network (VPN) gateways that communicate with remote network management platform 320 (see below).

[0062] The managed network 300 may also include one or more proxy servers 312. An embodiment of the proxy server 312 may be a server device that facilitates communication and data movement between the managed network 300, the remote network management platform 320, and the third-party network 340. In particular, the proxy server 312 may be capable of establishing and maintaining secure communication sessions with one or more computing instances of the remote network management platform 320. Via such sessions, the remote network management platform 320 may be able to discover and manage aspects of the architecture and configuration of the managed network 300 and its components. Possibly with the aid of the proxy server 312, the remote network management platform 320 may also be able to discover and manage aspects of the third-party network 340 used by the managed network 300.

[0063] A firewall, such as firewall 310, typically denies all communication sessions coming in via Internet 350 unless such sessions are ultimately initiated from behind the firewall (i.e., from a device on managed network 300) or unless the firewall is explicitly configured to support the sessions. By placing proxy server 312 behind firewall 310 (e.g., within managed network 300 and protected by firewall 310), proxy server 312 may be able to initiate these communication sessions through firewall 310. Thus, firewall 310 may not need to be specially configured to support incoming sessions from remote network management platform 320, thereby avoiding a potential security risk to managed network 300.

[0064] In some cases, the managed network 300 may consist of just a few devices and a small number of networks. In other deployments, the managed network 300 may span multiple physical locations and include hundreds of networks and hundreds of thousands of devices. Thus, the architecture depicted in Figure 3 can be scaled up or down by orders of magnitude.

[0065] Furthermore, depending on the size, architecture, and connectivity of the managed network 300, various numbers of proxy servers 312 may be deployed therein. For example, each one of the proxy servers 312 may be responsible for communicating with the remote network management platform 320 for a portion of the managed network 300. Alternatively or additionally, a set of two or more proxy servers may be assigned to that portion of the managed network 300 for purposes of load balancing, redundancy, and / or high availability.

[0066] The remote network management platform 320 is a hosted environment that provides aPaaS services to users, particularly operators of the managed network 300. These services may illustratively take the form of a web-based portal. Thus, users may securely access the remote network management platform 320, illustratively from client devices 302, or potentially from client devices external to the managed network 300. Via the web-based portal, users may design, test, and deploy applications, generate reports, view analytics, and perform other tasks.

[0067] As shown in FIG. 3 , remote network management platform 320 includes four compute instances: 322, 324, 326, and 328. Each of these instances may represent one or more server devices and / or one or more databases that provide a set of web portals, services, and applications (e.g., a fully functional aPaaS system) available to a particular customer. In some cases, a single customer may use multiple compute instances. For example, managed network 300 may be an enterprise customer of remote network management platform 320 and may use compute instances 322, 324, and 326. A reason for providing multiple instances to a single customer is that the customer may want to independently develop, test, and deploy its applications and services. Thus, compute instance 322 may be dedicated to application development related to managed network 300, compute instance 324 may be dedicated to testing these applications, and compute instance 326 may be dedicated to live operation of the tested applications and services. The compute instances may also be referred to as hosted instances, remote instances, customer instances, or some other designation. Any application deployed on a compute instance may be a scoped application in that its access to the database within the compute instance may be limited to certain elements therein (e.g., one or more specific database tables or specific rows with one or more database tables).

[0068] For purposes of clarity, this disclosure refers to the physical hardware, software, and their arrangement as a "compute instance." Note that a user may colloquially refer to the graphical user interface provided thereby as an "instance." However, unless otherwise defined herein, a "compute instance" is a computing system deployed within remote network management platform 320.

[0069] The multi-instance architecture of the remote network management platform 320 contrasts with traditional multi-tenant architectures, where multi-instance architectures offer several advantages. In a multi-tenant architecture, data from different customers (e.g., businesses) is mixed in a single database. These customers' data is isolated from one another, but this isolation is enforced by the software operating the single database. As a result, a security breach in this system could affect the data of all customers, creating additional risk, especially for entities subject to government, healthcare, and / or financial regulations. Furthermore, any database operation that affects one customer will likely affect all customers who share that database. Thus, if there is an outage due to a hardware or software error, this outage affects all such customers. Similarly, if a database is upgraded to meet the needs of one customer, it will be unavailable to all customers during the upgrade process. Often, due to the size of the shared database, such maintenance windows will be long.

[0070] In contrast, a multi-instance architecture provides each customer with its own database within a dedicated computing instance. This prevents intermingling of customer data and allows each instance to be managed independently. For example, if one customer's instance experiences an outage due to an error or upgrade, other compute instances are not affected. Because the database contains only one customer's data, maintenance downtime is limited. Furthermore, the simpler design of a multi-instance architecture allows redundant copies of each customer database and instance to be deployed in a geographically diverse manner. This facilitates high availability; if a failure is detected or maintenance is performed, the live version of a customer's instance can be taken down.

[0071] In some embodiments, remote network management platform 320 may include one or more central instances controlled by the entity operating the platform. Similar to the compute instances, the central instances may include any number of physical or virtual servers and database devices. Such central instances may serve as repositories for data that may be shared among at least some of the compute instances. Illustratively, definitions of common security threats that may occur on the compute instances, software packages commonly found on the compute instances, and / or an application store for applications that may be deployed to the compute instances may reside within the central instance. The compute instances may communicate with the central instance via well-defined interfaces to obtain this data.

[0072] To support multiple compute instances in an efficient manner, remote network management platform 320 may implement these multiple instances on a single hardware platform. For example, if the aPaaS system is implemented on a server cluster, such as server cluster 200, it may run virtual machines that dedicate varying amounts of compute, storage, and communication resources to the instances. However, full virtualization of server cluster 200 may not be necessary, and other mechanisms may be used to isolate instances. In some examples, each instance may have a dedicated account on server cluster 200 and one or more dedicated databases. Alternatively, compute instances 322 may span multiple physical devices.

[0073] In some cases, a single server cluster of remote network management platform 320 may support multiple independent enterprises. Additionally, as described below, remote network management platform 320 may include multiple server clusters deployed in geographically diverse data centers to facilitate load balancing, redundancy, and / or high availability.

[0074] Third-party network 340 may be remote server devices (e.g., multiple server clusters, such as server cluster 200) that may be used for outsourced computation, data storage, communications, and service hosting operations. These servers may be virtualized (i.e., the servers may be virtual machines). Examples of third-party network 340 may include AMAZON WEB SERVICES® and MICROSOFT® AZURE®. Similar to remote network management platform 320, multiple server clusters supporting third-party network 340 may be deployed in different geographic locations for purposes of load balancing, redundancy, and / or high availability.

[0075] Managed network 300 may use one or more of third-party networks 340 to deploy applications and services to its clients and customers. By way of illustration, if managed network 300 provides an online music streaming service, third-party network 340 may store music files and provide a web interface and streaming capabilities. In this way, the businesses in managed network 300 do not have to build and maintain their own servers for these operations.

[0076] The remote network management platform 320 may include modules that integrate with third-party networks 340 to expose virtual machines and the managed services therein to the managed network 300. The modules may enable users to request virtual resources and provide flexible reports to the third-party networks. To establish this functionality, a user from the managed network 300 may first establish an account with the third-party network 340 and request a set of associated resources. The user may then enter the account information into the appropriate modules of the remote network management platform 320. These modules may then automatically discover the manageable resources within the account and may also provide reports related to usage, performance, and billing.

[0077] Internet 350 may represent a portion of the global Internet, however, Internet 350 may instead represent a different type of network, such as a private wide-area or local-area packet-switched network.

[0078] 4 further illustrates the communication environment between managed network 300 and compute instance 322, introducing additional features and alternative embodiments. In FIG. 4, compute instance 322 is replicated across data centers 400A and 400B. These data centers may be geographically distant from one another, perhaps in different cities or different countries. Each data center includes supporting equipment that facilitates communication with managed network 300 and remote users.

[0079] In data center 400A, network traffic to and from external devices flows through either VPN gateway 402A or firewall 404A. VPN gateway 402A may peer with VPN gateway 412 of managed network 300 via security protocols such as Internet Protocol Security (IPSEC) or Transport Layer Security (TLS). Firewall 404A may be configured to allow access from authorized users, such as user 414 and remote user 416, and to deny access to unauthorized users. Through firewall 404A, these users may access compute instance 322 and potentially other compute instances. Load balancer 406A may be used to distribute traffic among one or more physical or virtual server devices hosting compute instance 322. Load balancer 406A may simplify user access by hiding the internal configuration of data center 400A (e.g., compute instance 322) from client devices. By way of illustration, if compute instance 322 includes multiple physical or virtual computing devices that share access to multiple databases, load balancer 406A may distribute network traffic and processing tasks across these computing devices and databases so that some computing devices or databases are not significantly busier than others. In some embodiments, compute instance 322 may include VPN gateway 402A, firewall 404A, and load balancer 406A.

[0080] Data center 400B may include its own versions of components in data center 400A. Thus, VPN gateway 402B, firewall 404B, and load balancer 406B may perform the same or similar operations as VPN gateway 402A, firewall 404A, and load balancer 406A, respectively. Furthermore, via real-time or near-real-time database replication and / or other operations, compute instances 322 may exist simultaneously in data centers 400A and 400B.

[0081] Data centers 400A and 400B, as shown in Figure 4, can facilitate redundancy and high availability. In the configuration of Figure 4, data center 400A is active and data center 400B is passive. Thus, data center 400A serves all traffic to and from managed network 300, while versions of compute instances 322 in data center 400B are updated in approximately real time. Other configurations, such as a configuration in which both data centers are active, can be supported.

[0082] If data center 400A fails or otherwise becomes unavailable to users in some way, data center 400B may take over as the active data center. For example, a domain name system (DNS) server that associates the domain name of compute instance 322 with one or more Internet Protocol (IP) addresses of data center 400A may reassociate the domain name with one or more IP addresses of data center 400B. After this reassociation is complete (which may take less than a second or a few seconds), users may access compute instance 322 via data center 400B.

[0083] FIG. 4 also illustrates possible configurations of managed network 300. As noted above, proxy server 312 and user 414 may access compute instance 322 through firewall 310. Proxy server 312 may also access configuration item 410. In FIG. 4, configuration item 410 may refer to any or all of client device 302, server device 304, router 306, and virtual machine 308, any applications or services running on them, and any or all of the relationships among the devices, applications, and services. Thus, the term “configuration item” may be shorthand for any physical or virtual device, or any application or service remotely discoverable or managed by compute instance 322, or the relationships among the discovered devices, applications, and services. Configuration items may be represented in a configuration management database (CMDB) of compute instance 322.

[0084] As described above, VPN gateway 412 may provide a dedicated VPN for VPN gateway 402A. Such a VPN may be useful when there is a large amount of traffic between managed network 300 and compute instance 322, or when security policies suggest or require the use of a VPN between these sites. In some embodiments, any devices within managed network 300 and / or compute instance 322 that communicate directly via the VPN are assigned public IP addresses. Other devices within managed network 300 and / or compute instance 322 may be assigned private IP addresses (e.g., IP addresses selected from the ranges 10.0.0.0 to 10.255.255.255 or 192.168.0.0 to 192.168.255.255, represented in shorthand as subnets 10.0.0.0 / 8 and 192.168.0.0 / 16, respectively).

[0085] IV. Device, Application, and Service Discovery Examples In order for the remote network management platform 320 to manage the devices, applications, and services of the managed network 300, the remote network management platform 320 may first determine what devices exist in the managed network 300, the configuration and operational status of those devices, the applications and services provided by the devices, and the relationships between the discovered devices, applications, and services. As noted above, each device, application, service, and relationship may be referred to as a configuration item. The process of defining configuration items within the managed network 300 is referred to as discovery and may be facilitated at least in part by the proxy server 312.

[0086] For purposes of embodiments herein, an "application" may refer to one or more processes, threads, programs, client modules, server modules, or any other software executing on a device or group of devices. A "service" may refer to a high-level capability provided by multiple applications executing on one or more devices working in conjunction with each other. For example, a high-level web service may include multiple web application server threads executing on one device and accessing information from a database application running on another device.

[0087] 5A provides a logical depiction of how configuration items may be discovered and how information related to the discovered configuration items may be stored. For simplicity, remote network management platform 320, third-party network 340, and Internet 350 are not shown.

[0088] 5A, the CMDB 500 and task list 502 are stored within the compute instance 322. The compute instance 322 may send a discovery command to the proxy server 312. In response, the proxy server 312 may send probes to various devices, applications, and services within the managed network 300. These devices, applications, and services may send responses to the proxy server 312, which may then provide information about the discovered configuration items to the CMDB 500 for storage therein. The configuration items stored in the CMDB 500 represent the environment of the managed network 300.

[0089] Task list 502 represents a list of activities that proxy server 312 performs on behalf of compute instance 322. As discoveries are made, task list 502 is populated. Proxy server 312 repeatedly queries task list 502 to obtain the next task therein and performs this task until task list 502 is empty or another stopping condition is reached.

[0090] To facilitate discovery, proxy server 312 may be configured with information about one or more subnets in managed network 300 that are reachable via proxy server 312. By way of illustration, proxy server 312 may be given the IP address range 192.168.0 / 24 as a subnet. Compute instance 322 may then store this information in CMDB 500 and place tasks in task list 502 for discovery of devices at each of these addresses.

[0091] 5A also depicts devices, applications, and services in managed network 300 as configuration items 504, 506, 508, 510, and 512. As noted above, these configuration items represent physical and / or virtual devices (e.g., client devices, server devices, routers, or virtual machines), the applications running on them (e.g., web servers, email servers, databases, or storage arrays), the relationships between them, and sets of services that include multiple individual configuration items.

[0092] Placing a task in task list 502 may trigger discovery or otherwise cause proxy server 312 to initiate discovery. Alternatively or additionally, discovery may be triggered manually or automatically based on a triggering event (e.g., discovery may start automatically once a day at a specific time).

[0093] Generally, discovery may proceed in four logical phases: scanning, classification, identification, and exploration. Each phase of discovery involves different types of probe messages being sent by proxy server 312 to one or more devices in managed network 300. Responses to these probes may be received and processed by proxy server 312, and representations thereof may be sent to CMDB 500. Thus, each phase may result in more configuration items being discovered and stored in CMDB 500.

[0094] During the scan phase, proxy server 312 may probe each IP address within a specified range of IP addresses for open Transmission Control Protocol (TCP) and / or User Datagram Protocol (UDP) ports to determine the general type of device. The presence of such open ports at an IP address may indicate that a particular application is running on the device to which the IP address is assigned, which in turn may identify the operating system used by the device. For example, if TCP port 135 is open, the device may be running a WINDOWS® operating system. Similarly, if TCP port 22 is open, the device may be running a UNIX® operating system, such as LINUX®. If UDP port 161 is open, the device may be further identifiable through Simple Network Management Protocol (SNMP). Other possibilities exist. Once the presence of a device at a particular IP address and its open ports is discovered, these configuration items are stored in CMDB 500.

[0095] During the classification phase, proxy server 312 may further probe each discovered device to determine its operating system version. The probes used for a particular device are based on information collected about the device during the scan phase. For example, if a device is found with TCP port 22 open, a set of UNIX-specific probes may be used. Similarly, if a device is found with TCP port 135 open, a set of Windows-specific probes may be used. In either case, an appropriate set of tasks may be placed in task list 502 for proxy server 312 to execute. These tasks may cause proxy server 312 to log on to or otherwise access information from a particular device. By way of illustration, if TCP port 22 is open, proxy server 312 may be instructed to initiate a Secure Shell (SSH) connection to the particular device and retrieve information about the operating system thereon from a particular location in the file system. Based on this information, the operating system may be determined. As an example, a UNIX® device with TCP port 22 open may be classified as AIX®, HPUX, LINUX®, MACOS®, or SOLARIS®. This classification information may be stored in CMDB 500 as one or more configuration items.

[0096] During the identification phase, the proxy server 312 may determine specific details about the classified device. The probes used during this phase may be based on information collected about the particular device during the classification phase. For example, if the device is classified as LINUX®, a set of LINUX®-specific probes may be used. Similarly, if the device is classified as WINDOWS® 2012, a set of WINDOWS® 2012-specific probes may be used. As with the classification phase, an appropriate set of tasks may be placed in the task list 502 for the proxy server 312 to execute. These tasks may result in the proxy server 312 retrieving information from the particular device, such as basic input / output system (BIOS) information, serial number, network interface information, media access control addresses assigned to those network interfaces, and the IP address used by the particular device. This identification information may be stored in the CMDB 500 as one or more configuration items.

[0097] During the discovery phase, the proxy server 312 may determine further details about the operational state of the classified device. The probes used during this phase may be based on information collected about the particular device during the classification and / or identification phases. Again, an appropriate set of tasks may be placed in the task list 502 for the proxy server 312 to execute. These tasks may result in the proxy server 312 retrieving additional information from the particular device, such as processor information, memory information, and a list of running processes (applications). Again, the discovered information may be stored in the CMDB 500 as one or more configuration items.

[0098] Performing discovery on a network device, such as a router, may utilize SNMP. Instead of, or in addition to, determining a list of running processes or other application-related information, discovery may determine additional subnets known to the router and the operational state of the router's network interfaces (e.g., active, inactive, queue length, number of dropped packets, etc.). The IP addresses of the additional subnets may be candidates for further discovery procedures. Thus, discovery may proceed iteratively or recursively.

[0099] Once discovery is complete, a snapshot representation of each discovered device, application, and service is available in CMDB 500. For example, after discovery, operating system versions, hardware configurations, and network configuration details for client devices, server devices, and routers in managed network 300, as well as the applications running on them, may be stored. This collected information may be presented to the user in a variety of ways to allow the user to view the hardware configuration and operational status of devices and service characteristics across multiple devices and applications.

[0100] Additionally, CMDB 500 may include entries regarding dependencies and relationships between configuration items. More specifically, an application running on a particular server device and the services that depend on this application may be represented as such in CMDB 500. As an illustrative example, assume that a database application runs on a server device and that this database application is used by a new employee onboarding service and a payroll service. Therefore, if the server device goes down for maintenance, it is clear that the employee onboarding service and the payroll service will be affected. Similarly, dependencies and relationships between configuration items may be able to represent the services that will be affected if a particular router fails.

[0101] Generally, dependencies and relationships between configuration items may be displayed on a web-based interface and represented in a hierarchical manner, such that adding, modifying, or deleting such dependencies and relationships may be accomplished via this interface.

[0102] Additionally, users from managed network 300 may develop workflows that allow certain coordinated activities to be performed across multiple discovered devices. As an illustration, an IT workflow may allow a user to change the administrator password common to all discovered Linux devices in a single action.

[0103] For discovery to occur in the manner described above, proxy server 312, CMDB 500, and / or one or more credential stores may be configured with credentials for one or more of the devices to be discovered. The credentials may include any type of information necessary to access the device. These may include user ID / password pairs, credentials, etc. In some embodiments, these credentials may be stored in encrypted fields in CMDB 500. Proxy server 312 may include decryption keys for the credentials so that proxy server 312 can use these credentials to log on to or otherwise access the devices being discovered.

[0104] The discovery process is depicted as a flowchart in FIG. 5B. In block 520, a task list within a compute instance is illustratively populated with a range of IP addresses. In block 522, a scanning phase occurs. The proxy server then probes the IP addresses for devices using these IP addresses and attempts to determine the operating systems running on these devices. In block 524, a classification phase occurs. The proxy server attempts to determine the operating system versions of the discovered devices. In block 526, an identification phase occurs. The proxy server attempts to determine the hardware and / or software configuration of the discovered devices. In block 528, a discovery phase occurs. The proxy server attempts to determine the operational state and applications running on the discovered devices. In block 530, further editing of the configuration items representing the discovered devices and applications can occur. This editing can be automated in nature and / or performed manually.

[0105] The blocks depicted in Figure 5B are for illustrative purposes. Discovery can be a highly configurable procedure that can have more or fewer phases, and the operation of each phase can vary. In some cases, one or more phases can be customized or otherwise deviate from the exemplary description above.

[0106] V. Natural Language Processing of Text Queries Natural language processing is a field that involves, among other activities, using computers to understand the structure and meaning of human language. This determined structure and meaning may be applicable to the processing of IT incidents, as described below.

[0107] Each incident may be represented as an incident report. Although incident reports may exist in a variety of formats and contain a variety of types of information, an exemplary incident report 600 is shown in Figure 6. Incident report 600 consists of several fields in the left column, at least some of which are associated with values ​​in the right column.

[0108] Field 602 identifies the originator of the incident, in this case Bob Smith. Field 604 identifies the time the incident was created, in this case February 7, 2018 at 9:56 AM. Field 605 is a text string providing a brief description of the problem. Field 606 identifies the description of the problem as provided by the originator. Thus, field 606 can be a free-form text string containing anywhere from a few words to a few sentences or more. Field 608 is the categorization of the incident, in this case an email. This categorization can be provided by the originator, the IT person to whom the incident is assigned, or can be automatically provided based on the context of the problem description field.

[0109] Field 610 (if applicable) identifies the IT person to whom the incident is assigned, in this case, Alice Jones. Field 612 identifies the status of the incident. The status may illustratively be one of "open," "assigned," "in progress," or "resolved." Field 614 (if applicable) identifies how the incident was resolved. This field may be completed by the IT person to whom the incident is assigned or another individual. Field 616 identifies the time the incident was resolved, in this case, February 7, 2018, at 10:10 AM. Field 618 (if applicable) specifies the incident's closure code, which may take values ​​such as "closed (permanent)," "closed (workaround)," or "closed (cannot reproduce)." Field 620 identifies any additional notes added to the record, such as by the IT person to whom the incident is assigned. Field 622 identifies a link to an online article that may help the user avoid having to deal with a similar problem in the future.

[0110] Incident report 600 is presented for purposes of example. Other types of incident reports may be used, and these reports may include more, fewer, and / or different fields.

[0111] Incident reports such as incident report 600 may be created in a variety of ways, including, for example, via a web form, email sent to a specified address, and a voicemail box using speech-to-text conversion. These incident reports may be stored in a queryable incident report database. For example, a query in the form of a text string may return one or more incident reports that contain words in the text string. Additionally or alternatively, one or more elements of the incident report (e.g., the "short description" field) may be used to query a database of knowledge base articles, other incident reports, or some other corpus of text. This may be done to identify other incident reports, past incident reports that have been resolved, reports on the resolution of past problems, knowledge base articles, or other information that may be relevant to the incident report to facilitate resolution of the problem represented in the incident report.

[0112] This process is illustrated in Figure 7. A text query may be entered into a web interface 700. This web interface may be served via a compute instance of the remote network management platform 320. The web interface 700 converts the text query into a database query (e.g., an SQL query) and provides the SQL query to a database 702. This database may be the CMDB 500 or some other database. The database 702 contains several incident reports with problem description fields as shown in Figure 6. The database 702 nevertheless performs the query and returns matching results to the web interface 700. One or more such results may be returned. The web interface 700 presents these results as a web page.

[0113] For example, if the text query is "email," the web interface 700 may translate this query into an SQL query of the database 702. For example, the query may look at the problem description field of a table containing incident reports. Any such incident reports that match the query, i.e., contain the term "email," may be provided in the query results. Thus, incident reports with problem descriptions of "my email client is not downloading new emails," "email crashed," and "can't connect to email" may be provided, while incident reports with the problem description "VPN timed out" are not returned.

[0114] This matching technique is simple and has several drawbacks. It only considers the presence of the query text within the incident. Therefore, it does not consider contextual information such as words that appear before or after the query text. Also, synonyms of the query text (e.g., "email" or "message") and misspellings of the query text (e.g., "emial") would not return any results in this example.

[0115] Furthermore, deploying such a solution would involve the use of an inefficient sparse matrix, with one dimension of entries for each word in the English language and another dimension of entries for the problem description of each incident. The exact number of English words is debatable, but there are at least 150,000 to 200,000, with less than about 20,000 in common usage. Given that a busy IT department may have a database of tens of thousands of incidents, this matrix would be prohibitively large and wasteful to store, even if it contained only the 20,000 most commonly used words.

[0116] Thus, the above comparison methods may be replaced and / or augmented with various methods for comparing the semantic content and / or context of text samples. These methods may improve upon various machine learning techniques to facilitate natural language processing. Such techniques may include determining word and / or paragraph vectors from text samples, applying artificial neural networks or other deep learning algorithms, sentiment analysis, or other techniques for determining similarity between text samples. For example, these or other natural language processing techniques may be applied to determine similarity between one or more text fields of an incident report and other incident reports, resolved incident reports, knowledge base articles, or other potentially related text samples.

[0117] VI. Natural Language Processing of Text Queries Based on Semantic Content The degree of similarity between two samples of text may be determined in various ways. The two samples of text may be a text field of an incident report and a text field of another incident report, a text field of a resolved incident report, a knowledge base article, or any other sample of text that may be related to the resolution, classification, or other aspects of the incident report. Additionally or alternatively, one or both of the samples may be a segment of text within a larger sample of text. As noted above, the degree of overlap between the two samples of text and / or the identity of the words present in the word matrix may be used to determine the degree of similarity. Additionally or alternatively, one or more techniques of natural language processing may be applied to compare the samples of text, such that the context or other semantic content of the text affects the determined similarity value between the samples of text.

[0118] Such techniques can be applied to improve matching of text queries related to incident reports. These techniques can include various machine learning algorithms that can be trained based on text samples. The text samples used for training can include historical examples of incident reports, knowledge base articles, or other text samples of the same nature as the text samples to which the trained model will be applied. This has the advantage of providing a model that is uniquely adapted to the common vocabulary, topics, and idiomatic word usage within its intended application.

[0119] Such techniques may include determining word and / or paragraph vectors from samples of text, applying ANN or other deep learning algorithms, performing sentiment analysis, or other techniques for determining similarity between samples of text, grouping multiple samples of text together according to topic or content, dividing samples of text into distinct inter-related segments, determining statistical associations between words, or performing some other language processing task. Below, a specific method for determining similarity values ​​between samples of text using an ANN model that provides a compact semantic representation of words and text strings is provided as a non-limiting example of such a technique. However, as applied elsewhere herein, other techniques can be applied to generate similarity values ​​between samples of text. In the following discussion, there are two approaches for training an ANN model to represent the semantic meaning of words: word vectors and paragraph vectors. These techniques can be combined with each other or with other techniques.

[0120] A. Word Vectors A "word vector" can be determined for each word present in a corpus of text records, such that words with similar meanings (or "semantic content") are associated with word vectors that are near each other in a semantically encoded vector space. Such vectors can have tens, hundreds, or more elements. These word vectors allow the underlying meanings of words to be compared or otherwise acted upon by a computing device. Thus, the use of word vectors can be a significant improvement over simpler word list or word matrix methods.

[0121] The word vectors can be used to quickly and efficiently compare the overall semantic content of the text samples, allowing a similarity value to be determined between the text samples. This may include determining the distance, cosine similarity, or some other measure of similarity between the word vectors of the words in each of the text samples. For example, the average of the word vectors in each of the text samples may be determined, and the cosine similarity between the averages may then be used as a measure of similarity between the text samples. Additionally or alternatively, the word vectors may be provided as input to an ANN, support vector machine, decision tree, or some other machine learning algorithm to perform sentiment analysis, classify or cluster the text samples, determine the level of similarity between the text samples, or perform some other language processing task.

[0122] Word vectors can be determined for a set of words in a variety of ways. In one example, a matrix of word vectors can be the input layer of an ANN. The ANN (containing the matrix of word vectors) can then be trained with a large number of text strings from a database to determine contextual relationships between words that appear in these text strings. Such an ANN 800 is shown in FIG. 8A. The ANN 800 includes an input layer 802, which feeds a hidden layer 804, which in turn feeds an output layer 806. The number of nodes in the input layer 802 and the output layer 806 may be equal to the number of words in a predefined vocabulary or dictionary (e.g., 20,000 words, 50,000 words, or 100,000 words). The number of nodes in the hidden layer 804 can be much smaller (e.g., 64 as shown in FIG. 8A, or other values ​​such as 16, 32, 128, 512, 1024, etc.).

[0123] For each text string in the database, the ANN 800 is trained with one or more arrangements of words. By way of illustration, in FIG. 8B , the ANN 800 is shown being trained with the input word "email" and the output (context) words "can't," "connect," and "to." The output words serve as ground truth outputs against which results produced by the output layer 806 are compared. This arrangement reflects the fact that "email" appears in close proximity to "can't," "connect," and "to" in the text strings in the database 702.

[0124] In an implementation, this may be represented as node 12 receiving an input of 1 and all other nodes in the input layer 802 receiving an input of 0. Similarly, node O1 may be associated with a ground truth value of "cannot", node O2 may be associated with a ground truth value of "connect", and node O3 may be associated with a ground truth value of "to". In an implementation, this may be represented as nodes O1, O2, and O3 being associated with ground truth values ​​of 1, and all other nodes in the output layer 806 being associated with ground truth values ​​of 0. The loss function may be, for example, the sum of squared errors between the outputs generated by the output layer 806 in response to the above inputs and vectors containing the ground truth values ​​associated with the output layer nodes.

[0125] Other arrangements of this text string from database 702 can be used to train the ANN. By way of illustration, as shown in Figure 9A, the input word to input layer 902 of ANN 900 can be "can't," and the output words of output layer 906 can be "connect," "to," and "email." In another example, as shown in Figure 9B, the input word can be "connect," and the output words can be "can't," "to," and "email."

[0126] In general, these arrangements may be selected such that the output word is within w words of the input word (e.g., w is 1, 2, 3, 5, etc.), the output word is in the same sentence as the input word, the output word is in the same paragraph as the input word, etc. Furthermore, various word arrangements of each text string in database 702 may be used to train an ANN (e.g., ANN800 or ANN900). These text strings may be selected from the short description field 605, the problem description field 606, the category field 608, the solution field 614, the notes field 620, and / or any other field or combination of fields in an incident report.

[0127] After the ANN (e.g., 800, 900) is trained using these arrangements of text strings, the hidden layer (e.g., 804, 904) of the ANN becomes a compact vector representation of the context and meaning of the input words. That is, the weights from a particular node in the input layer (e.g., I3) to the hidden layer represent elements of the word vector for the word (e.g., "can't") corresponding to that particular node. For example, assuming the ANN is fully trained with a corpus of text strings on the order of 10,000 words (although more or fewer text strings may be used), an input word of "email" may have a similar vector representation to an input word of "mail." Intuitively, because it is only in the hidden layer that the ANN must determine the context of the input words, if two words have similar contexts, they are more likely to have similar vector representations.

[0128] In some embodiments, the ANN has output nodes O1...O n and the input words associated with input nodes I1···I n and the associated output (context) words. This arrangement may yield the same or similar vectors for the hidden layer.

[0129] Furthermore, the vectors generated in this manner are additive. Thus, subtracting the vector representation of "mail" from the vector representation of "email" is expected to produce a vector whose value is close to 0. However, subtracting the vector representation of "VPN" from the vector representation of "email" is expected to produce a vector whose value is higher. Thus, the model indicates that "email" and "mail" are closer in meaning than "email" and "VPN."

[0130] Once vector representations have been determined for all words of interest, a linear and / or multiplicative aggregation of these vectors can be used to represent the text string. As an illustration, the vector for the text string "cannot connect to email" can be found by adding together the individual vectors for the words "cannot", "connect", "to", and "email". In some cases, an average or some other operation can be applied to the vectors for the words. This can be done by dividing each entry in m vectors v by m. i where i = {1 m}. However, other possibilities exist, such as weighted averaging.

number

[0131] Regardless of how aggregation is determined, this general technique allows for finding a vector representation for each text string in database 702. These vector representations may also be stored in database 702, together with their associated text strings or separately. These vector representations may then be used to compare text strings, cluster or group text strings, train some other machine learning classifier, or perform some other task. For example, matching text strings for a particular query text may be determined by determining a cosine similarity or other similarity value between the vector representation of the query text and the stored vector representations of samples of text in database 702.

[0132] The comparison may identify one or more text string vectors from database 702 that "match" in this manner. In some cases, this may be the k text string vectors with the highest similarity, or any text string vectors with a similarity greater than a predetermined value. The identified text string vectors may correspond to a subset of incident reports within a larger corpus of incident reports recorded in database 702, related to additional incident reports that correspond to the query text string vector. For each identified text string vector, an associated text string may be looked up in database 702 and provided as an output text string. In some cases, an associated incident report may also be provided.

[0133] In some cases, only incident reports that are no older than a predetermined age are provided. Illustratively, the system may be configured to identify text string vectors only from incident reports that have been resolved within the past three, six, or twelve months. Alternatively, the system may be configured to identify text string vectors only from incident reports that have been opened within the past three, six, or twelve months.

[0134] In this way, incident reports with problem descriptions similar to that of the entered text string can be quickly identified. In particular, the system provides contextual results that are more likely to be relevant and meaningful to the entered text string. As a result, individuals can review these incident reports to determine how problems similar to those in the problem description have been reported and addressed in the past. This can result in a significant reduction in the time it takes to resolve incidents.

[0135] Additionally or alternatively, these embodiments may be applied to detect and identify clusters of semantically and / or contextually similar incident reports within a corpus of incident reports. For example, a cluster of incident reports related to similar problems that may affect users of an IT system, an ongoing misconfiguration of one or more aspects of the IT system, progressive hardware failures in components of the IT system, or some other recurring problem within the IT system. Identifying such clusters of related incident reports may enable the IT system to be repaired or upgraded (e.g., by replacing and / or reconfiguring faulty or inconsistently performing hardware or software), users to be trained to avoid common mistakes, infrequent hardware or software problems to be detected and corrected, or other benefits.

[0136] Such clusters of related incident reports may be detected and / or identified by identifying aggregated word (and / or paragraph) vectors in the semantically encoded vector space that correspond to the incident reports. To detect such clusters in the semantically encoded vector space, various methods may be used, such as k-means clustering, support vector machines, ANNs (e.g., unsupervised ANNs configured and / or trained to identify related subsets of training examples in a corpus of available training examples), or any other classification or other method for identifying clusters of related vectors in a vector space.

[0137] B. Paragraph Vector As previously discussed, ANN models (e.g., 800, 900) use surrounding context to provide compact, semantically relevant vector representations of words. After training, words with similar meanings may map to similar locations in vector space. For example, vectors for "powerful" and "strong" may appear near each other, while vectors for "powerful" and "Paris" may be further apart. Addition and subtraction between word vectors are also meaningful. Using vector algebra on the determined word vectors, analogy questions such as "king" - "man" + "woman" = "queen" can be answered.

[0138] However, the complete semantic meaning of a sentence or other clause (e.g., a phrase, multiple sentences, a paragraph, a text segment within a larger sample of text, or a document) cannot always be captured from the individual word vectors of the sentence (e.g., by applying vector algebra). Word vectors may represent the semantic content of individual words and may be trained using a short context window. Thus, the semantic content of word order, and any information outside the short context window, is lost when operating solely based on word vectors.

[0139] For example, consider the sentence "I want a big green cell now." In this case, simple vector algebra of individual words may fail to provide the correct semantic meaning of the word "cell" because the word "cell" has multiple possible meanings and can therefore be ambiguous. Depending on the context, a "cell" may be a biological cell, a living cell, or a cell in a communication network. Therefore, the paragraph, sentence, or phrase from which a given word is sampled can provide important contextual information.

[0140] In another example, given the sentence "Where are you_", it is easy to predict the missing word as "Romeo" if we are told that the sentence comes from a paragraph about Shakespeare. Thus, learning a semantic vector representation of the entire paragraph can help predict the context of words sampled from that paragraph.

[0141] Similar to the above method for learning word vectors, an ANN or other machine learning structure can be trained using multiple paragraphs in a corpus to determine the contextual meaning of entire paragraphs, sentences, phrases, or other multi-word text samples, as well as the meaning of individual words that make up the paragraphs in the corpus. Such an ANN 1000 is shown in FIG. 10A. The ANN 1000 includes an input layer 1002, which feeds a hidden layer 1004, which in turn feeds an output layer 1006. The input layer 1002 represents two types of input substructures: words (input nodes I1...I2), and input nodes (input nodes I3...I4). m and an upper substructure 1010 (consisting of input nodes D1...D m Note that the output layer 1006 and the top input layer substructure 1008 consist of a top substructure 1010 (consisting of a top substructure 1010 consisting of a bottom ...

[0142] For each paragraph in the corpus, the ANN 1000 is trained using a fixed-length context generated from moving a sliding window across the paragraph. Thus, a given paragraph vector is shared among all training contexts created from that source paragraph, but not among training contexts created from other paragraphs. Word vectors are shared among training contexts created from all paragraphs; for example, the vector for “can’t” is the same for all paragraphs. There is no limit to the size of paragraphs; they can be as large as an entire document or as small as a sentence or phrase. In FIG. 10A , the ANN 1000 is shown in a single training iteration, trained using input word contexts “can’t,” “connect,” and “to,” input paragraph context DOC1, and output word “email.” The output word serves as a ground truth output against which results produced by the output layer 1006 are compared. This placement reflects that “email” appears in close proximity to “can’t,” “connect,” and “to” and is located within DOC1.

[0143] In an implementation, this may be represented as output node O4 receiving a ground truth value of 1 and all other nodes in the output layer 1006 having ground truth values ​​of 0. Similarly, node I1 has a ground truth value of “cannot”, node I2 has a ground truth value of “connect”, node I3 has a ground truth value of “to”, and node D1 has a ground truth value of DOC1. In an implementation, this may be represented as nodes I1, I2, I3, and D1 being associated with a value of 1 and all other nodes in the input layer 1002 having a value of 0. The loss function may be, for example, the sum of squared errors between the output of the output layer 1006 and a vector containing the ground truth values. The weight values ​​of the corresponding word and paragraph vectors, as well as all output layer parameters (e.g., softmax weights), are updated (e.g., via backpropagation) based on the loss function.

[0144] 10B shows ANN 1000 being trained with a subsequent context window, derived from the same document but shifting the words earlier in the document, using the input word contexts "connect," "to," and "email," the input paragraph context DOC1, and the output word "server." In implementation, these inputs and outputs may be coded with ground truth values, similar to those described above.

[0145] Figure 10C shows an instance of ANN 1000 trained using another document in the corpus. A context window is derived from this document, using the input word contexts "can't", "load", and "my", the input paragraph context DOC2, and the output word "database". In implementation, these inputs and outputs may be coded with ground truth values, similar to those described above.

[0146] After ANN 1000 is trained, the weights associated with hidden layer 1004 become compact vector representations of the context and meaning of the input words and paragraphs. For example, if ANN 1000 is fully trained with a corpus of 1,000 paragraphs, and the entire corpus contains 10,000 unique words, then each paragraph and each word can be represented by a unique vector with length equal to the number of hidden nodes in hidden layer 1004. These unique vectors encode the contextual meaning of the words within the paragraph or the paragraph itself.

[0147] 10D shows the ANN 1000 at prediction time, performing an inference step to compute a paragraph vector for a new, previously unseen paragraph. This inference step begins by adding an additional input node 1012 representing the unseen paragraph (DOC M+1) to the input layer substructure 1010. During this inference process, the coefficients of the word vector substructure 1008 and the learned weights between the hidden layer 1004 and the output layer 1006 remain fixed. Thus, the model generates an additional paragraph vector 1012 corresponding to the unseen paragraph in the input paragraph vector substructure 1010 to obtain a new semantic vector representation of the unseen paragraph. Additional unseen paragraphs can be trained through a similar process by adding input nodes to the input layer substructure 1010.

[0148] Alternatively, paragraph vectors can be trained by ignoring word contexts in the input layer, using only paragraph vectors as inputs, and having the model predict different word contexts randomly sampled from paragraphs in the output layer. The input layer of such an ANN consists only of paragraph vectors, while the output layer represents a single context window randomly generated from a given paragraph. Training such an ANN may yield vector representations for the semantic content of paragraphs in a corpus, but it will not necessarily provide any semantic vector representations for the words therein.

[0149] Once vector representations are determined for the paragraphs in the corpus, linear and / or multiplicative aggregations of these vectors can be used to represent topics of interest. Furthermore, as shown in ANN 1000, if the dimensions of the paragraph vectors are the same as the dimensions of the word vectors, linear and multiplicative aggregations between word vectors and paragraph vectors can be obtained. For example, finding the Chinese equivalent of "Julius Caesar" using an encyclopedia as a corpus can be achieved by the vector operation PV("Julius Caesar")-WV("Roman")+WV("Chinese"), where PV is the paragraph vector (representing the entire Wikipedia article) and WV is the word vector. Thus, paragraph vectors can achieve the same kind of inference as word vectors with more context-based results.

[0150] In practice, such learned paragraph vectors can be used as inputs to other supervised learning models, such as sentiment prediction models. In such models, which may include, but are not limited to, ANNs, support vector machines (SVMs), or naive Bayes classifiers, the paragraph vectors are used as inputs with corresponding sentiment labels as outputs. Other metrics, such as cosine similarity and nearest neighbor clustering algorithms, can be applied to the paragraph vectors to find or group paragraphs related to similar topics within a corpus of paragraphs.

[0151] In this embodiment, the combination of learned word vectors and paragraph vectors can help determine the structure and meaning of an incident report, such as incident report 600 shown in FIG. 6 . Incident report 600 consists of multiple fields in the left column, at least some of which are associated with values ​​in the right column. For longer text fields, such as short description field 605, problem description field 606, solution field 614, and notes field 620, it may be preferable to represent the associated right-column text as a paragraph vector or multiple paragraph vectors corresponding to distinct text segments within the right-column text to obtain more contextual meaning, rather than aggregating the individual word vectors that make up the text. Incident report 600 is presented for illustrative purposes. Various fields in an incident report can be arranged to be represented as paragraph vectors, word vectors, or a weighted combination of the two. Other types of incident reports, problem reports, case files, or knowledge base articles can also be used, and these reports may include more, fewer, and / or different fields.

[0152] After representing different fields as paragraph vectors, word vectors, or a weighted combination of the two, a single vector to represent the entire incident can be generated by concatenating, generating vector sums, or otherwise aggregating the word and / or paragraph vector representations of the individual incident fields. Using a single aggregate incident vector representation, the system can be configured to identify similar aggregate vectors (and thus similar incident reports) based on cosine similarity or other metrics as discussed above. Alternatively, a search for similar incident reports may use only the paragraph text of one or more individual fields. In this way, text from one or more individual fields within an incident report can be combined into a single paragraph of text. A paragraph vector can then be generated from this single large paragraph of concatenated text and used to search for similar incidents.

[0153] This process can be described in terms of the previously described ANN structure. First, text strings are obtained from database 702 of FIG. 7. As mentioned above, these text strings may be from portions of incident reports. Words are then extracted from the text strings. The extracted words may be all of the words in the text strings or some of these words. These extracted words are provided as input to ANN 900 of FIGS. 9A-9B. Substring contexts for these words are extracted from the text strings. A substring context may be one or more substrings that include words before, after, or surrounding the extracted associated words. These vector representations may then be used to compare (e.g., using cosine similarity) with their individual text samples.

[0154] The comparison may identify one or more incident reports from database 702 that "match" in this manner. In some cases, this may be the k most similar incident reports, or any incident reports whose similarity is greater than a predetermined value. The user may be provided with these identified incident reports or references to them.

[0155] In some cases, only incident reports that are not older than a predetermined age are provided. Illustratively, the system may be configured to identify only incident reports that have been resolved within the past three, six, or twelve months. Alternatively, the system may be configured to identify only incident reports that have been opened within the past three, six, or twelve months.

[0156] In this way, incident reports with content similar to that of the entered incident report can be quickly identified. Individuals can then review these incident reports to determine how issues similar to that of the incident have been reported and addressed in the past. This can result in a significant reduction in the time it takes to resolve an incident.

[0157] Although this section describes some possible implementations of word and paragraph vectors, other implementations are possible, for example, different ANN structures and different training procedures may be used.

[0158] VII. Text Query or Other Record Clustering Queries, incident reports, knowledge base articles, and / or other text or non-text records may be clustered together. Such clustering may be performed to provide various benefits. For example, clustering may be applied to a set of records to identify patterns or groups within the set of records that have relevance to the operation of a system or organization. Such groups may facilitate tracking of ongoing problems (e.g., network outages, disruptions to users interfacing with network-based services) by measuring the time dependency of records assigned to a particular cluster associated with the ongoing problem. Such groups may facilitate early identification of emerging problems, for example, by identifying similarities between newly received reports. In some examples, clustering may enable similar reports (e.g., reports corresponding to the same cluster) to be commonly manipulated to reduce the time required to respond to a set of similar reports. For example, all reports related to a network outage and assigned to a single cluster may be resolved in a single action following resolution of the network outage.

[0159] In some examples, clustering may facilitate the assignment of reports to technicians according to expertise, familiarity, or other factors. Additionally or alternatively, knowledge base articles, resolution flowcharts, or other materials may be created for each identified cluster to facilitate resolution of reports as they are assigned to a cluster. Identifying clusters within a set of queries, incident reports, or other text or non-text records may provide additional or alternative benefits.

[0160] Incident reports, queries, knowledge base articles, or any type of record that may contain textual and / or non-textual elements may be grouped into clusters in a variety of ways. Such clustering may be performed in a supervised manner to generate a clustering algorithm that assigns new records to clusters according to a manually classified set of training records. Additionally or alternatively, clustering may be performed in an unsupervised manner to generate clusters without the need for manually labeled records, to identify previously unidentified clusters in training data, or to provide some other advantage.

[0161] Various methods and / or machine learning algorithms may be applied to identify clusters within a set of records and / or to assign records (e.g., newly received or generated records) to already identified clusters. For example, decision trees, artificial neural networks, k-means, support vector machines, independent component analysis, principal component analysis, or other methods may be trained based on a set of available records to generate an ML model for classifying the available records and / or for classifying records not present in the training set of available records. Inputs to such ML models may include various features of the records. Such features may be present within the records (e.g., date and time, status flag, user ID) and / or may be determined from information already present in the records (e.g., word vectors, paragraph vectors). Input features may include date and time or other numerical information associated with the records. Input features may include categorical information such as a user ID number or a status flag (e.g., “open,” “closed-resolved,” “closed-unresolved”). Input features may include information related to the textual information of the records (e.g., the “problem description” field). For example, the input features may relate to word and / or paragraph vectors generated from text fields of a record and / or other features generated using natural language processing. The input features may undergo mapping (e.g., nonlinear transformations, dimensionality reduction) before being applied to the ML model.

[0162] 11A shows an example set of records 1100. For purposes of illustrating record clustering, each record is represented by a distinct location in a two-dimensional space. The location of a given record in space may be related to the values ​​of two features of the given record (e.g., the creation time of the record and the resolution time of the record, two dimensions of a paragraph vector generated from the text of the record). Alternatively, the location of a given record may be related to the projection of three or more features of the record into the two-dimensional space using linear or nonlinear dimensionality reduction techniques or other mapping methods.

[0163] Similarity values ​​may be determined for pairs of records in the set of records 1100. These similarity values ​​may be used to generate an ML model for clustering the records 1100 so that records that are “similar” in some sense are assigned to the same cluster, while records that are very “dissimilar” are assigned to different clusters. Such similarity values ​​may correspond to a distance measure between the records in some space, such as the two-dimensional space of FIG. 11A , a semantically encoded vector space related to word and / or paragraph vectors determined from the textual aspects of the records, a vector space including dimensions related to the generation time of the records or other numerical and / or categorical information of the records, etc. Such distance may be Euclidean distance, Manhattan distance, or some other distance measure according to the application.

[0164] 11B shows each record in set of records 1100 assigned to one of three clusters 1110A, 1120A, 1130A identified from set of records 1100. Each record's assignment is indicated by the shape used to indicate the record: a filled circle for the first cluster 1110A, an open circle for the second cluster 1120A, and a square for the third cluster 1130A. Note that all of the records are assigned to clusters, and in fact the ML model used to assign records to clusters may optionally assign the record to the remaining set of records or take some other action instead of assigning the record to a cluster. This may be done, for example, to avoid assigning a record to a cluster if the similarity between the record and the potential cluster is below a minimum degree of similarity.

[0165] As described above, various methods can be used to generate an ML model that assigns records to two or more clusters and / or to the set of remaining unassigned records. Once the ML model is determined, it can be applied to assign additional records to the identified clusters represented by the ML model and / or to assign records to the set of remaining records. The ML model can include parameter values, neural network hyperparameters, cluster centroid locations in mechanism space, cluster boundary locations in mechanism space, threshold similarity values, or other information used by the ML model to determine which cluster to assign a record to and / or to determine that a record should not be assigned to a cluster (e.g., should be stored in the set of remaining unassigned records). Such information can define a region in mechanism space corresponding to each cluster. That is, the information in the ML model can be such that the ML model assigns a record to a particular cluster if the record's features correspond to a location in mechanism space within the region defined for that particular cluster. The defined region can be closed (completely surrounded by a boundary) or open (having one or more boundaries but extending outward indefinitely in one or more directions in mechanism space).

[0166] 11C illustrates such regions 1110B, 1120B, 1130B in the example two-dimensional mechanism space of FIG. 11A. Records in set of records 1100 lie within regions 1110B, 1120B, 1130B and therefore would be assigned by the ML model to clusters corresponding to regions 1110B, 1120B, 1130B. Records with locations outside regions 1110B, 1120B, 1130B may be assigned to the set of remaining records. The ML model may include information indicating the location, shape, extent, or other information for a spline, hyperplane, cell, hypersphere, or other shape that explicitly defines the boundaries and / or extent of regions 1110B, 1120B, 1130B in mechanism space that correspond to the clusters represented by the ML model. Additionally or alternatively, the ML model may include network weight parameters, output unit scaling parameters, or other information about the neural network structure or other algorithm that indirectly define the boundaries and / or ranges of the regions 1110B, 1120B, 1130B.

[0167] In some examples, the ML model may include a centroid or other location information that indicates the location of the cluster in mechanism space. The centroid may be the location of the arithmetic or geometric mean of the locations of the records in the cluster, the determined geometric center, or other defining location of a hypersphere, hyperellipsoid, or other shape fitted to the records of the cluster, or some other location related to the overall location and / or extent of the cluster in mechanism space. In such examples, a record may be assigned to a particular cluster if its location in mechanism space is closer to the centroid of that cluster than it is to the centroid of any other cluster. An ML model organized in this manner may include a k-means classifier. As an illustrative example, FIG. 11D includes three centroids 1115C, 1125C, and 1135C in mechanism space. Each centroid corresponds to a separate cluster. A record from set of records 1100 may be assigned to a cluster based on which of the centroids 1115C, 1125C, and 1135C is closest to the record's location in mechanism space. Under such an arrangement, records would be assigned to clusters based on whether the record's location is within an open area 1110C, 1120C, or 1130C.

[0168] In some examples, a record may be excluded from assignment to a particular cluster unless the degree of similarity between the cluster and the record is greater than a similarity threshold. This may include the distance between the record's location and the cluster's centroid or other characteristic location being less than a threshold distance. Records excluded from inclusion in any cluster may be added to the set of remaining records. As an illustrative example, FIG. 11E includes three centroids 1115D, 1125D, and 1135D in mechanism space. Each centroid corresponds to a separate cluster. A record from the set of records 1100 may be assigned to a cluster based on which of the centroids 1115D, 1125D, and 1135D is closest to the record's location in mechanism space and less than a threshold distance d from the record's location. Under such an arrangement, a record would be assigned to a cluster based on which of the closed regions 1110D, 1120D, and 1130D the record's location falls within. Note that the threshold distance may vary between clusters. For example, the threshold distance may be based on the variance of records assigned to each cluster during training of the ML model.

[0169] In some examples, the ML model may operate in a specified order to determine whether a record should be assigned to each cluster. For example, the ML model may first determine whether a record should be assigned to a first cluster (e.g., by comparing the distance between the centroid of the first cluster and the location of the record to a threshold distance). If it is determined that the record should not be assigned to the first cluster, the ML model may operate to determine whether the record should be assigned to a second cluster, and so on. Such a method may have the advantage of reducing the expected computational cost of assigning a record to a cluster (or determining that a record should not be assigned to any cluster). Additionally or alternatively, such a method may allow additional clusters to be added to the model without reallocating any region of mechanism space to the new cluster that was previously associated with any of the existing clusters. This may be done by placing the newly added cluster at the end of a sequence for determining whether a record should be assigned to any of the clusters.

[0170] As an illustrative example, Figure 11F includes three centroids 1115E, 1125E, and 1135E in mechanism space. Each centroid corresponds to a separate cluster. Records from the set of records 1100 may be assigned to clusters according to the specified sequence of clusters by determining whether the record's location in mechanism space is less than a threshold distance from the location of the cluster's centroid. Under such an arrangement, a record will be assigned to a cluster based on whether the record's location is within one of the closed regions 1110E, 1120E, or 1130E. Note that the threshold distance varies between clusters.

[0171] The clustering and ML model training operations described above may be distributed among elements of a remote network management platform, as described elsewhere herein. For example, an end-user computing instance within a remote network management platform may be dedicated to managing a particular user and configured to serve web pages, manage the generation and storage of incident reports, maintain and serve a set of knowledge base articles, manage the managed network's servers, or perform some other operation related to the maintenance and operation of the user's managed network.

[0172] An additional set of computational resources may be allocated for generating and / or applying ML models based on incident reports, knowledge base articles, or other text records related to end-user compute instances and / or the managed network. This distribution of tasks allows ML-related tasks, which may be memory- and computationally intensive, to be performed separately from other network management tasks (e.g., serving web pages, managing servers, generating incident reports), so as not to degrade the performance of those other network management tasks when generating or applying ML models. This distribution of tasks may also allow resources used to generate and apply ML models to be shared among different users of the remote network management platform to increase efficiency.

[0173] In some examples, a training computing instance may initially generate an ML model. This may be done in response to a request for such from an end-user computing instance. The training instance may subsequently access an incident report or other set of records associated with the end-user computing instance (e.g., by receiving such information from the end-user computing instance) and use that information to generate the ML model. The generated ML model may then be stored by the training instance for later use, transmitted to the end-user instance for storage / later use, and / or stored in some other location for later use.

[0174] Additional incident reports or other records may subsequently be assigned to clusters and / or added to the set of remaining records using the trained ML model. This may be done by the end user instance. Alternatively, a predictive computing instance within a remote network management platform may receive a request for such classification from an end user instance. Such a request may include an indication of the incident report or record to be classified. The predictive instance may then apply the ML model to assign the incident report or other record to a cluster or to the set of remaining records. The predictive instance may access the ML model from local memory or from a database that is not part of the predictive instance. For example, the predictive instance may determine that the associated ML model does not reside in the predictive instance's memory (e.g., hard drive, database), and in response to that determination, the predictive instance may send a request for a copy of the ML model to the end user instance. Once the predictive instance assigns an incident report or other record to a particular cluster, the predictive instance may send a representation of the particular cluster to the end user instance.

[0175] Transmitting a representation of one or more clusters of text records within a corpus of text records may include transmitting various information related to the represented cluster. This may include transmitting an identification number (e.g., a GUID), a location in a database, or other identifying information that may enable an end user instance to identify the represented cluster of text records. Additionally or alternatively, a representation of the content of the cluster of text records may be transmitted. This may include transmitting identifying information for the text records within the cluster (e.g., a GUID, a location in a database), all or part of the content of the text records within the cluster (e.g., text from the “problem resolution” field of incident reports within the cluster), the content of one or more representative text records associated with the cluster (e.g., a knowledge base article associated with the cluster, the “problem resolution” field, or other content of incident reports near the centroid of the cluster), or some other content associated with the cluster.

[0176] Transmitting a representation of one or more text records in the corpus of text records may include transmitting a variety of different information. In some examples, transmitting a representation of a text record includes transmitting an identification number (e.g., a globally unique identifier (GUID)), a date and time stamp, a location within a database, or some other identifying information that may enable an end user instance to identify or otherwise access the represented text record within a database. Additionally or alternatively, transmitting a representation of a text record may include transmitting a copy of the text record itself or a portion thereof. For example, transmitting a representation of a text record may include transmitting a copy of a “problem resolution” field of an incident report.

[0177] VIII. Text Query or Other Record Clustering Generating an ML model to identify related clusters of incident reports (or other records) within a training set of incident reports and using that ML model to later assign newly generated incident reports to the identified clusters may provide various advantages. However, the process of initially generating an ML model can be computationally expensive. Thus, an ML model may be used for an extended period of time after its initial generation. However, as new incident reports are generated over time, the effectiveness of the ML model may decrease. This may be related to the presence of new clusters of related incident reports within newly generated incident reports, changes in the properties of the identified clusters over time (e.g., movement within mechanism space), or other processes. Thus, an ML model may be regenerated, in whole or in part, based on newly generated incident reports, according to a specified schedule or other criteria.

[0178] The accuracy of the ML model between these updates may be further improved by performing partial updates based on residual incident reports that have not been assigned to clusters already identified in the ML model. These residual incident reports may have been added to the set of residual records due to their similarity to all of the identified clusters being lower than a similarity threshold. For example, all of the residual incident reports may be greater than the maximum distance in mechanism space from the centroid or other location of the identified clusters. These residual records may therefore represent unidentified clusters in the data and may be analyzed periodically between full regeneration of the ML model to determine whether additional clusters should be added to the ML model. This solution may have the advantage of being computationally less expensive than a full regeneration of the ML model, while still allowing the ML model to detect clusters that appear in newly generated incident reports.

[0179] 12A illustrates an example set of records 1200. For purposes of illustrating record clustering, each record is represented by a distinct location in two-dimensional space. Similarity values ​​can be determined for pairs of records in the set of records 1200 and used to generate an ML model for clustering the records 1200 such that records that are "similar" in some sense are assigned to the same cluster, while records that are very "dissimilar" are assigned to different clusters.

[0180] 12B illustrates each record in the set of records 1200 and a set of three clusters 1210, 1220, and 1230 identified from the set of records 1200. Subsets of the records (indicated by solid circles) are assigned to the clusters 1210, 1220, and 1230. For example, disjoint subsets of records 1200 are added to the set of remaining incident reports (indicated by open circles) because those records are less similar than a threshold degree of similarity to any of the identified clusters 1210, 1220, and 1230. As shown in FIG. 12B, this threshold degree of similarity may correspond to a maximum distance in the mechanism space, and the threshold level of similarity may vary between clusters.

[0181] Over time, additional incident reports may be generated. Figure 12C shows the original set 1200 of incident reports (indicated by the filled circles), the identified clusters 1210, 1220, 1230, and a set of newly generated incident reports (indicated by the filled boxes). An ML model has been applied to the additional incident reports (filled boxes) to assign some of the additional incident reports to corresponding clusters. Additional incident reports determined not to correspond to any of the identified clusters 1210, 1220, 1230 have been added to the set of remaining incident reports (open boxes).

[0182] Prior to fully regenerating the ML model, the set of residual incident reports may be analyzed to identify additional clusters not already present in the ML model. This analysis may occur according to a set schedule (e.g., every 15 minutes). Additionally or alternatively, the analysis to identify additional clusters from the set of residual incident reports may occur in response to some other criteria. For example, the analysis may occur in response to a specified number of additional incident reports being added to the set of residual incident reports.

[0183] Figure 12D shows an additional cluster 1240 identified from the set of remaining records (open circles / open squares) shown in Figure 12C. The incident reports from the set of remaining incident reports that correspond to the newly identified cluster 1240 have been removed from the set of remaining incident reports and assigned to the newly identified cluster 1240. Note that the incident reports assigned to the newly identified cluster 1240 may include both incident reports that were part of the original training set of incident reports and incident reports that were subsequently generated.

[0184] The algorithm used to identify additional clusters from the set of residual incident reports may be the same algorithm used to originally generate the ML model, or another method may be applied. The ML model may be extended to include the newly identified clusters. This extended ML model may be sent to a corresponding end-user computing instance (e.g., for distribution to other prediction and / or training instances, for purposes of protecting end-user data), or to some other database for later use by the prediction instance or by other systems or components.

[0185] The process described above may function to iteratively add clusters to the ML model based on the residual set of incident reports. As new clusters are identified, incident reports belonging to the new cluster are removed from the set of residual incident reports. Thus, the order in which incident reports may be applied to the ML model and thus added to the set of residual incident reports may affect the clusters identified through this method. To ensure that the same set of clusters is identified by multiple different prediction instances (or other instances performing these operations), the order of the incident reports may be recorded and made available for prediction instances.

[0186] For example, a prediction instance may determine that it has an outdated version of an ML model. The version of the ML model may be outdated because it does not represent all of the relevant incident reports that have been generated. In response to determining that its local version of the ML model is outdated, the prediction instance may request information from the end user instance about newly generated incident reports and the order in which those incident reports were received. The prediction instance may then apply the newly generated incident reports to update the outdated version of the ML model. Under such an update scheme, multiple different prediction instances may independently generate the same updated ML model.

[0187] The above methods may be applied to update an ML model based on incident reports that were not used to initially generate the ML model. Periodically, an ML model updated via these methods may be completely regenerated. This may be done according to a specified schedule or in response to other criteria (“refresh criteria”) to provide more accurate and up-to-date clusters or to provide other benefits. In some instances, it may be beneficial to retain one or more of the clusters identified from a previous ML model in some way. This may be done to preserve clusters that have become useful, are the focus of significant development efforts (e.g., to create knowledge base articles, resolution workflows, or other initiatives), are related to ongoing concerns or events (e.g., network or service outages), or are otherwise desirable. In such instances, a user may specify one or more clusters to be retained, and any information related to the specified clusters may be used to regenerate the ML model.

[0188] Generating an ML model to maintain one or more specified clusters from a previous model may include various processes. In some examples, corresponding clusters in the new ML model may be set according to the specified clusters. This may include defining corresponding clusters in the new ML model to have the same centroids, maximum distance thresholds, or other properties according to corresponding properties of the specified clusters from the previous ML model. Additionally or alternatively, the initial state or seed of one or more clusters in the new ML model may be set to the specified cluster's information so that the corresponding clusters in the new ML model may differ slightly from the specified clusters (e.g., to account for changes in properties of the underlying process associated with the specified clusters).

[0189] In some examples, a new ML model may be generated such that some or all of the members of a specified cluster remain within the same cluster. This may involve using a supervised model generation algorithm with the members of the specified cluster tagged as having the same "true" output classification. The ML model generation algorithm may then operate based on these tags and other information. This may be a hard requirement, requiring that all of the members of the specified cluster be commonly assigned to a single cluster by the new ML model. Alternatively, it may be a soft requirement, allowing some of the members of the specified cluster to be separated among multiple different clusters in the new ML model.

[0190] IX. Working Example 13 is a flowchart illustrating an exemplary embodiment. The process illustrated in FIG. 13 may be performed by a computing device, such as computing device 100, and / or a cluster of computing devices, such as server cluster 200. However, the process may be performed by other types of devices or device subsystems. For example, the process may be performed by a portable computer, such as a laptop or tablet device.

[0191] The embodiments of Figure 13 may be simplified by removing any one or more of the features shown therein. Additionally, these embodiments may be combined with features, aspects, and / or implementations of any of the previous figures and otherwise described herein.

[0192] The example method of Figure 13 includes receiving 1300 a first text record by a predictive computing instance and from an end-user computing instance, the end-user computing instance being dedicated to a managed network, and both the predictive computing instance and the end-user computing instance being deployed within a remote network management platform.

[0193] The example method of Figure 13 further includes applying 1300 the ML pipeline to the target dataset. The example method of Figure 13 further includes determining 1302 that the first text record corresponds to a particular cluster selected from the set of clusters of text records by an ML clustering model of the predictive computing instance that represents the set of clusters of text records. The example method of Figure 13 further includes transmitting 1304 a representation of the particular cluster by the predictive computing instance and to the end-user computing instance.

[0194] The example method of Figure 13 further includes receiving 1306 a second text record by the predictive computing instance and from the end-user computing instance. The example method of Figure 13 also includes determining 1308, by the ML clustering model, that the second text record does not correspond to any cluster of text records in the set of clusters of text records. The example method of Figure 13 further includes 1310, by the predictive computing instance, adding the second text record to the stored set of remaining text records in response to determining that the second text record does not correspond to any cluster of text records in the set of clusters of text records.

[0195] The example method of Figure 13 further includes identifying 1312, by the prediction computing instance, additional clusters of text records based on the stored set of residual text records. The example method of Figure 13 also includes transmitting 1314, by the prediction computing instance and to the end-user computing instance, a representation of the additional clusters of text records.

[0196] The example method of FIG. 13 may include additional or alternative steps.In some examples, the prediction computation instance is a first prediction computation instance, the ML clustering model is a first ML clustering model, and the example method of FIG. 13 includes: (i) receiving, by the first prediction computation instance and from the end user computation instance prior to receiving the first text record, a first plurality of text records; (ii) using a non-stochastic iterative algorithm by the first prediction computation instance to determine a first ML clustering model based on the first plurality of text records prior to receiving the first text record; (iii) receiving, by a second prediction computation instance and from the end user computation instance, a second text record; (iv) determining, by the second prediction computation instance, that the second prediction computation instance does not include an up-to-date ML clustering model corresponding to the end user computation instance; and (v) determining, in response to determining that the second prediction computation instance does not include an up-to-date ML clustering model corresponding to the end user computation instance, that the second prediction computation instance does not include an up-to-date ML clustering model corresponding to the end user computation instance. (vi) using a non-probabilistic iterative algorithm by a second predictive computing instance to determine a second ML clustering model representing a set of clusters of text records based on the first plurality of text records and the first text records, wherein using the non-probabilistic iterative algorithm to determine the second ML clustering model includes using each text record in the first plurality of text records and the first text record to update the second ML clustering model via the non-probabilistic iterative algorithm; (vii) determining by the second ML clustering model that the second text record corresponds to a particular cluster selected from the set of clusters of text records; and (viii) sending a representation of the particular cluster by the second predictive computing instance and to the end-user computing instance.Using a non-stochastic iterative algorithm to determine the first ML clustering model includes using each text record in the first plurality of text records to update the first ML clustering model via the non-stochastic iterative algorithm.

[0197] In some examples, determining by the ML clustering model that the first text record corresponds to a particular cluster selected from the set of clusters of text records includes at least one of: (i) using the ML model to determine word vectors in a first semantically-encoded vector space that describe the meaning of individual words of the first text record and comparing the word vectors to at least one of the locations or volumes in the first semantically-encoded vector space that corresponds to the particular cluster; or (ii) using the ML model to determine paragraph vectors in a second semantically-encoded vector space that describe the meaning of multiple words of the first text record and comparing the paragraph vectors to one of the locations or volumes in the second semantically-encoded vector space.

[0198] In some examples, the ML clustering model includes a distinct location in the vector space for each set of clusters of text records represented by the ML clustering model. In such examples, determining, by the ML clustering model, that a first text record corresponds to a particular cluster selected from the set of clusters of text records may include (i) determining, based on the first text record, a location of the first text record in the vector space, and (ii) determining that a distance in the vector space between the location of the first text record and the location of the particular cluster is less than a threshold distance.

[0199] 14 is a flowchart illustrating an exemplary embodiment. The process illustrated by FIG. 14 may be performed by a computing device, such as computing device 100, and / or a cluster of computing devices, such as server cluster 200. However, the process may be performed by other types of devices or device subsystems. For example, the process may be performed by a portable computer, such as a laptop or tablet device.

[0200] The embodiments of Figure 14 may be simplified by removing any one or more of the features shown therein. Additionally, these embodiments may be combined with features, aspects, and / or implementations of any of the previous figures and otherwise described herein.

[0201] The example method of Figure 14 includes obtaining a plurality of text records divided into clusters and a residual set of text records, where the ML clustering model has divided the plurality of text records based on a similarity metric (1400). The example method of Figure 14 further includes receiving a particular text record representing a query from a client device (1402). The example method of Figure 14 also includes determining (1404), via the ML clustering model and based on the similarity metric, that the particular text record does not fit into any of the clusters. The example method of Figure 14 further includes adding (1406) the particular text record to the residual set of text records in response to determining that the particular text record does not fit into any of the clusters. The example method of Figure 14 may include additional or alternative procedures. 14 further includes (i) receiving, from the client device, a second particular text record representing a second query; (ii) determining, via the ML clustering model, that the second particular text record matches a particular cluster of the clusters based on a similarity metric; and (iii) adding the second particular text record to the particular cluster in response to determining that the second particular text record matches a particular cluster of the clusters. In such examples, determining that the particular text record matches a particular cluster may include determining that the similarity metric indicates that the particular text record matches a particular cluster better than the particular text record matches all other clusters. Additionally or alternatively, determining that the particular text record matches a particular cluster may include determining that the similarity metric indicates that the particular text record matches a particular cluster to an extent that exceeds a specified threshold similarity.Additionally or alternatively, the ML clustering model may include a distinct location in the vector space for each cluster of text records, and determining that the second particular text record fits into the particular cluster may include (i) determining a location of the second text record in the vector space based on the second particular text record, and (ii) determining that the distance in the vector space between the location of the second text record and the location of the particular cluster is less than a threshold distance.

[0202] 14 may further include (i) identifying, via an ML clustering model, that the remaining set of text records includes an additional cluster, (ii) adding the additional cluster to the cluster of text records, and (iii) removing the constituent text records of the additional cluster from the remaining set of text records. In some examples, determining that the particular text record does not fit into any of the clusters may include at least one of (i) using an ML clustering model to determine word vectors in a first semantically-encoded vector space that describe the meaning of individual words of the particular text record and comparing the word vectors with at least one of the locations or volumes in the first semantically-encoded vector space that each correspond to a cluster, or (ii) using an ML model to determine paragraph vectors in a second semantically-encoded vector space that describe the meaning of multiple words of the particular text record and comparing the paragraph vectors with at least one of the locations or volumes in the second semantically-encoded vector space that each correspond to a cluster.

[0203] X. Conclusion The present disclosure should not be limited with respect to the particular embodiments described in this application, which are intended as illustrations of various aspects. As will be apparent to those skilled in the art, many modifications and variations can be made without departing from its scope. Functionally equivalent methods and apparatuses within the scope of the disclosure, in addition to those described herein, will be apparent to those skilled in the art from the foregoing description. Such modifications and variations are intended to be within the scope of the appended claims.

[0204] The above detailed description, with reference to the accompanying figures, describes various features and operations of the disclosed systems, devices, and methods. The exemplary embodiments described herein and in the figures are not meant to be limiting. Other embodiments may be utilized, and other changes may be made, without departing from the scope of the subject matter presented herein. It will be readily understood that aspects of the present disclosure, as generally described herein and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations.

[0205] With respect to any or all of the message flow diagrams, scenarios, and flowcharts in the figures, as discussed herein, each step, block, and / or communication may represent the processing of information and / or the transmission of information according to the exemplary embodiments. Alternative embodiments are included within the scope of these exemplary embodiments. In these alternative embodiments, for example, operations described as steps, blocks, transmissions, communications, requests, responses, and / or messages may be executed in an order different from that shown or discussed, including substantially simultaneously or in reverse order, depending on the functionality involved. Furthermore, more or fewer blocks and / or operations may be used with any of the message flow diagrams, scenarios, and flowcharts discussed herein, and these message flow diagrams, scenarios, and flowcharts may be combined with each other, either partially or in whole.

[0206] Steps or blocks representing the processing of information may correspond to circuitry that can be configured to perform specific logical functions of the methods or techniques described herein. Alternatively or additionally, steps or blocks representing the processing of information may correspond to modules, segments, or portions of program code (including associated data). The program code may include one or more instructions executable by a processor to implement specific logical operations or acts in the methods or techniques. The program code and / or associated data may be stored on any type of computer-readable medium, such as a storage device, including a RAM, a disk drive, a solid-state drive, or another storage medium.

[0207] Computer-readable media may also include non-transitory computer-readable media, such as computer-readable media that store data for a short period of time, such as register memory and processor cache. Computer-readable media may further include non-transitory computer-readable media that store program code and / or data for a longer period of time. Thus, computer-readable media may include secondary or permanent long-term storage, such as, for example, a ROM, an optical or magnetic disk, a solid-state drive, or a compact disk read-only memory (CD-ROM). Computer-readable media may also be any other volatile or non-volatile storage system. Computer-readable media may be considered, for example, a computer-readable storage medium or a tangible storage device.

[0208] Additionally, steps or blocks representing one or more information transmissions may correspond to information transmissions between software and / or hardware modules within the same physical device, however, other information transmissions may be between software and / or hardware modules in different physical devices.

[0209] The particular arrangement shown in the figures should not be considered limiting. It should be understood that other embodiments may include more or less of each element shown in a given figure. Furthermore, some of the described elements may be combined or omitted. Furthermore, example embodiments may include elements not described in the figures.

[0210] While various aspects and embodiments are disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for purposes of illustration and are not intended to be limiting, the true scope being indicated by the following claims.

Claims

1. a processor circuit; at least one non-transitory memory device accessible by the processor circuitry, which, when executed by the processor circuitry, receiving a first text record containing information about the incident; determining, by a machine learning (ML) clustering model, a first position of the first text record in a vector space based on a time when the incident of the first text record was created and a time when the incident of the first text record was resolved; determining, with the ML clustering model, respective distances in the vector space between the first position of the first text record and respective positions of a plurality of clusters of text records in the vector space; determining, by the ML clustering model, that the first text record is not associated with any of the plurality of clusters of text records based on the individual distances between the first location of the first text record and the individual locations of the plurality of clusters exceeding a threshold distance; in response to determining that the first text record is not associated with any of the plurality of clusters of text records, adding the first text record to a set of remaining records stored in the at least one non-transitory memory device; identifying a new cluster of text records to add to the plurality of clusters of text records in the vector space based on the number of text records in the set of remaining records exceeding a threshold number of text records; the at least one non-transitory memory device having stored thereon a program that causes the processor circuit to perform operations including: A system including:

2. 2. The system of claim 1, wherein the operations include extending, with a predictive computation instance, the ML clustering model to represent the new cluster of text records as one of the plurality of clusters of text records in the vector space.

3. The system of claim 1 , wherein the first text record is received from an end-user computing instance dedicated to a managed network.

4. The system of claim 3 , wherein the actions include sending to the end-user computing instance individual representations of the plurality of clusters and an additional representation of the new cluster of text records.

5. The system of claim 1 , wherein the processor circuitry executes predictive computation instances.

6. The system of claim 5 , wherein the prediction computing instance is configured to receive the ML clustering model from the end-user computing instance before receiving the first text record from the end-user computing instance.

7. Receiving a first text record containing information about an incident; determining, by a machine learning (ML) clustering model, a first position of the first text record in a vector space based on a time when the incident of the first text record was created and a time when the incident of the first text record was resolved; determining, with the ML clustering model, respective distances in the vector space between the first position of the first text record and respective positions of a plurality of clusters of text records in the vector space; determining, by the ML clustering model, that the first text record is not associated with any of the plurality of clusters of text records based on the individual distances between the first location of the first text record and the individual locations of the plurality of clusters exceeding a threshold distance; in response to determining that the first text record is not associated with any of the plurality of clusters of text records, adding the first text record to a set of remaining records stored in a memory; identifying a new cluster of text records to add to the plurality of clusters of text records in the vector space based on the number of text records in the set of remaining records exceeding a threshold number of text records; A computer-implemented method comprising:

8. 8. The computer-implemented method of claim 7, wherein the first text record is received from an end-user computing instance dedicated to a managed network, and the computer-implemented method is performed by a prediction computing instance, the prediction computing instance configured to receive the ML clustering model from the end-user computing instance before receiving the first text record from the end-user computing instance.

9. When executed by the processor circuitry, receiving a first text record containing information about the incident; determining, by a machine learning (ML) clustering model, a first position of the first text record in a vector space based on a time when the incident of the first text record was created and a time when the incident of the first text record was resolved; determining, with the ML clustering model, respective distances in the vector space between the first position of the first text record and respective positions of a plurality of clusters of text records in the vector space; determining, by the ML clustering model, that the first text record is not associated with any of the plurality of clusters of text records based on the individual distances between the first location of the first text record and the individual locations of the plurality of clusters exceeding a threshold distance; in response to determining that the first text record is not associated with any of the plurality of clusters of text records, adding the first text record to a set of remaining records stored in a memory; identifying a new cluster of text records to add to the plurality of clusters of text records in the vector space based on the number of text records in the set of remaining records exceeding a threshold number of text records; A tangible, non-transitory computer-readable recording medium storing a program that causes the processor circuit to perform the above.

10. The program, when executed, extending the ML clustering model to represent the new cluster of text records as one of the plurality of clusters of text records in the vector space with a predictive computation instance.

10. The tangible, non-transitory computer-readable storage medium of claim 9, which causes the processor circuit to perform the following:

11. the vector space includes a first semantically encoded vector space and a second semantically encoded vector space, and determining that the first text record is not associated with any of the plurality of clusters of text records includes: (i) using the ML clustering model to determine word vectors in the first semantically encoded vector space that describe the meaning of individual words of the first text record, and comparing the word vectors to at least one of the locations or volumes in the first semantically encoded vector space that individually correspond to the plurality of clusters of text records; or (ii) using the ML clustering model to determine paragraph vectors that describe the meaning of a plurality of words of the first text record in a second semantically encoded vector space, and comparing the paragraph vectors to at least one of the locations or volumes in the second semantically encoded vector space that individually correspond to the plurality of clusters of the text record; 10. The tangible, non-transitory computer-readable storage medium of claim 9, comprising at least one of:

12. The system of claim 1, wherein the ML clustering model includes cluster boundary positions that define individual regions corresponding to the individual positions in the vector space for each cluster of the plurality of clusters of text records.

13. The system described in claim 12, wherein the individual regions for each cluster of the plurality of clusters of text records include an open boundary having one or more boundaries extending outward in one or more directions within the vector space.

14. The system of claim 12, wherein the individual region for each cluster of the plurality of clusters of text records includes a closed boundary, the closed boundary completely surrounding the individual region.

15. The system described in claim 14, wherein the ML clustering model includes information indicating the shape of the closed boundary.

16. The system described in claim 1, wherein the individual positions of each cluster of the plurality of clusters correspond to a centroid comprising a geometric mean of the individual positions of each of the text records within each cluster of the plurality of clusters of text records.

17. The system of claim 1, wherein each cluster of the plurality of clusters of text records is arranged in a specified order, and the ML clustering model is configured to iterate through the specified order when it determines that the first text record is not associated with any of the plurality of clusters of text records.

18. The system of claim 1, including a training computation instance configured to generate the ML clustering model based on a set of incident reports associated with end-user computation instances.

19. The system of claim 1, wherein the operation includes, in response to identifying the new cluster of text records, removing each of the text records included in the new cluster of text records from the set of remaining records.

20. The operation is receiving a plurality of additional text records after receiving the first text record; applying the plurality of additional text records to the ML clustering model in a specified order to determine whether any of the plurality of additional text records are associated with any of the plurality of clusters of text records; The system of claim 1 , comprising:

Images