Determining the semantic similarity of a text based on its subsections

Abstract

To provide systems and methods to compare a target sample of text with a set of textual records.SOLUTION: Each textual record includes a sample of text and an indication of one or more segments of text within the sample of text. Semantic similarity values between the target sample of text and each of the textual records are determined. Determining a particular semantic similarity value between the target sample of text and a particular textual record of a corpus includes (i) determining individual semantic similarity values between the target sample of text and each of the segments of text indicated by the particular textual record, and (ii) generating the particular semantic similarity value between the target sample of text and the particular textual record on the basis of the individual semantic similarity values. A textual record is then selected on the basis of the semantic similarities.SELECTED DRAWING: Figure 11

Description

[Background technology]

[0001] Determining the similarity between or otherwise comparing samples of text is useful in a variety of applications. This can be done to identify one or more samples from a corpus of text samples that are similar in some respect to a subject sample of text. For example, the subject sample of text may include a user's description of an information technology problem, and the subject sample may be compared to articles in a database that describe various information technology topics and / or solutions to the information technology problem. By comparing the user's description of the problem to the articles in the database, one or more "most relevant" or otherwise similar articles can be provided to the user and / or engineer to efficiently guide the user and / or engineer to a solution. Summary of the Invention

[0002] Natural language processing or other methods can be used to compare the samples of text. This may be done to find patterns within the samples of text, to identify samples that are related to the query or otherwise similar to the subject sample of text, or to provide some other benefit. For example, a query may include a sample of text that describes a problem the user is experiencing. A similarity value can then be determined between the query text and the text of articles in the database, allowing the user to be provided with one or more articles that are most similar to the query. This comparison can improve the quality of results provided to the user and / or reduce the time the user spends before determining and implementing a solution to the problem.

[0003] However, when an article or other sample of text is large (e.g., relative to the query text), it can be difficult to compare the article to the query text in a way that produces useful results. For example, a longer article may contain subsections that are relevant to a particular query text, but also contain other parts that are not relevant. As a result, the influence of the irrelevant parts of the article may prevent such relevant articles from being detected.

[0004] To address this problem, articles or other large samples of text can be divided into segments. The segments can be determined according to sentences, paragraphs, or other punctuation within the large sample of text, using a segmentation algorithm trained on samples of query text, or according to some other method. This can be done so that the size of the segments of text is similar to the size of the queries to which they are likely to be compared. Similarity values ​​can then be created between the query text and each of the segments in each of the large samples of text. An overall similarity value for a particular large sample of text can then be determined based on the similarity values ​​for each of the text segments in that large sample of text. This can include summing the segment similarity values, determining a maximum segment similarity value, determining how many segment similarity values ​​exceed a threshold, or some other method. The overall similarity value of the large sample of text can then be used to select large samples of text that are related to the query text or to facilitate some other use.

[0005] Accordingly, a first exemplary embodiment may include a computer-implemented method including: (i) obtaining, by a server device, a corpus of text records, each of the text records including a sample of text and an indication of one or more segments of text within the sample of text, at least one of the text records including an indication of at least two segments of text within the sample of text; (ii) obtaining, by the server device, a subject sample of text from a client device; (iii) determining, by the server device, a semantic similarity value between the subject sample of text and each of the text records; (iv) based on the semantic similarity value, selecting, by the server device, a text record from the corpus having a semantic similarity indicating that the text record is more similar to the subject sample of text than any other text record; and (v) providing, by the server device, a representation of the text record to the client device. Determining a particular semantic similarity value between the subject sample of text and a particular text record of the corpus includes (i) determining individual semantic similarity values ​​between the subject sample of text and each of the segments of text represented by the particular text record, and (ii) generating a particular semantic similarity value between the subject sample of text and the particular text record based on the individual semantic similarity values.

[0006] In a second exemplary embodiment, an article of manufacture may include a non-transitory computer-readable medium having stored thereon program instructions that, when executed by a computing system, cause the computing system to perform operations in accordance with the first exemplary embodiment.

[0007] In a third exemplary embodiment, a computing system may include at least one processor, as well as a memory and program instructions that are stored in the memory and that, when executed by the at least one processor, cause the computing system to perform operations according to the first exemplary embodiment.

[0008] In the fourth exemplary embodiment, the system may include various means for performing each of the operations of the first exemplary embodiment.

[0009] In some embodiments, each of the text records includes a distinct indication of at least two segments of text within a distinct sample of text. That is, all of the text records may include multiple segments of text. Alternatively, one or more of the text records may include only a single segment of text. For example, a shorter sample of text may not be divided into multiple segments.

[0010] In some embodiments, determining an individual semantic similarity value between the subject sample of text and a particular segment of text in a particular text record comprises: (i) obtaining a vector representation of the subject sample of text, wherein the vector representation of the subject sample of text includes at least one of (a) word vectors describing the meaning of each distinct word of the subject sample of text in a first semantically encoded vector space, or (b) paragraph vectors describing the meaning of multiple words of the subject sample of text in a second semantically encoded vector space; (ii) obtaining a vector representation of the particular segment of text, wherein the vector representation of the particular segment of text includes at least one of (a) word vectors describing the meaning of each distinct word of the particular segment of text in the first semantically encoded vector space, or (ii) paragraph vectors describing the meaning of multiple words of the particular segment of text in the second semantically encoded vector space; and (iii) determining a semantic similarity value between the vector representation of the subject sample of text and the vector representation of the particular segment of text.

[0011] In some embodiments, generating a specific semantic similarity value between the target sample of text and the particular text record based on the individual semantic similarity values ​​includes (i) comparing each of the individual semantic similarity values ​​between the target sample of text and each of the segments of text represented by the particular text record to a threshold similarity level, and (ii) determining the number of individual semantic similarity values ​​that exceed the threshold similarity level as the specific semantic similarity value.

[0012] In some embodiments, generating a particular semantic similarity value between the subject sample of text and the particular text record based on the individual semantic similarity values ​​includes (i) generating a sum of the individual semantic similarity values ​​between the subject sample of text and each of the segments of text represented by the particular text record, and (ii) normalizing the sum to the number of segments of text represented by the particular text record.

[0013] In some embodiments, the indicators of one or more segments of text within each sample of text indicate non-overlapping segments of text.

[0014] In some embodiments, the indicators of one or more segments of text within each sample of text indicate segments of text that each represent one or more separate sentences.

[0015] In some embodiments, the method performed further includes: (i) receiving a plurality of queries from one or more client devices, each query including a sample of text; (ii) training a machine learning model based on the plurality of queries to predict associated segments of text within the sample of text; and (iii) applying the trained machine learning model to the corpus of text records to generate indicators of one or more segments of text within the sample of text of the corpus of text records.

[0016] These and other embodiments, aspects, advantages, and alternatives will become apparent to those skilled in the art upon reading the following detailed description, with appropriate reference to the accompanying drawings. Moreover, this summary and other descriptions and illustrations provided herein are intended to illustrate embodiments by way of example only, and therefore, many variations are possible. For example, structural elements and processing steps may be rearranged, combined, distributed, or removed, or otherwise modified while remaining within the scope of the claimed embodiments. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 1 is a schematic diagram of a computing device in accordance with an illustrative embodiment. [Figure 2] FIG. 1 is a schematic diagram of a server device cluster according to an example embodiment. [Figure 3] FIG. 1 illustrates a remote network management architecture in accordance with an example embodiment. [Figure 4] FIG. 1 illustrates a communication environment including a remote network management architecture in accordance with an exemplary embodiment. [Figure 5A] FIG. 1 illustrates another communication environment including a remote network management architecture in accordance with an example embodiment. [Figure 5B] 1 is a flowchart according to an example embodiment. [Figure 6] FIG. 10 illustrates an incident report in accordance with an exemplary embodiment. [Figure 7] FIG. 1 illustrates a database query architecture in accordance with an example embodiment. [Figure 8] FIG. 1 is an illustration of a sample text in accordance with an exemplary embodiment; [Figure 9A] FIG. 1 illustrates an artificial neural network (ANN) configured to learn the contextual meaning of words, according to an exemplary embodiment. [Figure 9B] FIG. 9B illustrates a set of training data for the ANN of FIG. 9A in accordance with an exemplary embodiment. [Figure 9C] FIG. 9B illustrates a set of training data for the ANN of FIG. 9A in accordance with an exemplary embodiment. [Figure 9D] FIG. 9B illustrates a set of training data for the ANN of FIG. 9A in accordance with an exemplary embodiment. [Figure 10A] FIG. 10 illustrates training an ANN for paragraph vectors, according to an example embodiment; [Figure 10B] FIG. 10 illustrates training an ANN for paragraph vectors, according to an example embodiment; [Figure 10C]FIG. 10 illustrates training an ANN for paragraph vectors, according to an example embodiment; [Figure 10D] FIG. 10 illustrates the use of a trained ANN to determine paragraph vectors for previously unseen paragraphs, according to an exemplary embodiment. [Figure 11] 1 is a flowchart according to an example embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0018] Exemplary methods, apparatus, and systems are described herein. The words "example" and "exemplary" should be understood as used herein to mean "serving as an example, instance, or illustration." Any embodiment or feature described herein as "example" or "exemplary" should not necessarily be construed as preferred or advantageous over other embodiments or features, unless so stated. Accordingly, other embodiments may be used, and other changes may be made, without departing from the scope of the subject matter presented herein.

[0019] Accordingly, the exemplary embodiments described herein are not intended to be limiting. It will be readily understood that aspects of the present disclosure, as generally described and illustrated herein, may be arranged, substituted, combined, separated, and designed in a variety of different forms. For example, the separation of features into "client" and "server" components may occur in many ways.

[0020] Furthermore, unless the context suggests otherwise, features shown in each figure may be used in combination with one another. Thus, the figures should generally be considered as component aspects of one or more overall embodiments, with the understanding that not all illustrated features are required for each embodiment.

[0021] Furthermore, recitation of elements, blocks, or steps in the specification or claims is for purposes of clarity, and therefore, such recitation should not be construed as requiring or implying that these elements, blocks, or steps be fixed in a particular arrangement or performed in a particular order.

[0022] I. Introduction Large companies are complex entities that include many interrelated activities. Some of these are found across the enterprise, such as human resources (HR), supply chain, information technology (IT), and finance. However, each company also has its own unique activities that provide essential functions and / or create competitive advantage.

[0023] To support commonly performed operations, enterprises typically use off-the-shelf software applications, such as customer relationship management (CRM) and human capital management (HCM) packages. However, enterprises may also require custom software applications to meet their unique requirements. Large enterprises often have dozens to hundreds of such custom software applications. Nevertheless, the benefits provided by embodiments herein are not limited to large enterprises, but may be applied to enterprises of any size or any other type of organization.

[0024] Many such software applications are developed by individual departments within an enterprise. These range from simple spreadsheets to custom software tools and databases. However, the proliferation of siloed, custom software applications has many drawbacks. It negatively impacts an enterprise's ability to operate, grow, innovate, and meet regulatory requirements. Without a single system that integrates its subsystems and data, an enterprise can find it difficult to integrate, streamline, and enhance its operations.

[0025] To efficiently create custom applications, businesses would benefit from a remotely hosted application platform that eliminates unnecessary development complexity. The goal of such a platform is to reduce time-consuming, repetitive application development tasks, allowing software engineers and individuals in other roles to focus on developing unique, high-value functionality.

[0026] To achieve this goal, the concept of Application Platform as a Service (aPaaS) has been introduced to intelligently automate workflows across an enterprise. aPaaS systems are hosted remotely from the enterprise but can access data, applications, and services within the enterprise via a secure connection. Such aPaaS systems can have several advantageous capabilities and features that have the potential to improve enterprise operations and workflows related to IT, HR, CRM, customer service, application development, and security.

[0027] aPaaS systems can support the development and execution of Model-View-Controller (MVC) applications. MVC applications separate their functionality into three interconnected parts (model, view, and controller) to separate the representation of information from how that information is presented to the user, thereby enabling efficient code reuse and parallel development. These applications can be web-based and can provide create, read, update, and delete (CRUD) capabilities, making it possible to build new applications on a common application infrastructure.

[0028] An aPaaS system can support standardized application components, such as a standardized set of widgets for graphical user interface (GUI) development. In this way, applications built using an aPaaS system have a common look and feel. Other software components and modules may also be standardized. In some cases, this look and feel may be branded or skinned with a company's custom logo and / or color scheme.

[0029] aPaaS systems can support the ability to configure application behavior using metadata, allowing application behavior to be quickly adapted to specific needs. Such an approach reduces development time and increases flexibility. Additionally, aPaaS systems support GUI tools that make it easier to create and manage metadata, thereby reducing errors in the metadata.

[0030] aPaaS systems support well-defined interfaces between applications, allowing software developers to avoid unwanted inter-application dependencies. Thus, aPaaS systems can implement a service layer where persistent state information and other data are stored.

[0031] aPaaS systems support a rich set of integration capabilities, allowing applications on the system to interact with legacy and third-party applications. For example, an aPaaS system can support a custom employee onboarding system that integrates with legacy HR, IT, and accounting systems.

[0032] aPaaS systems can support enterprise-level security. Because aPaaS systems may be remotely hosted, they should also utilize security procedures when interacting with systems within the enterprise or with third-party networks and services hosted outside the enterprise. For example, aPaaS systems can be configured to share data between the enterprise and other parties to detect and identify common security threats.

[0033] An aPaaS system may have other features, functionality, and advantages; this description is for illustrative purposes and not intended to be limiting.

[0034] As an example of an aPaaS development process, a software developer may be tasked with creating a new application using an aPaaS system. First, the developer may define a data model, thereby specifying the types of data the application will use and the relationships between them. Then, through the aPaaS system's GUI, the developer inputs (e.g., uploads) the data model. The aPaaS system automatically creates all of the corresponding database tables, fields, and relationships, which can be accessed through an object-oriented services layer.

[0035] Additionally, aPaaS systems can build fully functional MVC applications with client-side interfaces and server-side CRUD logic. This generated application can serve as a foundation for further development for users. Advantageously, developers do not need to spend a significant amount of time on basic application functionality. Furthermore, applications can be web-based and therefore accessible from any internet-enabled client device. Alternatively or additionally, a local copy of the application may be accessible when, for example, internet service is unavailable.

[0036] aPaaS systems can also support a rich set of predefined functionality that can be added to applications, including support for search, email, templates, workflow design, reporting, analytics, social media, scripting, mobile-friendly output, and customized GUIs.

[0037] The following embodiments describe the architecture and functional aspects of an exemplary aPaaS system, as well as its features and advantages.

[0038] II. Exemplary Computing Devices and Cloud-Based Computing Environments 1 is a simplified block diagram illustrating a computing device 100, showing some of the components that may be included in a computing device configured to operate in accordance with embodiments herein. Computing device 100 may be a client device (e.g., a device actively operated by a user), a server device (e.g., a device that provides computing services to client devices), or some other type of computing platform. Some server devices may act as client devices at times to perform certain operations, and some client devices may incorporate server functionality.

[0039] In this example, computing device 100 includes a processor 102, memory 104, a network interface 106, and input / output units 108, all of which may be coupled by a system bus 110 or similar mechanism. In some embodiments, computing device 100 may include other components and / or peripheral devices (e.g., removable storage, printers, etc.).

[0040] The processor 102 may be one or more of any type of computer processing element, such as a central processing unit (CPU), a coprocessor (e.g., a mathematical, graphics, or encryption coprocessor), a digital signal processor (DSP), a network processor, and / or in the form of an integrated circuit or controller that performs processor operations. In some cases, the processor 102 may be one or more single-core processors. In other cases, the processor 102 may be one or more multi-core processors having multiple independent processing units. The processor 102 may also include register memory for temporarily storing instructions and associated data being executed, and cache memory for temporarily storing recently used instructions and data.

[0041] Memory 104 may be any form of computer-usable memory, including, but not limited to, random access memory (RAM), read-only memory (ROM), and non-volatile memory (e.g., flash memory, hard disk drives, solid state drives, compact discs (CDs), digital video discs (DVDs), and / or tape storage). Memory 104 thus represents both primary and long-term storage. Other types of memory may include biological memory.

[0042] The memory 104 may store program instructions and / or data on which the program instructions may operate. By way of example, the memory 104 may store these program instructions on a non-transitory computer-readable medium such that these instructions are executable by the processor 102 to perform any method, process, or operation disclosed herein or in the accompanying drawings.

[0043] As shown in FIG. 1 , memory 104 may include firmware 104A, kernel 104B, and / or applications 104C. Firmware 104A may be program code used to boot or otherwise activate part or all of computing device 100. Kernel 104B may be an operating system, including modules for memory management, process scheduling and management, input / output, and communications. Kernel 104B may also include device drivers that allow the operating system to communicate with hardware modules (e.g., memory units, networking interfaces, ports, and buses) of computing device 100. Applications 104C may be one or more user-space software programs, such as a web browser or email client, and any software libraries used by these programs. Memory 104 may also store data used by these and other programs and applications.

[0044] Network interface 106 may take the form of one or more wired interfaces, such as Ethernet (e.g., Fast Ethernet, Gigabit Ethernet). Network interface 106 may also support communication over one or more non-Ethernet media, such as coaxial cable or power line, or over wide area media, such as Synchronous Optical Network (SONET) or Digital Subscriber Line (DSL) technology. Network interface 106 may also take the form of one or more wireless interfaces, such as IEEE 802.11 (Wi-Fi), BLUETOOTH, Global Positioning System (GPS), or wide area wireless interfaces. However, other forms of physical layer interfaces and other types of standard or proprietary communication protocols may be used via network interface 106. Furthermore, network interface 106 may include multiple physical interfaces. For example, some embodiments of computing device 100 may include Ethernet, BLUETOOTH, and Wi-Fi interfaces.

[0045] The input / output unit 108 may facilitate interaction between the computing device 100 and users and peripheral devices. The input / output unit 108 may include one or more types of input devices, such as a keyboard, a mouse, a touchscreen, etc. Similarly, the input / output unit 108 may include one or more types of output devices, such as a screen, a monitor, a printer, and / or one or more light-emitting diodes (LEDs). Additionally or alternatively, the computing device 100 may communicate with other devices using, for example, a universal serial bus (USB) or a high-definition multimedia interface (HDMI) port interface.

[0046] In some embodiments, one or more computing devices, such as computing device 100, may be deployed to support an aPaaS architecture. The exact physical location, connectivity, and configuration of these computing devices may be unknown and / or unimportant to the client devices. Thus, the computing devices are sometimes referred to as "cloud-based" devices that may be housed in various remote data center locations.

[0047] 2 illustrates a cloud-based server cluster 200 according to an exemplary embodiment. In FIG. 2, the operations of a computing device (e.g., computing device 100) may be distributed among server devices 202, data storage 204, and routers 206, all of which may be connected by a local cluster network 208. The number of server devices 202, data storage 204, and routers 206 in server cluster 200 may depend on the computing tasks and / or applications assigned to server cluster 200.

[0048] For example, server device 202 may be configured to perform various computing tasks for computing device 100. Accordingly, computing tasks may be distributed among one or more server devices 202. To the extent that these computing tasks can be performed in parallel, such distribution of tasks may reduce the total time to complete these tasks and return results. For simplicity, both server cluster 200 and individual server devices 202 may be referred to as "server devices." This naming should be understood to suggest that one or more separate server devices, data storage devices, and cluster routers may be involved in server device operations.

[0049] Data storage 204 may be a data storage array including a drive array controller configured to manage read and write access to a group of hard disk drives and / or solid state drives. The drive array controller, alone or in conjunction with server device 202, may also be configured to manage backup or redundant copies of data stored in data storage 204 to protect against drive failures or other types of failures that prevent one or more server devices 202 from accessing units of data storage 204. Other types of memory besides drives may also be used.

[0050] Router 206 may include network equipment configured to provide internal and external communications for server cluster 200. For example, router 206 may include one or more packet switching and / or routing devices (including switches and / or gateways) configured to provide (i) network communications between server device 202 and data storage 204 via local cluster network 208, and / or (ii) network communications between server cluster 200 and other devices via communication link 210 to network 212.

[0051] Additionally, the configuration of router 206 may be based, at least in part, on the data communication requirements of server device 202 and data storage 204, the latency and throughput of local cluster network 208, the latency, throughput, and cost of communication link 210, and / or other factors that may contribute to cost, speed, fault tolerance, resilience, efficiency, and / or other design goals of the system architecture.

[0052] As a possible example, data storage 204 may include any form of database, such as a Structured Query Language (SQL) database. Information may be stored in such a database using various types of data structures, including, but not limited to, tables, arrays, lists, trees, and tuples. Furthermore, any database in data storage 204 may be monolithic or distributed across multiple physical devices.

[0053] Server device 202 may be configured to send and receive data to and from data storage 204. This sending and retrieval may take the form of SQL queries or other types of database queries, and the output of such queries, respectively. Additional text, images, video, and / or audio may also be included. Furthermore, server device 202 may organize the received data into web page representations. Such representations may take the form of a markup language, such as Hypertext Markup Language (HTML), Extensible Markup Language (XML), or some other standardized or proprietary format. Furthermore, server device 202 may be capable of executing various types of computerized scripting languages, such as, but not limited to, Perl, Python, PHP Hypertext Preprocessor (PHP), Active Server Pages (ASP), JavaScript, and the like. Computer program code written in these languages ​​may facilitate the provision of web pages to client devices and their interaction.

[0054] III. Exemplary Remote Network Management Architecture 3 illustrates a remote network management architecture according to an example embodiment. The architecture includes three main components: a managed network 300, a remote network management platform 320, and a third-party network 340, all connected via the Internet 350.

[0055] Managed network 300 may be, for example, an enterprise network used by an entity for computing and communication tasks and data storage. Thus, managed network 300 may include client devices 302, server devices 304, routers 306, virtual machines 308, firewalls 310, and / or proxy servers 312. Client devices 302 may be embodied by computing devices 100, server devices 304 may be embodied by computing devices 100 or server clusters 200, and routers 306 may be any type of router, switch, or gateway.

[0056] Virtual machine 308 may be embodied by one or more of computing devices 100 or server cluster 200. Generally, a virtual machine is an emulation of a computing system that mimics the functionality (e.g., processor, memory, and communication resources) of a physical computer. A single physical computing system, such as server cluster 200, can support up to thousands of individual virtual machines. In some embodiments, virtual machines 308 may be managed by a centralized server device or application that facilitates allocation of physical computing resources to individual virtual machines, as well as performance and error reporting. Enterprises often use virtual machines to efficiently allocate computing resources as needed. Providers of virtualized computing systems include VMWARE® and MICROSOFT®.

[0057] Firewall 310 may be one or more dedicated router or server devices that protect managed network 300 from unauthorized attempts to access devices, applications, and services therein while allowing authorized communications initiated from managed network 300. Firewall 310 may also provide intrusion detection, web filtering, virus scanning, application layer gateways, and other applications or services. In some embodiments not shown in FIG. 3, managed network 300 may include one or more virtual private network (VPN) gateways used to communicate with remote network management platform 320 (see below).

[0058] The managed network 300 may also include one or more proxy servers 312. One embodiment of the proxy server 312 may be a server device that facilitates communication and movement of data between the managed network 300, the remote network management platform 320, and the third-party network 340. In particular, the proxy server 312 may be capable of establishing and maintaining secure communication sessions with one or more computing instances of the remote network management platform 320. Such sessions may enable the remote network management platform 320 to discover and manage aspects of the architecture and configuration of the managed network 300 and its components. In some cases, with the aid of the proxy server 312, the remote network management platform 320 may also be capable of discovering and managing aspects of the third-party network 340 used by the managed network 300.

[0059] A firewall, such as firewall 310, typically denies all communication sessions coming in via Internet 350 unless such sessions are ultimately initiated from behind the firewall (i.e., from a device on managed network 300) or the firewall is explicitly configured to support the sessions. By locating proxy server 312 behind firewall 310 (e.g., within managed network 300 and protected by firewall 310), proxy server 312 may be able to initiate these communication sessions through firewall 310. Thus, firewall 310 does not need to be specifically configured to support incoming sessions from remote network management platform 320, thereby avoiding a potential security risk to managed network 300.

[0060] In some cases, managed network 300 may consist of just a few devices and a small number of networks. In other deployments, managed network 300 may span multiple physical locations and include hundreds of networks and hundreds of thousands of devices. Thus, the architecture shown in Figure 3 can be scaled by orders of magnitude.

[0061] Furthermore, depending on the size, architecture, and connectivity of the managed network 300, various numbers of proxy servers 312 may be deployed therein. For example, each of the proxy servers 312 may be responsible for communications with the remote network management platform 320 for a portion of the managed network 300. Alternatively or additionally, a set of two or more proxy servers may be assigned to such portion of the managed network 300 for load balancing, redundancy, and / or high availability purposes.

[0062] Remote network management platform 320 is a hosted environment that provides aPaaS services to users, particularly operators of managed network 300. These services may take the form of, for example, a web-based portal. Thus, users may securely access remote network management platform 320 from, for example, client device 302, or potentially from client devices external to managed network 300. Through the web-based portal, users may design, test, and deploy applications, generate reports, view analytics, and perform other tasks.

[0063] As shown in FIG. 3 , remote network management platform 320 includes four compute instances: 322, 324, 326, and 328. Each of these instances may represent one or more server devices and / or one or more databases that provide a set of web portals, services, and applications (e.g., a fully functional aPaaS system) available to a particular customer. In some cases, a single customer may use multiple compute instances. For example, managed network 300 may be an enterprise customer of remote network management platform 320 and may use compute instances 322, 324, and 326. The reason for providing multiple instances to a single customer is that the customer may wish to independently develop, test, and deploy its applications and services. Thus, compute instance 322 may be dedicated to developing applications related to managed network 300, compute instance 324 may be dedicated to testing those applications, and compute instance 326 may be dedicated to live operation of the tested applications and services. A compute instance may be referred to as a host instance, a remote instance, a customer instance, or some other name. Any application deployed on a compute instance may be a scoped application in that access to a database within the compute instance may be limited to specific elements therein (e.g., one or more specific database tables or specific rows with one or more database tables).

[0064] For clarity, this disclosure refers to the physical hardware, software, and their arrangement as a "compute instance." Note that the graphical user interface provided thereby by a user may also be colloquially referred to as an "instance." However, unless otherwise defined herein, a "compute instance" is a computing system arranged within remote network management platform 320.

[0065] The multi-instance architecture of the remote network management platform 320 contrasts with traditional multi-tenant architectures, which offer several advantages. In a multi-tenant architecture, data from different customers (e.g., businesses) is combined into a single database. These customers' data is isolated from one another, but that isolation is enforced by the software that runs the single database. As a result, a security breach in this system can affect the data of all customers, potentially creating additional risk, especially for entities subject to government, healthcare, and / or financial regulations. Furthermore, any database operation that affects one customer can potentially affect all customers that share that database. Thus, if there is an outage due to a hardware or software error, the outage affects all such customers. Similarly, if a database is upgraded to meet the needs of one customer, it becomes unavailable to all customers during the upgrade process. Often, such maintenance periods are lengthy due to the size of the shared database.

[0066] In contrast, a multi-instance architecture provides each customer with its own database on a dedicated computing instance. This prevents intermingling of customer data and allows each instance to be managed independently. For example, if one customer's instance experiences an outage due to an error or upgrade, other compute instances are not affected. Because the database contains only one customer's data, maintenance downtime is limited. Furthermore, the simpler design of a multi-instance architecture allows redundant copies of each customer database and instance to be deployed in a geographically diverse manner. This promotes high availability, allowing the live version of a customer's instance to be moved when a failure is detected or maintenance is being performed.

[0067] In some embodiments, remote network management platform 320 can include one or more central instances controlled by the entity operating the platform. Similar to the compute instances, the central instances can include any number of physical or virtual server and database devices. Such central instances can serve as repositories for data that can be shared among at least some of the compute instances. For example, definitions of common security threats that may occur on the compute instances, software packages commonly found on the compute instances, and / or an application store for applications that may be deployed to the compute instances can reside within the central instance. The compute instances can communicate with the central instance through well-defined interfaces to obtain this data.

[0068] To support multiple compute instances in an efficient manner, remote network management platform 320 can implement multiple of these instances on a single hardware platform. For example, if an aPaaS system is implemented on a server cluster, such as server cluster 200, it can run virtual machines that dedicate varying amounts of compute, storage, and communication resources to the instances. However, full virtualization of server cluster 200 may not be necessary, and other mechanisms may be used to isolate instances. In some examples, each instance may have a dedicated account and one or more dedicated databases on server cluster 200. Alternatively, compute instances 322 may span multiple physical devices.

[0069] In some cases, a single server cluster of remote network management platform 320 may support multiple independent enterprises. Additionally, as described below, remote network management platform 320 may include multiple server clusters deployed in geographically diverse data centers to facilitate load balancing, redundancy, and / or high availability.

[0070] Third-party network 340 may be a remote server device (e.g., multiple server clusters, such as server cluster 200) that can be used for outsourced computing, data storage, communication, and service hosting operations. These servers may be virtualized (i.e., the servers may be virtual machines). Examples of third-party network 340 may include AMAZON WEB SERVICES® and MICROSOFT® AZURE®. Similar to remote network management platform 320, multiple server clusters supporting third-party network 340 may be deployed in various geographic locations for purposes of load balancing, redundancy, and / or high availability.

[0071] Managed network 300 can deploy applications and services to its clients and customers using one or more of third-party networks 340. For example, if managed network 300 provides an online music streaming service, third-party network 340 can store the music files and provide the web interface and streaming functionality. In this way, the businesses in managed network 300 do not have to build and maintain their own servers for these operations.

[0072] The remote network management platform 320 may include modules that integrate with third-party networks 340 to expose the virtual machines and managed services therein to the managed network 300. The modules may enable users to request virtual resources and provide flexible reporting about the third-party networks 340. To establish this functionality, a user from the managed network 300 may first establish an account with the third-party networks 340 and request a set of related resources. The user may then enter the account information into the appropriate modules of the remote network management platform 320. These modules may then automatically discover manageable resources within the account and may also provide reports on usage, performance, and billing.

[0073] Internet 350 may represent a portion of the global Internet, but may alternatively represent a different type of network, such as a private wide-area or local-area packet-switched network.

[0074] 4 further illustrates a communication environment between managed network 300 and compute instance 322, introducing additional features and alternative embodiments. In FIG. 4, compute instance 322 is replicated across data centers 400A and 400B. These data centers are geographically separated from one another and may be located in different cities or even different countries. Each data center includes supporting equipment that facilitates communication with managed network 300 and remote users.

[0075] In data center 400A, network traffic to and from external devices flows through either VPN gateway 402A or firewall 404A. VPN gateway 402A may be peered with VPN gateway 412 of managed network 300 via security protocols such as Internet Protocol Security (IPSEC) or Transport Layer Security (TLS). Firewall 404A may be configured to allow access from authorized users, such as user 414 and remote user 416, and to deny access to unauthorized users. Through firewall 404A, these users can access compute instance 322 and possibly other compute instances. Load balancer 406A may be used to distribute traffic among one or more physical or virtual server devices hosting compute instance 322. Load balancer 406A can simplify user access by hiding the internal configuration of data center 400A (e.g., compute instance 322) from client devices. For example, if compute instance 322 includes multiple physical or virtual computing devices that share access to multiple databases, load balancer 406A can distribute network traffic and processing tasks across these computing devices and databases so that no computing device or database is significantly busier than the others. In some embodiments, compute instance 322 may include VPN gateway 402A, firewall 404A, and load balancer 406A.

[0076] Data center 400B may contain its own versions of the components in data center 400A. Thus, VPN gateway 402B, firewall 404B, and load balancer 406B may perform the same or similar operations as VPN gateway 402A, firewall 404A, and load balancer 406A, respectively. Furthermore, through real-time or near-real-time database replication and / or other operations, compute instances 322 may exist simultaneously in data centers 400A and 400B.

[0077] Data centers 400A and 400B as shown in Figure 4 can facilitate redundancy and high availability. In the configuration of Figure 4, data center 400A is active and data center 400B is passive. Thus, data center 400A handles all traffic to and from managed network 300 while versions of compute instances 322 in data center 400B are updated in near real time. Other configurations in which both data centers are active can also be supported.

[0078] If data center 400A fails or becomes unavailable to users in some manner, data center 400B can take over as the active data center. For example, a Domain Name System (DNS) server that associates the domain name of compute instance 322 with one or more Internet Protocol (IP) addresses of data center 400A can reassociate that domain name with one or more IP addresses of data center 400B. After this reassociation is complete (which may take less than a second or several seconds), users can access compute instance 322 through data center 400B.

[0079] FIG. 4 also illustrates possible configurations of managed network 300. As described above, proxy server 312 and user 414 can access compute instance 322 through firewall 310. Proxy server 312 can also access configuration items 410. In FIG. 4, configuration items 410 can refer to any or all of client device 302, server device 304, router 306, virtual machine 308, any applications or services running on them, and relationships between devices, applications, and services. Thus, the term “configuration item” can be shorthand for any physical or virtual device, or any application or service that is remotely discoverable or managed by compute instance 322, or relationships between discovered devices, applications, and services. Configuration items can be represented in a configuration management database (CMDB) of compute instance 322.

[0080] As mentioned above, VPN gateway 412 can provide a dedicated VPN to VPN gateway 402A. Such a VPN can be useful if there is a significant amount of traffic between managed network 300 and compute instance 322 or if security policies recommend or require the use of a VPN between these sites. In some embodiments, any devices in managed network 300 and / or compute instance 322 that communicate directly via the VPN are assigned public IP addresses. Other devices in managed network 300 and / or compute instance 322 can be assigned private IP addresses (e.g., IP addresses selected from the ranges 10.0.0.0 to 10.255.255.255 or 192.168.0.0 to 192.168.255.255, abbreviated as subnets 10.0.0.0 / 8 and 192.168.0.0 / 16, respectively).

[0081] IV. Exemplary Device, Application, and Service Discovery In order for the remote network management platform 320 to manage the devices, applications, and services of the managed network 300, the remote network management platform 320 can first determine what devices exist in the managed network 300, the configuration and operational state of those devices, the applications and services provided by those devices, and the relationships between the discovered devices, applications, and services. As mentioned above, each device, application, service, and relationship may be referred to as a configuration item. The process of defining configuration items in the managed network 300 is referred to as discovery and may be facilitated at least in part by the proxy server 312.

[0082] For purposes of embodiments herein, an "application" may refer to one or more processes, threads, programs, client modules, server modules, or any other software running on a device or group of devices. A "service" may refer to a high-level capability provided by multiple applications running on one or more devices working in conjunction with each other. For example, a high-level web service may include multiple web application server threads running on one device and accessing information from a database application running on another device.

[0083] 5A shows a logical representation of how configuration items can be discovered and how information related to discovered configuration items is stored. For simplicity, remote network management platform 320, third-party network 340, and Internet 350 are not shown.

[0084] 5A, the CMDB 500 and task list 502 are stored within the compute instance 322. The compute instance 322 can send discovery commands to the proxy server 312. In response, the proxy server 312 can send probes to various devices, applications, and services within the managed network 300. These devices, applications, and services can send responses to the proxy server 312, which can then provide information about the discovered configuration items to the CMDB 500 for storage. The configuration items stored in the CMDB 500 represent the environment of the managed network 300.

[0085] Task list 502 represents a list of activities that proxy server 312 performs on behalf of compute instance 322. Once discovery occurs, task list 502 is populated. Proxy server 312 repeatedly queries task list 502 to obtain the next task in the list and executes this task until task list 502 is empty or another stopping condition is reached.

[0086] To facilitate discovery, proxy server 312 may be configured with information about one or more subnets in managed network 300 that are reachable through proxy server 312. For example, proxy server 312 may be given the IP address range 192.168.0 / 24 as a subnet. Compute instance 322 may then store this information in CMDB 500 and place tasks in task list 502 for discovery of devices at each of these addresses.

[0087] 5A also illustrates devices, applications, and services within managed network 300 as configuration items 504, 506, 508, 510, and 512. As discussed above, these configuration items represent a set of physical and / or virtual devices (e.g., client devices, server devices, routers, or virtual machines), the applications running on them (e.g., web servers, email servers, databases, or storage arrays), the relationships between them, and services that include multiple individual configuration items.

[0088] Placing a task in task list 502 triggers or otherwise causes proxy server 312 to initiate discovery. Alternatively or additionally, discovery may be triggered manually or automatically based on a triggering event (e.g., discovery may begin automatically once a day at a specific time).

[0089] In general, discovery can proceed in four logical phases: scanning, classification, identification, and exploration. Each phase of discovery involves different types of probe messages being sent by proxy server 312 to one or more devices in managed network 300. Responses to these probes are received and processed by proxy server 312, and representations thereof may be sent to CMDB 500. Thus, each phase may result in more configuration items being discovered and stored in CMDB 500.

[0090] During the scan phase, the proxy server 312 can probe each IP address in a specified range for open Transmission Control Protocol (TCP) and / or User Datagram Protocol (UDP) ports to determine the device's general type. The presence of such open ports at an IP address can indicate that a particular application is running on the device assigned that IP address, thereby identifying the operating system used by the device. For example, if TCP port 135 is open, the device is likely running a WINDOWS® operating system. Similarly, if TCP port 22 is open, the device is likely running a UNIX® operating system, such as LINUX®. If UDP port 161 is open, the device may be able to be further identified via Simple Network Management Protocol (SNMP). Other possibilities exist. Once the presence of a device and its open ports at a particular IP address is discovered, these configuration items are stored in the CMDB 500.

[0091] During the classification phase, the proxy server 312 may further probe each discovered device to determine its operating system version. The probes used for a particular device are based on information collected about that device during the scan phase. For example, if a device is found with TCP port 22 open, a set of UNIX-specific probes may be used. Similarly, if a device is found with TCP port 135 open, a set of WINDOWS-specific probes may be used. In either case, an appropriate set of tasks may be placed in the task list 502 for the proxy server 312 to execute. These tasks may result in the proxy server 312 logging on or otherwise accessing information from the particular device. For example, if TCP port 22 is open, the proxy server 312 may be instructed to open a Secure Shell (SSH) connection to the particular device and obtain information about its operating system from a particular location in the file system. Based on this information, the operating system may be determined. As an example, a UNIX device with TCP port 22 open may be classified as AIX, HPUX, LINUX, MACOS, or SOLARIS. This classification information may be stored in the CMDB 500 as one or more configuration items.

[0092] During the identification phase, the proxy server 312 can determine specific details about the classified device. The probes used during this phase can be based on information collected about the particular device during the classification phase. For example, if the device is classified as LINUX®, a set of LINUX®-specific probes can be used. Similarly, if the device is classified as WINDOWS® 2012, a set of WINDOWS® 2012-specific probes can be used. As with the classification phase, an appropriate set of tasks can be placed in the task list 502 for the proxy server 312 to execute. As a result of these tasks, the proxy server 312 can read information from the particular device, such as basic input / output system (BIOS) information, serial number, network interface information, media access control addresses assigned to those network interfaces, and the IP address used by the particular device. This identification information can be stored as one or more configuration items in the CMDB 500.

[0093] During the exploration phase, the proxy server 312 may determine further details regarding the operational state of the classified device. The probes used during this phase may be based on information gathered about the particular device during the classification and / or identification phases. Again, an appropriate set of tasks may be placed in the task list 502 for the proxy server 312 to perform. These tasks may result in the proxy server 312 reading additional information from the particular device, such as processor information, memory information, a list of running processes (applications), etc. Again, the discovered information may be stored as one or more configuration items in the CMDB 500.

[0094] SNMP can be used to perform discovery on network devices such as routers. Discovery may determine additional subnets known to the router and the operational state of the router's network interfaces (e.g., active, inactive, queue lengths, number of dropped packets, etc.) instead of or in addition to determining a list of running processes or other application-related information. The IP addresses of the additional subnets may be candidates for further discovery procedures. Thus, discovery may proceed iteratively or recursively.

[0095] Once discovery is complete, a snapshot representation of each discovered device, application, and service is available in CMDB 500. For example, operating system versions, hardware configurations, and network configuration details for client devices, server devices, and routers in managed network 300, as well as the applications running on them, may be stored after discovery. This collected information may be presented to the user in a variety of ways to allow the user to view the hardware configuration and operational status of devices and the characteristics of services across multiple devices and applications.

[0096] Additionally, CMDB 500 may include entries about dependencies and relationships between configuration items. More specifically, applications running on a particular server device and services that depend on the applications may be represented as such in CMDB 500. For example, assume that a database application runs on a server device and is used by a new employee onboarding service and a payroll service. Therefore, if the server device is taken out of service for maintenance, it is clear that the employee onboarding service and the payroll service will be affected. Similarly, dependencies and relationships between configuration items may be able to represent services that are affected when a particular router fails.

[0097] Generally, dependencies and relationships between configuration items may be displayed on a web-based interface and represented in a hierarchical manner, and thus adding, modifying, or deleting such dependencies and relationships may be accomplished through this interface.

[0098] Additionally, users of managed network 300 can develop workflows that allow specific collaborative activities to occur across multiple discovered devices. For example, an IT workflow may allow a user to change a common administrator password for all discovered LINUX devices in a single operation.

[0099] To perform discovery in the manner described above, proxy server 312, CMDB 500, and / or one or more credential stores may be configured with credentials for one or more devices to be discovered. Credentials may include any type of information needed to access a device. These may include user ID / password pairs, certificates, etc. In some embodiments, these credentials may be stored in encrypted fields in CMDB 500. Proxy server 312 may contain a decryption key for the credentials so that proxy server 312 can use these credentials to log on to or otherwise access the devices to be discovered.

[0100] The discovery process is shown as a flowchart in FIG. 5B. At block 520, a task list within a compute instance is populated with, for example, a range of IP addresses. At block 522, a scanning phase occurs, whereby the proxy server uses these IP addresses to probe the IP addresses of devices in an attempt to determine the operating systems running on those devices. At block 524, a classification phase occurs, whereby the proxy server attempts to determine the operating system versions of the discovered devices. At block 526, an identification phase occurs, whereby the proxy server attempts to determine the hardware and / or software configurations of the discovered devices. At block 528, a probing phase occurs, whereby the proxy server attempts to determine the operational state and applications running on the discovered devices. At block 530, further editing of the configuration items representing the discovered devices and applications can occur. This editing can be automatic and / or manual in nature.

[0101] The blocks depicted in Figure 5B are for illustrative purposes. Discovery can be a highly configurable procedure that may have more or fewer phases, and the operation of each phase may differ. In some cases, one or more phases may be customized or may otherwise deviate from the exemplary description above.

[0102] V. Natural Language Processing of Text Queries Natural language processing is a field that involves, among other activities, using computers to understand the structure and meaning of human language. This determined structure and meaning may be applicable to the processing of IT incidents, as described below.

[0103] Each incident may be represented as an incident report, which may exist in a variety of formats and contain a variety of types of information, with an exemplary incident report 600 shown in Figure 6. Incident report 600 consists of multiple fields in the left column, at least some of which are associated with values ​​in the right column.

[0104] Field 602 identifies the originator of the incident, in this case Bob Smith. Field 604 identifies the time the incident was created, in this case 9:56 AM on February 7, 2018. Field 605 is a text string providing a short description of the problem. Field 606 identifies the problem description provided by the originator. Thus, field 606 can be a free-form text string of a few words to a few sentences or longer. Field 608 is the categorization of the incident, in this case email. This categorization can be provided by the originator, the IT person assigned to the incident, or automatically based on the context of the problem description field.

[0105] Field 610 identifies the IT person (if applicable) to whom the incident is assigned, in this case, Alice Jones. Field 612 identifies the status of the incident. The status can be, for example, "open," "assigned," "working," or "resolved." Field 614 identifies how the incident was resolved (if applicable). This field can be filled out by the IT person assigned to the incident or another individual. Field 616 identifies the time the incident was resolved, in this case, 10:10 AM on February 7, 2018. Field 618 specifies the incident's closure code (if applicable), which can take values ​​such as "Closed (Permanent)," "Closed (Workaround)," or "Closed (Cannot Reproduce)." Field 620 identifies any additional notes added to the record by the IT person assigned to the incident or others. Field 622 identifies a link to an online article that may help the user avoid having to deal with a similar problem in the future.

[0106] Incident report 600 is presented for illustrative purposes. Other types of incident reports may be used, and these reports may include more, fewer, and / or different fields.

[0107] Incident reports such as incident report 600 may be created in a variety of ways, such as via a web form, email sent to a specified address, a voicemail box using speech-to-text conversion, etc. These incident reports may be stored in a queryable incident report database. As an example, a query in the form of a text string may return one or more incident reports that contain the words in the text string. Additionally or alternatively, one or more elements of the incident report (e.g., the "short description" field) may be used to query a database of knowledge base articles, other incident reports, or some other corpus of text. This may be done to identify other incident reports, past incident reports that have been resolved, reports on the resolution of past problems, knowledge base articles, or other information that may be relevant to the incident report to facilitate the resolution of the problem represented by the incident report.

[0108] This process is illustrated in Figure 7. A text query may be entered into a web interface 700. This web interface may be served through a compute instance of the remote network management platform 320. The web interface 700 converts the text query into a database query (e.g., an SQL query) and provides the SQL query to a database 702. This database may be the CMDB 500 or some other database. The database 702 contains multiple incident reports with problem description fields, such as those shown in Figure 6. In either case, the database 702 executes the query and returns matching results to the web interface 700. One or more such results may be returned. The web interface 700 presents these results as a web page.

[0109] For example, if the text query is "email," the web interface 700 can translate this query into an SQL query of the database 702. For example, the query can reference the problem description field of a table containing incident reports. Those incident reports that match the query (i.e., contain the word "email") can be provided in the query results. Thus, incident reports with problem descriptions of "My email client is not downloading new emails," "Email crashed," and "Can't connect to email" might be provided, but incident reports with a problem description of "VPN timed out" would not be returned.

[0110] This matching technique is simple and has several drawbacks. It only considers the presence of the query text in the incident. Therefore, it does not consider contextual information such as words that appear before or after the query text. Also, synonyms of the query text (e.g., "email" or "message") and misspellings of the query text (e.g., "emial") will not return results in this example.

[0111] Furthermore, implementing such a solution would involve the use of an inefficient sparse matrix, with an entry in one dimension for each word in the English language and an entry in the other dimension for the problem description of each incident. The exact number of words in the English language is open to debate, but is estimated to be at least 150,000-200,000, with less than approximately 20,000 words in common use. Given that a busy IT department may have a database of tens of thousands of incidents, this matrix would be quite large and wasteful to store, even if it contained only the 20,000 most commonly used words.

[0112] Thus, the above-described comparison methods may be replaced and / or augmented by various methods for comparing the semantic content and / or context of text samples. These methods may improve upon various machine learning techniques to facilitate natural language processing. Such techniques may include determining word and / or paragraph vectors from the text samples and applying artificial neural networks or other deep learning algorithms, sentiment analysis, or other techniques to determine similarities between the text samples. For example, these or other natural language processing techniques may be applied to determine similarities between one or more text fields of an incident report and other incident reports, resolved incident reports, knowledge base articles, or other potentially related samples of text.

[0113] However, particular segments of text within a larger sample of text may vary in similarity or relevance to the query text sample. For example, a particular article may contain a segment that is highly relevant to a particular query text, while the remainder of the article may have very low similarity to the query text. In such instances, determining a similarity value for the entire article may result in discarding articles that contain relevant subsections. Additionally, providing a user with a large sample of a particular article or other text that is "related" overall may result in lost time as the user reads through the article to find a particular subsection that is relevant to the user's problem. Furthermore, it may be beneficial to compare a text query to sections of text that are similar in size to the text query.

[0114] Therefore, it may be beneficial to divide a knowledge base article or other large sample of text into multiple segments. The similarity between the query text and each of the segments of the large text may then be determined and used to determine whether the large text is relevant to the query text. This may include using the similarity values ​​for each of the segments of the large text to determine an overall similarity value for the large text. Additionally or alternatively, the segments of the large text may be treated as independent text samples. In such an example, from the set of segments of the text sample, the segment that is "most relevant" or otherwise most similar to the query text may be selected, rather than limiting the selection to the entire text sample.

[0115] Determining similarity and / or relatedness at the segment level may enable more fine-grained detection of relevant portions of text within a corpus of text samples and may prevent discarding relevant portions of a larger text sample because they constitute a relatively small portion of the larger text sample of which they are a part. Furthermore, comparing a text query to text segments having a similar size (e.g., number of words, number of clauses, number of sentences) to the text query may allow a more appropriate comparison to be made than a comparison between a text query and an unsegmented sample of text that is significantly larger in size than the text query. Furthermore, these methods may allow the determination of the overall similarity of large samples of text to be tailored to specific applications. For example, to be more thorough in detecting all relevant portions of text within a corpus of text samples, a similarity value for a particular text sample may be determined as the maximum of the similarity values ​​determined for segments of text within the particular text sample. In another example, an overall more "topical" article may be selected by determining the average similarity values ​​of the segments of text within each article, such that for each article, articles containing only a single "similar" segment are disfavored compared to articles containing multiple "similar" segments.

[0116] 8 shows exemplary query text sample 810, first large text sample 820, and second large text sample 830. First large text sample 820 is divided into three segments of text 820A, 820B, and 820C, and second large text sample 830 is divided into three segments of text 830A, 830B, and 830C. Query text sample 810 may be a sample of text from an incident report. For example, query text sample 810 may be the problem description field 606 from incident report 600 shown in FIG. 6. The large text sample may be an article in a knowledge base or other database, or another field of an incident report.

[0117] Segments within such samples of text may be determined in a variety of ways. In some examples, segments may be generated manually, for example, by an author of a knowledge base article who annotates the article to indicate conceptually distinct segments of the article. Additionally or alternatively, segments may be generated automatically, which may include determining segments within a sample of text based on line breaks, punctuation, headers, or other information. Machine learning algorithms may be trained to identify segments within a sample of text, for example, based on a training set of query text.

[0118] As shown in Figure 8, the segments of text may not overlap and each may represent a separate set of sentences. However, Figure 8 is intended as a non-limiting example of segments within a sample of text, and other types of segmentation of text are contemplated. For example, a segment of text may include a portion of a sentence. That is, a segment of text may end and / or begin in the middle of a sentence. In some instances, segments of text within a particular larger sample of text may overlap. Such overlap may allow a transitional portion of text to be associated with both an earlier portion of text and a later portion of text.

[0119] As described above, a similarity value may be determined between the query text and each segment within each sample of text in a corpus of text samples (e.g., between one or more fields of an incident report and a corpus of articles describing the resolution of past incident reports). These similarity values ​​may then be used to select and present to the user the segments that are most similar to the query text. The segments may be presented by themselves or as part of the article of which they are a part (e.g., the selected segments are highlighted or otherwise indicated within the article).

[0120] Additionally or alternatively, the set of similarity values ​​determined for segments of a text sample may be used to determine an overall similarity value for the text sample. The overall similarity value may then be used to select and present to the user the text samples that are most similar to the query text. The method used to generate the overall similarity value from the segment similarity values ​​may be selected to affect the sorting of the selected text samples.

[0121] In some examples, an overall similarity value for a text sample may be determined as the maximum of the similarity values ​​of the text segments within the text sample. This method may be used to identify highly relevant segments of text, even if those segments are part of a text sample that is otherwise less relevant.

[0122] In some examples, the sum of the similarity values ​​of the text segments may be determined as the overall similarity value of the text sample. This method may be used to identify text samples that contain "more" segments of text that are "more" relevant to the query text. The sum may be normalized to the number of segments in the text sample (i.e., the overall similarity value may be the average of the segment similarity values) to highlight text samples that are more "topical" in content related to the query text.

[0123] In some examples, a weighted sum of some or all of the segment similarity values ​​may be determined as an overall similarity value for the text sample. For example, the overall similarity value may be determined as the sum of 100% of the similarity value of the most similar segment, 80% of the similarity value of the second most similar segment, 60% of the similarity value of the third most similar segment, etc. In another example, an iterative method may be applied to add together weighted versions of the similarity of each segment. An example of this iterative method may be expressed as follows for similarity values ​​between 0 and 1: Sort segment similarity values ​​from high to low Overall similarity score ← highest segment score for segment[i] remaining=2 to end: if similarity value for segment[i] >= overall similarity value / ρ Overall similarity value ← Overall similarity value + (1 - Overall similarity value) * Similarity value for segment[i]

[0124] The parameter ρ is advantageously set to a value between 1 and 2. If a value of 1 is selected, only the highest similarity value is considered. The weighting parameter ρ is provided so that a text sample with multiple low-value segments does not end up with a higher overall score than a text sample with a single segment with a high similarity value. The method also ensures that the overall similarity value will not be greater than 1, regardless of the number of segments in a particular text sample.

[0125] These methods can be used to identify text samples that have a few "highly relevant" segments, as well as to identify text samples that contain multiple segments that are individually less relevant to the query text but may be relevant overall. The number of segment similarity values ​​used to generate the weighted sum can be limited, for example, to the top three segments. This can be done to emphasize text samples that contain fewer segments with higher similarity scores over text samples with a larger number of relatively less relevant segments.

[0126] In some examples, text segment similarities may be compared to a threshold to determine an overall similarity value for the text sample. For example, only segment similarity values ​​that exceed a threshold similarity value may be summed to generate an overall text sample similarity value. In another example, the number of segment similarity values ​​that exceed a threshold similarity value may be determined as the overall text sample similarity value. Such a threshold comparison method may be used to base an assessment of the overall relevance of a text sample only on segments that may actually be related to the query text. Similarity values ​​of individual segments greater than a threshold level may be used as a proxy for such likelihood. The count of similarity values ​​that exceed the threshold may be normalized to the number of segments in the text sample to highlight text samples whose portions contribute significantly to content related to the query text.

[0127] VI. Natural Language Processing of Text Queries Based on Semantic Content The degree of similarity between two samples of text may be determined in various ways. The two samples of text may be a text field of an incident report and a text field of another incident report, a text field of a resolved incident report, a knowledge base article, or any other sample of text that may be related to the resolution, classification, or other aspects of the incident report. Additionally or alternatively, one or both of the samples may be a segment of text within a larger sample of text. As described above, the degree of overlap between the identities of words present in the two samples of text and / or word matrix methods may be used to determine the degree of similarity. Additionally or alternatively, one or more techniques of natural language processing may be applied to compare the samples of text so that the context or other semantic content of the text affects the determined similarity value between the samples of text.

[0128] Such techniques can be applied to improve text query matching associated with incident reports. These techniques can include various machine learning algorithms that can be trained based on text samples. The text samples used for training can include historical examples of incident reports, knowledge base articles, or other text samples of the same nature as the text samples to which the trained model will be applied. This has the advantage of providing a model that is specifically adapted to the vocabulary, topics, and idiomatic usage common to its intended use.

[0129] Such techniques may include determining word and / or paragraph vectors from text samples, applying ANN or other deep learning algorithms, performing sentiment analysis, or other techniques to determine similarities between text samples, group multiple text samples together according to topic or content, divide text samples into separate, interrelated segments, determine statistical associations between words, or perform any other language processing task. Below, a specific method for determining similarity values ​​between text samples using an ANN model that provides a compact semantic representation of words and text strings is provided as a non-limiting example of such a technique. However, other techniques may be applied to generate similarity values ​​between text samples that are applied elsewhere in this specification. In the following discussion, there are two approaches for training an ANN model to represent the semantic meaning of words: word vectors and paragraph vectors. These techniques may be combined with each other or with other techniques.

[0130] These techniques can also be applied to divide a segment of text into multiple text segments. Such division can enable more fine-grained detection of similarity between a query text (e.g., the problem description field of an incident report) and a larger sample of text because a similarity value can be determined between the query text and each segment of the larger sample of text. As a result, the relevance of a sample of text that is largely unrelated to the query text but includes subsections that are highly relevant to the query text can be detected.

[0131] A. Word Vector A "word vector" may be determined for each word present in a corpus of text records such that words with similar meanings (or "semantic content") are associated with word vectors that are close to each other in a semantically encoded vector space. Such vectors may have tens, hundreds, or even more elements. These word vectors allow the underlying meanings of words to be compared or otherwise manipulated by a computing device. Thus, the use of word vectors may allow for significant improvements over simpler word list or word matrix methods.

[0132] The word vectors can be used to quickly and efficiently compare the overall semantic content of the text samples, allowing a similarity value to be determined between the text samples. This can include determining the distance, cosine similarity, or some other measure of similarity between the word vectors of the words in each of the text samples. For example, the average of the word vectors in each of the text samples can be determined, and then the cosine similarity between the averages can be used as a measure of similarity between the text samples. Additionally or alternatively, the word vectors can be provided as input to an ANN, support vector machine, decision tree, or some other machine learning algorithm to perform sentiment analysis, classify or cluster the text samples, determine the level of similarity between the text samples, or perform other language processing tasks.

[0133] Word vectors can be determined for a set of words in various ways. In one example, a matrix of word vectors can be the input layer of an ANN. The ANN (including the matrix of word vectors) can then be trained with a large number of text strings from a database to determine contextual relationships between words that appear in these text strings. Such an ANN 900 is shown in FIG. 9A. The ANN 900 includes an input layer 902 that feeds a hidden layer 904, which feeds an output layer 906. The number of nodes in the input layer 902 and the output layer 906 can be equivalent to the number of words in a predefined vocabulary or dictionary (e.g., 20,000, 50,000, or 100,000). The number of nodes in the hidden layer 904 can be much smaller (e.g., 64 as shown in FIG. 9A, or other values ​​such as 16, 32, 128, 512, 1024, etc.).

[0134] ANN 900 is trained with one or more word configurations for each text string in the database. For example, in FIG. 9B , ANN 900 is shown being trained using the input word "email" and the output (context) words "can't," "connect," and "to." The output words serve as ground truth output values ​​against which results produced by output layer 906 are compared. This configuration reflects that "email" appears near "can't," "connect," and "to" in the text strings in database 702.

[0135] In an implementation, this may be represented as node 12 receiving an input of 1, while all other nodes in the input layer 902 receive an input of 0. Similarly, node O1 is associated with a ground truth value of "can't," node O2 is associated with a ground truth value of "connect," and node O3 is associated with a ground truth value of "to." In an implementation, this is represented as nodes O1, O2, and O3 being associated with a ground truth value of 1, while all other nodes in the output layer 906 are associated with a ground truth value of 0. The loss function may be, for example, the sum of squared errors between the outputs produced by the output layer 906 in response to the above inputs and vectors containing the ground truth values ​​associated with the output layer nodes.

[0136] Other arrangements of this text string from database 702 may be used to train ANN 900. For example, as shown in Figure 9C, the input word may be "can't" and the output words may be "connect," "to," and "email." In another example shown in Figure 9D, the input word may be "connect" and the output words may be "can't," "to," and "email."

[0137] In general, these arrangements may be selected such that the output word is within w words of the input word (e.g., w is 1, 2, 3, 5, etc.), the output word is in the same sentence as the input word, the output word is in the same paragraph as the input word, etc. Additionally, various word arrangements of each text string in database 702 may be used to train ANN 900. These text strings may be selected from short description field 605, problem description field 606, category field 608, resolution field 614, notes field 620, and / or any other field or combination of fields in an incident report.

[0138] After ANN 900 has been trained on these arrangements of text strings, hidden layer 904 becomes a compact vector representation of the context and meaning of the input words. That is, the weights from a particular node (e.g., I3) in input layer 902 to hidden layer 904 represent elements of the word vector for the word corresponding to the particular node (e.g., "can't"). For example, assuming ANN 900 is fully trained on a corpus of around 10,000 text strings (although more or fewer text strings may be used), an input word of "email" may have a similar vector representation to an input word of "mail." Intuitively, since hidden layer 904 is all that ANN 900 needs to determine the context of an input word, if two words have similar contexts, they are likely to have similar vector representations.

[0139] In some embodiments, the ANN 900 includes output nodes O1...O n and the input words associated with input nodes I1...I n , which can produce the same or similar vectors for the hidden layer 904.

[0140] Furthermore, vectors generated in this manner are additive. Thus, subtracting the vector representation of "mail" from the vector representation of "email" is expected to produce a vector with a value close to 0. However, subtracting the vector representation of "VPN" from the vector representation of "email" is expected to produce a vector with a higher value. Thus, the model indicates that "email" and "mail" are closer in meaning than "email" and "VPN."

[0141] Once vector representations have been determined for all words of interest, linear and / or multiplicative aggregations of these vectors can be used to represent text strings. For example, the vector of the text string "can't connect to email" can be found by summing the individual vectors of the words "can't," "connect," "to," and "email." In some cases, an average or some other operation can be applied to the vectors of words. This can be done by summing m vectors v where each entry is divided by m, as follows: i where i={1...m}, but other possibilities exist, such as weighted averaging.

number

[0142] Regardless of how aggregations are determined, this general technique allows for finding a vector representation of each text string in database 702. These vector representations may similarly be stored in database 702, together with associated text strings or separately. These vector representations may then be used to compare text strings, cluster or group text strings, train some other machine learning classifier, or perform other tasks. For example, matching text strings for a particular query text may be determined by determining a cosine similarity or other similarity value between the vector representation of the query text and the stored vector representations of samples of text in database 702.

[0143] The comparison can identify one or more text string vectors from database 702 that "match" in this manner. In some cases, this can be the k most similar text string vectors, or any text string vectors that have a similarity greater than a predetermined value. The identified text string vectors can correspond to a subset of incident reports within a larger corpus of incident reports recorded in database 702, and they are associated with additional incident reports that correspond to the query text string vector. For each identified text string vector, an associated text string can be looked up in database 702 and provided as an output text string. In some cases, an associated incident report can also be provided.

[0144] In some cases, only incident reports that are less than a predetermined age are provided. For example, the system may be configured to identify text string vectors from only incident reports that have been resolved within the past three, six, or twelve months. Alternatively, the system may be configured to identify text string vectors from only incident reports that have been opened within the past three, six, or twelve months.

[0145] In this way, incident reports with problem descriptions similar to the input text string can be quickly identified. Notably, the system provides contextual results that are likely to be relevant and meaningful to the input text string. As a result, individuals can review these incident reports to determine how issues similar to the one in the problem description have been reported and addressed in the past. This can dramatically reduce the amount of time it takes to resolve an incident.

[0146] Additionally or alternatively, these embodiments can be applied to detect and identify clusters of semantically and / or contextually similar incident reports within a corpus of incident reports. For example, the clusters of incident reports may relate to similar issues that may affect users of an IT system, persistent misconfigurations of one or more aspects of the IT system, progressive hardware failures in components of the IT system, or some other recurring problem within the IT system. Identifying such clusters of related incident reports may enable the IT system to be repaired or upgraded (e.g., by replacing and / or reconfiguring faulty or inconsistent running hardware or software), users to be trained to avoid common mistakes, detection and correction of infrequent hardware or software problems, or other benefits.

[0147] Such clusters of related incident reports can be detected and / or identified by identifying aggregated word (and / or paragraph) vectors corresponding to the incident reports within the semantically encoded vector space. Various methods may be used to detect such clusters within the semantically encoded vector space, such as k-means, support vector machines, ANNs (e.g., unsupervised ANNs configured and / or trained to identify related subsets of training examples within a corpus of available training examples), or some other classifier or other method for identifying clusters of related vectors within a vector space.

[0148] B. Paragraph Vector As discussed above, the ANN model 900 uses surrounding context to provide compact, semantically related vector representations of words. After training, words with similar meanings can be mapped to similar locations in vector space. For example, the vectors for "powerful" and "strong" may appear close to each other, while the vectors for "powerful" and "Paris" may be further apart. Addition and subtraction between word vectors are also meaningful. Vector algebra can be used on the determined word vectors to answer analogy problems such as "king" - "man" + "woman" = "queen."

[0149] However, the complete meaning of a sentence or other clause (e.g., a phrase, multiple sentences, a paragraph, a text segment within a larger sample of text, or a document) cannot necessarily be captured from the individual word vectors of a sentence (e.g., by applying vector algebra). Word vectors can represent the semantic content of individual words and can be trained using a short context window. Thus, when operating solely on word vectors, the semantic content of word order and any information outside the short context window is lost.

[0150] For example, take the sentence "I want a big green cell right now," in which case simple vector algebra on individual words may not provide the correct meaning of the word "cell," since the word "cell" has multiple possible meanings and may therefore be ambiguous. Depending on the context, a "cell" may be a biological cell, a prison cell, or a cell in a cellular communications network. Thus, the paragraph, sentence, or phrase from which a given word is sampled can provide important contextual information.

[0151] In another example, given the sentence "Where art thou ___", it is easy to predict that the missing word is "Romeo" if the sentence is said to be taken from a paragraph about Shakespeare. Thus, learning a semantic vector representation of the entire paragraph can help predict the context of words sampled from that paragraph.

[0152] Similar to the above method for learning word vectors, an ANN or other machine learning structure can be trained using many paragraphs in a corpus to determine the contextual meaning of entire paragraphs, sentences, phrases, or other multi-word text samples, as well as to determine the meaning of individual words that make up paragraphs in the corpus. Such an ANN 1000 is shown in FIG. 10A. The ANN 1000 includes an input layer 1002 that feeds a hidden layer 1004, which in turn feeds an output layer 1006. The input layer 1002 has two types of input substructures: an upper substructure 1008 (input nodes I1...I2) that represent words; n , and a lower substructure 1010 (composed of input nodes D1...D m Note that the output layer 1006 and upper input layer substructure 1008 may be composed of nodes (composed of a set of words, a set of sentences, a set of paragraphs, a set of words ...

[0153] For each paragraph in the corpus, the ANN 1000 is trained using fixed-length contexts generated by moving a sliding window over the paragraph. Thus, a given paragraph vector is shared across all training contexts created from that source paragraph, but not across training contexts created from other paragraphs. Word vectors are shared across training contexts created from all paragraphs; for example, the vector for "cannot" is the same for all paragraphs. Paragraphs are not limited in size and can be as large as an entire document or as small as a sentence or phrase. Figure 10A shows the ANN 1000 being trained in a single training iteration with input word contexts "can't," "connect," and "to," input paragraph context DOC1, and output word "email." The output word serves as a ground truth output value against which results generated by the output layer 1006 are compared. This arrangement reflects that "email" appears near "can't," "connect," and "to" and is within DOC1.

[0154] In an implementation, this may be represented as output node O4 receiving a ground truth value of 1, and all other nodes in the output layer 1006 having a ground truth value of 0. Similarly, node I1 has a ground truth value of "can't," node I2 has a ground truth value of "connect," node I3 has a ground truth value of "to," and node D1 has a ground truth value of DOC1. In an implementation, this may be represented as nodes I1, I2, I3, and D1 being associated with a value of 1, and all other nodes in the input layer 1002 having a value of 0. The loss function may be, for example, the sum of squared errors between the output of the output layer 1006 and a vector containing the ground truth values. The weight values ​​of the corresponding word and paragraph vectors, as well as the weight values ​​of all output layer parameters (e.g., softmax weights), are updated based on the loss function (e.g., via backpropagation).

[0155] 10B shows ANN 1000 being trained on a subsequent context window, which derives from the same document but shifts the words earlier in the document, using input word contexts "connect," "to," and "email," input paragraph context "DOC1," and output word "server." In implementation, these inputs and outputs may be coded with ground truth values ​​in the same manner as described above.

[0156] 10C shows an example of ANN 1000 trained on another document in the corpus. A context window is derived from this document, using the input word contexts "can't," "load," and "my," the input paragraph context DOC2, and the output word "database." In implementation, these inputs and outputs may be coded with ground truth values, similar to those described above.

[0157] After ANN 1000 is trained, the weights associated with hidden layer 1004 become compact vector representations of the context and meaning of the input words and paragraphs. For example, assuming ANN 1000 is fully trained on a corpus of 1,000 paragraphs, and the entire corpus contains 10,000 unique words, each paragraph and each word can be represented by a unique vector with a length equal to the number of hidden nodes in hidden layer 1004. These unique vectors encode the contextual meaning of the words within the paragraph or the paragraph itself.

[0158] 10D shows the ANN 1000 at prediction time, where it is performing an inference step to calculate a paragraph vector for a new, previously unseen paragraph. This inference step begins by adding an additional input node 1012 to the input layer substructure 1010 representing the unseen paragraph (DOC M+1). During this inference process, the coefficients of the word vector substructure 1008, as well as the learned weights between the hidden layer 1004 and the output layer 1006, are fixed. Thus, the model generates an additional paragraph vector 1012 corresponding to the unseen paragraph in the input paragraph vector substructure 1010, obtaining a new semantic vector representation for the unseen paragraph. Any additional unseen paragraphs can be trained via a similar process by adding input nodes to the input layer substructure 1010.

[0159] Alternatively, paragraph vectors can be trained by ignoring word contexts in the input layer, using only paragraph vectors as input, and having the model predict different word contexts randomly sampled from a paragraph in the output layer. The input layer of such an ANN consists only of paragraph vectors, and the output layer represents a single context window randomly generated from a given paragraph. Training such an ANN may yield vector representations for the semantic content of paragraphs in a corpus, but does not necessarily provide semantic vector representations for the words therein.

[0160] Once vector representations are determined for paragraphs in a corpus, linear and / or multiplicative aggregations of these vectors can be used to represent a topic of interest. Furthermore, as shown in ANN 1000, if the dimensions of the paragraph vectors are the same as the dimensions of the word vectors, linear and multiplicative aggregations between word vectors and paragraph vectors can be obtained. For example, finding the Chinese version of "Julius Caesar" using an encyclopedia as a corpus can be achieved by the vector operation PV("Julius Caesar") - WV("Rome") + WV("China"), where PV is the paragraph vector (representing the entire Wikipedia article) and WV is the word vector. Thus, paragraph vectors can achieve the same kind of inference as word vectors, with context-based results.

[0161] Indeed, such learned paragraph vectors can be used as inputs to other supervised learning models, such as sentiment prediction models. Such models, which may include but are not limited to ANNs, support vector machines (SVMs), or naive Bayes classifiers, use paragraph vectors as inputs and have corresponding sentiment labels as outputs. Other metrics, such as cosine similarity or nearest neighbor clustering algorithms, can be applied to paragraph vectors to find or group paragraphs of similar topics within a corpus of paragraphs.

[0162] In this embodiment, the combination of learned word vectors and paragraph vectors can help determine the structure and meaning of an incident report, such as incident report 600, as shown in FIG. 6. Incident report 600 consists of multiple fields in the left column, at least some of which are associated with values ​​in the right column. For long text fields, such as short description field 605, problem description field 606, solution field 614, and notes field 620, it may be preferable to represent the associated right-column text as a paragraph vector or multiple paragraph vectors corresponding to each separate text segment within the right-column text to obtain more contextual meaning than aggregating the individual word vectors that form the text. Incident report 600 is presented for illustrative purposes. Various fields in an incident report may be arranged to be represented as paragraph vectors, word vectors, or a weighted combination of the two. Other types of incident reports, problem reports, case files, and knowledge base articles may also be used, and these reports may include more, fewer, and / or different fields.

[0163] After representing the different fields as paragraph vectors, word vectors, or a weighted combination of the two, a single vector representing the entire incident can be generated by concatenating, generating a vector sum, or aggregating the word and / or paragraph vector representations of the individual incident fields in other forms. Using a single aggregate incident vector representation, the system can be configured to identify similar aggregate vectors (and thus similar incident reports) based on cosine similarity or other metrics described above. Alternatively, a search for similar incident reports may use only the paragraph text of one or more individual fields. In this way, text from one or more individual fields in an incident report may be combined into a single paragraph of text. A paragraph vector can then be generated from this single large paragraph of concatenated text and used to search for similar incidents.

[0164] This process can be explained based on the aforementioned ANN structure. First, text strings are obtained from database 702 of FIG. 7. As mentioned above, these text strings may be from portions of incident reports. Then, words are extracted from the text strings. The extracted words may be all words in the text strings or portions of these words. These extracted words are provided as input to ANN 900 of FIGS. 9A-9D. Substring contexts of these words are extracted from the text strings. A substring context may be one or more substrings that include words before, after, or around the extracted associated word. These vector representations may then be used to compare each of these distinct text samples (e.g., using cosine similarity).

[0165] The comparison can identify one or more incident reports from database 702 that "match" in this manner. In some cases, this can be the k incident reports with the highest similarity, or incident reports with similarity greater than a predetermined value. The user can be provided with these identified incident reports or references to them.

[0166] In some cases, only incident reports that are less than a predetermined age are provided. For example, the system may be configured to identify only incident reports that have been resolved within the past three, six, or twelve months. Alternatively, the system may be configured to identify only incident reports that have been opened within the past three, six, or twelve months.

[0167] In this manner, incident reports with content similar to the input incident report can be quickly identified. As a result, individuals can review these incident reports to determine how issues similar to the one in the incident have been reported and addressed in the past. This can dramatically reduce the amount of time it takes to resolve an incident.

[0168] Although this section describes some possible implementations of word and paragraph vectors, other implementations are possible, e.g., different ANN structures and different training procedures may be used.

[0169] C. Splitting the text sample As noted above, rather than or in addition to determining the similarity between the query text and the comparison text sample as a whole, it may be beneficial to determine the similarity between the query text and individual segments of the comparison text sample. The segments may or may not overlap. Segments may be restricted to include separate entire paragraphs or sentences, or may be specified to include portions of sentences and / or portions of paragraphs.

[0170] Segments may be determined manually. For example, a user may annotate a text sample to indicate the extent of conceptually distinct portions of the text sample. In some examples, this may include inserting section headings, which are later used as indicators of segments within the text sample. Additionally or alternatively, an automatic algorithm may be used to segment the text sample.

[0171] Such an automatic algorithm may operate based on the punctuation present in the text sample. For example, the algorithm may segment the text sample based on tabs, carriage returns, and / or any other punctuation present in the text, such that each paragraph and / or sentence in the text sample is assigned to a separate segment.

[0172] Additionally or alternatively, such algorithms may incorporate natural language processing techniques to partition the text sample. For example, the algorithm may operate to select partition boundaries to minimize a cost function related to the semantic similarity between words within each segment. This may include minimizing the variance between the word vectors of the words within each segment.

[0173] In another example, a machine learning algorithm (e.g., an ANN) may be trained to receive a sample of text and generate segments therefrom. Such a machine learning algorithm may be trained to detect semantically related segments of text within a larger sample of text by being provided with training data containing examples of short samples of text related to a single concept. For example, a machine learning algorithm may be provided with a set of previously received text queries (e.g., problem description fields from incident reports) and trained to identify segments within the sample of text that exhibit similar characteristics. In another example, a machine learning algorithm may be trained to recognize words preceding punctuation marks that signal the end of a sentence (e.g., !?.), such that segments are determined based solely on punctuation marks, while avoiding segmentation based on non-final punctuation marks (e.g., the period in "Dr."). In yet another example, a machine learning algorithm may be trained to detect portions of speech tags to detect sentence boundaries (e.g., for languages ​​without punctuation).

[0174] VII. EXEMPLARY OPERATIONS 11 is a flowchart illustrating an exemplary embodiment. The process illustrated in FIG. 11 may be performed by a computing device, such as computing device 100, and / or a cluster of computing devices, such as server cluster 200. However, the process may also be performed by other types of devices or device subsystems. For example, the process may be performed by a portable computer, such as a laptop or tablet device.

[0175] The embodiments of Figure 11 may be simplified by eliminating any one or more of the features shown therein. Additionally, these embodiments may be combined with features, aspects, and / or implementations of any of the preceding figures or otherwise described herein.

[0176] The example embodiment of Figure 11 includes obtaining, by a server device, a corpus of text records (1100). Each of the text records includes a sample of text and an indication of one or more segments of text within the sample of text. At least one of the text records includes an indication of at least two segments of text within the sample of text.

[0177] The example embodiment of FIG. 11 further includes obtaining, by the server device, a subject sample of text from the client device (1102).

[0178] 1 further includes determining, by the server device, a semantic similarity value between the subject sample of text and each of the text records (1104). Determining a specific semantic similarity value between the subject sample of text and a particular text record of the corpus includes (i) determining an individual semantic similarity value between the subject sample of text and each of the segments of text represented by the particular text record, and (ii) generating a specific semantic similarity value between the subject sample of text and the particular text record based on the individual semantic similarity values.

[0179] The example embodiment of FIG. 11 further includes selecting, by the server device, based on the semantic similarity values, text records from the corpus that have semantic similarities indicating that the text records are more similar to the subject sample of text than any other text records (1106).

[0180] The example embodiment of FIG. 11 further includes providing, by the server device, a representation of the text record to the client device (1108).

[0181] VIII. Conclusion The present disclosure is not limited in terms of the specific embodiments described herein, but is intended as an illustration of various aspects. Many modifications and variations can be made without departing from its scope, as will be apparent to those skilled in the art. Functionally equivalent methods and apparatuses within the scope of the disclosure, in addition to those described herein, will be apparent to those skilled in the art from the foregoing description. Such modifications and variations are intended to be included within the scope of the appended claims.

[0182] The above detailed description, with reference to the accompanying drawings, sets forth various features and operations of the disclosed systems, apparatus, and methods. The illustrative embodiments described herein and in the drawings are not intended to be limiting. Other embodiments may be used, and other changes may be made, without departing from the scope of the subject matter presented herein. It will be readily understood that aspects of the present disclosure, as generally described and illustrated herein, may be arranged, substituted, combined, separated, and designed in a variety of different forms.

[0183] With respect to any or all of the message flow diagrams, scenarios, and flowcharts in the figures and described herein, each step, block, and / or communication may represent the processing of information and / or the transmission of information in accordance with the example embodiments. Alternative embodiments are included within the scope of these example embodiments. In these alternative embodiments, for example, operations described as steps, blocks, transmissions, communications, requests, responses, and / or messages may be executed in a different order than that shown or described, including substantially simultaneously or in reverse order, depending on the functionality involved. Furthermore, more or fewer blocks and / or operations may be used with any of the message flow diagrams, scenarios, and flowcharts described herein, and these message flow diagrams, scenarios, and flowcharts may be combined with each other, in part or in whole.

[0184] Steps or blocks representing the processing of information may correspond to circuitry that can be configured to perform specific logical functions of the methods or techniques described herein. Alternatively or additionally, steps or blocks representing the processing of information may correspond to modules, segments, or portions of program code (including associated data). The program code may include one or more instructions that are executable by a processor to implement specific logical operations or actions in the method or technique. The program code and / or associated data may be stored in any type of computer-readable medium, such as a storage device, including a RAM, a disk drive, a solid-state drive, or other storage medium.

[0185] Computer-readable media may also include non-transitory computer-readable media, such as computer-readable media that store data for a short period of time, such as register memory and processor cache. Computer-readable media may also include non-transitory computer-readable media that store program code and / or data for a longer period of time. Thus, computer-readable media may include, for example, secondary or permanent long-term storage devices, such as ROMs, optical or magnetic disks, solid-state drives, and compact disc read-only memories (CD-ROMs). Computer-readable media may also be any other volatile or non-volatile storage system. Computer-readable media may be considered, for example, to be computer-readable storage media or tangible storage devices.

[0186] Additionally, steps or blocks representing one or more information transfers may correspond to information transfers between software and / or hardware modules within the same physical device, although other information transfers may be between software and / or hardware modules in different physical devices.

[0187] The particular arrangement shown in the figures should not be considered limiting. It should be understood that other embodiments may include more or fewer elements than each element shown in a given figure. Furthermore, some of the illustrated elements may be combined or omitted. Furthermore, example embodiments may include elements not shown in the figures.

[0188] While various aspects and embodiments are disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are intended to be illustrative and not limiting, with the true scope being indicated by the appended claims.

Claims

1. persistent storage including a corpus of text records, each of the text records including a sample of text and an indication of one or more segments of text within the sample of text, at least one of the text records including an indication of at least two segments of text within the sample of text, and a machine learning model generating the indications of the one or more segments of text within the sample of text of the corpus of text records; Computing equipment and wherein the computing device comprises: obtaining a subject sample of text from a client device; determining a semantic similarity value between the subject sample of text and each of the text records, wherein determining a specific semantic similarity value between the subject sample of text and a particular text record of the corpus includes: (i) determining an individual semantic similarity value between the subject sample of text and each of the one or more segments of text represented by the particular text record; (ii) comparing each of the individual semantic similarity values ​​between the subject sample of text and each of the one or more segments of text represented by the particular text record to a threshold similarity level; and (iii) determining the number of the individual semantic similarity values ​​that exceed the threshold similarity level as the specific semantic similarity value; selecting, based on the semantic similarity values, text records from the corpus having semantic similarities that indicate the text records are more similar to the target sample of text than any other of the text records; providing a representation of the text record to the client device; and A system configured to:

2. Determining a respective semantic similarity value between the subject sample of text and a particular segment of text within the particular text record, obtaining a vector representation of the subject sample of text, the vector representation of the subject sample of text comprising at least one of (i) a word vector describing the meaning of each distinct word of the subject sample of text in a first semantically encoded vector space, or (ii) a paragraph vector describing the meaning of multiple words of the subject sample of text in a second semantically encoded vector space; obtaining a vector representation of the particular segment of text, the vector representation of the particular segment of text comprising at least one of (i) a word vector describing the meaning of each distinct word of the particular segment of text in the first semantically encoded vector space, or (ii) a paragraph vector describing the meaning of multiple words of the particular segment of text in the second semantically encoded vector space; determining a semantic similarity value between the vector representation of the subject sample of text and the vector representation of the particular segment of text; The system of claim 1 , comprising:

3. 3. The system of claim 1 or 2, wherein the machine learning model is configured to identify the indications of the one or more segments of text within each sample of text as non-overlapping segments of text.

4. 3. The system of claim 1 or 2, wherein the machine learning model is configured to identify the indications of the one or more segments of text in each sample of text as segments of text each representing one or more separate sentences.

5. the computing device, obtaining a plurality of queries from one or more client devices, each query including a query sample of text; training the machine learning model based on the plurality of queries to predict associated segments of text within a sample of text; The system of claim 1 , configured to:

6. applying, by a server device, a machine learning model to a corpus of text records to generate indicators of one or more segments of text within each distinct sample of text of the corpus of text records, each of the text records including a respective distinct sample of text, and at least one of the text records including indicators of at least two segments of text within the sample of text; obtaining, by the server device, a subject sample of text from a client device; determining, by the server device, a semantic similarity value between the subject sample of text and each of the text records, wherein determining a specific semantic similarity value between the subject sample of text and a particular text record of the corpus includes: (i) determining an individual semantic similarity value between the subject sample of text and each of the one or more segments of text represented by the particular text record; (ii) comparing each of the individual semantic similarity values ​​between the subject sample of text and each of the one or more segments of text represented by the particular text record to a threshold similarity level; and (iii) determining as the specific semantic similarity value the number of the individual semantic similarity values ​​that exceed the threshold similarity level; selecting, by the server device, based on the semantic similarity values, from the corpus text records having semantic similarities indicating that the text records are more similar to the target sample of text than any other of the text records; providing, by the server device, to the client device, a representation of the text record; 10. A computer-implemented method comprising:

7. determining a respective semantic similarity value between the subject sample of text and a particular segment of text within the particular text record, obtaining a vector representation of the subject sample of text, the vector representation of the subject sample of text comprising at least one of (i) word vectors describing the meaning of each distinct word of the subject sample of text in a first semantically encoded vector space, or (ii) paragraph vectors describing the meaning of multiple words of the subject sample of text in a second semantically encoded vector space; obtaining a vector representation of the particular segment of text, the vector representation of the particular segment of text comprising at least one of (i) a word vector describing the meaning of each distinct word of the particular segment of text in the first semantically encoded vector space, or (ii) a paragraph vector describing the meaning of multiple words of the particular segment of text in the second semantically encoded vector space; determining a semantic similarity value between the vector representation of the target sample of text and the vector representation of the particular segment of text; The computer-implemented method of claim 6 , comprising:

8. 8. The computer-implemented method of claim 6 or 7, wherein the machine learning model is configured to identify the indications of the one or more segments of text within each sample of text as non-overlapping segments of text.

9. 8. The computer-implemented method of claim 6 or 7, wherein the machine learning model is configured to identify the indications of the one or more segments of text in each sample of text as segments of text each representing one or more separate sentences.

10. obtaining a plurality of queries from one or more client devices, each query including a query sample of text; training the machine learning model based on the plurality of queries to predict related segments of text within a sample of text; 10. A computer-implemented method according to any one of claims 6 to 9, comprising:

11. A non-transitory computer-readable medium having stored thereon program instructions that, when executed by a computing system, obtaining a corpus of text records, each of the text records including a sample of text and an indication of one or more segments of text within the sample of text, wherein a machine learning model generates the indications of the one or more segments of text within the sample of text; acquiring a subject sample of text from a client device, at least one of the text records including an indication of at least two segments of text within the sample of text; determining a semantic similarity value between the subject sample of text and each of the text records, wherein determining a specific semantic similarity value between the subject sample of text and a particular text record of the corpus includes: (i) determining an individual semantic similarity value between the subject sample of text and each of the one or more segments of text represented by the particular text record; (ii) comparing each of the individual semantic similarity values ​​between the subject sample of text and each of the one or more segments of text represented by the particular text record to a threshold similarity level; and (iii) determining the number of the individual semantic similarity values ​​that exceed the threshold similarity level as the specific semantic similarity value; selecting, based on the semantic similarity values, text records from the corpus having semantic similarities that indicate the text records are more similar to the target sample of text than any other of the text records; providing a representation of the text record to the client device; and a non-transitory computer-readable medium that causes the computing system to perform operations including:

12. Determining a respective semantic similarity value between the subject sample of text and a particular segment of text within the particular text record, obtaining a vector representation of the subject sample of text, the vector representation of the subject sample of text comprising at least one of (i) a word vector describing the meaning of each distinct word of the subject sample of text in a first semantically encoded vector space, or (ii) a paragraph vector describing the meaning of multiple words of the subject sample of text in a second semantically encoded vector space; obtaining a vector representation of the particular segment of text, the vector representation of the particular segment of text comprising at least one of (i) a word vector describing the meaning of each distinct word of the particular segment of text in the first semantically encoded vector space, or (ii) a paragraph vector describing the meaning of multiple words of the particular segment of text in the second semantically encoded vector space; determining a semantic similarity value between the vector representation of the subject sample of text and the vector representation of the particular segment of text; 12. The non-transitory computer-readable medium of claim 11, comprising:

13. 13. The non-transitory computer-readable medium of claim 11 or 12, wherein the machine learning model is configured to identify the indications of the one or more segments of text within each sample of text as non-overlapping segments of text.

14. The operation is obtaining a plurality of queries from one or more client devices, each query including a query sample of text; training the machine learning model based on the plurality of queries to predict associated segments of text within a sample of text; 14. The non-transitory computer-readable medium of claim 11, comprising:

15. a processor; a memory accessible by said processor; the memory storing instructions that, when executed by the processor, cause the processor to perform operations, the operations including: accessing a corpus containing a plurality of text records; generating an indicator for each of one or more distinct segments of text within each of the plurality of text records in the corpus based on a machine learning model; obtaining a subject sample of text from a client device; generating a respective separate record semantic similarity value between the subject sample of text and each of the text records in the corpus, for each of the text records in the corpus: determining one or more respective distinct segment semantic similarity values ​​between the subject sample of text and the one or more respective distinct segments of text in the text record; and generating each distinct record semantic similarity value between the subject sample of text and the text record by comparing each of the distinct segment semantic similarity values ​​to a threshold similarity level and determining the number of the distinct segment semantic similarity values ​​that exceed the threshold similarity level as the distinct record semantic similarity value; generating a selecting, from the corpus, a particular text record having the greatest distinct record semantic similarity value for the target sample of text based on each of the generated distinct record semantic similarity values; providing a representation of the particular text record to the client device; and Including, the system.

16. Determining the one or more respective distinct segment semantic similarity values ​​between the subject sample of text and the one or more respective distinct segments of text in the text record includes: receiving a vector representation of the subject sample of text, the vector representation of the subject sample of text comprising word vectors describing the meaning of each distinct word of the subject sample of text in a first semantically encoded vector space, or paragraph vectors describing the meaning of multiple words of the subject sample of text in a second semantically encoded vector space, or both; receiving one or more vector representations of the one or more distinct segments of text in the text record, the one or more vector representations of the one or more distinct segments of text in the text record comprising: word vectors describing, in the first semantically encoded vector space, the meaning of each distinct word of the one or more distinct segments of text in the text record; or paragraph vectors describing, in the second semantically encoded vector space, the meaning of words in the one or more distinct segments of text in the text record; determining a vector semantic similarity value between the vector representation of the subject sample of text and each distinct vector representation of the one or more segments of text within the text record; The system of claim 15, comprising:

17. 16. The system of claim 15, wherein the machine learning model is configured to identify the indicators of each of the one or more distinct segments of text in each of the text records as non-overlapping segments of text.

18. 16. The system of claim 15, wherein the machine learning model is configured to identify each of the one or more distinct segments of text in each of the text records as one or more separate sentences.

19. The system of claim 15 , wherein each of the text records includes an indication of a timestamp within a predetermined time threshold.

20. accessing, by a server device, a corpus comprising a plurality of text records; generating an indicator for each of one or more distinct segments of text within each of the plurality of text records in the corpus based on a machine learning model; receiving, by the server device, a subject sample of text from a client device; generating, by the server device, a respective separate record semantic similarity value between the subject sample of text and each of the text records in the corpus, for each of the text records in the corpus: determining one or more respective distinct segment semantic similarity values ​​between the subject sample of text and the one or more respective distinct segments of text in the text record; and generating each distinct record semantic similarity value between the subject sample of text and the text record by comparing each of the distinct segment semantic similarity values ​​to a threshold similarity level and determining the number of the distinct segment semantic similarity values ​​that exceed the threshold similarity level as the distinct record semantic similarity value; and selecting, from the corpus, a particular text record having the greatest distinct semantic similarity value for the target sample of text based on the generated semantic similarity values ​​for each distinct record; providing, by the server device, to the client device, a representation of the particular text record; 10. A computer-implemented method comprising:

21. Determining the one or more respective distinct segment semantic similarity values ​​between the subject sample of text and the one or more respective distinct segments of text in the text record comprises: receiving a vector representation of the subject sample of text, the vector representation of the subject sample of text comprising word vectors describing the meaning of each distinct word of the subject sample of text in a first semantically encoded vector space, or paragraph vectors describing the meaning of multiple words of the subject sample of text in a second semantically encoded vector space, or both; receiving one or more vector representations of the one or more distinct segments of text in the text record, the one or more vector representations of the one or more distinct segments of text in the text record comprising: word vectors describing, in the first semantically encoded vector space, the meaning of each distinct word of the one or more distinct segments of text in the text record; or paragraph vectors describing, in the second semantically encoded vector space, the meaning of words in the one or more distinct segments of text in the text record; determining a vector semantic similarity value between the vector representation of the subject sample of text and the vector representation of each of the one or more distinct segments of text within the text record; 21. The computer-implemented method of claim 20, comprising:

22. 21. The computer-implemented method of claim 20, wherein generating an indication of the one or more distinct segments of text within each of the text records in the corpus is based on one or more document properties, the method including dividing each of the text records into the one or more distinct segments of text based on the indication.

23. 21. The computer-implemented method of claim 20, wherein the machine learning model is configured to identify the indicators of each of the one or more distinct segments of text in each of the text records as non-overlapping segments of text.

24. 21. The computer-implemented method of claim 20, wherein the machine learning model is configured to identify each of the one or more distinct segments of text in each of the text records as one or more separate sentences.

25. selecting two or more text records from the corpus based on the generated semantic similarity values ​​for each distinct record; providing a representation of the two or more text records to the client device; 21. The computer-implemented method of claim 20, comprising:

26. A non-transitory computer-readable medium having stored thereon program instructions that, when executed by a computing system, accessing, by a server device, a corpus including a plurality of text records; generating an indicator for each of one or more distinct segments of text within each of the plurality of text records in the corpus based on a machine learning model; receiving, by the server device, a subject sample of text from a client device; generating, by the server device, a respective separate record semantic similarity value between the subject sample of text and each of the text records in the corpus, for each of the text records in the corpus: determining one or more respective distinct segment semantic similarity values ​​between the subject sample of text and the one or more respective distinct segments of text in the text record; and generating each distinct record semantic similarity value between the subject sample of text and the text record by comparing each of the distinct segment semantic similarity values ​​to a threshold similarity level and determining the number of the distinct segment semantic similarity values ​​that exceed the threshold similarity level as the distinct record semantic similarity value; generating a selecting from the corpus a particular text record having the greatest distinct semantic similarity value for the target sample of text based on the generated semantic similarity values ​​for each distinct record; providing, by said server device, to said client device, a representation of said particular text record; a non-transitory computer-readable medium that causes the computing system to perform operations including:

27. Determining the one or more respective distinct segment semantic similarity values ​​between the subject sample of text and the one or more respective distinct segments of text in the text record includes: receiving a vector representation of the subject sample of text, the vector representation of the subject sample of text comprising word vectors describing the meaning of each distinct word of the subject sample of text in a first semantically encoded vector space, or paragraph vectors describing the meaning of multiple words of the subject sample of text in a second semantically encoded vector space, or both; receiving one or more vector representations of the one or more distinct segments of text in the text record, the one or more vector representations of the one or more distinct segments of text in the text record comprising: word vectors describing, in the first semantically encoded vector space, the meaning of each distinct word of the one or more distinct segments of text in the text record; or paragraph vectors describing, in the second semantically encoded vector space, the meaning of words in the one or more distinct segments of text in the text record; determining a vector semantic similarity value between the vector representation of the subject sample of text and the vector representation of each of the one or more distinct segments of text within the text record; 27. The non-transitory computer-readable medium of claim 26, comprising:

28. 27. The non-transitory computer-readable medium of claim 26, wherein each of the text records includes an indication of a timestamp within a predetermined time threshold.

29. 27. The non-transitory computer-readable medium of claim 26, wherein the machine learning model is configured to identify the indicators of each of the one or more distinct segments of text in each of the text records as non-overlapping segments of text.

30. The operation is selecting two or more text records from the corpus based on the generated respective distinct record semantic similarity values; providing a representation of the two or more text records to the client device; and 27. The non-transitory computer-readable medium of claim 26, comprising:

31. The step of determining the number of the distinct semantic similarity values ​​that exceed the threshold similarity level as the particular semantic similarity value, comprising: generating a sum of the individual semantic similarity values ​​that exceed the threshold similarity level; normalizing said sum to the number of segments of text represented by said particular text record; The computer-implemented method of claim 6 , comprising:

32. The method of claim 31, wherein the step of determining the number of each distinct segment semantic similarity value that exceeds the threshold similarity level as the each distinct record semantic similarity value comprises: generating a sum of each of the distinct segment semantic similarity values ​​that exceed the threshold similarity level; normalizing said sum to the number of segments of text represented by said text record; 21. The computer-implemented method of claim 20, comprising:

Images