Method and apparatus for determining acceptability of a remedy for a supply plan deviation using a machine learning model

Abstract

The present disclosure relates to determining acceptability of a remedy for a supply plan deviation with a machine learning model. A system for analyzing a supplier communication regarding a deviation from a supply plan is described. The system can determine a severity of the deviation and determine an impact on a supply chain or inventory level caused by the deviation. A remedy in the supplier communication can be identified, and the system can determine whether the remedy is acceptable to address the deviation. The analysis of the supply plan deviation, the severity of the deviation, the acceptability of the remedy, and / or other factors can be used to generate a supplier score.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to analyzing communications related to supply plan deviations. In particular, the present disclosure relates to using a machine learning model to determine whether a remediation is an acceptable remediation of a supply chain deviation. BACKGROUND

[0002] Communications can be transmitted between different entities through various communication channels. Examples of communication channels include email, instant messaging, social media platforms, project collaboration applications, gaming applications, electronic photo albums, and specialized supply chain applications that facilitate the transmission of purchase orders, order fulfillment, and financial transactions. As used herein, the term “entity” can refer to a company or organization (such as a supplier company or a customer company) and / or a person.

[0003] In some cases, an entity has multiple options for entities with which to have a conversation about a particular topic. As an example, a customer can have multiple suppliers from which the customer can purchase a desired product or service. As another example, a company’s purchasing agent can have multiple sales representatives with which the purchasing agent can discuss a product issue. As another example, a social media user can have the option to reach out to multiple other users for advice and / or recommendations.

[0004] The methods described in this section are methods that can be employed, but are not necessarily the methods that have been previously conceived or employed. Therefore, unless otherwise indicated herein, the methods should not be deemed to be inherent or prior art in the art solely to their inclusion in this section. BRIEF DESCRIPTION OF DRAWINGS

[0005] In the drawings, which are included by way of example and not limitation, embodiments are illustrated. It should be noted that references to “an embodiment” or “one embodiment” in this disclosure are not necessarily to the same embodiment, and they mean at least one. In the drawings:

[0006] Figure 1 illustrates an example system including a trained machine learning model that can analyze supplier communications to generate a supplier score and / or determine whether a remediation proposed by a supplier in response to a supply plan deviation meets an acceptance criteria, in accordance with one or more embodiments;

[0007] Figure 2 illustrates an example set of operations for training a machine learning model to analyze communications related to a supply plan, in accordance with one or more embodiments;

[0008] Figure 3A illustrates an example set of operations for identifying a supply plan deviation and generating a supplier score based on an analysis of supplier communications, in accordance with one or more embodiments;

[0009] Figure 3B FIGURE 1 illustrates an example set of operations for determining a deviation attribute, in accordance with one or more embodiments;

[0010] Figure 4A FIGURE 2 illustrates an example set of operations for determining whether a remedial action meets acceptance criteria, in accordance with one or more embodiments;

[0011] Figure 4B FIGURE 3 illustrates an example set of operations for training a machine learning model to determine acceptance criteria for remedial actions used in response to a supply plan deviation, in accordance with one or more embodiments;

[0012] Figure 5 FIGURE 4 illustrates an example user interface for presenting a supplier score and a reply recommendation based on one or more message ratings, in accordance with one or more embodiments; and

[0013] Figure 6 FIGURE 5 shows a block diagram illustrating a computer system in accordance with one or more embodiments. DETAILED DESCRIPTION

[0014] In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding. One or more embodiments can be practiced without these specific details. Features described in one embodiment can be combined with features described in another embodiment. In some examples, well-known structures and devices are described with reference to a block diagram form in order to avoid unnecessarily obscuring the present description.

[0015] 1. OVERALL SUMMARY

[0016] 2. SYSTEM ARCHITECTURE

[0017] 2.1 Terminology

[0018] 2.2 Supply notification analysis system architecture

[0019] 3. SUPPLY PLAN DEVIATION ANALYSIS TECHNIQUES

[0020] 3.1 Machine learning engine training

[0021] 3.2 Supply plan deviation analysis

[0022] 4. DETERMINING ACCEPTABILITY OF REMEDIAL ACTIONS

[0023] 5. EXAMPLE EMBODIMENTS

[0024] 6. COMPUTER NETWORKS AND CLOUD NETWORKS

[0025] 7. HARDWARE OVERVIEW

[0026] 8. OTHER MATTERS; EXTENSIONS

[0027] 1. OVERALL SUMMARY

[0028] One or more embodiments described herein include analyzing supplier communications regarding deviations from a supply plan. Some embodiments can determine a severity of the deviation. For example, some embodiments can determine an impact on a supply chain, inventory levels, or production operations caused by the deviation. Some embodiments can also analyze the supplier communication to identify a remedial action submitted by the supplier to address the deviation from the supply plan and further determine whether the remedial action is acceptable. Some embodiments can use the analysis of the supply deviation, the severity of the deviation, the acceptability of the remedial action, and / or other factors to generate a supplier score. In some cases, this score can be generated relative to similar analysis performed for other suppliers, or the score is normalized based on any number of relevant factors to improve the basis for comparing scores.

[0029] A machine learning model classifies one or more attributes corresponding to a supply plan deviation and / or a remedial action. These attributes can be identified in a communication to or from a supplier. The communication can inform a buyer or recipient (equivalently referred to herein as an “entity”) of the deviation and / or the remedial action, or confirm the existence of the deviation and / or implementation of the remedial action by the supplier or the entity. The attributes can be used to determine a severity level of the deviation.

[0030] Example attributes include, but are not limited to, a type of impact associated with the deviation, a frequency of all types of deviations associated with a supplier, a frequency of the same type of deviation associated with a supplier or a particular product. The machine learning model can be trained based on a set(s) of communications from one or more suppliers communicating supply plan deviations and / or remedial actions. For supervised machine learning models, the machine learning model can be provided with associations between attributes and types of deviations, and acceptability of remedial actions by senior supply chain personnel with rich relevant experience.

[0031] In conjunction with machine learning, embodiments described herein may use natural language processing (NLP), keyword analysis, and / or sentiment analysis to analyze supplier notifications to detect sentiment associated with the messages. NLP (e.g., in combination with machine learning techniques) may be used to determine the attributes of supplier notifications, and the relationship between notifications of deviations and corresponding notifications of remedial actions. For example, the system may use NLP techniques to identify notifications from suppliers indicating remedial actions of delayed shipments and price discounts. The trained machine learning model may associate the information identified by the NLP with the attributes and further identify that the price discount is insufficient to remedy the supply disruption. This analysis may be used to generate a low value for the acceptability score of the remedial action. This information may also be used to generate supplier scores. Supplier scores may be generated for multiple suppliers and compared to each other to distinguish the performance of different suppliers. The system described herein may compare supplier scores relative to one or more thresholds to rate supplier performance. In some embodiments, the scores may be limited to different suppliers of the same product, thereby further improving the relevance of the comparison of different suppliers.

[0032] One or more embodiments described in the specification and / or set forth in the claims may not be included in this general summary section.

[0033] 2. System Architecture

[0034] Figure 1 Illustrated is an example system architecture of a system 100 configured to analyze supplier communications, identify deviations from a supply plan and remedial actions, determine whether the remedial actions are acceptable, and generate an acceptability score and a supplier score.

[0035] Before describing the example system 100 in detail, an explanation of various terms follows.

[0036] 2.1 Terminology

[0037] In one or more embodiments, a communication channel refers to a method or means for transmitting messages between users. Examples of communication channels include email, instant messaging, social media platforms, project collaboration applications, gaming applications, electronic photo albums, and specialized supply chain applications that facilitate the transmission of purchase orders, order fulfillment, and financial transactions. A correspondence chain or communication chain may include one or more messages transmitted between two or more users. These users may be associated with a purchasing entity ("entity") and that entity's suppliers.

[0038] In one or more embodiments, information (such as messages) obtained from one or more communication channels can be stored in one or more data repositories. A data repository is any type of storage unit and / or device (for example, a file system, database, collection of tables, or any other storage mechanism) for storing data. Additionally, a data repository can include multiple different storage units and / or devices. The multiple different storage units and / or devices can or can not be of the same type or located at the same physical site. Additionally, a data repository can be implemented or performed on the same computing system as the supply plan deviation system. Alternatively or additionally, a data repository can be implemented or performed on a separate computing system from the supply plan deviation system. A data repository can be communicatively coupled to the supply plan deviation system via a direct connection or via a network.

[0039] In one or more embodiments, a supply plan can correspond to an agreement between a supplier and a recipient for the supplier to supply one or more products under a set of conditions. Example conditions that can be used to delineate a supply plan include any one or more of the following: (1) one or more product identifiers, product descriptions, and / or product characteristics (e.g., color, size, units of sale packaging, units of sale quantity); (2) unit price; (3) quantity tier-based discounts on unit price; (4) discount percentage or value; (5) delivery method (e.g., ground, air, rush, direct); (6) ship date; (7) arrival date; (8) minimum quality level. These are provided for illustration only. A supply plan can be instantiated as a purchase order, a supply contract identifying periodic shipment quantities, a combination thereof, or other agreement by which a supplier supplies products to another entity using prearranged terms.

[0040] In one or more embodiments, a deviation from a supply plan can correspond to any condition delineated in the supply plan that is not met or will not be met during shipment. Example deviations include a drop in quality level, a product quantity or product characteristic that differs from those previously agreed upon, use of a substitute product that differs from that agreed upon in the supply plan (i.e., different part number or SKU), a change in price or applicable discount, a delayed or inappropriately expedited delivery, and the like.

[0041] In one or more embodiments, a remedial action is any action taken by a supplier in response to a deviation from a supply plan. Example remedial actions include, but are not limited to, offering a discount, revising a sales price, offering a substitute product in the event that a contract product cannot be supplied, expediting shipment of a product, offering a substitute shipment of a product that does not meet a quality level or product characteristic, and combinations thereof.

[0042] In one or more embodiments, an attribute associated with a supply plan deviation refers to a type of deviation. Examples of a deviation type include, but are not limited to, a change in one of the following: quantity of product to be supplied; price of product; delivery time; delivery location; bulk discount rate; quality level deficiency; and incorrect product. Another attribute includes a frequency of the deviation associated with one or more of the product and the supplier and a duration of the deviation. More specific examples of attributes (or, alternatively or additionally, data used to quantify the attributes) include timeliness of delivery of a product and / or service, price variance between a purchase order (PO) price and an invoice price, quality of a product and / or service provided by a supplier, and fulfillment rate. The price variance between the PO and invoice price can account for a currency difference between the PO and invoice price and / or a difference in number between the PO and invoice price. The quality of the product and / or service can account for a number of requested returns and / or a number of recalls. The fulfillment rate can account for a number of orders not delivered, a number of partially fulfilled, and / or a number of items canceled.

[0043] In some examples, in addition to or instead of product supply attributes, data directly related to attributes of the supplier itself (alternatively referred to as “external information”) can be considered. Supplier entity attributes include, for example, license factors and corporate diversity. License factors can account for sanctions imposed on the entity, data breaches, bankruptcy risk, and third-party assessments of credit such as FRISK score, FICO score, D&B score, Equifax score, number of employees employed by the entity, licenses of employees. Corporate diversity can account for minority ownership and / or percentage of employees hired.

[0044] In one or more embodiments, an emotional level of a supply plan deviation and / or remedial action can be identified. The emotional level reflects whether the sentiment expressed in a message is positive, negative, or neutral. In some embodiments, the emotional level can also indicate a severity level of the supply plan deviation and / or a confirmation of severity by the supplier aspect that caused the deviation. In embodiments, the sentiment analysis can identify a responsiveness level of a particular message. For example, a supplier can transmit a message (e.g., a message via a channel) to reply to an initiating message from a purchaser that identifies a supply issue. The system can also use sentiment analysis to determine a responsiveness level exhibited by a remedial action to a supply plan deviation. In one example, a customer can transmit a message requesting a refund. The supplier can reply with a message that provides an apology and the requested refund. The supplier’s message will be associated with a high responsiveness level. Additional and / or alternative attributes of the message can be reflected by a message rating.

[0045] In one or more embodiments, a supplier score represents a supplier's performance and / or excellence level. The supplier score represents the degree to which the supplier has met, satisfied, and / or will meet the terms of the agreed-upon supply plan (e.g., the expectations identified in a long-term contract or a discrete PO). As an example, a supplier's supplier score may represent the extent to which the supplier delivers high-quality products in a timely manner at a low price. As another example, the supplier score may be proportional to the frequency and severity of deviations from the supply plan generated by the supplier.

[0046] In one or more embodiments, the keyword library includes a collection of words, phrases, sentences and / or variables that can be used to analyze supply plans, supply plan deviations and remedial measures. Different keyword libraries can be associated with messages with different message attributes. The keyword library can indicate the relationship between (a) words and phrases and (b) deviation attributes (and particularly deviation type and severity). As described below, the keyword library can operate in coordination with the semantic analysis system and the machine learning system.

[0047] 2.2 Supply Notification Analysis System Architecture

[0048] Figure 1 The system 100 is illustrated in accordance with one or more embodiments. In some embodiments, the system 100 may determine a severity level of a supply plan deviation based on the type of impact on the supply status of a product caused by the corresponding supply plan deviation. In one or more embodiments, the system 100 may generate a supplier score that may be an absolute value, a relative value (i.e., a score of a particular supplier normalized relative to other suppliers of the same product(s), limited to a particular product, and combinations thereof. In some embodiments, the system may also generate an acceptability score for remedial actions proposed in response to the corresponding deviation. The system may include subsystems that perform machine learning, natural language processing, and / or sentiment analysis techniques.

[0049] like Figure 1 As shown in FIG, system 100 includes clients 102A, 102B, a machine learning (ML) application 104, a data repository 128, and external resources 124A, 124B. In one or more embodiments, system 100 may include Figure 1 More or fewer components than shown. Figure 1 The components shown in can be local to each other or remote from each other. Figure 1 The components shown in the figure can be implemented in software and / or hardware. Each component can be distributed across multiple applications and / or machines. Multiple components can be combined into one application and / or machine. The operations described with respect to one component can be performed by another component instead.

[0050] In some examples, the clients 102A, 102B can be web browsers, mobile applications, or other software applications communicatively coupled to a network. In other examples, the clients 102A, 102B can be associated with a human user (e.g., a system administrator, an inventory manager, a procurement or supply specialist), or associated with another application, such as a shell or client application. In some examples, the clients 102A, 102B are interfaces for communication between systems (e.g., a supplier communication channel, a product management system).

[0051] The clients can interact with embodiments of the machine learning application 104 instantiated as a cloud service using one or more communication protocols, such as HTTP and / or other communication protocols of the Internet Protocol (IP) suite. In other embodiments, where the ML application 104 can be instantiated as a local system (e.g., via an “on-premise” computer system), the clients 102A, 102B can be desktop or other standalone applications that can access the ML application 104.

[0052] Figure 1 The example ML application 104 shown in FIG. 1 includes a supplier communication system 106, a product management system 108, a machine learning engine 110, a front-end interface 118, and an action interface 120. In some embodiments, the ML application 104 is a cloud service, such as a software as a service (SaaS) or a web service. In other embodiments, the ML application 104 is an operating system on a dedicated system (e.g., in a private network, an “on-premise” computer system, a private distributed computer network system).

[0053] The supplier communication system 106 of the ML application 104 can facilitate communication with various suppliers as a dedicated supplier communication channel and / or as a single interface through which different channels can be operated or analyzed. The supplier communication system 106 can be used to place orders (e.g., POs) to suppliers, monitor notifications and / or messages regarding shipment and / or delivery of products, monitor fulfillment status of periodic shipments associated with long-term supply contracts, track order status, and / or track shipment status. The supplier communication system 106 can also receive notifications from suppliers regarding deviations from supply plans.

[0054] The supplier communication system 106 can communicate with other elements of the machine learning application 104. Upon receiving a supplier notification or other communication indicating a deviation, the supplier communication system 106 can process the notification and optionally share the notification with other elements of the machine learning application 104. The other elements of the machine learning application 104 can then apply the techniques described below (e.g., sentiment analysis, trained machine learning model analysis, supplier scoring) to analyze the notification, identify the deviation, identify a severity level of the deviation and a corresponding acceptability of a remediation measure, score the supplier, and other analysis operations.

[0055] In some examples, the product management system 108 can communicate with the data repository 128, monitor inventory levels, inventory consumption rates, expected re-supply quantities and receipt dates, production and outbound shipping schedules, and process both inbound and outbound purchase orders. In some cases, in addition to communicating with other elements of the machine learning application 104, the product management system 108 can perform many functions associated with inventory management systems. For example, the product management system 108 can receive queries about inventory levels submitted by clients (e.g., as operated by a supply manager) and present query results. The product management system 108 can monitor current inventory levels and can be used for general inventory management functions, such as performing quarterly “closes,” and storing customer and supplier profiles (e.g., addresses, financial information, payment history). The product management system 108 can also request, receive, and store schedules and shipping durations related to requests for submitted products as part of managing inbound supply of products and outbound delivery of products. For example, the product management system 108 can track receipt dates, shipping progress, and other timing and quantity aspects used to coordinate inbound and outbound product orders.

[0056] In some embodiments, the product management system 108 can receive analysis of a supplier notification from the machine learning engine 110 (described below) and use the analysis to estimate a supply impact for an associated product. For example, the product management system 108 can receive an indication from the machine learning engine that a particular product will be delivered two weeks after a contract delivery date. The product management system 108 can then query the data repository 128 to identify current inventory levels and perform an analysis of historical shipping and / or supply consumption data for the product over time. They can be compared to determine whether existing inventory for the product is sufficient to meet projected demand. The product management system can also store outstanding but unfulfilled product orders and use them as part of its analysis. Once performed, the estimated supply impact can be passed back to (or shared with) the machine learning engine 110 as a factor for identifying a severity of a supply plan deviation and / or an acceptability of a remediation measure.

[0057] The machine learning engine 110 includes training logic 112, communication analysis logic 113, acceptability evaluator logic 114, and vendor scoring logic 116. The training logic 112 can be used to train the machine learning engine 110 to identify associations between deviations from a supply plan and corresponding remedial measures proposed in response to the deviation. For example, the training logic 112 of the ML engine 110 can be trained by analyzing training data sets. These training data sets can include one or more notifications indicating a supply plan deviation, a remedial measure provided in response to the deviation, and an acceptability level of the remedial measure. The acceptability of the remedial measure can be indicated by, for example, a communication indicating an acceptance level (e.g., “resolve issue,” “is acceptable,” or “is not acceptable”) or by an applied label in a supervised learning model (and detected by NLP).

[0058] The machine learning model can also be trained to a determined acceptance standard. This training is described below in the context of Figure 4B The training can reflect different acceptability levels for different vendors, different supply products, and / or different supply metrics (alternatively referred to as “supply conditions” or “supply states”). Once the associations between ordered products and corresponding alternative products are established by the training logic 112, they can be stored in other elements of the system 100, such as the product management system 108 and / or the data repository 128.

[0059] The training logic 112 can identify and learn patterns by generating feature vectors of the analyzed communications and / or labels. That is, the ML engine 110 can include logic that identifies and extracts features from the communications and / or labels. These features can include, for example, customer identifiers, product identifiers, prices, order dates, scheduled fulfillment dates, deviation and remedial measure descriptions, sentiment, content identified by NLP. In some embodiments, using sentiment analysis and keyword analysis, the training logic 112 can also identify a severity level associated with the deviation.

[0060] The communication analysis logic 113 can use keywords (such as those stored in the keyword library 136 of the data repository 128), NLP, trained ML models, sentiment analysis techniques, and combinations thereof, to identify various aspects of communications to and from the vendor communication system 106. The communication analysis logic 113 can be configured to intercept or scrape one or more communication channels for messages identified by the system as being related to supply plan deviations and remedial measures. Once a communication is identified, the communication analysis logic 113 can identify the product that is the subject of the communication (e.g., by identifying a predetermined character pattern), the vendor (using keyword matching), and the deviation and corresponding remedial measure (using trained ML models, keyword analysis, NLP).

[0061] The communication analysis logic 113 can also perform the operations described in the context of one or more of Figure 3A 、 Figure 3B and FIG. 4 to determine bias and other aspects of remediation, such as various attributes.

[0062] The acceptability evaluator logic 114 can use information from other elements of the system 100 to evaluate whether a proposed remediation is acceptable. For example, the acceptability evaluator logic 114 can generate (or use previously generated) feature vectors for the bias description and the remediation description to determine whether the remediation is related to the bias. If the two are not related (e.g., a discount is offered in response to a missed delivery date), then the remediation can be unacceptable.

[0063] The acceptability evaluator logic 114 can also store rules and / or perform similarity analysis or other types of analysis for determining a severity level of a bias. The acceptability evaluator logic 114 can communicate with other elements of the system 100 or elements outside of the system 100 to determine a type and level of impact of a bias as part of determining whether a remediation is acceptable. For example, the acceptability evaluator logic 114 can communicate with the product management system 108 to identify whether a bias resulted in a production interruption. This information can be compared to the remediation to determine whether the remediation is acceptable.

[0064] The supplier score logic 116 can use bias attributes, such as severity level, frequency, type, etc., to generate a supplier score. In some cases, different factors and weights for bias attributes can be used to emphasize the importance of some aspects over others. In some cases, an entity can allow different sub-entities (e.g., departments, divisions, subsidiaries) to select different weights according to the priorities and preferences of the different sub-entities. In some cases, these weights can be selected on a product-by-product and / or sub-entity-by-sub-entity basis. For example, in some contexts, the weight of timely delivering an ordered product is the highest, while the weight of price is lower. Thus, a bias that causes delayed delivery is listed as highly severe, thereby greatly reducing the supplier score, while a price increase does not have a significant impact on bias severity or the supplier score.

[0065] In some examples, one or more elements of the machine learning engine 110 can use a machine learning algorithm to identify the above-described patterns. A machine learning algorithm is an algorithm that can be iterated using a set of training data to learn an optimal model f that maps a set of input variables to an output variable. The machine learning algorithm can include supervised components and / or unsupervised components. Various types of algorithms can be used, such as linear regression, logistic regression, linear discriminant analysis, classification and regression trees, Naive Bayes, k-nearest neighbors, learning vector quantization, support vector machines, bagging and random forests, boosting, backpropagation, and / or clustering.

[0066] In embodiments, the set of training data includes a dataset and associated labels. The dataset is associated with input variables for the target model f (e.g., target product identifier, target product description, customer identifier). The associated labels are associated with output variables of the target model f (e.g., replacement product identifier, replacement product description). The training data can be updated based on feedback, for example, regarding the accuracy of the current target model f. The updated training data is fed back into the machine learning algorithm, which in turn updates the target model f.

[0067] The machine learning algorithm generates the target model f such that the target model f best fits the dataset of the training data to the labels of the training data. Additionally or alternatively, the machine learning algorithm generates the target model f such that a maximum number of results determined by the target model f match the labels of the training data when the target model f is applied to the dataset of the training data.

[0068] The front-end interface 118 manages interactions between the ML application 104 and the clients 102A, 102B. For example, a client can submit requests to perform various functions and view results through the front-end interface 118. In some embodiments, the front-end interface 118 is a presentation layer in a multi-tiered application. The front-end interface 118 can process requests received from clients such as the clients 102A, 102B and translate results from other application layers into a format that the clients can understand or process. The front-end interface 118 can be configured to render user interface elements and receive input via the user interface elements. For example, the front-end interface 118 can generate web pages and / or other graphical user interface (GUI) objects. A client application such as a web browser can access and render interactive displays according to protocols of the Internet Protocol (IP) suite. Additionally or alternatively, the front-end interface 118 can provide other types of user interfaces, including hardware and / or software configured to facilitate communication between a user and the application. Example interfaces include, without limitation, GUIs, web interfaces, command line interfaces (CLIs), haptic interfaces, and voice command interfaces. Example user interface elements include, without limitation, checkboxes, radio buttons, drop-down lists, list boxes, buttons, toggle switches, text fields, date and time selectors, command lines, sliders, pages, and forms.

[0069] The action interface 120 provides an interface for performing actions using computing resources such as the external resources 124A, 124B. The action interface 120 can include an API, a CLI, or other interface for invoking functions to perform actions. One or more of these functions can be provided by a cloud service or other application that can be external to the ML application 104. For example, one or more components of the system 100 can invoke an API to communicate with a vendor via a communication channel. In another example, one or more components of the system 100 can invoke an API to a inventory, financial, or production system, which can provide information in response to a query related to a deviation (e.g., a severity level).

[0070] In some embodiments, the external resources 124A, 124B are network services external to the ML application 104. Example cloud services can include, but are not limited to, social media platforms, email services, short messaging services, enterprise management systems, verbal communication systems (e.g., voice over internet communications, text chat communications, POTS communication systems), and other cloud applications. The action interface 120 can serve as an API endpoint for invoking cloud services. For example, the action interface 120 can generate outbound requests that conform to protocols ingestible by the external resources 124A, 124B. The action interface 120 can process and translate inbound requests to allow further processing by other components of the ML engine 110. The action interface 120 can store, negotiate, and / or otherwise manage authentication information for accessing the external resources 124A, 124B. Example authentication information can include, but is not limited to, digital certificates, cryptographic keys, usernames and passwords. The action interface 120 can include authentication information in requests to invoke functionality provided by the external resources 124A, 124B.

[0071] In one or more embodiments, the system 100 can include one or more data repositories 128. A data repository is any type of storage unit and / or device (e.g., a file system, database, collection of tables, or any other storage mechanism) for storing data. Additionally, a data repository can include multiple different storage units and / or devices. The multiple different storage units and / or devices can or can not be of the same type or located at the same physical site.

[0072] A data repository, such as the illustrated data repository 128, can be implemented or executed on the same computing system as the machine learning application 104. The data repository 128 can be communicatively coupled to the machine learning application 104 via a direct connection or via a network.

[0073] An example data repository 128 includes a data partition 132 that stores inventory levels and product locations within a supply system. Storing this data enables other elements of the machine learning application 104 to identify current inventory levels, historical inventory levels, consumption rates, shipment arrival dates, and various other inventory management functions described above (e.g., in the context of the product management system 108).

[0074] An example data repository 128 also includes a data partition that stores a library of keywords 136. Keywords of the library of keywords 136 can be associated with different vendors, sentiments, biases, remedial actions, and bias attributes such as severity, type, frequency.

[0075] Examples of operations for training the machine learning application 104 are described below with reference to Figure 2 and Figure 4B Examples of operations for training the machine learning application 104 are described below with reference toFigure 3A and Figure 3B Examples of operations for using the machine learning application 104 are described.

[0076] In embodiments, the system 100, including the machine learning application 104, is implemented on one or more digital devices. The term “digital device” generally refers to any hardware device that includes a processor. A digital device can refer to a physical device that executes an application or a virtual machine. Examples of digital devices include a computer, a tablet, a laptop, a desktop, a netbook, a server, a web server, a network policy server, a proxy server, a general purpose computer, a function-specific hardware device, a hardware router, a hardware switch, a hardware firewall, a hardware firewall, a hardware network address translator (NAT), a hardware load balancer, a mainframe, a television, a content receiver, a set-top box, a printer, a mobile phone, a smart phone, a personal digital assistant (“PDA”), a wireless receiver and / or transmitter, a base station, a communication management device, a router, a switch, a controller, an access point, and / or a client device.

[0077] In one or more embodiments, the term “interface” refers to hardware and / or software configured to facilitate communication between a digital device or a user and the system 100. An interface can render user interface elements and receive input via the user interface elements. Examples of interfaces include those indicated above in the context of the system 100.

[0078] In embodiments, different components of an example interface can be specified in different languages. The behavior of a user interface element is specified in a dynamic programming language, such as JavaScript. The content of a user interface element is specified in a markup language, such as HyperText Markup Language (HTML) or XML User Interface Language (XUL). The layout of a user interface element is specified in a style sheet language, such as Cascading Style Sheets (CSS). In some examples, an interface can be specified in one or more other languages, such as Java, C, or C++.

[0079] Additional embodiments and / or examples related to computer networks are described in Section 6 below, entitled “Computer Networks and Cloud Networks.”

[0080] 3. Supply Plan Deviation Analysis Techniques

[0081] The following sections describe example techniques that can be used to train machine learning models in preparation for their application to the analysis of supply deviation notifications, the detection of the severity of supply deviations, remediation measures and their acceptability in resolving deviations, supplier scoring, and other aspects. Following the description of the training techniques, a description of example application techniques follows.

[0082] 3.1 Machine Learning Engine Training

[0083] Figure 2 FIG. illustrates an example set of operations (shown as method 200) for training a machine learning model to analyze deviations from a supply plan and determine the acceptability of remedial measures submitted in association with corresponding deviations, in accordance with one or more embodiments. The method 200 can begin with identifying a training set of training data, the training set including sets of corresponding supply plan deviations, remedial measures, and remedial measure acceptability. The acceptability of remedial measures in these training sets can be detected using sentiment analysis and / or indicated by labels supplied within the context of a supervised learning model (operation 204).

[0084] In some examples, various natural language processing (NLP) techniques and sentiment analysis tools can be used to analyze the training data set in order to efficiently identify notifications of the training data set. In other words, NLP can be used to identify relevant communications within a set of communications (e.g., associated with a particular product number, supplier, and / or a particular deviation), such that the relevant communications can be grouped together in a set. Such grouping can improve the accuracy of a trained machine learning model.

[0085] The system can optionally further analyze the training set to identify attributes corresponding to the severity level of the deviation, the type of deviation, the frequency, and the acceptability of the remedial measure (operation 208). For example, NLP, sentiment analysis, and keyword matching (matching to a library of keywords associated with sentiment) can be used to identify indications of the severity of the deviation. Communications (whether a notification from a supplier, a response to a notification from an entity, or other communications) including phrases such as “requires immediate attention,” “not acceptable,” “please resolve as soon as possible,” “please check and let us know,” “correct next shipment,” “reject,” “repeat error,” and the like imply different levels of urgency and severity. Similarly, terms and phrases such as “late,” “delayed,” “missed target” imply a type of deviation related to shipment or receipt times.

[0086] The system can also apply the above techniques to determine whether to communicate additional urgency to indicate that the purchasing entity is unable to operate as planned. For example, “out of stock,” “missed shipment,” and the like can indicate a business disruption due to the deviation, thereby increasing the severity of the deviation.

[0087] The frequency attribute can be determined by searching the training data set for a supplier identifier (name, number, ID, etc.), a product identifier, a contract number, and / or combinations thereof. The frequency attribute can be associated with a product, a supplier, or a contract.

[0088] Once the various training materials have been identified, an ML algorithm can be applied to the training data sets (operation 212). The ML algorithm analyzes the training data sets to identify data and patterns indicative of supply plan deviations and deviation attributes, remedial actions, and acceptability of remedial actions, as described above. Types of ML models include, but are not limited to, linear regression, logistic regression, linear discriminant analysis, classification and regression trees, Naive Bayes, k-nearest neighbors, learning vector quantization, support vector machines, bagging and random forests, boosting, backpropagation, and / or clustering.

[0089] In examples of supervised ML algorithms, the system can optionally obtain feedback regarding various aspects of the above-described analysis (operation 216). For example, the feedback can confirm or revise the severity of a deviation, the acceptability of a remedial action, revise (e.g., add or delete) attributes associated with a deviation, among other aspects. In some examples, the feedback can be limited to those authorized to provide feedback, such as supply chain managers, procurement managers, and the like (operation 220). In some cases, those authorized to provide feedback to the supervised learning model are the enforcement layer or otherwise have experience and authorization to judge the severity of a deviation.

[0090] Based on the associations identified by the machine learning model and / or feedback, the ML training set can be updated, thereby improving the accuracy of its analysis (operation 224). One benefit of using a trained machine learning model in this context is that it can improve the accuracy of the communication analysis, thereby reducing “false positive” and “false negative” interpretations of communications. That is, the system is less likely to incorrectly identify a communication as reporting a deviation, and is also less likely to incorrectly fail to identify a communication that properly reports a deviation.

[0091] Once updated, the ML model can be further trained by optionally applying it to additional training materials.

[0092] 3.2 Supply Plan Deviation Analysis

[0093] Figure 3A FIG. illustrates an example set of operations (shown as method 300) for applying a trained machine learning model to analyze deviations from a supply plan and determine the acceptability of remedial actions submitted in association with corresponding deviations, in accordance with one or more embodiments. The method 300 can begin with receiving a communication from a supplier that includes at least a remedial action (operation 304).

[0094] In some embodiments, communications may be received through a general communication channel, such as a general email interface. In some embodiments, communications may be received through a communication channel configured to manage supply plan communications. In some examples, the system may be configured to intercept or scrape one or more communication channels for messages that are identified or inferred by the system as being related to supply plan deviations and remedial actions. In some examples, the system may infer one or both of supply plan deviations and remedial actions in communications sent by an entity to its suppliers (e.g., from an entity stating "We noticed that this price is too high. Will you correct the PO price?"). Additionally or alternatively, the communication channel may transmit a copy of the message to the system for analysis. Such a communication channel may communicate with an inventory management system, a financial operations system, a purchase order management system, and the like.

[0095] In some examples, the system can identify separate communications related to a set of corresponding supply plan deviations and remedial actions by identifying one or more of a common purchase order number, a common part identifier (e.g., part number, part name, part description, SKU), a notification date, and a supplier identifier (name, account number, address) in the communications. Once the system identifies separate but related notifications, the system can perform the analysis described below.

[0096] Regardless of the channel through which the communication is received, one or more of the above techniques may be applied to the communication to determine a level of acceptability of the remedial action (operation 308). As part of determining the acceptability of the remedial action (operation 308), the system may also apply these techniques, alone or in various combinations thereof, to identify deviations and analyze the deviations to determine attributes associated with the deviations (operation 312). Figure 3B Various operations within operation 312 are illustrated, further illustrating various attributes that may be identified and used by the system to analyze deviations.

[0097] For example, go to Figure 3BOne attribute associated with a deviation that the system can detect is a severity level (operation 320). The system can determine some severity levels based on the parameters of the deviation itself (e.g., time, cost, part number) and without reference to any other aspects of the associated supply plan or remedial measures. For example, the system can compare the deviation to a set of rules that identify deviation parameter values ​​that indicate a severity level (operation 322). In some examples, the system can use NLP, keyword analysis, and trained machine learning models to extract parameter values ​​from communications so that the parameter values ​​can be compared to the rules. For example, the system can identify in a notification that a product will be received one month after the contracted delivery date. Based on a rule that associates late shipment with a severity level, the system can identify this delay as severe. The system can store and use similar rules that identify unexpected price increases greater than a threshold as severe (e.g., greater than 5%, greater than 10%). In another example, the system can identify that a product provided by a supplier is not the same part number as the ordered product. Based on the rules, this can also be identified as a severe deviation.

[0098] In other examples, the system may analyze the deviation and / or remedial action notification to determine the type of impact (operation 324). This analysis may also be performed using NLP, machine learning, sentiment analysis, keyword analysis, rules, and combinations thereof. For example, the system may use these techniques to determine that the type of deviation is any one or more of the following: (a) an unexpected price increase; (b) an inappropriate delivery date for a product; (c) an incorrect quality level; (d) an incorrect part number or product; or (e) an incorrect quantity. These example deviations are provided as examples and for ease of explanation. Based on the techniques described herein, the system may also identify other types of deviations.

[0099] Each of the illustrated supply plan deviations can have a different impact (or multiple different impacts) on the receiving entity based on the specific circumstances of the entity. In some examples, similar to the description given in the context of operation 320, the system can store a set of rules or can train a machine learning model to associate one or more of these deviations with a type of deviation (operation 324). The severity level associated with different types of deviations can be selected on a product-by-product or sub-entity-by-sub-entity basis.

[0100] In other examples, rules cannot easily specify the severity of deviations. For example, severity levels may vary depending on the entity's own use of the received product, its supply chain, operations, materials management processes, inventory status, production rates, and financial situation. These variables and their fluctuations can make fixed rules too imprecise to be useful.

[0101] In some examples, as part of the process for determining a severity level, the system can compare the determined impact type to one or more supply metrics (operation 328). Example supply metrics for characterizing a supply state can include, but are not limited to, the various supply chain conditions indicated above, such as production rate, inventory level, yield rate, outstanding orders, cash on hand, and so forth.

[0102] The system can vectorize the communication reporting the deviation, e.g., using NLP and machine learning techniques, to determine an impact type (operation 332). The system can then perform a query to determine various supply metrics related to the deviation (e.g., identify a part number in the deviation notification and search a product management system for a bill of materials that includes the part number). The system can then perform a similarity analysis (e.g., cosine similarity) to compare the deviation impact type to the various supply metrics (operation 332). This comparison can determine whether the deviation is similar to the supply metrics related to the deviation. For example, if the deviation indicates that a delivery will be delayed by one day, then if the current inventory within the receiving entity is less than the production rate multiplied by the duration of the delay (in this case, one day), then this can be identified as a severe deviation. In other words, production using the delayed component will cease due to the delay. This is a severe deviation despite the relatively short delay. If there are outstanding orders for the product that has ceased production due to the delay, then the severity can be further increased. Conversely, if the inventory level within the entity can accommodate production for multiple days, thereby avoiding a production disruption due to the delay, then a one-day delay can be non-severe.

[0103] A similarity score can be generated between the deviation and one or more supply metrics (e.g., inventory level, production rate, yield rate associated with the product, outstanding orders). In some examples, separate scores can be generated between the deviation and one or more supply metrics, and the resulting similarity scores are added together.

[0104] The similarity scores can be used to determine a severity level (operation 336). The higher the score, the more similar the deviation is to one or more supply metrics. The score can indicate the number of supply metrics with a similarity score above a threshold, thereby indicating a more widespread impact of the deviation. A higher score can indicate a high similarity to one supply metric, and the supply metric itself can indicate a severe deviation. Regardless, one or more thresholds can be established to progressively delineate higher severity levels of the deviation.

[0105] In addition to type and severity level, some embodiments can include one or more frequency of deviation as an attribute (operation 340). For example, in some examples, a frequency of deviation can be generated for individual suppliers (operation 344). This embodiment can include deviations on all products supplied by a supplier, thereby capturing the overall performance of the supplier. The system can identify a supplier identifier (e.g., email address, account number, unique identifier) and then use the supplier identifier to search for deviations associated with that supplier identifier within various data storage systems and communication channels.

[0106] In another example, the system can generate a frequency of deviation for a single product across multiple suppliers providing the product (operation 348). The system can identify a part identifier of interest and then use the part identifier to search for deviations associated with the part number within various data storage systems and communication channels. A Pareto chart (or similar assessment tool) can be generated to facilitate comparison of the performance of different suppliers to one another.

[0107] In some examples, any of the previous frequencies of deviation can be normalized to enable scaled comparisons between parts and / or suppliers (operation 352). For example, the frequency of deviation can be divided by the total number of units received (for a single supplier or across suppliers), divided by a revenue level, or any other convenient normalization factor that elevates the basis of comparison.

[0108] In the analysis, the frequency of deviation can be combined with one or both of the type of deviation and the severity level. For example, a deviation with a high severity level (e.g., above a corresponding threshold) but that does not occur frequently can be considered both serious and / or reduce the supplier score. A deviation with a low severity level but that occurs frequently can cause the system to generate a severity level similar to a rare but serious deviation. As with the other aspects described above, the frequency threshold can be established generally or can be adjusted according to the supply metric. For example, a critical part that is essential to continue production of multiple products, a component used in a revenue-generating product, or a part that has historically experienced supply planning deviations can have a low frequency of deviation threshold (i.e., trigger action with a low frequency of occurrence), while a less critical part can have a high threshold to indicate a serious deviation.

[0109] In some examples, the operations associated with operation 312 can optionally include comparing the type of impact determined in operation 324 to the remediation measures received in the notification of operation 304 to determine the degree to which the remediation measures respond to the deviation (operation 356). The system can perform this comparison using the techniques described above, including NLP, sentiment analysis, machine learning, keyword analysis, and similarity scoring. In this comparison, the system can identify a similarity between the remediation measures and the type of impact to determine whether the remediation measures proposed by the supplier are responsive to the deviation or related to the deviation. For example, if the remediation measures in the notification are discounts and the type of deviation identified by the system is a supply disruption, then the similarity analysis would indicate that the remediation measures do not actually reduce the impact of the deviation. In contrast, remediation measures in the notification that propose providing alternative products in response to a supply disruption deviation are responsive to the deviation.

[0110] Based on the comparison and the determined responsiveness, the system can generate a resolution value based on the similarity of the remediation measures (e.g., proportional to the co-signing similarity value). The system can use the resolution value to revise the severity level of the deviation (operation 360). In other words, if the remediation measures are responsive to the deviation, then the system can decrease the severity level of the deviation. If the remediation measures are not responsive to the deviation, then the system can maintain or increase the severity level of the deviation.

[0111] Returning to Figure 3A and method 300, after determining the various deviation attributes, the system can calculate an acceptability level of the remediation measures based on the deviation attributes and the associated remediation measures (operation 316). Much like the aspects described above, the acceptability can be based on one or more similarity scores determined between the deviation attributes and the corresponding remediation measures. The similarity can be determined based on one or more of keyword analysis, NLP analysis, trained machine learning analysis, etc. The more similar the attributes are to the remediation measures, the more acceptable the remediation measures are. In other words, the more likely the remediation measures are to reduce the impact caused by the deviation, the higher the acceptability score.

[0112] In some embodiments, the system can use the previous analysis to generate a supplier score based on one or more of the deviation attributes, severity levels, remediation measure acceptability (operation 318). For example, the supplier score can be generated proportional to the frequency of deviations, the absolute number of deviations normalized by the number of units purchased, the number of purchase orders issued, the amount of spend, the sum of the product of the deviations and the corresponding severity. Other techniques can be used to generate the supplier score. In particular, the normalized score can be used to compare the performance of different suppliers to one another. This in turn can be used to guide supplier management strategies.

[0113] For example, the system can use the various elements described above (e.g., type of deviation, severity of deviation, frequency of deviation (or multiple frequencies of deviation), appropriateness of remedial action) to generate a numerical score for each supplier. These scores can be normalized. Each normalized numerical score can be referred to as a "normalized supplier score." In some examples, the elements used to generate the numerical score can be weighted such that some elements contribute more to the score than other elements.

[0114] As an example, there can be a high percentage (e.g., greater than 10%, greater than 25%) price difference between the PO price and the invoice price. The customer can transmit a message to the supplier to identify the problem. The supplier can respond immediately with a message that includes an apology and a correction for the problem. While the deviation itself can be severe, the appropriateness of the remedial action and the low post-revision severity (assuming the remedial action completely resolved the deviation) can have a high weight in the scoring operation. While the existence of any deviation tends to lower the supplier score, the higher weighted elements reduce the impact of the deviation on the score.

[0115] In some embodiments, the supplier score can be revised if the transmitted deviation did not actually occur. For example, the remedial action can have resolved the deviation such that the price was adjusted, inventory was not depleted, production was not halted (as the deviation analysis initially indicated). In these cases, any drop in the supplier score can be reversed.

[0116] In embodiments, the supplier score can be presented on a user interface. The supplier score can be presented as a numerical value. Additionally or alternatively, the supplier score can be presented as a graph. The supplier score can be presented, for example, using a bar. The supplier score can be represented by a level filled with a bar.

[0117] In some examples, the supplier score can be presented on a user interface as part of a supplier profile. A user can gain an overall understanding of a supplier based on the supplier profile. In some examples, the system can present the supplier score as part of a candidate list of suppliers available to provide a product and / or service. The user can thereby compare suppliers and select the most appropriate supplier to purchase a product and / or service. In some examples, the system can present the supplier score as a tag associated with any occurrence of the supplier name on a user interface. The tag can be presented below the supplier name, or only when the mouse hovers over the supplier name. The user can thereby quickly review the level of performance associated with a supplier within any user interface with which the user interacts (such as an email application user interface, a web browser, a word document).

[0118] 4. Determine acceptability of remedial action

[0119] Figure 4A Example operations of an example method 400 by which remediation measures can be determined are illustrated. In one example, the method 400 can begin with receiving a notification from a supplier indicating (1) a particular deviation from a corresponding particular supply plan and (2) a particular remediation measure corresponding to the particular deviation (operation 404). Operation 404 is similar to the analogous operations described above.

[0120] The system can determine a severity level of the deviation by analyzing the deviation in the context of both the supply status associated with the product(s) indicated in the deviation and the remediation measure (operation 408). As described above in the context of Figure 3B the severity level can be determined using rules and using machine learning techniques that determine the impact of the deviation on, for example, revenue, productivity, customer shipment, cost, and other similar downstream effects caused by the deviation. In particular, operation 408 determines the severity level in the context of the supply status of the product associated with the deviation. The supply status metric is also described above in the context of Figure 3B

[0121] The system can then generate a remediation measure score based on the similarity of the remediation measure to the deviation (operation 412). As described above, the system can do so by generating feature vectors for the deviation and the remediation measure and performing a similarity analysis between the vectors. The more similar the remediation measure is to the deviation, the more likely the remediation measure is to address the deviation, and thus the higher the remediation measure score. For example, if the deviation and the remediation measure both involve the same part number, thereby suggesting that the remediation measure addresses the same component affected by the deviation, then the similarity score will be high. Similarity between the quantity and the deviation / remediation measure type (e.g., financial, quality, delivery date) will also improve the similarity score, and thus the remediation measure score.

[0122] The system then generates an indication of whether the remediation measure meets an acceptance criterion based on the severity level of the deviation and the remediation measure score (operation 416). For example, various thresholds can be established that gradually indicate the acceptability of the remediation measure score with the severity of the deviation. For example, for deviations with a high severity level, the acceptability threshold for the remediation measure can be based on a similarity between the remediation measure and the deviation that is greater than 0.8 or 0.9 (out of 1). This establishes a standard that requires an acceptable remediation measure to directly address the severe deviation and reduce the impact of the severe deviation. Less severe deviations can be associated with lower acceptability thresholds, whereby less similar remediation measures are designated as acceptable.

[0123] In variations of (and / or in addition to) the training techniques described above in the context of Figure 2 Figure 4B ​​The example operations of method 420 are illustrated by which a machine learning model can be trained to determine acceptance criteria for a remedial action. The system can apply many of the same techniques described above. In some embodiments, such techniques can be applied to the models used in method 400.

[0124] For example, the system can obtain (or be provided) communications with the supplier (operation 424). These communications can include deviation notifications (or other types of communications identifying deviations) and corresponding deviation attributes. The techniques described above in the context of Figure 3B identifying deviation attributes can be applied to operation 424.

[0125] The system can then identify training sets in the communications, where each training set includes communications of a corresponding supply plan deviation and a remedial action, and an indication of the appropriateness of the remedial action (operation 428). The training sets can be associated with supply conditions related to the product(s) associated with the remedial action and the deviation (operation 428). The supply conditions can be provided to the learning model, or the system itself can identify such information by identifying part numbers or product references within the communications and querying other elements of the system (e.g., via the supplier communication system 106 and / or the product management system 108 shown in FIG. 1) to obtain the information. Figure 1

[0126] In some examples, the system itself can determine the acceptability of the remedial action from the supply conditions. For example, the system can determine whether a delayed shipment would result in an out-of-stock condition for the product, and further identify the inappropriateness of a financial discount or the appropriateness of an alternative product for the supply. For example, as described above, the system (e.g., using the acceptability evaluator logic 114) can generate vectors representing the remedial action and the corresponding supply conditions, and use these vectors to generate a similarity score. The system can then use a threshold to establish acceptance criteria for the remedial action. A similarity score above the threshold can be used to indicate that the supply conditions and the remedial action are similar. This in turn can indicate that the remedial action addresses the supply conditions related to the deviation, and is therefore likely to meet the acceptability criteria.

[0127] ​The system can also analyze various combinations of supply conditions associated with a product and compare them to remediation plans using similarity analysis. The system can use this type of combination analysis to generate acceptance criteria for remediation plans that combine multiple factors associated with a product. For example, the system can check the inventory level of a product and the rate of consumption of inventory and extrapolate this data to determine whether a delayed shipment is likely to have an impact on production. In another example, the system can identify a replacement product number in a remediation action and search production records to identify whether the replacement product can be used in production (e.g., search a database identifying replacement products that are allowed to be used in production, such as via a "bill of materials" database.)

[0128] In other examples, the acceptability of remediation actions according to supply conditions is determined by user-supplied labels or similar supervised training techniques. The system can store labels related to various supply conditions according to associated supply plan deviations and remediation actions. These labels can be used to indicate one or more acceptance criteria for remediation actions. System-based techniques and user-based techniques can also be used in combination with one another.

[0129] The system can use one or more of the labels and / or similarity scores to train a machine learning model to determine acceptance criteria based on supply conditions, supply plan deviations, and remediation actions (operation 432).

[0130] In some examples, the acceptability is further determined based on user preferences, optionally (operation 436). For example, some organizations can require that certain materials be supplied constantly. In this case, a financially-based remediation action is less likely to match acceptance criteria, but a just-in-time delivery of an allowed replacement product is likely to match acceptance criteria. For other users, the opposite can be reflected in user preferences.

[0131] 5. Example Embodiments

[0132] For clarity, the following description details examples. The components and / or operations described below should be understood as one specific example that can not apply to certain embodiments. As such, the components and / or operations described below should not be interpreted as limiting the scope of any claims.

[0133] Figure 5 FIG. illustrates an example user interface 500 for presenting a supplier score and a communication indicating a deviation and a remediation action, in accordance with one or more embodiments.

[0134] As shown, the user interface 500 displays a supplier profile for the supplier Lee Supplies. The user interface 500 presents a numeric supplier score 502 and a graphical supplier score 504 for Lee Supplies. As indicated, the numeric supplier score 502 for Lee Supplies is “95.” At the same time, 95% of the bar that is graphically represented as the supplier score 504 is filled to represent the numeric supplier score 502 of “95.”

[0135] The user interface 500 presents information about a particular agreement with Lee Supplies, Agreement #A123. The user interface 500 indicates that, according to the agreement, PO #P456 is matched with invoice #I789. The user interface 500 presents information about a price variance between the PO and the invoice. The user interface 500 also shows communications exchanged with Lee Supplies about the price variance.

[0136] One price variance involves surgical gloves. In this example, a message 506 is transmitted to Lee Supplies to identify the issue. In this example, the system can use the message 506 from the entity that ordered the surgical gloves from the supplier Lee Supplies to identify the deviation as an improper price variance for the gloves. In some examples, a message 508 transmitted from Lee Supplies confirms the deviation and proposes a remedy (i.e., a price reduction). As described above, the system can analyze these messages via one or more of a purchase order (PO) number, an invoice number, an agreement number, and a supplier name to identify deviations and remedies.

[0137] The system can also analyze these communications to generate and / or update the supplier scores 502, 504. The system can generate and / or update the supplier scores 502, 504 based on the severity of the deviation and the acceptability of the remedy that directly addresses the deviation (i.e., correcting the incorrect price by reducing it). Moreover, a timely response to the deviation communication and a timely proposal of an acceptable remedy can also contribute to an improved supplier score. In one example, the responsiveness and acceptability of the remedy can further improve the supplier scores 502, 504.

[0138] In some examples, the system can even generate a recommendation 510 to include content in the message that replies to the message 508.

[0139] 6. Computer networks and cloud networks

[0140] In one or more embodiments, a computer network provides connectivity between a set of nodes. The nodes can be local to each other and / or remote from each other. The nodes are connected by a set of links. Examples of links include coaxial cables, unshielded twisted cables, copper wires, optical fibers, and virtual links.

[0141] A subset of the nodes implements the computer network. Examples of such nodes include switches, routers, firewalls, and network address translators (NATs). Another subset of the nodes uses the computer network. Such nodes (also referred to as “hosts”) can execute client processes and / or server processes. Client processes make requests for computational services, such as execution of a particular application and / or storage of a particular amount of data. Server processes respond by executing the requested services and / or returning corresponding data.

[0142] The computer network can be a physical network, including physical nodes connected by physical links. A physical node is any digital device. A physical node can be a function-specific hardware device, such as a hardware switch, a hardware router, a hardware firewall, and a hardware NAT. Additionally or alternatively, a physical node can be a general-purpose machine configured to execute various virtual machines and / or applications that perform corresponding functions. A physical link is a physical medium connecting two or more physical nodes. Examples of links include coaxial cables, unshielded twisted cables, copper wires, and optical fibers.

[0143] The computer network can be an overlay network. An overlay network is a logical network implemented on top of another network, such as a physical network. Each node in the overlay network corresponds to a respective node in the underlying network. Thus, each node in the overlay network is associated with both an overlay address (addressing the overlay node) and an underlay address (addressing the underlay node that implements the overlay node). An overlay node can be a digital device and / or a software process, such as a virtual machine, an application instance, or a thread. A link connecting overlay nodes is implemented as a tunnel through the underlay network. Overlay nodes at either end of the tunnel treat the underlying multi-hop path between them as a single logical link. Tunneling is performed by encapsulation and decapsulation.

[0144] In embodiments, a client can be local to the computer network and / or remote from the computer network. The client can access the computer network through other computer networks, such as a private network or the Internet. The client can transmit requests to the computer network using a communication protocol, such as the Hypertext Transfer Protocol (HTTP). The requests are transmitted through an interface, such as a client interface (such as a web browser), a programmatic interface, or an application programming interface (API).

[0145] In embodiments, a computer network provides connectivity between clients and network resources. Network resources include hardware and / or software configured to execute server processes. Examples of network resources include processors, data storage devices, virtual machines, containers, and / or software applications. Network resources are shared among multiple clients. Clients independently of each other request computing services from the computer network. Network resources are dynamically allocated to requests and / or clients on demand. Network resources allocated to each request and / or client can scale up or down based on, for example, (a) the computing service requested by a particular client, (b) the aggregate computing services requested by a particular tenant, and / or (c) the requested aggregate computing services of the computer network. Such a computer network can be referred to as a “cloud network.”

[0146] In embodiments, a service provider provides a cloud network to one or more end users. The cloud network can implement various service models, including but not limited to software as a service (SaaS), platform as a service (PaaS), and infrastructure as a service (IaaS). In SaaS, the service provider provides end users the ability to use an application executing on network resources of the service provider. In PaaS, the service provider provides end users the ability to deploy custom applications onto network resources. The custom applications can be created using programming languages, libraries, services, and tools supported by the service provider. In IaaS, the service provider provides end users the ability to provision processing, storage, networks, and other basic computing resources provided by network resources. Any arbitrary application, including an operating system, can be deployed on the network resources.

[0147] In embodiments, a computer network can implement various deployment models, including but not limited to private cloud, public cloud, and hybrid cloud. In a private cloud, network resources are provisioned for the exclusive use of a particular group of one or more entities (as the term “entity” is used herein to refer to a business, organization, individual, or other entity). The network resources can be local to the premises of the particular group of entities and / or remote from the premises of the particular group of entities. In a public cloud, cloud resources are provisioned for multiple entities (also referred to as “tenants” or “customers”) independently of each other. The computer network and its network resources are accessed by clients corresponding to different tenants. Such a computer network can be referred to as a “multi-tenant computer network.” Several tenants can use the same particular network resource at different times and / or at the same time. The network resources can be local to the premises of the tenants and / or remote from the premises of the tenants. In a hybrid cloud, a computer network includes a private cloud and a public cloud. An interface between the private cloud and the public cloud allows for portability of data and applications. Data stored at the private cloud and data stored at the public cloud can be exchanged through the interface. Applications implemented at the private cloud and applications implemented at the public cloud can have dependencies on each other. Calls from an application at the private cloud to an application at the public cloud (and vice versa) can be performed through the interface.

[0148] In embodiments, tenants of a multi-tenant computer network are independent of each other. For example, a tenant's business or operations can be separate from another tenant's business or operations. Different tenants can have different network requirements for the computer network. Examples of network requirements include processing speed, amount of data storage, security requirements, performance requirements, throughput requirements, latency requirements, resiliency requirements, quality of service (QoS) requirements, tenant isolation, and / or consistency. The same computer network can need to implement different network requirements required by different tenants.

[0149] In one or more embodiments, in a multi-tenant computer network, tenant isolation is implemented to ensure that applications and / or data of different tenants are not shared with each other. Various tenant isolation methods can be used.

[0150] In embodiments, each tenant is associated with a tenant ID. Each network resource of the multi-tenant computer network is tagged with a tenant ID. A tenant is only allowed to access a particular network resource if the tenant and the particular network resource are associated with the same tenant ID.

[0151] In embodiments, each tenant is associated with a tenant ID. Each application implemented by the computer network is tagged with a tenant ID. Additionally or alternatively, each data structure and / or data set stored by the computer network is tagged with a tenant ID. A tenant is only allowed to access a particular application, data structure, and / or data set if the tenant and the particular application, data structure, and / or data set are associated with the same tenant ID.

[0152] As an example, each database implemented by the multi-tenant computer network can be tagged with a tenant ID. Only a tenant associated with a corresponding tenant ID can access data of a particular database. As another example, each entry in a database implemented by the multi-tenant computer network can be tagged with a tenant ID. Only a tenant associated with a corresponding tenant ID can access data of a particular entry. However, the database can be shared by multiple tenants.

[0153] In embodiments, a subscription list indicates which tenants have access to which applications. For each application, a list of tenant IDs of tenants authorized to access the application is stored. A tenant is only allowed to access a particular application if the tenant's tenant ID is included in the subscription list corresponding to the particular application.

[0154] In embodiments, network resources corresponding to different tenants, such as digital devices, virtual machines, application instances, and threads, are isolated to tenant-specific overlay networks maintained by the multi-tenant computer network. As an example, a data packet from any source device in a tenant overlay network can only be sent to other devices within the same tenant overlay network. Encapsulation tunnels are used to prohibit any transmissions from a source device on a tenant overlay network to devices in other tenant overlay networks. Specifically, a data packet received from a source device is encapsulated within an outer data packet. The outer data packet is sent from a first encapsulation tunnel endpoint (in communication with the source device in the tenant overlay network) to a second encapsulation tunnel endpoint (in communication with a destination device in the tenant overlay network). The second encapsulation tunnel endpoint decapsulates the outer data packet to obtain the original data packet sent by the source device. The original data packet is sent from the second encapsulation tunnel endpoint to the destination device in the same specific overlay network.

[0155] 7. Hardware Overview

[0156] According to one embodiment, the technology described herein is implemented by one or more special-purpose computing devices. The special-purpose computing devices can be hardwired to perform the present technology, or can include digital electronic devices such as one or more application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or network processing units (NPUs) that are permanently programmed to perform the present technology, or can include one or more general purpose hardware processors programmed to perform the present technology pursuant to program instructions in firmware, memory, other storage, or a combination. Such special-purpose computing devices can also combine custom hard-wired logic, ASICs, FPGAs, or NPUs with custom programming to accomplish the present technology. The special-purpose computing devices can be a desktop computer system, a portable computer system, a handheld device, a networking device or any other device that incorporates hard-wired and / or program logic to implement the technology.

[0157] For example, Figure 6 FIG. 6 is a block diagram that illustrates a computer system 600 upon which an embodiment of the application can be implemented. Computer system 600 includes a bus 602 or other communication mechanism for communicating information, and a hardware processor 604 coupled with bus 602 for processing information. Hardware processor 604 can be, for example, a general purpose microprocessor.

[0158] Computer system 600 also includes a main memory 606, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 602 for storing information and instructions to be executed by processor 604. Main memory 606 also can be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor 604. Such instructions can be stored or implemented in non-transitory storage media accessible by processing system 600, such as storage device 610, hard disk, or other memory storage device. Such instructions when stored in non-transitory storage media accessible by processing system 600, make the computer system 600 into a special-purpose machine that is customized to perform the operations specified in the instructions.

[0159] Computer system 600 further includes a read only memory (ROM) 608 or other static storage device coupled to bus 602 for storing static information and instructions for processor 604. A storage device 610, such as a magnetic disk or optical disk, is provided and coupled to bus 602 for storing information and instructions.

[0160] Computer system 600 can be coupled via bus 602 to a display 612, such as a cathode ray tube (CRT), for displaying information to a computer user. An input device 614, including alphanumeric and other keys, is coupled to bus 602 for communicating information and command selections to processor 604. Another type of user input device is cursor control 616, such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to processor 604 and for

[0161] Computer system 600 can implement the techniques described herein using customized hard-wired logic, one or more ASICs or FPGAs, firmware and / or program logic which in combination with the computer system causes or programs computer system 600 to be a special-purpose machine. According to one embodiment, the techniques herein are performed by computer system 600 in response to processor 604 executing one or more sequences of instructions contained in main memory 606. Such instructions can be read into main memory 606 from another storage medium, such as storage device 610. Execution of the sequences of instructions contained in main memory 606 causes processor 604 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry can be used in place of or in combination with software instructions.

[0162] The term "storage media" as used herein refers to any non-transitory media that store data and / or instructions that cause a machine to operate in a specific fashion. Such storage media can comprise non-volatile media and / or volatile media. Non-volatile media includes, for example, optical disks or magnetic disks, such as storage device 610. Volatile media includes dynamic memory, such as main memory 606. Common forms of storage media include, for example, a floppy disk, a flexible disk, a hard disk, a solid state drive, magnetic tape, or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, NVRAM, any other memory chip or cartridge, content addressable memory (CAM), and ternary content addressable memory (TCAM).

[0163] Storage media is distinct from, but can be used in combination with, transmission media. Transmission media participates in transferring information between storage media. For example, transmission media includes coaxial cables, copper wire, and fiber optic cables, including wires that comprise bus 602. Transmission media can also take the form of acoustic or light waves, such as those generated during radio frequency and infrared data communications.

[0164] Various forms of media can be involved in carrying one or more sequences of one or more instructions to processor 604 for execution. For example, the instructions can initially be carried on a magnetic disk or solid state drive of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to computer system 600 can receive the data on the telephone line and use an infra-red transmitter to convert the data to an infra-red signal. An infra-red detector can receive the data carried in the infra-red signal and appropriate circuitry can place the data on bus 602. Bus 602 carries the data to main memory 606, from which processor 604 retrieves and executes the instructions. The instructions received by main memory 606 can optionally be stored on storage device 610 either before or after execution by processor 604.

[0165] Computer system 600 also includes a communication interface 618 coupled to bus 602. Communication interface 618 provides a two-way data communication coupling to a network link 620 that is connected to a local network 622. For example, communication interface 618 can be an integrated services digital network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, communication interface 618 can be a local area network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links can also be implemented. In any such implementation, communication interface 618 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.

[0166] Network link 620 typically provides data communication through one or more networks to other data devices. For example, network link 620 can provide a connection through local network 622 to a host computer 624 or to data equipment operated by an Internet Service Provider (ISP) 626. ISP 626 in turn provides data communication services through the world wide packet data communication network now commonly referred to as the "Internet" 628. Local network 622 and Internet 628 both use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on network link 620 and through communication interface 618, which carry the digital data to and from computer system 600, are example forms of transmission media for digital data.

[0167] Computer system 600 can send messages and receive data, including program code, through the network(s), network link 620 and communication interface 618. In the Internet example, a server 630 might transmit a requested code for an application program through Internet 628, ISP 626, local network 622 and communication interface 618.

[0168] The received code can be executed by processor 604 as it is received, and / or stored in storage device 610, or other non-volatile storage for later execution.

[0169] 8. Other items; extensions

[0170] Embodiments are directed to a system with one or more devices that include hardware processors and are configured to perform any of the operations described herein and / or recited in any of the claims below.

[0171] In embodiments, a non-transitory computer-readable storage medium includes instructions that, when executed by one or more hardware processors, cause performance of any of the operations described herein and / or recited in any of the claims.

[0172] According to one or more embodiments, any combination of the features and functions described herein can be used. In the foregoing specification, embodiments have been described with reference to numerous specific details that can vary from implementation to implementation. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense. The sole and exclusive indicator of the scope of the application, and what is intended by the applicants to be the scope of the application, is the literal and equivalent scope of the claims issued from this application, in whatever form that claim(s) can be issued, including any subsequent correction(s).

Claims

1. One or more non-transitory machine-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to facilitate a plurality of operations comprising: training a machine learning model to compute an acceptability level of a remedial action corresponding to a supply plan deviation, the training comprising: (a) obtaining a training dataset, each training dataset of historical data comprising: attributes of a supply plan deviation by a supplier; a remedial action agreed to or provided by the supplier for the supply plan deviation; and an acceptability level of the remedial action for the supply plan deviation; and (b) training the machine learning model based on the training dataset; receiving a communication from the supplier, the communication comprising a first remedial action for a first deviation from a first supply plan; and applying the machine learning model to compute a first acceptability level of the first remedial action for the first deviation from the first supply plan, the applying comprising: (a) analyzing the first deviation to determine a first set of attributes associated with the first deviation; and (b) computing the first acceptability level based on one or more similarity scores between the first set of attributes associated with the first deviation and the first remedial action.

2. The one or more non-transitory machine-readable media of claim 1, wherein the operations further comprise generating a supplier score for the supplier, the supplier score based on at least the first set of attributes and the first acceptability level.

3. The one or more non-transitory machine-readable media of claim 1, wherein computing the first acceptability level based on one or more similarity scores between the first set of attributes associated with the first deviation and the first remedial action comprises: computing a severity level of the first deviation based on the first set of attributes associated with the first deviation; and computing the first acceptability level using the severity level and the first remedial action.

4. The one or more non-transitory machine-readable media of claim 3, wherein the operations further comprise: analyzing the first remedial action to determine a resolution value for the first deviation; and changing a first value of the severity level of the first deviation to a second value of the severity level based on the resolution value.

5. The one or more non-transitory machine-readable media of claim 3, wherein determining the severity level of the first deviation comprises: analyzing the communication to determine a type of impact; comparing the determined type of impact to a set of supply metrics; generating a similarity score between the type of impact and the set of supply metrics; and determining the severity level of the first deviation using the set of supply metrics having a value of the similarity score above a threshold.

6. The one or more non-transitory machine-readable media of claim 5, wherein: the attributes used to determine the severity level comprise one or more of a frequency of total deviations, a frequency of a set of deviations sharing one or more attributes, and the type of impact.

7. The one or more non-transitory machine-readable media of claim 5, wherein the type of impact comprises one or more of a delayed shipment and a price fluctuation. ​ ​ ​ 8. The one or more non-transitory machine-readable media of claim 5, wherein the operations further comprise: comparing the type of the impact to a corresponding first remediation measure; and determining whether the first remediation measure resolves the type of impact.

9. The one or more non-transitory machine-readable media of claim 5, wherein comparing the determined type of the impact to the set of supply metrics comprises: identifying a product reference in the communication that is used to identify a supply status of a product associated with the product reference; based on comparing the type of the impact to the supply status of the product, determining whether the supply status is changed by the first deviation in the communication.

10. The one or more non-transitory machine-readable media of claim 3, wherein the frequency used to determine the severity level is normalized using other deviations from other suppliers different from the supplier that share the same attribute.

11. The one or more non-transitory machine-readable media of claim 1, wherein training the machine learning model further comprises: providing the machine learning model with a classification of supply plan deviations that identifies: an attribute for the supply plan deviation; and a severity level associated with the attribute of the corresponding supply plan deviation.

12. The one or more non-transitory machine-readable media of claim 1, wherein: calculating the first acceptability level based on one or more similarity scores between the first set of attributes associated with the first deviation and the first remediation measure comprises: calculating a severity level of the first deviation based on the first set of attributes associated with the first deviation; using the severity level with the first remediation measure to calculate the first acceptability level; analyzing the first remediation measure to determine a resolution value for the first deviation; and based on the resolution value, changing a first value of the severity level of the first deviation to a second value of the severity level, and the operations further comprise: training the machine learning model by providing the machine learning model with a classification of supply plan deviations that identifies an attribute of the supply plan deviation and a severity level associated with the attribute of the corresponding supply plan deviation; generating a supplier score for the supplier based on at least the first set of attributes and the first acceptability level; determining the severity level of the first deviation by analyzing the communication to determine a type of impact, comparing the determined type of the impact to the set of supply metrics, generating a similarity score between the type of the impact and the set of supply metrics, and using the set of supply metrics with values of the similarity score above a threshold to determine the severity level of the first deviation; comparing the type of the impact to a corresponding first remediation measure, and determining whether the first remediation measure resolves the type of impact, wherein the attributes used to determine the severity level include one or more of a frequency of total deviations, a frequency of a set of deviations that share one or more attributes, a normalized frequency, and the type of the impact, and wherein the type of the impact includes one or both of a delayed shipment and a price fluctuation. ​ identifying a product reference in the communication, the product reference used to identify a supply status of a product associated with the product reference; and determining whether the supply status is changed by a first deviation in the communication based on comparing the type of the impact to the supply status of the product.

13. A system having one or more processors configured to facilitate a plurality of operations comprising: receiving a notification from a supplier, the notification indicating a deviation from a corresponding supply plan and indicating a remedial action corresponding to the deviation; determining a severity level of the deviation by analyzing the deviation, the remedial action, and a supply status related to the deviation; generating a remedial action score based on a similarity of the remedial action to the deviation; and based on the severity level and the remedial action score, generating an indication of whether the remedial action meets an acceptance criteria.

14. The system of claim 13, wherein generating the indication of whether the remedial action meets the acceptance criteria comprises applying a trained machine learning model to the notification to analyze the deviation and the remedial action to determine the severity level and the acceptance criteria.

15. The system of claim 14, wherein the operations further comprise training the machine learning model, the training comprising: obtaining communications between an entity and a plurality of suppliers, including notifications of deviations from corresponding supply plans, the deviations including one or more attributes; identifying a training data set in the communications, the training data set including deviations, remedial actions corresponding to the deviations, and supply status information associated with the corresponding supply plans; and training the machine learning model to determine the acceptance criteria based on the associated supply status information, the deviations, and the remedial actions.

16. The system of claim 15, wherein the training comprises receiving user preferences from a supply chain supervisor regarding the acceptance criteria for remedial actions with respect to a set of supply status information elements.

17. A method for determining acceptability of a remedial action for a supply plan deviation, comprising: training a machine learning model to compute an acceptability level of remedial actions corresponding to supply plan deviations, the training comprising: (a) obtaining training data sets of historical data, each training data set including: attributes of a supply plan deviation of a supplier; a remedial action agreed to or provided by the supplier for the supply plan deviation; and an acceptability level of the remedial action for the supply plan deviation; and (b) training the machine learning model based on the training data sets; receiving a communication from a supplier, the communication including a first remedial action for a first deviation from a first supply plan; and applying the machine learning model to compute a first acceptability level for the first remedial action for the first deviation from the first supply plan, the applying comprising: (a) analyzing the first deviation to determine a first set of attributes associated with the first deviation; and (b) computing the first acceptability level based on one or more similarity scores between the first set of attributes associated with the first deviation and the first remedial action. ​ 18. The method of claim 17, wherein calculating the first acceptability level based on one or more similarity scores between the first set of attributes associated with the first deviation and the first remedial action comprises: calculating a severity level of the first deviation based on the first set of attributes associated with the first deviation; and using the severity level with the first remedial action to calculate the first acceptability level.

19. The method of claim 18, further comprising: analyzing the first remedial action to determine a resolution value for the first deviation; and based on the resolution value, changing a first value of the severity level of the first deviation to a second value of the severity level.

20. The method of claim 17, wherein: calculating the first acceptability level based on one or more similarity scores between the first set of attributes associated with the first deviation and the first remedial action comprises: (a) calculating a severity level of the first deviation based on the first set of attributes associated with the first deviation, (b) using the severity level with the first remedial action to calculate the first acceptability level, (c) analyzing the first remedial action to determine a resolution value for the first deviation, and (d) based on the resolution value, changing a first value of the severity level of the first deviation to a second value of the severity level, and further comprising: training a machine learning model by providing a classification of a supply plan deviation to the machine learning model, the classification identifying attributes of the supply plan deviation and a severity level associated with the attributes of the corresponding supply plan deviation; generating a supplier score for the supplier, the supplier score based on at least the first set of attributes and the first acceptability level; determining the severity level of the first deviation by analyzing the communication to determine a type of impact, comparing the determined type of impact to a set of supply metrics, generating a similarity score between the type of impact and the set of supply metrics, and using the set of supply metrics having a value of the similarity score above a threshold to determine the severity level of the first deviation; comparing the type of impact to a corresponding first remedial action; and determining whether the first remedial action resolves the type of impact; wherein the attributes used to determine the severity level include one or more of a frequency of total deviations, a frequency of a set of deviations sharing one or more attributes, a frequency normalized, and the type of impact, and the type of impact includes one or both of a delayed shipment and a price fluctuation; identifying a product reference in the communication, the product reference used to identify a supply status of a product associated with the product reference; and based on comparing the type of impact to the supply status of the product, determining whether the supply status is changed by the first deviation in the communication. ​ ​

Images