Risk assessment system

Abstract

Systems and methods for assessing enterprise-level risk assessment of a vendor. The method comprises the following steps: receiving an indication of a supplier; generating influence scores of the suppliers; generating a likelihood score of the supplier; generating a combined risk score based on the generated impact score and the generated likelihood score; evaluating the generated combined risk score relative to a dynamic risk threshold; and generating an output including the generated combined risk score based on the evaluation.

Description

(19) State Intellectual Property Office (12) Invention Patent Application (10) Publication Number (43) Publication Date (21) Application Number 202480066500.7 (22) Application Date 2024.10.09 (30) Priority Data 63 / 544,308 2023.10.16 US (85) PCT International Application Entering National Phase Date 2026.04.16 (86) PCT International Application Application Data PCT / IB2024 / 059885 2024.10.09 (87) PCT International Application Publication Data WO2025 / 083514 EN 2025.04.24 (71) Applicant: Kof Brand LLC, Address: New Jersey, USA (72) Inventors: D. Merritt, K. Dusso, J.P. Harlembeck, D.A. Yarosinski, P. Tendick, C.J. Van Schkindl (74) Patent Agency: Shanghai Patent & Trademark Agency Co., Ltd., 31100 Patent Attorneys: Gao Jian, Huang Songquan (51) Int.Cl. G06Q 10 / 0635 (2006.01) (54) Invention Title: Risk Assessment System (57) Abstract: A system and method for assessing enterprise-level risk of suppliers. The method includes: receiving instructions to suppliers; generating an impact score for suppliers; generating a probability score for suppliers; generating a combined risk score based on the generated impact score and the generated probability score; evaluating the generated combined risk score relative to a dynamic risk threshold; and generating an output including the generated combined risk score based on the evaluation. Claims 3 pages, Description 18 pages, Drawings 7 pages, CN 122055735 A 2026.05.15 CN 1 22 05 57 35 A 1. A computer-implemented method, the computer-implemented method comprising: receiving an instruction to a supplier; generating an influence score for the supplier; generating a probability score for the supplier; generating a combined risk score based on the generated influence score and the generated probability score; evaluating the generated combined risk score relative to a dynamic risk threshold; and generating an output including the generated combined risk score based on the evaluation. 2. The computer-implemented method of claim 1, further comprising: determining the dynamic risk threshold based on one or more of the following: risk tolerance; supplier type; services supplied by the supplier; time of year; supplier history; availability of alternative suppliers; data type to be provided to or received from the supplier; whether an intrusion has been identified at the supplier, and if so, how long ago; and the amount to be paid to the supplier for the services supplied by the supplier.3. The computer-implemented method of claim 1, further comprising: determining that the supplier is an existing supplier of the organization; and retrieving previously generated influence scores and previously generated probability scores of the existing supplier. 4. The computer-implemented method of claim 3, further comprising: determining to update the previously generated influence scores and previously generated probability scores of the existing supplier; updating the previously generated influence scores and previously generated probability scores of the existing supplier; generating an updated combined risk score based on the updated influence scores and updated probability scores; evaluating the generated combined risk score relative to the dynamic risk threshold; and generating an updated output including the updated combined risk score based on the evaluation. 5. The computer-implemented method of claim 3, further comprising: determining not to update the previously generated influence scores and previously generated probability scores of the existing supplier; retrieving the previously generated combined risk score of the existing supplier; evaluating the previously generated combined risk score relative to the dynamic risk threshold; and generating a new output including the previously generated combined risk score based on the evaluation. 6. The computer-implemented method of claim 1, further comprising: triggering additional analysis on the supplier based on the generated combined risk threshold being lower than the dynamic risk threshold; and performing a second assessment of the generated combined risk score relative to the dynamic risk threshold based on the additional analysis. 7. The computer-implemented method of claim 6, wherein the additional analysis comprises: generating a questionnaire including at least one question that, when answered, provides additional information about at least one of the impact or risk probability of an event concerning the supplier; sending the questionnaire to the supplier; receiving a response from the supplier, the received response including the additional information in response to the at least one question; and analyzing the received additional information received from the supplier. Claims 1 / 3 Page 2 CN 122055735 A 8. The computer-implemented method of claim 1, further comprising: implementing a first machine learning (ML) model to generate the impact score; and implementing a second ML model to generate the probability score. 9. The computer-implemented method of claim 8, further comprising: receiving feedback on the generated influence score; and updating the first ML model based on the received feedback.10. The computer-implemented method of claim 8, further comprising: receiving feedback on the generated probability score; and updating the second ML model based on the received feedback. 11. The computer-implemented method of claim 1, wherein the probability score is the probability that the supplier will become a victim of a cyberattack. 12. The computer-implemented method of claim 1, wherein the probability score is the probability that the supplier will become a victim of an operational failure. 13. A system comprising: a memory; and a processor coupled to the memory, the processor configured to: receive an instruction to a supplier; generate an impact score for the supplier; generate a probability score for the supplier, the generated probability score indicating the probability that a new supplier will become a victim of one or more of a cyberattack or an operational failure; generate a combined risk score based on the generated impact score and the generated probability score, the generated combined risk score indicating the overall risk of the supplier based on the supplier's impact and the generated probability score; evaluate the generated combined risk score relative to a dynamic risk threshold; and generate an output including the generated combined risk score based on the evaluation. 14. The system of claim 13, wherein the processor is further configured to: determine the dynamic risk threshold based on one or more of the following: risk tolerance; supplier type; services supplied by the supplier; time of year; supplier history; availability of a replacement supplier; data type to be provided to or received from the supplier; whether an intrusion has been identified at the supplier, and if so, how long ago; and the amount to be paid to the supplier for the services supplied by the supplier. 15. The system of claim 13, wherein the processor is further configured to: determine that the supplier is an existing supplier of the organization; and retrieve previously generated impact scores and previously generated probability scores of the existing supplier. 16. The system of claim 15, wherein the processor is further configured to: determine to update the previously generated impact scores and previously generated probability scores of the existing supplier; update the previously generated impact scores and previously generated probability scores of the existing supplier; generate an updated combined risk score based on the updated impact scores and updated probability scores; evaluate the generated combined risk score relative to the dynamic risk threshold; and generate an updated output including the updated combined risk score based on the evaluation.17. The system of claim 15, wherein the processor is further configured to: determine not to update the previously generated impact score and the previously generated probability score of the existing supplier; retrieve the previously generated combined risk score of the existing supplier; evaluate the previously generated combined risk score relative to the dynamic risk threshold; and generate a new output including the previously generated combined risk score based on the evaluation. 18. One or more non-transitory computer-readable media, the one or more non-transitory computer-readable media storing instructions, which, when executed by a processor, cause the processor to: receive an instruction for a new supplier; generate an impact score for the new supplier; generate a probability score for the new supplier, the generated probability score indicating the likelihood that the new supplier will become a victim of one or more of a cyberattack or operational failure; generate a combined risk score based on the generated impact score and the generated probability score, the generated combined risk score indicating the overall risk of the supplier based on the supplier's impact and the generated probability score; trigger a questionnaire to be sent to the supplier based on the generated combined risk score being greater than a dynamic risk threshold; update the combined risk score based on a response received from the supplier in response to the sent questionnaire; and generate an output including the combined risk score based on the updated combined risk score. 19. The one or more computer-readable media of claim 18, further storing instructions that, when executed by the processor, cause the processor to: determine the dynamic risk threshold based on one or more of the following: risk tolerance; supplier type; services supplied by the supplier; time of year; the supplier's history; availability of alternative suppliers; data type to be provided to or received from the supplier; whether an intrusion has been identified at the supplier, and if so, how long ago; and the amount to be paid to the supplier for the services supplied by the supplier. 20. The one or more computer-readable media of claim 18, further storing instructions that, when executed by the processor, cause the processor to: implement a first machine learning (ML) model to generate the influence score; receive feedback on the generated influence score; update the first ML model based on the received feedback; implement a second ML model to generate the probability score; receive feedback on the generated probability score; and update the second ML model based on the received feedback.Claims 3 / 3 Page 4 CN 122055735 A Risk Assessment System

[0001] Cross-Reference to Related Applications

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 544,308, filed October 16, 2023, the entire contents of which are incorporated herein by reference. Background Art

[0003] Organizations assess suppliers of goods and services to determine the risks of using or not using the goods and / or services provided by the suppliers. Risk assessment may include cyber risk assessment, operational risk assessment, financial risk assessment, etc. Risk assessment assesses both the impact of an event on the organization and the likelihood of the event occurring. Risks are often manually derived, categorical, or numerical and are not well-tailored to the organization. Forecasted risks are assessed at the organizational level and characterize the likelihood of organizational loss due to adverse events or activities affecting one or more of the following aspects of organizational health: operations, finance, cybersecurity, compliance, regulatory, health and safety, supply chain, operational technology, enterprise technology, social, and environment. Risk assessments must meet several requirements, including being scale-adjustable, accurate, consistent, reliable, and customizable to an organization’s goals, priorities, and risk tolerance. Risk assessments must also be flexible and scalable enough to account for potential changes in an organization’s goals, priorities, and risk tolerance. Summary of the Invention

[0004] This summary is provided in a simplified form to introduce some concepts that will be further described in the detailed description below. This summary is not intended to identify key or essential features of the subject matter protected by the claims, nor is it intended to help determine the scope of the subject matter protected by the claims.

[0005] In one example, a computer-implemented method is provided. The method includes: receiving instructions to a supplier; generating an influence score for the supplier; generating a probability score for the supplier; generating a combined risk score based on the generated influence score and the generated probability score; evaluating the generated combined risk score relative to a dynamic risk threshold; and generating an output including the generated combined risk score based on the evaluation.

[0006] In another example, a system is provided. The system includes: a memory; and a processor coupled to the memory, the processor being configured to: receive an instruction to a supplier; generate a supplier impact score; generate a supplier probability score, the generated probability score indicating the likelihood that a new supplier will become a victim of one or more cyberattacks or operational failures; generate a combined risk score based on the generated impact score and the generated probability score, the generated combined risk score indicating the overall risk of the supplier based on the supplier's impact and the generated probability score; evaluate the generated combined risk score relative to a dynamic risk threshold; and based on the evaluation, generate an output including the generated combined risk score.

[0007] In another example, a computer-readable storage medium is provided. The computer-readable storage medium stores instructions that, when executed by a processor, cause the processor to: receive instructions to a supplier; generate an influence score for the supplier; generate a probability score for the supplier, the generated probability score indicating the likelihood that a new supplier will become a victim of one or more cyberattacks or operational failures; generate a combined risk score based on the generated influence score and the generated probability score, the generated combined risk score indicating the overall risk of the supplier based on the supplier's influence and the generated probability score; trigger a questionnaire to be sent to the supplier based on the generated combined risk score being greater than a dynamic risk threshold; update the combined risk score based on a response received from the supplier in response to the sent questionnaire; and generate an output including the combined risk score based on the updated combined risk score. Other technical features will be apparent to those skilled in the art from the following figures, description, and claims.

[0008] Brief Description of the Drawings The following detailed description will provide a better understanding of this specification in view of the accompanying drawings, in which: Figure 1 illustrates an example system for assessing the risk of a supplier or vendor; Figure 2 illustrates an example of a current state challenge for assessing risk; Figure 3 illustrates a system for generating the output of a risk assessment; Figure 4 illustrates an example computer-implemented method for generating the output of a risk assessment; Figures 5A to 5B illustrate example computer-implemented methods for generating the output of a risk assessment; and Figure 6 is a block diagram illustrating an example computing environment suitable for implementing one or more of the various examples disclosed herein.

[0009] In the drawings, corresponding reference numerals indicate corresponding parts. In Figures 1 to 6, the system is illustrated schematically. The drawings may not be drawn to scale. Detailed Description of the Embodiments

[0010] Various specific embodiments and examples will be described in detail with reference to the accompanying drawings. Where possible, the same reference numerals will be used throughout the drawings to refer to the same or similar parts. References made throughout this disclosure relating to specific examples and particular implementations are provided for illustrative purposes only, but are not intended to limit all examples unless indicated otherwise.

[0011] As described herein, for each supplier an organization introduces or considers introducing, a risk assessment is performed to assess both the impact of an event on the organization and the likelihood of the event occurring. However, particularly in organizations contracting with hundreds or thousands of different suppliers, performing a thorough risk assessment for each potential supplier is impractical or even impossible due to the rigor of performing each assessment and the amount of assessment to be completed.For example, each risk assessment must be tailored to the specific organization's objectives, priorities, and risk tolerance, and be flexible and scalable to account for changes in the organization's objectives, priorities, and risk tolerance. Risk assessments need to be current and not susceptible to inaccuracies due to obsolescence. Assessments must also be reasonably affordable relative to the size of the organization performing them. Assessment systems must include sufficient and appropriate automation to improve the effectiveness and scaleability of assessments without excessive or burdensome human intervention, while also providing a mechanism to capture guidance from human experts in customizing their risk score weights / factors. Therefore, there is a need for systems and methods that perform risk assessments in a timely, scale-adjustable, and dynamic manner.

[0012] Various examples of this disclosure recognize and take into account these challenges and provide risk assessment systems and methods that predict the impact of an event, the likelihood of the event, and the supplier's overall risk score for a specific supplier claiming to provide goods or services to a specific organization. The method includes: generating a supplier's influence score, generating a supplier's probability score, generating a combined risk score based on the influence score and probability score, evaluating the combined risk score relative to a dynamic threshold, and generating an output including the combined risk score based on the evaluation. Therefore, various examples of this disclosure provide a risk assessment system designed to allow it to be included as a bidirectional component in a larger, comprehensive enterprise risk quantification system. The output from the risk assessment system is appropriately structured to be fed into the enterprise risk quantification system, which then outputs objectives, priorities, and risk tolerance to be used by the risk assessment system. This allows the risk assessment system to be automatically tailored to the organization's objectives, priorities, and risk tolerance without excessive human intervention.

[0013] Various examples of this disclosure provide a risk assessment system that operates in an unconventional manner by implementing multiple predictive models that work collaboratively to determine the extent of a supplier's influence on the organization, determine the probability of risk events, generate a combined risk score for the supplier, and determine a dynamic risk threshold for the supplier. Dynamic risk thresholds are used to analyze combined risk scores to create a comprehensive risk picture of suppliers, enabling the assessment and balancing of risks and opportunities in partnering with them. Furthermore, as circumstances change, including changes in suppliers, payments to suppliers, whether actual intrusions are discovered at suppliers, and any changes in supplier involvement, such as whether suppliers are now receiving more or more sensitive data, or whether potential alternative suppliers have been lost or added, the predictive model utilizes dynamic risk thresholds based on various supplier-centric dynamics.Furthermore, as circumstances change, including changes in organizational goals, priorities, risk tolerance, etc., the predictive model uses dynamic risk thresholds based on various organization-centric dynamics.

[0014] The systems and methods described herein provide a technical solution to the inherent technical problems of performing risk assessments of technology vendors at scale, including generating new data structures to store and present vendor and organizational risk data, reducing the burden of human input or other human-computer interactions in traditional risk assessment processes, and reducing computational resource consumption by identifying existing vendors and performing simplified, efficient analysis that identifies updates to vendor and organizational dynamics and generates updated risk scores accordingly. Specifically, the various examples described herein depict performing an initial analysis for each potential new vendor, storing an initial risk score, and continuously monitoring vendors with risk scores above a dynamic risk threshold in response to events that may change the risk score. In the event of an event, the risk score is updated from the initial risk score and reassessed based on the change in risk score, rather than generating a new risk score every time a vendor is identified for analysis, thereby reducing the burden of computational resources when assessing the risk of a vendor or potential vendor.

[0015] Figure 1 illustrates an example system for assessing the risk of a vendor or reseller. The system 100 illustrated in Figure 1 is provided for illustrative purposes only. Other examples of the system 100 may be used without departing from the scope of this disclosure. In some examples, the system 100 generates risk assessments for suppliers or vendors, including but not limited to network risk assessments.

[0016] The system 100 includes a computing device 102, an external device 134, a server 136, and a network 138. The computing device 102 represents any device that executes computer-executable instructions 106 (e.g., as an application, operating system function, or both) to implement operations and functions associated with the computing device 102. In some examples, the computing device 102 includes a mobile computing device or any other portable device. Mobile computing devices include, for example, but not limited to, mobile phones, laptop computers, tablet computers, computing boards, netbooks, gaming devices, and / or portable media players. The computing device 102 may also include less portable devices such as servers, desktop personal computers, self-service kiosks, or desktop devices. Additionally, the computing device 102 may represent a set of processing units or other computing devices.

[0017] In some examples, computing device 102 includes at least one processor 108, memory 104 (which includes computer-executable instructions 106), and user interface device 110. Processor 108 includes any number of processing units and is programmed to execute computer-executable instructions 106. Computer-executable instructions 106 are executed by processor 108, by multiple processors within computing device 102, or by a processor external to computing device 102.In some examples, processor 108 is programmed to execute computer-executable instructions 106, such as those illustrated in the accompanying drawings (such as FIG. 6) described herein. In various examples, processor 108 is configured to execute computer-executable instructions such as those of a risk assessor 118 as described herein.

[0018] Memory 104 includes any number of media associated with or accessible by computing device 102. In some examples, memory 104 is located inside computing device 102. In other examples, memory 104 is located outside computing device 102, or both inside and outside computing device 102. For example, memory 104 may include both memory components located inside computing device 102 and memory components (such as server 136) located outside computing device 102. Memory 104 stores data, such as one or more applications 107. Applications 107 operate when executed by processor 108 to perform various functions on computing device 102. Application 107 may communicate with peer applications or services, such as network services accessible via network 138. In the example, application 107 represents a server-side service of an application running in the cloud, such as cloud server 136. In some examples, application 107 is an application for assessing the risk of a supplier or vendor and generating a risk assessment score for the supplier or vendor.

[0019] User interface device 110 includes a graphics card for displaying data to and receiving data from a user. User interface device 110 may also include computer-executable instructions (e.g., drivers) for operating the graphics card. In addition, user interface device 110 may include a display (e.g., a touchscreen display or a natural user interface) and / or computer-executable instructions (e.g., drivers) for operating the display. User interface device 110 may also include one or more of the following to provide data to or receive data from a user: a speaker, a sound card, a camera, a microphone, a vibration motor, one or more accelerometers, a Bluetooth® communication module, global positioning system (GPS) hardware, and a light-sensitive sensor. In a non-limiting example, a user inputs commands or manipulates data by moving the computing device 102 in one or more ways.

[0020] The computing device 102 further includes a communication interface device 112. The communication interface device 112 includes a network interface card and / or computer-executable instructions (such as drivers) for operating the network interface card. Communication between the computing device 102 and other devices (such as, but not limited to, user device 136) can occur using any protocol or mechanism over any wired or wireless connection.

[0021] The computing device 102 also includes a data storage device 114 for storing data 116.Data 116 includes, but is not limited to, databases storing influence scores associated with one or more suppliers or vendors, risk scores associated with one or more suppliers or vendors, and determined risk thresholds associated with one or more suppliers or vendors; one or more template questionnaires to be provided to suppliers or vendors; and questionnaires already received from one or more suppliers or vendors and including responses to questions contained in the questionnaires. In some examples, the database also includes various data associated with suppliers or vendors, including but not limited to expenditure data, i.e., the amount provided to suppliers or vendors in exchange for services; supplier categories; supplier quality data associated with suppliers; supplier data security classifications; supplier cybersecurity vulnerability risk data; and supplier operational, financial, environmental, regulatory, or logistical risk data. In some examples, the data is dynamically updated from authoritative data sources within and / or outside the organization, such as an internal procurement database containing all invoice data from third parties; an internal supplier quality database containing product and manufacturing quality data from all third parties in the supply chain; an internal asset management database containing IT metadata for assets hosted by third parties; an external cybersecurity monitoring database containing security risk data for all known companies with publicly accessible networked assets; and an external supply chain intelligence database containing operational risk data for companies commonly found in the global product supply chain.

[0022] Data storage device 114 may include one or more different types of data storage devices, such as, for example, one or more rotating disk drives, one or more solid-state drives (SSDs), and / or any other type of data storage device. In some non-limiting examples, data storage device 114 includes a redundant array of independent disks (RAID). In other examples, data storage device 114 includes a database. In this example, data storage device 114 is included within, attached to, inserted into, or otherwise associated with computing device 102. In other examples, data storage device 114 includes a remote data storage device, such as server 136, accessible by computing device 102 via network 138, which may be a remote data storage device, a data storage device in a remote data center, a cloud storage device, etc.

[0023] Computing device 102 also includes a risk assessor 118. In some examples, risk assessor 118 is an example of a dedicated processor or processing unit implemented on processor specification page 4 / 18, 8 CN 122055735 A 108. Risk assessor 118 assesses specific types of risk for a particular supplier or distributor.The risk assessor 118 includes an influence score generator 120, a probability score generator 122, a combined score generator 123, a threshold determiner 124, a risk analyzer 126, a questionnaire generator 128, an output generator 130, a notification generator 131, and a feedback receiver 132. Each of the influence score generator 120, probability score generator 122, combined score generator 123, threshold determiner 124, risk analyzer 126, questionnaire generator 128, output generator 130, notification generator 131, and feedback receiver 132 is a separate example of a dedicated processor or processing unit implemented on the risk assessor 118.

[0024] The risk assessor 118 assesses a specific type of risk for a particular supplier by: generating an impact score via an impact score generator 120, generating a probability score via a probability score generator 122, generating a combined risk score for the supplier via a combined score generator 123 based on specific data input, identifying one or more risk thresholds for the supplier via a threshold determiner 124, and evaluating the generated combined risk score relative to one or more risk thresholds. If the generated combined risk score exceeds one of the risk thresholds, the risk analyzer 126 determines that further analysis is needed and generates a questionnaire to be sent to the supplier. Based on additional information received in response to the questionnaire, the risk analyzer 126 performs additional analysis on the supplier. If the generated combined risk score does not exceed one of the risk thresholds, or after the additional analysis on the supplier, the output generator 130 generates an output containing the supplier's risk assessment. In some examples, the generated output triggers automated responses, including but not limited to supplier approval, limited supplier approval, supplier rejection, notifications generated by the notification generator 131 on user interface device 110 indicating that the generated output is available, one or more automatically generated actions to reduce supplier risk, such as automatically scheduling a desktop exercise or cybersecurity training session to show the supplier the greatest risk, automatically adding high-risk third parties to the organization's risk register for visibility and follow-up actions, and automatically adjusting relevant security controls to limit the scope of supplier access to sensitive data and systems.

[0025] Upon receiving an indication for a new supplier, influence score generator 120 generates an initial influence score for the supplier. In some examples, the score is related to the magnitude and strength of the potential negative impact the supplier may have on the organization operating risk assessor 118.In some examples, the impact score generator 120 is an example of a machine learning (ML) or artificial intelligence (AI) model that generates impact scores based on multiple weighted variables, including but not limited to supplier-related data 116, such as expenditure data, supplier category, supplier-related supplier quality data, supplier data security classification, supplier's predetermined criticality level, and concentration data. The predetermined criticality level is determined by the organization of the operational risk assessor 118, and the concentration data characterizes the uniqueness of the supplier and how many other suppliers can supply the same goods or services. Expenditure data is the amount provided to suppliers or vendors in exchange for goods or services. Supplier category is a category that defines the type of supplier, including but not limited to compound suppliers, artificial intelligence (AI) service providers, information technology (IT) service providers, human resources (HR) companies, law firms, logistics providers, call center providers, etc. Supplier quality data is a critical score that measures the criticality or importance of supply chain and manufacturing supplier sites based on several factors, including but not limited to manufacturing quality and performance, compliance, and operational risk. Data security classification is a classification of how suppliers securely protect a company’s sensitive data (such as intellectual property (IP) and personally identifiable information (PII)). However, it should be understood that these examples are provided for illustration only and should not be construed as limiting. Various examples of variables are possible, including but not limited to additional variables such as the number or quality of alternative suppliers, the type of data being accessed or provided to suppliers, the frequency of variable changes and / or the degree of dynamism of variables, etc. It should be understood that because data 116 is dynamically refreshed, the impact score generator dynamically and continuously updates its scoring output. Therefore, various examples in this disclosure include continuous monitoring of entities or suppliers and dynamically and automatically adjusting the scoring output of entities or suppliers in real time based on received updated data, which in turn leads to automatic updates of the risk scores and profiles of entities or suppliers.

[0026] The impact score generator 120 generates an initial impact score based at least in part on weighted variables. For example, if the supplier category, supplier quality data, and data security classification are the same for two suppliers, the first supplier for which the organization spends more will have a greater influence score than the second supplier for which the organization spends less. In some examples, the generated initial influence score is represented as a numerical value, such as a value between 0 and 10, 0 and 100, etc., where 0 represents the supplier's lowest risk, and the maximum value and 10, 100, etc., represent the supplier's highest risk.In other examples, the generated initial impact score is represented as a classification score, such as A, B, C, etc., where A indicates low risk of using the supplier. In other examples, the generated initial impact score is represented as a color, such as green, yellow, or red, where green indicates low risk, yellow indicates medium risk, and red indicates high risk.

[0027] Upon receiving an instruction for a new supplier, the probability score generator 122 generates an initial probability score for the supplier, which is related to the probability of events that the supplier will experience through the organization affected by the operational risk assessor 118. In some examples, the probability score generator 122 is an example of an ML or AI model based on multiple weighted variables to generate the probability score, including but not limited to data 116 associated with the supplier, such as cybersecurity risks due to known vulnerabilities, previous intrusions, poor security practices, or operational risks due to financial, logistical, regulatory, governmental, social, criminal, or other environmental factors. In some examples, the event is a network-related event, such as a cyberattack. In other examples, the event is operational, such as a supplier failing to deliver goods or services to the organization. For example, if a supplier has been a victim of cyberattacks in the past, the probability score will be higher than if the supplier has not been a victim of cyberattacks in the past. As another example, if a supplier has been a victim of multiple cyberattacks that have put confidential information at risk, or has recently been a victim of many cyberattacks, the probability score will be higher than if the supplier has been a victim of a single cyberattack, or has recently been a victim of fewer cyberattacks. As another example, if a supplier's location is subject to geopolitical restrictions that interfere with the supplier's ability to manufacture or ship goods across borders, the probability score will be higher than if the supplier has no geopolitical restrictions. Because data 116 is dynamically updated, the probability score generator dynamically and continuously updates its scoring output, as described herein.

[0028] The combined score generator 123 generates a combined risk score based on the generated initial impact score and the generated initial probability score. The combined risk score measures the supplier's overall risk based on both the impact the supplier has or will have on the organization and the probability that the supplier will be affected by events in the affected organization. By generating a combined risk score based on both impact and likelihood, rather than one or the other, the risk assessor 118 considers the nuances between suppliers who have a high impact on the organization but may or may not have a high risk likelihood, suppliers who have a low impact on the organization but may or may not have a high risk likelihood, and every variation between them.

[0029] In some examples, the combined score generator 123 generates a combined risk score by retrieving a matrix stored as data 116 in the data storage device 114 and mapping the generated initial impact score and the generated initial likelihood score onto the matrix.For example, a supplier with a high impact score and a high probability score will produce a high combined risk score indicating high risk. Conversely, a supplier with a low impact score and a low probability score will produce a low combined risk score indicating low risk. In some examples, the combined risk score is represented as a numerical value, such as a value between 0 and 10, 0 and 100, etc., where 0 represents the supplier's lowest risk, and maximum values ​​such as 10, 100, etc., represent the supplier's highest risk. In other examples, the combined risk score is represented as a classification score, such as A, B, C, etc., where A represents low risk using the supplier. In other examples, the combined risk score is represented as a color, such as green, yellow, or red, where green represents low risk, yellow represents medium risk, and red represents high risk. These examples are presented for illustration only and should not be construed as limiting. Specification 6 / 18 pages 10 CN 122055735 A

[0030] Threshold determiner 124 determines one or more thresholds to be used as thresholds for analyzing the risk represented by the combined risk score. In some examples, threshold determiner 124 determines a single threshold to be used. In other examples, threshold determiner 124 determines more than one threshold to use, such as two thresholds: a higher threshold and a lower threshold. In some examples, the determined thresholds are dynamic. In other words, even for the same supplier, the determined thresholds may change over time depending on changes in factors, including but not limited to the risk tolerance, goals, or priorities of the organization or individual implementing computing device 102, changes in focus on different or specific types of suppliers or services, the time of year, the supplier's history, etc. Threshold determiner 124 determines one or more thresholds by implementing an ML or AI model that considers various factors, including factors used to generate a combined risk score, and the risk tolerance, goals, or priorities of the organization or individual implementing computing device 102, the type of supplier, the services provided by the supplier, the time of year, the supplier's history, the availability of alternative suppliers for the services provided, the type of supplier involvement, including but not limited to the types of data to be provided to and / or received from the supplier, whether an intrusion has been identified at the supplier, and if so, how long ago, the amount to be paid to the supplier for the services, etc. In some examples, these factors use organizational goals, priorities, and risk tolerance data 116 retrieved from an external enterprise risk quantification system, such as a comprehensive risk analyzer 135 implemented on an external device (such as external device 134). Because data 116 is dynamically refreshed, the threshold determiner updates its thresholds dynamically and continuously in real time.

[0031] Risk analyzer 126 evaluates a combined risk score relative to one or more determined thresholds and analyzes the evaluation results relative to one or more thresholds.In the example where threshold determiner 124 determines a single threshold, risk analyzer 126 evaluates a combined risk score relative to the determined threshold. If the generated initial impact score is greater than the determined threshold, risk analyzer 126 marks the supplier as high-risk, and if the combined risk score is not greater than the determined threshold, risk analyzer 126 marks the supplier as not high-risk. In the example where threshold determiner 124 determines more than one threshold (such as two thresholds), risk analyzer 126 evaluates a combined risk score relative to each of the determined thresholds. If the combined risk score is greater than the higher determined threshold, risk analyzer 126 marks the supplier as high-risk. If the combined risk score is not greater than the higher determined threshold but greater than the lower determined threshold, risk analyzer 126 marks the supplier as medium-risk. If the combined risk score is not greater than the lower determined threshold, risk analyzer 126 marks the supplier as low-risk.

[0032] In the example where the supplier is marked as high-risk, questionnaire generator 128 generates a questionnaire to be sent to the supplier. The generated questionnaire can be dynamically updated, tailored to the risks identified by the risk assessor 118, and used to collect additional information from the supplier as part of additional due diligence for further analysis of supplier risk. For example, the generated questionnaire may include questions about the supplier's general cybersecurity posture, including how the supplier encrypts and stores data, how the supplier handles privacy-related information, and how the supplier handles cybersecurity incidents. The output generator 130 generates an output including a combined risk score, as an analysis result of the risk analyzer 126 identifying the supplier as high-risk, low-risk, or medium-risk, or having other identified risk scores (such as numerical scores). In some examples, the generated output may also include one or more of the following: a generated initial impact score, a generated initial probability score, one or more thresholds determined for analyzing the combined risk score, data values ​​and weights for each factor among the factors used to determine the one or more thresholds, details about supplier involvement, whether a questionnaire was generated and output to the supplier, and if so, whether answers or responses were received from the supplier, and potential follow-up actions taken by the organization to document and address the identified risks. In some examples, the output from the output generator 130 is generated automatically and dynamically. In some examples, the output is presented on computing device 102, such as on user interface device 112. In some examples, the output is sent to external device 134 via network 138. In some examples, the output is structured as supplier risk control effectiveness data, which can be used as downstream input to enterprise risk quantification systems (such as a comprehensive risk analyzer 135 implemented on external device 134).

[0033] The notification generator 131 evaluates the output generated by the output generator 130 relative to the notification trigger data 116 to determine whether the notification should be delivered to one or more recipients via the external device 134. The notification trigger data 116 includes, but is not limited to, factors related to which topics will be used for triggering, thresholds for triggering for each topic, and the destination of the notification. In some examples, topics include, but are not limited to, the type of technology used by the vendor, the privacy or sensitivity of the data used by the vendor, etc. In some examples, thresholds include, but are not limited to, whether the services provided by the vendor to the organization include the use of AI, the organization's PII data, the organization's intellectual property, etc. In some examples, the destination of the notification includes various teams or groups within the organization, such as the AI ​​governance team, the procurement team, the privacy governance team, the data protection legal team, etc. In the example where the notification generator 131 delivers the notification to the recipients, the notification may be delivered in the form of emails delivered to individuals and teams, push notifications, internal enterprise messaging systems, etc.

[0034] The feedback receiver 132 receives feedback on the generated output. In some examples, the risk assessor 118 includes an ML or AI model that uses feedback received from human experts in the form of training data 116 and is updated to continuously improve aspects of the risk assessor 118, including but not limited to an influence score generator 120, a probability score generator 122, a combined score generator 123, a threshold determiner 124, a risk analyzer 126, an output generator 130, and a feedback receiver 132 itself. For example, the influence score generator 120 continuously improves its model to generate improved influence score predictions over time, and the probability generator 122 continuously improves its model to generate improved probability score predictions over time. Because the data 116 is dynamically refreshed, the feedback receiver 132 provides the risk assessor 118 with the data needed to continuously improve itself, especially in cases where the ML or AI model is automatically and dynamically retrained for human expert data.

[0035] In some examples, training data 116 includes details about the organization's relationships with third-party vendors, including vendor costs, criticality, quality, type of products or services, and whether the vendors handle sensitive data about the organization or its personnel. In some examples, training data 116 also includes ground-based real-world data on the highest-risk vendors, such as human expert labels, and labels for vendors involved in real-world cybersecurity vulnerabilities. The training data is used to train a risk assessor 118, and the output is an AI or ML model that predicts one or more risk thresholds tailored to the organization's risk priorities.As noted above, because data 116 is dynamically refreshed, feedback receiver 132 provides risk assessor 118 with the data needed to continuously improve itself, especially in cases where the ML or AI model is automatically and dynamically retrained against human expert data.

[0036] In some examples, one or more components of risk assessor 118 are designed to tend to mitigate or, in some examples, completely eliminate false negatives. In other words, the ML model of risk assessor 118 is tuned during training so that suppliers that are actually high-risk are not labeled as low-risk. Conversely, while this may tend to identify some suppliers as high-risk when, upon further review, these suppliers are actually medium- or low-risk, this design accepts the risk of performing additional analyses on some suppliers for which it would not otherwise be necessary, in order to prevent higher-risk suppliers from avoiding such additional analyses.

[0037] Furthermore, in some examples, the ML model of risk assessor 118 is trained using training data that identifies suppliers, various data values ​​associated with suppliers as described herein, and reported real-world results. For example, risk assessor 118 is trained on data that identifies a particular type of supplier as having a high impact on the organization and therefore a high risk score due to the amount spent on the supplier's contract, the sensitivity of the data the supplier receives from the organization, and the anticipated high impact on the organization if the supplier needs to be replaced due to the lack of other similar suppliers performing the same functions at the same scale. Various combinations of data values ​​are used to train risk assessor 118 to identify the risk of various types of suppliers. Specification 8 / 18 pages 12 CN 122055735 A

[0038] In some examples, risk assessor 118 is trained based on a particular organization or type of organization or a team of organizations that has deployed or is expected to deploy risk assessor 118 on computing device 102. For example, a small organization selling low-risk goods and services (such as groceries and personal care products) in a small market is expected to analyze and assess risk, which is very different from a large organization that uses software and personal information to market consumer health products or security software. Furthermore, even within an organization, different suppliers are treated very differently depending on the department or group that uses risk assessor 118 to analyze its suppliers. For example, a supplier for the human resources (HR) or legal department might have access to an employee's PII, which is subject to and necessarily protected by strict guidelines and regulations, while a supply chain provider might only accept purchase orders for a specific number of products. As described herein, training data 116 is used to train risk assessor 118, which identifies these differences and treats different categories of suppliers differently accordingly.

[0039] External device 134 is another example of a computing device that is separate from and located outside of computing device 102. In some examples, user device 134 includes a mobile computing device or any other portable device. Mobile computing devices include, for example, but not limited to, mobile phones, laptop computers, tablet computers, computing boards, netbooks, gaming devices, and / or portable media players. User device 134 may also include less portable devices such as servers, desktop personal computers, self-service terminals, or desktop devices. Additionally, user device 134 may represent a set of processing units or other computing devices. In some examples, server 136 is an example of an external storage device, a remote data storage device, a data storage device in a remote data center, or a cloud storage device. In one example, external device 134 is an enterprise risk quantification system that can use input from computing device 102 and transmit output to that computing device. In another example, external device 134 is an email server that receives email notifications from notification generator 131.

[0040] In an example where external device 134 is a comprehensive enterprise risk quantification system, the external device includes a comprehensive risk analyzer 135 that takes as input the outputs received from risk assessor 118 (such as those received from output generator 130 via communication interface device 112). The comprehensive risk analyzer 135 uses the outputs of risk assessor 118 (which is customized supplier risk data) as well as outputs from other risk assessment systems throughout the organization. Examples of other risk assessment systems used throughout the organization may include cybersecurity risk assessors, financial risk assessors, regulatory / compliance risk assessors, operational risk assessors, etc. Thus, risk assessor 118 is designed to fit a larger risk quantification ecosystem of systems, suppliers, sites, people, and other risk areas.

[0041] Figure 2 illustrates the technical problems presented by the current state of existing risk assessment systems. Figure 200 includes a first box (1), which illustrates that traditional manual risk assessment is only about 25% effective because it is performed manually, takes a lot of time to complete, and fails to capture important risk areas. Figure 200 also includes a second box (2) illustrating an existing monitoring service that captures some risks not captured in the first box (1), some of which are also captured by the first box (1), but also misses some of which are captured by the first box (1). Neither the first box (1) nor the second box (2) captures the risks included in the third box (3) illustrated in Figure 200.In other words, the methods illustrated in the first box (1) and the second box (2) cannot capture the essential parts of actual risk on their own. They illustrate the need for an improved risk assessment system capable of performing risk assessments of technology suppliers at scale, which generates new data structures to store and present supplier and organizational risk data, reducing the burden of manual input or other human-computer interaction in the traditional risk assessment process, and reducing the consumption of computing resources. The examples provided herein offer a technical solution to this inherent technical problem by implementing simplified and efficient analysis that generates a comprehensive risk profile for new suppliers and identifies updates to supplier dynamics and organizational dynamics for existing suppliers and generates updated risk scores accordingly.

[0042] Figure 3 illustrates a system for generating risk assessment outputs. System 300 is presented for illustration only and should not be construed as limiting. In some examples, system 300 may be implemented by one or more electronic devices described herein (such as computing device 102, as described on page 9 / 18 of the specification, CN 122055735 A).

[0043] System 300 includes one or more data inputs 302. One or more data inputs 302 stored as data 116 in data storage device 114 include, but are not limited to, expenditure data (i.e., the amount provided to suppliers or vendors in exchange for services), supplier categories, supplier quality data associated with suppliers, and supplier data security classifications. Data inputs 302 are fed into an impact score generator 304 and a probability score generator 306. In some examples, impact score generator 304 is an example of impact score generator 120, and probability score generator 306 is an example of probability score generator 122. Impact score generator 304 is a predictive ML model applied to all supplier risk functions, predicting the extent of supplier influence on the organization based on dynamically available data inputs. The level of influence is the generated impact score. In other words, the impact score is a measure of the level of impact the organization will feel if the supplier experiences a significant adverse event (e.g., intrusion, inoperability, or the need for replacement). As described in this paper, the impact score is quantified by one or more variables, such as the amount of money the organization spends on suppliers, the substitutability of suppliers, the number of other suppliers that can supply the same products, goods, or services, whether the supplier handles the organization's sensitive data or intellectual property (IP), and the type of supplier (e.g., supply chain provider, IT / software provider, law firm, HR provider, logistics or packaging company, marketing company). The probability score generator 306 is a predictive ML model applied to all supplier risk functions, predicting the likelihood that a supplier will be affected by events in the organization, such as cyberattacks, logistical incidents, bankruptcy, etc.A probability score is a measure of the likelihood of an event occurring and is quantified through self-verification and / or through objective verification of one or more variables, such as the frequency of the supplier's events, the severity of the events, the recentity of one or more events, the supplier's observable footprint or vulnerabilities (physical or digital), etc., as described herein.

[0044] The impact score generated by the impact score generator 304 and the probability score generated by the probability score generator 306 are provided to the risk score generator 308, which generates a risk score for the supplier. In some examples, the risk score generator 308 is an example of a combined score generator 123. As described herein, the generated risk score is a measure of the supplier's overall risk based on both the impact the supplier has or will have on the organization and the likelihood that the supplier will experience events affecting the organization. In some examples, a dynamically adjustable weighting of the impact score and the probability score is used to generate the generated risk score. The generated risk score is adjustable because it can be changed, and it is dynamic because it can be automatically adjusted based on triggers from updated or recently available data 116 (e.g., the type of supplier, the importance of the supplier in a specific time period, a measure of the quality or comprehensiveness of the data input, changes in organizational operations and / or strategy, etc.).

[0045] Risk analyzer 310 analyzes the generated risk score by evaluating the combined risk score relative to one or more determined thresholds and analyzing the results of the evaluation relative to one or more thresholds. In some examples, risk analyzer 310 is an example of risk analyzer 126. The generated risk score is included in output 310, which, in addition to the generated risk score, includes one or more determined thresholds for analyzing the generated risk score, data values ​​and weights for each of the factors used to determine the one or more thresholds, and, if a questionnaire is generated for the supplier, details of the questionnaire response and its associated comments and automatically generated scores. In some examples, output 310 is presented on computing device 102, such as on user interface device 112. In some examples, output 310 is sent to an external device, such as external device 134, via communication interface device 112.

[0046] FIG4 illustrates an example computer-implemented method for generating risk assessment output. This computer-implemented method 400 is presented for illustration only and should not be construed as limiting. Other examples of the computer-implemented method 400 may be used without departing from the scope of this disclosure. The computer-implemented method 400 may be implemented by one or more electronic devices (such as computing device 102) described herein. Specification 10 / 18 pages 14 CN 122055735 A

[0047] Method 400 begins in operation 402 with the risk assessor 118 receiving instructions to the supplier.Receiving instructions to a supplier triggers a risk analysis, which ultimately results in the generation of a detailed risk assessment and recommendation for the supplier as output. Instructions to suppliers can be received through various mechanisms. For example, the instructions can be received from manual input based on registration by the supplier or organizational stakeholders via a web or mobile application, or automatically based on the determination that potential or actual suppliers have been identified, respectively, in the organization's purchasing or invoicing systems.

[0048] In operation 404, the risk assessor 118 determines whether the identified supplier is a new supplier. A list of existing suppliers is stored as data 116 in data storage device 114 and / or server 136, containing unique alphanumeric identifiers of the suppliers identified in operation 402. In operation 404, the identifier of the supplier in question is cross-referenced with the supplier identifier stored as data 116 to determine whether the supplier is a new supplier. In the example where the supplier is not a new supplier, method 400 proceeds to operation 406, where risk assessor 118 retrieves the supplier's previously generated impact score, and in operation 408, risk assessor 118 retrieves the supplier's previously generated probability score. The previously generated impact score and probability score are each stored as data 116 in data storage device 114 and / or server 136, and are retrieved based on the supplier not being new (i.e., being an existing supplier). The previously generated impact score and probability score are each identified using an identifier label (such as reference number, name, etc.) that identifies the supplier. In operation 410, risk assessor 118 determines whether the additional data received along with the indication in operation 402 indicates that the previously generated impact score and / or probability score has substantially changed since a previous time when the data was stored in data storage device 116 or server 136. In the example where an update is required, method 400 proceeds to operation 412, and in the example where an update is not required, method 400 proceeds to operation 418.

[0049] In the example where the supplier is identified as new in operation 404 and / or determined to need updating in operation 410, in operation 412, the risk assessor 118 generates a new, updated impact score for the supplier. For example, the impact score generator 120 generates an initial impact score for the supplier based on multiple weighted variables, including but not limited to data 116 associated with the supplier, such as expenditure data, supplier category, supplier quality data associated with the supplier, and the supplier's data security classification, as described herein. In operation 414, the risk assessor 118 generates a new likelihood score for the supplier as described herein.

[0050] In operation 416, the risk assessor 118 generates a new combined risk score for the supplier based on the generated new impact score and the generated new likelihood score.As described herein, the combined risk score measures the supplier’s overall risk based on both the supplier’s influence on the organization and the likelihood of events affecting the organization.

[0051] In operation 418, risk assessor 118 determines one or more dynamic risk thresholds for the supplier. For example, threshold determiner 124 determines one or more thresholds to be used to analyze the risk represented by the combined risk score. In some examples, threshold determiner 124 determines a single threshold to be used. In other examples, threshold determiner 124 determines more than one threshold to be used, such as two thresholds, namely a higher threshold and a lower threshold. For example, an organization may need multiple risk thresholds to increase the fidelity of risk determination. Threshold determiner 124 meets this need by identifying and implementing multiple thresholds, and thus can be customized for the organization’s risk priorities.

[0052] In operation 420, risk assessor 118 evaluates the generated new combined risk score relative to the determined risk thresholds and determines whether the generated new combined risk score is greater than the determined risk thresholds. In some examples, the determined risk threshold on which the risk score is evaluated is the lowest of the determined one or more thresholds. In other examples, the risk threshold on which the risk score is based is the highest of one or more determined thresholds. In examples where the risk score is greater than the threshold, method 400 proceeds to operation 422. In operation 422, the risk assessor 118 determines whether the risk is acceptable based on data from automated or manual input. In an example where the risk is determined to be unacceptable (see page 11 / 18 of specification CN 122055735 A), in operation 424, the risk assessor 118 generates an output that triggers additional analysis of the supplier, i.e., a first output. For example, the generated output may trigger the generation and sending of a questionnaire to the supplier, which collects additional information about the supplier, its processes, internal systems, etc.

[0053] In operation 426, the risk analyzer 126 performs additional analysis. For example, the additional analysis performed may include analysis of a received questionnaire that includes answers to questions. In some examples, the additional analysis includes updating one or more of the impact score and the probability score, and generating an updated combined risk score. After the additional analysis, method 400 returns to operation 420, and risk assessor 118 evaluates the updated risk score relative to a threshold.

[0054] In the example where the risk score determined by operation 420 is below the threshold or operation 422 determines that the risk score is acceptable, method 400 proceeds to operation 428, and risk assessor 118 generates an output that includes at least the generated risk score, i.e., a second output.For example, output generator 130 generates an output including a combined risk score as an analysis result of risk analyzer 126 identifying the supplier as high-risk, low-risk, or medium-risk. In some examples, the generated output also includes one or more of the following: a generated initial impact score, a generated initial probability score, one or more thresholds determined for analyzing the combined risk score, data values ​​and weights for each of the factors used to determine the one or more thresholds, details about whether to generate a questionnaire and output it to the supplier, and a recommendation for the supplier. In some examples, the recommendation includes recommended methods for mitigating risks posed by the supplier, contractual language to be implemented to mitigate risks posed by the supplier, historical risks and events involving the supplier, etc. After generating the output in operation 428, method 400 terminates.

[0055] Figures 5A and 5B illustrate an example computer-implemented method for generating risk assessment output. This computer-implemented method 500 is presented for illustration only and should not be construed as limiting. Other examples of the computer-implemented method 500 may be used without departing from the scope of this disclosure. The computer-implemented method 500 may be implemented by one or more electronic devices (such as computing device 102) described herein. In some examples, the computer-implemented method 500 is implemented in the example of a supplier registration and being a new supplier (i.e., not an existing supplier). It should be understood that FIG5B is a continuation of FIG5A, and FIG5A and FIG5B together illustrate the computer-implemented method 500.

[0056] The computer-implemented method 500 begins in operation 502 by a risk assessor 118 registering an entity for a risk assessment system. For example, the risk assessor 118 may be an example aspect of a risk assessment system to which the entity registers, such as a cybersecurity risk assessment system, a financial risk assessment system, or the like. In some examples, the entity is a supplier, user, company, organization, etc.

[0057] In operation 504, an influence score generator 120 generates an initial risk score for the entity, and a threshold determiner 124 determines a risk threshold for the entity. In some examples, the influence score generator 120 generates an initial risk profile as described in operation 418. For example, in operation 418, threshold determiner 124 determines one or more risk thresholds that will be used as thresholds for analyzing risk represented by combined risk scores. In some examples, threshold determiner 124 determines a single threshold to use. In other examples, threshold determiner 124 determines more than one threshold to use, such as two thresholds, namely a higher threshold and a lower threshold.

[0058] In operation 506, risk assessor 118 evaluates the generated initial risk score relative to the determined risk thresholds and determines, as in operation 420, whether the generated initial risk score is greater than the determined risk thresholds.For example, the risk threshold on which the risk score is based is the lowest of one or more determined thresholds. In other examples, the risk threshold on which the risk score is based is the highest of one or more determined thresholds. In examples where the risk score is not greater than the threshold, the entity is determined to have minimal risk or no risk, and method 500 terminates. In examples where the risk score is greater than the threshold, method 500 continues to operation 508. Specification 12 / 18 pages 16 CN 122055735 A

[0059] In operation 508, risk assessor 118 distributes a questionnaire to the entity. In some examples, questionnaire generator 128 generates a questionnaire, and output generator 130 generates an output including the generated questionnaire, which is distributed to external devices associated with the entity via communication interface device 112. As described herein, the generated questionnaire includes questions about the entity's general cybersecurity posture, including how to encrypt and store data, how to handle privacy-related information, how to handle cybersecurity incidents, etc. In some examples, the external device associated with the entity is external device 134. In other examples, the external device associated with the entity is an additional external device such as external device 134 or server 136.

[0060] In operation 510, feedback receiver 132 receives responses to the questionnaire from the device associated with the entity via communication interface device 112. For example, a user of the entity may enter responses to the questionnaire questions and then send them to computing device 102 via communication interface 112. In operation 512, combined score generator 123 generates an updated risk score based on the received responses. The responses to the questionnaire provide additional details about the entity's profile, which may increase or decrease the entity's risk score.

[0061] In operation 514, risk analyzer 126 evaluates the generated new combined risk score relative to a determined risk threshold and determines whether the generated new combined risk score is greater than the determined risk threshold, as in operation 420. In the example where the risk score is not greater than the threshold, the entity is determined to have an acceptable risk level, and method 500 terminates. In the example where the risk score is greater than the threshold, method 500 continues to operation 516. In operation 516, the generated new combined risk score is stored, for example, as data 116, in data storage device 114 or on an external device (such as server 136).

[0062] In operation 518, risk analyzer 126 determines whether existing contracts with the entity are in effect. For example, risk analyzer 126 may determine whether existing contracts such as supply agreements, purchase orders, etc., are stored as data 116 in data storage device 114 or on server 136.If an existing contract is found, the risk analyzer 126 determines whether the found contract is still in effect, such as by determining the contract's termination date (which may be stored as metadata) and evaluating the termination date relative to the current date. If the current date is before the termination date, the contract is determined to be in effect, and the computer-implemented method 500 proceeds to operation 522. If the current date is after the termination date, the risk analyzer 126 determines that no existing contract is in effect and proceeds to operation 520, where the notification generator 131 generates a notification to an external device (such as external device 134) that may require a contract with the entity. In operation 520, if the contract is not in effect, the computer-implemented method 500 terminates. If the contract is in effect, the computer-implemented method 500 proceeds to operation 522.

[0063] In operation 522, the risk analyzer 126 continuously monitors for transactions or events that may affect the entity's risk profile. For example, the risk analyzer 126 monitors events for the entity, including but not limited to cybersecurity incidents, data breaches, etc. In operation 524, the combined score generator 123 determines whether an event has caused a change in the risk score. For example, the combined score generator 123 generates an updated combined score and then determines whether the updated combined score differs from the previous risk score, i.e., causing a change in the previous risk score. If the risk score has not changed, method 500 returns to operation 522, and the risk analyzer 126 continues to monitor transactions or events that may affect the entity's risk profile. If the risk score has changed, method 500 continues to operation 526, where the risk analyzer 126 determines whether the risk score has increased relative to the previous risk score.

[0064] If the risk score has increased in operation 526, in operation 528, the notification generator 131 appropriately generates a corresponding notification. In some examples, the appropriate notification is an alert indicating that the increased score requires manual review. In other examples, the appropriate notification is an alert triggering an increased score on a questionnaire to be distributed or an updated questionnaire, providing additional information about the event in response to that questionnaire or updated questionnaire. In some examples, the notification triggers additional steps to mitigate risk, including but not limited to meeting with the entity, continuous monitoring of the entity, etc. In some examples, an increase in the risk score due to the entity's existence at risk leads to the termination of the relationship with the entity. In some examples, an increase in the risk score triggers additional considerations to protect the organization that has deployed the risk assessor 118 due to the relationship with the entity. After operation 528, method 500 terminates.

[0065] In the case where the risk score has not increased, in operation 530, the risk analyzer 126 determines the degree of reduction in the risk score.Based on this reduction, in operation 532, risk analyzer 126 evaluates the generated new combined risk score relative to the determined risk threshold and determines whether the generated new combined risk score is greater than the determined risk threshold, as described herein. If the risk score is not greater than the threshold, notification generator 131 appropriately generates a corresponding notification. For example, an appropriate notification could be a notification that the change in risk score is insufficient to change the entity's risk profile. In the example where the risk score is greater than the threshold, method 500 proceeds to operation 536. In operation 536, the generated new combined risk score is stored, for example, as data 116 in data storage device 114 or on an external device (such as server 136). After operation 536, method 500 returns to operation 522, and risk analyzer 126 continues to continuously monitor transactions or events that may affect the entity's risk profile.

[0066] Example Operating Environment

[0067] FIG6 is a block diagram of an example computing device 600 for implementing the aspects disclosed herein, and this example computing device is generally referred to as computing device 600. Computing device 600 is an example of a suitable computing environment and is not intended to impose any limitation on the scope or functionality of the examples disclosed herein. Computing device 600 should also not be construed as having any dependencies or requirements relating to any of the components / modules or combinations shown. The examples disclosed herein can be described in the general context of computer code or machine-usable instructions, including computer-executable instructions (such as program components), executed by a computer or other machine (such as a personal data assistant or other handheld device). Typically, program components (including routines, programs, objects, parts, data structures, etc.) refer to code that performs a specific task or implements a specific abstract data type. The disclosed examples can be practiced in a variety of system configurations, including personal computers, laptops, smartphones, mobile tablets, handheld devices, consumer electronics, dedicated computing devices, etc. The disclosed examples can also be practiced in distributed computing environments when the task is performed by a remote processing device linked via a communication network.

[0068] The computing device 600 includes a bus 620 that is directly or indirectly coupled to the following devices: computer storage memory 602, one or more processors 608, one or more presentation units 610, I / O ports 614, I / O components 616, power supply 618, and network components 612. Although the computing device 600 is depicted as a single device, multiple computing devices 600 may work together and share the depicted device resources. For example, the memory 602 may be distributed across multiple devices, and the processor 608 may be housed with different devices.

[0069] The bus 620 indicates that it may be one or more buses (such as an address bus, a data bus, or a combination thereof).Although the various boxes in Figure 6 are shown with lines for clarity, alternative representations can be used to depict the various components. For example, in some examples, presentation components (such as display devices) are I / O components, and some examples of processors have their own memory. No distinction is made between categories such as “workstation,” “server,” “laptop,” and “handheld device,” as all of these are contemplated within the scope of Figure 6 and the scope of this document’s reference to “computing device.” Memory 602 may take the form of the computer storage medium reference below and operatively provides storage for computer-readable instructions, data structures, program modules, and other data for computing device 600. In some examples, memory 602 stores one or more of an operating system, a general-purpose application platform, or other program modules and program data. Thus, memory 602 is capable of storing and accessing data 604 and instructions 606, which can be executed by processor 608 and configured to perform the various operations disclosed herein.

[0070] In some examples, memory 602 includes computer storage media in the form of volatile and / or non-volatile memory, removable or non-removable memory, data disks in a virtual environment, or combinations thereof. Memory 602 may include any number of memories associated with or accessible by computing device 600. Memory 602 may be located inside computing device 600 (as shown in FIG. 6), outside computing device 600, or both. Examples of memory 602 include, but are not limited to, random access memory (RAM); read-only memory (ROM); electrically erasable programmable read-only memory (EEPROM); flash memory or other memory technologies; CD-ROM, digital universal disc (DVD) or other optical or holographic media; magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices; memory wired to an analog computing device; or any other medium for encoding desired information and making it accessible to computing device 600. Alternatively or additionally, memory 602 may be distributed across multiple computing devices 600, for example, in a virtualized environment where instruction processing is performed on multiple computing devices 600. For the purposes of this disclosure, “computer storage medium,” “computer storage memory,” “memory,” and “memory device” are synonymous terms for computer storage memory 602, and none of these terms include a carrier wave or propagation signaling.

[0071] Processor 608 may include any number of processing units that read data from various entities such as memory 602 or I / O components 616, and may include a CPU and / or a GPU. Specifically, processor 608 is programmed to execute computer-executable instructions for implementing aspects of this disclosure.These instructions can be executed by a processor, by multiple processors within computing device 600, or by a processor external to client computing device 600. In some examples, processor 608 is programmed to execute instructions such as those shown in the accompanying drawings. Furthermore, in some examples, processor 608 represents an implementation of an analog technique for performing the operations described herein. For example, the operations can be performed by analog client computing device 600 and / or digital client computing device 600. Presentation components 610 present data instructions to a user or other device. Exemplary presentation components include display devices, speakers, printing components, vibrating components, etc. Those skilled in the art will understand and recognize that computer data can be presented in a variety of ways, such as visually in a graphical user interface (GUI), audibly through speakers, wirelessly between computing devices 600, via a wired connection, or otherwise. I / O ports 614 allow computing device 600 to be logically coupled to other devices including I / O components 616, some of which may be built-in. Example I / O component 616 includes, for example, but not limited to, microphones, joysticks, game controllers, satellite antennas, scanners, printers, wireless devices, etc.

[0072] Computing device 600 can operate in a networked environment via network component 612 using a logical connection to one or more remote computers. In some examples, network component 612 includes a network interface card and / or computer-executable instructions (e.g., drivers) for operating the network interface card. Communication between computing device 600 and other devices can occur using any protocol or mechanism via any wired or wireless connection. In some examples, network component 612 is operable to transmit data via public, private, or hybrid (public and private) transport protocols, wirelessly transmit data between devices using short-range communication technologies (e.g., Near Field Communication (NFC), Bluetooth™ Branded Communication, etc.), or combinations thereof. Network component 612 communicates with cloud resource 624 across network 626 via wireless communication link 622 and / or wired communication link 622a. Various examples of communication links 622 and 622a include wireless connections, wired connections, and / or dedicated links, and in some examples, at least a portion is routed over the Internet.

[0073] Although described in conjunction with example computing device 700, the examples of this disclosure can be implemented with many other general-purpose or special-purpose computing system environments, configurations, or devices.Examples of well-known computing systems, environments, and / or configurations suitable for use with any aspect of this disclosure include, but are not limited to, smartphones, mobile tablets, mobile computing devices, personal computers, server computers, handheld or laptop devices, multiprocessor systems, game consoles, microprocessor-based systems, set-top boxes, programmable consumer electronics, mobile phones, wearable or accessory form factor mobile computing and / or communication devices (e.g., watches, glasses, headsets or headphones), network PCs, minicomputers, mainframe computers, distributed computing environments including any of the aforementioned systems or devices, virtual reality (VR) devices, augmented reality (AR) devices, mixed reality devices, holographic devices, etc. Such systems or devices may accept input from a user in any manner, including via input devices (such as keyboards or pointer devices), gesture input, proximity input (such as by hover), and / or voice input.

[0074] Examples of this disclosure can be described in the general context of computer-executable instructions (such as program modules) that are executed by one or more computers or other devices as software, firmware, hardware, or a combination thereof. Computer-executable instructions may be organized into one or more computer-executable parts or modules. Typically, program modules include, but are not limited to, routines, programs, objects, components, and data structures that perform a particular task or implement a particular abstract data type. Aspects of this disclosure may be implemented with any number and organization of such parts or modules. For example, aspects of this disclosure are not limited to specific computer-executable instructions, or specific parts or modules illustrated in the figures and described herein. Other examples of this disclosure may include different computer-executable instructions or parts having more or fewer functions than shown and described herein. In examples involving general-purpose computers, aspects of this disclosure transform a general-purpose computer into a special-purpose computing device when configured to execute the instructions described herein.

[0075] By way of example and not limitation, computer-readable media include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable memory implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, etc. Computer storage media are tangible and mutually exclusive with communication media. Computer storage media are implemented in hardware and are non-transient, i.e., they do not include carrier waves and propagating signals. The computer storage media used for the purposes of this disclosure are themselves not signals.Exemplary computer storage media include hard disks, flash drives, solid-state storage, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, optical disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage devices, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by a computing device. In contrast, communication media typically embody computer-readable instructions, data structures, program modules, etc., in the form of modulated data signals (such as carrier waves or other transmission mechanisms), and include any information delivery medium.

[0076] In some examples, a computer-implemented method includes: receiving instructions to a supplier; generating an influence score for the supplier; generating a probability score for the supplier; generating a combined risk score based on the generated influence score and the generated probability score; evaluating the generated combined risk score relative to a dynamic risk threshold; and generating an output including the generated combined risk score based on the evaluation.

[0077] In some examples, a system includes: a memory; and a processor coupled to the memory, the processor being configured to: receive an instruction to a supplier; generate a supplier impact score; generate a supplier probability score, the generated probability score indicating the likelihood that a new supplier will become a victim of one or more of a cyberattack or operational failure; generate a combined risk score based on the generated impact score and the generated probability score, the generated combined risk score indicating the overall risk of the supplier based on the supplier's impact and the generated probability score; evaluate the generated combined risk score relative to a dynamic risk threshold; and based on the evaluation, generate an output including the generated combined risk score.

[0078] In some examples, a computer-readable storage medium stores instructions that, when executed by a processor, cause the processor to: receive instructions to a supplier; generate an influence score for the supplier; generate a probability score for the supplier, the generated probability score indicating the likelihood that a new supplier will become a victim of one or more of a cyberattack or operational failure; generate a combined risk score based on the generated influence score and the generated probability score, the generated combined risk score indicating the overall risk of the supplier based on the supplier's influence and the generated probability score; trigger a questionnaire to be sent to the supplier based on the generated combined risk score being greater than a dynamic risk threshold; update the combined risk score based on responses received from the supplier in response to the sent questionnaire; and generate an output including the combined risk score based on the updated combined risk score.

[0079] Other examples are described herein.

[0080] Various examples also include one or more of the following: determining a dynamic risk threshold based on one or more of the following: risk tolerance; supplier type; services being supplied by the supplier; time of year; supplier history; availability of a replacement supplier; data type to be provided to or received from the supplier; whether an intrusion has been identified at the supplier, and if so, how long ago; and the amount to be paid to the supplier for the services supplied; determining that the supplier is an existing supplier of the organization; retrieving previously generated impact scores and previously generated probability scores of existing suppliers; determining to update previously generated impact scores and previously generated probability scores of existing suppliers; updating previously generated impact scores and previously generated probability scores of existing suppliers; generating an updated combined risk score based on the updated impact scores and updated probability scores; evaluating the generated combined risk score relative to a dynamic risk threshold; generating an updated output including the updated combined risk score based on the evaluation; determining not to update previously generated impact scores and previously generated probability scores of existing suppliers; retrieving previously generated combined risk scores of existing suppliers; evaluating the previously generated combined risk scores relative to a dynamic risk threshold; Based on the assessment, a new output including the previously generated combined risk score is generated; based on the fact that the generated combined risk threshold is lower than the dynamic risk threshold, additional analysis of the supplier is triggered; based on the additional analysis, a second assessment of the generated combined risk score is performed relative to the dynamic risk threshold; a questionnaire is generated, which includes at least one question that, when answered, provides additional information about at least one of the impact or risk probability of an event on the supplier; the questionnaire is sent to the supplier; a response is received from the supplier, the received response including additional information in response to at least one question; the received additional information received from the supplier is analyzed; a first machine learning (ML) model is implemented to generate an impact score; a second ML model is implemented to generate a probability score; feedback on the generated impact score is received; the first ML model is updated based on the received feedback; feedback on the generated probability score is received; the second ML model is updated based on the received feedback; wherein the probability score is the likelihood that the supplier will become a victim of a cyberattack; and wherein the probability score is the likelihood that the supplier will become a victim of an operational failure.

[0081] The order in which the operations in the examples of this disclosure illustrated and described herein are performed or carried out is not required and may be performed in different orders in various examples. For example, it is conceivable that a particular operation is performed or carried out before, simultaneously with, or after another operation within the scope of this disclosure.When elements of any aspect of this disclosure or examples thereof are introduced, as described on pages 17 / 18 of CN 122055735 A, the articles “a,” “an,” “the,” and “the” are intended to mean the presence of one or more of that element. The terms “comprising,” “including,” and “having” are intended to be inclusive and mean that additional elements may be present in addition to the listed elements. The term “exemplary” is intended to mean “an example of…”. The phrase “one or more of the following: A, B, and C” means “at least one A and / or at least one B and / or at least one C.”

[0082] Having described aspects of this disclosure in detail, it will be apparent that modifications and variations are possible without departing from the scope of the aspects of this disclosure as defined in the appended claims. Since various changes may be made to the above structures, products, and methods without departing from the scope of the aspects of this disclosure, everything contained in the above description or shown in the drawings should be interpreted as illustrative and not limiting. Instruction manual 18 / 18 page 22 CN 122055735 A Figure 1 Instruction manual Figure 1 / 7 page 23 CN 122055735 A Figure 2 Instruction manual Figure 2 / 7 page 24 CN 122055735 A Figure 3 Instruction manual Figure 3 / 7 page 25 CN 122055735 A Figure 4 Instruction manual Figure 4 / 7 page 26 CN 122055735 A Figure 5A Instruction manual Figure 5 / 7 page 27 CN 122055735 A Figure 5B Instruction manual Figure 6 / 7 page 28 CN 122055735 A Figure 6 Instruction manual Figure 7 / 7 page 29 CN 122055735 A.

Claims

1. A computer-implemented method, the computer-implemented method comprising: Receive instructions from suppliers; Generate the influence score of the supplier; Generate the probability score of the supplier; A portfolio risk score is generated based on the generated impact score and the generated probability score. The generated portfolio risk score is evaluated relative to a dynamic risk threshold; as well as Based on the assessment, an output including the generated combined risk score is generated.

2. The computer-implemented method according to claim 1, further comprising: The dynamic risk threshold is determined based on one or more of the following: risk tolerance; Supplier type; Services supplied by the aforementioned supplier; The time of year; The history of the supplier; availability of alternative suppliers; The data type to be provided to or received from the supplier; Whether the intrusion has been detected at the supplier's location, and if so, how long ago; and the amount to be paid to the supplier for the services provided.

3. The computer-implemented method according to claim 1, further comprising: It has been determined that the supplier is an existing supplier of the organization; as well as Retrieve the previously generated impact score and previously generated probability score of the existing supplier.

4. The computer-implemented method according to claim 3, further comprising: Determine and update the previously generated impact score and the previously generated probability score of the existing supplier; Update the previously generated impact score and the previously generated probability score of the existing supplier; An updated portfolio risk score is generated based on the updated impact score and the updated probability score; The generated combined risk score is evaluated relative to the dynamic risk threshold; as well as Based on the assessment, an updated output is generated, including the updated combined risk score.

5. The computer-implemented method according to claim 3, further comprising: Determine not to update the previously generated impact score and the previously generated probability score of the existing supplier; Retrieve previously generated combined risk scores from the existing suppliers; The previously generated combined risk score is evaluated relative to the dynamic risk threshold; as well as Based on the assessment, a new output is generated that includes the previously generated combined risk score.

6. The computer-implemented method according to claim 1, further comprising: If the generated combined risk threshold is lower than the dynamic risk threshold, additional analysis of the supplier is triggered. as well as Based on the additional analysis, a second assessment of the generated combined risk score is performed relative to the dynamic risk threshold.

7. The computer-implemented method of claim 6, wherein the additional analysis comprises: Generate a questionnaire that includes at least one question, which, when answered, provides additional information about at least one of the potential impact or risk of an event involving the supplier. Send the questionnaire to the supplier; Receive a response from the supplier, the received response including the additional information in response to the at least one question; as well as Analyze the additional information received from the supplier.

8. The computer-implemented method according to claim 1, further comprising: Implement the first machine learning (ML) model to generate the influence score; as well as Implement a second ML model to generate the probability scores.

9. The computer-implemented method according to claim 8, further comprising: Receive feedback on the generated impact scores; as well as The first ML model is updated based on the received feedback.

10. The computer-implemented method according to claim 8, further comprising: Receive feedback on the generated probability scores; as well as The second ML model is updated based on the received feedback.

11. The computer-implemented method of claim 1, wherein the probability score is the likelihood that the supplier will become a victim of a cyberattack.

12. The computer-implemented method of claim 1, wherein the probability score is the likelihood that the supplier will become a victim of operational failure.

13. A system comprising: Memory; and A processor, coupled to the memory, is configured to: Receive instructions from suppliers; Generate the influence score of the supplier; A probability score is generated for the supplier, which indicates the likelihood that the new supplier will become a victim of one or more cyberattacks or operational failures. A combined risk score is generated based on the generated impact score and the generated probability score. The generated combined risk score indicates the overall risk of the supplier based on the supplier's impact and the generated probability score. The generated portfolio risk score is evaluated relative to a dynamic risk threshold; as well as Based on the assessment, an output including the generated combined risk score is generated.

14. The system of claim 13, wherein the processor is further configured to: The dynamic risk threshold is determined based on one or more of the following: risk tolerance; supplier type; services supplied by the supplier; and time of year. The history of the supplier; availability of alternative suppliers; The data type to be provided to or received from the supplier; Whether the intrusion has been detected at the supplier's location, and if so, how long ago; and the amount to be paid to the supplier for the services provided.

15. The system of claim 13, wherein the processor is further configured to: It is determined that the supplier is an existing supplier of the organization; and Retrieve the previously generated impact score and previously generated probability score of the existing supplier.

16. The system of claim 15, wherein the processor is further configured to: Determine and update the previously generated impact score and the previously generated probability score of the existing supplier; Update the previously generated impact score and the previously generated probability score of the existing supplier; An updated portfolio risk score is generated based on the updated impact score and the updated probability score; The generated combined risk score is evaluated relative to the dynamic risk threshold; as well as Based on the assessment, an updated output is generated, including the updated combined risk score.

17. The system of claim 15, wherein the processor is further configured to: Determine not to update the previously generated impact score and the previously generated probability score of the existing supplier; Retrieve previously generated combined risk scores from the existing suppliers; The previously generated combined risk score is evaluated relative to the dynamic risk threshold; as well as Based on the assessment, a new output is generated that includes the previously generated combined risk score.

18. One or more non-transitory computer-readable media, said one or more non-transitory computer-readable media storing instructions, said instructions causing the processor, when executed by a processor, to: Receive instructions for new suppliers; Generate the impact score of the new supplier; A probability score is generated for the new supplier, which indicates the likelihood that the new supplier will become a victim of one or more cyberattacks or operational failures. A combined risk score is generated based on the generated impact score and the generated probability score. The generated combined risk score indicates the overall risk of the supplier based on the supplier's impact and the generated probability score. If the generated combined risk score is greater than the dynamic risk threshold, a questionnaire to be sent to the supplier is triggered. The combined risk score is updated based on the responses received from the supplier in response to the sent questionnaire; as well as Based on the updated combined risk score, an output including the combined risk score is generated.

19. The one or more computer-readable media of claim 18, wherein the one or more computer-readable media further stores instructions that, when executed by the processor, cause the processor to: The dynamic risk threshold is determined based on one or more of the following: risk tolerance; supplier type; services supplied by the supplier; and time of year. The history of the supplier; availability of alternative suppliers; The data type to be provided to or received from the supplier; Whether the intrusion has been detected at the supplier's location, and if so, how long ago; and the amount to be paid to the supplier for the services provided.

20. The one or more computer-readable media of claim 18, wherein the one or more computer-readable media further stores instructions that, when executed by the processor, cause the processor to: Implement the first machine learning (ML) model to generate the influence score; Receive feedback on the generated impact scores; Based on the received feedback, update the first ML model; Implement the second ML model to generate the probability scores; Receive feedback on the generated probability scores; and The second ML model is updated based on the received feedback.

Images