Shift-left topology construction and information augmentation using machine learning

Abstract

Methods, computer systems, and computer program products are provided for shift-left topology construction. Embodiments of the present invention may include collecting a dataset. Embodiments of the present invention may include extracting a topology entity from the dataset. Embodiments of the present invention may include correlating multiple data from the topology entity. Embodiments of the present invention may include mapping the topology entity. Embodiments of the present invention may include marking entry points of multiple subgraphs of the topology entity. Embodiments of the present invention may include constructing a topology graph.

Description

[Technical Field]

[0001] The present invention relates generally to the field of computing, and more specifically to topology construction and machine learning. Microservices architectures can be implemented in a distributed manner using many different topologies. Some include the use of centralized messaging topologies, Representational State Transfer (REST)-based topologies, and Representational Transfer Application Program Interface (REST-API)-based topologies. Summary of the Invention

[0002] Embodiments of the present invention disclose methods, computer systems, and computer program products for shift-left topology construction. Embodiments of the present invention may include collecting a dataset. Embodiments of the present invention may include extracting a topology entity from the dataset. Embodiments of the present invention may include correlating multiple data from the topology entity. Embodiments of the present invention may include mapping the topology entity. Embodiments of the present invention may include marking entry points of multiple subgraphs of the topology entity. Embodiments of the present invention may include constructing a topology graph. [Brief explanation of the drawings]

[0003] These and other objects, features, and advantages of the present invention will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompanying drawings. Since the illustrations are for clarity purposes only to facilitate understanding of the invention by those skilled in the art in conjunction with the detailed description, various features of the drawings are not to scale. The drawings are as follows:

[0004] [Figure 1] 1 illustrates a networked computing environment in accordance with at least one embodiment.

[0005] [Figure 2] FIG. 1 is a block diagram example of an constructed microservices topology according to at least one embodiment.

[0006] [Figure 3] 1 is an operational flowchart illustrating a process of topology construction and information augmentation using machine learning, according to at least one embodiment.

[0007] [Figure 4] FIG. 2 is a block diagram of internal and external components of the computer and server shown in FIG. 1 according to at least one embodiment.

[0008] [Figure 5] 2 is a block diagram of an exemplary cloud computing environment including the computer system shown in FIG. 1, according to one embodiment of the present disclosure.

[0009] [Figure 6] FIG. 6 is a block diagram of functional layers of the exemplary cloud computing environment of FIG. 5, in accordance with one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0010] Although detailed embodiments of the claimed structures and methods are disclosed herein, it should be understood that the disclosed embodiments are merely exemplary of the claimed structures and methods, which may be embodied in various forms. However, the present invention may be embodied in many different forms and should not be construed as limited to the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. In the description, details of well-known features and techniques may be omitted to avoid unnecessarily obscuring the presented embodiments.

[0011] As mentioned above, a microservices architecture can be implemented in a distributed manner using many different topologies. Some include the use of a centralized messaging topology, a Representational State Transfer (REST)-based topology, and a Representational Transfer Application Program Interface (REST-API)-based topology. An application program interface (API) that uses a Representational State Transfer (REST) ​​architecture can post, delete, get, or put data using HyperText Transfer Protocol (HTTP) requests. An application program interface (API) allows applications or software programs to communicate.

[0012] A microservices framework can be structured using smaller, independent architectural components that operate at high speeds. Microservices allow applications to be more easily managed and scalable. Independent components within a microservices framework may perform centralized operations, while other independent components within the same microservices framework may operate using different technologies. The independent scalability of microservices allows services to use a larger number of resources to provide multiple cloud-based products.

[0013] Many microservices frameworks operate using a container-based infrastructure. Containers can include isolated infrastructure that uses minimal resources, shares a host operating system, and is easy to integrate. Containers also allow for minimal software configuration, for example, because they can use the same software code in a delivery pipeline. Different containers can operate within the same microservices framework and run on alternative software code with minimal configuration. An application program interface (API) can include, for example, a set of functions or a communication protocol that allows a container to access data on a virtual machine, operating system, service, or application.

[0014] While topology information or topology data obtained from microservices is essential for microservice management stacks and analysis, a common component may not exist in a microservice or container framework to build the topology without deploying the entire microservice. For example, building the topology for 500 or more microservices is a major task during runtime. Microservices may be deployed with distributed tracing tools, and service mesh tools may be added to build the topology and additional information, such as latency, application program interface (API) connections, and overall resource consumption. Distributed tracing tools may include, for example, Jaeger, which is open-source software that performs end-to-end distributed tracing. Service mesh tools may include, for example, Istio, which is open-source software that helps create a load-balanced network of deployed services.

[0015] Topology information is built incrementally as microservice systems report data. Thus, management tools, such as distributed tracing and service mesh tools, use the incrementally built topology information as data is reported, but the management tools cannot function until the entire topology is built. Also, analysis services within a microservice framework typically require topology information before deploying the entire service, especially when training is offline. Therefore, it would be advantageous to create systems, methods, and program products for, among other things, continuously monitoring a microservice framework for changes in information, updating the topology graph, and predicting dependencies between microservices.

[0016] The exemplary embodiments described below provide a system, method, and program product for topology construction using machine learning. As such, embodiments of the present invention have the ability to improve the technical field of microservice topology construction by using machine learning to learn the behavioral characteristics of microservices in a microservice framework using static and dynamic information. More specifically, machine learning is used to learn the behavioral characteristics from data such as logs, communication transmissions, documentation, and subject matter expert (SME) input and feedback to predict a topology graph that can be used by analytical services and users.

[0017] According to one embodiment, shift-left analysis and machine learning can be used to build a microservice topology and validate topology information. Shift-left analysis can be obtained from source code and can include, for example, deployment configurations, deployment specifications, or environment variables to augment more detailed data for building the topology. An aspect of machine learning can be used to build a microservice topology by analyzing the deployment configuration or specification without runtime information, or by analyzing offline. Offline analysis can provide information that can be used to extract performance estimates for building the topology.

[0018] According to one embodiment, various types of machine learning (ML) models may be constructed and used to create predictions for building a topology of services and microservices. Data or information from different domains providing services and microservices may be acquired to build the topology. The domains may include, for example, retail, entertainment, social media, business, technology, healthcare, academia, government, industry, food chain, law, or automotive. Machine learning (ML) models may also include deep learning models and artificial intelligence (AI). Training and updating machine learning (ML) models may include supervised, unsupervised, and semi-supervised machine learning (ML) procedures. Supervised learning may use labeled datasets or labeled training sets to build, train, and update models. Unsupervised learning may use all unlabeled data to train models. Semi-supervised learning may use both labeled and unlabeled datasets to train models.

[0019] Supervised and semi-supervised learning may integrate feedback or ground truth by having individuals check the accuracy of the data, data labels, and data classifications. The individuals are typically subject matter experts (SMEs) with extensive knowledge in a specific area of ​​the dataset. The subject matter experts' (SMEs') input may represent the ground truth for the machine learning (ML) model, and the provided ground truth may increase the accuracy and predictive power of the model. The subject matter experts (SMEs) may correct, revise, update, or remove the classification of the data or data labels by manually updating the labeled dataset. Using subject matter experts (SMEs) to provide feedback to the model may improve the accuracy of the model as the dataset is updated or corrected.

[0020] According to one embodiment, supervised or semi-supervised machine learning (ML) may be used to allow an individual (e.g., a user, expert, or administrator) to have the ability to verify, modify, update, or alter the training set, thereby providing some effect improvement to the training of the model. Where a user may not provide input to the data, an expert may provide input or feedback to the model by modifying the training set, as opposed to an unsupervised machine learning (ML) environment. The training set of data may include classifier parameters, or labels, for learning purposes, and a supervised or semi-supervised machine learning (ML) environment may allow a user to update the labels or classifiers of the training set.

[0021] Cognitive analytics, such as natural language processing (NLP) and semantic analysis, can be used during model building and training. Cognitive analytics can analyze both structured and unstructured data to be integrated into the machine learning (ML) process. Natural language processing (NLP) can be used to analyze the quality of data or feedback received for training and model building. Structured data can include highly organized data, such as spreadsheets, relational databases, or data stored in fixed fields. Unstructured data can include data that is unorganized and has a non-traditional internal structure, such as portable document formats (PDFs), images, presentations, web pages, video content, audio content, emails, word processing documents, or multimedia content.

[0022] Received data may be processed using natural language processing (NLP) to extract meaningful information for training and building models. Semantic analysis may be used to infer the complexity, meaning, and intent of the received information or data to identify service dependencies within a microservice network. Semantic analysis may also consider the syntactic structure of the information at various levels to infer the meaning of words, phrases, sentences, and paragraphs of the received, collected, or extracted data.

[0023] According to one embodiment, part of the framework for a shift-left topology program may include collecting data, extracting entities, retrieving meta-information, correlating information, and building a topology. The collected data or information may be used for analysis and training models to improve microservices and microservice management. Data may be added to, stored in, or retrieved from one or more databases, repositories, or corpora. For example, accessible data repositories may include databases that hold configuration data, test case data, open-source data, DevOps pipeline logs, and other static and dynamic information associated with microservices. Feedback, such as subject matter expert (SME) feedback provided to models during training or after they have been fully trained, may also be stored in one or more databases.

[0024] Some examples of collected data may include test documentation, automation data, deployment scripts, configuration parameters, data from Swagger, Travis logs, or Jenkins logs. Data may be obtained from a variety of sources. Another example of collected data may be data received based on microservice deployment, such as how frequently or for how long users are using the microservice. Some data may be received from devices such as Internet of Things (IoT) devices, Global Positioning Systems (GPS), sensors, smart watches, smartphones, smart tablets, personal computers, or automotive devices. The collected data or information may be used to create datasets for model training and model usage. Some of the collected data may be considered static data, dynamic data, or real-time data, and this information may be widely available to microservice technologies and frameworks.

[0025] Microservice deployments, microservices, or containers may have a life expectancy that decreases over time. Because many microservices, such as social media microservices, have a life expectancy (i.e., the amount of time a user accesses the service) of less than five minutes, topology information may be abstracted at a higher level to ensure continuous updates to the topology. Connectivity or dependencies may be constructed and updated between microservices using the abstracted information, and analysis may be provided without actual deployment of the microservices. The semantic context of services and dependencies may be identified using the abstracted information and used to extend and update the constructed topology graph.

[0026] Example use cases may include microservice management systems and analytics components. For example, topology information may be used to train analytics models for alert groups or localization. Topology information may also be used for validation and configuration information and deployment planning. Deployment specifications may be generated, such as SIG applications or kube config. Fine-grained topology information (data), such as application program interface (API) upstream services, is essential for microservice operation. For example, a microservice used by a consumer relies on or depends on an upstream service.

[0027] Referring to FIG. 1 , an exemplary networked computing environment 100 according to one embodiment is shown. The networked computing environment 100 may include a computer 102 having a processor 104 and a data storage device 106 enabled to execute a software program 108, and a shift-left topology program 110a. The networked computing environment 100 may also include a server 112 enabled to execute a shift-left topology program 110b, which may interact with a database 114 and a communications network 116. The computer 102 may also be known as a client computer, and the server 112 may also be known as a server computer. The networked computing environment 100 may include multiple computers 102 and servers 112, only one of which is shown. The communications network 116 may include various types of communications networks, such as a wide area network (WAN), a local area network (LAN), a telecommunications network, a wireless network, a public switched network, and / or a satellite network. It should be appreciated that Figure 1 is intended to illustrate only one implementation and is not intended to suggest any limitations with regard to the environments in which different embodiments may be implemented. Many modifications to the depicted environments may be made based on design and implementation requirements.

[0028] The computer 102 may communicate with the server 112 via a communications network 116. The communications network 116 may include connections such as wired, wireless communication links, or fiber optic cables. As described with reference to FIG. 4 , the server 112 may include internal components 902a and external components 904a, respectively, and the computer 102 may include internal components 902b and external components 904b, respectively. The server 112 may also operate within a cloud computing service model, such as Software as a Service (SaaS), Analytics as a Service (AaaS), Blockchain as a Service (BaaS), Platform as a Service (PaaS), or Infrastructure as a Service (IaaS). The server 112 may also be located within a cloud computing deployment model, such as a private cloud, a community cloud, a public cloud, or a hybrid cloud. The computer 102 may be, for example, a mobile device, a phone, a personal digital assistant, a netbook, a laptop computer, a tablet computer, a desktop computer, or any type of computing device capable of running programs, accessing a network, and accessing a database 114. According to various implementations of the present embodiment, the shift-left topology programs 110a, 110b may interact with the database 114, which may be embedded in various storage devices such as, but not limited to, the computer / mobile device 102, the server 112, or a cloud storage service.

[0029] According to this embodiment, a user operating computer 102 or server 112 may use shift-left topology programs 110a, 110b (respectively) to construct topologies using shift-left analysis and machine learning. The shift-left topology method is described in more detail below with respect to Figures 2 and 3.

[0030] Referring now to Figure 2, an example block diagram of a constructed microservice topology 200 used by the shift-left topology programs 110a, 110b is shown in accordance with at least one embodiment. Topology entities are shown correlated to application program interface (API) routes. The microservice topology 200 shows an example result of mapping topology entities to source code. The microservice topology 200 may represent a call graph that identifies relationships between services, application program interface (API) connections, and resource connections.

[0031] Service API 202 provides an interface that allows services to interact, transmit, or receive data. Services 1 through 5 (204, 206, 208, 210, and 212, respectively) may represent services that are application program interface (API) endpoints. The application program interface (API) endpoints for each microservice may include / auth, / login, / home, / search, / recommend, or / member. For example, Service 1 204 is / auth, Service 2 206 is / get_info, Service 3 208 is / recommend, Service 4 210 is / graph, and Service 5 212 is / nodes. The entity abstractions and application program interface (API) endpoints may form a call control graph or call graph. A call graph may include a graph that represents the call flow relationships between computer programs.

[0032] Edge AB is a directional connection from A to B and refers to a message or transmission being sent from A to B. Blocking or non-blocking can refer to input / output (IO) operations during the process of executing an operation within a microservices framework. For example, a blocking operation has a longer processing period because it waits for a response to return before moving on to the next operation. Therefore, blocking operations can have a significant impact on performance, such as latency. Non-blocking operations can be an efficient way to operate on network and storage to avoid the additional wait time required in blocking operations. For example, during the time a blocking operation is waiting for a response to proceed, other processes can run and execute until the response returns to the blocking operation.

[0033] Databases 1-2 (214 and 216, respectively) may include specialized network storage databases that block one or more processes until a result or response is returned. File system 218 is an additional blocking device or system that holds one or more processes until a result or response is returned from the file system. Databases 1-2 and file system 218 may be examples of blocking systems used by shift-left topology programs 110a, 110b.

[0034] Referring now to FIG. 3, an operational flowchart illustrating an exemplary topology construction and information expansion process 300 used by the shift-left topology programs 110a, 110b in accordance with at least one embodiment is shown.

[0035] At 302, a dataset is collected. Static and real-time data may be collected from and stored in one or more repositories (e.g., databases 1-2). The collected data may be static or dynamic information associated with the microservice, such as configuration data, test case data, open source data, or DevOps pipeline logs. Other examples of collected data may include test documentation, automation data, deployment scripts, configuration parameters, data from Swagger, Travis logs, or Jenkins logs, and data may be obtained from various source databases. Additional data may be collected based on microservice deployment information. The collected data may be used as a dataset for multiple purposes, such as data extraction, model building or training, validation, topology graph construction, and refinement of built models using ground truth to update the dataset.

[0036] At 304, topology entities are extracted from the dataset. Entity extraction may include extracting topology entities, such as permission entities, login entities, front-end entities, ElasticSearch entities, graph database entities, recommendation entities, or member entities, associated with the microservice network from the dataset. The associated topology entities may be accessed to extract permission information, login information, front-end information, search information, recommendation information, or member information. The entity extraction process obtains endpoints for each of the associated topology entity microservices, for example, using keyword search, language modeling, or simple field extraction. See FIG. 2 for an example of endpoints.

[0037] For each application, service, or microservice, a top list of representative keywords is identified. The top list of representative keywords is based on an analysis of the distribution of words in the collected data for each topic. For example, for each microservice, topological keywords are identified from the collected dataset of documentation and source code configurations. Based on the identified keywords and the microservice, an unsupervised model can be built and used to identify important or relevant topics or names of the microservice. For example, topic modeling, clustering, or a priori models can be used to identify topics for the service.

[0038] For example, the following expressions can be used to extract keywords and identify related topics:

number

number

number

[0039] At 306, data is correlated from the extracted topology entities. Data may be correlated to match the same topology entities from different sources. Correlations of topology entities with application program interface (API) routes may be identified. Data may be correlated to each microservice application through data obtained for each application, from each application program interface (API), or both.

[0040] At 308, topology entities are mapped. The correlated data is mapped as a graph. For example, topology entities may be mapped to source code, and relationships may be identified from the mapped topology entities, such as caller-callee and application program interface (API) routes. Topology entities that have not been called at any location may be removed. As an example, one application program interface (API) call may have multiple service calls, in sequence or in parallel, and may be identified as blocking or non-blocking. See FIG. 2 for an example of the correlation of mapped topology entities, service calls, and application program interface (API) routes.

[0041] At 310, central processing unit (CPU) cycles of the subtree are marked for each entry point. Meta information can be marked for each application program interface (API) entry point. Meta information or metadata can be obtained from the call control graph and the call control graph program flow. Meta information can be obtained for each application program interface (API). The meta information obtained from each application program interface (API) can include, for example, central processing unit (CPU) cycles, processing time, estimated maximum calls per second, blocking code for network or storage access, threads, or parallel processing. Central processing unit (CPU) cycles can be marked for each entry point. The entry point can be determined by each application program interface (API). Processing time can be calculated for each subtree. The subtree can include traces of each application program interface (API) and a portion of the entire program.

[0042] A learning model can be used to obtain meta-information from a subtree or subgraph. A subgraph for a node can have data represented as labels, such as central processing unit (CPU) cycles, memory, network read / write, or disk read / write. A graph convolution operation can be used to average a node's features with the features of its neighbors. Unlike image or text data, a node's neighborhood can be unordered and variable in size. For example, subgraphs node A and node X can be processed through one or more graph convolutional networks (GCNs), followed by a pooling function and then a multilayer perceptron (MLP) network. The output can provide a softmax layer that represents meta-information of one or more node features.

[0043] At 312, a topology graph is constructed. The entire topology graph may be constructed using the correlation phase, the mapping phase, and the learning model. Additional information, such as performance metadata, may be marked for nodes in the entire topology graph, such as the estimated latency of each link (i.e., edge), endpoint routes, resource consumption, or estimated processing time. The represented nodes may be service endpoints, and the edges may be dependencies between endpoints. Additional information may be marked on edges or links and may include information such as latency, application program interface (API) connections, or round-trip time (RTT). The entire topology graph may be constructed by connecting subgraphs. Additional information, such as the estimated latency of each edge in a program, may be marked on subtrees or application program interfaces (APIs). See FIG. 2 for an example of an entire topology graph of nodes, edges, routes, and endpoints.

[0044] At 314, the ongoing topology information is validated. The entire constructed topology may be stored, and the information may be continuously monitored as new code is committed or configuration changes are made. The constructed topology may be validated with additional information, changes to the information, and updates to the information. Updates or changes to the information may be reflected in the topology graph. For example, the entire topology service graph may be stored and maintained in a database, and changes or modifications may be made to the topology graph over time. Changes or extensions to the stored topology graph may be made using, for example, updated data, static logs, configuration data, source code, or pipeline data.

[0045] At 316, additional or alternative embodiments enable the insertion of an active learning feedback model or component. Continuous validation of the constructed topology graph with updated static data, real-time information, subject matter expert (SME) feedback, or updated temporary topology data can be used as an active learning component. The active learning component can leverage supervised and semi-supervised models to introduce real-time information or expert feedback that is added as ground truth. Active learning can be used, for example, by obtaining subject matter expert (SME) or software developer feedback as ground truth during model training, testing, and validation phases. User knowledge or expert knowledge can be captured as real-time data.

[0046] 2 and 3 provide only an illustration of one embodiment and do not imply any limitations on how different embodiments may be implemented. Many modifications to the illustrated embodiment may be made based on design and implementation requirements.

[0047] Figure 4 is a block diagram 900 of the internal and external components of the computer shown in Figure 1, in accordance with an exemplary embodiment of the present invention. It should be understood that Figure 4 is intended to illustrate only one implementation and is not intended to suggest any limitation with regard to the environments in which different embodiments may be implemented. Many modifications to the depicted environment may be made based on design and implementation requirements.

[0048] Data processing systems 902, 904 represent any electronic device capable of executing machine-readable program instructions. Data processing systems 902, 904 may represent smartphones, computer systems, PDAs, or other electronic devices. Examples of computing systems, environments, and / or configurations that may be represented by data processing systems 902, 904 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, network PCs, minicomputer systems, and distributed cloud computing environments that include any of the above systems or devices.

[0049] The computer 102 and the server 112 may include respective sets of internal components 902a, b and external components 904a, b shown in Figure 4. Each of the sets of internal components 902a, b includes one or more processors 906 on one or more buses 912, one or more computer-readable RAMs 908 and one or more computer-readable ROMs 910, and one or more operating systems 914 and one or more computer-readable tangible storage devices 916. The one or more operating systems 914, software programs 108, and the shift-left topology program 110a in the client computer 102 and the shift-left topology program 110b in the network server 112 may be stored in one or more computer-readable tangible storage devices 916 for execution by the one or more processors 906 via one or more RAMs 908 (which typically include cache memory). In the embodiment shown in Figure 4, each of the computer-readable tangible storage devices 916 is an internal hard drive magnetic disk storage device. Alternatively, each of the computer-readable tangible storage devices 916 is a semiconductor storage device, such as a ROM 910, an EPROM, a flash memory, or any other computer-readable tangible storage device capable of storing computer programs and digital information.

[0050] Each set of internal components 902a, b also includes a R / W drive or interface 918 for reading from or writing to one or more portable computer-readable tangible storage devices 920, such as a CD-ROM, DVD, memory stick, magnetic tape, magnetic disk, optical disk, or semiconductor storage device. Software programs, such as software program 108 and shift-left topology programs 110a, 110b, may be stored on one or more of the respective portable computer-readable tangible storage devices 920, read via the respective R / W drive or interface 918, and loaded onto the respective hard drive 916.

[0051] Each set of internal components 902a,b may also include a network adapter (or switch port card) or interface 922, such as a TCP / IP adapter card, a wireless Wi-Fi interface card, a 3G / 4G / 5G wireless interface card, or other wired or wireless communication link. The software program 108 and the shift-left topology program 110a in the computer 102 and the shift-left topology program 110b in the network server 112 may be downloaded from an external computer (e.g., a server) and their respective network adapters or interfaces 922 via a network (e.g., the Internet, a local area network, or other wide area network). From the network adapters (or switch port adapters) or interfaces 922, the software program 108 and the shift-left topology program 110a in the computer 102 and the shift-left topology program 110b in the network server computer 112 are loaded onto their respective hard drives 916. The network may include copper wire, optical fiber, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers.

[0052] Each of the multiple sets of external components 904 a,b may include a computer display monitor 924, a keyboard 926, and a computer mouse 928. The external components 904 a,b may also include touch screens, virtual keyboards, touchpads, pointing devices, and other human interface devices. Each of the set of internal components 902 a,b also includes a device driver 930 for interfacing to the computer display monitor 924, the keyboard 926, and the computer mouse 928. The device driver 930, the R / W drive or interface 918, and the network adapter or interface 922 include hardware and software (stored in the storage device 916 and / or ROM 910).

[0053] Although this disclosure includes detailed descriptions of cloud computing, it is understood in advance that implementation of the teachings described herein is not limited to cloud computing environments. Rather, embodiments of the present invention can be implemented in conjunction with any other type of computing environment now known or later developed.

[0054] Cloud computing is a model of service delivery for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort or interaction with the provider of the service. The cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0055] The characteristics are as follows: On-Demand Self-Service: Cloud consumers can unilaterally provision computing capacity, such as server time and network storage, automatically as needed, without requiring human interaction with the provider of the service. Wide network access: The capabilities are available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (eg, cell phones, laptops, and PDAs). Resource Pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically allocated and reallocated according to demand. Consumers generally have no control or knowledge over the exact location of the provided resources, but there is a sense of location independence in that they may be able to specify location at a higher level of abstraction (e.g., country, state, or data center). Rapid Elasticity: This capacity can be rapidly and elastically provisioned, sometimes automatically, to quickly scale out, and rapidly released to quickly scale in. To the consumer, the capacity available for provisioning often appears unlimited, and can be purchased in any quantity at any point in time. Metered Services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage is monitored, controlled, and reported, providing transparency to both providers and consumers of the services utilized.

[0056] The service models are as follows: Software as a Service (SaaS): The functionality offered to the consumer is the use of the provider's applications running on a cloud infrastructure or hybrid cloud infrastructure. These applications are accessible from a variety of client devices through thin-client interfaces such as web browsers (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings. Platform as a Service (PaaS): The ability offered to consumers is to deploy consumer-created or acquired applications on a cloud infrastructure, written using programming languages ​​and tools supported by the provider. The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, or storage, but does control the deployed applications and, in some cases, the application hosting environment configuration. Analytics as a Service (AaaS): The ability offered to consumers is to access an analytics platform using a web-based or cloud-based network (i.e., infrastructure). The analytics platform may include access to analytics software resources or may include access to associated databases, corpora, servers, operating systems, or storage. The consumer does not manage or control the underlying web-based or cloud-based infrastructure, including the databases, corpora, servers, operating systems, or storage, but does control the deployed applications and, in some cases, the application hosting environment configuration. Infrastructure as a Service (IaaS): The ability offered to consumers is to provision processing, storage, network, and other basic computing resources on which they can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but does control the operating system, storage, deployed applications, and in some cases, limited control over selected networking components (e.g., host firewalls).

[0057] The deployment model is as follows: Private Cloud: This cloud infrastructure operates solely for an organization. It may be managed by the organization or a third party and may exist on-premise or off-premise. Community Cloud: This cloud infrastructure is shared by several organizations and supports a specific community with shared interests (e.g., mission, security requirements, policies and compliance considerations). The community cloud may be managed by these organizations or a third party and may exist on-premises or off-premises. Public cloud: The cloud infrastructure is made available to the general public or large industry organizations and is owned by an organization that sells cloud services. Hybrid cloud: The cloud infrastructure is a composite of two or more clouds (private, community, or public) that remain unique entities but are tied together by standardized or proprietary technologies that allow for data and application portability (e.g., cloud bursting for load balancing between clouds).

[0058] A cloud computing environment is a service oriented environment with an emphasis on statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure that comprises a network of interconnected nodes.

[0059] Referring now to FIG. 5, an exemplary cloud computing environment 1000 is shown. As shown, the cloud computing environment 1000 includes one or more cloud computing nodes 1050 with which local computing devices used by cloud consumers, such as, for example, a personal digital assistant (PDA) or cellular phone 1000A, a desktop computer 1000B, a laptop computer 1000C, and / or an automobile computer system 1000N, may communicate. The cloud computing nodes 1050 may communicate with each other. They may be physically or virtually grouped within one or more networks, such as the private, community, public, or hybrid clouds described above, or combinations thereof (not shown). This enables the cloud computing environment 1000 to provide infrastructure, platforms, and / or software as a service, eliminating the need for cloud consumers to maintain resources on their local computing devices. It is understood that the types of computing devices 1000A-N shown in FIG. 5 are intended as examples only, and that computing node 1050 and cloud computing environment 1000 may communicate with any type of computerized device via any type of network and / or network-addressable connection (e.g., using a web browser).

[0060] Referring now to Figure 6, there is shown a set of functional abstraction layers 1100 provided by cloud computing environment 1000. It should be understood in advance that the components, layers, and functions shown in Figure 6 are intended to be illustrative only, and that embodiments of the invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0061] Hardware and software layer 1102 includes hardware and software components. Examples of hardware components include mainframe 1104, reduced instruction set computer (RISC) architecture-based server 1106, server 1108, blade server 1110, storage device 1112, and network and networking components 1114. In some embodiments, software components include network application server software 1116 and database software 1118.

[0062] The virtualization layer 1120 provides an abstraction layer over which the following examples of virtual entities can be provided: virtual servers 1122, virtual storage 1124, virtual networks including virtual private networks 1126, virtual applications and operating systems 1128, and virtual clients 1130.

[0063] In one example, management layer 1132 may provide the functions described below. Resource provisioning 1134 provides dynamic procurement of computing and other resources utilized to execute tasks within the cloud computing environment. Metering and pricing 1136 provides cost tracking as resources are utilized within the cloud computing environment and charging or billing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides identity verification for cloud consumers and tasks, as well as protection for data and other resources. User portal 1138 provides consumers and system administrators with access to the cloud computing environment. Service level management 1140 provides cloud computing resource allocation and management so that required service levels are met. Service level agreement (SLA) planning and fulfillment 1142 provides pre-configuration and procurement of cloud computing resources where future requirements are forecasted according to SLAs.

[0064] The workload layer 1144 provides examples of functions for which a cloud computing environment can be utilized. Examples of workloads and functions that can be provided from this layer include mapping and navigation 1146, software development and lifecycle management 1148, virtual classroom instruction delivery 1150, data analytics processing 1152, transaction processing 1154, and shift-left topology construction 1156. The shift-left topology programs 110a, 110b provide a way to use machine learning to learn the behavioral characteristics of microservices and construct a topology representation of the services and service characteristics.

[0065] The present invention may be a system, method and / or computer program product at any possible level of technical detail integration. The computer program product may include a computer-readable storage medium (or multiple computer-readable storage media) having computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0066] A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. The computer-readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, punch cards, or mechanically encoded devices such as ridge structures in grooves that record instructions, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted through wires.

[0067] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing / processing device.

[0068] The computer-readable program instructions for carrying out the operations of the present invention may be either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages, including, for example, object-oriented programming languages ​​such as Smalltalk® or C++, procedural programming languages ​​such as the “C” programming language, the Python programming language, or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (e.g., through the Internet using an Internet Service Provider). In some embodiments, to carry out aspects of the present invention, an electronic circuit including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to personalize the electronic circuit.

[0069] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0070] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to create a machine, such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions may also be stored on a computer-readable storage medium that can instruct a computer, programmable data processing apparatus, and / or other device to function in a particular manner, such that the computer-readable storage medium having stored instructions comprises an article of manufacture containing instructions that implement aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0071] The computer-readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be executed on the computer, other programmable apparatus, or other device to generate a computer-implemented process, whereby the instructions executing on the computer, other programmable apparatus, or other device implement the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0072] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions that implement a specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or in some cases, the blocks may be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a dedicated hardware-based system that performs the specified functions or operations or executes a combination of dedicated hardware and computer instructions.

[0073] The description of various embodiments of the present invention has been presented for purposes of illustration, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terms used herein are chosen to best explain the principles of the embodiments, practical applications or technical improvements found in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A computer-implemented method for shift-left topology construction, comprising: collecting datasets, including using machine learning to collect static, dynamic, and real-time data associated with applications and services, including microservices; extracting a plurality of topology entities from the dataset related to a microservices network using an unsupervised machine learning (ML) model; correlating a plurality of data from the plurality of topology entities, the correlating comprising identifying correlations of the plurality of topology entities with application program interface (API) routes; mapping the plurality of topology entities, the mapping step including mapping the plurality of topology entities to source code and identifying relationships from the mapped plurality of topology entities; marking entry points of a plurality of subgraphs of the plurality of topology entities; and constructing a topology graph using machine learning, wherein nodes represent service endpoints and edges represent dependencies between the service endpoints. A method comprising:

2. updating the data set with static data; and continuously validating the updated data set. The method of claim 1 further comprising:

3. updating the dataset using real-time data from subject matter experts (SMEs) as an active learning feedback model; and continuously validating the updated data set. The method of claim 1 or 2, further comprising:

4. The method of claim 1 , wherein extracting the plurality of topology entities comprises extracting authorization entities, login entities, front-end entities, elasticsearch entities, and graph database entities from the dataset.

5. 5. The method of claim 1, wherein extracting the plurality of topological entities comprises using the unsupervised machine learning (ML) model to identify keywords and topics distributed among the collected dataset.

6. 6. The method of claim 1, wherein correlating the plurality of data from the plurality of topology entities comprises matching the plurality of topology entities from a plurality of different sources, wherein the application program interface (API) routes are identified between the plurality of topology entities.

7. The method of claim 1 , wherein marking the entry points of the plurality of subgraphs comprises marking central processing unit (CPU) cycles of each subgraph at the entry point.

8. The method of claim 1 , wherein a learning model is used to obtain meta-information from the plurality of sub-graphs.

9. 9. The method of claim 2, wherein the step of continually validating the updated data set occurs when a configuration change occurs.

10. A computer system for constructing a topology with a shift-left approach, comprising: a computer system comprising one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage media, and program instructions stored on at least one of the one or more computer-readable tangible storage media for execution by at least one of the one or more processors via at least one of the one or more computer-readable memories; collecting datasets, including using machine learning to collect static, dynamic, and real-time data associated with applications and services, including microservices; extracting a plurality of topology entities from the dataset related to a microservices network using an unsupervised machine learning (ML) model; correlating data from the plurality of topology entities, the correlating step including identifying correlations of the plurality of topology entities with application program interface (API) routes; a step of mapping the plurality of topology entities, the step including: mapping the plurality of topology entities to source code; and identifying relationships from the mapped plurality of topology entities; marking entry points of a plurality of subgraphs of the plurality of topology entities; and A method for constructing a topology graph using machine learning, wherein nodes represent service endpoints and edges represent dependencies between the service endpoints. A computer system capable of performing a method including:

11. The method comprises: updating the data set with static data; and and continuously validating said updated data set. The computer system of claim 10 further comprising:

12. The method comprises: updating the dataset using real-time data from subject matter experts (SMEs) as an active learning feedback model; and and continuously validating said updated data set. further comprising:

12. A computer system according to claim 10 or 11.

13. 13. The computer system of claim 10, wherein extracting the plurality of topology entities comprises extracting authorization entities, login entities, front-end entities, elasticsearch entities, and graph database entities from the dataset.

14. 14. The computer system of claim 10, wherein extracting the plurality of topological entities comprises using the unsupervised machine learning (ML) model to identify keywords and topics distributed among the collected dataset.

15. 15. The computer system of claim 10, wherein correlating the plurality of data from the plurality of topology entities comprises matching the plurality of topology entities from a plurality of different sources, wherein the application program interface (API) routes are identified between the plurality of topology entities.

16. 16. The computer system of claim 10, wherein marking the entry points of the plurality of subgraphs comprises marking a central processing unit (CPU) cycle of each of the subgraphs at the entry point.

17. The computer system of claim 10 , wherein a learning model is used to obtain meta-information from the plurality of subgraphs.

18. 18. The computer system of claim 11, wherein the step of continually validating the updated data set occurs when a configuration change occurs.

19. A computer program for shift-left topology construction, comprising: The processor collecting datasets, including using machine learning to collect static, dynamic, and real-time data associated with applications and services, including microservices; extracting a plurality of topology entities from the dataset related to a microservices network using an unsupervised machine learning (ML) model; correlating data from the plurality of topology entities, the correlating step including identifying correlations of the plurality of topology entities with application program interface (API) routes; a step of mapping the plurality of topology entities, the step including: mapping the plurality of topology entities to source code; and identifying relationships from the mapped plurality of topology entities; marking entry points of a plurality of subgraphs of the plurality of topology entities; and A method for constructing a topology graph using machine learning, wherein nodes represent service endpoints and edges represent dependencies between the service endpoints. A computer program for executing

20. the processor, updating the data set with static data; and and continuously validating said updated data set. Further execute 20. A computer program according to claim 19.

21. the processor, updating the dataset using real-time data from subject matter experts (SMEs) as an active learning feedback model; and and continuously validating said updated data set. Further execute 21. A computer program according to any one of claims 19 to 20.

22. 22. The computer program product of claim 19, wherein extracting the plurality of topology entities comprises extracting authorization entities, login entities, front-end entities, elasticsearch entities, and graph database entities from the dataset.

23. 23. The computer program product of claim 19, wherein extracting the plurality of topological entities comprises using the unsupervised machine learning (ML) model to identify keywords and topics distributed among the collected dataset.

24. 24. The computer program product of claim 19, wherein correlating the plurality of data from the plurality of topology entities comprises matching the plurality of topology entities from a plurality of different sources, wherein an application program interface (API) route is identified between the plurality of topology entities.

25. 25. The computer program of claim 19, wherein a learning model is used to obtain meta-information from the plurality of sub-graphs.

Images