Method and system for marketplace test automation

Abstract

Systems, methods, and devices for managing tests in a marketplace are provided. The system includes a first module, the first module having a memory storage storing computer-executable instructions and a processor communicatively coupled to the memory storage, the processor configured to execute the computer-executable instructions to cause the first module to present information of one or more tests associated with a service to a user, the test information including at least one mandatory test and at least one optional test, receive selections for the one or more tests from the user, collect test information associated with the user's selections, and output the collected test information.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of Provisional Patent Application No. 202241049596, entitled "METHOD AND SYSTEM FOR AUTOMATION AND ROLE-BASED CONTROL IN MARKETPLACE," filed with the Indian Patent Office on August 30, 2022, the disclosure of which is incorporated herein by reference in its entirety.

[0002] Methods and systems consistent with exemplary embodiments of the present disclosure relate to marketplaces, and more particularly to the automation of one or more tests in a marketplace. [Background technology]

[0003] Traditionally, telecommunication systems have been primarily hardware-based, and telecommunication services have been primarily provided in the form of hardware, which was typically proprietary or managed by a particular vendor, service provider, and / or network operator. In recent years, advances in telecommunication technology have made it possible to provide telecommunication services in the form of software. For example, telecommunication network services can be defined and provided in software-based forms or virtual network services, such as Virtualized Network Functions (VNFs) and Software Defined Networking (SDN), among others.

[0004] Accordingly, telecommunication services may be deployed and offered through a software-based marketplace system together with other non-telecommunication services (e.g., enterprise software, business support applications, etc.). Offering services through a marketplace offers various advantages, such as enabling telecommunication services to be integrated with and operate alongside non-telecommunication services, thereby providing unique user experiences and enhanced functionality, but the end-to-end process of services in the marketplace needs to be further improved.

[0005] Specifically, handling the end-to-end process, including but not limited to publishing, deploying, and operating a product or service, is one of the most complex and cumbersome processes for marketplace participants. In fact, the prior art does not provide an end-to-end solution for integrating and managing the end-to-end process of a product or service in a marketplace.

[0006] Furthermore, the separation of service provisioning and service management in a marketplace (i.e., service provisioning and management are no longer proprietary or managed by a limited number of users) significantly increases the number of users (e.g., network operators, vendors, service providers, customers, etc.), each of which may use the marketplace for different purposes and have different levels of requirements for services. Accordingly, prior art marketplaces can be highly fragmented, with different users (e.g., network operators, marketplace managers, etc.) requiring different information levels and formats from other users (e.g., vendors, service providers, developers, etc.) to onboard and manage services. Meeting the information and service requirements is a difficult challenge for all users.

[0007] In particular, the testing process for end-to-end processes can be particularly difficult to manage and provide in an efficient and effective manner. Specifically, various tests may be required throughout the end-to-end process, and the testing process in the prior art marketplace is time-consuming, inefficient, and burdensome for users.

[0008] First, the prior art does not provide any functionality that allows users to quickly and effectively set up or configure tests. Specifically, whenever testing of a service is needed, a vendor, developer, service provider, or any other appropriate person associated with the service (hereinafter referred to as a "test requester") may need to manually contact a person managing or executing the test (hereinafter referred to as a "test executor") to discuss the test. For example, whenever a test requester wants to onboard a service to a marketplace, the service may need to be tested by a test executor, such as a system administrator (e.g., a network operator, a marketplace manager, etc.), to ensure that the service complies with minimum system requirements (e.g., functionality, security, etc.). If the test requester subsequently wants to integrate the service with another service in the marketplace, the test requester may need to contact another test executor, such as the owner or manager of the other service, to arrange for integration testing.

[0009] To this end, in prior art marketplaces, whenever one or more tests are needed, the test requester may be required to contact each of the testers individually or separately, for example, by sending emails, making phone calls, creating tickets (via JIRA® or any other suitable ticket management system), etc. That is, multiple back-and-forth communications may be required before the tests can actually be performed. The process of setting up or configuring one or more tests can be time-consuming, inefficient, and burdensome for the test requester and / or tester, especially when the test requester and / or tester are inexperienced (e.g., the test requester may be unfamiliar with the testing process, the tester may not be clear on how to configure a particular test, etc.) and / or are simultaneously managing a significant number of services or tests.

[0010] Meanwhile, the related art also fails to provide any functionality for efficiently or effectively setting up or configuring test(s). Specifically, a tester may receive a significant number of requests (from one or more requesters), each with specific test requirements. The tester may then need to manually review each request, contact the requester for discussion as needed, and then set up a customized test for the requester. Thus, responding to test requests is burdensome for the tester, resulting in a long turnaround time for responding to test requests, which may delay testing and affect the deployment of the associated service(s).

[0011] Furthermore, the related art also does not provide effective and efficient test information management (especially test status management and test result management) to test requesters and test implementers.

[0012] Specifically, in related technology marketplaces, when a requester submits a test request, the requester may need to closely monitor the status of the requested tests. In this regard, if the requester submits multiple test requests to multiple testers (e.g., X requests submitted to a system administrator, Y requests submitted to a manager of service B, etc.), the requester may first need to create a new request for status updates for each of the test requests (which may further increase the burden on the requester and tester in managing the requests, such as managing multiple correspondence or tickets, etc.), and may then need to simultaneously note the status or record of the requests and / or tests in different channels (e.g., the system administrator may provide status updates via comments on the associated ticket, the manager of service B may provide status updates via multiple emails, etc.). Furthermore, testers may provide test status information in different ways (e.g., a system administrator may use system A to generate and provide test status information in format A, a manager of service B may use system B to generate and provide test information in format B, etc.), which may increase the difficulty for requesters to quickly understand test status. Correspondingly, it may be difficult for requesters to quickly obtain a comprehensive view of test status, and requesters may miss one or more status updates, which may result in inefficient communication with testers and cause delays in service deployment.

[0013] Similarly, related art marketplaces also do not provide an effective and efficient way to generate and manage test reports or logs. Specifically, whenever a requester wants to obtain test report(s) or log(s), the requester may need to create one or more new requests for them. Furthermore, the requester may need to manage test reports or logs provided by different personnel, which may be in different formats. Furthermore, because the test report(s) or log(s) are generated by the tester (i.e., the requester does not, by default, have control over the content of the test report or log), the provided test report(s) or log(s) may contain too much detailed information, some of which may not be of interest to the requester, or too little information, without including the information intended by the requester. As a result, requesters may further request that testers provide customized test report(s) or log(s), and testers may receive a significant amount of requests for that customization, which may further increase the burden on testers. Summary of the Invention

[0014] According to the exemplary embodiments, a system, method, and device are provided for efficiently and effectively setting up or configuring one or more tests for a user. Specifically, the exemplary embodiments enable a user to configure or customize one or more tests on the user's side according to requirements, and then automatically execute the one or more customized tests. This eliminates the multiple back-and-forth communications between multiple parties involved in configuring tests in the related art, simplifying the associated processes. Accordingly, the exemplary embodiments effectively reduce the burden on users (e.g., test requesters and test implementers), improve the process efficiency of test customization, and simplify the process of test execution.

[0015] Furthermore, embodiments provide systems, methods, and devices for efficiently and effectively managing test information, such as test status and / or test results. Specifically, the systems and methods of the exemplary embodiments provide test status and / or test results in real time or near real time without the involvement of a test administrator, and automatically generate and provide test status and / or test results to a user upon request, such as providing test status and / or test results in aggregate for multiple tests. Thus, test status and / or test results can be provided in a timely and effective manner.

[0016] According to an embodiment, a system may include a first module, the first module including a memory storage storing computer-executable instructions and a processor communicatively connected to the memory storage, the processor executing the computer-executable instructions and configured to cause the first module to present information of one or more tests associated with a service to a user, which may include at least one mandatory test and at least one optional test, receive selections for the one or more tests from the user, collect test information associated with the user's selections, and output the collected test information.

[0017] The processor of the first module may be configured to execute computer-executable instructions to cause the first module to generate a graphical user interface (GUI) and present information of the one or more tests via the GUI.

[0018] The GUI may include one or more interactive elements associated with one or more tests, and the processor of the first module may be configured to execute computer-executable instructions to cause the first module to receive a user selection by determining one or more user interactions with the one or more interactive elements.

[0019] The one or more interactive elements may be associated with a test profile ID, and the processor of the first module may be configured to execute computer-executable instructions to cause the first module to collect test information related to the user's selection by collecting the test profile IDs of the user's selected tests.

[0020] The system may further include a second module, the second module including a memory storage storing computer-executable instructions and a processor communicatively coupled to the memory storage, the processor configured to execute the computer-executable instructions to cause the second module to receive the collected test information from the first module, generate a test scheme based on the collected test information, and output the generated test scheme.

[0021] The system may further include a third module, the third module including a memory storage storing computer-executable instructions and a processor communicatively coupled to the memory storage, the processor configured to execute the computer-executable instructions to cause the third module to receive the generated test scheme from the second module, execute the one or more tests selected by the user based on the test scheme, and store information related to the tests.

[0022] The system may further include a fourth module, the fourth module including a memory storage storing computer-executable instructions and a processor communicatively connected to the memory storage, the processor configured to execute the computer-executable instructions to cause the fourth module to receive the information related to the test from the third module, store the received information, and provide the stored information to the first module.

[0023] According to an embodiment, a method that may be executed by at least one processor includes: presenting, by a first module, information of one or more tests associated with a service to a user, including at least one mandatory test and at least one optional test; receiving, by the first module, selections for the one or more tests from the user; collecting, by the first module, test information associated with the user's selections; and outputting, by the first module, the collected test information.

[0024] Presenting the information of the one or more tests may include generating, by the first module, a graphical user interface (GUI) and presenting, by the first module, the information of the one or more tests via the GUI.

[0025] The GUI may include one or more interactive elements associated with one or more tests, and receiving the user selection may include determining, by the first module, one or more user interactions with the one or more interactive elements.

[0026] The one or more interactive elements may be associated with a test profile ID, and collecting test information related to the user selection may include collecting, by the first module, a test profile ID for the test selected by the user.

[0027] The method may further include receiving, by a second module, the collected test information from the first module; generating, by the second module, a test scheme based on the collected test information; and outputting, by the second module, the generated test scheme.

[0028] The method may further include receiving, by a third module, the generated test scheme from the second module; executing, by the third module, one or more tests selected by the user based on the test scheme; and storing, by the third module, information related to the tests.

[0029] The method may further include receiving, by a fourth module, information related to the test from the third module, storing, by the fourth module, the received information, and providing, by the fourth module, the stored information to the first module.

[0030] According to an embodiment, a non-transitory computer-readable storage medium has stored thereon instructions executable by a processor that cause the processor to perform a method including: presenting, by a first module, information on one or more tests associated with a service to a user, including at least one mandatory test and at least one optional test; receiving, by the first module, selections for the one or more tests from the user; collecting, by the first module, test information associated with the user's selections; and outputting, by the first module, the collected test information.

[0031] Presenting the information of the one or more tests may include generating, by the first module, a graphical user interface (GUI) and presenting, by the first module, the information of the one or more tests via the GUI.

[0032] The GUI may include one or more interactive elements associated with one or more tests, and receiving the user selection may include determining, by the first module, one or more user interactions with the one or more interactive elements.

[0033] The one or more interactive elements may be associated with a test profile ID, and collecting test information related to the user selection may include collecting, by the first module, a test profile ID for the test selected by the user.

[0034] The method may further include receiving, by a second module, the collected test information from the first module; generating, by the second module, a test scheme based on the collected test information; and outputting, by the second module, the generated test scheme.

[0035] The method may further include receiving, by a third module, the generated test scheme from the second module; executing, by the third module, one or more tests selected by the user based on the test scheme; and storing, by the third module, information related to the tests.

[0036] The method may further include receiving, by a fourth module, information related to the test from the third module, storing, by the fourth module, the received information, and providing, by the fourth module, the stored information to the first module.

[0037] Additional aspects will be set forth in part in the description that follows, and in part will be apparent from the description, or may be learned by practice of presented embodiments of the present disclosure.

[0038] The features, advantages, and significance of exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which like reference numerals refer to like elements. [Brief explanation of the drawings]

[0039] [Figure 1] 1 shows a diagram of a general system architecture according to one or more embodiments.

[0040] [Figure 2] 1 illustrates a diagram of a centralized marketplace according to one or more embodiments.

[0041] [Figure 3] 1 illustrates an example of a system module for managing one or more tests according to one or more embodiments.

[0042] [Figure 4] FIG. 1 illustrates an example block diagram of system modules for managing one or more end-to-end processes, according to one or more embodiments.

[0043] [Figure 5] 1 illustrates an example block diagram of a system for managing end-to-end testing according to one or more embodiments.

[0044] [Figure 6] 1 illustrates a generalized GUI of a test workspace according to one or more embodiments.

[0045] [Figure 7] 1 illustrates an example GUI of a test workspace according to one or more embodiments.

[0046] [Figure 8] 1 illustrates another example of a generalized GUI of a test workspace according to one or more embodiments.

[0047] [Figure 9]10 illustrates yet another example of a generalized GUI of a test workspace according to one or more embodiments.

[0048] [Figure 10] FIG. 1 illustrates an example flow diagram of a method for interacting with a user to configure one or more tests, according to one or more embodiments.

[0049] [Figure 11] 1 illustrates an exemplary flow diagram of a method for generating a test scheme according to one or more embodiments.

[0050] [Figure 12] FIG. 1 illustrates an example flow diagram of a method for administering one or more tests according to one or more embodiments.

[0051] [Figure 13] FIG. 1 is a diagram of an example environment in which the systems and / or methods described herein may be implemented.

[0052] [Figure 14] FIG. 2 is a diagram of exemplary components of a device according to one or more embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0053] The following detailed description of exemplary embodiments refers to the accompanying drawings. The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit implementations to the precise form disclosed. Modifications and variations are possible in light of the above disclosure or may be acquired from practice of the implementations. Moreover, one or more features or components of one embodiment may be incorporated into or combined with another embodiment (or one or more features of another embodiment). Additionally, in the flowcharts and descriptions of operations provided below, it is understood that one or more operations may be omitted, one or more operations may be added, one or more operations may be performed (at least partially) concurrently, and the order of one or more operations may be permuted.

[0054] It will be apparent that the systems and / or methods described herein may be implemented in various forms of hardware, firmware, or combinations of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods is not intended to limit the implementation. Thus, the operation and behavior of the systems and / or methods are described herein without reference to specific software code, and it will be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0055] Although particular combinations of features are recited in the claims and / or disclosed herein, these combinations are not intended to limit the disclosure of possible implementations. Indeed, many of these features may be combined in ways not specifically recited in the claims and / or disclosed herein. Although each dependent claim listed below may depend directly on only one claim, the disclosure of possible implementations includes each dependent claim in combination with all other claims in the claim set.

[0056] No element, act, or instruction used herein should be construed as critical or required unless expressly stated to be so. Also, as used herein, the articles "a" and "an" are intended to include one or more items and may be used interchangeably with "one or more." Where only one item is intended, the term "one" or similar language is used. Also, as used herein, terms such as "has," "have," "having," "include," and "including" are intended to be open-ended terms. Furthermore, the phrase "based on" is intended to mean "based at least in part on," unless specifically stated otherwise. Furthermore, phrases such as "at least one of [A] and [B]" or "at least one of [A] or [B]" should be understood to include A only, B only, or both A and B.

[0057] Throughout this specification, references to "one embodiment," "an embodiment," "a non-limiting exemplary embodiment," or similar language mean that a particular feature, structure, or characteristic described in connection with the illustrated embodiment is included in at least one embodiment of the solution. Thus, throughout this specification, the phrases "in one embodiment," "in an embodiment," "in one non-limiting exemplary embodiment," and similar language may, but do not necessarily, all refer to the same embodiment.

[0058] Furthermore, the features, advantages, and characteristics described in the present disclosure may be combined in any suitable manner in one or more embodiments. Those skilled in the art will recognize, in light of the description herein, that the present disclosure can be practiced without one or more of the specific features or advantages of a particular embodiment. In other instances, additional features and advantages may be recognized in certain embodiments that may not be present in all embodiments of the present disclosure.

[0059] In one implementation of the disclosure described herein, a display page can include information residing in the memory of a computing device, and this information can be transmitted from the computing device to a database center or vice versa over a network. The information can be stored in memory at the computing device, in data storage at the edge of the network, or in a server at the database center. A computing device or a mobile device can accept non-transitory computer-readable media that can contain instructions, logic, data, or code that can be stored in persistent or temporary memory on the mobile device or that can affect or initiate actions by the mobile device. Similarly, one or more servers can communicate with one or more mobile devices over a network and can transmit computer files residing in memory. The network can include, for example, the Internet, a wireless communication network, or any other network for connecting one or more mobile devices to one or more servers.

[0060] Any description of a computing or mobile device may also apply to any type of networked device, including, but not limited to, mobile devices such as mobile phones (e.g., any "smartphone"), personal computers, server computers, or laptop computers, and telephones, personal digital assistants (PDAs), roaming devices such as network-attached roaming devices, wireless devices such as wireless email devices or other devices capable of wirelessly communicating with a computer network, or any other type of networked device that may communicate over a network and process electronic transactions. Any description of any mobile device mentioned may also apply to other devices, such as short-range ultra-high frequency (UHF) devices, near-field communication (NFC), infrared (IR), and devices including Wi-Fi capabilities, among others.

[0061] "Software," "application," "app," and "firmware" and similar words and terms may include any non-transitory computer-readable medium that stores a program that, when executed by a computer, causes the computer to perform a method, function, or control operation.

[0062] "Network" and similar phrases and terms may include one or more data links that enable the transport of electronic data between computer systems and / or modules. When information is transferred or provided to a computer over a network or another communications connection (either hardwired, wireless, or a combination of hardwired or wireless), the computer uses the connection as a computer-readable medium. Thus, by way of example and not limitation, a computer-readable medium may also include a network or data link that may be used to carry or store desired program code means in the form of computer-executable instructions or data structures and that may be accessed by a general-purpose or special-purpose computer.

[0063] Similar phrases and terms such as "portal" or "terminal" may include an intranet page, an Internet page, locally residing software or application, a mobile device graphical user interface, or a digital presentation for a user. A portal may also be any graphical user interface for accessing the various modules, components, features, options, and / or attributes of the present disclosure described herein. For example, a portal may be a web page accessed with a web browser, a mobile device application, or any application or software residing on a computing device.

[0064] "Marketplace," "marketplace platform," "marketplace system," or "marketplace module" and similar phrases and terms may refer to a system in which multiple users may offer, receive, and / or use one or more services.

[0065] "Tester" and similar phrases and terms may include any suitable person who can configure or manage testing of a service, such as a system administrator, which may include a user from a network operator, a user managing a marketplace, a service owner, etc. Meanwhile, "test requestor" and similar phrases and terms may include any suitable person who can request testing of a service, such as a vendor, developer, service provider, service owner, etc.

[0066] Phrases and terms like "products" or "services" may include products, services, and applications that can be virtualized / defined in the form of software and deployed via the marketplace. Additionally, while the exemplary embodiments may refer to telecommunications-related products and services, it is understood that the disclosure is not limited thereto and that the marketplace may be for telecommunications and / or non-telecommunications-related products. For example, the marketplace may offer applications and services generally for businesses (e.g., customer service-related applications, inventory management applications, financial forecasting or planning applications, accounting applications, etc.).

[0067] Exemplary embodiments of the present disclosure provide a system and method for efficiently and effectively setting up or configuring one or more tests for a user. Specifically, the system and method of the exemplary embodiments enable a user to configure or customize one or more tests on the user's side according to requirements, and can automatically execute the one or more customized tests. This eliminates the need for multiple back-and-forth communications between multiple parties involved in configuring tests in the related art, simplifying the associated processes. Accordingly, the exemplary embodiments of the present disclosure effectively reduce the burden on users (e.g., test requesters and test implementers), improve the process efficiency of test customization, and simplify the process of test execution.

[0068] Furthermore, exemplary embodiments of the present disclosure provide systems and methods for efficiently and effectively managing test information, such as test status and / or test results / logs. Specifically, the systems and methods of the exemplary embodiments provide test status and / or test results in real time or near real time without the involvement of a test administrator, and automatically generate and provide test status and / or test results to a user upon request, such as providing test status and / or test results in aggregate for multiple tests. Thus, test status and / or test results can be provided in a timely and effective manner.

[0069] FIG. 1 illustrates a diagram of an exemplary general network architecture, according to one or more embodiments. Referring to FIG. 1, users 110, 120, and 130 can bidirectionally communicate with a central server or application server 100 via a secure network, according to one or more embodiments. Additionally, users 110, 120, and 130 can also bidirectionally communicate directly with each other via a marketplace platform network system, according to one or more embodiments. Here, users 110 can be any type of vendor, service provider, or developer, such as Vendor A, Vendor B, or Vendor C, offering similar, identical, or different services or products to either user 120 or user 130 with respect to a network operator or service operator. In particular, users 110 can provide any type of telecommunications and / or non-telecommunications services or products, such as cellular service network deployment, network capacity updates, operational monitoring, data analysis and reporting, performance analysis, and any other suitable services. Each of users 110 can communicate with server 100 via a respective terminal or portal. The users 120 may be network operator A, network operator B, and service provider C, which may offer similar, identical, or different services or products to user 110 or user 130. Here, user 110 or 120 may be, among others, any type of vendor or network service provider, network operator, carrier, broadband provider, Unified Communications as a Service (UCaaS) provider, wired or wireless cellular network service provider, radio access network (RAN), web host, or any network-related or telecommunications-related service provider or network provider. Each of the users 120 may communicate with the server 100 through a respective terminal or portal.Users 130 may be any type of end user or customer of user 120 or user 110, such as end user A or customer B and customer C, or end users who purchase and / or receive services from user 120 or user 110. Each of users 130 may communicate with server 100 through a respective terminal or portal.

[0070] 1 , the central server 100 of the marketplace platform system, according to one or more embodiments, can further bi-directionally communicate with an administration terminal / dashboard 140. Here, the administration terminal / dashboard 140 can provide various tools to any of the users 110, 120, and 130 or a sales or content / product creation team to manage various customers / end users and customer leads, including, among other things, creating, editing, and promoting various types of service or product promotional campaigns, advertisements, offers, and ordering options for customers and other users of the marketplace platform, according to one or more embodiments. In addition, the administration terminal / dashboard 140 can also include various types of access privileges for various users of the marketplace platform system, according to one or more embodiments. Furthermore, the central server 100, according to one or more embodiments, can further bi-directionally communicate with a database / third-party server 150. Here, the server 150 may provide various types of data storage (e.g., cloud-based storage), web services, content creation tools, data streams, data feeds, and / or various types of third-party support services to the marketplace platform central server 100. However, within the scope of the present disclosure described herein, it is contemplated that the network services marketplace platform processes and systems according to one or more embodiments may include any type of general network architecture.

[0071] With further reference to FIG. 1, one or more of the servers or terminals of elements 100-150 may include a personal computer (PC), a printed circuit board with a computing device, a minicomputer, a mainframe computer, a microcomputer, a telephone computing device, a wired / wireless computing device (e.g., a smartphone, a personal digital assistant (PDA)), a laptop, a tablet, a smart device, a wearable device, or any other similarly functional device.

[0072] 1, one or more of the servers, terminals, and users 100-150 may include a set of components such as a processor, memory, storage components, input components, output components, communication interfaces, and JSON UI rendering components. The set of components of a device may be communicatively coupled via a bus.

[0073] A bus may comprise one or more components that enable communication between a set of components, such as one or more of the servers or terminals of elements 100-150. For example, a bus may be a communications bus, a crossover bar, a network, etc. A bus may be implemented using a single or multiple (two or more) connections between a set of components, such as one or more of the servers or terminals of elements 100-150. The present disclosure is not limited in this respect.

[0074] One or more of the servers or terminals of elements 100-150 may include one or more processors. The one or more processors may be implemented in hardware, firmware, and / or a combination of hardware and software. For example, the one or more processors may include a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a general-purpose single-chip or multi-chip processor, or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, or any conventional processor, controller, microcontroller, or state machine. The one or more processors may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. In some embodiments, particular processes and methods may be performed by circuitry that is specific to a given function.

[0075] One or more processors may control the overall operation of one or more of the servers or terminals of elements 100-150 and / or a set of components (e.g., memory, storage components, input components, output components, communication interfaces, rendering components) of one or more of the servers or terminals of elements 100-150.

[0076] One or more of the servers or terminals of elements 100-150 may further comprise memory. In some embodiments, the memory may comprise random access memory (RAM), read only memory (ROM), electrically erasable programmable ROM (EEPROM), flash memory, magnetic memory, optical memory, and / or another type of dynamic or static storage device. The memory may store information and / or instructions for use (e.g., execution) by the processor.

[0077] A storage component of one or more of the servers or terminals of elements 100-150 may store information and / or computer-readable instructions and / or code related to the operation and use of one or more of the servers or terminals of elements 100-150. For example, the storage component may include a hard disk (e.g., a magnetic disk, optical disk, magneto-optical disk, and / or solid-state disk), a compact disc (CD), a digital versatile disc (DVD), a universal serial bus (USB) flash drive, a Personal Computer Memory Card International Association (PCMCIA) card, a floppy disk, a cartridge, a magnetic tape, and / or another type of non-transitory computer-readable medium, along with a corresponding drive.

[0078] One or more of the servers or terminals of elements 100-150 may further comprise an input component. The input component may include one or more components that enable the server and one or more of terminals 110-140 to receive information, such as via user input (e.g., a touchscreen, keyboard, keypad, mouse, stylus, button, switch, microphone, camera, etc.). Alternatively, or in addition, the input component may include a sensor for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, an actuator, etc.).

[0079] The output components of one or more of the servers or terminals of elements 100-150 may include one or more components that can provide output information from device 100 (e.g., a display, a liquid crystal display (LCD), light-emitting diodes (LEDs), organic light-emitting diodes (OLEDs), a haptic feedback device, a speaker, etc.).

[0080] One or more of the servers or terminals of elements 100-150 may further comprise a communications interface. The communications interface may include a receiver component, a transmitter component, and / or a transceiver component. The communications interface may enable one or more of the servers or terminals of elements 100-150 to establish connections and / or transfer communications with other devices (e.g., a server, another device). Communications may be enabled via a wired connection, a wireless connection, or a combination of wired and wireless connections. The communications interface may enable one or more of the servers or terminals of elements 100-150 to receive information from and / or provide information to another device. In some embodiments, the communication interface may provide for communication with another device over a network such as a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a private network, an ad-hoc network, an intranet, the Internet, an optical fiber-based network, a cellular network (e.g., a fifth generation (5G) network, a long-term evolution (LTE) network, a third generation (3G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a telephone network (e.g., a public switched telephone network (PSTN)), etc., and / or combinations of these or other types of networks.Alternatively, or in addition, the communication interface may provide for communication with another device via a device-to-device (D2D) communication link, such as FlashLinQ, WiMedia, Bluetooth, ZigBee, Wi-Fi, LTE, 5G, etc. In other embodiments, the communication interface may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, etc.

[0081] FIG. 2 illustrates an exemplary system for a centralized marketplace platform, according to one or more embodiments. Referring to FIG. 2 , marketplace platform module or portal 200 may be a portal for the marketplace platform and may include a Customer Management System (CMS) delivery node module 202 in bidirectional communication with a Business Support System (BSS) product stack module or portal 210 (which may include any one or more of modules 212-220 shown in FIG. 2 ), as well as a CMS integration module or portal 206 for content management and integration with various repositories. A user / customer module or portal 224 (e.g., a user terminal or client device) may access the marketplace through communication with CMS delivery node module or portal 202. In some embodiments, user / customer module or portal 224 may also communicate with a Content Distribution Network (CDN) module or portal, which further communicates with modules or portals 202 and 206. In some embodiments, the CMS delivery node module or portal 202 can handle a first customer interaction when the customer accesses one or more embodiments of the marketplace platform 200. For example, the CMS delivery node module or portal 202 can receive a request from the customer's user terminal or portal 224 to browse available products / services, retrieve product / service data from a product catalog, and generate / present a graphical user interface (GUI) representation (e.g., columns, lists, etc.) showing available products / services of the marketplace platform or one or more service providers or vendors.Additionally, product / content team modules or portals 222 (e.g., terminals or client devices of sellers, service providers, vendors, etc.) may also communicate with the marketplace platform module or portal through communication with the CMS integration module or portal 206. In this case, the CMS integration module or portal 206 may generate and / or present GUIs related to the sellers, service providers, vendors, etc. (e.g., user interfaces for listing products for sale, configuring sales listings, creating or editing customer or product configuration profiles, facilitating sales activities, etc.).

[0082] 2 , the marketplace platform module or portal 200, according to one or more embodiments, may also include a BSS product stack module or portal 210. Here, module 210 may further include a product catalog module or portal 212, which may be a repository / data storage for storing product / service data. Additionally, module 212 may also include and store product / service-level promotional offers, or price drop indicators or events associated with one or more stored products / services. Module 210 may also include a customer information management (CIM) module or portal 214, which may be a repository / data storage for storing customer information (e.g., name, contact information, ID, customer interest list, customer-level promotional offers or price drop events, etc.). Module 210 may also include a customer interaction management module or portal 216, which may be a repository / data storage that stores information about customer interactions with the marketplace platform (e.g., customer interaction history with interactive elements of a graphical user interface (GUI), communications between the customer and service provider(s) or other customer(s), etc.), according to one or more embodiments.

[0083] 2 , module 210 may also include a billing / invoicing module or portal 218 that stores information associated with customer billing / invoicing for services offered on the marketplace platform (e.g., previously generated invoices, billing history, etc.). In addition, module 210 may also include a customer care / sales module or portal 220 that stores information on sales or customer-related activities (e.g., sales campaign history, promotions for specific types of customers, sales history analysis, etc.). In some embodiments, module 220 may identify data related to sales activities and user experiences, retrieve / collect data from external data storage or from any of modules or portals 212-218, store such data in a repository included in module 220, and provide such data to network service providers / vendors as needed. For example, module 220 may provide sales-related information or data to sales team module 228 so that the sales team utilizing sales team module 228 can use the information or data as needed. Alternatively, module 220 may also communicate with product / content team module or portal 222. Here, the product / content team module or portal 222 can be a user or portal for users (e.g., vendors / service providers) who want to promote / sell network services. Additionally, the product / content team module or portal 222 can be a lead conversion system or a user who wants to convert customer leads, analyze customer activity, and facilitate pre-sales and post-sales activities and marketing campaigns.

[0084] Continuing with reference to FIG. 2 , marketplace platform module or portal 200 can also communicate with orchestrator module 230 (e.g., via an application programming interface (API)). In particular, any one or more of modules or portals 212-220 of modules 210 can communicate with and / or be supported by any one of modules or portals 238-240 of modules 230. Specifically, orchestrator module 230 can include product / service management module 238 and cloud orchestration module 240. In particular, any of modules 238-240 can be containerized and stored in a “cloud” or cloud cluster on external servers and databases. In some embodiments, product / service management module 238 comprises a repository / data storage that stores products and services offered by vendors / service providers. In some embodiments, the products and services are virtualized or software-based. In some embodiments, the cloud orchestration module 240 is configured to schedule and offer products and services to users of the network services marketplace platform.

[0085] The modules shown in Figure 2 are associated with functionality for transacting (e.g., selling, buying, promoting, etc.) products or services in the marketplace platform. In addition to the modules of Figure 2, the marketplace platform may also include modules for managing the end-to-end process for the product or service (e.g., a sign-up process, a publishing process, a deployment process, an operational process, etc.), as well as modules for managing one or more tests associated with the end-to-end process.

[0086] 3 illustrates an example block diagram of a system module for managing one or more tests, according to one or more embodiments. The marketplace platform module 310 of FIG. 3 may correspond to the marketplace platform module 200 of FIG. 2 and may include one or more modules shown in FIG. 2, and thus, relevant redundant description may be omitted below for the sake of brevity.

[0087] 3, marketplace platform module 310 may include a testing module 320. Testing module 320 may be configured to perform and / or manage one or more tests of one or more services within the marketplace platform through an end-to-end process of the one or more services (the one or more tests hereinafter referred to as an “end-to-end test”).

[0088] Test module 320 may be configured to provide a virtualized test environment and process (e.g., execute, manage, etc.) one or more end-to-end tests in the virtualized test environment. The virtualized test environment may provide a secure test environment (e.g., a test environment isolated from existing system infrastructure and system operations) for running one or more tests on a service before and / or after deploying the service on the marketplace platform.

[0089] The virtualized test environment may include multiple types of test environments, each of which may be provided for processing one or more tests at different stages. For example, a lab or sandbox test environment isolated from other running or deployed services within the marketplace platform may be provided for processing tests in early stages (e.g., service design, initial commissioning of the service, etc.) to enable estimation of resources required for service optimization / development and validation of service configurations without affecting the operation of the marketplace platform, while mitigating system failures and / or avoiding the spread of service vulnerabilities during testing. Meanwhile, a staging test environment, which may be a replica (with more or fewer resources) of the required system infrastructure and / or required system resources, may be provided for processing tests before actual deployment (e.g., acceptance testing to ensure that the service meets acceptance criteria, etc.) to enable simulation of the actual performance and functionality of the service. Similarly, an operational test environment, which may be a replica of the current system infrastructure and / or available system resources, may be provided for processing tests after the service is deployed and operational.

[0090] 3, test module 320 may include a mandatory test module 321 and an optional test module 322. Mandatory test module 321 may be configured to handle (e.g., initiate, run, configure, manage, monitor, etc.) one or more mandatory tests, and optional test module 322 may be configured to handle (e.g., initiate, run, configure, manage, monitor, etc.) one or more optional tests.

[0091] The mandatory test module 321 and / or the optional test module 322 may be configured to run one or more of the associated tests based on a predetermined order. For example, the mandatory test module 321 may be configured to run one or more mandatory tests on the service before one or more optional tests (e.g., automatically as soon as the service is uploaded to the marketplace platform, in response to an event, etc.). Meanwhile, the optional test module 322 may be configured to run one or more optional tests after operations associated with the mandatory test module 321 are completed. In some embodiments, the mandatory test module 321 and / or the optional test module 322 may be configured to run one or more of the associated tests based on a priority level and / or urgency level defined by a test requester and / or test implementer. For example, when a test requester uploads a service to the marketplace, the test requester may specify (e.g., via a GUI) a priority level of the service (e.g., “critical,” “medium,” etc.) and / or a urgency level of the tests (e.g., “urgent,” “non-urgent,” etc.). Similarly, a tester can intervene in the testing process and adjust the urgency level of the tests appropriately. Based on the specified priority and / or urgency levels, the mandatory test module 321 and the optional test module 322 can then be configured to execute the relevant tests accordingly (e.g., tests with higher priority / urgency levels are executed before tests with lower priority / urgency levels, etc.).

[0092] As an example, the one or more mandatory tests may include one or more static tests, one or more acceptance tests, and / or one or more mandatory security tests.

[0093] One or more static tests may be performed to validate a service without implementing or running the service. For example, one or more static tests may be performed to (a) scan one or more programming code associated with the service (without executing the programming code) to check for defects or deficiencies in the programming code, (b) validate the design of the service based on one or more design documents, and (c) confirm that the service meets one or more required system requirements. Because static tests consume few resources and can be effective in detecting major defects and errors (which may result in a reduction in the overall cost and time of testing and effectively reduce defects and errors in subsequent testing stages), static tests may be performed during early stages of testing (e.g., before running other tests).

[0094] In an example embodiment, the one or more static tests may include, but are not limited to, scanning static objects (e.g., images, programming code, requirements specifications, design documents, logs, matrix documents, Helm charts, Docker, etc.). For example, images included in the service may be scanned to ensure that the image format conforms to a system-supported format, to ensure that the image does not contain sensitive words or content, to prevent the image from being accompanied by malicious items, etc. Additionally, Helm charts may be scanned to ensure proper indentation, to ensure zero indentation errors, etc. Additionally, one or more source codes may be scanned to ensure that the source code is readable by the system, to verify whether there are any credentials exposed in the source code as plain text, etc. Furthermore, one or more Dockers may be scanned to ensure that the Docker version is up to date, to ensure that the Docker contents comply with system requirements, etc.

[0095] Meanwhile, one or more acceptance tests may be performed to verify the functionality and performance of the service to ensure that the service meets one or more acceptance criteria before being deployed and used in the marketplace. The acceptance criteria are conditions that must be met for the service to be accepted by personnel (e.g., customers, system administrators, etc.) and may be defined by personnel based on, for example, customer needs, feedback from other tests, real-world conditions or requirements, service quality or performance as stated in a service level agreement (SLA), conditions defined in a standard operating environment (SOE), etc. The one or more acceptance tests may include performance tests, load tests, throughput tests, functional tests, style tests, etc.

[0096] Additionally, one or more mandatory security tests may be performed to verify whether a service complies with one or more minimum security requirements. For example, the one or more mandatory security tests may include a quick scan of specific areas or criteria of the service (e.g., criteria deemed important by a system administrator, areas of the service that could affect the stability or operation of the marketplace if compromised, areas of the service that are most likely to contain threats or malicious files, such as malware, spyware, viruses, etc.). The one or more mandatory security tests may provide mandatory security validation for the service in a short period of time to reduce the risk that the service will cause security issues after being deployed and used in the marketplace. Furthermore, the one or more mandatory security tests may identify potential security issues (e.g., bugs, design flaws, etc.) in the service so that relevant personnel can patch or update the service to address potential security issues before the service is deployed and used in the marketplace.

[0097] It should be apparent that the above static tests, acceptance tests, and mandatory security tests are merely examples of one or more mandatory tests, and it is contemplated that the one or more mandatory tests may further include any other suitable tests that are deemed mandatory by the tester without departing from the scope of this disclosure.

[0098] Meanwhile, by way of example, the one or more optional tests may include one or more integration tests, one or more dynamic tests, and one or more application tests.

[0099] One or more integration tests may be performed to validate and clarify service integration in the marketplace, such as integrating the service with one or more system elements, integrating the service with other dependent service(s) in the marketplace (e.g., status monitoring, performance management, fault detection, inventory management, etc.), etc. The one or more integration tests may validate that services can interoperate with each other (e.g., that multiple services can operate or work together to provide an intended functionality, etc.), determine the effectiveness or performance of different services when interoperating, and determine potential defects or errors when the service is integrated with other existing elements of the marketplace.

[0100] In an exemplary embodiment, the one or more integration tests may include one or more configuration tests, one or more container tests, one or more reachability tests, and one or more service compatibility tests. The one or more configuration tests may be performed to analyze the configuration associated with the service to identify potential error(s) or inconsistency(ies) in the configuration. For example, the one or more configuration tests may include a linting process that processes and analyzes code and files associated with the configuration of the service to determine potential program and / or style errors. Meanwhile, the one or more container tests are performed to ensure that the container(s) associated with the service can operate properly (e.g., start up, function, etc.) when utilized. For example, the one or more container tests may include a container startup test, in which a test run of a container(s) associated with the service is performed to determine whether the container(s) can operate as expected when integrated into the marketplace platform, and the status of the container(s), such as the status of the pod(s) associated with the container(s), is validated after the test run. Additionally, one or more reachability tests may be performed to verify connectivity between the service and other elements of the marketplace to verify whether the service is reachable by other elements after it is implemented. For example, the one or more reachability tests may include a ping test between two elements (e.g., a service, a container, a pod, etc.), a telnet connectivity test to determine available ports (e.g., transmission control protocol (TCP) ports, etc.), a secure shell (SSH) connectivity test, a connection timeout test, an API test, etc. Additionally, one or more service compatibility tests may be performed to verify whether a service is compatible with other services in the marketplace platform and to check interoperability between services.For example, one or more service compatibility tests may include infrastructure or resource compatibility tests (e.g., tests to verify whether infrastructure or resources required by a service are compatible with the configurations of other services), operating system compatibility tests (e.g., tests to verify whether a service is compatible with the operating systems of other services), software compatibility tests (e.g., testing whether software coding is compatible with other services), etc.

[0101] One or more static tests may be performed to validate the service through execution of the service. Specifically, one or more dynamic tests may be performed to validate the service by testing the behavior of the service at runtime using dynamic variables and comparing the actual runtime behavior with expected behavior. For example, the one or more dynamic tests may include one or more functional tests (e.g., unit tests, system tests, etc.) and / or one or more non-functional tests (e.g., recovery tests, usability tests, etc.). In some embodiments, the one or more dynamic tests may include security tests, such as dynamic application security testing (DAST).

[0102] One or more application tests may be executed to validate the overall software application behavior of the service. In some embodiments, the one or more application tests may include one or more front-end functionality tests, one or more back-end functionality tests, and / or one or more end-to-end functionality tests. For example, the one or more application tests may include GUI tests, database tests, load tests, etc. The one or more application tests may vary depending on the type of application. For example, tests for a Python-based application may have a different workflow or pipeline compared to a JSON-based application, a Jenkins-based application, etc.

[0103] It should be apparent that the above-described integration tests, dynamic tests, and application tests are merely examples of the one or more optional tests, and it is contemplated that the one or more optional tests may further include any other suitable tests deemed optional by a tester without departing from the scope of the present disclosure. Furthermore, one or more of the optional tests may enable a test requester to use one or more specific test scripts, which may be triggered as part of pre- or post-deployment of a service. To this end, a test requester may upload different types of scripts (e.g., Java, YAML, shell, JSON, Python, etc.) to test a service. These scripts may be stored in a specific storage location from which the test requester can initiate application tests.

[0104] Referring back to FIG. 3, test module 320 (or modules 321 and / or 322 included therein) may be configured to automatically execute one or more of the above tests based on a respective test process, workflow, and / or pipeline.

[0105] For example, each test can have a test profile associated with it, which can include information and configurations for respective test processes, rules and policies, workflows, settings, etc. Some of the test profiles can be predefined by a user (a tester and / or a test requester) before a test and / or configured by a user during or after a test. Alternatively, some of the test profiles can be generated or adjusted by the test module 320 based on, for example, user requirements, test history, system conditions, etc. For example, a user can specify requirements via GUI(s), and the test module 320 can search for appropriate test algorithms, workflows, etc. from an internal database (e.g., retrieve any test profile(s) with similar user requirements, etc.) and / or an external database (e.g., search for test profile(s) based on keywords, etc.), and then generate the test profile accordingly. Also, each test profile can be assigned a unique profile ID.

[0106] In some embodiments, test module 320 may be configured to collect and compile test-related information and / or materials and provide them to users (test administrators and / or test requesters) who wish to monitor or manage tests so that they can critique, inspect, and / or adjust tests as needed (e.g., by adjusting test profiles, etc.).

[0107] 4 shows an example block diagram of system modules for managing one or more end-to-end processes, according to one or more embodiments. Marketplace platform module 410 may correspond to marketplace platform module 200 of FIG. 2 and / or marketplace platform module 310 of FIG. 3 and may include one or more modules shown in FIG. 2 and / or FIG. 3, and thus, redundant description thereof may be omitted below for the sake of brevity.

[0108] 4, marketplace platform module 410 may include an end-to-end process management module 420 that may be configured to handle end-to-end processes for the service, such as a sign-up process, a publication process, a deployment process, and an operation process. Additionally, end-to-end process management module 420 may be communicatively coupled to one or more of the above-mentioned modules in the marketplace platform module (e.g., the modules shown in FIGS. 2 and / or 3) in handling the end-to-end processes. For example, end-to-end process management module 420 may be communicatively coupled to testing module 320 of FIG. 3 to provide testing or related features in one or more end-to-end processes (described further below).

[0109] In an exemplary embodiment, end-to-end process management module 420 and its various operations are provided by executing instructions embodied in a microservices architecture, i.e., an application architecture comprised of multiple independently executable services. It is understood that one or more other embodiments are not so limited, and end-to-end process management module 420 may utilize a different application or software design architecture (e.g., monolithic).

[0110] In an example embodiment, end-to-end process management module 420 can include a sign-up process module 421, a publication process module 422, a deployment process module 423, and an operation process module 424, each of which can be configured to perform a respective end-to-end process, i.e., a sign-up process, a publication process, a deployment process, and an operation process. Each of the sign-up process, the publication process, the deployment process, and the operation process can be defined in the form of one or more microservices.

[0111] Different types of users (i.e., users with different roles) may be involved in one or more of the end-to-end processes. For example, all users may be involved in the sign-up process, only users with developer (e.g., vendor, service provider, etc.) and reviewer roles may be involved in the publishing process, and only users with operator (e.g., customer), administrator, and / or reviewer roles may be involved in the deployment and operations processes.

[0112] In this regard, the sign-up process module 421 may be configured to guide users through a sign-up process to enable them to be onboarded and / or authenticated as users in a standardized manner (i.e., in a predefined user schema with predefined roles and privileges). Further, the publication process module 422 may be configured to provide a secure and rapid publication process for services in the marketplace. Further, the deployment process module 423 may be configured to provide a secure and rapid deployment process for services in the marketplace. Additionally, the operation process module 424 may be configured to provide a product and service monitoring and lifecycle control process throughout the operation of the products and services in the marketplace.

[0113] Additionally, each of sign-up process module 421, publication process module 422, deployment process module 423, and operational process module 424 may be configured to provide different sets of operations and / or GUI(s) to associated personnel during the corresponding process. For example, publication process module 422 may provide a first set of operations and / or GUI(s) to users with a developer role and a second set of operations and / or GUI(s) to users with a reviewer role, with one or more differences between the two sets. Similar processes may be applicable to other end-to-end process modules (i.e., module 422, module 423, and / or module 424).

[0114] In an example embodiment, publication process module 422 may be configured to provide a set of operations and / or GUI(s) that enable users to upload their products or services (e.g., product or service programming packages), product information, parameters, artifacts, etc. to the marketplace. Similarly, deployment process module 423 may be configured to provide a set of operations and / or GUI(s) that enable users to deploy published products or services to the marketplace. For example, via GUI(s), a user (e.g., a user with a developer role) may make a request to deploy a service, and another user (e.g., a user with a reviewer role) may approve the request. Similarly, operational process module 424 may be configured to provide a set of operations and / or GUI(s) that enable users to connect to and select one or more services, manage the lifecycle and operations of the selected service(s), etc.

[0115] Additionally, one or more of the above modules of the end-to-end process management module 420 may be configured to communicatively couple to a testing module (e.g., module 320 of FIG. 3 ) and may be configured to provide one or more tests or related features in the respective end-to-end processes. For example, the publication process (handled by the publication process module 422) may include one or more mandatory tests (e.g., static tests, mandatory security tests, etc.) and / or one or more optional tests (e.g., integration tests, etc.) on the service before publication or publishing of the service in the marketplace. Similarly, the deployment process (handled by the deployment process module 423) may include one or more mandatory tests (e.g., static tests, acceptance tests, etc.) and / or one or more optional tests (e.g., integration tests, application tests, etc.) on the service before deploying the service in the marketplace and allowing other users to use the service. Similarly, the operational process (handled by the operational process module 424) may include one or more mandatory tests (e.g., static tests, mandatory security tests, etc.) and / or one or more optional tests (e.g., dynamic tests, application tests, etc.) before, during, or after the service is operational.

[0116] As an example, the GUI(s) provided by the modules of the end-to-end process management module 420 may allow a user to configure, customize, and / or manage one or more tests throughout the end-to-end process of a service. For example, the publishing process module 422 may receive input from a test requester via the GUI(s) and provide the test requester's input to other modules so that a customized test can be calculated based on the input, and the test module can then execute the customized test accordingly. Furthermore, the publishing process module 422 may be configured to receive information or data from other modules and update the GUI(s) to include the latest test information (e.g., test configuration, test status, test logs, etc.). Upon successful completion of a test, the product or service may be published in the marketplace by the publishing process module 422. It is apparent that one or more tests involved in other end-to-end processes may be similarly provided and managed.

[0117] FIG. 5 illustrates an exemplary block diagram of a system for managing end-to-end testing, according to one or more embodiments.

[0118] 5, the marketplace platform module 510 may comprise a GUI module 520, an automation module 530, a testing module 540, and a storage module 550. Additionally, the marketplace platform module 510 may be communicatively coupled to a user module 560.

[0119] To this end, marketplace platform module 510 may correspond to marketplace platform module 200 of FIG. 2, marketplace platform module 310 of FIG. 3, and / or marketplace platform module 410 of FIG. 4. Additionally, test module 540 may correspond to test module 320 of FIG. 3, GUI module 520 may correspond to any module capable of generating and presenting one or more GUIs (e.g., CMS delivery node module 202 or CMS integration module 206 of FIG. 2, end-to-end process management module 420 of FIG. 4, etc.), and user module 560 may correspond to any module capable of communicating with a user (e.g., module 222, module 224, or module 228 of FIG. 2, etc.). Thus, the relevant operations and features described above are equally applicable herein unless otherwise specified, and redundant descriptions related thereto may be omitted below for brevity.

[0120] 5 , GUI module 520 may be communicatively coupled to user module 560, automation module 530, and storage module 550. In an example embodiment, GUI module 520 may be configured to generate and present one or more GUIs to a user (e.g., a test requester, such as a vendor, service provider, or developer, and / or a test implementer, such as a system administrator) via user module 560. The one or more GUIs may include information of a service associated with the user. For example, the one or more GUIs may include application information for the service, documentation and specifications related to the service, configuration and settings for the service, network or computing resources allocated to or required by the service, process status (e.g., uploaded, published, deployed, etc.), test-related information (e.g., test status, test logs, test results, potential issues identified from the tests, system recommendations based on the test results, etc.), and any other suitable information that can provide a user with a comprehensive view of the service.

[0121] Additionally, the one or more GUIs may include one or more interactive elements (e.g., checkboxes, buttons, sliders, etc.) with which a user can interact to configure and customize one or more tests. Specifically, one or more of the interactive elements in the one or more GUIs may be mapped to respective test information (e.g., test profiles) such that, upon interaction with a user, the GUI module 520 may be configured to collect or compile information of the user's intended tests and provide it to the automation module 530 for further processing.

[0122] As an example, assume that a user wants to run two optional tests on a service, such as Integration Test 1 and Dynamic Test 2. In this regard, the user can simply select Integration Test 1 and Dynamic Test 2 on the GUI (e.g., by clicking their respective checkboxes) and click the "Run Tests" button. In response, GUI module 520 can collect the test profile IDs of Integration Test 1 and Dynamic Test 2 and provide the test profile IDs to automation module 530.

[0123] Automation module 530 may be communicatively coupled to GUI module 520 and testing module 540. In an exemplary embodiment, automation module 530 may be configured to receive information from GUI module 520 (e.g., test profile IDs associated with user-selected tests, etc.) and may be configured to automatically generate a testing scheme based on the information received from GUI module 520. In this manner, the testing scheme may define a customized testing process or workflow, such as one or more tests selected by a user, an order for running the one or more tests, start and / or end times for running the one or more tests, priority and / or urgency levels of the tests, specific criteria to be tested during the one or more tests, a schedule for periodically or periodically repeating the one or more tests, etc.

[0124] The automation module 530 may then be configured to provide the generated test scheme to the test module 540, which may execute the associated tests accordingly. In an example embodiment, the automation module 530 may be configured to generate an API (e.g., a real-time API, a FastAPI, etc.) call based on the received information (e.g., a test profile ID, etc.) to provide the test scheme or associated information to the test module 540. In another embodiment, the automation module 530 may be configured to include queries (within the test scheme), such as structured query language (SQL) queries, language integrated query (LinQ), Hibernate query language (HQL) queries, Scala query language (ScalaQL) queries, etc. In some embodiments, the automation module 530 may comprise one or more automation servers (e.g., Jenkins, GitLab, etc.).

[0125] Test module 540 may be communicatively coupled to automation module 530 and storage module 550 and may include a mandatory test module 541 and an optional test module 542 (which may correspond to mandatory test module 321 and optional test module 322, respectively, of FIG. 3 ). Test module 540 may be configured to receive a test scheme or related information (e.g., in the form of API calls, interactive queries, etc.) from automation module 530 and may be configured to execute one or more tests in accordance with the test scheme.

[0126] For example, test module 540 may be configured to obtain test profile(s) (e.g., received from automation module 530) associated with one or more tests defined in the test scheme and execute one or more tests based on the associated test profile(s). Additionally, test module 540 may be configured to validate the test scheme before executing one or more tests defined in the test scheme to ensure that algorithms or requirements defined in the test scheme are accurate and / or feasible.

[0127] As an example, automation module 530 may determine that the test scheme indicates that mandatory tests should be run after optional tests, while the configuration profile for mandatory tests indicates that mandatory tests should be run before optional tests. In that case, test module 540 may provide a response to automation module 530 so that automation module 530 can adjust or update the test scheme accordingly, or may provide a recommendation regarding a modified test scheme and provide it to automation module 530. In response, automation module 530 may provide the revised test scheme to GUI module 520, and GUI module 520 may update the GUI to inform the user about the recommended test scheme.

[0128] In some embodiments, the GUI(s) provided to the user may include a search box that allows the user to easily define the tests they need (e.g., by inserting keywords, selecting tags, etc.). The GUI module 520 may then receive the test requirements and send them to the automation module 530, which may process the test requirements and generate queries, API calls, etc. to request relevant tests based on the keywords, tags, etc. In this regard, the automation module 530 may also be configured to search for one or more relevant tests and provide test recommendations to the user. As an example, assume that a user wants to perform integration testing of their service with cloud storage A, but cloud storage A is not available in the marketplace, the user can simply specify their intended features (e.g., “cloud native,” “hosting on the cloud,” “backup storage,” etc.) via the GUI(s) provided by the GUI module 520, and the automation module 530 and test module 540 may automatically determine and recommend service(s) with similar features and relevant tests to the user via the GUI(s).

[0129] As tests are performed, the test module 540 may be configured to continuously or periodically store test information, such as test status (e.g., pending, in progress, successful, failed, etc.) and test results (e.g., test logs, test criteria, service performance during the test, reasons for test failure (if any), warnings about potential defects or errors detected during the test, etc.) in the storage module 550.

[0130] The storage module 550 may be communicatively coupled to the testing module 540 and configured to receive test information or data (e.g., test status, test results, etc.) therefrom and store the received information or data in one or more associated storage media. In an exemplary embodiment, the storage module 550 may comprise at least two storage media (e.g., a database, a server, etc.), where the test status may be stored in one of the at least two storage media and the test results / logs may be stored in the other of the at least two storage media. In this regard, the test media that stores the test status may be communicatively coupled to or accessible by other services within the marketplace platform (e.g., an observability or monitoring service, etc.) so that the test status stored therein is available to the other services. In some embodiments, the storage medium that stores the test results or logs may be object storage (e.g., MinIO, etc.). Additionally, while storage module 550 is shown in FIG. 5 as being included in marketplace platform module 510, it is contemplated that the storage module may be any suitable storage medium external to marketplace platform module 510 without departing from the scope of the present disclosure.

[0131] Also, storage module 550 may be communicatively coupled to GUI module 520 and configured to provide stored information or data to GUI module 520. For example, storage module 550 may be configured to continuously or periodically provide test status and / or test report / log information or data to GUI module 520. Alternatively, GUI module 520 may be configured to continuously or periodically scan storage module 550 for new information or data related to one or more GUIs presented to the user, and then send requests to storage module 550 to retrieve the new information or data, if any. In an exemplary embodiment, storage module 550 may be configured to provide the information or data in data batches or data bundles.

[0132] Upon receiving information or data from the storage module 550, the GUI module 520 may be configured to generate new GUI(s) or update the presented GUI(s) to include the relevant information. For example, the GUI module 520 may be configured to update the GUI(s) of the test workspace to present the latest or updated status of each test selected by the user.

[0133] On the other hand, upon determining that one or more test reports or logs are available in the storage module 550, the GUI module 520 may be configured to update the GUI(s) of the test workspace such that the test workspace includes one or more interactive elements (e.g., buttons, clickable icons, etc.) such that a user can interact with the interactive elements (e.g., double-click, press a key, etc.) to view the status report or log (e.g., view locally via download, view online via the test workspace, etc.).

[0134] To this end, the above-described embodiments provide systems and methods that enable efficient processing of one or more tests. For example, the GUI module 520 can provide one or more GUIs forming a test workspace that provides comprehensive test information, such as test options, test status, test type, and test results, while allowing a user to easily and efficiently manage or customize one or more tests for a service. Based on the user's interaction with the one or more GUIs, the automation module 530 can automatically or dynamically generate a customized test scheme for the user, and the test module 540 can automatically execute one or more tests according to the test scheme. Detailed test status can be automatically and efficiently presented to the user via the test workspace, allowing the user to have a comprehensive view and improving the user experience. Furthermore, test reports or logs can be easily viewed or retrieved by the user.

[0135] Some exemplary embodiments of the test workspace are described below with reference to the associated GUI.

[0136] FIG. 6 illustrates a generalized GUI of a test workspace, according to one or more embodiments, while FIG. 7 illustrates an example GUI of a test workspace, according to one or more embodiments. For purposes of explanation, some description of FIG. 6 may be provided with reference to FIG. 7. Additionally, some elements of FIG. 7 may overlap with elements of FIG. 6, and therefore, descriptions of such overlapping elements may be omitted below for the sake of brevity.

[0137] Further, GUI 600 may be generated and presented by GUI module 520 of Figure 5. For example, whenever a user accesses the marketplace (e.g., via user module 560, etc.), GUI module 520 may collect test information associated with the user (e.g., from user module 560, etc.) and generate GUI 600 accordingly.

[0138] 6, GUI 600 may include a first partition 610 and a second partition 620. First partition 610 may include one or more blocks showing one or more available test categories, which may be determined by GUI module 520 based on user information. In the exemplary embodiment of FIG. 6, first partition 610 includes a first block 611 showing available mandatory test categories and a second block 612 showing available optional test categories. Nevertheless, it is contemplated that different users may have different available test category(s) presented in first partition 610. As an example, User 1's test workspace may have two mandatory test categories (e.g., static tests, acceptance tests, etc.) and one optional test category (e.g., integration tests, etc.) presented in the associated first partition, while User 2's test workspace may have one mandatory test category (e.g., mandatory security tests, etc.) and two optional test categories (e.g., dynamic tests, application tests, etc.) presented in the associated first partition.

[0139] 6, each of the first block 611 and the second block 612 may comprise a checkbox that is interactable or selectable by the user. In the exemplary embodiment of FIG. 6, selection of the second block 612 is disabled (e.g., by the GUI module 520) (i.e., selection of the second block 612 may only be available after the required tests have been configured) because the associated test administrator may require the user to complete configuration of the required tests before configuring the optional tests. However, it is contemplated that selection of the second block 612 may be enabled along with or before selection of the first block 611 without departing from the scope of the present disclosure.

[0140] It should be further apparent that the first partition 610 may include fewer or more blocks than those shown in FIG. 6. For example, referring to FIG. 7, the first partition 710 may include a first block 711 of static tests, a second block 712 of acceptance tests, a third block 713 of integration tests, a fourth block 714 of dynamic tests, and a fifth block 715 of application tests. In this case, blocks 711-712 are classified into one test category (i.e., the mandatory test category), and blocks 713-715 are classified into another test category (i.e., the optional test category). A user can choose to configure either the mandatory tests, i.e., static tests, or acceptance tests, by clicking the respective checkboxes. The selected block is presented (e.g., highlighted, filled with a pattern, etc.) in a manner that distinguishes it from other block(s) belonging to the same test category. In the exemplary embodiment of FIG. 7, a static test is selected, and therefore the associated block (i.e., the first block 711) is configured to be filled with diagonal lines, and the associated checkbox is updated to contain a check mark.

[0141] 6, second partition 620 may include test description 621, subwindow 630, and multiple interactive elements 623-624. The contents of the second partition may be associated with the test category selected by the user in partition 610. Specifically, based on the user's selection in first partition 610, GUI module 520 may generate or update the contents of second partition 620 accordingly. As an example, based on determining that the user selected a static test in the first partition, GUI module 520 may generate or update the second partition to include information about the static test associated with the user (as shown in FIG. 7).

[0142] Test description 621 may include a test title and a summary or brief description of the test (e.g., the corresponding portion of FIG. 7 shows an example test description related to a static test). The content of test description 621 may be predefined by the tester or any other appropriate personnel. In some embodiments, test description 621 may include a summary of the test history. For example, test description 621 may include information such as tests previously run, tests that are not completed, errors or defects determined from previous tests, etc. In this way, an inexperienced user can gain a comprehensive understanding of the selected test, which may be useful to the user when determining how to configure or customize tests for a service.

[0143] 6, sub-window 630 may include multiple columns, each defining an information group (e.g., test name, test type, test status, test result, etc.), and may include multiple rows, each containing information for one or more available tests (e.g., tests associated with the test category selected by the user in partition 610). For example, referring to FIG. 7, because static tests are selected in first partition 710, information for available static tests (e.g., configuration test 1, reachability test 1, compatibility test, etc.) is presented in sub-window 730 of second partition 720.

[0144] Referring back to FIG. 6, sub-window 630 may include one or more interactive elements 631 , one or more interactive elements 632 , and one or more interactive elements 633 .

[0145] The one or more interactive elements 631 may be icons that present information related to them when interacted with (e.g., hovered, scrolled, clicked, etc.) by a user. For example, when a user interacts with an interactive element 631 associated with the “Results” column (e.g., when the user taps the interactive element 631 via a screen of the user module 560, etc.), a summary of the test results is presented, for example, in a hover window, a pop-up window, etc. It is contemplated that the interactive elements 631 may also be associated with any other information within the test workspace. For example, one or more interactive elements 631 may be associated with each of the tests listed in the sub-window 630 such that information related thereto is presented when a user interacts with the one or more interactive elements 631. By way of example, with reference to FIG. 7 , one or more of the Configuration Test 1, Reachability Test 1, and Compatibility Test may have an interactive element 631 associated therewith such that a summary or description of the test is presented when a user interacts with the respective interactive element 631.

[0146] 6 , one or more interactive elements 632 may be one or more icons that, when interacted with (e.g., clicked, etc.) by a user, rearrange the sequential display of information in sub-window 630. For example, when a user clicks on an up icon of element 632 associated with a test type, GUI module 520 may be configured to generate a new GUI or update GUI 600, such that the test information included in sub-window 630 is sorted according to test type (e.g., tests according to test type 1 are presented first, followed by tests according to test type 2, etc.), alphabetically by test, etc. Similar interactive elements may be associated with other test information, such as test name, test status, test result, etc., and the information in sub-window 630 may similarly be sorted according to test name, test status, test result, etc.

[0147] One or more interactive elements 633 may be one or more icons that, when interacted with (e.g., clicked on, etc.) by a user, retrieve the user's associated test results or logs. For example, when a user clicks element 632 associated with Forced Test 1, GUI module 520 may be configured to retrieve the test results or logs associated with Forced Test 1 (e.g., from storage module 550, etc.) and then present the retrieved test results or logs to the user (e.g., in a pop-up window, etc.). Furthermore, upon detecting user interaction with element 632, the associated test results or logs may be downloaded by user module 560. In some embodiments, upon detecting user interaction with element 632, the test results or logs for all tests presented in sub-window 630 (or one or more tests selected in sub-window 630) may be downloaded in a compiled document.

[0148] 6 , check boxes associated with tests may be pre-populated by GUI module 520. For example, during generation of GUI 600, GUI module 520 can determine which associated mandatory tests have completed successfully and which have not, and then pre-populate the check boxes for the mandatory tests that have not yet completed. As an example, GUI module 520 may determine that mandatory test 1, mandatory test 3.1, and mandatory test 3.2 have completed successfully, but other mandatory tests have not yet completed. Thus, when generating sub-window 630, GUI module 520 can be configured to pre-populate the check boxes for tests other than mandatory test 1, mandatory test 3.1, and mandatory test 3.2.

[0149] Additionally, interactive elements 623 and 624 in second partition 620 perform their associated actions when interacted with (e.g., hovered over, scrolled, clicked, etc.) by a user. For example, after selecting or configuring one or more tests to be run, a user can interact with element 623 to initiate the testing process. Upon detecting user interaction with element 623, GUI module 520 may be configured to compile information about the selected tests (e.g., generate a list of the selected tests, collect a test profile ID for each selected test, etc.) and provide the information to other modules. For example, the information may be provided to automation module 530, which may be configured to generate a test scheme based on the user's selected tests and then provide the generated test scheme to test module 540, such that test module 540 runs the user's selected tests based on the test scheme.

[0150] Instead, the user may continue to configure one or more tests in the next category (e.g., optional tests, 612, etc.) by interacting with element 624. For example, upon detecting that the user has interacted with element 624, the GUI module may update the test workspace of GUI 600 such that the contents of first partition 610 and second partition 620 are updated accordingly.

[0151] 8 illustrates another example generalized GUI of a test workspace, according to one or more embodiments. GUI 800 of FIG. 8 may represent the test workspace upon detecting a user's interaction with interactive element 624 (i.e., the "Next" button) of FIG. 6, indicating that the user wishes to continue configuring optional tests following the configuration of mandatory tests. Additionally, some elements of FIG. 8 may overlap with elements of FIG. 6, and therefore, descriptions of such overlapping elements may be omitted below for the sake of brevity.

[0152] Referring to FIG. 8, the first block 811 and the second block 812 in the first partition 810 may be presented differently from the first block 611 and the second block 612 in the first partition 610 of FIG. 6 to inform the user that the configuration of the test category of the first block (e.g., mandatory tests) is complete and the content shown in the test workspace is associated with the test category of the second block (e.g., optional tests).

[0153] The second partition 820 may include content, information, and interactive elements related to the optional tests defined in the second block 612, similar to those described in connection with the second partition 620 of Figure 6. Additionally, the second partition 620 may include an additional interactive element 822 that, when interacted with by the user, redirects the user to a previous test workspace (e.g., GUI 600 of Figure 6) so that the user can adjust or reconfigure the tests configured in the previous test workspace. Furthermore, the second partition 820 may also include a status summary 825 that can present the status of the tests configured in the previous test workspace.

[0154] 8, the forced test has not yet completed, so the user can choose to continue configuring other available tests (e.g., optional tests) or simply wait for the forced test to complete before the user can deploy the service.

[0155] 9 shows another example of a generalized GUI of a test workspace according to one or more embodiments. GUI 900 of FIG. 9 may show a test workspace similar to that of GUI 800 after a forced test has been completed. Additionally, some elements of FIG. 9 may overlap with elements of FIG. 6, and therefore, descriptions of such overlapping elements may be omitted below for brevity.

[0156] 9, the second partition 920 may include a status summary 925. The status summary 925 may be an updated version of the status summary 825 of FIG. 8, which may present an updated status of the tests configured in the previous test workspace. Additionally, the second partition 920 may include an additional interactive element 926 upon detecting that the service is ready to be deployed (e.g., that the forced tests have completed, etc.). In response, when the interactive element 926 is interacted with by a user, the GUI module 520 may trigger a deployment process for the service (e.g., via the deployment process module 423 of FIG. 4).

[0157] 6-9 and the associated description are provided herein for illustrative purposes only, and it is contemplated that the GUI of the test workspace should not be limited thereto. Specifically, without departing from the scope of this disclosure, the GUI of the test workspace may be presented differently (e.g., a first partition may be presented in an upper portion of the GUI, a second partition may be presented in a lower portion of the GUI, etc.), the GUI may include less or more information, the GUI may include fewer or more interactive elements, etc. Furthermore, the test workspace may be applicable to other end-to-end processes, such as a release process, an operational process, etc., without departing from the scope of this disclosure.

[0158] 10 illustrates an example flow diagram of a method 1000 for interacting with a user to configure one or more tests, according to one or more embodiments. Method 1000 may be performed by GUI module 520 of FIG. 5.

[0159] 10, at operation S1010, test information is presented to a user. Specifically, a GUI module (e.g., module 520) may be configured to generate a GUI that includes information of one or more tests associated with the user's service and then present the GUI to the user via a user module (e.g., module 560).

[0160] For example, the GUI module may be configured to receive user information (e.g., a user ID, a user profile, etc.) from the user module, determine a service associated with the user based on the user information, collect test information related to the determined service (e.g., tests previously run on the service, tests available for the service, test status of the service, etc.), and generate a GUI based on the collected test information.

[0161] In some embodiments, the generated GUI may include a first partition (e.g., first partition 610, etc.) and a second partition (e.g., second partition 620, etc.), where the first partition may include one or more blocks (e.g., block 611, block 612, etc.), each of which may include information about an available test category (e.g., mandatory tests, optional tests, etc.), and the second partition may include (a) a description of the test category selected in the first partition (e.g., test description 621), (b) a sub-window (e.g., sub-window 630) that includes one or more available tests associated with the test category selected in the first partition and one or more interactive elements (e.g., element 631, element 632, element 633, etc.), and (c) one or more interactive elements (e.g., element 623, element 624, etc.). Each of the one or more interactive elements in the sub-window of the second partition may be associated with one or more available tests (e.g., mapped with a test profile ID associated with the respective test, etc.). For example, the Force Test 1 checkbox may be mapped to the test profile ID of Test Profile 1.

[0162] 10 , at operation S1020, user input is received. Specifically, the GUI module may be configured to receive user selection of one or more tests. In some embodiments, the GUI module may be configured to receive user interaction with one or more interactive elements (e.g., checkboxes, etc.) associated with the available tests.

[0163] At operation S1030, test information associated with the user's input is collected. Specifically, the GUI module may be configured to collect information about tests selected by the user. For example, the GUI module may be configured to determine which tests were selected by the user based on the user's input. As an example, based on a determination that the user interacted (e.g., checked, clicked, etc.) with the checkboxes for Mandatory Test 1, Mandatory Test 2, and Optional Test 3, the GUI module may determine that the user selected Mandatory Test 1, Mandatory Test 2, and Optional Test 3, and may then collect the test profile ID for Mandatory Test 1, the test profile ID for Mandatory Test 2, and the test profile ID for Optional Test 3.

[0164] The collected test information is output at operation 1040. Specifically, the GUI module may be configured to output the collected test information to another module (e.g., automation module 530, etc.). For example, the GUI module may be configured to provide the collected test profile IDs to the other module (e.g., in a compiled document, in a list, in computer-readable instructions, etc.).

[0165] 11 illustrates an example flow diagram of a method 1100 for generating a test scheme, according to one or more embodiments. Method 1100 may be performed by automation module 530 of FIG.

[0166] 11 , at operation S1110, test information is received. Specifically, an automation module (e.g., module 530) may be configured to receive test information from another module (e.g., GUI module 520). The test information may correspond to the collected test information provided at operation S1040 of FIG. 10.

[0167] At operation S1120, a test scheme is generated. Specifically, the automation module may be configured to generate the test scheme based on the received test information. For example, the automation module may obtain (from the test module or any suitable storage medium) test information (e.g., test type, test priority level, test urgency level, etc.) based on the test profile ID, configure or sequence how the tests should specifically be executed (e.g., which test should be executed first, which test variables should be adjusted, etc.), and then generate a test scheme that defines or specifies how the tests should specifically be executed.

[0168] As an example, assuming the received test information includes test profile IDs for mandatory test 1, mandatory test 2, and optional test 3, the automation module can request corresponding test profiles from a test module (e.g., module 540) based on the respective test profile IDs. Thus, the automation module can be configured to receive the test profile for mandatory test 1 (defining the test workflow and pipeline for mandatory test 1) and the test profile for mandatory test 2 (defining the test workflow and pipeline for mandatory test 2) from the mandatory test module (e.g., mandatory test module 541) of the test module, and the test profile for optional test 3 (defining the test workflow and pipeline for optional test 3) from the optional test module (e.g., optional test module 542) of the test module.

[0169] The automation module may then be configured to order the tests based on test category and / or test priority. For example, the automation module may determine that the mandatory tests (i.e., mandatory test 1 and mandatory test 2) should be executed before the optional test (i.e., optional test 3), and may further determine that mandatory tests with higher priority (e.g., more urgent, more important to system stability or security, consume fewer resources, etc.) should be executed first. Assuming that mandatory test 1 has a higher test priority than mandatory test 2, the automation module may determine that the three tests selected by the user should be executed in the following order: first, mandatory test 1, followed by mandatory test 2, and then optional test 3. The above are merely examples of possible embodiments, and it is contemplated that the tests may be ordered differently (e.g., mandatory test 1 and mandatory test 2 are executed simultaneously, optional test 3 should be executed first under certain conditions, etc.) without departing from the scope of the present disclosure.

[0170] In operation S1130, the test scheme is output. Specifically, the automation module may be configured to output the generated test scheme to another module (e.g., a test module, etc.).

[0171] 12 illustrates an example flow diagram of a method 1200 for managing one or more tests, according to one or more embodiments. Method 1200 may be performed by test module 310 of FIG. 3 or test module 540 of FIG. 5.

[0172] 12, at operation S1210, a test scheme is received. Specifically, the test module may be configured to receive a generated test scheme from another module (e.g., automation module 530). The received test scheme may correspond to the test scheme provided by the automation module at operation S1130 of FIG.

[0173] In operation S1220, one or more tests are executed. Specifically, the test module may be configured to execute the tests based on the received test scheme. For example, based on determining (from the test scheme) that mandatory test 1 should be executed first, followed by mandatory test 2, and then optional test 3, the test module (or the mandatory test module and optional test modules included therein) may be configured to schedule, execute, and record the tests according to the configurations and settings defined in the test scheme.

[0174] At operation S1230, the status of one or more tests is determined. Specifically, the test module may be configured to determine whether any of the one or more tests are complete. Based on determining that the test is complete, the process proceeds to operation S1250. Alternatively, based on determining that the test is not complete, the process proceeds to operation S1240.

[0175] At operation S1240, the status of the test (e.g., pending, completed, failed, elapsed time, etc.) is stored. Specifically, the test module (or each sub-module, such as the mandatory test module and optional test module) may be configured to monitor the status of each test and may be configured to continuously or periodically store the test status in a storage module (e.g., storage module 550).

[0176] Meanwhile, in operation S1250, a test report or log is stored. Specifically, the test module (or each sub-module, such as the mandatory test module and the optional test module) may be configured to generate or update a report or log for each executed test, and may be configured to store the report or log in a storage module upon detecting that the associated test has finished (e.g., completed, failed, rejected, etc.).

[0177] In an exemplary embodiment, the test module may be configured to store the test status in a first storage of the storage module and store the test report or log in a second storage of the storage module. It will further be understood that operations S1240 and S1250 may be performed simultaneously. As an example, assuming multiple tests (e.g., mandatory test 1 and mandatory test 2) are running simultaneously, based on determining that mandatory test 1 is complete but mandatory test 2 is not, the test module may be configured to store the test status of mandatory test 2 in the first storage of the storage module and store the test report / log of mandatory test 1 in the second storage of the storage module.

[0178] In view of the above, exemplary embodiments of the present disclosure provide a system and method for efficiently and effectively setting up or configuring one or more tests for a user. Specifically, the system and method of the exemplary embodiments enable a user to configure or customize one or more tests on the user's side according to requirements, and can automatically execute the one or more customized tests. Therefore, the multiple back-and-forth communications between multiple parties involved in configuring a test in the related art can be eliminated, simplifying the related process. Accordingly, the exemplary embodiments of the present disclosure effectively reduce the burden on users (e.g., test requesters and test implementers), improve the process efficiency of test customization, and simplify the process of test execution.

[0179] Furthermore, exemplary embodiments of the present disclosure provide systems and methods for efficiently and effectively managing test information, such as test status and / or test results / logs. Specifically, the systems and methods of the exemplary embodiments provide test status and / or test results in real time or near real time without the involvement of a test administrator, and automatically generate and provide test status and / or test results to a user upon request, such as providing test status and / or test results in aggregate for multiple tests. Thus, test status and / or test results can be provided in a timely and effective manner.

[0180] 13 is a diagram of an example environment 1300 in which the systems and / or methods described herein may be implemented. As shown in FIG. 13, environment 1300 may include a user device 1310, a platform 1320, and a network 1330. The devices in environment 1300 may be interconnected via wired connections, wireless connections, or a combination of wired and wireless connections. In embodiments, any of the functions and operations described with reference to FIGS. 1-12 above may be performed by any combination of elements shown in FIG. 13.

[0181] User device 1310 includes one or more devices capable of receiving, generating, storing, processing, and / or providing information related to platform 1320. For example, user device 1310 may include a computing device (e.g., a desktop computer, a laptop computer, a tablet computer, a handheld computer, a smart speaker, a server, etc.), a mobile phone (e.g., a smartphone, a wireless phone, etc.), a wearable device (e.g., smart glasses or a smart watch), or a similar device. In some implementations, user device 1310 may receive information from and / or transmit information to platform 1320.

[0182] Platform 1320 includes one or more devices capable of receiving, generating, storing, processing, and / or providing information. In some implementations, platform 1320 may include a cloud server or a group of cloud servers. In some implementations, platform 1320 may be designed to be modular, such that particular software components can be swapped in or out depending on particular needs. Thus, platform 1320 may be easily and / or quickly reconfigured for different uses.

[0183] In some implementations, as shown, platform 1320 may be hosted in a cloud computing environment 1322. In particular, although the implementations described herein describe platform 1320 as being hosted within cloud computing environment 1322, in some implementations platform 1320 may not be cloud-based (i.e., may be implemented outside of a cloud computing environment) or may be partially cloud-based.

[0184] Cloud computing environment 1322 includes an environment that hosts platform 1320. Cloud computing environment 1322 can provide services such as computation, software, data access, storage, etc., without requiring end-user (e.g., user device 1310) knowledge of the physical location and configuration of the system(s) and / or device(s) that host platform 1320. As shown, cloud computing environment 1322 can include a group of computing resources 1324 (collectively referred to as “computing resources 1324” and individually referred to as “computing resource 1324”).

[0185] Computing resources 1324 include one or more personal computers, clusters of computing devices, workstation computers, server devices, or other types of computing and / or communication devices. In some implementations, computing resources 1324 can host platform 1320. Cloud resources may include compute instances running within computing resources 1324, storage devices provided within computing resources 1324, data transfer devices provided by computing resources 1324, etc. In some implementations, computing resources 1324 may communicate with other computing resources 1324 via wired connections, wireless connections, or a combination of wired and wireless connections.

[0186] As further shown in FIG. 13, computing resources 1324 include a group of cloud resources such as one or more applications (“APP”) 1324-1, one or more virtual machines (“VM”) 1324-2, virtualized storage (“VS”) 1324-3, and one or more hypervisors (“HYP”) 1324-4.

[0187] Applications 1324-1 include one or more software applications that may be provided to or accessed by user device 1310. Applications 1324-1 may obviate the need to install and run software applications on user device 1310. For example, applications 1324-1 may include software associated with platform 1320 and / or any other software that may be provided via cloud computing environment 1322. In some implementations, one application 1324-1 may send information to or receive information from one or more other applications 1324-1 via virtual machine 1324-2.

[0188] Virtual machine 1324-2 includes a software-implemented machine (e.g., a computer) that executes programs like a physical machine. Virtual machine 1324-2 can be either a system virtual machine or a process virtual machine, depending on the application and the degree to which virtual machine 1324-2 represents an actual machine. A system virtual machine can provide a complete system platform that supports the execution of a complete operating system (“OS”). A process virtual machine can execute a single program and support a single process. In some implementations, virtual machine 1324-2 can run on behalf of a user (e.g., user device 1310) and manage the infrastructure of cloud computing environment 1322, such as data management, synchronization, or long-term data transfer.

[0189] Virtualized storage 1324-3 includes one or more storage systems and / or one or more devices that use virtualization technology within the storage systems or devices of computing resources 1324. In some implementations, in the context of storage systems, types of virtualization can include block virtualization and file virtualization. Block virtualization can refer to the abstraction (or separation) of logical storage from physical storage such that the storage system can be accessed regardless of the physical storage or heterogeneous structure. The separation can allow administrators flexibility in how they manage the storage for end users. File virtualization can eliminate the dependency between data accessed at the file level and where the file is physically stored. This can enable optimization of storage usage, server consolidation, and / or non-disruptive file migration.

[0190] The hypervisor 1324-4 may provide hardware virtualization technology that allows multiple operating systems (e.g., "guest operating systems") to run simultaneously on a host computer, such as computing resource 1324. The hypervisor 1324-4 may provide a virtual operating platform for the guest operating systems and may manage the execution of the guest operating systems. Multiple instances of different operating systems may share virtualized hardware resources.

[0191] Network 1330 may include one or more wired and / or wireless networks. For example, network 1330 may include a cellular network (e.g., a fifth-generation (5G) network, a long-term evolution (LTE) network, a third-generation (3G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., a public switched telephone network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, an optical fiber-based network, etc., and / or a combination of these or other types of networks.

[0192] The number and arrangement of devices and networks shown in Figure 13 are provided as an example. In practice, there may be additional, fewer, different, or differently arranged devices and / or networks than those shown in Figure 13. Furthermore, two or more devices shown in Figure 13 may be implemented within a single device, or a single device shown in Figure 13 may be implemented as multiple distributed devices. Additionally, or instead, a set of devices (e.g., one or more devices) of environment 1300 may perform one or more functions that are described as being performed by another set of devices of environment 1300.

[0193] Figure 14 is a diagram of example components of a device 1400. The device 1400 may correspond to the user device 1310 and / or the platform 1320 of Figure 13. As shown in Figure 14, the device 1400 may include a bus 1410, a processor 1420, a memory 1430, a storage component 1440, an input component 1450, an output component 1460, and a communication interface 1470.

[0194] The bus 1410 includes components that enable communication between the components of the device 1400. The processor 1420 may be implemented in hardware, firmware, or a combination of hardware and software. The processor 1420 may be a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or another type of processing component. In some implementations, the processor 1420 includes one or more processors that can be programmed to perform functions. The memory 1430 includes random access memory (RAM), read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, and / or optical memory) that stores information and / or instructions for use by the processor 1420.

[0195] The storage component 1440 stores information and / or software related to the operation and use of the device 1400. For example, the storage component 1440 may include a hard disk (e.g., a magnetic disk, optical disk, magneto-optical disk, and / or solid-state disk), a compact disk (CD), a digital versatile disk (DVD), a floppy disk, a cartridge, magnetic tape, and / or another type of non-transitory computer-readable medium, along with a corresponding drive. The input component 1450 includes components that enable the device 1400 to receive information, such as via user input (e.g., a touchscreen display, a keyboard, a keypad, a mouse, buttons, switches, and / or a microphone). Additionally or alternatively, the input component 1450 may include sensors for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, and / or an actuator). The output component 1460 includes components that provide output information from the device 1400 (e.g., a display, a speaker, and / or one or more light-emitting diodes (LEDs)).

[0196] The communication interface 1470 includes transceiver-like components (e.g., a transceiver and / or a separate receiver and transmitter) that enable the device 1400 to communicate with other devices, such as via a wired connection, a wireless connection, or a combination of wired and wireless connections. The communication interface 1470 may enable the device 1400 to receive information from another device and / or provide information to another device. For example, the communication interface 1470 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, etc.

[0197] Device 1400 may perform one or more processes described herein. Device 1400 may perform these processes in response to processor 1420 executing software instructions stored by a non-transitory computer-readable medium, such as memory 1430 and / or storage component 1440. A computer-readable medium is defined herein as a non-transitory memory device. A memory device includes memory space within a single physical storage device or memory space spread across multiple physical storage devices.

[0198] The software instructions may be loaded into memory 1430 and / or storage component 1440 from another computer-readable medium or from another device via communication interface 1470. When executed, the software instructions stored in memory 1430 and / or storage component 1440 may cause processor 1420 to perform one or more processes described herein.

[0199] Additionally, or instead, hardwired circuitry may be used in place of or in combination with software instructions to implement one or more processes described herein. Thus, implementations described herein are not limited to any specific combination of hardware circuitry and software.

[0200] The number and arrangement of components shown in Figure 14 are provided as an example. In practice, device 1400 may include additional, fewer, different, or differently arranged components than those shown in Figure 14. Additionally or alternatively, a set of components (e.g., one or more components) of device 1400 may perform one or more functions that are described as being performed by another set of components of device 1400.

[0201] In embodiments, any one of the operations or processes of FIGS. 1-12 may be performed by or using any one of the elements shown in FIGS.

[0202] Exemplary embodiments of the present disclosure provide methods and systems in which the enrollment and onboarding process for network services is simplified and automated, as opposed to manually onboarding network services on-site. A centralized enrollment and onboarding process can save time and resources, and automation, among other benefits, minimizes human error and compatibility issues, improves customer satisfaction, and reduces business risk for users (e.g., service providers). Furthermore, by automating the enrollment and onboarding of users (e.g., service providers) in a centralized and guided manner, online marketplace platforms provide customers (e.g., network operators) with an easy, flexible, cost-effective, efficient, and quickly implementable solution for network service selection. For users (e.g., service providers), such a network service enrollment and onboarding process provides a low-risk testing environment with minimal impact on customers (e.g., network operators) to provide high-quality services without imposing large demands on customers.

[0203] It is understood that the specific order or hierarchy of blocks in the processes / flowcharts disclosed herein is an example of an example approach. It is understood that the specific order or hierarchy of blocks in the processes / flowcharts may be rearranged based on design preferences. Furthermore, some blocks may be combined or omitted. Although the accompanying method claims present elements of various blocks in an example order, the elements of the blocks are not limited to the specific order or hierarchy shown.

[0204] Some embodiments may relate to systems, methods, and / or computer-readable media at any possible level of technical detail. Furthermore, one or more of the components described above may be implemented as instructions stored on a computer-readable medium and executable by at least one processor (and / or may include at least one processor). The computer-readable medium may include a computer-readable non-transitory storage medium(s) having computer-readable program instructions for causing a processor to perform operations.

[0205] A computer-readable storage medium may be any tangible device capable of retaining and storing instructions for use by an instruction-execution device. A computer-readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge-in-groove structures on which instructions are recorded, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted through wires.

[0206] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or storage device over a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface within each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage on a computer-readable storage medium within the respective computing / processing device.

[0207] The computer-readable program code / instructions for carrying out operations may be either source code or object code written in any combination of one or more programming languages, including assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, integrated circuit configuration data, or object-oriented programming languages ​​such as Smalltalk, C++, and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to personalize the electronic circuitry to perform an aspect or operation.

[0208] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, executing on the processor of the computer or other programmable data processing apparatus, produce means for implementing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions may also be stored on a computer-readable storage medium capable of directing a computer, programmable data processing apparatus, and / or other device to function in a particular manner, such that the computer-readable storage medium on which the instructions are stored comprises an article of manufacture containing instructions that implement aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0209] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to generate a computer-implemented process, such that the instructions executing on the computer, other programmable apparatus, or other device perform the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0210] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer-readable media according to various embodiments. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a particular logical function(s). The methods, computer systems, and computer-readable media may include additional, fewer, different, or differently arranged blocks compared to the blocks shown in the figures. In some alternative implementations, the functions noted in the blocks may occur in a different order than noted in the figures. For example, two blocks shown in succession may actually be executed concurrently or substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, can be implemented by a dedicated hardware-based system that performs the specified functions or actions or executes a combination of dedicated hardware and computer instructions.

[0211] It will be apparent that the systems and / or methods described herein may be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual specialized control hardware or software code used to implement these systems and / or methods is not intended to limit the implementation. Thus, the operation and behavior of the systems and / or methods are described herein without reference to specific software code, and it will be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

Claims

1. a first module, the first module comprising: memory storage storing computer executable instructions; a processor communicatively coupled to the memory storage, the processor executing the computer-executable instructions to cause the first module to: presenting to the user information regarding one or more tests associated with the service, including at least one mandatory test and at least one optional test; receiving a selection for the one or more tests from the user; collecting test information related to said user selection; The system is configured to output the collected test information.

2. the processor of the first module is configured to execute the computer-executable instructions to cause the first module to generate a graphical user interface (GUI) and present the information of the one or more tests via the GUI. The system of claim 1 .

3. 3. The system of claim 2, wherein the GUI includes one or more interactive elements associated with the one or more tests, and the processor of the first module is configured to execute the computer-executable instructions to cause the first module to receive the user's selections by determining one or more user interactions with the one or more interactive elements.

4. 4. The system of claim 3, wherein the one or more interactive elements are associated with a test profile ID, and the processor of the first module is configured to execute the computer-executable instructions to cause the first module to collect the test information related to the user's selection by collecting a test profile ID of the user's selected test.

5. further comprising a second module, the second module comprising: memory storage storing computer executable instructions; a processor communicatively coupled to the memory storage, wherein the processor executes the computer-executable instructions to cause the second module to: receiving the collected test information from the first module; generating a test scheme based on the collected test information; The system of claim 1 , configured to output the generated test scheme.

6. further comprising a third module, the third module comprising: memory storage storing computer executable instructions; a processor communicatively coupled to the memory storage, wherein the processor executes the computer-executable instructions to cause the third module to: receiving the generated test scheme from the second module; executing the one or more tests selected by the user based on the test scheme; The system of claim 5 , configured to store information related to the test.

7. further comprising a fourth module, the fourth module comprising: memory storage storing computer executable instructions; a processor communicatively coupled to the memory storage, wherein the processor executes the computer-executable instructions to cause the fourth module to: receiving the information related to the test from the third module; storing the received information; The system of claim 6 , configured to cause the first module to provide the stored information.

8. 1. A method executed by at least one processor, comprising: presenting, by a first module, information of one or more tests associated with the service to a user, the test information including at least one mandatory test and at least one optional test; receiving, by the one module, a selection for the one or more tests from the user; collecting, by the first module, test information related to the user's selection; and outputting, by the first module, the collected test information.

9. presenting the information for the one or more tests, generating, by the first module, a graphical user interface (GUI); and presenting, by the first module, the information for the one or more tests via the GUI.

10. 10. The method of claim 9, wherein the GUI includes one or more interactive elements associated with the one or more tests, and wherein receiving the user selection includes determining, by the first module, one or more user interactions with the one or more interactive elements.

11. 11. The method of claim 10, wherein the one or more interactive elements are associated with a test profile ID, and collecting the test information related to the user selection includes collecting, by the first module, a test profile ID for a test selected by the user.

12. receiving, by a second module, the collected test information from the first module; generating, by the second module, a test scheme based on the collected test information; and outputting, by the second module, the generated test scheme.

13. receiving, by a third module, the generated test scheme from the second module; executing, by the third module, the one or more tests selected by the user based on the test scheme; 13. The method of claim 12, further comprising storing, by the third module, information related to the test.

14. receiving, by a fourth module, the information related to the test from the third module; storing, by the fourth module, the received information; and The method of claim 13 further comprising providing, by the fourth module, the stored information to the first module.

15. A non-transitory computer-readable storage medium, comprising: presenting, by a first module, information of one or more tests associated with the service to a user, the test information including at least one mandatory test and at least one optional test; receiving, by the one module, a selection for the one or more tests from the user; collecting, by the first module, test information related to the user's selection; and outputting, by the first module, the collected test information.

16. presenting the information for the one or more tests, generating, by the first module, a graphical user interface (GUI); and presenting, by the first module, the information for the one or more tests via the GUI.

17. 17. The non-transitory computer-readable medium of claim 16, wherein the GUI includes one or more interactive elements associated with the one or more tests, and wherein receiving the user selection includes determining, by the first module, one or more user interactions with the one or more interactive elements.

18. 20. The non-transitory computer-readable storage medium of claim 17, wherein the one or more interactive elements are associated with a test profile ID, and collecting the test information related to the user selection includes collecting, by the first module, a test profile ID for a user-selected test.

19. The method comprises: receiving, by a second module, the collected test information from the first module; generating, by the second module, a test scheme based on the collected test information; and outputting, by the second module, the generated test scheme.

20. The method comprises: receiving, by a third module, the generated test scheme from the second module; executing, by the third module, the one or more tests selected by the user based on the test scheme; 20. The non-transitory computer-readable medium of claim 19, further comprising: storing, by the third module, information related to the test.

Images