Electronic apparatus providing testbed and method therefor

Abstract

This invention discloses an electronic device and method for providing a test bench. According to one disclosed embodiment, the test bench provision method implemented by the electronic device may include the following steps: receiving a test bench generation request for a specific application-based test target scenario; identifying a plurality of services associated with the test target scenario; and generating a test bench that sets whether the features of the plurality of services have changed or not, and the test paths between the plurality of services, for each of the plurality of features that may change according to the test in the test target scenario.

Description

[Technical Field]

[0001] This invention relates to an electronic device and method for providing a test environment suitable for end-to-end (E2E) testing. [Previous Technology]

[0002] To evaluate the complete process of an application from start to finish, E2E testing is required. Through E2E testing, developers can simulate real-world scenarios and verify the interaction between various components and the system. These components include the user interface, API (Application Programming Interface), and databases.

[0003] However, E2E testing is complex to design and maintain, and requires significant resources. Specifically, due to the fragmented implementation of localized testing within existing modular testing environments, the following problems exist: If E2E tests with long call paths are implemented, a considerable amount of code needs to be modified. Furthermore, most E2E tests require large test teams and excessive manpower to prepare the test environment. In addition, most test programs are implemented manually and require collaboration between multiple teams, further increasing resource investment.

[0004] In this regard, see previous literature such as KR1267836B1. [Summary of the Invention]

[0005] [Problem to be Solved by the Invention] The present invention is proposed to solve the above-mentioned problem and aims to provide an integrated E2E testing environment, thereby reducing the resources used for E2E testing environment preparation and E2E testing execution, and minimizing code modifications during E2E testing.

[0006] The technical problem to be solved by the present invention is not limited to the technical problem described above, and other technical problems can be deduced by analogy based on the following embodiments. [Technical Means for Solving the Problem]

[0007] The test bench providing method of the present invention may include the following steps: receiving a test bench generation request for a specific test target scenario with an application; identifying a plurality of services associated with the test target scenario; and generating a test bench that sets whether the features of the plurality of services are changed or not, and the test paths between the plurality of services, for each of the plurality of features that can be changed according to the test in the test target scenario.

[0008] In one embodiment, the above-mentioned test target scenario can be specified by selecting a source webpage implemented within the product sales service application, the source webpage including a Search Result Page (SRP) and a Single Detail Page (SDP).

[0009] In one embodiment, the above-mentioned test bench providing method may further include the following steps: receiving the setting modification request of the test bench; and based on the setting modification request, modifying whether the characteristics of each of the plurality of services are changed, and at least a portion of the test paths among the plurality of services.

[0010] In one embodiment, the above-described test bench providing method may further include the following steps: along with generating the test bench, providing a dedicated URL (Uniform Resource Locator) that can access the test bench. Furthermore, the dedicated URL may be generated according to features that can change based on the test in the target test scenario, and the string constituting the dedicated URL may also include the name of the feature corresponding to the dedicated URL.

[0011] In one embodiment, the above-mentioned test bench providing method may further include the following steps: receiving the execution request of the above-mentioned test target scenario; executing the above-mentioned test target scenario under the above-mentioned test bench; and providing the execution result of the above-mentioned test target scenario.

[0012] As a related example, the steps for executing the above-mentioned test target scenario may include the following steps: executing a basic image corresponding to the released version of the application for a service whose features have not changed according to the above test; and executing a test image reflecting the changed features to the basic image for a service whose features have changed according to the above test. In this case, the basic image may be collected and stored by a distributed streaming platform in response to the above-mentioned execution request. Furthermore, the steps for executing the above-mentioned test image may also include the following steps for services among the above-mentioned plurality of services that are set up in an A / B testing manner: transmitting the above-mentioned basic image to the first pod and the second pod; executing the above-mentioned basic image using the first-1 pod within the first pod; executing the first test image with the features of version A using the first-2 pod within the first pod; executing the above-mentioned basic image using the second-1 pod within the second pod; and executing the second test image with the features of version B using the second-2 pod within the second pod.

[0013] On the other hand, as another example, the above execution result may be as follows: In a dashboard, the difference in performance parameters before and after the test corresponding to the feature changes of each of the above plurality of services is displayed separately. For the services among the above plurality of services that have undergone A / B testing, the difference in performance parameters between the group applying the features of version A and the group applying the features of version B is displayed separately. Specifically, the above execution result may further include a plot, which is displayed in the dashboard separately according to a plurality of types for the execution results during the test period. In this case, the plurality of types displayed in the plot may include at least a portion of test completed, no difference before and after test, difference before and after test, and test failure due to error.

[0014] In one embodiment, the method for providing the test bench may further include the steps of: receiving an adjustment input for the validity period of the test bench; and adjusting the validity period based on the adjustment input. Relatedly, when the validity period has elapsed since the test bench was created, external access to the test bench will be blocked.

[0015] According to the present invention, the electronic device providing the test bench may include a memory for storing commands and a processor, wherein the processor is connected to the memory to receive a test bench generation request for a specific test target scenario with an application; identifies a plurality of services associated with the test target scenario; and generates a test bench that sets whether the features of each of the plurality of services are changed or not, and the test paths between the plurality of services, for each of the plurality of features that may change according to the test in the test target scenario.

[0016] Specific details of other embodiments are included in the detailed description and drawings. [Effects of the Invention]

[0017] According to the present invention, an integrated E2E test bench that combines multiple services associated with a test target scenario can be provided, thereby minimizing code modifications even if the features accompanying the test change.

[0018] Furthermore, according to the present invention, when generating the E2E test bench, only the test path between services is set, and the constraint of whether the features between services are compatible is eliminated. In this way, the resources required for collaboration between test groups or teams can be greatly reduced in the step of constructing the test environment.

[0019] The effects of the invention are not limited to those mentioned above. Other effects not mentioned can be clearly understood by those skilled in the art based on the description in the claims of the invention.

Implementation Method

[0021] Hereinafter, specific embodiments will be described with reference to the drawings. The following detailed description is provided to aid in a comprehensive understanding of the methods, apparatus and / or systems described in this specification. However, it is merely an example, and the disclosed embodiments are not limited thereto.

[0022] When describing the embodiments, if it is determined that a detailed description of relevant well-known descriptions may unnecessarily obscure the essence of the disclosed embodiments, such detailed descriptions will be omitted. Furthermore, the terminology used below is defined in consideration of the functionality in the disclosed embodiments, and such terminology may vary depending on the intentions or practices of users, operators, etc. Therefore, its definition should be based on the overall content of this specification. The terminology used in the detailed description is only for describing the embodiments and should not be limited. Unless explicitly used otherwise, the singular form includes the meaning of the plural form. In this description, expressions such as "comprising" or "possessing" are used to refer to a portion or combination of certain features, numbers, steps, actions, elements, etc., and should not be construed as excluding the presence or possibility of one or more other features, numbers, steps, actions, elements, etc., other than those described.

[0023] The terminology used in the embodiments is selected as widely used common terms as possible to consider the functions of this invention, but may vary depending on the intentions or precedents of those skilled in the art, the emergence of new technologies, etc. Furthermore, in certain situations, there may be terms arbitrarily selected by the applicant; in such cases, their meanings will be described in detail in the corresponding explanatory section. Therefore, the terminology used in this invention should be defined based on the meaning of the term and the entire content of this invention, and not simply based on the name of the term.

[0024] Throughout this specification, when a part is described as "including" a certain component, it means that, unless otherwise specified, other components may be included, rather than excluding other components. Furthermore, terms such as "...part" and "...module" used in the specification refer to a unit that processes at least one function or action, which may be implemented by hardware or software, or by a combination of both. Unlike the examples shown, specific actions may not be explicitly distinguished.

[0025] The expression “at least one of a, b and c” as recorded throughout the specification may include “a alone”, “b alone”, “c alone”, “a and b”, “a and c”, “b and c”, or “a, b and c”.

[0026] The "terminal" mentioned below can be a computer or a portable terminal that can connect to a server or other terminal via a network. Here, a computer includes, for example, a laptop computer, a desktop computer, or a laptop computer equipped with a web browser. A portable terminal is a wireless communication device that ensures portability and mobility, and may include, for example, IMT (International Mobile Telecommunication), CDMA (Code Division Multiple Access), W-CDMA (W-Code Division Multiple Access), LTE (Long Term Evolution) and other communication terminals, as well as all types of handheld wireless communication devices such as smartphones and tablets.

[0027] In the following description, the terms “transmission”, “communication”, “sending”, “receiving” and other terms with similar meanings for signals or information include not only the direct transmission of signals or information from one component to another, but also the transmission of signals or information to another component through other components.

[0028] In particular, "transmitting" or "sending" a signal or information to a constituent element means indicating the final destination of the signal or information, and not the immediate destination. The same applies to "receiving" a signal or information. Furthermore, in this specification, "related" two or more pieces of data or information means that if one piece of data (or information) is obtained, at least a portion of the other data (or information) can be obtained based on it.

[0029] Furthermore, terms such as '1' and '2' can be used to describe various constituent elements, but the constituent elements mentioned above should not be limited to the terms mentioned above. The terms mentioned above are used for the purpose of distinguishing one constituent element from other constituent elements.

[0030] For example, without departing from the scope of the present invention, the first constituent element may be named the second constituent element, and similarly, the second constituent element may be named the first constituent element.

[0031] In describing the embodiments, descriptions of technical content well-known in the technical field to which this invention pertains and not directly related to this invention are omitted. The reason for this is that by omitting redundant descriptions, the main idea of ​​this invention is clearly conveyed, thereby avoiding confusion regarding the main idea of ​​this invention.

[0032] For the same reason, some constituent elements are exaggerated, omitted, or approximated in the accompanying drawings. Furthermore, the dimensions of each constituent element do not fully reflect the actual dimensions. In each drawing, the same or corresponding constituent elements are assigned the same reference number.

[0033] The advantages and features of the present invention, as well as the methods for achieving such advantages and features, will become clear upon reference to the accompanying drawings and the embodiments described in detail below. However, the present invention is not limited to the embodiments disclosed below and can be implemented in various different forms. These embodiments are provided only to fully disclose the present invention and to enable those skilled in the art to fully understand the scope of the invention. The present invention is defined solely by technical solutions. Throughout this specification, the same reference numerals refer to the same constituent elements.

[0034] It is understood that the combination of blocks in a flowchart and the flowchart itself can be executed by computer program instructions. These computer program instructions can be loaded onto the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing equipment. Therefore, the instructions executed by the processor of the computer or other programmable data processing equipment produce methods for performing the functions described in the blocks of the flowchart. To implement the functions in a specific manner, these computer program instructions can be stored in computer-usable or computer-readable memory that is accessible to a computer or other programmable data processing equipment. Therefore, the instructions stored in the computer-usable or computer-readable memory can also produce manufactured items containing instruction methods that perform the functions described in the blocks of the flowchart. The computer program instructions can also be loaded onto a computer or other programmable data processing equipment. Therefore, a series of action steps are performed on the computer or other programmable data processing equipment to produce a process executed by the computer. Thus, the instructions executed by the computer or other programmable data processing equipment can also provide steps for performing the functions described in the blocks of the flowchart.

[0035] Furthermore, each block may represent a module, segment, or portion of code comprising one or more executable instructions for performing a specific logical function. It should also be noted that in some alternative execution examples, the functions mentioned in the blocks may be generated out of order. For example, two blocks represented successively may be executed simultaneously, or occasionally in reverse order depending on their corresponding functions.

[0036] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement the invention. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. The methods shown in Figures 2 to 5 can be implemented, for example, by using the electronic device 110 described with reference to Figure 8. In the present invention, the electronic device 110 serves as an exemplary device for providing a test bench. This device may be a single device, but according to embodiments, it may also be a system comprising a plurality of detailed devices that communicate with each other via wired or wireless networks.

[0037] Figure 1 is a diagram of a system for providing a test bench according to an embodiment.

[0038] Referring to FIG1, the system may include at least a portion of electronic device 110, user terminal 120, and network 130. The system shown in FIG1 only illustrates the components relevant to this embodiment. Therefore, those skilled in the art related to this embodiment will understand that, in addition to the components shown in FIG1, other common components may be included.

[0039] Electronic device 110 is a device that provides various information. Upon receiving a request from user terminal 120, electronic device 110 can generate a testbed constituting a test environment, modify settings related to the testbed, execute, interrupt, or terminate tests on the testbed, or remove the generated testbed. In this invention, a "testbed" is an environment or setting specifically designed for testing and evaluating the performance, functionality, or stability of a system, component, or software, and can support the infrastructure, tools, and conditions required for conducting tests. For example, a testbed may include the tools, frameworks, and components required for testing software applications under various operating systems, databases, and network conditions.

[0040] In one embodiment, to process test-related requests, electronic device 110 may utilize tools provided in a container orchestration logic platform (e.g., Kubernetes, Docker Swarm) to virtually execute an application reflecting the characteristics to be tested. Alternatively, to process test-related requests, electronic device 110 may utilize a distributed streaming platform, event monitoring and alerting applications, or visualization applications.

[0041] In one embodiment, the electronic device 110 may be a physical server, but according to an embodiment, it may also refer to a collection of multiple servers that communicate with each other electrically. Furthermore, data collected, processed, or generated by the electronic device 110 during the processing of test requests may be stored in the storage space within the electronic device 110, but according to an embodiment, it may also be stored in a separate medium (e.g., a cloud database) connected to the electronic device 110.

[0042] The user terminal 120 is a device that receives user input through an input interface, transmits test-related requests to the electronic device 110, and receives responses corresponding to the requests from the electronic device 110 and provides them to the user through an output interface.

[0043] According to one embodiment, the user terminal 120 may support software that provides users with a program library and command-line tools for inputting test-related requests. For example, the user terminal 120 may support services such as cURL and POSTMAN, thereby guiding users to more easily input test-related requests. The user terminal 120 may be a portable terminal providing an active environment (e.g., a smartphone, tablet, etc.) or a client supporting a web browser, and the electronic device 110 may provide a test bench in a format corresponding to the user terminal 120 to the user terminal 120.

[0044] According to one embodiment, user terminal 120 can transmit commands written using Gradle tools to electronic device 110, such commands being used to create an application, generate a test bench, or perform a test. Conversely, electronic device 110 can provide user terminal 120 with responses corresponding to the respective commands via software that supports Gradle tools.

[0045] Electronic device 110 and user terminal 120 can communicate via network 130 to send and receive information related to the generation, modification, and deletion of test equipment or information related to the execution, interruption, resumption, and termination of tests. Network 130 may include Local Area Network (LAN), Wide Area Network (WAN), Value Added Network (VAN), Mobile Radio Communication Network, Satellite Communication Network, and combinations thereof. This network is a comprehensive data communication network that enables the constituent elements of the networks shown in Figure 1 to communicate smoothly with each other, including wired internet, wireless internet, and mobile wireless communication networks. Wireless communication may include, but is not limited to, wireless LAN (Wi-Fi), Bluetooth, Bluetooth Low Energy, Wi-Fi Direct (WFD), UWB (ultra-wideband), infrared communication (IrDA), and NFC (Near Field Communication).

[0046] Figure 2 is a flowchart illustrating a method for providing a test bench according to an embodiment. Hereinafter, the method shown in Figure 2 will be described as being implemented using the aforementioned electronic device 110. However, this is merely an example; the method shown in Figure 2 can be implemented using other devices, or using a combination of the electronic device 110 and individual devices. The same applies to the description of the methods shown in Figures 3 to 5 below.

[0047] In step S210, the electronic device 110 can receive a request generated by the test bench of a specific test target scenario with an application.

[0048] According to one embodiment, as a user selects a specific source webpage from a pre-existing source webpage option via user terminal 120, electronic device 110 can specify a test target scenario. As an example, the test target scenario can be specified by selecting a source webpage implemented within a product sales service application, which includes a Search Result Page (SRP) and a Single Detail Page (SDP).

[0049] Specifically, the electronic device 110 can provide pre-existing source webpage options via a drop-down menu through the user terminal 120. If the user selects a specific source webpage, a specific test target scenario is mapped to the selected source webpage. At this time, the source webpage and the test target scenario can have a one-to-one mapping relationship, but depending on the situation, different source webpages can also be mapped to the same test target scenario.

[0050] In this invention, the "test target scenario" may include settings for the types of features that can be changed during the test, the service types that support each feature, and the test paths between services. For example, when the user selects an SRP, the electronic device 110 may specify a test target scenario in the SRP. This specific test target scenario is specifically configured with features x, y, and z that can be changed according to the test, and services a, b, c, and d that are required to provide item retrieval results. Test paths are set for services a to d for each feature.

[0051] In step S220, the electronic device 110 can identify a plurality of services associated with the test target scenario. For example, the electronic device 110 can refer in advance to a mapping table of test target scenarios and services to identify the services associated with the specific test target scenario in step S210. The corresponding mapping table can be stored in the electronic device 110, but according to the embodiment, it can also be stored in a database in the electronic device 110 with access rights.

[0052] In step S230, the electronic device 110 can generate a test bench that sets whether the features of the multiple services have changed and the test paths between the multiple services for each of the multiple features that can be changed according to the test in the test target scenario.

[0053] According to one embodiment, after temporarily generating a test bench, the electronic device 110 can provide information on whether the characteristics of each of the currently set plurality of services have changed, and the test paths between the plurality of services, to the user terminal 120. Based on input from the user terminal 120, it can modify whether the characteristics of at least some services have changed, or modify at least some test paths. Alternatively, if a final request for generating the test bench is received from the user terminal 120, the test bench can be generated according to the currently set parameters.

[0054] According to one embodiment, as a test bench is generated, the electronic device 110 can provide a directory of files (sets) corresponding to the generated test bench to the user terminal 120.

[0055] According to one embodiment, when a user requests to modify the settings of the test bench through the user terminal 120, the electronic device 110 can receive the setting modification request and, based on the setting modification request, modify at least a portion of the parameters belonging to the test bench (e.g., whether the characteristics of each of the plurality of services have changed or the test path between the plurality of services).

[0056] According to one embodiment, as a test bench is generated, the electronic device 110 can provide a dedicated URL (Uniform Resource Locator) accessible to the test bench to the user terminal 120. For example, the electronic device 110 can generate dedicated URLs for features that can change according to the test in the target test scenario. A dedicated URL generated in this manner may include a string containing the name of the feature corresponding to the dedicated URL. In this case, the name of the feature included in the dedicated URL may include at least a portion of the names of the sub-features constituting the corresponding feature. On the other hand, the electronic device 110 can identify the identifier of the account or the identifier of the machine IP that attempts to connect to the reference location via the dedicated URL. If the identified identifier is one of the identifiers allowed to connect to the test bench, it connects the identified identifier to the platform providing the test bench; if the identified identifier does not exist in the list of identifiers allowed to connect to the test bench, it blocks the connection of the corresponding account or machine. The electronic device 110 can configure accounts or machines allowed to connect to the test bench as follows: whether they can execute the test target scenario (execution permission), whether they can modify the test bench settings (modification permission), and whether they can receive the execution results of the test target scenario (viewing permission), etc. These permission settings can be set based on the account or machine identifier when the test bench is created, or based on a dedicated URL.

[0057] According to one embodiment, as the user terminal 120 receives an adjustment input regarding the validity period of the test station, the electronic device 110 can adjust the validity period of the test station based on the received adjustment input. When a preset or subsequently adjusted validity period has elapsed since the test station was created, the electronic device 110 can block external access to the test station. Furthermore, for accounts or machines providing a dedicated URL corresponding to the test station, if the validity period of the corresponding test station has elapsed, the electronic device 110 can transmit a message indicating that access to the corresponding test station has been blocked.

[0058] According to one embodiment, when the expiration time of the preset validity period of the test station is close to a fixed number of days or a fixed time, the electronic device 110 may request the user terminal 120 that has requested the generation of the test station to input the adjustment of the validity period of the test station.

[0059] Figure 3 is a flowchart illustrating the method of providing a test bench according to an additional embodiment.

[0060] In step S310, the electronic device 110 may receive a request generated by the test bench of a specific test target scenario with an application.

[0061] In step S320, the electronic device 110 can identify a plurality of services associated with the test target scenario.

[0062] In step S330, the electronic device 110 can generate a test bench that sets whether the features of the multiple services have changed and the test paths between the multiple services for each of the multiple features that can be changed according to the test in the test target scenario.

[0063] In step S340, the electronic device 110 may receive an execution request for the test target scenario.

[0064] In step S350, the electronic device 110 can execute the test target scenario under the test bench.

[0065] In step S360, the electronic device 110 can provide the execution results of the test target scenario.

[0066] According to one embodiment, the electronic device 110 can provide the execution results of the test target scenario in the form of a dashboard displayed by the user terminal 120. Specifically, the electronic device 110 can control the user terminal 120 in such a way that, as the execution results of the test target scenario, the dashboard can distinguish and display the difference in performance parameters before and after the test corresponding to feature changes of each of the plurality of services. In particular, for services that have undergone A / B testing among the plurality of services, the difference in performance parameters between groups applying features of version A and groups applying features of version B can be distinguished and displayed separately. At this time, the difference in performance parameters before and after the test corresponding to feature changes, and the difference in performance parameters between groups corresponding to A / B tests can be distinguished and displayed separately in one dashboard. The so-called "difference in performance parameters" refers to the difference in performance parameters under the following conditions: all variables except for the change of the benchmark feature or the difference between the A / B test groups are controlled.

[0067] According to one embodiment, the electronic device 110 can control the user terminal 120 in the following manner: as the execution result of the test target scenario, it displays a plot on the dashboard, which distinguishes the execution results within the test time period according to a plurality of types (at this time, the y-axis of the plot may be the number of features executed according to the service path). At this time, the plurality of types may include at least a portion of test completion, no difference before and after the test, difference before and after the test, and test failure due to error. Relatedly, when the difference in performance parameters before and after the test is above a set threshold, the electronic device 110 can determine that there is a difference before and after the test; if the threshold is not reached, it is determined that there is no difference before and after the test. Furthermore, when an error belonging to a preset error list occurs during the execution of a test image reflecting the changed features to the basic image, the electronic device 110 can determine that it is a test failure due to an error. Furthermore, the electronic device 110 can also provide the user terminal 120 with information on the cause of failure for the execution result corresponding to the test failure caused by the error.

[0068] Figure 4 is a flowchart illustrating an embodiment of executing a test target scenario under a test bench.

[0069] When performing tests in a test target scenario, the electronic device 110 can distinguish between services whose characteristics have not been changed (remained) according to the test and services that have been changed, thereby making the images executed in the container different.

[0070] In step S410, the electronic device 110 can execute a basic image corresponding to the published version of the application for services whose characteristics have not been changed according to the test.

[0071] According to one embodiment, the electronic device 110 can request the collection and storage of basic images from a distributed streaming platform (e.g., Kafka) in response to the execution request of the test target scenario, and can transmit the basic images from the corresponding distributed streaming platform to a preset container (or pod) and execute them.

[0072] In step S420, the electronic device 110 can execute a test image that reflects the changed features to the basic image for the service of changing features according to the test.

[0073] According to one embodiment, the electronic device 110 can compare and analyze the execution results of the basic image and the execution results of the test image to generate the execution results of the test target scene.

[0074] On the other hand, for services set up in a manner that implements A / B testing among a plurality of services, the electronic device 110 can distinguish between pods (container sets, pods) that implement A testing and pods that implement B testing, and execute the basic image and test image in each pod respectively. The specific process related to this will be explained below with reference to FIG5.

[0075] Figure 5 is a flowchart illustrating an embodiment of executing a test image in the embodiment of Figure 4. Specifically, Figure 5 relates to an embodiment in which electronic device 110 executes a base image and a test image as a Docker image in a preset container set (pod) using Kubernetes.

[0076] In step S510, the electronic device 110 can transmit the basic image to the first pod and the second pod.

[0077] In step S520, the electronic device 110 may use the first-1 pod within the first pod to execute the basic image.

[0078] In step S530, the electronic device 110 can use the first-second pods in the first pod to execute the first test image with the features of version A.

[0079] In step S540, the electronic device 110 may use the second-1 pod within the second pod to execute the basic image.

[0080] In step S550, the electronic device 110 may use the second-2nd pod within the second pod to execute the second test image with the features of version B.

[0081] In Figure 5, although steps S520 and S530 are shown in an earlier order than steps S540 and S550, this is only for the purpose of explanation. According to the embodiment, steps S520 and S530 may also be performed in parallel with steps S540 and S550, or in a later order.

[0082] According to one embodiment, the electronic device 110 can compare the execution result of the base image with the execution result of the first test image in steps S520 and S530 to calculate the difference in performance parameters when the features of version A are applied, and compare the execution result of the base image with the execution result of the second test image in steps S540 and S550 to calculate the difference in performance parameters when the features of version B are applied. Afterwards, the electronic device 110 can compare and analyze the difference in performance parameters when the features of version A are applied with the difference in performance parameters when the features of version B are applied, thereby generating the execution result of the AB test.

[0083] Figure 6 is an illustrative conceptual diagram of executing a test target scenario under a test bench. The test bench represents the characteristics of multiple services associated with the test target scenario and the test paths between the multiple services.

[0084] Referring to Figure 6, the user (engineer) can request the generation of a test bench via the user terminal 120. When generating the request, the user can use the A / B flag to specify the features to be performed in the A / B test, and can specify the feature names included in the string of the dedicated URL.

[0085] On the other hand, other users can access the test console by requesting a dedicated URL in the transmission platform (e.g., cURL, POSTMAN, mobile application software, web browser) to request the execution of the target test scenario. At this time, since each of the features (feature 1, feature 2-A, feature 2-B, feature 3) that can be changed according to the test is assigned a header, the test path can also be identified between different services (modular-list, search-api, materialization-api, materialization-service)). Here, feature 2-A and feature 2-B represent features that are versioned according to A / B testing.

[0086] Referring again to Figure 6, in the Module-Manifestation Service, Features 1, 2-A, 2-B, and 3 are all changed based on testing. In the Retrieval-API Service, Features 1 and 3 will change based on testing, but Feature 2 remains unchanged. Furthermore, in the Concretization-API Service, Feature 2 will be changed to Feature 2-A and Feature 2-B respectively based on testing, but Features 1 and 3 remain unchanged. In the Concretization-API Service, although testing has been performed, all features remain unchanged.

[0087] Thus, by implementing a test bench with pre-defined test paths between different services and whether or not features have changed, for example, even if the code related to feature 2 is modified in the retrieval-api service, it will not affect the testing in other services. Similarly, even if the code related to any feature is modified in the materialization-service, testing can be performed without any problems in other services. Furthermore, during the process of modifying code or generating the test bench, the engineering teams managing each service do not need to collaborate, thereby reducing the labor required when designing the test bench.

[0088] Figure 7 is an illustrative conceptual diagram showing a test image of a service set up in an A / B testing manner.

[0089] As shown in FIG7, the electronic device 110 can collect and store Docker images (hereinafter, "basic images") corresponding to the corresponding application from the pod of the product (application) that is actually released via a distributed streaming platform. For example, the electronic device 110 can collect and store basic images driven in a web server container supported by the Ngnix Sidecar tool.

[0090] Subsequently, the electronic device 110 can transmit the basic image from the distributed streaming platform to the pressure source pods of the first and second pods, and then transmit the basic image from the pressure source pods to the first-1 and second-1 pods. On the other hand, the electronic device 110 can transmit a test image obtained by reflecting the changed features to the basic image to the first-2 and second-2 pods. At this time, the test image transmitted to the first-2 pod can be the first test image with features of version A, and the test image transmitted to the second-2 pod can be the second test image with features of version B.

[0091] Subsequently, the electronic device 110 can transmit the following analysis results to the event monitoring and warning application: analysis results comparing the execution results of the basic image in pod 1-1 with the execution results of the first test image in pod 1-2, and analysis results comparing the execution results of the basic image in pod 2-1 with the execution results of the second test image in pod 2-2. Then, the electronic device 110 can extract events occurring due to the AB test based on the analysis results using the event monitoring and warning application, and send a warning (notification) to the user terminal when an event that meets a preset benchmark occurs. Afterwards, the electronic device 110 can distinguish the performance parameter differences between groups applying features of version A and groups applying features of version B based on the extracted events using a visualization application, thereby providing the results to the user terminal in the form of a dashboard.

[0092] The operation of the above-mentioned electronic device 110 can be achieved by commands generated in the business environment, including the software and development tools provided by the user terminal.

[0093] Figure 8 is a block diagram showing an electronic device according to an embodiment.

[0094] According to one embodiment, the electronic device 110 may include a memory 111 and a processor 113. The electronic device 110 shown in FIG8 only illustrates components relevant to this embodiment. Therefore, those skilled in the art related to this embodiment will understand that, in addition to the components shown in FIG8, other common components may be included. According to the embodiment, the electronic device 110, as a device for implementing wired / wireless communication, may further include a transceiver (not shown) capable of communicating with an external electronic device. The external electronic device may be a terminal or a server. Furthermore, the communication technologies utilized by the transceiver may include GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), LTE (Long Term Evolution), 5G, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Bluetooth™, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), ZigBee, NFC (Near Field Communication), etc. In this embodiment, the transceiver may be included in the communication device.

[0095] The processor 113 can control the overall operation of the electronic device 110 and process data and signals. In an embodiment, the processor 113 may be included in a controller. The processor 113 may include at least one hardware unit. Furthermore, the processor 113 may operate by executing one or more software modules generated by executing program code stored in memory 111. The processor 113 may include memory, and the processor 113 can execute program code stored in memory to control the overall operation of the electronic device 110 and process data and signals.

[0096] The processor 113 can receive a test bench generation request for a specific test target scenario with an application; identify a plurality of services associated with the test target scenario; and generate a test bench that sets whether the features of the plurality of services have changed and the test paths between the plurality of services for each of the plurality of features that can be changed according to the test in the test target scenario.

[0097] The electronic device of the above embodiments may include a processor, memory for storing and executing program data, permanent storage such as a disk drive, a communication port for communicating with external devices, and a user interface device such as a touch panel, key, or button. Computer-readable code or program commands that can be executed on the processor can be stored on a computer-readable recording medium by means of software modules or algorithms. Here, computer-readable recording media include magnetic storage media (e.g., ROM (read-only memory), RAM (random-access memory), floppy disks, hard disks, etc.) and optical reading media (e.g., CD-ROM, DVD). The computer-readable recording medium is distributed across a network-connected computer system, thereby enabling distributed storage and execution of computer-readable code. The medium can be read by a computer, stored in memory, and executed in a processor.

[0098] This embodiment can be represented by functional blocks and various processing steps. These functional blocks can be implemented by different numbers of hardware and / or software components performing specific functions. For example, the embodiment can employ integrated circuits capable of performing various functions via the control of one or more microprocessors or other control devices, such as memory, processing, logic, lookup tables, etc. The constituent elements can be executed by software programming or software components. Similarly, this embodiment includes algorithms implemented by combinations of data structures, programs, conventions, or other programming structures, and therefore can be implemented using programming or scripting languages ​​such as C, C++, Java, and assemblers. Functionally, it can be implemented by algorithms executed in one or more processors. Furthermore, this embodiment can employ prior art for electronic environment setup, signal processing, and / or data processing. Terms such as "mechanism," "component," "structure," and "construction" are used broadly and are not limited to mechanical or physical structures. The above terms may be associated with processors and include the meaning of a series of software processes.

[0099] The above embodiments can be implemented as artificial intelligence (AI) using the processor and memory of an electronic device. The processor may include one or more processors, which may be general-purpose processors such as CPUs, APs, and DSPs (digital signal processors), graphics-specific processors such as GPUs and VPUs (vision processing units), or AI-specific processors such as NPUs. One or more processors can be controlled to process input data according to predefined action rules or AI models stored in memory. Alternatively, when one or more processors are AI-specific processors, the AI-specific processors can be designed with hardware specifically for processing specific AI models.

[0100] The predefined action rules or artificial intelligence models are characterized by being created through learning. Here, "created through learning" means creating predefined action rules or artificial intelligence models in such a way that a basic artificial intelligence model is learned using a learning algorithm with a large amount of learning data, thereby implementing the desired characteristics (or, purpose). This learning can be implemented in the electronic device implementing artificial intelligence of the present invention itself, or it can be implemented by a separate server and / or system. Examples of learning algorithms may include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the above examples.

[0101] An artificial intelligence model may include a plurality of neural network layers. Each of the plurality of neural network layers may have a plurality of weight values, and neural network operations are performed by operations between the results of previous layers and the plurality of weight values. The plurality of weight values ​​of the plurality of neural network layers can be optimized by the learning results of the artificial intelligence model. For example, the plurality of weight values ​​can be updated in a way that reduces or minimizes the loss or cost values ​​obtained from the artificial intelligence model during the learning process. Artificial neural networks may include deep neural networks (DNN), convolutional neural networks (CNN), recurrent neural networks (RNN), restricted Boltzmann machines (RBM), deep belief networks (DBN), bidirectional recurrent deep neural networks (BRDNN), or deep Q-networks, but are not limited to the examples mentioned above.

[0102] The above embodiment is only an example, and other embodiments can be implemented within the scope of the technical solutions described below. [Simplified Explanation of the Diagram]

[0020] Figure 1 is a diagram illustrating a system for providing a test bench according to an embodiment. Figure 2 is a flowchart illustrating a method for providing a test bench according to an embodiment. Figure 3 is a flowchart illustrating a method for providing a test bench according to an additional embodiment. Figure 4 is a flowchart illustrating an embodiment of executing a test target scenario under a test bench. Figure 5 is a flowchart illustrating an embodiment of executing test images in the embodiment of Figure 4. Figure 6 is an exemplary conceptual diagram illustrating the execution of a test target scenario under a test bench, which represents a test bench with multiple services associated with the test target scenario, including whether the features of each service have changed and the test paths between the multiple services. Figure 7 is an exemplary conceptual diagram illustrating the execution of test images for services configured in an A / B testing manner. Figure 8 is a block diagram illustrating an electronic device according to an embodiment.

Claims

1. A test bench providing method, implemented via an electronic device, comprising the steps of: receiving a test bench generation request for a specific test target scenario by selecting a source webpage executed within an application; identifying a plurality of services associated with the test target scenario; generating a test bench configured with the status of feature changes for each of the plurality of services and test paths between the plurality of services, for each of the plurality of features that can be changed according to testing in the test target scenario, wherein each of the plurality of services supports at least one of the plurality of features; and executing the test target scenario on the test bench, wherein the execution steps include: For services whose features have not been changed according to the above tests, execute a basic image corresponding to the released version of the application; and for services whose features have been changed according to the above tests, execute a test image reflecting the changed features to the basic image.

2. As requested in the test bench method of item 1, wherein the source webpage includes a search results webpage and a single detail webpage implemented within the product sales service application.

3. The test bench providing method as requested in item 1 further includes the following steps: receiving a configuration modification request for the test bench; and based on the configuration modification request, modifying whether the characteristics of each of the plurality of services are changed, and at least a portion of the test paths among the plurality of services.

4. The test bench provision method of request item 1 further includes the following steps: along with the creation of the test bench, providing a dedicated URL that can access the test bench.

5. As requested in the test bench method of item 4, wherein the aforementioned dedicated URL is generated according to each of the features that can be changed according to the test in the aforementioned test target scenario, and the string constituting the aforementioned dedicated URL includes the name of the feature corresponding to the aforementioned dedicated URL.

6. The test bench providing method as requested in item 1 further includes the following steps: receiving the execution request of the above-mentioned test target scenario; and providing the execution result of the above-mentioned test target scenario.

7. The test bench provision method as described in request item 4 further includes the following steps: Based on the above-mentioned dedicated URL, setting at least one permission setting related to the above-mentioned test bench, wherein the at least one permission setting includes execution permission, modification permission and viewing permission.

8. The test bench provided by Request 1, wherein the step of executing the test image for the service configured to perform A / B testing among the plurality of services includes the following steps: transmitting the basic image to Pod 1 and Pod 2; executing the basic image using Pod 1-1 within Pod 1; executing a first test image with features of version A using Pod 1-2 within Pod 1; executing the basic image using Pod 2-1 within Pod 2; and executing a second test image with features of version B using Pod 2-2 within Pod 2.

9. The test bench provided by Request 1, wherein the aforementioned basic image is collected and stored by a distributed streaming platform in response to the aforementioned execution request.

10. As provided in the test bench of request item 6, the above execution results are as follows: In a dashboard, the performance parameter differences before and after the test corresponding to the feature changes of each of the above plurality of services are displayed separately; for the services among the above plurality of services that have undergone A / B testing, the performance parameter differences between the group applying the features of version A and the group applying the features of version B are displayed separately.

11. The test bench providing method as requested in Request 10, wherein the execution results further include a graph, which is displayed in the dashboard according to a plurality of types to differentiate the execution results within a time period during which the test was performed; the plurality of types include at least a portion of test completed, no difference before and after the test, difference before and after the test, and test failure due to an error.

12. The method for providing a test bench as requested in claim 1 further includes the following steps: receiving an adjustment input for the validity period of the test bench; adjusting the validity period based on the adjustment input; and blocking external access to the test bench when the validity period has elapsed since the point in time when the test bench was created.

13. A non-transitory computer-readable recording medium recording a program for performing the method as described in claim 1 on a computer.

14. An electronic device providing a test bench, comprising a memory storing at least one command and a processor, the processor being configured to: be connected to the memory and execute the at least one command to receive a test bench request for a specific test target scenario by selecting a source webpage executed within an application; identify a plurality of services associated with the test target scenario; generate a test bench for each of a plurality of features that may change according to testing in the test target scenario, specifying whether the features of each of the plurality of services have changed and the test paths between the plurality of services, wherein each of the plurality of services supports at least one of the plurality of features; and execute the test target scenario in the test bench, wherein the processor further executes the at least one command to: execute a base image corresponding to a published version of the application for services whose features have not changed according to the testing; and execute a test image reflecting the changed features to the base image for services whose features have changed according to the testing.

Images