Method for managing service-related test, apparatus for the same and non-transitory computer-readable recording medium for performing the method

Abstract

This invention discloses a method for managing service-related testing in electronic devices. Specifically, the method for managing service-related testing may include: a step of confirming first information of a set A / B test; a step of confirming at least one domain associated with the A / B test based on the first information; a step of confirming at least one automated test associated with the at least one domain; a step of confirming tester identification information of the at least one automated test; and a step of distributing at least a portion of the users corresponding to the confirmed tester identification information to the user group set for the A / B test.

Description

Technical Field

[0001] The embodiments of this specification relate to a method and apparatus for managing service-related testing. The embodiments of this specification relate to a method and apparatus for managing service-related testing by minimizing the impact between A / B testing and at least one automated test associated with A / B testing. Prior Technology

[0002] In recent years, various tests have been implemented in online marketing and other online environments to analyze user behavior data regarding existing and new versions. Among these, A / B testing compares the existing version (A) with new versions (B, C, ...) to determine which version performs better. The benchmarks for comparing the results of the existing and new versions can include click-through rate (CTR) or purchase conversion rate. Except for changes in specific factors that might affect user behavior, the existing and new versions can be identical. Therefore, the results of A / B testing can reveal how changes in specific factors affect user CTR or purchase conversion rate.

[0003] On the other hand, unlike A / B testing, automated testing of existing versions can also be implemented. However, if the users assigned to the test group for A / B testing overlap with the users assigned to the test group for automated testing, the reliability of automated testing may be compromised. Therefore, methods and apparatuses are needed to solve this problem.

[0004] For related information, please refer to previous documents such as KR102259945B1 and KR102309715B1. Summary of the Invention

[0005] [The problem the invention aims to solve] This invention is proposed to solve the above-mentioned problems and aims to provide a method and apparatus for testing related to management and services.

[0006] More specifically, the present invention aims to provide a method and apparatus for distributing at least a portion of users corresponding to the tester identification information to a user group set by the A / B tester by confirming the tester identification information of at least one automated test related to A / B testing.

[0007] Therefore, the present invention aims to provide a method and apparatus for ensuring the reliability of automated testing even when performing A / B testing by distributing at least a portion of the users corresponding to the tester identification information to the user group set by the A / B tester.

[0008] The technical problem to be solved in this embodiment is not limited to the above-mentioned technical problem, and other technical problems can be deduced from the following embodiments. [Technical means to solve the problem]

[0009] As a technical method for achieving the above-mentioned objectives, the method for managing and service-related testing in the electronic device of the first embodiment of the present invention may include: a step of confirming first information of the set A / B test; a step of confirming at least one domain associated with the A / B test based on the first information; a step of confirming at least one automated test associated with at least one domain; a step of confirming tester identification information of at least one automated test; and a step of distributing at least a portion of the users corresponding to the confirmed tester identification information to the user group set for the A / B test.

[0010] According to one embodiment, a method for managing service-related tests in an electronic device may include the step of providing a page that displays information about at least one A / B test related to a domain on the electronic device.

[0011] According to one embodiment, the information of at least one A / B test may include at least one of the following: source code information corresponding to at least one A / B test, domain information of at least one A / B test, and status information of the action of at least one A / B test.

[0012] According to one embodiment, the step of distributing at least a portion of users to the user group defined in the A / B test may include the step of distributing at least a portion of users to the control group of the A / B test.

[0013] According to one embodiment, the step of distributing at least a portion of users to the control group of an A / B test may include: identifying the user identification information assigned to at least one experimental group of the A / B test; identifying a first tester identification information that matches the user identification information in the tester identification information; and reassigning the user corresponding to the first tester identification information from the at least one experimental group to the control group.

[0014] According to one embodiment, the step of identifying at least one domain may include: identifying at least one domain associated with the A / B test based on the source code of the A / B test.

[0015] According to one embodiment, a method for managing service-related testing in an electronic device may further include: after distributing at least a portion of users to a user group defined for A / B testing, identifying at least one first automated test related to at least one domain within a set time period; confirming the first tester identification information of at least one first automated test; and distributing at least a portion of the users corresponding to the first tester identification information to the user group.

[0016] According to one embodiment, the step of distributing at least a portion of the users corresponding to the first tester identification information to a user group may include: confirming the identification information of users assigned to at least one experimental group in an A / B test; confirming the second tester identification information in the first tester identification information that matches the user's identification information; and reassigning the users corresponding to the second tester identification information from at least one experimental group to a control group.

[0017] According to one embodiment, a method for managing service-related tests in an electronic device may include the step of confirming the status information of an A / B test as being generated, executed, or completed via an API (Application Programming Interface).

[0018] According to one embodiment, the step of confirming the first information of the A / B test may include: confirming the status information of the A / B test as generated, or confirming that the status information of the A / B test has changed from generated to executed.

[0019] According to one embodiment, a method for managing service-related testing in an electronic device may further include the step of initiating the distribution of at least a portion of users to a user group as the status information of the A / B test is confirmed to be complete.

[0020] According to one embodiment, at least one automated test may be a test for which an A / B test corresponding to at least one automated test has been completed.

[0021] According to one embodiment, a method for managing service-related testing in an electronic device may further include the steps of: providing a page showing the current service version to the terminals of users in a control group of an A / B test, and providing a page showing a service version matching the at least one experimental group to the terminals of users in at least one experimental group of the A / B test.

[0022] According to one embodiment, a method for managing service-related testing in an electronic device may further include the step of providing A / B test result information that includes information on at least a portion of the users.

[0023] According to one embodiment, a method for managing service-related testing in an electronic device may further include the step of providing A / B test result information to users in a test group based on A / B testing, excluding the user corresponding to the identification information of the second tester.

[0024] The electronic device for managing and servicing related tests in the second aspect of the present invention may include: a transceiver; a storage device storing one or more commands; and a processor that confirms first information of a set A / B test, confirms at least one domain associated with the A / B test based on the first information, confirms at least one automated test associated with the at least one domain, confirms tester identification information of the at least one automated test, and distributes at least a portion of the users corresponding to the confirmed tester identification information to the user group set for the A / B test.

[0025] The recording medium of the third aspect of the present invention may be a non-transitory computer-readable recording medium for recording programs that can be executed in a computer. [Effects of the Invention]

[0026] According to embodiments of this specification, an electronic device can identify at least one automated test in at least one domain associated with an A / B test based on first information from the A / B test. As the electronic device distributes at least a portion of the tester identification information for at least one automated test to a user group defined for the A / B test, the reliability of the automated test results can be guaranteed regardless of the execution of the A / B test.

[0027] The effects of this invention are not limited to those mentioned above. Those skilled in the art can clearly understand other effects not mentioned based on the description in the claims. Simple Explanation of the Diagram

[0028] Figure 1 is a system diagram illustrating various embodiments of a method for conducting tests related to electronic device management and services. Figure 2 is a flowchart illustrating the testing methods related to the management and service of electronic devices. Figure 3 is a diagram illustrating in more detail the methods for implementing management and service-related testing of electronic devices. Figure 4 is a diagram illustrating in detail the page that displays information about at least one A / B test related to the domain. Figure 5 is a diagram illustrating the page used to override user groups in at least one A / B test. Figure 6 is a detailed page showing the method for prioritizing user groups in the 1A / B test. Figure 7 is a detailed page showing the method for prioritizing user groups in the 4A / B test. Figure 8 is a block diagram illustrating an embodiment of an electronic device used for managing and servicing related tests. Implementation

[0029] The terminology used in the embodiments takes into account the functions of this invention and selects currently widely used common terms. However, this may vary depending on the intent of those skilled in the art, precedents, or the emergence of new technologies. Furthermore, in certain situations, there may be terms arbitrarily chosen by the applicant; in such cases, their meanings will be described in detail in the corresponding description. Therefore, the terminology used in this invention should be defined based on its meaning and the entire content of this invention, and not simply on the name of the term.

[0030] Throughout this specification, when a section "includes" a certain component, it means that, unless otherwise specified, other components may be included, rather than excluding them. Furthermore, terms such as "...section" and "...module" used in the specification refer to a unit that processes at least one function or action, which can be implemented by hardware or software, or by a combination of hardware and software.

[0031] The expression “at least one of a, b and c” in the entire instruction manual may include “a alone”, “b alone”, “c alone”, “a and b”, “a and c”, “b and c” or “a, b and c”.

[0032] The term "terminal" as used below can refer to a computer or portable terminal that connects to a server or other terminal via a network. Here, "computer" includes laptops, desktops, and laptops equipped with a web browser. "Portable terminal," as a wireless communication device ensuring portability and mobility, can include communication terminals such as IMT (International Mobile Telecommunication), CDMA (Code Division Multiple Access), W-CDMA (W-Code Division Multiple Access), and LTE (Long Term Evolution), as well as all types of handheld wireless communication devices such as smartphones and tablets.

[0033] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings to enable those skilled in the art to readily implement the invention. However, the present invention can be implemented in various different ways and is not limited to the embodiments described herein.

[0034] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0035] In the description of the embodiments, descriptions of technical content well-known in the technical field to which this invention pertains and not directly related to this invention will be omitted. By omitting unnecessary descriptions, the main points of this invention are conveyed more clearly and explicitly.

[0036] For the same reason, some components in the accompanying drawings are exaggerated, omitted, or shown in a simplified manner. Furthermore, the size of each component does not fully reflect its actual size. Identical or corresponding components in each drawing are labeled with the same reference symbols.

[0037] The advantages and features of the present invention, as well as the methods for achieving these advantages and features, will become clear by referring to the accompanying drawings and the embodiments described in detail below. However, the present invention is not limited to the embodiments disclosed below and can be implemented in various other ways. These embodiments are provided only to make the disclosure of the present invention complete and to enable those skilled in the art to fully understand the scope of the invention. The present invention is defined only according to the claims. Throughout this specification, the same reference numerals refer to the same constituent elements.

[0038] At this point, it can be understood that the combination of the blocks in the flowchart and the flowchart itself can be performed using computer program instructions. These computer program instructions can be installed on the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thus enabling the execution of methods for performing the functions described in the flowchart blocks by instructions executed by the processor of the computer or other programmable data processing device. To implement functions in a specific manner, these computer program instructions can be stored in computer-usable or computer-readable memory of a specified computer or other programmable data processing device. Therefore, instructions stored in computer-usable or computer-readable memory can also manufacture products including methods for executing the functions described in the flowchart blocks. Computer program instructions can be installed on a computer or other programmable data processing device, performing a series of actions on the computer or other programmable data processing device to produce a flow executed by the computer. The instructions executed by the processor of the computer or other programmable data processing device provide steps for executing the functions described in the flowchart blocks.

[0039] Furthermore, each block may represent a portion of a module, fragment, or code, which includes one or more executable instructions for performing a specific logical function. It should also be noted that in some alternative embodiments, the functions mentioned in the blocks may not occur sequentially. For example, two consecutively listed blocks may actually occur simultaneously, or they may occur in reverse order according to their respective functions.

[0040] Figure 1 is a diagram illustrating a system of methods for implementing management and service-related testing using electronic devices in various embodiments.

[0041] Referring to FIG1, the system 10 of various embodiments can be implemented by various types of devices. For example, system 10 may include electronic device 100, terminal 110, and terminal 120. The system 10 shown in FIG1 only illustrates the components related to this embodiment. Therefore, those skilled in the art related to this embodiment will understand that, in addition to the components shown in FIG1, other general components may be included.

[0042] Electronic device 100, terminal 110, and terminal 120 may each include a transceiver, a memory, and a processor. Furthermore, electronic device 100, terminal 110, and terminal 120 may each refer to a unit that processes at least one function or action, which may be implemented by hardware or software, or a combination of hardware and software. On the other hand, throughout this specification, electronic device 100, terminal 110, and terminal 120 are each referred to as separate devices or servers, but this may be a logically separate construction, where at least a portion may be implemented by functions separate from a single device or server.

[0043] According to one embodiment, electronic device 100, terminal 110, and terminal 120 may include multiple computer systems or computer software implemented by a network server. For example, at least a portion of electronic device 100, terminal 110, and terminal 120 may refer to computer systems and computer software that connect to a lower-level device to receive job execution requests, execute the job execution requests, and provide execution results. The lower-level device may communicate with other network servers via a computer network such as an intranet or the Internet. Furthermore, at least a portion of electronic device 100, terminal 110, and terminal 120 can be understood as a broad concept, including a series of applications operable on a network server, and various databases built internally or on other connected nodes. For example, at least a portion of electronic device 100, terminal 110, and terminal 120 may be implemented using various network server programs provided by operating systems such as Disk Operating System (DOS), Windows Operating System (Windows), Linux Operating System (Linux), UNIX Operating System (UNIX), or Macintosh Operating System (MacOS).

[0044] Electronic device 100, terminal 110, and terminal 120 can communicate with each other via a network (not shown). The network includes Local Area Network (LAN), Wide Area Network (WAN), Value Added Network (VAN), mobile radio communication network, satellite mobile communication network, and combinations thereof. It is a comprehensive data communication network that enables the constituent entities of the network shown in Figure 1 to communicate smoothly with each other, and may include wired internet, wireless internet, and mobile radio communication networks. Wireless communication may include, for example, Wi-Fi, Bluetooth, Bluetooth Low Energy (BLE), Zigbee, WFD (Wi-Fi Direct), UWB (ultra-wideband), IrDA (infrared data association), NFC (Near Field Communication), etc., but is not limited to these.

[0045] According to one embodiment, the method for managing and service-related tests in electronic device 100 can confirm first information of a set A / B test, confirm at least one domain associated with the A / B test based on the first information, confirm at least one automated test associated with the at least one domain, confirm tester identification information of the at least one automated test, and distribute at least a portion of the users corresponding to the confirmed tester identification information to the user group set for the A / B test. Furthermore, electronic device 100 can provide a page of the current service version to the terminals 110 of users distributed to the control group of the A / B test, and can provide a page of the service version matching the at least one experimental group to the terminals 120 of users distributed to the at least one experimental group of the A / B test. Moreover, when a user's terminal 110 is the object of an automated test, electronic device 100 can ensure the integrity of the A / B test by distributing the terminals 110 of users assigned to the at least one experimental group of the A / B test to the control group of the A / B test, regardless of the execution of the A / B test. As described above, automated testing can be applied to the code of the current service. When a part of the functionality of the current service is changed, in order to compare the effect of the current service and the changed code, A / B testing can provide the current service to users designated as Group 1 and the service with the changed code to users designated as Group 2, thereby determining whether to apply the changed service based on the results. However, in the A / B testing described above, the users distributed to the control group and the experimental group can be dynamically determined. When the user information of the terminal executing the automated test is distributed to the experimental group, the problem of the automated test object not being the current service may arise. Therefore, the above method can ensure the integrity of the automated test. Specific embodiments are detailed below.

[0046] Figure 2 is a flowchart illustrating the testing methods related to the management and service of electronic devices.

[0047] Referring to Figure 2, it can be clearly understood that, within the scope of common knowledge in the art to which this invention pertains, some actions of the tests related to electronic device management and services may be changed, replaced, or the order of some actions may be changed.

[0048] In step S210, the electronic device 100 can confirm the first information of the set A / B test.

[0049] According to one embodiment, before confirming the first information of the set A / B test, the electronic device 100 can confirm the status information of the A / B test via an API. More specifically, the electronic device 100 can confirm the status information of the A / B test as generated, executed, or completed via the XPC (XNU inter-process communication) service API. For example, the electronic device 100 can confirm that the status information of the A / B test is generated or confirm that the status information of the A / B test has changed from generated to executed. In step S210, the set A / B test can be an A / B test with the following A / B test values ​​set and completed: 1) assigning users who are the objects of the A / B test to the test group of the A / B test, and 2) service distribution. In this invention, the test group of the A / B test can be a group that includes all control groups (A) and at least one experimental group (B, C, D, ...).

[0050] In the automated testing performed in electronic device 100, there may be automated tests that are affected by the execution of A / B testing. In order to identify the automated tests affected by the execution of A / B testing, electronic device 100 can identify the first information of the set A / B test. Here, the first information is information used to identify at least one field affected by A / B testing, which may be the source code of A / B testing or information set in A / B testing.

[0051] In step S220, the electronic device 100 can identify at least one field associated with the A / B test based on the first information.

[0052] According to one embodiment, electronic device 100 can identify at least one domain associated with A / B testing based on the source code of A / B testing. More specifically, electronic device 100 can identify at least one affected domain based on the source code of executing A / B testing. Here, the at least one domain identified based on the source code of A / B testing can be a domain directly affected by the execution of A / B testing. Furthermore, there can be a plurality of domains associated with A / B testing, but it is not limited to this.

[0053] In step S230, the electronic device 100 can confirm at least one automated test associated with at least one domain.

[0054] Automated testing refers to the completed A / B testing corresponding to automated testing, which can be a test of the existing version of the user interface provided to the user's terminal. More specifically, the existing version of the user interface provided to the user's terminal can be a control group for A / B testing. Therefore, if the users included in the test group of automated testing are assigned to at least one experimental group of A / B testing, the object of automated testing will change, and in this case, the reliability of automated testing may be compromised by A / B testing. Therefore, electronic device 100 needs to verify at least one automated test related to at least one domain.

[0055] According to one embodiment, when performing automated testing, the electronic device 100 can pre-confirm information about the domains associated with the automated test. This confirmation can be performed based on at least one of the automated test code and values ​​set for the automated test.

[0056] Therefore, the electronic device 100 can confirm at least one automated test associated with at least one domain. Furthermore, when performing a domain-related automated test, the user subject to the automated test can be at least a portion of the users in a user group defined for each domain. User groups corresponding to each domain can be defined.

[0057] In step S240, the electronic device 100 can confirm the identification information of at least one automated tester.

[0058] According to one embodiment, electronic device 100 can pre-distribute identification information of users corresponding to at least one domain. More specifically, in order to distinguish the tests performed in each domain, electronic device 100 can pre-set different user groups for each domain. For example, electronic device 100 can distribute a first user group to the shopping cart domain and a second user group to the checkout domain. Furthermore, the user who becomes the object of an automated test associated with a domain can be at least a portion of the users in the user group corresponding to the domain. Therefore, electronic device 100 can confirm the tester identification information of at least one automated test and can confirm the user who becomes the object of the automated test corresponding to the electronic tester identification information. In this invention, the identification information is information used to distinguish a user from other users and can be at least one of the user's ID (identifier) ​​information or the user's unique identifier information.

[0059] In step S250, the electronic device 100 may distribute at least a portion of the users corresponding to the confirmed tester identification information to the user group set for the A / B test.

[0060] The automated testing is an existing version of the test. Among the users corresponding to the tester identification information of the automated test, the users included in the A / B test need to be assigned to the control group and at least one experimental group. More specifically, if users who are included in the test group of the A / B test among the users corresponding to the tester identification information of the automated test are assigned to at least one experimental group, the reliability of the automated test may be compromised. However, the allocation of the test groups in the A / B test (assigned to the control group and at least one experimental group) is randomly determined, so users who are included in the test group of the A / B test among the users corresponding to the tester identification information of the automated test can be assigned to at least one experimental group. Therefore, among the users corresponding to the tester identification information of the automated test, users assigned to at least one experimental group of the A / B test need to be reassigned to the control group. Therefore, the electronic device 100 can distribute at least a portion of the users corresponding to the tester identification information to the control group of the A / B test. More specifically, the electronic device 100 can confirm the identification information of users assigned to at least one experimental group in an A / B test, confirm the first tester identification information that matches the identification information of users assigned to at least one experimental group, and reassign the user corresponding to the first tester identification information from the at least one experimental group to the control group. In this invention, the reassignment corresponds to the order in which user groups are preferentially distributed in the A / B test.

[0061] In this invention, the control group may correspond to the existing version implemented in the A / B test, and at least one experimental group may correspond to the new version that serves as the object of comparison of results with the existing version in the A / B test.

[0062] Figure 3 is a diagram illustrating in more detail the methods for implementing management and service-related testing of electronic devices.

[0063] Referring to Figure 3, the electronic device 100 can function as a device for managing tests related to the service of the electronic device 100.

[0064] In step S310, the electronic device 100 can confirm that the status information of the A / B test is "generated". More specifically, the electronic device 100 can use an API to confirm the status information of the A / B test as "generated", "executed", or "completion in progress". The set A / B test can be an A / B test that has been set up and completed, such as: 1) assigning users who are the subjects of the A / B test to the test group of the A / B test, and 2) service distribution, etc., and can also be set to a predetermined execution time of the A / B test. For example, out of 10,000 users who are the subjects of the A / B test, 5,000 can be randomly assigned to the control group of group A, and the remaining 5,000 can be randomly assigned to the experimental group of group B. Referring to Figure 3, the predetermined execution time of the A / B test can be time 1, and the execution time of the A / B test can be time 2.

[0065] During A / B testing, changes to the test group of users who are the subjects of the A / B test may compromise the reliability of the A / B test results. Therefore, before executing A / B testing, a portion of the users corresponding to the tester identification information of the automated test should be distributed to the user group defined for the A / B test.

[0066] In step S311, the electronic device 100 may distribute at least a portion of the users corresponding to the tester identification information to the user group defined for the A / B test. More specifically, the electronic device 100 may distribute at least a portion of the users corresponding to the A / B tester identification information to the user group defined for the A / B test based on the predetermined execution time of the A / B test. For example, the electronic device 100 may distribute at least a portion of the users corresponding to the A / B tester identification information to the user group defined for the A / B test before the first moment of the predetermined execution time of the A / B test.

[0067] For example, referring to Figure 3, the electronic device 100 can identify the identification information of 5,000 users assigned to the experimental group B in the A / B test. Furthermore, the electronic device 100 can identify the first tester identification information that matches the user identification information from the tester identification information of at least one automated test. Here, the number of users corresponding to the first tester identification information can be 100. Therefore, the electronic device 100 can reassign the users corresponding to the first tester identification information from the experimental group B to the control group A. Thus, the control group A may include 5,100 users, and the experimental group B may include 4,900 users.

[0068] In step S320, the electronic device 100 can confirm that the status information of the A / B test is "Executing". More specifically, the electronic device 100 can confirm the status information of the A / B test as "Executing" in the stages of generation, execution, and completion via API. More specifically, the electronic device 100 can provide the current service version page to the terminals of the 5100 users in the control group (Group A) and provide the service version page matching the experimental group to the terminals of the 4900 users in the experimental group (Group B).

[0069] In step S321, the electronic device 100 may distribute at least a portion of the users corresponding to the tester identification information to the user group set for the A / B test. The specific action of distributing at least a portion of the users corresponding to the tester identification information to the user group set for the A / B test may be similar to the embodiment shown in step S311 and FIG2.

[0070] For example, referring to Figure 3, the electronic device 100 can identify the identification information of 4,900 users assigned to the experimental group B in the A / B test. Furthermore, the electronic device 100 can identify the first tester identification information that matches the user identification information from the tester identification information of at least one automated test. Here, the number of users corresponding to the first tester identification information can be 50. Therefore, the electronic device 100 can reassign the users corresponding to the first tester identification information from the experimental group B to the control group A. Thus, the number of users included in the control group A can be 5,150, and the number of users included in the experimental group B can be 4,850.

[0071] According to one embodiment, after at least a portion of users are distributed to the user group defined for A / B testing, the electronic device 100 can also identify at least one first automated test related to at least one domain within a set time period. More specifically, there may be at least one first automated test that is additionally executed or generated after at least a portion of users are distributed to the user group defined for A / B testing. In this case, the electronic device 100 needs to confirm the additionally executed or generated first automated test. Therefore, the electronic device 100 can identify at least one first automated test related to at least one domain at each set time period. Here, the set time period may be 1 hour, but is not limited to this.

[0072] According to one embodiment, electronic device 100 can confirm at least one first tester identification information for a first automated test. Furthermore, electronic device 100 can distribute at least a portion of the users corresponding to the first tester identification information to a user group. More specifically, electronic device 100 can preferentially distribute users among the users corresponding to the first tester identification information who match users in at least one experimental group of an A / B test to a control group. In this invention, preferential distribution can be the forced distribution of a portion of the users in at least one experimental group of an A / B test to a control group.

[0073] For example, referring to Figure 3, the electronic device 100 can identify the identification information of 4,850 users assigned to the experimental group B in the A / B test. Furthermore, the electronic device 100 can identify the first tester identification information that matches the user identification information in at least one first automated tester identification information. Here, the number of users corresponding to the first tester identification information can be 50. Therefore, the electronic device 100 can reassign the users corresponding to the first tester identification information from the experimental group B to the control group A. Thus, the control group A includes 5,200 users, and the experimental group B includes 4,800 users.

[0074] Therefore, for the terminals of 100 users who were reassigned from the experimental group B to the control group A, the electronic device 100 can change the page of the service version matched to the experimental group to the page of the current service version.

[0075] In step S330, the electronic device 100 can confirm that the status information of the A / B test is "complete". More specifically, the electronic device 100 can confirm the status information of the A / B test as "complete" in the process of generation, execution and completion via API.

[0076] In step S331, the electronic device 100 can confirm the completion of the A / B test status information and initialize the distribution of at least a portion of the users to the user group. For example, referring to FIG3, the electronic device 100 can initialize the distribution of 200 users from the experimental group of group B to the control group of group A.

[0077] According to one embodiment, the electronic device 100 can provide A / B test result information. Here, the A / B test result information may include information on which test group in the control group and at least one experimental group recorded a higher user purchase conversion rate. To ensure the reliability of the A / B test results, the electronic device 100 may provide the A / B test result information according to the following embodiments.

[0078] For example, even while an A / B test is being performed, the electronic device 100 can reassign a portion of the users included in the experimental group of the A / B test to the control group. As some users in the experimental group of the A / B test are reassigned to the control group, the results of the A / B test may be distorted. Therefore, the electronic device 100 can also simultaneously provide information on users in the test group of the A / B test who have undergone test group changes. For example, referring to Figure 3, the electronic device 100 can simultaneously provide information on the 200 users reassigned from group B of the experimental group to group A of the control group in steps S311 and S321.

[0079] Furthermore, for example, electronic device 100 can identify at least one first user who performed a change in the test group during the A / B test, and provide A / B test result information based on users other than at least one first user included in the test group of the A / B test. For example, electronic device 100 can provide A / B test result information based on 9900 users other than the 100 users who were reassigned from group B of the experimental group to the control group in step S321.

[0080] However, the allocation of test groups in the A / B test embodiments of the present invention is not limited to this. For example, the electronic device 100 distributes at least a portion of the users corresponding to the tester identification information to the control group of the A / B test, so the allocation ratio of the test groups in the A / B test may differ from the set allocation ratio. Therefore, the electronic device 100 may correspond to distributing at least a portion of the users corresponding to the tester identification information to the control group of the A / B test, and distributing at least a portion of the users included in the control group to the experimental group of the A / B test. For example, the electronic device 100 may correspond to distributing 200 users included in the experimental group of group B to the control group of group A, and randomly selecting 200 users from the original control group of group A to the experimental group of group B. Therefore, the electronic device 100 can maintain the same number of users distributed to the control group and the experimental group as set in the A / B test.

[0081] Figure 4 is a diagram illustrating in detail the page that displays information about at least one A / B test related to the domain.

[0082] Referring to Figure 4, the electronic device 100 can provide a page 400 displaying information about at least one A / B test related to a domain. Page 400 can be provided for searching the shopping cart domain 410 within the domain and for the status information "Execution" 411 of the shopping cart domain 410. For example, page 400 may include information about A / B tests with index numbers 19022, 19023, 21156, 21157, and 21511. Referring to Figure 4, page 400 can display information about the third A / B test 420 with index number 21156.

[0083] Furthermore, page 400 may include information about each A / B test. More specifically, the A / B test information may include at least one of the following: source code information corresponding to at least one A / B test, domain information of at least one A / B test, and status information of the actions of at least one A / B test. Referring to Figure 4, the A / B test information may include: A / B test domain information 421, information about the domains associated with the A / B test identified based on the A / B test source code 422, status information of the actions of the A / B test 423, execution time information of the A / B test 424, information about the test group to which the test is prioritized 425, and update information 426.

[0084] For example, information 421 regarding the A / B test field can be information about the format field related to the A / B test. Furthermore, the electronic device 100 can store information about the fields associated with each A / B test. Referring to Figure 4, the information in the field of the third A / B test 420 can be "checkout".

[0085] For example, information 422 related to the A / B test, confirmed based on the A / B test source code, may be information about the domain actually affected by the A / B test. Furthermore, information 422 related to the A / B test, confirmed based on the A / B test source code, may differ from information about the formal domains of the A / B test. Referring to Figure 4, information 421 "checkout" for the A / B test domain with index number 21511 and information 422 "cart" for the domain associated with the A / B test may differ.

[0086] For example, the status information 423 of the A / B test action can be the status information of the A / B test, which may include "created", "calculated", and "calculation_stopped". Referring to Figure 4, the status information of the action 420 of the third A / B test can be "calculation stopped".

[0087] For example, the execution time information 424A / B of the A / B test can be not only the year / month / day of the test execution, but also the time information. Referring to Figure 4, the third A / B test 420 can be the A / B test executed at 13:49:30 on March 25, 2022.

[0088] For example, the information 425 of the priority distribution test group may be information that distributes at least a portion of the users corresponding to the tester identification information to the user group. In this invention, the information 425 of the priority distribution test group may primarily be the control group of group A, but is not limited thereto. Referring to FIG4, the information of the priority distribution test group in the 3A / B test 420 may be group A.

[0089] For example, the updated information 426 could be information about the time at which at least a portion of the users corresponding to the tester identification information are distributed to the user group, with the time period set in step S321. Referring to Figure 4, the updated information for test 3A / B 420 could be 16:00:11 on May 19, 2022.

[0090] Furthermore, the embodiment in Figure 4 has one field corresponding to at least one A / B test, but is not limited to this. For example, there may be multiple fields corresponding to one A / B test, and the multiple fields may also include a shopping cart field, a checkout field, a recommendation algorithm field, and a details page field.

[0091] Figure 5 is a diagram illustrating the page used to override user groups in at least one A / B test.

[0092] Referring to Figure 5, the electronic device 100 can provide a page 500 for priority distribution of at least one A / B test. For example, the priority distribution page can display information on the priority distribution of at least one A / B test generated, executed, and completed in the electronic device 100.

[0093] The priority distribution of this invention can be performed simultaneously with the allocation of test groups in A / B testing, or it can be performed after the allocation of test groups in A / B testing. 1) For example, the allocation of test groups in the first A / B test 510 may not have been performed yet. Specifically, the "group" information of the first A / B test 510 may be information indicating that 100% of users are assigned to the control group of group A and 0% of users are assigned to the experimental group of group B. The electronic device 100 may prioritize the allocation of at least a portion of the users corresponding to the tester identification information of at least one automated test associated with the first A / B test 510 to the control group of group A in the A / B test, and allocate the users included in the test group of the first A / B test 510 to either the control group of group A or the experimental group of group B. 2) Furthermore, the fourth A / B test 520 may be an A / B test in which test group allocation is performed. Specifically, the information for the 4A / B test 520 "group" may include status information showing that 50% of users are assigned to the control group (Group A), 25% of users are assigned to the experimental group (Group B), and 25% of users are assigned to the experimental group (Group C). The electronic device 100 may preferentially assign at least a portion of the users corresponding to the tester identification information of at least one automated test related to the 4A / B test 520 to the control group (Group A) of the A / B test.

[0094] Page 500 may include the input point image 511 of the detailed page for priority distribution of the first A / B test 510 and the input point image 521 of the detailed page for priority distribution of the fourth A / B test 520. The detailed pages for priority distribution of each A / B test are described below.

[0095] Figure 6 is a detailed page showing the method for prioritizing user groups in the 1A / B test.

[0096] The detailed page 600 for priority distribution in the 1A / B test may be the priority distribution page for the 1A / B test 510 in Figure 5. Here, the test group for the 1A / B test may include the control group of group A and the experimental group of group B. Furthermore, it may be a state where 100% of users are distributed to the control group of group A in the 1A / B test, and 0% of users are distributed to the experimental group of group B. The electronic device 100 may prioritize the distribution of at least a portion of the users corresponding to the tester identification information of at least one automated test associated with the 1A / B test 510 to the control group of group A in the A / B test. For example, the electronic device 100 may reassign the user with user ID 1283 among the users corresponding to the tester identification information of at least one automated test to the control group of group A.

[0097] Electronic device 100 can verify user identification information, which serves as user ID, to distinguish users included in the test group of the 1A / B test. Here, the user identification information included in the test group of the 1A / B test can be information of the user equipment, namely user SRL (shift register latch) information.

[0098] Figure 7 is a detailed page showing the method for prioritizing user groups in the 4A / B test.

[0099] The detailed page 700 for priority distribution in the 4A / B test may be the page for priority distribution in the 4A / B test 520 shown in Figure 5. Here, the test group for the 4A / B test may include the control group of group A, the experimental group of group B, and the experimental group of group C. Furthermore, it may be a situation where 50% of users are distributed to the control group of group A in the 1A / B test, 25% of users are distributed to the experimental group of group B, and 25% of users are distributed to the experimental group of group C. However, the electronic device 100 may prioritize the distribution of at least a portion of the users corresponding to the tester identification information of at least one automated test associated with the 4A / B test 520 to the control group of group A in the A / B test. For example, the electronic device 100 may reassign the user with user ID 1483, who corresponds to the tester identification information of at least one automated test, from the experimental group of group B to the control group of group A. Furthermore, the electronic device 100 can reassign the user with user ID 2579 from the experimental group of group C to the control group of group A, among the users corresponding to the tester identification information of at least one automated test.

[0100] Electronic device 100 can verify user identification information such as user ID to distinguish users included in the test group of test 1A / B. Here, the user identification information included in the test group of test 4A / B can be the user's computing device information PCID (Process-Context Identifier) ​​or the user's account identification information in the service.

[0101] Figure 8 is a block diagram illustrating an embodiment of an electronic device used for managing and servicing related tests.

[0102] The electronic device 800 in Figure 8 can correspond to the electronic device 100 in the specification of this invention.

[0103] According to one embodiment, the electronic device 800 of the present invention may include a transceiver 810, a processor 820, and a storage device 830. The components shown in FIG8 are not essential for implementing the electronic device, and those skilled in the art related to this embodiment will understand that the electronic device 800 described herein may have more or fewer components than those listed above. Furthermore, in this embodiment, the processor 820 may include at least one processor. Also, the electronic device 800 may operate as a server.

[0104] Transceiver 810 can communicate with external devices using wired / wireless communication, including transceiver 810 itself. External devices can be terminals or servers belonging to developers, quality assurance teams, etc. Furthermore, the communication technologies used by transceiver 810 can include, but are not limited to, GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), LTE (Long Term Evolution), 5G (5th generation mobile network), WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Bluetooth, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), Zigbee, and NFC (Near Field Communication).

[0105] According to one embodiment, transceiver 810 can communicate with a terminal to send and receive information. Transceiver 810 can provide a page displaying information about at least one A / B test related to a domain on electronic device 100. Furthermore, transceiver 810 can provide a page showing the current service version to terminals 110 of users assigned to the control group in the A / B test. Also, transceiver 810 can provide a page showing the service version matching at least one experimental group to terminals 120 of users assigned to at least one experimental group in the A / B test. Furthermore, transceiver 810 can provide A / B test result information including information about at least a portion of the users corresponding to the tester identification information.

[0106] Processor 820 can control the overall operation of electronic device 800 and process data and signals. Processor 820 can implement one of the methods described in Figures 1 to 8. Processor 820 can control the interaction of constituent elements to control the implementation of an embodiment of electronic device 800, which may further include transceiver 810, storage 830 and electronic device 800. According to one embodiment, processor 820 can confirm first information of a set A / B test, confirm at least one domain associated with the A / B test based on the first information, confirm at least one automated test associated with the at least one domain, confirm tester identification information of at least one automated test, and distribute at least a portion of the users corresponding to the confirmed tester identification information to the user group set for the A / B test.

[0107] Storage 830 can store information implemented by at least one of the methods described above using Figures 1 to 8. Storage 830 can be referred to as memory, and can be volatile memory or non-volatile memory. For example, storage 830 can include the data storage of Figure 3. For example, storage 830 can store identification information corresponding to a user. Furthermore, storage 830 can include information from at least one A / B test of electronic device 800 and information from at least one field corresponding to at least one A / B test.

[0108] On the other hand, preferred embodiments of the invention have been described in this specification and accompanying drawings. Although specific terms have been used, these terms are used only in a general sense to illustrate the technical content of the invention and to aid in understanding the invention, and are not intended to limit the scope of the invention. In addition to the embodiments disclosed herein, those skilled in the art can certainly implement other variations based on the technical concept of the invention.

[0109] The electronic device or terminal described in the above embodiments may include: a processor, memory for storing and executing program data, permanent storage such as a disk drive, a communication port for communicating with external devices, a touch panel, keys, buttons, and other user interface devices. The method implemented using software modules or algorithms can store executable code or program instructions on a computer-readable recording medium as executable code on the processor. Here, computer-readable recording media include magnetic storage media (e.g., ROM (read-only memory), RAM (random-access memory), floppy disks, hard disks, etc.) and optical reading media (e.g., CD-ROM, DVD, etc.). The computer-readable recording media are distributed across a networked computer system, storing and executing computer-readable code in a distributed manner. The media can be read by a computer, stored in memory, and executed on the processor.

[0110] This embodiment can be represented by functional blocks and various processing steps. Such functional blocks can be implemented by different hardware and / or software components that perform specific functions. For example, the embodiment can employ integrated circuits capable of performing various functions via the control of one or more microprocessors or other control devices, such as memory, processing, logic, lookup tables, etc. The constituent elements can be executed by software programs or software components. Similarly, this embodiment includes algorithms implemented by combinations of data structures, programs, conventions, or other program structures, and therefore can be implemented using programming or scripting languages ​​such as C, C++, Java, assemblers, Python, etc. Functionally, it can be implemented by algorithms executed in one or more processors. Furthermore, this embodiment can employ prior art for electronic environment setup, signal processing, and / or data processing. Terms such as "mechanism," "component," "structure," and "construction" are used broadly and are not limited to mechanical and physical structures. The above terms may be associated with processors and include the meaning of a series of processes in software.

[0111] The above embodiment is only one example, and other embodiments can be implemented in the patent application scope described below.

[0112] 10: System 100: Electronic devices 110: Terminal 120: Terminal 400: Page 410: Domain 411: Execution 420: Test 3A / B 421: Information on the Domain of A / B Testing 422: Information on domains related to A / B testing 423: Status Information of Actions in A / B Testing 424: Information on the execution time of A / B tests 425: Information on the priority distribution of test groups 426: Updated Information 500: Page 510: Test 1A / B 511: Input point image 520: Test 4A / B 521: Input point image 600: Details Page 700: Details Page 800: Electronic Devices 810: Transceiver 820: Processor 830: Storage S210: Steps S220: Steps S230: Steps S240: Steps S250: Steps S310: Steps S311: Steps S320: Steps S321: Steps S331: Steps

Claims

1. A method for conducting management and service-related tests using an electronic device, comprising: The steps to confirm the first piece of information for the set A / B test; The steps include: confirming at least one domain associated with the set A / B test based on the first information mentioned above; confirming at least one automated test associated with the at least one domain, wherein the at least one automated test is a completed A / B test corresponding to the at least one automated test; confirming the tester identification information of the at least one automated test, wherein each user corresponding to the tester identification information is randomly assigned to the control group or the experimental group of the set A / B test; and reassigning one of the users corresponding to the tester identification information and assigned to the at least one experimental group from the at least one experimental group back to the control group.

2. The testing methods related to management and services, as described in Request 1, include: The steps include providing a page that displays information about at least one A / B test related to a domain on the aforementioned electronic device.

3. The method for testing related to management and services as described in Request Item 2, wherein the information of the at least one A / B test includes at least one of the following: source code information corresponding to the at least one A / B test, domain information of the at least one A / B test, and status information of the actions of the at least one A / B test.

4. As in Request Item 1, the method for testing related to management and services, wherein the steps for identifying at least one domain include: Based on the source code of the aforementioned A / B test, the step of identifying at least one domain associated with the aforementioned A / B test is performed.

5. The testing methods related to management and services, as described in Request 1, further include: After the users are reassigned, the steps include: identifying at least one first automated test related to at least one domain within a set time period; confirming the first tester identification information of the at least one first automated test; and reassigning one of the users assigned to the at least one experimental group that corresponds to the first tester identification information from the at least one experimental group to the control group.

6. The testing methods related to management and services, as described in Request 1, include: The API is used to confirm the status information of the above-mentioned A / B test as one of the steps of generation, execution and completion.

7. As in Request 1, the method for testing related to management and services, wherein the step of confirming the first information of the aforementioned A / B test includes: The steps to confirm the status information of the above-mentioned A / B test as generated or to confirm the status information of the above-mentioned A / B test as changed from generated to executed.

8. The testing methods related to management and services, as described in Request 1, further include: Once the status information of the aforementioned A / B test is confirmed as complete, the step of reassigning from at least one experimental group to the aforementioned control group for initialization will be performed.

9. The method for testing management and services as described in Request 1 further includes the following steps: providing a page showing the current service version to the terminals of users distributed in the control group of the aforementioned A / B test, and providing a page showing the service version matching the at least one experimental group to the terminals of users distributed in at least one experimental group of the aforementioned A / B test.

10. The testing methods related to management and services, as described in Request 1, further include: Provide the result information of the A / B test set above, including user information of users who have been randomly assigned to at least one experimental group of the A / B test among the users corresponding to the tester identification information.

11. An electronic device for managing and servicing related tests, comprising: transceiver; A storage device that stores one or more commands; and a processor, wherein the processor is configured to: confirm first information of a set A / B test; confirm at least one domain associated with the set A / B test based on the first information; confirm at least one automated test associated with the at least one domain, wherein the at least one automated test is a test for which the corresponding A / B test has been completed; confirm tester identification information of the at least one automated test, wherein each user corresponding to the tester identification information is randomly assigned to a control group or at least one experimental group of the set A / B test; and one of the users corresponding to the tester identification information and already assigned to the at least one experimental group is reassigned from the at least one experimental group to the control group.

12. A non-transitory computer-readable recording medium recording a program for performing the method as described in claim 1 on a computer.

Images