Error handling method and system therefor
The method and system for handling software errors by selectively pausing relevant A/B tests address the challenge of service interruptions, ensuring efficient error management and minimizing disruptions in software applications, thereby enhancing operational efficiency and user experience.
Patent Information
- Application Number
- PCT/KR2023/006905
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-12
- Filing Date
- 2023-05-22
- Publication Date
- 2024-11-21
AI Technical Summary
Existing software error handling methods are inadequate for quickly resolving faults in software applications, leading to potential service interruptions and significant losses for operators and users, especially in applications serving multiple users on smartphones.
A method and system that receive error information in target software, select relevant A/B tests based on error occurrence history and function call information, and pause specific tests to minimize service disruptions, allowing for temporary suspension and resumption of tests to manage errors without terminating them entirely.
This approach effectively reduces service outages by pausing only affected A/B tests, maintaining overall software functionality, and minimizes software deployments causing interruptions, while improving operational efficiency by using pause actions over termination, allowing for accurate error handling and gradual software updates.
Smart Images

Figure KR2023006905_21112024_PF_FP_ABST
Abstract
Description
Error handling method and system
[0001] The present disclosure relates to a method and system for handling errors occurring in software.
[0002] Typically, various faults or errors occur during software use. Because these errors are difficult to completely resolve through testing (e.g., functional testing, integration testing, etc.) conducted during the software development phase, the ability to quickly handle errors during software maintenance is crucial.
[0003] Meanwhile, it's become common for many businesses to provide services to a large number of users through applications installed on smartphones. However, if errors occurring in these applications aren't promptly resolved, they can lead to service interruptions, resulting in significant losses for both businesses and users.
[0004] A technical problem to be solved through some embodiments of the present disclosure is to provide a method for effectively handling a fault or error occurring in target software and a system for performing the method.
[0005] Another technical challenge that some embodiments of the present disclosure seek to address is to provide a method for incrementally updating target software while minimizing error occurrence and service interruption.
[0006] The technical problems of the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art of the present disclosure from the description below.
[0007] According to some embodiments of the present disclosure for solving the above-described technical problem, an error handling method may be performed by at least one computing device, and may include: receiving information about an error occurring in target software; selecting, in response to the receiving, a test associated with the error from among a plurality of ongoing tests, wherein the plurality of tests are A / B tests performed on modules in the target software; and pausing at least some of the selected tests.
[0008] In some embodiments, the step of selecting a test associated with the error may include selecting the associated test based on an error occurrence history of the plurality of tests.
[0009] In some embodiments, the received error information includes function call information, and the step of selecting a test associated with the error may include the step of identifying a module in the target software associated with the error based on the function call information, and the step of selecting a test targeting the identified module among the plurality of tests.
[0010] In some embodiments, the target software is an application running on a user terminal, and the step of selecting a test associated with the error may include a step of selecting the associated test based on a determination that the occurrence rate of the error relative to the number of active users using the application is greater than or equal to a reference value.
[0011] In some embodiments, the step of selecting a test associated with the error may be performed when the severity level of the error is below a reference level, and may further include a step of pausing all of the plurality of tests based on a determination that the severity level of the error is above the reference level.
[0012] In some embodiments, the step of suspending may include the step of suspending a test among the selected tests whose correlation with the error is greater than or equal to a threshold value, and the step of withholding the suspension action for a test whose correlation with the error is less than or equal to the threshold value and providing test information to an administrator.
[0013] In some embodiments, a particular test among the selected tests is performed on an existing module and a modified module of the target software, and the pausing step may include a step of deactivating the modified module.
[0014] In some embodiments, the target software is an application running on a user terminal, and the step of disabling the modified module may include a step of transmitting a control command to a terminal of a user exposed to the modified module among users participating in the specific test to disable the modified module and activate the existing module.
[0015] In some embodiments, the method may further include the step of resuming the paused test in response to a preset period of time elapsed.
[0016] In some embodiments, the method may further include, in response to the first time period elapsed, resuming the paused test; determining, if the error occurs again, whether the number of pauses associated with the error is less than or equal to a preset maximum number; and re-pausing the resumed test for a second time period that is equal to or longer than the first time period based on the determination that the number of pauses is less than or equal to the maximum number.
[0017] According to some embodiments of the present disclosure for solving the above-described technical problem, an error handling system includes a memory storing one or more processors and a computer program executed by the one or more processors, wherein the computer program may include instructions for receiving information about an error occurring in target software, selecting a test associated with the error from among a plurality of ongoing tests in response to the receiving, wherein the plurality of tests are A / B tests performed on modules in the target software, and pausing at least some of the selected tests.
[0018] According to some embodiments of the present disclosure for solving the above-described technical problem, a computer program may be stored in a computer-readable recording medium to be combined with a computing device and execute the steps of: receiving information about an error occurring in target software; selecting a test associated with the error from among a plurality of ongoing tests in response to the receiving of the information; wherein the plurality of tests are A / B tests conducted on modules in the target software; and pausing at least some of the selected tests.
[0019] According to some embodiments of the present disclosure, various A / B tests can be conducted on modules within the target software, and when an error occurs, actions such as pausing the associated A / B test can be taken. In this case, since error handling can be performed without significantly affecting the overall function of the target software, the service interruption problem can be significantly alleviated (because only some associated A / B tests, not the entire A / B test, are paused, and most functions are provided through the existing modules even if the A / B test is paused). Furthermore, by directly reflecting the modified module into the target software upon completion of the A / B test, the number of software deployments that cause service interruptions can be minimized.
[0020] Additionally, by adopting a pause action instead of a test termination (or interruption) action as the default mechanism for error handling, the operational efficiency of A / B testing can be utilized (e.g., excessive termination of A / B tests, duplicate execution of A / B tests with the same purpose, etc. are prevented).
[0021] Additionally, the A / B test associated with the error can be accurately selected by using the error occurrence history for each A / B test and the function call information related to the error.
[0022] Additionally, error handling can be made more efficient by performing differential actions depending on the type of error (e.g., general error, critical error), severity level, etc.
[0023] The effects according to the technical idea of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.
[0024] FIG. 1 is an exemplary diagram schematically illustrating an error handling system and its operating environment according to some embodiments of the present disclosure.
[0025] FIG. 2 and FIG. 3 are exemplary drawings for further explaining the operating environment of an error handling system according to some embodiments of the present disclosure.
[0026] FIG. 4 is an exemplary flowchart illustrating an error handling method according to some embodiments of the present disclosure.
[0027] FIG. 5 is an exemplary diagram illustrating a method for selecting a related test according to some embodiments of the present disclosure.
[0028] FIG. 6 is an exemplary diagram illustrating an implementation method of a test pause action according to some embodiments of the present disclosure.
[0029] Figure 7 is an exemplary flowchart showing the detailed process of the action execution step illustrated in Figure 4.
[0030] FIG. 8 is an exemplary flowchart illustrating an error handling method according to some other embodiments of the present disclosure.
[0031] Figure 9 is an exemplary flowchart showing the detailed process of the important error processing step illustrated in Figure 8.
[0032] Figure 10 is an exemplary drawing to further explain the detailed process of the important error processing step illustrated in Figure 8.
[0033] FIG. 11 illustrates an exemplary computing device that may implement an error handling system according to some embodiments of the present disclosure.
[0034] Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the attached drawings. The advantages and features of the present disclosure, and methods for achieving them, will become clear with reference to the embodiments described in detail below together with the attached drawings. However, the technical idea of the present disclosure is not limited to the following embodiments and may be implemented in various different forms. The following embodiments are provided only to complete the technical idea of the present disclosure and to fully inform those skilled in the art of the present disclosure of the scope of the present disclosure, and the technical idea of the present disclosure is defined only by the scope of the claims.
[0035] In describing various embodiments of the present disclosure, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present disclosure, the detailed description will be omitted.
[0036] Unless otherwise defined, the terms (including technical and scientific terms) used in the following examples may be used with meanings commonly understood by those of ordinary skill in the art to which this disclosure pertains; however, this may vary depending on the intentions of engineers working in the relevant field, precedents, the emergence of new technologies, etc. The terminology used in this disclosure is for the purpose of describing the embodiments and is not intended to limit the scope of this disclosure.
[0037] In the following examples, singular expressions include plural concepts unless the context clearly specifies that they are singular. Furthermore, plural expressions include singular concepts unless the context clearly specifies that they are plural.
[0038] In addition, terms such as first, second, A, B, (a), (b), etc. used in the following embodiments are only used to distinguish certain components from other components, and the nature, order, or sequence of the components are not limited by the terms.
[0039] Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the attached drawings.
[0040] FIG. 1 is an exemplary drawing schematically illustrating an error handling system (10) and its operating environment according to some embodiments of the present disclosure.
[0041] As illustrated in FIG. 1, the error handling system (10) according to the embodiments can operate in an environment in which a plurality of user terminals (12-1 to 12-N) and an administrator terminal (11) exist. However, FIG. 1 assumes that the error handling system (10) processes (handles) errors occurring in software (e.g., apps) installed in the user terminals (12-1 to 12-N). In other words, FIG. 1 assumes that the target software is software (e.g., applications) installed in the user terminals (12-1 to 12-N). Accordingly, if the target software is software of a device (e.g., server-side device) other than the user terminal (e.g., 12-1), the user terminal (e.g., 12-1) illustrated in FIG. 1 may be changed to the corresponding device. That is, in some other embodiments, the error handling system (10) may perform error processing on server-side devices.
[0042] Hereinafter, for convenience of explanation, the reference number '12' will be used both when referring to any user terminal (e.g., 12-1 or 12-2) or when referring to all user terminals (12-1 to 12-N), and the error processing system (10) will be abbreviated as 'processing system (10)'.
[0043] The processing system (10) may refer to a computing device / system equipped with an error handling function. For example, the processing system (10) may receive information about errors occurring in target software in real time from user terminals (12) and take appropriate action on the errors based on the received information.
[0044] For reference, a fault may be termed as an error, bug, defect, failure, issue, crash, incident, etc. depending on the viewpoint, and the term 'fault' used in this disclosure can be understood to encompass all concepts of the exemplified terms.
[0045] According to various embodiments of the present disclosure, the processing system (10) can perform error handling in an environment where A / B testing of target software is conducted. For better understanding, this will be further explained with reference to FIGS. 2 and 3.
[0046] As illustrated in FIG. 2, various A / B tests (21-1 to 21-K) can be conducted on modules within the target software. The management entity of these A / B tests (21-1 to 21-K) may be the processing system (10) or another device. However, the following description will continue on the assumption that the management entity of the A / B tests is the processing system (10). In addition, for the convenience of explanation, the reference number '21' will be used both when referring to an arbitrary A / B test (e.g., 21-1 or 21-2) or when referring to all A / B tests (21-1 to 21-K) in general.
[0047] The processing system (10) can conduct (perform) an A / B test (21) on two or more modules (e.g., module A, module B, module C, etc.) within the target software. At this time, module A refers to an existing module (e.g., a module providing an existing function, a module whose reliability has been verified, a module that constitutes the current version of the target software, etc.), and modules B and C may refer to modified modules compared to the existing modules (e.g., a module providing a new function, a module requiring reliability verification, a module to be installed in the next version of the target software, etc.), but the scope of the present disclosure is not limited thereto. In the following, for the convenience of understanding, the explanation will continue by assuming that an A / B test is conducted on two modules (i.e., modules A and B), and that modules A and B are 'existing modules' and 'modified modules', respectively.
[0048] The processing system (10) can conduct an A / B test (21) by exposing module A to a first user group and module B to a second user group. For example, the processing system (10) can control the terminals of the first user group (e.g., 12-1) to execute the target software with module A activated, and control the terminals of the second user group (e.g., 12-2) to execute the target software with module B activated (e.g., when distributing a specific version of the target software, modules A and B are loaded and distributed together, and each module can be activated or deactivated through a control command of the processing system (10). Through such control, each module can be exposed to the corresponding user group. However, the scope of the present disclosure is not limited to this method.
[0049] Meanwhile, the method for determining the first and second user groups can be designed in various ways. For example, the processing system (10) can set the ratio of the second user group to be lower than that of the first user group (because errors typically occur in module B, which is a transformation module) and gradually increase the ratio of the second user group until the ratios of the two user groups become equal. As another example, the initial values for the ratios of the first and second user groups can be set to appropriate values by the administrator of the A / B test (21).
[0050] In some cases, the processing system (10) may adjust the ratio of the two user groups based on the degree of error occurrence. For example, if the degree of error occurrence (e.g., number of error occurrences, rate of increase in number, etc.) of a specific A / B test (21) is above a reference value, the processing system (10) may decrease the ratio of the second user group for the corresponding A / B test (21) (because errors usually occur in module B, which is a modified module), and in the opposite case, may increase the ratio of the second user group for the purpose of quickly completing the corresponding A / B test (21).
[0051] The above-described A / B test (21) can be performed for various purposes. For example, the A / B test (21) can be performed for purposes such as verifying new functions (or modules) (e.g., verifying reliability), confirming user reactions, etc. However, the scope of the present disclosure is not limited thereto, and the purpose of the A / B test, the dependent variable of the A / B test, etc. can vary greatly. Those skilled in the art are likely already familiar with the concept of A / B testing, and therefore, further explanation thereof will be omitted.
[0052] As illustrated in FIG. 2, the processing system (10) can manage error occurrence histories (22-1 to 22-K) for each A / B test (21). That is, the processing system (10) can monitor and manage information regarding what type of error occurred and to what extent in a specific A / B test (21). The error occurrence histories (22-1 to 22-K) can be used to select an A / B test (21) associated with a specific error, which will be described later with reference to FIG. 4 and the like.
[0053] Error occurrence history (e.g., 22-1) may include information such as the name of the error, type, identifier, type (e.g., general error, critical error, etc.), severity level, function (method) call information such as stack trace, number of occurrences (frequency), occurrence ratio (e.g., number of occurrences of a specific error compared to the total number of occurrences of errors, number of users who experienced an error compared to the total number of users who participated in the test), occurrence time, etc., but the scope of the present disclosure is not limited thereto.
[0054] In the environment illustrated in FIG. 2, the processing system (10) can perform error processing by receiving error information in real time from the user terminal (12) and pausing the A / B test (21) associated with the error. For example, as illustrated in FIG. 3, let us assume that the target software is an application (31) installed on the user terminal (12) and that an A / B test is being conducted targeting the A module (32) and the B module (33). In addition, let us assume that the user terminal (12) exposes the B module (33) to the user (i.e., the user of the terminal (12) is a second user group). In this case, the processing system (10) can pause the A / B test by transmitting a control command (35) to the user terminal (12) to deactivate the B module (33) in response to receiving the error information (34). However, the scope of the present disclosure is not limited thereto. In some cases, the processing system (10) may deactivate both the A module (32) and the B module (33) via a control command (35).
[0055] Note that a pause differs from a termination (or suspension) action in that it ensures the resumption of an A / B test (i.e., a pause can ensure the continuity of the A / B test). Therefore, utilizing a pause can prevent excessive termination of A / B tests and duplicate A / B tests, significantly improving the operational efficiency of A / B tests.
[0056] Upon completion of the A / B test (21), the processing system (10) may reflect the B module in the target software based on the test results (e.g., continuously activating the B module via a control command). In this case, the target software can be gradually updated without going through a software deployment process that causes service interruption (i.e., the number of service interruptions, software deployments, etc. can be minimized), and the user can avoid the inconvenience of repeatedly downloading new versions of the target software.
[0057] This is explained again with reference to Figure 1.
[0058] The processing system (10) described above may be implemented with at least one computing device. For example, all functions of the processing system (10) may be implemented in a single computing device, or a first function of the processing system (10) may be implemented in a first computing device and a second function may be implemented in a second computing device. Alternatively, specific functions of the processing system (10) may be implemented in multiple computing devices.
[0059] A computing device may include any device having computing capabilities, and for an example of such a device, see FIG. 11.
[0060] A computing device is a collection of interacting components (e.g., memory, processor, etc.), and so may sometimes be referred to as a "computing system." Of course, the term "computing system" can also encompass the concept of a collection of interacting computing devices.
[0061] The administrator terminal (11) may refer to a terminal of a person who manages the target software or a service provided through the target software. The administrator terminal (11) may be implemented using any computing device. The administrator may be, for example, a developer, a tester, a test manager, a service provider, etc., but the scope of the present disclosure is not limited thereto.
[0062] The administrator can perform error management, such as designating critical errors and defining / designating / changing severity levels, through the terminal (11), and can also manage A / B tests (e.g., designing, creating, conducting, terminating, etc.). Furthermore, the administrator can receive various types of information, such as error information and target software information, from the processing system (10) through the terminal (11).
[0063] Next, the user terminal (12) may refer to a user terminal utilizing the target software. The user terminal (12) may be, for example, a mobile terminal such as a smartphone or tablet. However, the scope of the present disclosure is not limited thereto, and the user terminal (12) may also be a fixed terminal such as a desktop. In other words, the user terminal (12) may be implemented as a computing device of various types.
[0064] Users naturally participate in A / B testing while using the target software installed on the terminal (12), and through these processes, service interruption can be minimized while the target software is being updated.
[0065] As illustrated in FIG. 1, a user terminal (12) and a processing system (10) and / or an administrator terminal (11) and a processing system (10) can communicate via a network. Here, the network can be implemented as any type of wired / wireless network, such as a local area network (LAN), a wide area network (WAN), a mobile radio communication network, Wibro (Wireless Broadband Internet), etc.
[0066] So far, the processing system (10) and its operating environment according to some embodiments of the present disclosure have been schematically described with reference to FIGS. 1 to 3. Hereinafter, various methods performed for error processing in the processing system (10) will be described with reference to the drawings including FIG. 4 and below.
[0067] Hereinafter, for ease of understanding, all steps / operations of the methods to be described below will be described assuming that they are performed in the processing system (10) of the environment illustrated in FIG. 1. Accordingly, if the subject of a specific step / operation is omitted, it can be understood that it is performed in the processing system (10). However, in an actual environment, some steps / operations of the methods to be described below may be performed in other computing devices.
[0068] Figure 4 is an exemplary flowchart illustrating an error handling method according to some embodiments of the present disclosure. However, this is merely a preferred embodiment for achieving the purpose of the present disclosure, and it is understood that some steps may be added or deleted as needed.
[0069] As illustrated in FIG. 4, the present embodiments may begin with step S41, which receives information about an error occurring in the target software. For example, the processing system (10) may receive error information (e.g., log information about the error) in real time from multiple user terminals (12). The method by which the user terminals (12) report the error information may be any method.
[0070] Error information may include, for example, the name, type, type, identifier, severity level of the error, function (method) call information such as a stack trace, target software information (e.g., release / distribution information of the corresponding version), etc. However, the scope of the present disclosure is not limited thereto.
[0071] In step S42, it can be determined whether the degree of error occurrence exceeds a threshold. For example, the processing system (10) can determine whether the degree of error occurrence exceeds a threshold. However, the specific method for determining the degree of error occurrence may vary depending on the embodiment.
[0072] In some embodiments, it may be determined whether the number of times an error occurs (frequency) is greater than or equal to a threshold. For example, the processing system (10) may determine whether the number of times a specific error occurs (frequency) per unit time is greater than or equal to a threshold.
[0073] In some other embodiments, it may be determined whether the error occurrence rate relative to the number of active users (e.g., the number of users currently using the target software) is greater than or equal to a threshold. For example, the processing system (10) may determine whether the number of times a specific error occurs (e.g., the number of times an error occurs per unit time) relative to the total number of active users is greater than or equal to a threshold. Alternatively, the processing system (10) may determine whether the number of users experiencing the error relative to the total number of active users is greater than or equal to a threshold. In such cases, by considering the number of active users currently using the target software, the degree of error occurrence can be determined more accurately.
[0074] In some other embodiments, it may be determined whether the duration of an error occurrence exceeds a threshold. For example, the processing system (10) may determine whether the number of time intervals during which the error occurs continuously (or consecutively) exceeds a threshold. Here, the length of a unit time interval may be predetermined (e.g., 1 hour, etc.).
[0075] In some other embodiments, the extent of error occurrence may be determined based on various combinations of the aforementioned embodiments. For example, the processing system (10) may determine whether the error occurrence rate relative to the number of active users is greater than or equal to a first threshold and whether the duration of error occurrence is greater than or equal to a second threshold.
[0076] The above-described threshold may be a preset fixed value or a value that varies depending on the situation. For example, the threshold may be a value that varies based on the time zone, the load of the processing system (10), the number of active users, the elapsed time since the target software was distributed, etc. As a more specific example, the threshold may be lowered during times with a large number of active users (e.g., daytime, evening time, etc.), and may be raised during opposite times (e.g., early morning time, etc.). As another example, when the elapsed time from the time of distribution of the target software (e.g., a new version of the software) is not long (i.e., less than a certain amount of time), the threshold may be lowered in order to take prompt action against various errors. In the opposite case, the threshold may be raised.
[0077] In some cases, the above-described step S42 may be omitted (e.g., when the reference value is set to '0'). In such cases, the processing system (10) may operate to perform actions corresponding to the error whenever an error occurs.
[0078] In step S43, a test associated with the error may be selected from among the ongoing A / B tests. However, the specific method for selecting the associated test may vary depending on the embodiment.
[0079] In some embodiments, related tests may be selected based on the error occurrence history of each A / B test. In other words, the correlation between the error occurrence history of the A / B test and the A / B test may be determined based on the error occurrence history of the A / B test. Specifically, as illustrated in FIG. 5 , the processing system (10) may determine the correlation between the A / B test (51-1 to 51-K) and the error (53) by referring to the error occurrence history (52-1 to 52-K) of the A / B tests (51-1 to 51-K). For example, the processing system (10) can determine the correlation between a specific A / B test (e.g., 52-1) and the corresponding error (53) based on the type, number of times, and rate of errors that occurred in the specific A / B test (e.g., 52-1) (e.g., the correlation is determined to be a higher value as the number of times or rate of occurrence of the corresponding error (53) occurs in the specific A / B test is higher, the number of users who have occurred the corresponding error (53) in the specific A / B test is higher or the rate is higher, the correlation is determined to be a higher value). In addition, the processing system (10) can select an A / B test with a correlation higher than a reference value as a test associated with the corresponding error (53).
[0080] In some other embodiments, associated tests may be selected based on the function call information associated with the error. Specifically, the processing system (10) may identify a module within the target software associated with the error (e.g., a module that called the function) based on the function call information associated with the error. Furthermore, the processing system (10) may select an A / B test targeting the identified module as the test associated with the error.
[0081] In some other embodiments, the scope of the A / B test associated with the error may vary depending on the type of the error (e.g., general error, critical error). For example, if the error is not a critical error, the processing system (10) may select an A / B test whose correlation with the error is greater than a first threshold as the relevant test. Conversely, if the error is a critical error, the processing system (10) may select an A / B test whose correlation with the error is greater than a second threshold but lower than the first threshold as the relevant test (i.e., the scope of the relevant test is expanded).
[0082] In some other embodiments, the scope of the A / B test associated with the error may vary depending on the severity level of the error. For example, if the severity level of the error is lower than a reference level, the processing system (10) may select an A / B test having a correlation with the error greater than or equal to a first reference level as the associated test. Conversely, if the severity level of the error is higher than or equal to the reference level, the processing system (10) may select an A / B test having a correlation with the error greater than or equal to a second reference level that is lower than the first reference level as the associated test (i.e., the scope of the associated test is expanded).
[0083] In some other embodiments, association tests may be selected based on various combinations of the embodiments described above.
[0084] In step S44, preset actions may be performed on the selected A / B test. However, the detailed process of this step may vary depending on the embodiment.
[0085] In some embodiments, a pause action may be taken for a selected A / B test. For example, as illustrated in FIG. 6, assume that the selected A / B test (61) is a test performed on module A (63) and module B (64). In this case, the processing system (10) may transmit a control command to a terminal (12) of a user who is exposed to module B (64) among the users participating in the selected A / B test (61), to deactivate module B (64) and activate module A (63). As a result, the selected A / B test (61) may be paused.
[0086] In some other embodiments, different actions may be taken depending on the correlation between the error and the selected A / B test. For example, as illustrated in FIG. 7, the processing system (10) may perform a pause action for a test among the selected A / B tests whose correlation is higher than a threshold value (e.g., a value higher than the threshold value of FIG. 4) (S71, S72). In addition, the processing system (10) may withhold the pause action for a test whose correlation is lower than the threshold value and provide only test information (e.g., a list of A / B tests, etc.) to the administrator (S71, S73). The processing system (10) may provide the test information to the administrator through various collaboration channels (e.g., messenger, bulletin board, SMS, phone, etc.), but the scope of the present disclosure is not limited thereto. As another example, the processing system (10) may perform a termination action for a test among the selected A / B tests whose correlation is higher than a threshold value and a pause action for a test whose correlation is lower than the threshold value. As another example, the processing system (10) may perform an action of pausing for a first time period for a test among the selected A / B tests whose correlation is higher than a threshold value, and may perform an action of pausing for a second time period shorter than the first time period for a test whose correlation is lower than the threshold value (i.e., the pause time may be determined based on the correlation). In some cases, the pause time may be determined based on the severity level of the corresponding error (e.g., the higher the severity level, the longer the pause time). According to the present embodiments, error handling can be performed more efficiently by performing differential actions depending on the type of error (e.g., general error, critical error).
[0087] In some other embodiments, differential actions may be taken based on the severity level of the error. See the description in Figure 9 for details. According to these embodiments, by taking differential actions based on the severity level of the error, the efficiency of error handling can be improved.
[0088] In some other embodiments, differential actions may be taken based on the completion time of the selected A / B test. For example, the processing system (10) may terminate a first test among the selected A / B tests with a short remaining time until completion, and suspend a second test with a long remaining time. The completion time of an A / B test may be determined based on, for example, the test execution (elapsed) time, the difference in test dependent variable values for Module A and Module B, etc., but the scope of the present disclosure is not limited thereto.
[0089] In some other embodiments, the occurrence of the error can be monitored while alternately pausing and resuming the selected A / B tests. This process allows the A / B tests and modules closely related to the error to be accurately and quickly identified. As a more specific example, the processing system (10) can pause a first test among the selected A / B tests, monitor the occurrence of the error, resume the first test, and then pause the second test while monitoring the occurrence of the error. As another example, the processing system (10) can pause all selected A / B tests, then resume the first test, and monitor the occurrence of the error. Then, the processing system (10) can pause the first test again, resume the second test, and monitor the occurrence of the error.
[0090] In some other embodiments, actions corresponding to errors that occur may be taken based on various combinations of the embodiments described above.
[0091] Meanwhile, according to some embodiments of the present disclosure, a resume action may be performed in response to a predetermined condition being satisfied after a pause action. For example, the processing system (10) may resume a paused A / B test in response to the elapse of a preset period of time (i.e., a pause time). At this time, the pause time may be a preset fixed value (e.g., 10 minutes) or a value that may vary depending on the situation. For example, the pause time may be set to an appropriate value considering the time required for error analysis, the importance of the A / B test, etc. As another example, the pause time may be adjusted based on the extent to which the error has occurred since the pause action (e.g., if the same error continues to occur, the pause time may be reduced because the relevance of the paused A / B test to the error may be reduced, and conversely, the pause time may be increased). In some cases, the processing system (10) may also resume a paused A / B test in response to an administrator's pause release command.
[0092] Additionally, in some embodiments, a maximum number of pauses may be preset. The maximum number may or may not be set differently for each error, error type, and severity level. In the present embodiments, if the same error occurs again, the processing system (10) may determine whether the number of pauses for the error is less than or equal to the preset maximum number. Based on the determination that the number of pauses is less than or equal to the maximum number, the processing system (10) may again pause the A / B test associated with the error. At this time, the pause time may be set to be longer than or equal to the previous pause time. This is because if the error was not resolved during the previous pause time, it is likely a complex error requiring considerable processing time.
[0093] Meanwhile, while Figure 4 depicts that no action is taken when the error occurrence rate is below a threshold, this can be varied as needed. For example, even when the error occurrence rate is below a threshold, the processing system (10) may provide the administrator with information about the error, information about associated A / B tests, etc.
[0094] Heretofore, error handling methods according to some embodiments of the present disclosure have been described with reference to FIGS. 4 to 7. As described above, various A / B tests can be conducted on modules within the target software, and when an error occurs, actions such as pausing the associated A / B test can be taken. In this case, error handling can be performed without significantly affecting the overall function of the target software, so that the service interruption problem can be significantly alleviated (because only some associated A / B tests, not the entire A / B test, are paused, and even if the A / B test is paused, most functions are provided through the existing modules). Furthermore, by directly reflecting the modified module into the target software upon completion of the A / B test, the number of software deployments that cause service interruptions can be minimized.
[0095] Additionally, by adopting a pause action instead of a test termination (or interruption) action as the default mechanism for error handling, the operational efficiency of A / B tests can be significantly improved.
[0096] Additionally, the A / B test associated with the error can be accurately selected by using the error occurrence history for each A / B test and the function call information related to the error.
[0097] Hereinafter, error handling methods according to several other embodiments of the present disclosure will be described with reference to FIGS. 8 to 10. However, for clarity of the present disclosure, descriptions of content overlapping with previous embodiments will be omitted.
[0098] FIG. 8 is an exemplary flowchart illustrating an error handling method according to some other embodiments of the present disclosure.
[0099] As illustrated in FIG. 8, the present embodiments may also begin with step S81, which receives information about an error occurring in the target software. For this step, please refer to the description of step S41 described above.
[0100] In step S82, it may be determined whether the error that occurred corresponds to a pre-specified critical error. For example, the processing system (10) may determine whether the type (or grade) of the error that occurred corresponds to a pre-specified critical error. If it is determined that the error does not correspond to a critical error, step S83 may be performed, and if not, step S84 may be performed.
[0101] Meanwhile, the way to specify critical errors can be designed in various ways.
[0102] For example, a critical error may be designated by an administrator. An administrator may designate a critical error as an error that occurs repeatedly in the target software, an error that critically impacts the function (or user experience) of the target software, etc., but the scope of the present disclosure is not limited thereto.
[0103] As another example, critical errors can be automatically designated based on satisfying preset conditions. For example, the processing system (10) can designate as critical errors errors that occur more frequently than a threshold, errors that occur at a rate greater than a threshold (e.g., errors occurring in a majority of users), errors received through user communication channels (e.g., bulletin boards, messengers, SNS, etc.), errors related to core functions of the target software, and errors for which a temporary suspension action has been taken at least once.
[0104] As another example, critical errors may be designated based on various combinations of the examples described above. For example, the processing system (10) may designate as critical errors those errors among the errors designated by the manager that satisfy preset conditions.
[0105] In step S83, an error occurring according to the first error handling algorithm may be processed. For example, the processing system (10) may select a related A / B test and perform a preset action as illustrated in FIG. 4.
[0106] In step S84, an error occurring may be processed according to a second error handling algorithm. Here, the second error handling algorithm may be defined at least partially differently from the first error handling algorithm. The detailed process of step S84 (i.e., the second error handling algorithm) is illustrated in FIG. 9. The following description will be made with reference to FIG. 9.
[0107] In step S91, it can be determined whether the occurrence of an error (i.e., a critical error) exceeds a threshold. For this step, please refer to the description of step S42 described above. This step may be omitted in some cases.
[0108] In step S92, it can be determined whether the severity level of the error is higher than or equal to the reference level. If the severity level of the error is determined to be lower than the reference level, step S93 may be performed. If not, step S94 may be performed. The severity level may be defined in any form.
[0109] In step S93, information about the error may be provided to the administrator. For example, if the error is a minor error, the processing system (10) may complete error processing by providing error information to the administrator.
[0110] In step S94, preset actions may be performed on at least some of the ongoing A / B tests. However, the details of this step may vary depending on the embodiment.
[0111] In some embodiments, tests associated with the error may be selected from among ongoing A / B tests, and a pause action may be taken for the selected tests. For example, if the error is not a critical error, the processing system (10) may select an A / B test with a correlation greater than or equal to a first threshold as the relevant test, and if the error is a critical error, the processing system (10) may select an A / B test with a correlation greater than or equal to a second threshold but lower than the first threshold as the relevant test. By doing so, more stringent action may be taken for critical errors (because more A / B tests are selected as the relevant tests).
[0112] In some other embodiments, a pause may be performed on all A / B tests that use the same version of software as the target software in which the error occurred, without determining the correlation with the error. For example, as illustrated in FIG. 10 , the processing system (10) may select, among ongoing AB tests (102-1 to 102-K), tests (e.g., 102-1, 102-K) that use the same version of software as the target software in which the error occurred (S101). Then, the processing system (10) may perform a pause on all of the selected tests (e.g., 102-1, 102-K). By doing so, more stringent measures may be taken for critical errors.
[0113] In some other embodiments, the processing system (10) may identify a first module (e.g., module A, module B) associated with the error among ongoing A / B tests. Here, the first module may be identified through function call information or may be a module of a related A / B test. Next, the processing system (10) may identify a second module associated with the first module. Here, the second module may be, for example, a module that has a dependency with the first module, a module used on the same app page (or app screen) as the first module, a module that provides the same function (e.g., payment function, etc.) as the first module, etc. However, the scope of the present disclosure is not limited thereto. Next, the processing system (10) may perform a pause action on A / B tests targeting the first module or the second module. By doing so, more stringent measures can be taken for critical errors.
[0114] In some other embodiments, a pause action may be taken on all ongoing A / B tests.
[0115] Error handling methods according to several other embodiments of the present disclosure have been described with reference to FIGS. 8 through 10. As described above, error handling can be more efficiently achieved by performing differential actions depending on the type of error (e.g., general error, critical error).
[0116] Hereinafter, an exemplary computing device (110) capable of implementing the above-described processing system (10) will be described with reference to FIG. 11.
[0117] Figure 11 is an exemplary hardware configuration diagram showing a computing device (110).
[0118] As illustrated in FIG. 11, a computing device (110) may include one or more processors (111), a bus (113), a communication interface (114), a memory (112) for loading a computer program (116) executed by the processor (111), and a storage (115) for storing the computer program (116). However, only components related to the embodiment of the present disclosure are illustrated in FIG. 11. Therefore, a person skilled in the art to which the present disclosure pertains will appreciate that other general components may be included in addition to the components illustrated in FIG. 11. That is, the computing device (110) may further include various components in addition to the components illustrated in FIG. 11. In addition, in some cases, the computing device (110) may be configured in a form in which some of the components illustrated in FIG. 11 are omitted. Hereinafter, each component of the computing device (110) will be described.
[0119] The processor (111) can control the overall operation of each component of the computing device (110). The processor (111) can be configured to include at least one of a CPU (Central Processing Unit), an MPU (Micro Processor Unit), an MCU (Micro Controller Unit), a GPU (Graphics Processing Unit), an NPU (Neural Processing Unit), or any other type of processor well known in the technical field of the present disclosure. In addition, the processor (111) can perform operations for at least one application or program for executing operations / methods according to embodiments of the present disclosure. The computing device (110) can include one or more processors.
[0120] Next, the memory (112) can store various data, commands, and / or information. The memory (112) can load a computer program (116) from the storage (115) to execute operations / methods according to embodiments of the present disclosure. The memory (112) may be implemented as a volatile memory such as RAM, but the technical scope of the present disclosure is not limited thereto.
[0121] Next, the bus (113) can provide a communication function between components of the computing device (110). The bus (113) can be implemented as various types of buses such as an address bus, a data bus, and a control bus.
[0122] Next, the communication interface (114) can support wired and wireless Internet communication of the computing device (110). Furthermore, the communication interface (114) can also support various communication methods other than Internet communication. To this end, the communication interface (114) can be configured to include a communication module well known in the technical field of the present disclosure.
[0123] Next, the storage (115) can non-temporarily store one or more computer programs (116). The storage (115) can be configured to include non-volatile memory such as Read Only Memory (ROM), Erasable Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), flash memory, a hard disk, a removable disk, or any form of computer-readable recording medium well known in the art to which the present disclosure pertains.
[0124] Next, the computer program (116) may include instructions that, when loaded into the memory (112), cause the processor (111) to perform operations / methods according to various embodiments of the present disclosure. That is, the processor (111) may perform operations / methods according to various embodiments of the present disclosure by executing the loaded instructions.
[0125] For example, the computer program (116) may include instructions that cause the computer program to perform operations such as receiving information about an error occurring in the target software, selecting a test associated with the error from among a plurality of ongoing tests targeting modules within the target software, and pausing at least some of the selected tests. In such a case, the processing system (10) according to some embodiments of the present disclosure may be implemented via the computing device (110).
[0126] Meanwhile, in some embodiments, the computing device (110) illustrated in FIG. 11 may refer to a virtual machine implemented based on cloud technology. For example, the computing device (110) may be a virtual machine operating on one or more physical servers included in a server farm. In this case, at least some of the processor (111), memory (112), and storage (115) illustrated in FIG. 11 may be virtual hardware, and the communication interface (114) may also be implemented as a virtualized networking element, such as a virtual switch.
[0127] So far, an exemplary computing device (110) capable of implementing a processing system (10) according to some embodiments of the present disclosure has been described with reference to FIG. 11.
[0128] Various embodiments of the present disclosure and effects according to the embodiments have been described with reference to FIGS. 1 to 11. The effects according to the technical concept of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.
[0129] Furthermore, even though the above embodiments have described multiple components as being combined or operating in combination, the technical concept of the present disclosure is not necessarily limited to these embodiments. That is, within the scope of the technical concept of the present disclosure, all of the components may be selectively combined and operated one or more times.
[0130] The technical concepts of the present disclosure described so far can be implemented as computer-readable code on a computer-readable medium. A computer program recorded on a computer-readable recording medium can be transmitted to another computing device via a network such as the Internet, installed on said other computing device, and thus used on said other computing device.
[0131] Although the operations are depicted in a specific order in the drawings, it should not be understood that the operations must be performed in the specific order depicted, or in a sequential order, or that all depicted operations must be performed to achieve the desired result. In certain circumstances, multitasking and parallel processing may be advantageous. Although various embodiments of the present disclosure have been described above with reference to the attached drawings, those skilled in the art to which the present disclosure pertains will understand that the technical concepts of the present disclosure can be implemented in other specific forms without changing the technical concepts or essential characteristics thereof. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. The scope of protection of the present disclosure should be interpreted by the claims below, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of the technical ideas defined by the present disclosure.
Claims
1. A method performed by at least one computing device, A step of receiving information about an error that occurred in the target software; In response to the above receipt, a step of selecting a test associated with the error from among a plurality of ongoing tests, wherein the plurality of tests are A / B tests conducted on modules within the target software; and Comprising a step of pausing at least some of the above selected tests, How to handle errors.
2. In paragraph 1, The step of selecting the test associated with the above error is: A step of selecting the related test based on the error occurrence history of the plurality of tests, How to handle errors.
3. In paragraph 1, The above received error information includes function call information, The step of selecting the test associated with the above error is: A step of identifying a module within the target software associated with the error based on the function call information; and A step of selecting a test targeting the identified module from among the plurality of tests, How to handle errors.
4. In paragraph 1, The above target software is an application running on a user terminal, The step of selecting the test associated with the above error is: A step of selecting the related test based on a judgment that the occurrence rate of the error is greater than a reference value compared to the number of active users using the application, How to handle errors.
5. In paragraph 1, The step of selecting the test associated with the above error is: A step of selecting a test whose correlation with the error is greater than or equal to a first criterion as a related test based on a judgment that the error does not correspond to a pre-specified critical error; and A step of selecting a test whose correlation with the error is higher than a second criterion lower than a first criterion, as a related test, based on a judgment that the error corresponds to the critical error. How to handle errors.
6. In paragraph 1, The step of selecting a test associated with the above error is performed when the severity level of the above error is below the reference level, Further comprising a step of pausing all of the plurality of tests based on a determination that the severity level of the error is higher than the reference level. How to handle errors.
7. In paragraph 1, The above pause step is, A step of temporarily suspending a test among the above-mentioned selected tests whose correlation with the above-mentioned error is greater than a standard value; and A step of suspending a test for which the correlation with the above error is below the above criterion and providing the test information to the administrator is included. How to handle errors.
8. In paragraph 1, Among the above-mentioned selected tests, certain tests are conducted on existing modules and modified modules of the above-mentioned target software, The above pause step is, comprising a step of deactivating the above transformation module; How to handle errors.
9. In paragraph 8, The above target software is an application running on a user terminal, The step of disabling the above transformation module is: A step of transmitting a control command to deactivate the modified module and activate the existing module to the terminal of a user exposed to the modified module among the users participating in the specific test, How to handle errors.
10. In paragraph 1, Further comprising the step of resuming the paused test in response to the elapse of a preset time. How to handle errors.
11. In paragraph 10, The above preset time is adjusted based on the extent to which the above error has occurred since the pause action. How to handle errors.
12. In paragraph 1, In response to the first time period elapsed, a step of resuming the paused test; If the above error occurs again, a step of determining whether the number of pauses associated with the above error is less than or equal to a preset maximum number; and Further comprising a step of re-suspending the resumed test for a second time that is equal to or longer than the first time based on a judgment that the number of times is less than the maximum. How to handle errors.
13. In paragraph 1, The above pause step is, A step of suspending the first test among the above-mentioned selected tests and monitoring the degree of occurrence of the error; and A step of monitoring the occurrence degree of the error while pausing the second test among the selected tests, wherein the first test is resumed when the second test is paused. How to handle errors.
14. One or more processors; and A memory storing a computer program executed by one or more processors, The above computer program, The action of receiving information about errors that occurred in the target software; In response to the above receipt, an operation of selecting a test associated with the error from among a plurality of ongoing tests, wherein the plurality of tests are A / B tests conducted on modules within the target software, and comprising instructions for pausing at least some of the selected tests; Error handling system.
15. Combined with a computing device, A step of receiving information about an error that occurred in the target software; In response to the above receipt, a step of selecting a test associated with the error from among a plurality of ongoing tests, wherein the plurality of tests are A / B tests conducted on modules within the target software; and Stored on a computer-readable recording medium to execute a step of pausing at least some of the above-mentioned selected tests, Computer program.
Citation Information
Patent Citations
Software fault management device, test management device and program therefor
JP2009181536A
Calculator selecting test case and selection method of test case
JP2018180711A
Interactive software program repair
JP2019029015A
Application software error monitoring, statistics management service and solution method.
KR1020180037342A
A / B Testing Platform with Anomaly Detection
KR102309715B1
