System and Method for Controlling Service Restriction Suspension Based on Commitment Fulfillment History

Abstract

The present invention relates to a system and method for controlling a service restriction deferral based on a history of commitment fulfillment. The system according to the present invention includes: a data collection unit that collects conversation data and commitment-fulfillment log records in which past commitment event identifiers and their actual fulfillment results are recorded in a corresponding relationship; an intent analysis unit that extracts situational metadata including payment fulfillment commitment information using a language model; a correction coefficient calculation unit that calculates a fulfillment reliability correction coefficient by referencing the commitment-fulfillment log records and adjusts the reflection weight of the payment fulfillment commitment information based thereon; an evaluation unit that calculates a reliability grade by combining the situational metadata with the adjusted weight and payment reliability data; and a state control unit that transitions the application status of service restriction conditions to a suspended state for a deferral period when the reliability grade satisfies a reference value. According to the present invention, since statement information generated by an entity benefiting from a judgment result is corrected by the objective performance of the same entity, judgment distortion caused by manipulation of input is suppressed, and by reflecting an extraction certainty index calculated from the internal state information of the language model, the propagation of inference uncertainty to the judgment is blocked.

Description

Technology Field

[0001] The present invention relates to an operation management technology for artificial intelligence-based services, and more specifically, to a system and method for correcting payment fulfillment commitment information extracted from conversation data received from a user based on the user's past commitment fulfillment record, and determining whether to temporarily suspend the application of service restriction conditions and the length of the suspension period according to the correction result. Background Technology

[0002] In services based on continuous provision, such as subscription services, cloud services, and telecommunications services, payment failures frequently occur regardless of the user's intent due to reasons such as the expiration of payment methods, insufficient balances, or card company rejections. Although such involuntary payment failures are not caused by the user's intention to churn, conventional service operating systems have performed a standardized process of uniformly suspending accounts after a pre-set grace period has elapsed once a payment failure is detected.

[0003] Such uniform processing simultaneously results in two conflicting losses. Specifically, if services are suspended prematurely for users who possess both the willingness and ability to pay, both recoverable revenue and the users are lost; conversely, if a long grace period is granted to users with a low probability of recovery, computational and storage resources required for service provision remain occupied without any prospect of recovery, leading to a waste of system resources.

[0004] Accordingly, a system has recently been proposed that analyzes consultation conversations or inquiry messages with users using natural language processing technology to determine their payment intent and incorporates the results into the decision regarding payment deferral. This is because advancements in large-scale language models have made it technically possible to extract structured information, such as the timing of payment execution and the estimated payment amount, from free-form conversational data.

[0005] However, this approach has a fundamental vulnerability. The payment commitment information, which serves as the input for the deferral decision, consists of natural language statements generated by the party benefiting from the outcome of that decision. In other words, since the structure involves an entity generating the input that possesses complete incentive to manipulate the system's output to its advantage, the moment the system is deployed and its operational characteristics are learned by users, users intentionally generate statements that are favorable for obtaining the deferral. In this case, no matter how high the extraction accuracy of the natural language processing module may be, the predictive power of the deferral decision deteriorates rapidly because the extracted information itself is irrelevant to the facts.

[0006] Furthermore, since information extraction using language models is inherently a probabilistic process, extraction results can vary even for the same conversational data; additionally, if the conversational data contains ambiguous or double-meaning expressions, the model may generate unfounded values. Conventional systems treated the extraction results of language models as established facts and input them directly into subsequent judgments, so the uncertainty of the extraction itself was not reflected in the judgment results in any way.

[0007] Therefore, a service restriction deferral control technology is required that operates stably even for inputs with manipulation incentives by correcting self-statement information extracted from conversation data with objective performance data and further quantifying the uncertainty of the extraction process itself to reflect it in the judgment. The problem to be solved

[0008] The present invention aims to solve the aforementioned problems by providing a service restriction deferral control system that suppresses judgment distortion caused by manipulation of statements by correcting self-statement information generated by a user based on the user's past commitment fulfillment record.

[0009] Another objective of the present invention is to suppress resource occupancy without recovery prospects by dynamically calculating the length of the grace period based on the expected recovery value and the service resource occupancy costs incurred during the grace period, rather than assigning the length of the grace period as a fixed value.

[0010] Another objective of the present invention is to prevent the uncertainty from propagating to the judgment result when the extraction itself is uncertain, by calculating an extraction certainty index from internal state information generated during the inference process of a language model and reflecting it in the judgment.

[0011] Another objective of the present invention is to reduce the total amount of computational resources required for inference while maintaining judgment accuracy by adaptively adjusting the number of inference executions of a language model according to an extraction certainty index.

[0012] Another objective of the present invention is to enable the system to be continuously recalibrated in response to changes in the user's statement strategy by providing a closed-loop structure that updates evaluation criteria by retrieving whether actual payment has been made after a grace period has been granted. means of solving the problem

[0013] An intelligent service operation management system according to an embodiment of the present invention for solving the above problem comprises: a data collection unit that collects service usage history including conversation data received from a user terminal and a commitment-fulfillment log record in which a past commitment event identifier and an actual fulfillment result corresponding to the commitment event are recorded in a corresponding relationship; an intent analysis unit that extracts situational metadata from the conversation data using a language model; a correction coefficient calculation unit that calculates a fulfillment reliability correction coefficient by referring to the commitment-fulfillment log record and adjusts the reflection weight of payment fulfillment commitment information based thereon; an evaluation unit that calculates a reliability grade by combining the situational metadata with the adjusted weight and payment reliability data; and a state control unit that transitions the application state of a service restriction condition to a suspended state for a grace period when the reliability grade satisfies a reference value.

[0014] A language model-based service operation management system according to another embodiment of the present invention is characterized in that the intent analysis unit calculates an extraction certainty index based on internal state information generated during the inference process of the language model, and the reliability correction unit determines a reflection weight by combining the extraction certainty index and a fulfillment reliability correction coefficient, wherein the weight is determined to be smaller as the extraction certainty index is lower. Effects of the invention

[0015] According to the present invention, a correction coefficient derived from the correspondence between the same subject's past statements and their realization results is applied to statement information generated by the subject who benefits from the result of a judgment. This configuration corrects unverifiable subjective inputs based on verifiable objective results, so that even if the input subject changes the content of the statement, the magnitude of the influence that statement has on the judgment is limited by the results accumulated by the subject in the past. Therefore, the judgment result cannot be favorably altered solely by manipulating the input, and the predictive power of the judgment is maintained even in an environment where an incentive for manipulation exists.

[0016] Furthermore, according to the present invention, the period during which resource occupancy is permitted is determined by the relationship between the expected recovery value from the occupancy and the cost incurred for the occupancy. In all systems subject to resource allocation, terminating occupancy with no recovery prospects early and extending occupancy with high recovery prospects increases the total amount of recovery achievable with the same total amount of resources. Conventional methods that grant a fixed grace period do not make such differentiations, thereby generating a double inefficiency in which unnecessary resources are consumed for targets with low recovery prospects while failing to provide sufficient time for targets with high recovery prospects.

[0017] Furthermore, according to the present invention, the output of a probabilistic inference engine is not treated as a confirmed fact, but the uncertainty accompanying the output is quantified and transmitted together with the subsequent judgment. In a system where the reliability of the inference result is not transmitted to the subsequent processing, inference results with weak grounds and inference results with sufficient grounds exert the same influence, leading to the accumulation of judgment errors. Transmitting uncertainty together so that only an influence proportional to the strength of the grounds is exerted is a principle of error propagation suppression applicable to information processing systems in general having a multi-stage processing structure.

[0018] Furthermore, according to the present invention, iterative inference is omitted when the certainty of the inference result is sufficient, and iterative inference is performed only when certainty is insufficient; the number of iterations is also adjusted to be inversely proportional to the degree of certainty. This configuration selectively concentrates computational resources on inputs where uncertainty exists, thereby reducing the total number of inference calls required to achieve the same judgment accuracy compared to a method that distributes a uniform amount of computation to all inputs. In a system including a probabilistic inference engine as a component, since the number of inference calls directly determines the amount of computational resources occupied and the response delay time, this configuration has the effect of improving the processing capacity of the system itself.

[0019] Furthermore, according to the present invention, the actual implementation status resulting from the granting of a grace period is retrieved as input to the evaluation criteria, thereby updating the correspondence relationship between the certainty indicator and the reflection weight. In an environment where the decision-making subject and the decision target interact, the target's behavioral strategy changes in response to the judgment criteria; thus, judgments based on fixed criteria inevitably lose predictive power over time. A closed-loop structure that continuously retrieves the correspondence relationship between the judgment result and the actual result to recalibrate the criteria corresponds to a general requirement for maintaining judgment performance in such an adaptive environment. Brief explanation of the drawing

[0020] FIG. 1 is a block diagram showing the overall configuration of an intelligent service operation management system according to one embodiment of the present invention. FIG. 2 is a diagram showing the data structure of a commitment-fulfillment log record according to one embodiment of the present invention. FIG. 3 is a diagram showing the process of calculating a transition reliability correction coefficient with time decay weighting applied according to an embodiment of the present invention. FIG. 4 is a diagram illustrating the classification by amount range and the application of scale weights of a commitment event according to one embodiment of the present invention. FIG. 5 is a diagram showing the classification criteria for commitment types and the aggregation of implementation results by type according to one embodiment of the present invention. FIG. 6 is a diagram showing the flow of weight adjustment in the process of calculating a confidence level according to one embodiment of the present invention. FIG. 7 is an account state transition diagram of a state control unit according to one embodiment of the present invention. FIG. 8 is a diagram illustrating the dynamic calculation process of a grace period according to one embodiment of the present invention. FIG. 9 is a diagram showing the maintenance of a hold state and rollback processing upon expiration of a grace period according to an embodiment of the present invention. FIG. 10 is a diagram showing differential adjustment of the service usage limit according to the reliability grade according to one embodiment of the present invention. FIG. 11 is a diagram illustrating the process of recovering implementation results and updating the evaluation model according to one embodiment of the present invention. FIG. 12 is a block diagram showing the overall configuration of a language model-based service operation management system according to another embodiment of the present invention. FIG. 13 is a diagram illustrating the process of calculating an extraction certainty index from internal state information of a language model according to another embodiment of the present invention. FIG. 14 is a diagram showing the specificity of a token interval and the output probability distribution of the interval according to another embodiment of the present invention. FIG. 15 is a diagram illustrating the process of calculating the field-specific extraction certainty index and determining the minimum value of commitment information according to another embodiment of the present invention. FIG. 16 is a diagram illustrating the process of multiple inferences and agreement calculations based on different sampling conditions according to another embodiment of the present invention. FIG. 17 is a diagram showing the adaptive adjustment of the number of inference executions according to the extraction certainty index according to another embodiment of the present invention. FIG. 18 is a diagram showing similarity calculation and weight attenuation using a reference embedding repository according to another embodiment of the present invention. FIG. 19 is a diagram illustrating the process of updating a reference embedding repository and deleting expired embeddings according to another embodiment of the present invention. FIG. 20 is a diagram illustrating the process of excluding the calculation of the reliability grade of commitment information according to another embodiment of the present invention. FIG. 21 is a diagram showing the aggregation of actual implementation rates and the updating of corresponding relationships by certainty indicator intervals according to another embodiment of the present invention. FIG. 22 is a flowchart of an intelligent service operation management method according to one embodiment of the present invention. FIG. 23 is a flowchart of a language model-based service operation management method according to another embodiment of the present invention. Specific details for implementing the invention

[0021] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. Since the present invention is capable of various modifications and may have various embodiments, specific embodiments are illustrated in the drawings and described in detail. However, this is not intended to limit the present invention to specific embodiments, and it should be understood that it includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the present invention.

[0022] < Overall configuration of the first embodiment >

[0023] Referring to FIG. 1, an intelligent service operation management system (100) according to one embodiment of the present invention comprises a data collection unit (110), an intent analysis unit (120), a correction coefficient calculation unit (130), an evaluation unit (140), a state control unit (150), and a storage unit (170). The system (100) is connected to a user terminal (10), a service server (20), and a payment processing server (30) through a communication network.

[0024] A user terminal (10) is a device used by a user for service use and inquiries, such as a smartphone, tablet, or personal computer. The user terminal (10) transmits conversation data to the system (100) in the form of a chat interface, e-mail, or transcribed results of voice consultation.

[0025] The service server (20) is a server that provides actual services to user accounts and manages the service availability status and usage limit for each account. The state control unit (150) directly accesses the account status register of the service server (20) to transition the state of the corresponding account. In this embodiment, the state transition is performed by the system (100) itself and is not based on a request to an external system and its approval.

[0026] The payment processing server (30) notifies the system (100) of the results of the payment attempt, approval, and rejection. When a notification of payment failure is received, the application of service restriction conditions to the relevant account is scheduled, and from this time, the deferral judgment procedure of the system (100) begins.

[0027] Commitment-fulfillment log records (172) are stored in the storage unit (170). The storage unit (170) may be implemented as a relational database, a key-value store, or a time-series database, and it is preferable that it be configured to have an index with a commitment event identifier as the primary key so that the commitment history of a specific user can be retrieved within a constant time.

[0028] < Data Collection Section and Commitment-Fulfillment Log Records >

[0029] The data collection unit (110) collects conversation data received from the user terminal (10) and the user's service usage history. The service usage history includes payment reliability data and commitment-fulfillment log records (172). The payment reliability data includes the number of past successful payments, the number of overdue payments, the average number of overdue days, the total usage period, the cumulative payment amount, etc.

[0030] Referring to FIG. 2, the commitment-fulfillment log record (172) is configured with fields including a commitment event identifier, a user identifier, a time of commitment creation, a time of committed fulfillment, a committed amount, a means of commitment fulfillment, a commitment type code, a time of fulfillment confirmation, and a fulfillment result flag. The fulfillment result flag has a value of any one of fulfillment completed, partial fulfillment, non-fulfillment, or judgment pending.

[0031] The key feature of the commitment-fulfillment log record (172) is that the statement information generated by the user and the objective facts regarding whether the statement has been realized are combined in a corresponding relationship within a single record. If only the statement is recorded or only the payment result is recorded, it is impossible to determine retrospectively whether a specific statement has been realized, and thus it is impossible to calculate the correction factor. Therefore, in this embodiment, a commitment event identifier is assigned at the time the commitment is extracted, and after the time of the committed fulfillment has elapsed, the fulfillment result is entered into the same record using the identifier as a key.

[0032] Partial performance refers to cases where an amount less than the committed amount is settled, and is converted to the midpoint between completed and non-fulfilled performance when calculating the correction factor described below. Judgment pending refers to cases where the committed performance date has not yet arrived, and is excluded from the calculation of the correction factor.

[0033] Intention Analysis Department

[0034] The intent analysis unit (120) extracts context metadata from conversation data using a language model. The context metadata includes the user's intent to request service maintenance, payment fulfillment commitment information, and contextual urgency.

[0035] The intention to request service maintenance is a classification value regarding whether the user wishes to continue using the service or terminate it. If the intention to terminate is specified, there is no benefit in granting a grace period, so subsequent procedures are terminated.

[0036] Payment fulfillment commitment information is structured into three fields: the time of commitment, the committed amount, and the means of commitment. For example, if a user makes a statement indicating that they will change their card and pay in full on the following Wednesday, the time of commitment is extracted as that Wednesday, the committed amount as the total outstanding amount, and the means of commitment as the newly registered card.

[0037] Situational urgency is a value indicating the degree of impact that a service interruption has on the user, and it is derived from expressions such as references to business continuity, the existence of ongoing work, and the absence of alternative means. However, since situational urgency is also information generated by the user themselves, it is subject to the correction procedure described below.

[0038] The intent analysis unit (120) outputs the extraction results in the form of structured data that conforms to a predefined schema. If an output that does not conform to the schema is generated, re-extraction is performed, and if re-extraction fails more than a pre-set number of times, the conversation data is treated as not containing commitment information.

[0039] < Correction Factor Calculation Section >

[0040] The correction coefficient calculation unit (130) calculates a performance reliability correction coefficient for the user by referring to the commitment-performance log record (172), and adjusts the weights that reflect payment performance commitment information in the reliability grade calculation based on the calculated correction coefficient.

[0041] In the most basic form, the fulfillment reliability correction factor is calculated by dividing the number of records in the user’s commitment-fulfillment log records (172) in which the fulfillment result flag is fulfillment completed by the total number of records in which the determination is completed. Partial fulfillment records are converted to fulfillment completion by the ratio of the paid amount to the committed amount.

[0042] For new users who have no commitment history, a pre-set initial value is assigned. The initial value may be set to the average fulfillment rate of all users and is gradually replaced by a value based on individual performance as commitment history accumulates. In this embodiment, while the number of records with completed judgments falls below a pre-set threshold, the initial value and the individual performance value are mixed and used in a ratio proportional to the number of records.

[0043] Referring to FIG. 3, the correction coefficient calculation unit (130) may apply a time decay weight having a smaller value as the elapsed time from the current point in time increases for each commitment event. The time decay weight may be defined in the form of an exponential function, and the half-life is set in correspondence with the payment cycle of the service. For a monthly payment service, it is preferable to set the half-life to 6 to 12 months.

[0044] The reason for applying time decay is that users' payment ability and fulfillment tendencies change over time. If a history of non-fulfillment from several years ago were reflected with the same weight as a recent continuous fulfillment history, an unfairly low correction factor would be calculated for users whose status has improved, resulting in the loss of recoverable revenue.

[0045] Referring to FIG. 4, the correction coefficient calculation unit (130) classifies the target amount of each commitment event into one of a plurality of amount ranges, and can apply a relatively high scale weight to commitment events in the same range as the range to which the amount corresponding to the current commitment information belongs.

[0046] This is based on the observation that the track record of fulfilling small-value commitments does not guarantee the feasibility of fulfilling large-value commitments. Since it is possible to manipulate the system by accumulating high correction coefficients through the repeated fulfillment of small-value commitments and subsequently obtaining a grace period for large-value commitments, this manipulation pathway is blocked by differentiating weights based on corresponding price ranges.

[0047] Referring to FIG. 5, the correction coefficient calculation unit (130) can classify each commitment event into one of a plurality of commitment types based on the remaining period until the committed time of execution and whether the commitment utterance includes a conditional expression.

[0048] Classification based on the remaining period is divided into immediate, short-term, and long-term types. The immediate type refers to a commitment with a remaining period of 24 hours or less, the short-term type refers to a commitment with a remaining period of 7 days or less, and the long-term type refers to a commitment with a remaining period of longer than that. Classification based on the inclusion of conditional expressions is divided into unconditional and conditional types; it is classified as conditional when an antecedent event that the user cannot directly control, such as the receipt of salary, deposit from a third party, or the execution of a loan, is mentioned as a premise of the commitment.

[0049] Since the fulfillment of a conditional commitment depends on the realization of its underlying event, the fulfillment rate differs from that of an unconditional commitment, even for the same user. Therefore, when a current commitment is classified as conditional, the accuracy of the prediction can be improved by prioritizing the reference to the fulfillment results of past conditional commitments.

[0050] < Application of Evaluation Section and Weights >

[0051] Referring to FIG. 6, the evaluation unit (140) calculates a reliability rating by combining the payment reliability data with the situation metadata to which the weights adjusted by the correction coefficient calculation unit (130) are applied.

[0052] The application of weights is achieved by multiplying the partial score derived from payment commitment information by a correction factor. In other words, even if the commitment information itself is stated favorably, the maximum contribution that statement makes to the confidence rating is limited if the user's past performance is low.

[0053] The technical significance of this configuration lies in the fact that it structurally limits the utility of manipulation by combining a manipulable input and an unmanipulable input in a multiplicative relationship. When two inputs are combined in an additive relationship, the decision result can be changed simply by maximizing the value of the manipulable input, whereas in a multiplicative relationship, if the unmanipulable input is low, the combined result remains low regardless of the value of the manipulable input.

[0054] Payment reliability data is not subject to correction and is reflected in the reliability rating as is. This is because payment reliability data is an objective fact recorded within the system and cannot be manipulated by the user.

[0055] The calculated confidence level is compared with a pre-established threshold value, and if it exceeds the threshold value, the grace period granting procedure is initiated. The threshold value is set based on the characteristics of the service and operational policies, and it is desirable to determine it as the point where the expected loss resulting from granting the grace period balances the expected loss resulting from early suspension.

[0056] < Status Control Unit and Account Status Transition >

[0057] Referring to FIG. 7, the account status managed by the status control unit (150) is divided into four types: normal status, payment failure status, hold status, and restricted status.

[0058] When a notification of payment failure is received while in a normal state, the account transitions to a payment failure state. In the payment failure state, if the trust level meets the threshold, it transitions to a pending state; if it does not, it transitions to a restricted state. If payment fulfillment is confirmed within the grace period while in a pending state, it returns to a normal state; if payment fulfillment is not confirmed after the grace period has elapsed, it transitions to a restricted state.

[0059] The suspended state refers to a condition in which the application of service restriction conditions is scheduled but their execution is temporarily suspended. In the suspended state, the service continues to be provided, but the usage limit may be lowered as described below.

[0060] The state control unit (150) releases the hold state in response to the expiration of the grace period. By configuring the release of the hold state to occur automatically upon the expiration of a timer inside the system, a situation in which the grace period continues indefinitely and resources are occupied without prospect of recovery is prevented.

[0061] Referring to FIG. 8, the state control unit (150) may include a grace period calculation unit (152). The grace period calculation unit (152) does not set the length of the grace period to a fixed value, but dynamically calculates it based at least on an adjusted weight, an expected recovery value corresponding to a confidence level, and the cost of occupying service resources that occurs while maintaining the hold state.

[0062] The expected recovery value is calculated by multiplying the unpaid amount by the recovery probability, which is obtained by converting the confidence level into a probability value based on a pre-learned correspondence. The service resource occupancy cost is calculated as a cost per unit of time based on the amount of computational and storage resources allocated to the relevant account while the hold status is maintained.

[0063] Computational resource usage includes processor occupancy time, inference accelerator occupancy time, and network bandwidth consumed by the account, while storage resource usage includes the storage space allocated to the account and the capacity of its backup copies. In AI-based services, since the cost of inference accelerator occupancy often accounts for the majority of the total resource usage cost, it is desirable to calculate this as a separate term.

[0064] The grace period calculation unit (152) determines the upper limit of the grace period within the range where the expected recovery value exceeds the cumulative value of the resource occupancy cost per unit time. That is, the point at which the cumulative cost incurred by continuing the grace period exceeds the expected recovery value becomes the end point of the grace period.

[0065] Under this structure, long-term deferrals are granted to accounts with large arrears, high recovery probabilities, and low resource consumption, while only short-term deferrals are granted to accounts with small arrears, low recovery probabilities, and high resource consumption. This increases the total recovery amount achievable with the same total amount of resources.

[0066] The grace period calculation unit (152) may apply a pre-set minimum value and a maximum value to the calculated grace period. The minimum value is set by considering the time physically required for the user to change the payment method, and the maximum value is set according to the requirements for accounting processing and receivables management.

[0067] Referring to FIG. 9, the state control unit (150) maintains session state information of the corresponding user account while maintaining a pending state. The session state information includes intermediate results of ongoing operations, conversation context information, temporary files, and user-specific settings.

[0068] The reason for maintaining session state information is to allow the user to continue working without interruption when the payment is fulfilled and the system returns to a normal state. If session state information is destroyed, the user must perform the work again from the beginning even if the payment is fulfilled, which significantly reduces the benefit of granting a grace period.

[0069] The state control unit (150) stores the account state prior to the transition to a pending state as a restore point. The restore point includes the usage limit, feature activation list, resource allocation, and billing status prior to the transition.

[0070] If payment fulfillment is not confirmed by the expiration of the grace period, the state control unit (150) performs a rollback process to restore the account state changed due to the pending state based on the restoration point. The rollback process is performed atomically, and if a failure occurs during the process, the incomplete rollback is resumed after restart.

[0071] If rollback processing is not provided, settings temporarily changed while in a pending state persist even after transitioning to a restricted state, compromising the consistency of the account status. In particular, if the usage limit is lowered while in a pending state, an error occurs where the user is subject to the lowered limit even after payment if the restriction is lifted without a rollback.

[0072] Referring to FIG. 10, the state control unit (150) can set the service usage limit for the user account while maintaining the hold state to a value lower than the usage limit prior to the transition, but can adjust it differentially according to the trust level so that the value is set larger as the trust level is higher.

[0073] Lowering the usage limit constitutes a resource expenditure that entails a risk of non-recovery, as the pending state implies the provision of services while payments remain unpaid. However, if the limit is lowered excessively, users may not perceive the value of the service, thereby reducing the incentive to make payments; therefore, a balance between risk and incentive is sought by applying differential limits proportional to the trust level.

[0074] In AI-based services, usage limits can be specified by the number of inference requests per unit period, the total amount of processable input tokens, or the number of concurrently executable tasks. Lowering the limit may be implemented not by rejecting requests, but by lowering processing priority and allowing response delays.

[0075] Referring to FIG. 11, the system (100) may further include a feedback collection unit (160). The feedback collection unit (160) checks whether the user has fulfilled the payment after the expiration of the grace period and additionally records in the commitment-fulfillment log record (172) the confirmation of payment fulfillment status with the commitment event identifier corresponding to the payment fulfillment commitment information in a corresponding relationship.

[0076] The correction coefficient calculation unit (130) updates the evaluation model used to calculate the fulfillment reliability correction coefficient using additionally recorded commitment-fulfillment log records (172) as training data. The update is performed when the cumulative number of additionally recorded records reaches a preset number, or when the discrepancy rate between the evaluation model's predicted fulfillment status and the confirmed payment fulfillment status exceeds a preset value.

[0077] Updates based on cumulative counts correspond to periodic recalibration, while updates based on the discrepancy rate correspond to a response to a sharp decline in forecasting performance. The reason for incorporating the latter condition is that forecasting performance may deteriorate before the scheduled update cycle if the user's statement strategy changes or the external economic environment fluctuates.

[0078] < Overall configuration of the second embodiment >

[0079] Referring to FIG. 12, a language model-based service operation management system (200) according to another embodiment of the present invention includes a data collection unit (210), an intent analysis unit (220), a reliability correction unit (240), an evaluation unit (250), and a state control unit (260), and optionally further includes a consistency verification unit (230), a feedback collection unit (270), and a reference embedding repository (280). The intent analysis unit (220) calls a language model (500) to perform inference.

[0080] The second embodiment is distinguished in that, unlike the first embodiment which deals with corrections to the content of user statements, it quantifies the uncertainty of the process of extracting information from user statements and reflects it in the judgment. The two embodiments are not mutually exclusive and can be merged and implemented within a single system.

[0081] < Calculation of Extraction Certainty Indicators >

[0082] Referring to FIG. 13, the intent analysis unit (220) calculates an extraction certainty index for payment fulfillment commitment information based on internal state information generated during the inference process of the language model (500).

[0083] Internal state information refers to any numerical information incidentally generated by the language model (500) during the process of generating the final output text, and includes token-unit output probability distribution, logit vector of the output layer, attention weight distribution, activation value of the hidden layer, and output of a separately added certainty prediction head. The present invention is not limited to any one specific internal state information.

[0084] Referring to FIG. 14, when using a token unit output probability distribution as internal state information, the intent analysis unit (220) identifies a token segment corresponding to payment fulfillment commitment information among the token sequences output by the language model (500). When using a structured output format, the position of the token corresponding to the value of each field can be determined from the output schema.

[0085] For a specified token interval, an extraction certainty index is calculated based on at least one of the mean of the logarithm of the output probability of each token or the entropy of the output probability distribution of each token. The mean of the logarithm represents the degree of confidence the model had in selecting the corresponding value, and the entropy represents how dispersed the probability mass was among the unselected alternatives.

[0086] The reason it is desirable to use both together is that the two indicators capture different types of uncertainty. If the log probability is high but the entropy is also high, it means that multiple strong candidates exist; conversely, if both the log probability and entropy are low, it means that the model has selected a specific value with confidence.

[0087] The important point here is that the information used to calculate the certainty index does not appear in the final output text of the language model (500). A system that uses only the final output cannot distinguish between a value generated by the model with certainty and a value generated without grounds, and treats both as the same confirmed fact.

[0088] Referring to FIG. 15, when payment fulfillment commitment information consists of multiple fields for the time of commitment fulfillment, the amount subject to commitment, and the means of commitment fulfillment, the intent analysis unit (220) calculates an extraction certainty index for each token interval corresponding to each field, and determines the lowest value among the individually calculated indicators as the extraction certainty index for the entire commitment information.

[0089] The reason for adopting the lowest value is that the reliability of structured information is constrained by the most uncertain field among its constituent fields. For example, even if the committed amount is clearly extracted, if the timing of the commitment fulfillment is unclear, the calculation of the grace period itself becomes impossible, and thus the utility value of the entire commitment information is lost. If the average value is adopted, a large number of clear fields conceal a small number of unclear fields, so this constraint is not reflected.

[0090] < Consistency Verification >

[0091] Referring to FIG. 16, the consistency verification unit (230) performs inference of the language model (500) multiple times with different sampling conditions on the same conversation data to obtain multiple extraction results and calculates the degree of agreement between the multiple extraction results.

[0092] Different sampling conditions include mutually different temperature parameter values ​​or mutually different random number seed values. When the temperature parameter is changed, the sharpness of the output probability distribution is adjusted, which changes the possibility for the model to select a suboptimal candidate, and when the random number seed is changed, different samples are drawn from the same distribution.

[0093] The degree of agreement is calculated based on the variance of the result values ​​for the commitment fulfillment time and the committed amount included in multiple extraction results. For the commitment fulfillment time, the standard deviation in days may be used, and for the committed amount, the coefficient of variation may be used. For categorical fields such as the means of commitment fulfillment, the frequency of the mode is used.

[0094] A low agreement rate means that the model is producing different interpretations for the same input. This is a signal suggesting that the conversational data itself is ambiguous or that the content of the commitment is not clearly specified; it is a type of uncertainty that cannot be captured solely by the output probability of a single inference.

[0095] Calculating agreement by multiple inferences is an operation that cannot be performed by humans. This is because even if the same person reads the same document repeatedly, they cannot reproduce the process corresponding to probabilistic sampling under different random number seeds, nor can they quantify the variance of the result.

[0096] Referring to FIG. 17, the consistency verification unit (230) omits multiple inferences when the extraction certainty index is greater than or equal to a preset first threshold, and performs multiple inferences only when it is less than the first threshold, and increases the number of inferences as the extraction certainty index becomes lower.

[0097] This configuration concerns the allocation of computational resources. Since language model inference itself consumes significant computational resources and time, performing multiple inferences uniformly for all inputs results in an inference cost that increases proportionally to the number of inferences.

[0098] However, a significant portion of the actual conversational data processed contains clearly expressed commitments, allowing for sufficiently high certainty indices to be obtained through a single inference. Performing iterative inference on such inputs amounts to a waste of computational resources, as it merely reconfirms already definitive results.

[0099] Conversely, for inputs with very low certainty indicators, it is difficult to obtain a stable degree of agreement even with the usual number of iterations; therefore, it is necessary to increase the number of iterations to ensure the precision of the estimation.

[0100] Therefore, by using a certainty metric as a criterion for determining the number of iterations, computational resources can be selectively concentrated on inputs with uncertainty, while resource consumption for deterministic inputs can be eliminated. This reduces the total number of inference calls required to achieve the same judgment accuracy, thereby increasing the number of requests that can be processed per unit of time and shortening the average response latency.

[0101] The first threshold is determined from the correspondence between the certainty indicator and the actual extraction accuracy. If the actual extraction accuracy meets the required level in the range where the certainty indicator is above a specific value, that value is set as the first threshold.

[0102] < Weight Attenuation Using Reference Embedding Store >

[0103] Referring to FIG. 18, the system (200) may further include a reference embedding storage (280) that stores a set of embedding vectors of commitment utterances confirmed to have been previously unfulfilled. The reliability correction unit (240) calculates the similarity between the embedding vectors of commitment utterances included in the conversation data and the embedding vectors stored in the reference embedding storage (280), and if the similarity exceeds a preset threshold, it attenuates the reflection weight.

[0104] This construction is based on the observation that commitment utterances resulting in non-fulfillment form clusters in the semantic space. Statements generated for the purpose of obtaining a grace period tend to have similar semantic structures even if their expressions differ; in particular, when information is shared among users that a specific statement format is effective for obtaining a grace period, utterances following that format appear repeatedly.

[0105] Similarity can be calculated as cosine similarity, and the maximum similarity for all vectors in the reference embedding repository (280), or the average of the similarities for the top k vectors, may be used. For large-scale repositories, it is desirable to reduce search time by applying an approximate nearest neighbor search structure.

[0106] The degree of attenuation is set to be proportional to the magnitude of similarity. However, it is desirable to set a lower limit so that the weight after attenuation does not become zero, because the mere fact that expressions similar to past unfulfilled utterances were used cannot conclude that a current commitment will necessarily remain unfulfilled.

[0107] For the calculation of the embedding vector, the hidden representation of the language model (500) may be used directly, or a separate sentence embedding model may be used. However, in either case, it must be calculated in the same embedding space as the vector stored in the reference embedding storage (280).

[0108] Referring to FIG. 19, the reliability correction unit (240) adds the embedding vector of a user's commitment utterance whose payment fulfillment has not been confirmed after the expiration of the grace period to the reference embedding storage (280), and deletes the vector among the stored embedding vectors for which a pre-set period has elapsed after registration.

[0109] The addition of unfulfilled utterances corresponds to the accumulation of manipulation patterns, while the deletion of expiration vectors prevents the infinite expansion of the repository and simultaneously prevents past expression forms that are no longer in use from being continuously referenced. In environments where users' statement strategies change over time, the persistence of old patterns may actually lower the discriminability of current utterances.

[0110] Referring to FIG. 20, if the extraction certainty index is less than a preset lower limit value, the reliability correction unit (240) sets the reflection weight of the payment fulfillment commitment information to zero and excludes it from the calculation of the reliability grade. In this case, the evaluation unit (250) calculates the reliability grade based only on the payment reliability data.

[0111] This configuration is based on the principle that when the extraction itself is unreliable, completely excluding the result rather than partially reflecting it enhances the accuracy of the judgment. Since extraction results with extremely low certainty are practically close to random values, reflecting them even with low weight amounts to nothing more than introducing noise into the judgment.

[0112] Even if commitment information is excluded, payment reliability data is utilized as is, so judgment does not become impossible. In other words, if extraction from conversation data fails, the system automatically reverts to a judgment based solely on objective historical data, which corresponds to a judgment made with manipulable inputs removed.

[0113] Referring to FIG. 21, the system (200) may further include a feedback collection unit (270). The feedback collection unit (270) checks whether payment has been fulfilled after the expiration of the grace period and additionally records it in the commitment-fulfillment log record by matching it with a commitment event identifier.

[0114] The reliability correction unit (240) refers to additionally recorded log records to aggregate the actual fulfillment rate of payment fulfillment commitment information by interval of the extraction certainty indicator, and updates the correspondence relationship between the extraction certainty indicator and the reflection weight based on the aggregated actual fulfillment rate.

[0115] For example, if the certainty indicator is divided into deciles and the actual fulfillment rates of commitments in each segment are aggregated, and the fulfillment rate in a specific segment is found to be significantly lower than expected, the weighting assigned to that segment is adjusted downward.

[0116] The significance of this configuration lies in the fact that the degree to which an extraction result is trusted when the certainty index takes a certain value is not determined by a fixed rule in advance, but is decided retrospectively based on the observation of actual results. The output probability of a language model does not necessarily correspond to actual accuracy, and the correspondence shifts depending on changes in the model version, modifications to the prompt, or changes in the input distribution. Without re-correction based on observed results, such shifts directly lead to judgment errors.

[0117] < Method Example >

[0118] Referring to FIG. 22, an intelligent service operation management method according to one embodiment of the present invention includes the steps of: collecting service usage history including conversation data and commitment-fulfillment log records; extracting situational metadata using a language model; calculating a fulfillment reliability correction coefficient by referring to the commitment-fulfillment log records and adjusting the reflection weight; calculating a reliability grade by combining the situational metadata with the adjusted weight and payment reliability data; and transitioning the application state of a service restriction condition to a suspended state when the reliability grade meets a threshold value.

[0119] Referring to FIG. 23, a language model-based service operation management method according to another embodiment of the present invention is characterized by calculating an extraction certainty index based on internal state information of the language model in the step of extracting the situation metadata, and determining a weight by combining the extraction certainty index and a fulfillment confidence correction coefficient in the step of adjusting the reflection weight, wherein the weight is determined such that the lower the extraction certainty index, the smaller the weight.

[0120] < Implementation Type >

[0121] Each component of the present invention may be implemented as a program module executed on a computing device including a processor and memory. Each component may be distributed across a plurality of physically separated servers, and in particular, the language model (500) may be placed on a separate inference server equipped with an inference accelerator and utilized by remote calling.

[0122] However, since access to the internal state information of the language model (500) is required to calculate the extraction certainty index, the second embodiment of the present invention cannot be implemented if the language model (500) is used through an interface that returns only the final output text. Therefore, the language model (500) must be provided with an interface configured to return the internal state information together.

[0123] A method according to an embodiment of the present invention may be stored in the form of instructions that can be executed by a computer on a computer-readable recording medium. The recording medium includes a hardware device configured to store and execute program instructions, such as a hard disk, an optical recording medium, or a semiconductor storage device.

[0124] Although embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements by those skilled in the art using the basic concept of the present invention as defined in the following claims also fall within the scope of the present invention. Explanation of the symbols

[0125] 10: User terminal 20: Service Server 30: Payment processing server 100: Intelligent Service Operations Management System 110: Data Collection Unit 120: Intent Analysis Unit 130: Correction coefficient calculation unit 140: Evaluation Department 150: State control unit 152: Grace period calculation section 160: Feedback Collection Department 170: Storage section 172: Commitment-Fulfillment Log Record 200: Language Model-Based Service Operations Management System 210: Data Collection Unit 220: Intent Analysis Department 230: Consistency Verification Unit 240: Reliability Correction Unit 250: Evaluation Department 260: State control unit 270: Feedback Collection Department 280: Reference Embedding Repository 500: Language Model

Claims

Claim 1 A system for providing an AI-based service to a user comprises: a data collection unit that collects conversation data received from a user terminal and user context including the user's service usage history — the service usage history includes a past commitment event identifier and a commitment-fulfillment log record in which the actual fulfillment result corresponding to the commitment event is recorded in a corresponding relationship —; an intent analysis unit that extracts context metadata including the user's intention to request service maintenance, payment fulfillment commitment information, and situational urgency from the conversation data using a language model; a correction coefficient calculation unit that calculates a fulfillment reliability correction coefficient for the user by referring to the commitment-fulfillment log record and adjusts the weight to which the payment fulfillment commitment information is reflected in the reliability grade calculation based on the fulfillment reliability correction coefficient; and an evaluation unit that calculates a reliability grade for deferring the application of a pre-set service restriction condition by combining the context metadata to which the adjusted weight is applied and the payment reliability data included in the service usage history. An intelligent service operation management system characterized by including a state control unit that, when the above-mentioned trust level satisfies a preset threshold value, transitions the application status of the above-mentioned service restriction condition for the corresponding user account to a suspended state for a grace period, and releases the suspended state in response to the elapsed grace period. Claim 2 An intelligent service operation management system according to claim 1, wherein the correction coefficient calculation unit applies a time decay weight having a smaller value as the elapsed time from the current point in time increases for each commitment event recorded in the commitment-fulfillment log record, and calculates the fulfillment reliability correction coefficient based on the aggregate value of the fulfillment results to which the time decay weight is applied. Claim 3 An intelligent service operation management system according to claim 1, wherein the correction coefficient calculation unit classifies the target amount of each commitment event recorded in the commitment-fulfillment log record into one of a plurality of amount ranges, and calculates the fulfillment reliability correction coefficient by applying a higher scale weight to the fulfillment result of a commitment event classified into the same amount range as the amount range to which the amount corresponding to the payment fulfillment commitment information belongs, compared to a commitment event in a different amount range. Claim 4 An intelligent service operation management system according to claim 1, wherein the correction coefficient calculation unit classifies each commitment event recorded in the commitment-fulfillment log record into one of a plurality of commitment types based on the remaining period until the committed fulfillment time and whether the commitment utterance includes a conditional expression, and calculates the fulfillment reliability correction coefficient based on the fulfillment result of the commitment event corresponding to the classified commitment type of the payment fulfillment commitment information. Claim 5 An intelligent service operation management system according to claim 1, wherein the state control unit includes a grace period calculation unit that dynamically calculates the length of the grace period based at least on the adjusted weight, the expected recovery value corresponding to the confidence level, and the service resource occupancy cost incurred while maintaining the hold state, rather than setting the length of the grace period to a preset fixed value. Claim 6 An intelligent service operation management system characterized in that, in claim 5, the grace period calculation unit calculates the service resource occupancy cost as a cost per unit of time based on the amount of computational resources and storage resources allocated to the corresponding user account while maintaining the hold state, and determines an upper limit of the grace period within the range where the expected recovery value exceeds the cumulative value of the cost per unit of time. Claim 7 An intelligent service operation management system according to claim 1, wherein the state control unit maintains session state information of the user account while maintaining the hold state, stores the account state prior to the transition to the hold state as a restoration point, and performs a rollback process to restore the account state changed due to the hold state based on the restoration point if payment fulfillment is not confirmed until the expiration of the grace period. Claim 8 An intelligent service operation management system according to claim 1, wherein the state control unit sets the service usage limit for the corresponding user account during the maintenance of the hold state to a value lower than the usage limit prior to the transition to the hold state, and differentially adjusts the usage limit according to the trust level such that the lower value is set larger as the trust level is higher. Claim 9 An intelligent service operation management system according to claim 1, further comprising a feedback collection unit that checks whether the user has fulfilled the payment after the expiration of the grace period, and additionally records in the commitment-fulfillment log record with a commitment event identifier corresponding to the payment fulfillment commitment information and the confirmed payment fulfillment status as a corresponding relationship. Claim 10 An intelligent service operation management system characterized in that, in claim 9, the correction coefficient calculation unit updates an evaluation model used to calculate the fulfillment reliability correction coefficient using the additionally recorded commitment-fulfillment log records as training data, and performs the update when the cumulative number of the additionally recorded log records reaches a preset number or when the discrepancy rate between the predicted fulfillment status of the evaluation model and the confirmed payment fulfillment status exceeds a preset value. Claim 11 A system for providing an AI-based service to a user comprises: a data collection unit that collects a service usage history including conversation data received from a user terminal and a commitment-fulfillment log record in which a past commitment event identifier and an actual fulfillment result corresponding to the commitment event are recorded in a corresponding relationship; an intent analysis unit that extracts situation metadata including payment fulfillment commitment information from the conversation data using a language model, and together calculates an extraction certainty index for the payment fulfillment commitment information based on internal state information generated during the inference process of the language model; a reliability correction unit that calculates a fulfillment reliability correction coefficient by referencing the commitment-fulfillment log record, combines the extraction certainty index and the fulfillment reliability correction coefficient to determine a weight in which the payment fulfillment commitment information is reflected in the calculation of a reliability grade, wherein the weight is determined to be smaller as the extraction certainty index is lower; and an evaluation unit that calculates a reliability grade by combining the payment fulfillment commitment information to which the determined weight is applied and payment reliability data included in the service usage history. A language model-based service operation management system characterized by including a state control unit that, when the above-mentioned trust level satisfies a preset threshold value, transitions the application status of a preset service restriction condition for the corresponding user account to a suspended state for a grace period, and releases the suspended state in response to the expiration of the grace period. Claim 12 A language model-based service operation management system characterized in that, in claim 11, the internal state information includes a token unit output probability distribution output by the language model, and the intent analysis unit specifies a token interval corresponding to the payment fulfillment commitment information among the token sequences output by the language model, and calculates the extraction certainty index based on at least one of the average of the log value of the output probability of each token belonging to the specified token interval and the entropy of the output probability distribution of each token. Claim 13 A language model-based service operation management system according to claim 12, wherein the intent analysis unit, when the payment fulfillment commitment information is composed of multiple fields including the time of commitment fulfillment, the amount subject to commitment, and the means of commitment fulfillment, calculates the extraction certainty indicator individually for each token interval corresponding to each of the multiple fields, and determines the lowest value among the individually calculated extraction certainty indicators as the extraction certainty indicator for the entire payment fulfillment commitment information. Claim 14 A language model-based service operation management system according to claim 11, further comprising a consistency verification unit that obtains multiple extraction results by performing inference of the language model multiple times with different sampling conditions on the same conversation data and calculates the degree of agreement among the multiple extraction results, wherein the reliability correction unit determines the weight by combining the extraction certainty index, the implementation reliability correction coefficient, and the degree of agreement. Claim 15 A language model-based service operation management system according to claim 14, wherein the consistency verification unit sets at least one of mutually different temperature parameter values ​​and mutually different random number seed values ​​as the different sampling conditions in the language model to perform the multiple inferences, and calculates the degree of agreement based on the variance of result values ​​for the commitment fulfillment time and the commitment target amount included in the multiple extraction results. Claim 16 A language model-based service operation management system according to claim 14, wherein the consistency verification unit omits the multiple inferences when the extraction certainty indicator is greater than or equal to a preset first threshold, and performs the multiple inferences only when the extraction certainty indicator is less than the first threshold, and increases the number of inferences performed as the extraction certainty indicator becomes lower. Claim 17 A language model-based service operation management system according to claim 11, further comprising a reference embedding storage that stores a set of embedding vectors of commitment utterances confirmed to have been unfulfilled in the past, wherein the reliability correction unit calculates the similarity between the embedding vector of the commitment utterance included in the conversation data and the embedding vector stored in the reference embedding storage, and attenuates the weight when the similarity exceeds a preset threshold. Claim 18 A language model-based service operation management system according to claim 17, wherein the reliability correction unit adds the embedding vector of a user's commitment utterance whose payment fulfillment has not been confirmed after the expiration of the grace period to the reference embedding repository, and deletes the embedding vector among the embedding vectors stored in the reference embedding repository for which a pre-set period has elapsed after registration. Claim 19 A language model-based service operation management system according to claim 11, wherein the reliability correction unit sets the weight of the payment fulfillment commitment information to zero and excludes it from the calculation of the reliability grade when the extraction certainty indicator is less than a preset lower limit value, and the evaluation unit calculates the reliability grade based on the payment reliability data. Claim 20 A language model-based service operation management system according to claim 11, further comprising a feedback collection unit that checks whether the user has fulfilled the payment after the expiration of the grace period, and additionally records in the commitment-fulfillment log record with a commitment event identifier corresponding to the payment fulfillment commitment information and the confirmed payment fulfillment status as a correspondence relationship, wherein the reliability correction unit refers to the additionally recorded commitment-fulfillment log record to aggregate the actual fulfillment rate of the payment fulfillment commitment information by interval of the extracted certainty indicator, and updates the correspondence relationship between the extracted certainty indicator and the weight based on the aggregated actual fulfillment rate. Claim 21 An intelligent service operation management method performed by a system providing an artificial intelligence-based service to a user, comprising: a step of collecting a service usage history including conversation data received from a user terminal and a commitment-fulfillment log record in which a past commitment event identifier and an actual fulfillment result corresponding to the commitment event are recorded in a corresponding relationship; a step of extracting context metadata including payment fulfillment commitment information from the conversation data using a language model; a step of calculating a fulfillment reliability correction coefficient by referring to the commitment-fulfillment log record and adjusting a weight in which the payment fulfillment commitment information is reflected in the calculation of a reliability grade based on the fulfillment reliability correction coefficient; a step of calculating a reliability grade by combining the context metadata with the adjusted weight and payment reliability data included in the service usage history; and a step of, when the reliability grade satisfies a preset threshold value, transitioning the application status of a preset service restriction condition for the corresponding user account to a suspended state for a grace period and releasing the suspended state in response to the expiration of the grace period. Claim 22 A method for managing service operations performed by a system providing an artificial intelligence-based service to a user, comprising: a step of collecting service usage history including conversation data received from a user terminal and a commitment-fulfillment log record in which a past commitment event identifier and an actual fulfillment result corresponding to the commitment event are recorded in a corresponding relationship; a step of extracting situation metadata including payment fulfillment commitment information from the conversation data using a language model, and together calculating an extraction certainty index for the payment fulfillment commitment information based on internal state information generated during the inference process of the language model; a step of calculating a fulfillment reliability correction coefficient by referring to the commitment-fulfillment log record, and determining a weight to which the payment fulfillment commitment information is reflected in the calculation of a reliability grade by combining the extraction certainty index and the fulfillment reliability correction coefficient, wherein the weight is determined to be smaller as the extraction certainty index is lower; and a step of calculating a reliability grade by combining the payment fulfillment commitment information to which the determined weight is applied with payment reliability data. A language model-based service operation management method characterized by including the step of, when the above-mentioned trust level satisfies a pre-set threshold value, transitioning the application status of a pre-set service restriction condition for the corresponding user account to a suspended state for a grace period, and releasing the suspended state in response to the expiration of the grace period. Claim 23 A computer program stored on a computer-readable recording medium to execute the method of paragraph 21 or 22.

Images