Prediction system and its control method, and learning method of the prediction system

Abstract

The present invention relates to a prediction system, a method for controlling the same, and a method for learning the prediction system. More specifically, the present invention relates to a prediction system capable of predicting potential client companies (or potential customers) in B2B (Business-to-Business) and / or B2C (Business-to-Consumer) sales situations, a method for controlling the same, and a method for learning the prediction system.

Description

Technology Field

[0001] The present invention relates to a prediction system, a method for controlling the same, and a method for learning the prediction system. More specifically, the present invention relates to a prediction system capable of predicting potential client companies (or potential customers) in B2B (Business-to-Business) and / or B2C (Business-to-Consumer) sales situations, a method for controlling the same, and a method for learning the prediction system. Background Technology

[0002] With the recent advancement of artificial intelligence, there has been a surge in cases of outstanding performance across various industrial sectors. In particular, the development of machine learning (ML) and deep learning technologies is making a significant contribution to the development of artificial intelligence models that learn patterns from vast amounts of data and support prediction and decision-making.

[0003] Meanwhile, the quantity and quality of training data directly affect the generalization performance of artificial intelligence models. High-quality data enables models to make more accurate predictions, and the utility of the data can be maximized through the integration and preprocessing of various data sources.

[0004] On the other hand, data imbalance can degrade the prediction accuracy of artificial intelligence models. Most datasets exhibit an imbalance where data from specific classes is significantly more or less than that of others; this can lead to the problem of AI models becoming biased in their learning towards the classes that appear frequently. For example, in business data, positive outcomes (e.g., purchases) often occur relatively less frequently than negative outcomes (e.g., non-purchases). This can cause data imbalance and negatively impact the training and prediction performance of AI models.

[0005] Accordingly, efficient training of artificial intelligence models must be considered, and recently, research on methods to solve the problem of imbalanced data is actively underway. The problem to be solved

[0006] The present invention aims to solve the problem of unbalanced data and to provide a prediction system, a control method thereof, and a learning method for the prediction system that can be universally utilized in various industrial fields.

[0007] More specifically, the present invention is intended to provide a prediction system capable of predicting potential customers and establishing an optimal sales strategy, a control method thereof, and a learning method for the prediction system.

[0008] Furthermore, the present invention aims to provide a training method for a prediction model that can predict valid customers by analyzing various customer data. means of solving the problem

[0009] To solve the problem described above, a learning method for a prediction system, in which a memory and at least one processor according to the present invention cooperate, may include the steps of: specifying a learning data set; configuring a plurality of different sub-data sets using the learning data set; training a learning target prediction model for each of the different sub-data sets; obtaining a plurality of learned prediction models, each trained with the different sub-data sets, based on the learning; inputting input data to be predicted into each of the plurality of learned prediction models; obtaining a plurality of predicted values ​​for the input data from each of the plurality of learned prediction models; and specifying a final predicted value for the input data using the plurality of predicted values.

[0010] In an embodiment, the training data set is configured to include a plurality of records having values ​​for a plurality of different categories, and in the step of configuring the plurality of different sub-data sets, the plurality of different sub-data sets may be configured based on a value corresponding to a specific category among the plurality of categories.

[0011] In an embodiment, the training data set includes MQL (Marketing Qualified Lead) data configured to have values ​​for the plurality of different categories, and the specific category is a category indicating whether a customer has made a purchase conversion, and the value corresponding to the specific category may have a first value or a second value depending on whether a customer has made a purchase conversion.

[0012] In an embodiment, each of the plurality of learned prediction models may be configured to predict a value for the specific category.

[0013] In an embodiment, the method further includes the step of performing feature engineering on the training data set, and

[0014] In the step of performing the above feature engineering, a derived category is generated using at least some of the plurality of categories and values ​​corresponding to at least some of the categories, and a value corresponding to the generated derived category can be specified.

[0015] In the embodiment, the training data set may further include the derived category and the value corresponding to the derived category.

[0016] In an embodiment, the value corresponding to the specific category is configured to have the first value or the second value, and in the step of configuring the plurality of different sub-data sets, at least a portion of the plurality of records may be included in each of the plurality of different sub-data sets such that the composition ratio of the first record containing the first value for the specific category and the second record containing the second value for the specific category among the plurality of records satisfies a preset composition ratio standard.

[0017] In an embodiment, the pre-set configuration ratio standard may be related to having the same ratio between the number of first records containing the first value for the specific category and the number of second records containing the second value for the specific category for each of the plurality of different sub-data sets.

[0018] In an embodiment, the number of different sub-data sets may be determined based on the number of second records containing the second value for the specific category and the number of first records containing the first value for the specific category among the total number of records included in the training data set.

[0019] In an embodiment, the method further includes a step of determining the number of different sub-data sets, wherein the determining step may be based on a value obtained by dividing the number of second records containing the second value for the specific category by the number of first records containing the first value for the specific category.

[0020] In an embodiment, each of the plurality of different sub-data sets includes all of the first records having the first value for the specific category among the records included in the training data set, and a portion of the second records having the second value for the specific category among the records included in the training data set may be included in an amount equal to the number of first records included in each of the plurality of different sub-data sets.

[0021] In an embodiment, each of the plurality of different sub-data sets may include the same first record, and each of the plurality of different sub-data sets may include different second records.

[0022] In an embodiment, the learning target prediction model includes a plurality of prediction models based on the GBDT (Gradient Boosting Decision Tree) algorithm, and in the learning step, the plurality of prediction models are each trained for each of the different sub-data sets, and the plurality of trained prediction models trained for each of the different sub-data sets can be obtained.

[0023] In the embodiment, in the training step, as a result of training the plurality of prediction models for each of the different sub-data sets, the plurality of trained prediction models can be obtained in an amount equal to the product of the number of different sub-data sets N and the number of the plurality of prediction models M.

[0024] In an embodiment, the number of multiple predicted values ​​obtained from the plurality of learned prediction models may correspond to the value obtained by multiplying the number of different sub-data sets N and the number of multiple prediction models M.

[0025] In an embodiment, in the step of determining the final predicted value, soft voting based on the plurality of predicted values ​​may be performed to determine the final predicted value.

[0026] A method for predicting an effective customer according to the present invention, in which a memory and a processor cooperate, may include: receiving data of a customer to be predicted from a user terminal; inputting the customer to be predicted into each of a plurality of prediction models, each of which is trained on a different sub-data set divided based on the purchasing customer data among a training data set composed of purchasing customer data and non-purchasing customer data; obtaining a plurality of prediction values ​​as outputs of each of the plurality of prediction models, each of which represents the probability that a customer corresponding to the customer to be predicted is an effective customer; determining a final prediction value for the customer to be predicted using the plurality of prediction values; and providing information to the user terminal regarding whether a customer corresponding to the customer to be predicted is an effective customer using the determined final prediction value.

[0027] A system according to the present invention is a prediction system comprising a memory and at least one processor, wherein the memory and the processor cooperate to form a plurality of different sub-data sets using a learning data set, train a learning target prediction model for each of the different sub-data sets, obtain a plurality of trained prediction models each trained with the different sub-data sets based on the training, input data to be predicted to each of the plurality of trained prediction models, obtain a plurality of predicted values ​​for the input data from each of the plurality of trained prediction models, and determine a final predicted value for the input data using the plurality of predicted values.

[0028] A program according to the present invention is a program that is executed by one or more processes in an electronic device and can be stored on a computer-readable recording medium, wherein the program may include instructions for performing the steps of: specifying a learning data set; configuring a plurality of different sub-data sets using the learning data set; training a learning target prediction model for each of the different sub-data sets; obtaining a plurality of learned prediction models, each trained with the different sub-data sets, based on the learning; inputting input data to be predicted into each of the plurality of learned prediction models; obtaining a plurality of predicted values ​​for the input data from each of the plurality of learned prediction models; and specifying a final predicted value for the input data using the plurality of predicted values. Effects of the invention

[0029] As described above, according to the prediction system, the control method thereof, and the learning method of the prediction system of the present invention, by providing a prediction model that has learned various business data, it is possible to effectively respond to various business situations.

[0030] Furthermore, according to the prediction system, the control method thereof, and the learning method of the prediction system of the present invention, by resolving the problem of imbalance in various business data, it is possible to provide learning on balanced learning data. Through this, the prediction model learns balanced input data, thereby maintaining stable and high prediction performance even with various inputs during the actual usage phase.

[0031] Furthermore, according to the prediction system, the control method thereof, and the learning method of the prediction system of the present invention, balanced business data is learned to resolve the problem of unbalanced data in actual usage environments. That is, according to the present invention, the generalization performance of the prediction model is enhanced, enabling more accurate sales conversion prediction in actual sales environments, and making it possible to efficiently allocate sales resources and establish optimized sales strategies.

[0032] Furthermore, according to the prediction system, the control method thereof, and the learning method of the prediction system of the present invention, it is possible to provide an environment in which customized sales strategies tailored to customer characteristics can be established by analyzing various customer data. Through this, companies can flexibly respond to various customer types and market environments, thereby strengthening long-term relationships with customers and significantly improving the performance of various businesses. In addition, companies can optimize performance in the global market and establish customized strategies tailored to the characteristics of each country. In other words, according to the present invention, it can provide important insights for a company's strategic decision-making and contribute to improving long-term business performance. Brief explanation of the drawing

[0033] FIG. 1 is a conceptual diagram illustrating a prediction system and a control method thereof according to the present invention. FIGS. 2a and FIGS. 2b are conceptual diagrams for explaining a prediction system according to the present invention. FIG. 3 is a flowchart illustrating a learning method of a prediction system according to the present invention. FIGS. 4, FIGS. 5, FIGS. 6, FIGS. 7, FIGS. 8, and FIGS. 9 are conceptual diagrams for explaining a learning method of a prediction system according to the present invention. FIG. 10 is a flowchart illustrating a method for predicting valid customers of a prediction system according to the present invention. FIG. 11 is a conceptual diagram illustrating a method for predicting valid customers of a prediction system according to the present invention. Specific details for implementing the invention

[0034] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Identical or similar components are assigned the same reference number regardless of the drawing symbols, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably solely for the ease of drafting the specification and do not have distinct meanings or roles in themselves. Furthermore, in describing the embodiments disclosed in this specification, if it is determined that a detailed description of related prior art could obscure the essence of the embodiments disclosed in this specification, such detailed description will be omitted. Additionally, the attached drawings are intended only to facilitate understanding of the embodiments disclosed in this specification; the technical concept disclosed in this specification is not limited by the attached drawings, and it should be understood that they include all modifications, equivalents, and substitutions that fall within the spirit and technical scope of the present invention.

[0035] Terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. These terms are used solely for the purpose of distinguishing one component from another.

[0036] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0037] A singular expression includes a plural expression unless the context clearly indicates otherwise.

[0038] In this application, terms such as “comprising” or “having” are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0039] Hereinafter, the present invention will be examined in more detail with reference to the attached drawings. FIG. 1 is a conceptual diagram illustrating a prediction system and a control method thereof according to the present invention, and FIG. 2a and FIG. 2b are conceptual diagrams illustrating a prediction system according to the present invention. FIG. 3 is a flowchart illustrating a learning method of a prediction system according to the present invention, and FIG. 4, FIG. 5, FIG. 6, FIG. 7, FIG. 8 and FIG. 9 are conceptual diagrams illustrating a learning method of a prediction system according to the present invention. Furthermore, FIG. 10 is a flowchart illustrating a valid customer prediction method of a prediction system according to the present invention, and FIG. 11 is a conceptual diagram illustrating a valid customer prediction method of a prediction system according to the present invention.

[0040] The prediction system, the control method thereof, and the learning method of the prediction system according to the present invention can be usefully utilized in various environments or situations. For example, the prediction system can be usefully utilized in situations where efficient marketing strategies are established by predicting the purchasing probability of potential customers, or where inventory management and production planning are optimized by predicting market demand for specific products or services.

[0041] In one embodiment, the prediction system according to the present invention can be usefully utilized in a B2B (Business-to-Business) sales situation. Here, B2B may refer to transactions or commercial activities between companies. This may represent a business model in which a specific company provides products (or goods) or services to another company. For example, cases where a software company provides software solutions to another company, or where a parts manufacturer supplies parts to another company, may constitute a B2B transaction. In other words, unlike B2C (Business to Consumer), which targets consumers, B2B is characterized by being centered on relationships between companies.

[0042] Furthermore, the prediction system and control method according to the present invention, and the learning method of the prediction system, can be applied to various industries and services and utilized effectively. For example, a prediction model to which the learning method according to the present invention is applied can be utilized to diagnose rare diseases in the medical industry, to detect fraudulent transactions in the financial industry, or to detect and respond to security threats early in the security industry.

[0043] Meanwhile, although the purpose or application of the prediction system described in this specification is related to B2B and / or B2C sales, it is not necessarily limited thereto. For example, the prediction system according to the present invention can be applied to and effectively utilized in various industrial fields such as medical, financial, and security as discussed above. Furthermore, in the present invention, the term “customer” may be used interchangeably with “client company” to have the same meaning.

[0044] A prediction system and a control method according to the present invention will be briefly examined together with FIG. 1. As shown in FIG. 1, the prediction system (100) according to the present invention may include at least one of a data processing unit (110), a model unit (120), a prediction unit (130), and a control unit (140).

[0045] Although not illustrated, the prediction system (100) may include one or more processors, and such processors may include one or more general-purpose processors and / or one or more special-purpose processors (e.g., digital signal processors, tensor processing units (TPUs), graphics processing units (GPUs), neural network processing units (NPUs), application integrated circuits, application semiconductors (ASICs), etc.). One or more processors may be configured to execute instructions stored (or included) in memory, computer-readable instructions and / or other instructions described herein. Such answer generation systems and methods may perform data processing described below in cooperation with memory and at least one processor. The processor may perform a series of operations and data processing using data and information stored in memory.

[0046] Meanwhile, the data processing unit (110) may be configured to collect data from various sources (e.g., database, web crawling, API, a server linked to the prediction system (100), an external server, etc.) and perform preprocessing on the collected data.

[0047] The data processing unit (110) can collect various data used for training at least one model (121, 122, 123) included in the model unit (120). For example, as illustrated in FIGS. 2a and 2b, the data processing unit (110) can collect MQL (Marketing Qualified Lead) data (210) configured to have values ​​for multiple different categories from various sources.

[0048] Here, MQL data can be understood as information selected from potential customers (or clients) through marketing activities, used to identify potential customers who show interest in a product or service or who have the potential to make a purchase. Marketing eligible lead data may include various elements related to records and / or behaviors regarding a customer's interest in a product or service. For example, these various elements may include at least one of the following: customer (or client) information (e.g., customer's name, customer account (or customer identification number or code), contact information, email address, job title, location information, country of origin, company (or firm) affiliation, etc.), information about the company to which the customer belongs (e.g., company name, industry, size, etc.), the type and / or category of the product or service the customer showed interest in, records of customer events (e.g., customer's website visit history (or frequency), customer's purchase history, product page views, product inquiries, survey responses, etc.), and information related to the customer's purchase intent (e.g., information regarding expected budget, expected timing of purchase, etc.).

[0049] However, the collected data is not necessarily limited to the examples mentioned above. In one embodiment, the data processing unit (110) may collect at least one of the following in addition to the marketing eligible data (210): product data (e.g., product identification information (or code), product name and description, product price, inventory status, product category, product rating and review, product launch date, product specifications and features, etc.), sales process data (e.g., lead information, sales representative information, sales opportunity information, sales activity records, sales stages, contract information, performance indicators, etc.), and market trend data (e.g., market research reports, competitor information, industry trends, consumer behavior, economic indicators, technology trends, regional (or country) characteristics and regulatory information, etc.). For convenience of explanation, the collected data will not be separately distinguished and will all be described as “learning data set (or learning data, 210).”

[0050] Meanwhile, the data processing unit (110) can perform preprocessing on the training data set (210).

[0051] The data processing unit (110) can clean the training data set (210) to process errors or missing values ​​in the training data set (210) and detect (or identify) and remove outliers or duplicate records (or data). For example, the data processing unit (110) can replace or delete missing values ​​in the training data set (210) with average values, and detect and remove outliers (e.g., outliers) and duplicate records that have abnormally large or small values.

[0052] Additionally, if the training data set (210) includes categorical data (or variables), the data processing unit (110) can convert the categorical data into a numerical form that an artificial intelligence model can understand. For example, the data processing unit (110) can convert the categorical data into a multidimensional vector using at least one of one-hot encoding and / or label encoding.

[0053] Additionally, the data processing unit (110) can adjust the range of numeric data (or continuous data) so that all variables have the same range. For example, the data processing unit (110) can convert numeric data into data with a mean of 0 and a variance of 1 through normalization (e.g., Z-Score normalization) of numeric data, or convert continuous data into a range between 0 and 1 through scaling (e.g., Min-Max scaling) of continuous data. This can be understood as data processing to prevent the results from being distorted by the magnitude of a specific variable or the artificial intelligence model from being biased toward a specific feature when training an artificial intelligence model.

[0054] Furthermore, the data processing unit (110) can expand (or augment) the data that the artificial intelligence model can learn by generating new variables (derived variables) from the training data set (210) through feature engineering on the training data set (210) in which existing variables have been preprocessed.

[0055] In this case, the data processing unit (110) can generate derived variables (or derived categories) from the training data set (210) based on RFM (Recency-Frequency-Monetary) analysis during the feature engineering process.

[0056] RFM analysis is a marketing method used to evaluate and classify customers and may include recency, frequency, and monetary value. Here, recency refers to the time elapsed from the customer's last purchase to the present, frequency refers to the number of times a customer has made purchases over a certain period, and monetary value refers to the total amount spent by a customer during that period.

[0057] In one embodiment, the data processing unit (110) can extract specific data (or variables, e.g., sales representative (e.g., “lead_owner”), customer identification information (e.g., “customer_idx”), etc.) with high characteristic importance from a training data set (210) based on RFM analysis, and generate derived variables for the extracted specific data (e.g., a variable indicating the level or frequency of the representative’s experience (e.g., “lead_owner_job”), a variable indicating whether or not the customer makes repeat purchases or the frequency (e.g., “customer_idx_count”), a variable combining the sales representative’s experience and the customer’s revisit frequency (e.g., “oppty”), etc.).

[0058] In another embodiment, the data processing unit (110) can separate year and month information using date data (e.g., “lead_date”) included in the training data set (210) and generate a derived variable (e.g., “lead_date_yearmonth”) including the customer’s recent purchase activity.

[0059] Meanwhile, the data processing unit (110) can form at least one sub-data set using the training data set (210).

[0060] For example, training datasets may exhibit an imbalance where data containing specific values ​​is significantly more or less abundant than data containing other values. This can lead to a problem where the AI ​​model becomes biased toward classes that appear frequently. For instance, in MQL data, data corresponding to a customer purchase conversion often occurs relatively less frequently than data corresponding to a customer purchase non-conversion. This can cause data imbalance issues and negatively impact the training and prediction performance of the AI ​​model.

[0061] To resolve this data imbalance problem, the data processing unit (110) may configure multiple different sub-data sets such that the composition ratio of each of the multiple records containing different values ​​for a specific category among the multiple data (or records) included in the training data set (210) satisfies a preset composition ratio standard. More specific details regarding this will be explained later.

[0062] Meanwhile, the model unit (120) may be configured to include at least one learning target prediction model. For example, the model unit (120) may include at least one of a first model (121), a second model (122), and a third model (123) that are the subject of learning.

[0063] The first model (121) may also be named a “CatBoost model” and may be a model specialized for processing categorical data (or variables or features). The first model (121) can prevent target leakage problems that may occur in categorical data by using a regularization technique (or method) called “Ordered Target Statistics” and / or “Ordered Boosting.” Additionally, the first model (121) can distribute balanced data at each level of the tree using a symmetric tree structure. This first model (121) can prevent overfitting and have high predictive performance.

[0064] The second model (122) may also be named a “LightGBM (LGBM) model” and may be a model that maximizes learning speed and reduces memory usage while maintaining high prediction performance by utilizing “Gradient-based One-Side Sampling (GOSS)” and / or “Exclusive Feature Bundling (EFB)” methods. Gradient-based One-Side Sampling (GOSS) reduces computational load by sampling data based on the magnitude of gradients, and Exclusive Feature Bundling (EFB) can reduce the number of variables by bundling sparse features. Additionally, the second model (122) can learn deeply about specific parts of the data and better identify complex data patterns by utilizing a leaf-wise tree growth method.

[0065] The third model (123) may also be named the “XGBoost model” and may be a model based on the Gradient Boosting Decision Tree (GBDT) algorithm, which is optimized for high prediction performance and prevention of overfitting. The third model (123) may use regularization to prevent overfitting and tree pruning to reduce the complexity of the model by removing unnecessary branches. The third model (123) provides flexibility in handling missing values ​​and can perform broad learning by evenly dividing all nodes through a level-wise tree growth method so that various characteristics are well reflected.

[0066] As seen above, the first model (121), the second model (122), and the third model (123) may be models based on the Gradient Boosting Decision Tree (GBDT) algorithm, and may split data and perform learning based on the decision tree.

[0067] However, the models included in the model section (120) in the present invention are not necessarily limited to the models mentioned above, and various additional models may be included. It is obvious that the models included in the model section (120) in the present invention may be one or multiple, and may be changed in various ways depending on the case.

[0068] Meanwhile, a plurality of sub-data sets (221, 222, 223) generated from the data processing unit (110) may be input to each of the models (121, 122, 123) included in the model unit (120). In this case, each of the plurality of models (121, 122, 123) may receive the plurality of sub-data sets (221, 222, 223) as input and perform learning on each of the plurality of sub-data (221, 222, 223).

[0069] Specifically, the first model (121), the second model (122), and the third model (123) independently perform learning on each of the multiple sub-data sets (221, 222, 223), and when the learning of each model (121, 122, 123) is completed, multiple learned prediction models can be obtained.

[0070] Here, “multiple learned prediction models” may mean a number of learned prediction models equal to the product of the number of multiple different sub-data sets “N” and the number of multiple prediction models (121, 122, 123) “M” as a result of training each of the multiple prediction models (121, 122, 123) for each of the multiple sub-data sets (221, 222, 223).

[0071] That is, when the training of each of the multiple prediction models (121, 122, 123) for each of N different sub-data sets is completed, each of the multiple prediction models (121, 122, 123) may include multiple trained prediction models trained for each of the N sub-data sets. In the present invention, the prediction model may also be named a “Binary Classification Model” or a BalancedTreeMarketer Model.

[0072] Meanwhile, the prediction unit (130) may be configured to determine the final prediction result (or final prediction value) using the output value of at least one learned prediction model (or N learned models).

[0073] Specifically, the prediction unit (130) can perform soft voting based on multiple prediction values ​​output from each of the multiple learned prediction models to determine the final prediction value.

[0074] In one embodiment, the prediction unit (130) calculates (or produces) an averaged probability (e.g., “Sales Conversion Probability”, 230) by averaging a plurality of prediction values ​​(or prediction probabilities) that each of the plurality of learned prediction models independently predicted based on soft voting, and can determine a final prediction value (e.g., “Sales Conversion Predict”, or customer conversion, 240) based on the calculated probability (230).

[0075] Here, soft voting can be understood as one of the ensemble techniques, a technique that determines the final prediction by combining (or averaging) the results (or probabilities) that multiple artificial intelligence models each independently predict. That is, the prediction unit (130) can determine the final prediction result (or prediction value) by combining the result values ​​(or prediction values) output from each of the multiple learned prediction models.

[0076] Additionally, the averaged probability is the result of combining the predicted values ​​output by the learned prediction models, and can be understood as representing the probability value of the likelihood that a customer will purchase a product or service (purchase conversion likelihood). For example, the prediction unit (130) can express the probability value as a value between 0 and 1, in which case, a value of 0.7 may mean that the customer has a 70% probability of purchasing the product.

[0077] Furthermore, the final prediction value (240) can be understood as a prediction result that is finally extracted, representing whether a customer will purchase a product or service through binary classification. For example, the prediction unit (130) may express it as “purchased (1)” if it is predicted that the customer will purchase the product, and “not purchased (0)” if it is predicted that the customer will not purchase the product.

[0078] That is, the prediction unit (130) compares the averaged probability (230) with a preset threshold and can specify the final predicted value (240) as “purchased (1)” if the averaged probability (230) exceeds the preset threshold, and “not purchased (0)” if it does not exceed the threshold. More specific details regarding this will be explained later.

[0079] Meanwhile, the control unit (140) can perform the role of controlling the overall operation of the present invention and the prediction system (100). The control unit (140) can process signals, data, information, etc. that are input or output through the components described above, or perform a series of data processing to provide or process appropriate information and functions to the user.

[0080] In one embodiment, the control unit (140) may provide a service page (1000) to the user terminal (10). The service page (1000) may provide a list (or customer list) of at least one company that performs interactions (e.g., transactions, cooperation, etc.) with a specific company. In this case, the control unit (140) may provide information regarding the probability of each customer purchasing a specific product (e.g., “Puricare Objet Collection Water Purifier”), predicted by the prediction system (100), in a part of the service page (1000).

[0081] Meanwhile, the present invention aims to solve the problem of unbalanced data and provide a prediction system, a control method thereof, and a learning method for the prediction system that can be universally utilized in various industrial fields. More specifically, the present invention aims to provide a prediction system capable of predicting valid customers by analyzing various customer data, and the learning method of the prediction system (or prediction model) will be examined in more detail below.

[0082] First, in the present invention, a process of specifying a training data set may be performed (S310, see FIG. 3).

[0083] The control unit (140) can specify the training data set to be used for training the learning target prediction model.

[0084] In this case, the criteria for specifying the training data set may vary. The control unit (140) may specify the training data set to be used for training the target prediction model based on various criteria.

[0085] In one embodiment, the control unit (140) collects (or receives) a data set from at least one of various sources (e.g., a database (DB), web crawling, API, a server linked to the prediction system (100), an external server, etc.) and can specify the collected data set as a training data set to be used for training a prediction model to be trained.

[0086] In another embodiment, the control unit (140) may specify a data set stored in at least one of various storage units (e.g., storage unit, memory, database (DB), etc.) as a training data set to be used for training a learning target prediction model.

[0087] A training data set may include various types of data. For example, as illustrated in FIG. 4, a training data set (410) may include at least one of MQL data, product data, sales process data, and market trend data, and the data included in such a training data set may be in at least one form of numerical data, categorical data, and text data. However, the forms of data included in the training data set are not necessarily limited to the examples mentioned, and it is obvious that data in various forms may be included.

[0088] This training data set (410) may be configured to include multiple records having values ​​for multiple different categories.

[0089] Here, a record represents at least one unit of data, and a record may contain data values ​​of multiple categories (i.e., multiple fields or attributes). In a database, this can also be referred to as a "row." For example, in an Excel sheet, each row represents a single record, and each column of a row can represent data values ​​of various categories within the record.

[0090] That is, each piece of data included in a single data set, or a single unit of data containing multiple categories of data values, can be named a “record” or “sample.”

[0091] The training data set (410) may include MQL data configured to have values ​​for multiple different categories. Furthermore, in the present invention, categories may also be referred to as “categories,” “features,” “variables,” or “elements.”

[0092] Before examining the preprocessing process for the training data set (410), we will first examine the multiple categories and values ​​for the multiple categories included in the training data set (410) together with FIG. 5.

[0093] The first category (e.g., “ID”, 501) is a random value that uniquely identifies each data entry, and its primary purpose may be to calculate an f1 score by comparing it with the third category (e.g., “is_converted”, 539). Through the first category (501), accuracy can be measured by matching each prediction result with the actual result, and the first category (501) can also be used for evaluating model performance.

[0094] Category 2 (e.g., “bant_submit”, 502) is a variation of the BANT (Budget, Authority, Need, Timeline) framework that can be used to evaluate MQL quality. Here, “Budget” refers to the customer’s budget information, meaning funds available for projects or purchases. “Authority (Title)” refers to the customer’s position or rank, indicating whether they possess decision-making authority. Additionally, “Needs” refers to the customer’s specific requirements or the problems or goals that the product or service must solve, and “Timeline” refers to the delivery date desired by the customer.

[0095] The third category (e.g., “customer_country”, 503) represents the customer’s nationality, and the value can be expressed as “ / region / country” (e.g., Asia / Korea). The third category (503) can provide important information regarding regional sales strategies, the provision of localized services, and approaches based on legal / cultural understanding. Additionally, the third category (503) can be used to formulate strategies that take into account time differences, language barriers, and cultural differences that may occur in international business relationships.

[0096] Category 4 (ex: “customer_country.1”, 504) may refer to regions and countries, such as the corporate region of the responsible company.

[0097] The fifth category (e.g., “business_unit”, 505) refers to the business unit within the company corresponding to the product or service requested in the MQL, and can be divided into multiple (e.g., 5) categories (e.g., ID, AS, IT, Solution, CM). These categories are important for understanding the nature of the lead and assigning the appropriate sales team or expert, and can be utilized for performance analysis, resource allocation, and strategy formulation by business unit.

[0098] Category 6 (ex: “com_reg_ver_win_rate”, 506) is a weight calculated based on the opportunity ratio for a specific business area (Vertical Level 1), a specific business unit (or business unit), and a region, and can be used to predict the likelihood of future success based on past success rates.

[0099] Category 7 (e.g., “customer_idx”, 507) stores the customer company name and the number of times the customer company has submitted data, which can indirectly show the customer company’s level of engagement or interest. A high value indicates that the company frequently inquires or interacts, which may indicate a potentially high level of interest or purchase intent. For example, Category 7 (507) can be used for customer segmentation, prioritization, and establishing customized marketing strategies.

[0100] Category 8 (ex: “customer_type”, 508) is data classifying customers' occupations, which can be usefully utilized for target marketing or establishing customized sales strategies.

[0101] Category 9 (e.g., “enterprise”, 509) represents the size of the customer company and can be divided into Enterprise and SMB.

[0102] Category 10 (e.g., “historical_existing_cnt”, 510) may represent the number of times the customer (or company) has successfully converted to sales in the past. Category 10 (510) can be usefully utilized to evaluate customer loyalty or repurchase potential. A high value implies a strong business relationship with the customer and can be understood as indicating a high likelihood of future transactions.

[0103] Category 11 (e.g., “id_strategic_ver”, 511) may include weight values ​​representing the strategic importance of a combination of a specific Business Unit and a specific Business Area (Vertical Level 1). Category 11 (511) can be used to optimize resource allocation by reflecting the company’s strategic priorities and to increase concentration in a specific Business Area.

[0104] Category 12 (e.g., “it_strategic_ver”, 512) may include weight values ​​indicating strategic importance for a combination of a specific Business Unit and a specific Vertical Level 1, similar to Category 11 (511). This is a weight for a specific Business Unit (e.g., IT Business Unit), which allows for the efficient deployment and planning of technical personnel.

[0105] Category 13 (e.g., “idit_strategic_ver”, 513) may represent a composite indicator that combines Category 11 (511) and Category 12 (512). If at least one of Category 11 (511) and / or Category 12 (512) has a value of 1, Category 13 (513) may be assigned a weight of 1. Category 13 (513) provides an integrated strategic importance encompassing ID and IT domains and can be used as a factor in determining enterprise-wide resource allocation.

[0106] Category 14 (e.g., “customer_job”, 514) may refer to categorical data representing job groups. Through Category 14 (514), communication methods that consider job-specific characteristics can be adopted, and customer grouping based on job can be achieved.

[0107] Category 15 (e.g., “lead_desc_length”, 515) may refer to the total length of the lead description text written by the customer. This may indirectly indicate the customer’s level of interest or engagement and may reflect the complexity of the customer’s requirements or problems.

[0108] Category 16 (e.g., “inquiry_type”, 516) may include information classifying the types of customer inquiries. For example, Category 16 (516) may be divided into multiple (e.g., 71) different categories, including product information inquiries, purchase consultations, and quotation requests. This allows for the identification of the customer's purchasing stage and can be an important factor in establishing marketing strategies. Additionally, Category 16 (516) can assist in sales conversion by assigning an appropriate department or person in charge according to the type of inquiry.

[0109] Category 17 (e.g., “product_category”, 517) may refer to a parent category of the requested product. For example, Category 17 (517) may be divided into multiple categories (e.g., 357) including tablets, TVs, washing machines, refrigerators, etc. Through this, a marketing strategy focused on the category desired by the customer can be established.

[0110] Category 18 (e.g., “product_subcategory”, 518) can classify the subcategories of the requested product more specifically. For example, Category 18 (518) can include a detailed product classification system by being divided into multiple (e.g., 330) subcategories, such as OLED, QLED, and 8K within TVs. Through this, precise customer needs can be identified and more segmented marketing can be provided.

[0111] Category 19 (e.g., “product_modelname”, 519) may refer to the model name of a specific product requested by a customer. For instance, since customers provide very specific information, their interests can be accurately identified. Based on this, customized proposals can be created, and personalized sales approaches can be developed. As a result, customer satisfaction can be increased and sales conversion rates improved.

[0112] Category 20 (e.g., “customer_position”, 520) may refer to the position of the customer who made the inquiry within the company. This allows for the identification of the customer’s level of purchasing decision authority. Additionally, Category 20 (520) can be an important factor in establishing differentiated sales and marketing strategies based on position.

[0113] Category 21 (e.g., “response_corporate”, 521) may be string-type data representing the corporate name of the company responsible for customer inquiries or transactions. Category 21 (521) can play an important role in a corporate structure with multiple subsidiaries. By identifying which corporate entity is primarily involved in customer response or sales processes through Category 21 (521), the locus of responsibility between internal organizations can be clarified and consistency in customer management can be maintained. Additionally, through Category 21 (521), insights necessary for performance analysis by each corporate entity, optimization of resource allocation, and establishment of company-wide sales strategies can be obtained.

[0114] Category 22 (e.g., “expected_timeline”, 522) may refer to the deadline for processing tasks requested by the customer. This can be utilized as an important indicator in the prediction model. This is because the fact that the customer has presented a specific schedule may be a signal of high purchase intent. Additionally, the likelihood and speed of transaction completion can be estimated based on the urgency of Category 22 (522). For example, a short deadline may imply quick decision-making and a high conversion rate, while a long deadline may imply a larger transaction or a complex decision-making process. Effectively utilizing Category 22 (522) can help optimize resource allocation for the sales team and establish customer-tailored approach strategies. In other words, Category 22 (522) can be a factor contributing to the improvement of B2B sales conversion rates.

[0115] Category 23 (e.g., “ver_cus”, 523) may be a category that quantifies the impact of a combination of a specific business area and a customer type on sales conversion in B2B sales. A weight of 1 may be assigned if the customer type is an end consumer while belonging to a specific business area. This allows for the assessment of the potential for success of sales targeting direct end users in a specific business area. Category 23 (523) reflects the importance of customer segmentation in B2B sales strategies and can help identify business areas where an end-user-centric approach may be more effective.

[0116] Category 24 (e.g., “ver_pro”, 524) may be a category that assigns weights to combinations of a specific business area (Vertical Level 1) and a product type (Product Category). Category 24 (524) can be used to determine whether a specific product type shows a higher sales conversion rate in a specific business area. A combination assigned a weight of 1 may mean that the product type is competitive or in high demand in that business area. Category 24 (524) can be used to identify product groups to focus on for each business area and to establish customized sales strategies.

[0117] Category 25 (e.g., “ver_win_rate_x”, 525) may be a composite weighted category that simultaneously considers the relative importance and success rate of each Vertical. It is calculated by multiplying the proportion of the corresponding Vertical among the total leads by the sales conversion success rate within that Vertical. Category 25 (525) allows for a more balanced evaluation by considering not only the success rate but also the overall weight of the corresponding Vertical. Through this, the actual importance of each Vertical can be identified when allocating sales resources and formulating strategies.

[0118] Category 26 (e.g., “ver_win_ratio_per_bu”, 526) may be a category representing the sales conversion success rate for each business unit (or business division) within a specific business area. This can show how effectively each business unit is performing sales in a specific Vertical. Through Category 26 (526), ​​it is possible to identify which specific business unit is performing the highest in each Vertical, and this can be utilized for optimal process sharing and resource allocation optimization within the organization. Additionally, based on Category 26 (526), ​​customized sales strategies can be established to leverage the strengths of each business unit.

[0119] Category 27 (e.g., “business_area”, 527) may be a category representing the client company’s main business area. Category 27 (527) can play a key role in predicting B2B sales conversion rates. By understanding the client company’s business area through Category 27 (527), a customized approach strategy specialized for that industry can be established. Additionally, by analyzing past success patterns in a specific business area through Category 27 (527), sales strategies for new clients in similar industries can be optimized. Through this, efficient allocation of sales resources and improvement of conversion rates can be achieved.

[0120] Category 28 (e.g., “business_subarea”, 528) may be a category that classifies the detailed business areas of a client company more specifically. Category 28 (528) can help to more accurately identify the specific needs or requirements of a client company. By utilizing Category 28 (528) in the prediction model, a highly segmented market approach may be possible. Based on this Category 28 (528), more sophisticated sales strategies can be established and conversion rates can be increased.

[0121] Category 29 (e.g., “lead_owner”, 529) may be a category representing the name of the sales representative in charge of each sales opportunity. Category 29 (529) can play an important role in analyzing the performance of individuals and teams in the predictive model. Additionally, through Category 29 (529), the impact of a specific representative’s sales skills, experience, or expertise in a specific industry on the conversion rate can be identified. Furthermore, through Category 29 (529), an optimal lead allocation strategy can be established, and collaboration patterns among team members can be analyzed to improve overall sales performance.

[0122] Category 30 (e.g., “lead_date”, 530) may be a category representing the date on which a sales opportunity (lead) was first created. Category 30 (530) can be key to considering temporal factors in a prediction model. Additionally, through Category 30 (530), the time elapsed from lead creation to actual transaction completion, seasonal trends, and changes in performance over a specific period can be analyzed. Furthermore, through Category 30 (530), the impact of lead recency on the conversion rate can be identified, allowing for the establishment of effective follow-up strategies in a timely manner. Through this, the sales cycle can be optimized and the conversion rate increased.

[0123] Category 31 (e.g., “lead_from_channel”, 531) may be a category representing a marketing channel from which sales opportunity information was collected. Category 31 (531) may be important for evaluating the effectiveness of each marketing channel in a predictive model. By analyzing the quality and conversion rate of leads acquired through a specific channel based on Category 31 (531), the most effective marketing channel can be identified. Additionally, based on Category 31 (531), marketing budget allocation can be optimized and sales strategies customized for each channel can be established. As a result, lead quality can be improved and the overall sales conversion rate can be increased.

[0124] Category 32 (e.g., “event_name”, 532) may be a category representing the name of a specific marketing event where sales activities took place. Category 32 (532) may be important for evaluating the effectiveness of each marketing event in a predictive model. By analyzing the quality of leads generated through a specific event and the conversion rate based on Category 32 (532), the most successful event types can be identified. Additionally, based on Category 32 (532), future marketing event planning and resource allocation can be optimized, and customized follow-up sales strategies tailored to the characteristics of each event can be established. As a result, event ROI can be improved and the overall sales conversion rate can be increased.

[0125] Category 33 (e.g., “prefer_ver_count”, 533) may be a category representing the distribution ratio of cases converted from a specific business unit to a specific business area. Category 33 (533) may be important for identifying the strengths of each business unit in a predictive model. By analyzing which verticals a specific business unit shows a high success rate based on Category 33 (533), the most effective target markets for each business unit can be identified. Through this, specialized strategies can be established for each business unit. Consequently, the strengths of each business unit can be maximized to improve the overall sales conversion rate.

[0126] Category 34 (e.g., “prefer_ver_mean”, 534) is calculated based on criteria similar to Category 33 (533), but may be a category representing the ratio of profit values ​​instead of the number of simple samples. Category 34 (534) is important for identifying the strengths of each business unit in terms of profitability in the prediction model. By analyzing which verticals a specific business unit generates high profits based on Category 34 (534), strategies can be established that consider actual profit contribution rather than the number of simple successes. Through this, intensive sales activities can be carried out on high-profit verticals, and overall operating profitability can be improved.

[0127] Category 35 (e.g., “transfer_agreement”, 535) may be a category indicating whether a customer has agreed to the transfer of their lead information abroad. Category 35 (535) can be an important indicator in predictive models for evaluating a customer’s openness and potential for global collaboration. Customers who have agreed to the transfer of information are likely to be interested in a wider range of services or global solutions. Based on Category 35 (535), customized proposals for products or services requiring international collaboration can be made, and this can be utilized to establish global sales strategies.

[0128] Category 36 (e.g., “ver_win_rate_mean_upper”, 536) may be a category that uses the average value of each Vertical as a reference point, expressed as 1 if it is exceeded and 0 otherwise. Category 36 (536) may be important for evaluating relative performance within each Vertical in a prediction model. By analyzing the characteristics of cases that perform above average based on Category 36 (536), key elements of a successful sales strategy can be identified. Through this, best practices can be applied to other cases to improve overall sales performance.

[0129] Category 37 (e.g., “expected_budget”, 537) may be a category representing the budget range desired by the customer. Category 37 (537) can serve as an important indicator in the forecasting model to gauge the customer’s purchase intent and project scale. Based on Category 37 (537), suitable products or services can be proposed according to the budget size, and customized solutions that meet the customer’s financial expectations can be developed. Additionally, based on Category 37 (537), the optimal target segment can be identified through an analysis of conversion rates by budget range, and resource allocation can be optimized to improve overall sales performance. In particular, Category 37 (537) may be the category responsible for Money when applying a traditional RFM model.

[0130] Category 38 (e.g., “lead_description”, 538) may be a category containing requirements written directly by the customer. Category 38 (538) can play a key role in identifying the customer’s specific needs and interests in a predictive model. By analyzing Category 38 (538) using text mining and natural language processing (NLP) techniques, potential customer needs and preferences can be identified. Based on Category 38 (538), customized proposals can be created and personalized sales approaches can be developed. As a result, customer satisfaction can be increased and sales conversion rates improved.

[0131] Category 39 (e.g., “is_converted”, 539) is a key category representing the final result of a sales activity, and can express whether a sales activity (or whether a sale was successful) as a binary value (e.g., 1: success, 0: failure). This may be the target category (or specific category) that we ultimately intend to predict in the prediction model. Based on Category 39 (539), the influence of various categories can be analyzed, and the characteristics of successful sales cases can be identified. Additionally, through Category 39 (539), the prediction accuracy of the prediction model can be evaluated, and continuous model improvement and optimization can be performed. Consequently, by accurately predicting Category 39 (539), the overall B2B sales performance can be improved by supporting the efficient allocation of sales resources and strategic decision-making.

[0132] Category 40 (e.g., “len_expected_timeline”, 540) may be a derived category generated during the preprocessing of Category 22 (522). Based on Category 40 (540), the problem of inconsistency in Category 22 (522) can be resolved.

[0133] Category 41 (e.g., “countrycoinside”, 541) may be a derivative category indicating whether the customer’s nationality and the regional information (continent) based on the name of the company in charge are the same. Sales strategies considering regional characteristics can be established based on Category 41 (541).

[0134] Category 42 (e.g., “lead_owner_job”, 542) may be a derived category created by Category 29 (529) to quantify the experience and proficiency of sales representatives in a B2B sales environment. The frequency with which a sales representative appears in the dataset was counted, and a higher frequency was considered to indicate that more sales cases were handled. Based on Category 42 (542), resource allocation can be optimized by assigning experienced representatives to important leads or complex cases, thereby increasing customer satisfaction and improving sales conversion rates.

[0135] Category 43 (e.g., “customer_idx_count”, 543) may be an important indicator (or derived category) of customer loyalty and purchase intent. The number of times each customer appeared in Category 7 (507) was counted, and a high count may indicate that the customer frequently made inquiries for transactions. This indicates continued interest in the product or service and may reflect the intensity of potential purchase intent. Through Category 43 (543), key targets for building long-term business relationships can be identified, and it can be understood that they are highly likely to purchase the company's various products in the future.

[0136] Category 44 (e.g., “oppty”, 544) may be a derivative category designed to predict sales conversion rates in a B2B sales environment. Category 44 (544) can combine the frequency concept of the traditional RFM model with the sales representative’s experience (e.g., “lead_owner_job”) and the customer’s return frequency (e.g., “customer_idx_count”). By quantifying and calculating the synergy between experienced sales representatives and loyal customers, it becomes possible to predict sales performance more accurately by considering the qualitative aspects of the sales relationship, going beyond simple transaction frequency.

[0137] Category 45 (e.g., “vertical_level”, 545) may use an approach that identifies strategically important verticals in each business sector and assigns weights to them. Category 45 (545) may be a derived category created by analyzing existing weighting variables such as Category 11 (511), Category 12 (512), and Category 23 (523). In a specific industry sector, data that has not received weights may be considered as strategically less important verticals in that sector. Based on this logic, Category 45 (545) can filter out data of verticals that are not strategically important and assign additional weights to data corresponding to important verticals. Through this, the most promising verticals in each business sector can be effectively identified, and customized sales strategies can be established accordingly, thereby contributing to the improvement of overall business performance.

[0138] Category 46 (e.g., “weight_expected_timeline”, 546) is a derivative category used as an important indicator in the B2B sales process to predict the timeline for transactions with customers. The original data for Category 46 (546) included email addresses or consultation details that were irrelevant to the actual timeline. However, Category 46 (546) was improved based on the observation that, due to the nature of B2B business, the likelihood of an actual transaction occurring without a clear timeline agreement is low. Specifically, a method was applied to assign weights to data containing words indicating dates or periods. By doing so, higher importance was assigned to data that is likely to contain actual timeline information, thereby enabling more accurate prediction of the likelihood of sales conversion. This approach can increase the efficiency of the B2B sales process and contribute to the establishment of more accurate sales strategies.

[0139] Category 47 (e.g., “qcut”, 547) is a method of dividing numerical data into intervals based on quantiles. Traditional RFM models use qcut to divide data into a specific number of groups, ensuring that the number of data points in each group is equal. This allows for a better reflection of the characteristics of each group. The appropriate number of groups was determined by visualizing the importance of variables. The eight derived categories using qcut were generated by applying a method that splits various numerical data into multiple groups with equal frequency. The data is divided so that the number of data points in each group is nearly equal. This is a methodology frequently used in traditional RFM models. This approach minimizes the influence of extreme values ​​and enables effective comparison of characteristics between groups. The appropriate number of groups was determined based on the results of analyzing variable importance through visualization. This method allows for clearer revelation of the unique characteristics of each group and enables flexible application to various analysis techniques by leveraging the advantages of categorical data while maintaining the characteristics of continuous variables.

[0140] Category 48 (e.g., “lead_date_yearmonth”, 548) may be a time-based variable (derived category) created by combining the year and month at the time of a customer lead. The process of creating Category 48 (548) is as follows: Category 30 (530) was grouped into various time units such as months, years, half-years, and quarters, and then analyzed. Among the various time units, the form combining the year and month showed the highest correlation, so this was adopted. Through this, the business cycle of the company can be reflected. The yearly element takes into account that the company’s product lineup or strategy changes over the years, while the monthly element reflects the tendency of the client company’s purchasing cycle or budget execution pattern to be concentrated in specific months. Through Category 48 (548), customer behavior patterns over time can be captured more accurately, and useful insights can be provided for establishing marketing strategies for each period.

[0141] Category 49 (e.g., “second_event”, 549) may be a derived category created to independently utilize important information extracted from the existing Category 32 (532). Category 32 (532) may have a structure such as “(business_unit)(second_event)(lead_from_channel)(date)”. In this structure, other elements, excluding “second_event”, already existed as individual variables. However, “second_event” was not uniquely represented as an independent variable. The “event_name” variable consists of a combination of four elements and is characterized by being highly dispersed overall due to the various values ​​of each element. This can make it difficult to find meaningful patterns during data analysis or modeling. Therefore, by extracting “second_event” as a separate variable, important information can be utilized more effectively. This can contribute to more accurately reflecting the characteristics of the data and increasing the accuracy of the analysis.

[0142] As seen above, the training data set (410) may include MQL data configured to have values ​​for multiple different categories (501 to 550) and multiple categories (501 to 550).

[0143] Category 50 (e.g., “is_fresh”, 550) may be a derived category created to improve the accuracy of customer classification. Category 50 (550) can classify customers into types such as completely new customers, customers who have previously made inquiries but did not proceed with an actual transaction, and customers who have had a transaction experience in the past. This segmentation can provide important insights for establishing sales strategies. This is because the approach and potential for success differ depending on each customer type. In particular, the second type of customers may have different needs and expectations than completely new customers, so classifying them separately can help with effective customer management.

[0144] Meanwhile, the control unit (140) can perform preprocessing on the training data set (410) (see FIG. 4).

[0145] First, the control unit (140) can clean the training data set (410) to process errors or missing values ​​in the training data set (410) and detect and remove outliers or duplicate records.

[0146] In one embodiment, the control unit (140) can replace missing values ​​in the training data set (410) with average values ​​or delete them, and identify and remove outliers and duplicate records.

[0147] And, if the training data set (410) includes categorical data, the control unit (140) can convert the categorical data into numerical data that the prediction model can understand.

[0148] In one embodiment, the control unit (140) can convert a specific category (e.g., “is_converted”, 539) into numerical data (e.g., “1”, “0”) that a prediction model can understand by using at least one of one-hot encoding and / or label encoding.

[0149] Furthermore, the control unit (140) can adjust the range of numeric data and / or continuous data when at least one of numeric data and / or continuous data is included in the learning data set (410).

[0150] In one embodiment, the control unit (140) can convert numeric data into data with a mean of 0 and a variance of 1 through Z-Score normalization of numeric data, or convert continuous data into a range between 0 and 1 through Min-Max scaling of continuous data.

[0151] Meanwhile, the control unit (140) can generate a new category (or variable or data) from the training data set (410) through a feature engineering process on the training data set (410).

[0152] Specifically, the control unit (140) can perform feature engineering on a training data set (410) in which existing categories have been preprocessed (e.g., refined, normalized, scaled, etc.) to generate derived categories using at least some of the categories and values ​​corresponding to at least some of the categories included in the training data set (410).

[0153] Here, “creating a derived category” can be understood as extracting additional information (or meaning) from an existing category (or original category) or creating a new category (derived category).

[0154] First, the control unit (140) can generate a derived variable for at least one of a plurality of categories by considering the domain (see FIG. 4).

[0155] Specifically, the control unit (140) can generate a derived category based on specific domain knowledge (or an analysis technique specialized for a specific domain) using at least one category among a plurality of categories and a value corresponding to said at least one category. In this case, the control unit (140) can understand which category is important and which combination is meaningful through the specific domain knowledge.

[0156] In one embodiment, as illustrated in FIG. 5, the control unit (140) can generate a derived category (e.g., “lead_date_yearmonth”, 548) using an existing category (e.g., “lead_date”, 530) and a value corresponding to the existing category (e.g., “2024-08-09”) based on expertise in a specific domain (e.g., a marketing domain). The derived category (548) can be understood as a category used to analyze lead data at a specific point in time.

[0157] And, the control unit (140) can specify a value corresponding to a derived category based on the fact that a derived category is created from an existing category. For example, the control unit (140) can specify a value corresponding to a derived category (ex: “2024-08”) based on the fact that a derived category (ex: “lead_date_yearmonth”, 548) is created from an existing category (ex: “lead_date”, 530) and a value corresponding to the existing category (ex: “2024-08-09”).

[0158] In addition, as seen above, the present invention can generate derived categories from a training data set (410) based on RFM (Recency-Frequency-Monetary) analysis.

[0159] More specifically, the control unit (140) can extract at least one category with high feature importance and a value corresponding to at least one category from the training data set (410) based on RFM analysis, and generate a derived category using the extracted category and the value corresponding to the extracted category.

[0160] In one embodiment, as illustrated in FIG. 5, the control unit (140) can extract a seventh category (e.g., “customer_idx”, 507) and a value corresponding to the seventh category (e.g., “CompanyA-1”), a 29th category (e.g., “lead_owner”, 529) and a value corresponding to the 29th category (e.g., “John Doe”) from a training data set (410) based on RFM analysis, and generate a derived category using each of the extracted categories (507, 529) and the value corresponding to each of the extracted categories. In this case, at least one derived category may be created among Category 42, which indicates the level or frequency of the representative’s experience (e.g., “lead_owner_job”, 542), Category 43, which indicates whether or not the customer makes repeat purchases or the frequency (e.g., “customer_idx_count”, 543), and Category 44, which combines the sales representative’s experience and the customer’s revisit frequency (e.g., “oppty”, 544).

[0161] And, the control unit (140) can specify a value corresponding to (or corresponding to) a derived category. For example, the control unit (140) can specify a value (ex: “25”) corresponding to category 42 (ex: “lead_owner_job”, 542), a value (ex: “10”) corresponding to category 43 (ex: “customer_idx_count”, 543), and a value (ex: “0.85”) corresponding to category 44 (ex: “oppty”, 544).

[0162] Through this, the training data set (410) may further include derived categories and values ​​corresponding to the derived categories generated through the process of generating derived variables (or feature engineering).

[0163] In this way, the present invention can improve the performance of a prediction model by generating new derived variables from existing data, thereby enabling the prediction model to learn meaningful patterns.

[0164] Meanwhile, in the present invention, a process of configuring a plurality of different sub-data sets using a training data set may be carried out (S320, see FIG. 3).

[0165] The control unit (140) can configure multiple different sub-data sets using the learning data set (410).

[0166] In the present invention, “configuring a plurality of different sub-data sets” can be understood as configuring each of a plurality of different sub-data sets using at least some of a plurality of records such that the composition ratio of each record containing different values ​​for a specific category among a plurality of records satisfies a preset standard.

[0167] The control unit (140) can configure multiple different sub-data sets based on a value corresponding to a specific category among multiple categories (501 to 550).

[0168] To this end, the control unit (140) can specify a specific category that serves as a standard for constituting different sub-data sets among a plurality of categories (501 to 550).

[0169] Here, the specific category may be a category indicating whether a customer has made a purchase conversion. The control unit (140) may specify a category, specifically “Category 39 (e.g., “is_converted”, 539)”, which corresponds to the category indicating whether a customer has made a purchase conversion among the plurality of categories (501 to 550). For convenience of explanation, the specified “Category 39 (539)” will be referred to as “Specific Category (539)” for the following description.

[0170] As seen above, a specific category (539) is a category representing the final result of a sales activity, and the success or failure of the sales (e.g., when a sales goal such as signing a contract and / or purchasing a product is achieved) can be expressed as a binary value (e.g., success is “1”, failure is “0”).

[0171] In this case, specific categories (539) may have different values ​​depending on whether a customer's purchase conversion has occurred.

[0172] Here, different values ​​may include a first value and a second value. More specifically, the first value may be a value corresponding to the case where a customer's purchase conversion occurs, and the second value may be a value corresponding to the case where a customer's purchase conversion does not occur.

[0173] That is, the value corresponding to a specific category (539) may have a first value and a second value depending on whether the customer's purchase conversion has occurred.

[0174] Furthermore, a specific category (539) may correspond to a “target category” that the learning target prediction model intends to predict. For example, in the present invention, the influence of a plurality of categories (e.g., 501 to 538 and 540 to 550, etc.) can be analyzed based on a specific category (539), and the characteristics of a successful sales case can be identified.

[0175] However, in the present invention, the specific category is not necessarily limited to “Category 39 (539).” For example, the specific category and the value corresponding to the specific category may vary depending on the purpose or use of the prediction system (100), and the specific category may be specified as multiple categories rather than necessarily just one.

[0176] In one embodiment, let us assume that a 43rd category (e.g., “customer_idx_count”, 543) representing customer loyalty and purchase intent is specified as a specific category. In this case, the first and second values ​​corresponding to the 39th category (539) and the first and second values ​​corresponding to the 43rd category (543) specified as a specific category may be different from each other. The first value corresponding to the 43rd category (543) may be a value corresponding to a high purchase intent of the customer (e.g., high purchase intent “1”), and the second value may be a value corresponding to a low purchase intent of the customer (e.g., low purchase intent “0”).

[0177] Meanwhile, the control unit (140) can analyze the training data set (410) to form a plurality of different sub-data sets.

[0178] Here, “analyzing the training data set (410)” can be understood as understanding a plurality of records (or data) included in the training data set (410) and determining (or analyzing) what value each record has based on the results of understanding.

[0179] As seen above, records may contain data values ​​corresponding to each category.

[0180] Specifically, the control unit (140) can analyze a plurality of records included in a learning data set (410) based on a specific category (539), and based on the analysis results, classify each of the plurality of records into a first record containing a first value for the specific category (539) and a second record containing a second value for the specific category (539).

[0181] In one embodiment, let us assume that there are a total of “59,299” data items included in the training data set (410). Based on the analysis results for the training data set (410), the control unit (140) can classify records included in the training data set (410) that contain a first value for a specific category (539) as the first records, and records that contain a second value for a specific category (539) as the second records.

[0182] Next, the control unit (140) can calculate (or calculate) the ratio of the first record and the second record included in the learning data set (410) based on the classified first record and the second record.

[0183] More specifically, the control unit (140) can specify the number of classified first records and second records, respectively, and calculate (or derive) the ratio of first records and second records included in the learning data set (410) based on the specified number.

[0184] In one embodiment, the control unit (140) specifies the number of classified first records as “4,850” and the number of second records as “54,449”, and based on the specified number of first records and second records, can calculate the ratio of first records (e.g., 8.18%) and the ratio of second records (e.g., “91.82%”). In this case, the total ratio of first records and second records can be understood as “1:11”.

[0185] Furthermore, the control unit (140) can determine the number of different sub-data sets to include each of the first record and the second record based on the ratio (or coefficient) of the specified first record and the second record.

[0186] Here, the number of different sub-data sets can be determined based on the number of second records containing a second value for a specific category (539) and the number of first records containing a first value for a specific category (539) among the total number of records included in the training data set (410).

[0187] More specifically, the number of different sub-data sets can be determined based on the value obtained by dividing the number of second records containing a second value for a specific category (539) by the number of first records containing a first value for a specific category (539).

[0188] The control unit (140) can determine the number of different sub-data sets based on the value obtained by dividing the number of second records containing a second value for a specific category (539) included in the training data set (410) by the number of first records containing a first value for a specific category (539). For example, as illustrated in FIG. 6, let us assume that among the total number of records (e.g., “59,299”) included in the training data set (600), there are “4,850” first records containing a first value for a specific category (539) and “54,449” second records containing a second value for a specific category (539). Based on the value obtained by dividing the number of second records containing a second value for a specific category (539) (e.g., “54449”) by the number of first records containing a first value for a specific category (539) (e.g., “4850”), the number of different sub-data sets (601 to 611) can be determined to be “11”.

[0189] Meanwhile, the control unit (140) may include at least some of the records in each of the multiple different sub-data sets (601 to 611) so that the composition ratio of the first record and the second record included in the learning data set (600) satisfies a preset composition ratio standard.

[0190] Here, the pre-set configuration ratio standard may be related to having the same ratio between the number of first records containing a first value for a specific category (539) and the number of second records containing a second value for a specific category (539) for each of the plurality of different sub-data sets (601 to 611).

[0191] That is, the control unit (140) can configure a plurality of different sub-data sets (601 to 611) having the same ratio (e.g., balanced) of the number of first records and second records containing different values ​​for a specific category (539).

[0192] First, the control unit (140) can include a first record containing a first value for a specific category (539) in each of a plurality of different sub-data sets (601 to 611).

[0193] At this time, the control unit (140) can include the first record in each of a plurality of different sub-data sets (601 to 611) while maintaining the original number of first records containing the first value for a specific category (539).

[0194] More specifically, the control unit (140) can include the first records having a first value for a specific category (539) in each of a plurality of different sub-data sets (601 to 611) while maintaining the original number of first records having a first value for a specific category (539) so that all first records having a first value for a specific category (539) are included among the records included in the training data set (600).

[0195] In this case, it can be understood that each of the multiple different sub-data sets (601 to 611) contains the same first record.

[0196] Next, the control unit (140) may include a portion of the second record containing a second value for a specific category (539) among the records included in the training data set (600) in each of the plurality of different sub-data sets (601 to 611).

[0197] Here, the number of second records included in each of the plurality of different sub-data sets (601 to 611) can be determined based on the number of first records included in each of the plurality of different sub-data sets (601 to 611).

[0198] The control unit (140) can include a portion of the second record in each of the multiple different sub-data sets (601 to 611) such that the second record is included in equal numbers of the first records included in each of the multiple different sub-data sets (601 to 611).

[0199] In this case, in each of the plurality of different sub-data sets (601 to 611), different second records are extracted equal to the number of first records included in each of the plurality of different sub-data sets (601 to 611), and the extracted different second records may be included in each of the plurality of different sub-data sets (601 to 611).

[0200] In one embodiment, during the process of extracting a portion of the second record, the control unit (140) may extract each of the different second records a number of times corresponding to the number of multiple sub-data sets (601 to 611) based on the number of (e.g., “11”). The number of each of the different second records may correspond to the number of first records included in each of the multiple different sub-data sets (601 to 611), and the control unit (140) may include each of the different second records in each of the multiple different sub-data sets (601 to 611).

[0201] That is, each of the multiple different sub-data sets (601 to 611) may include as many different second records as the number of first records.

[0202] However, in the embodiment described above, the process of configuring (or determining) “11” different sub-data sets was explained, but the number of sub-data sets in the present invention is not necessarily limited thereto. The number of sub-data sets may vary depending on the total number of records included in the training data set or the ratio (or number) of the first record and the second record.

[0203] In one embodiment, let us assume that the total number of records included in the training data set is “60,000,” and among them, there are “8,000” first records containing a first value for a specific category (539) and “52,000” second records containing a second value for a specific category (539). The control unit (140) can determine the number of different sub-data sets to be “7” based on the value obtained by dividing the number of second records containing a second value for a specific category (539) (e.g., “52,000”) by the number of first records containing a first value for a specific category (539) (e.g., “8,000”).

[0204] In another embodiment, let us assume that the total number of records included in the training data set is “50,000,” and among them, there are “3,000” first records containing a first value for a specific category (539) and “47,000” second records containing a second value for a specific category (539). The control unit (140) can determine the number of different sub-data sets to be “16” based on the value obtained by dividing the number of second records containing a second value for a specific category (539) (e.g., “47,000”) by the number of first records containing a first value for a specific category (539) (e.g., “3,000”).

[0205] In this way, the present invention may form different sub-data sets in which first records and second records containing different values ​​have the same ratio, and each sub-data set may be used independently for model training. Through this, the present invention can solve the problem of data imbalance and prevent the model from overfitting to a specific class, thereby improving the predictive performance of the model.

[0206] Meanwhile, in the present invention, a process of training a target prediction model for each different sub-data set may be carried out (S330, see FIG. 3).

[0207] Furthermore, in the present invention, a process of obtaining a plurality of learned prediction models, each trained with different sub-data sets based on learning, may be carried out (S340, see FIG. 3).

[0208] As described above, the present invention may include at least one prediction model to be the subject of learning. For example, the prediction system (100) may include at least one of a first model (121), a second model (122), and a third model (123) to be the subject of learning.

[0209] In this case, the prediction model to be trained may be a prediction model based on the GBDT (Gradient Boosting Decision Tree) algorithm. However, it is obvious that the training method according to the present invention is not necessarily limited to prediction models based on the GBDT algorithm, but can be applied to various models.

[0210] As illustrated in FIGS. 6 and 7, the control unit (140) can process a plurality of different sub-data sets (601 to 611) as inputs to each of the plurality of prediction models (121, 122, 123) in order to independently train each of the plurality of prediction models (121, 122, 123).

[0211] Specifically, the control unit (140) can train a plurality of prediction models (121, 122, 123) for each of a plurality of different sub-data sets (601 to 611). In this case, each of the plurality of prediction models (121, 122, 123) can receive a plurality of different sub-data sets (601 to 611) as input and perform training on each of the plurality of different sub-data sets (601 to 611).

[0212] In one embodiment, each of the first model (121), the second model (122), and the third model (123) can independently perform learning on each of the plurality of different sub-data sets (601 to 611).

[0213] The control unit (140) trains a plurality of prediction models for each of the plurality of different sub-data sets (601 to 611), and when the training of the plurality of prediction models (121, 122, 123) is completed, it can obtain a plurality (e.g., N) of trained prediction models trained for each of the plurality of different sub-data sets (601 to 611) (see FIG. 4).

[0214] The control unit (140) can obtain a number of multiple learned prediction models (e.g., “33”) by multiplying the number of different sub-data sets (601 to 611) N (e.g., “11”) and the number of multiple prediction models (e.g., “3”) M (e.g., “3”) by which multiple prediction models (121, 122, 123) are trained.

[0215] First, when a plurality of different sub-data sets (601 to 611) are input to the first model (121), the first model (121) can perform learning on each of the plurality of different sub-data sets (601 to 611). In this case, the control unit (140) can obtain a plurality of learned prediction models (e.g., 11) each trained on the plurality of different sub-data sets (601 to 611) as a result of learning on each of the plurality of different sub-data sets (601 to 611) of the first model (121). For example, as illustrated in FIGS. 7 and 8, the control unit (140) can train a first model (121) for each of the first sub-data set (e.g., “DS 1 (Balanced Data Set 1)”, 601), the second sub-data set (e.g., “DS 2 (Balanced Data Set 2)”, 602) to the Nth sub-data set (e.g., “DS 2 (Balanced Data Set 2)”, or the 11th sub-data set, 611), and obtain a plurality of trained prediction models (121a, 121b, 121c) trained for each of the first sub-data set (601), the second sub-data set (602) to the Nth sub-data set (611).

[0216] Additionally, when multiple different sub-data sets (601 to 611) are input to the second model (122), the second model (122) can perform learning on each of the multiple different sub-data sets (601 to 611). In this case, the control unit (140) can obtain multiple learned prediction models (e.g., 11) each trained on the multiple different sub-data sets (601 to 611) as the result of learning on each of the multiple different sub-data sets (601 to 611) of the second model (122). For example, the control unit (140) can train a second model (122) for each of the first sub-data set (e.g., “DS 1 (Balanced Data Set 1)”, 601), the second sub-data set (e.g., “DS 2 (Balanced Data Set 2)”, 602) to the Nth sub-data set (e.g., “DS 2 (Balanced Data Set 2)”, or the 11th sub-data set, 611), and obtain a plurality of trained prediction models (122a, 122b, 122c) trained for each of the first sub-data set (601), the second sub-data set (602) to the Nth sub-data set (611).

[0217] Furthermore, when multiple different sub-data sets (601 to 611) are input to the third model (123), the third model (123) can perform learning on each of the multiple different sub-data sets (601 to 611). In this case, the control unit (140) can obtain multiple learned prediction models (e.g., 11) each trained on the multiple different sub-data sets (601 to 611) as a result of learning on each of the multiple different sub-data sets (601 to 611) of the third model (123). For example, the control unit (140) can train a third model (123) for each of the first sub-data set (e.g., “DS 1 (Balanced Data Set 1)”, 601), the second sub-data set (e.g., “DS 2 (Balanced Data Set 2)”, 602) to the Nth sub-data set (e.g., “DS 2 (Balanced Data Set 2)”, or the 11th sub-data set, 611), and obtain a plurality of trained prediction models (123a, 123b, 123c) trained for each of the first sub-data set (601), the second sub-data set (602) to the Nth sub-data set (611).

[0218] That is, when the training of each of the multiple prediction models (121, 122, 123) for each of the N different sub-data sets is completed, each of the multiple prediction models (121, 122, 123) may include multiple trained prediction models trained for each of the N sub-data sets. In this case, it can be understood that it includes multiple trained prediction models equal to the product of the number of different sub-data sets N and the number of multiple prediction models M.

[0219] Through the process described above, the control unit (140) can obtain a number of multiple learned prediction models (e.g., 33) equal to the number of different sub-data sets (601 to 611) “11” and the number of multiple prediction models (121, 122, 123) “3”.

[0220] However, the number of multiple learned prediction models obtained may vary depending on the number of sub-data sets (N) and the number of prediction models (M).

[0221] In one embodiment, let us assume that the number of different sub-data sets is “20” and the number of multiple prediction models is “2”. In this case, the number of acquired multiple trained prediction models may be “40”.

[0222] In another embodiment, let us assume that the number of different sub-data sets is “10” and the number of multiple prediction models is “5”. In this case, the number of acquired multiple trained prediction models may be “50”.

[0223] In this way, the present invention can maximize data diversity and improve the generalization performance of the model by independently training each model for different sub-data sets. That is, through the process described above, the present invention can contribute to reducing the problem of overfitting the model and improving generalization performance.

[0224] Meanwhile, in the present invention, the process of inputting the input data to be predicted into each of a plurality of trained prediction models may be carried out (S350, see FIG. 3).

[0225] Furthermore, in the present invention, a process of obtaining a plurality of predicted values ​​for input data from each of a plurality of learned prediction models may be performed (S360, see FIG. 6).

[0226] In this case, the input data input to the learned model may vary depending on the purpose or use of the prediction system (100). In the present invention, based on the premise that the purpose of use of the prediction system (100) is related to the field of “marketing and / or business,” the explanation is to be made on the premise that input data related to the field of “marketing and / or business” is input.

[0227] As illustrated in FIGS. 8 and 9, the control unit (140) can process at least one input data (e.g., “Input Data”, 810) as an input to each of a plurality of learned prediction models (121a, 121b, 121c, 122a, 122b, 122c, 123a, 123b, 123c). Here, the input data (810) is, for example, at least among i) categorical data representing the customer's job (e.g., “customer_job”, 811), ii) a variable representing the marketing channel from which sales opportunity information was collected (e.g., “lead_from_channel”, 812), iii) text data containing requirements (or needs) or interests written directly by the customer (e.g., “lead_description”, 813), iv) text data representing the customer's level of interest or engagement (e.g., “lead_desc_length”, 814), v) a variable representing the profit margin generated in a specific Vertical (e.g., “prefer_ver_mean”, 815), vi) a variable representing the top category of the product requested by the customer (e.g., “product_category”, 816), vii) a variable representing the subcategory of the product requested by the customer (e.g., “product_subcategory”, 817), and viii) a variable representing the model name of a specific product requested by the customer (e.g., “product_modelname”, 818). It can include one.

[0228] However, the information included in the input data is not limited to the examples mentioned above and may include various additional data. As one example, the input data may further include customer MQL data and / or customer lead data. As another example, the input data may further include data related to the various categories (501 to 550) discussed above (see FIG. 5).

[0229] The control unit (140) can obtain multiple predicted values ​​for input data from each of the multiple learned prediction models (121a, 121b, 121c, 122a, 122b, 122c, 123a, 123b, 123c).

[0230] More specifically, the control unit (140) can obtain multiple prediction values ​​output from each of the multiple learned prediction models (121a, 121b, 121c) obtained through learning of the first model (121), the multiple learned prediction models (122a, 122b, 122c) obtained through learning of the second model (122), and the multiple learned prediction models (123a, 123b, 123c) obtained through learning of the third model (123).

[0231] In one embodiment, as illustrated in FIGS. 8 and 9, when input data (810) is input to each of the plurality of learned prediction models (121a, 121b, 121c) obtained through the learning of the first model (121), each of the plurality of learned prediction models (121a, 121b, 121c) can output a prediction value for the input data (810). In this case, the prediction model (or the first model, 121a) learned with the first sub-data set (601) outputs a first prediction value (901), the prediction model (121b) learned with the second sub-data set (602) outputs a second prediction value (902), and the prediction model (121c) learned with the Nth sub-data set (611) outputs an Nth prediction value (903).

[0232] In another embodiment, when input data (810) is input to each of the plurality of learned prediction models (122a, 122b, 122c) obtained through the learning of the second model (122), each of the plurality of learned prediction models (122a, 122b, 122c) can output a prediction value for the input data (810). In this case, the prediction model (or second model, 122a) learned with the first sub-data set (601) outputs a first prediction value (911), the prediction model (122b) learned with the second sub-data set (602) outputs a second prediction value (912), and the prediction model (122c) learned with the Nth sub-data set (611) outputs an Nth prediction value (913).

[0233] In another embodiment, when input data (810) is input to each of the plurality of learned prediction models (123a, 123b, 123c) obtained through the learning of the third model (123), each of the plurality of learned prediction models (123a, 123b, 123c) can output a prediction value for the input data (810). In this case, the prediction model (or the third model, 123a) learned with the first sub-data set (601) outputs a first prediction value (921), the prediction model (123b) learned with the second sub-data set (602) outputs a second prediction value (922), and the prediction model (123c) learned with the Nth sub-data set (611) outputs an Nth prediction value (923).

[0234] In this case, the number of multiple prediction values ​​(901, 902, 903, 911, 912, 913, 921, 922, 923) obtained from multiple learned prediction models (121a, 121b, 121c, 122a, 122b, 122c, 123a, 123b, 123c) may correspond to the value obtained by multiplying the number of different sub-data sets N and the number of multiple prediction models M. For example, the control unit (140) may obtain a number of multiple prediction values ​​(ex: “33”) corresponding to the value obtained by multiplying the number of different sub-data sets (601 to 611) “11” and the number of multiple prediction models (121, 122, 123) “3”.

[0235] Meanwhile, in the present invention, a process of determining the final predicted value for the input data using a plurality of predicted values ​​may be performed (S370, see FIG. 3).

[0236] The control unit (140) can determine the final predicted value for the input data (810) using the output of at least one learned prediction model.

[0237] Each of the multiple learned prediction models (121a, 121b, 121c, 122a, 122b, 122c, 123a, 123b, 123c) described above can be configured to predict a value for a specific category (539). For example, each of the multiple learned prediction models (121a, 121b, 121c, 122a, 122b, 122c, 123a, 123b, 123c) can predict whether a customer will make a purchase conversion when input data is input.

[0238] Specifically, the control unit (140) can determine the final predicted value for the input data (810) using a plurality of predicted values ​​(901, 902, 903, 911, 912, 913, 921, 922, 923) obtained from each of a plurality of learned prediction models (121a, 121b, 121c, 122a, 122b, 122c, 123a, 123b, 123c).

[0239] First, the control unit (140) can perform soft voting based on a plurality of predicted values ​​(901, 902, 903, 911, 912, 913, 921, 922, 923) to determine the final predicted value (see FIG. 6).

[0240] Here, soft voting can be understood as one of the ensemble techniques, a method that determines the final prediction by averaging the results (or classes) predicted independently by multiple artificial intelligence models.

[0241] The control unit (140) can calculate (or produce) an averaged probability (or purchase conversion probability, sales conversion probability, final prediction probability, etc.) by performing an averaging of a plurality of prediction values ​​(901, 902, 903, 911, 912, 913, 921, 922, 923) based on soft voting.

[0242] Here, the averaged probability is the result of combining multiple prediction values ​​output by each of the multiple learned prediction models, and can be understood as representing the probability value of the likelihood of a customer purchasing a product or service (purchase conversion likelihood). For example, the control unit (140) can express the probability value as a value between 0 and 1, in which case, a value of 0.7 may mean that the customer has a 70% probability of purchasing the product.

[0243] Furthermore, the control unit (140) can determine the final predicted value (or sales conversion, purchase conversion, customer conversion, etc.) based on the averaged probability.

[0244] Here, the final prediction value can be understood as the prediction result that is finally extracted, representing whether the customer will purchase the product or service through binary classification. For example, the control unit (140) may express it as “purchased (1)” if it is predicted that the customer will purchase the product, and “not purchased (0)” if it is predicted that the customer will not purchase the product.

[0245] In this case, the control unit (140) can compare the averaged probability with a preset threshold and specify the final predicted value as “purchased (1)” if the averaged probability satisfies (or exceeds) the preset threshold, and “not purchased (0)” if it does not satisfy (or exceeds) the threshold.

[0246] The control unit can determine the final predicted value for the input data (810) based on the mathematical formula (930) shown in FIG. 9.

[0247] At this time, let us assume that the sales conversion probability is calculated as “0.7 (70%)” and that a preset threshold is set to “0.65 (65%) or higher”. The control unit (140) can determine whether the averaged probability (e.g., “70%)” satisfies the preset threshold (e.g., “65% or higher”).

[0248] In one embodiment, based on the judgment result, the control unit (140) can specify the final predicted value (e.g., “sales conversion prediction”, 940) as “purchased (1)” based on the fact that the averaged probability (e.g., “70%”) satisfies a preset threshold value (e.g., “65% or more”).

[0249] In another embodiment, let us assume that the averaged probability is calculated to be “0.6 (60%)”. Based on the judgment result, the control unit (140) may specify the final predicted value (e.g., “sales conversion prediction”, 940) as “not buying (0)” based on the fact that the averaged probability (e.g., “60%)” does not satisfy a preset threshold (e.g., “65% or more”).

[0250] In this way, the present invention can improve overall prediction accuracy by combining the output values ​​of each of a plurality of models, thereby offsetting the prediction errors of individual models. This enables more accurate prediction of a customer's purchase conversion potential, which can increase the efficiency of marketing and sales strategies.

[0251] In other words, the present invention calculates a final predicted value by averaging the prediction results of a plurality of learned models, thereby reducing the uncertainty that may arise when relying on a single model and providing an optimized prediction result by making the most of the characteristics of each model.

[0252] Meanwhile, in the inference stage, the method for predicting an effective customer of the prediction system according to the present invention, as illustrated in FIG. 10, can predict whether a customer related to the customer data entered by the user will purchase the company's products or services through the following steps: receiving (or inputting) customer data to be predicted from a user terminal (S1010); inputting the customer data to be predicted into each of a plurality of prediction models, each of which is trained with different sub-data sets divided based on the purchasing customer data among a training data set composed of purchasing customer data and non-purchasing customer data (S1020); obtaining a plurality of prediction values ​​representing the probability that a customer corresponding to the customer data to be predicted is an effective customer as the output of each of the plurality of prediction models (S1030); determining a final prediction value for the customer data to be predicted using the plurality of prediction values ​​(S1040); and providing information to the user terminal regarding whether a customer corresponding to the customer data to be predicted is an effective customer using the determined final prediction value (S1050).

[0253] Here, a valid customer (or valid client company) may refer to a customer who has a clear demand for a specific product or service of a specific company and is highly likely to purchase it.

[0254] In one embodiment, as illustrated in FIG. 11, when the control unit (140) receives the customer data to be predicted from the user terminal (10), it can input the customer data to be predicted into each of a plurality of prediction models, each of which is trained with different sub-data sets divided based on the customer data to be purchased among the training data sets composed of customer data to be purchased and customer data not purchased.

[0255] And, the control unit (140) obtains a plurality of prediction values ​​representing the probability that a customer corresponding to the customer data to be predicted is a valid customer as the output of each of the plurality of prediction models, and can determine a final prediction value for the customer data to be predicted using the plurality of prediction values.

[0256] Furthermore, the control unit (140) can provide information to the user terminal (10) regarding whether a customer corresponding to the predicted target customer data is a valid customer by using a specified final prediction value. For example, as illustrated in FIG. 11, the control unit (140) can provide a prediction result (1021, 1022, 1023) regarding whether a customer (or customer company, U1, U2, U3) related to the customer data (1020) entered by the user will purchase a specific product of a specific company (e.g., “Puricare Objet Collection Water Purifier”) through a service page (1000) displayed on the user terminal (10).

[0257] In this case, the first customer (U1) can be understood to have a very high purchase conversion rate for a specific product (1010) with a purchase probability of “80%”, and the third customer (U3) can be understood to have a low purchase conversion rate for a specific product (1010) with a purchase probability of “30%”.

[0258] As described above, according to the prediction system, the control method thereof, and the learning method of the prediction system of the present invention, by providing a prediction model that has learned various business data, it is possible to effectively respond to various business situations.

[0259] Furthermore, according to the prediction system, the control method thereof, and the learning method of the prediction system of the present invention, learning on balanced learning data can be provided by resolving the problem of imbalance in various business data. Through this, the prediction model learns balanced input data, thereby maintaining stable and high prediction performance even under various inputs (e.g., various situations without being biased toward specific data) during the actual usage phase.

[0260] Furthermore, according to the prediction system, the control method thereof, and the learning method of the prediction system of the present invention, balanced business data is learned to resolve the problem of unbalanced data in actual usage environments. That is, according to the present invention, the generalization performance of the prediction model is enhanced, enabling more accurate sales conversion prediction in actual sales environments, and making it possible to efficiently allocate sales resources and establish optimized sales strategies.

[0261] Furthermore, according to the prediction system, the control method thereof, and the learning method of the prediction system of the present invention, it is possible to provide an environment in which customized sales strategies tailored to customer characteristics can be established by analyzing various customer data. Through this, companies can flexibly respond to various customer types and market environments, thereby strengthening long-term relationships with customers and significantly improving the performance of various businesses. In addition, companies can optimize performance in the global market and establish customized strategies tailored to the characteristics of each country. In other words, according to the present invention, it can provide important insights for a company's strategic decision-making and contribute to improving long-term business performance.

[0262] Meanwhile, the present invention described above can be implemented as a program that is executed by one or more processes on a computer and can be stored on a computer-readable medium (or recording medium).

[0263] Furthermore, the present invention described above can be implemented as computer-readable code or instructions on a medium on which a program is recorded. That is, the present invention can be provided in the form of a program.

[0264] Meanwhile, computer-readable media include all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable media include HDD (Hard Disk Drive), SSD (Solid State Disk), SSD (Silicon Disk Drive), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

[0265] Furthermore, the computer-readable medium may be a server or cloud storage that includes a storage and is accessible to an electronic device via communication. In this case, the computer may download the program according to the present invention from the server or cloud storage via wired or wireless communication.

[0266] Furthermore, in the present invention, the computer described above is an electronic device equipped with a processor, namely a CPU (Central Processing Unit), and no special limitations are placed on its type.

[0267] Meanwhile, the above detailed description should not be interpreted restrictively in all respects but should be considered exemplary. The scope of the invention shall be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the invention are included within the scope of the invention.

Claims

Claim 1 A method in which a memory and at least one processor cooperate to perform the following steps: receiving customer data of a customer who is the subject of a purchase prediction for a specific product; inputting said customer data into at least one prediction model, which is associated with said specific product and is trained with a plurality of sub-data sets generated by dividing non-purchasing customer data based on purchasing customer data included in MQL (Marketing Qualified Lead) data; and the prediction model As output A method for predicting an effective customer, comprising: a step of obtaining a probability value that the customer is an effective customer; and a step of providing a prediction result regarding whether the customer is an effective customer to purchase the specific product through a service page displayed on a user terminal using the probability value, wherein the service page provides product information of the specific product and customer information related to the customer, wherein the customer information includes purchase probability information of the customer as the prediction result, and wherein the purchase probability information is calculated based on the probability value obtained from the prediction model trained on a plurality of sub-data sets generated by dividing the non-purchasing customer data. Claim 2 delete Claim 3 A valid customer prediction method according to claim 1, characterized in that, when there are multiple customers, the service page includes purchase probability information for the specific product for each of the multiple customers. Claim 4 delete Claim 5 delete Claim 6 A valid customer prediction method according to claim 1, characterized in that the plurality of sub-data are configured based on the purchasing customer data so that the purchasing customer data and the non-purchasing customer data satisfy a pre-set configuration ratio standard. Claim 7 A valid customer prediction method according to claim 6, characterized in that the above-mentioned pre-set configuration ratio standard relates to having the same ratio between the number of purchasing customer data and the number of non-purchasing customer data included in each of the plurality of different sub-data sets. Claim 8 A valid customer prediction method according to claim 7, characterized in that the number of the plurality of different sub-data sets is determined based on the number of purchasing customer data and the number of non-purchasing customer data among the total number of records included in the MQL data. Claim 9 A valid customer prediction method according to claim 1, wherein in the step of inputting customer data, the customer data is input to a plurality of prediction models each trained with the MQL data, and in the step of obtaining the probability value, a plurality of prediction values ​​are obtained from each of the plurality of prediction models, and a probability value is determined that the customer is a valid customer using the plurality of prediction values. Claim 10 A valid customer prediction method according to claim 9, wherein the plurality of prediction models are composed of prediction models based on the GBDT (Gradient Boosting Decision Tree) algorithm, and the plurality of prediction models are each trained for each of the plurality of sub-data sets. Claim 11 A valid customer prediction method according to claim 10, characterized in that, in the step of obtaining the probability value, soft voting is performed based on the plurality of prediction values ​​obtained from each of the plurality of prediction models. Claim 12 A valid customer prediction method according to claim 1, wherein the customer data includes at least one of the following: the customer's name, account, contact information, email address, job title, location information, country of affiliation, and company of affiliation. Claim 13 A valid customer prediction method according to claim 1, wherein the MQL data is collected from a source based on at least one of a pre-configured database, web crawling, API, or a pre-linked server. Claim 14 A valid customer prediction system, wherein the system comprises a memory and at least one processor, and the memory and the processor cooperate to receive customer data of a customer who is the target of a purchase prediction for a specific product, input the customer data into at least one prediction model which is trained on multiple sub-data sets generated by dividing non-purchasing customer data based on purchasing customer data included in MQL (Marketing Qualified Lead) data related to the specific product, and obtain a probability value that the customer is a valid customer as the output of the prediction model, and provide a prediction result regarding whether the customer is a valid customer to purchase the specific product through a service page displayed on a user terminal using the probability value, wherein the service page provides product information of the specific product and customer information related to the customer, and the customer information includes purchase probability information of the customer as the prediction result, and the purchase probability information is calculated based on the probability value obtained from the prediction model trained on multiple sub-data sets generated by dividing the non-purchasing customer data. Claim 15 A program stored on a computer-readable medium that is executed by one or more processes in an electronic device, the program comprising instructions for performing the steps of: receiving customer data of a customer who is the subject of a purchase prediction for a specific product; inputting the customer data into at least one prediction model, which is associated with the specific product and is trained on a plurality of sub-data sets generated by dividing non-purchasing customer data based on purchasing customer data included in MQL (Marketing Qualified Lead) data; obtaining a probability value that the customer is a valid customer as the output of the prediction model; and providing a prediction result regarding whether the customer is a valid customer to purchase the specific product through a service page displayed on a user terminal using the probability value, wherein the service page provides product information of the specific product and customer information related to the customer, and the customer information includes purchase probability information of the customer as the prediction result, and the purchase probability information is calculated based on the probability value obtained from the prediction model trained on a plurality of sub-data sets generated by dividing the non-purchasing customer data.

Images