Method and system for user intention estimation and performance metric linkage from user behavior after ad exposure

Abstract

A method and system for managing advertising performance are disclosed for estimating intent regarding user behavior after an ad exposure and linking it to performance indicators. An advertising performance management method according to one embodiment may include the steps of: generating a user behavior pattern by analyzing a user behavior log after an ad exposure; estimating user intent for each user behavior pattern and converting it into an indicator; selecting a new indicator based on the converted user intent; generating a list of clues based on data aggregating the new indicator and the user behavior pattern; and generating and providing a plan for performance improvement based on the new indicator and the list of clues.

Description

Technology Field

[0001] The following description concerns an advertising performance management method and system for estimating user behavior intent after ad exposure and linking it to performance indicators. Background Technology

[0002] Regarding technologies for managing advertising performance, there are technologies for analyzing advertising effectiveness that consider the click / conversion rate of ads displayed on the same page, ad-related keyword searches and purchases, technologies for providing customized advertising campaigns that consider user age, number of ad views, timing and frequency of related product purchases, and technologies for determining user responsiveness and managing ad execution based on exposure and click frequency for each ad.

[0003] These conventional technologies commonly aim to analyze user behavior after ad exposure and implement ad optimization techniques based on this analysis. However, these prior art technologies focus only on ad clicks, post-click conversions, and purchases, failing to consider detailed intentions that can be inferred from user behavior; consequently, they have limitations in providing clearer insights for performance analysis and improvement. Furthermore, while they mention various attributes to analyze user behavior, they fail to derive the meaning of these attributes and the correlations between them and performance indicators, thereby failing to propose improvement strategies that consider the trends between attributes and performance indicators.

[0004] [Prior Art No.]

[0005] Korean Patent Publication No. 10-2024-0153176 The problem to be solved

[0006] We provide an advertising performance management method and system for estimating intent regarding user behavior after ad exposure and linking it to performance indicators. means of solving the problem

[0007] The present invention provides an advertising performance management method for an advertising performance management system implemented by at least one computer device, wherein the at least one computer device comprises at least one processor, and the advertising performance management method comprises: a step of generating a user behavior pattern by analyzing a user behavior log after an advertisement exposure by the at least one processor; a step of estimating and quantifying user intentions for each user behavior pattern by the at least one processor; a step of selecting a new indicator based on the quantified user intention by the at least one processor; a step of generating a list of clues based on data aggregating the new indicators and the user behavior patterns by the at least one processor; and a step of generating and providing a performance improvement plan based on the new indicators and the list of clues by the at least one processor.

[0008] According to one aspect, the step of generating the user behavior pattern may be characterized by including: a step of collecting the user behavior log; and a step of analyzing the collected user behavior log to pattern the user behavior flow.

[0009] According to another aspect, the step of patterning the user behavior flow may be characterized by including: a step of listing user behaviors in the order of occurrence based on the user behavior log; a step of constructing a transition graph composed of major behaviors and predefined states of the listed user behaviors; and a step of generalizing and patterning each user behavior flow in the transition graph.

[0010] According to another aspect, the step of quantifying may be characterized by comprising: a step of estimating the user intention based on at least one of the final state of the user behavior pattern, the frequency of a specific behavior within the user behavior pattern, the time taken for the user behavior, and the meaning of the user behavior for each user behavior pattern; and a step of generating an indicator according to the estimated user intention by calculating at least one of the frequency, ratio, and cumulative number for the estimated user intention.

[0011] According to another aspect, the step of selecting the new indicator may be characterized by including a step of determining whether to select the new indicator of the indicator according to the user's intention based on at least one of similarity, trend, and correlation between the indicator according to the user's intention and a predefined existing performance indicator.

[0012] According to another aspect, the similarity may be characterized by being determined based on at least one of the average, standard deviation, and similarity calculated through a similarity calculation technique for the values ​​of the indicator according to the user intention and the predefined existing performance indicator over a preset period.

[0013] According to another aspect, the above trend may be characterized by being determined based on at least one of the frequency of increase and the frequency of decrease of the numerical value of the indicator according to the user intention and the predefined existing performance indicator during a preset period.

[0014] According to another aspect, the correlation may be characterized by being determined based on the strength of the correlation coefficient between the indicator according to the user's intention and a predefined existing performance indicator.

[0015] According to another aspect, the step of determining whether to select a new indicator based on the user intent may be characterized by comprising: a step of assuming as an expectation the expected relationship between the indicator based on the user intent and a predefined existing performance indicator; and a step of selecting as the new indicator an indicator in which at least one of the similarity, trend, and correlation satisfies the expectation.

[0016] According to another aspect, the step of determining whether to select a new indicator based on the user intent may be characterized by selecting an indicator as the new indicator in which at least one of the similarity, trend, and correlation exceeds a predefined threshold.

[0017] According to another aspect, the step of generating the above-mentioned list of clues may be characterized by comprising: a step of generating data linking the above-mentioned new indicator and user behavior patterns corresponding to the above-mentioned new indicator; and a step of generating a list of clues including the result of comparing the above-mentioned new indicator of a specific channel with the above-mentioned new indicator of another channel based on the generated data.

[0018] According to another aspect, the step of generating and providing the performance improvement plan may be characterized by generating and providing a performance improvement plan for the new indicator by reflecting the new indicator in a template of a performance improvement plan defined according to the list of clues.

[0019] According to another aspect, the step of generating and providing the performance improvement plan may be characterized by inputting a prompt including the meaning of the new indicator, the relationship between the new indicator and a predefined basic performance indicator, and the list of clues into a large language model, and generating and providing a performance improvement plan based on the output of the large language model.

[0020] According to another aspect, the step of generating and providing the performance improvement plan may be characterized by including: providing to a manager at least one performance improvement plan generated by reflecting the new indicator in a template of a performance improvement plan defined according to the clue list, and two or more performance improvement plans generated using two or more different big language models; and providing a performance improvement plan selected by the manager.

[0021] A computer program stored on a non-transient computer-readable recording medium is provided, which is combined with a computer device and executes the above method on the computer device.

[0022] A non-transient computer-readable recording medium is provided on which a computer program for executing the above method on a computer device is recorded.

[0023] An advertising performance management system implemented by at least one computer device, wherein the at least one computer device includes at least one processor, and the at least one processor analyzes user behavior logs after an advertisement exposure to generate user behavior patterns, estimates and quantifies user intentions for each user behavior pattern, selects new indicators based on the quantified user intentions, generates a list of clues based on data combining the new indicators and the user behavior patterns, and generates and provides a plan for performance improvement based on the new indicators and the list of clues. Effects of the invention

[0024] We can provide an advertising performance management method and system for estimating user behavior intent after ad exposure and linking it to performance indicators.

[0025] By patterning user behavior flows after ad exposure, specific intentions that can be estimated from this can be specified, and the frequency of patterns corresponding to each intention can be aggregated and converted into metrics.

[0026] Among the acquired indicators, those that have semantic similarity and numerical correlation with existing performance indicators are selected and used as new indicators, and performance improvement plans can be generated and provided by utilizing clues that can be derived from each indicator and related patterns. Brief explanation of the drawing

[0027] FIG. 1 is a drawing illustrating an example of a network environment according to an embodiment of the present invention. FIG. 2 is a block diagram illustrating an example of a computer device according to an embodiment of the present invention. FIG. 3 is a diagram illustrating an example of the overall process for advertising performance management in one embodiment of the present invention. FIG. 4 is a diagram illustrating an example of patterning user behavior flow in an embodiment of the present invention. FIG. 5 is a diagram illustrating an example of estimating user intent in an embodiment of the present invention. FIG. 6 is a diagram illustrating an example of indicating user intent in one embodiment of the present invention. FIG. 7 is a diagram illustrating an example of discovering a new indicator that affects performance indicators in an embodiment of the present invention. FIG. 8 is a diagram illustrating an example of combining a novel indicator and a user behavior pattern in an embodiment of the present invention. FIG. 9 is a drawing illustrating an example of a list of clues in an embodiment of the present invention. FIG. 10 is a drawing illustrating an example of a performance improvement method in one embodiment of the present invention. FIG. 11 is a flowchart illustrating an example of an advertising performance management method according to an embodiment of the present invention. Specific details for implementing the invention

[0028] Hereinafter, embodiments will be described in detail with reference to the attached drawings.

[0029] Embodiments of the present invention relate to an advertising performance management method and system for estimating intent regarding user behavior after advertising exposure and linking performance indicators. An advertising performance management system according to embodiments of the present invention may be implemented by at least one computer device. In this case, a computer program according to an embodiment of the present invention may be installed and run on at least one computer device, and at least one computer device may perform an advertising performance management method according to embodiments of the present invention under the control of the run computer program. The aforementioned computer program may be stored on a computer-readable recording medium to be combined with at least one computer device to execute the advertising performance management method on the computer.

[0030] FIG. 1 is a diagram illustrating an example of a network environment according to an embodiment of the present invention. The network environment of FIG. 1 illustrates an example including a plurality of electronic devices (110, 120, 130, 140), a plurality of servers (150, 160), and a network (170). FIG. 1 is an example for explaining the invention, and the number of electronic devices or servers is not limited to that shown in FIG. 1. Furthermore, the network environment of FIG. 1 is merely an example of one of the environments applicable to the present embodiments, and the environments applicable to the present embodiments are not limited to the network environment of FIG. 1.

[0031] Multiple electronic devices (110, 120, 130, 140) may be fixed terminals or mobile terminals implemented as computer devices. Examples of multiple electronic devices (110, 120, 130, 140) include smartphones, mobile phones, navigation systems, computers, laptops, digital broadcasting terminals, PDAs (Personal Digital Assistants), PMPs (Portable Multimedia Players), tablet PCs, etc. For example, FIG. 1 shows the shape of a smartphone as an example of an electronic device (110), but in embodiments of the present invention, the electronic device (110) may substantially refer to one of various physical computer devices capable of communicating with other electronic devices (120, 130, 140) and / or servers (150, 160) via a network (170) using a wireless or wired communication method.

[0032] The communication method is not limited and may include not only communication methods utilizing communication networks (e.g., mobile communication networks, wired internet, wireless internet, broadcasting networks) that the network (170) may include, but also short-range wireless communication between devices. For example, the network (170) may include any one or more networks such as a PAN (personal area network), LAN (local area network), CAN (campus area network), MAN (metropolitan area network), WAN (wide area network), BBN (broadband network), and the Internet. Additionally, the network (170) may include any one or more network topologies such as a bus network, star network, ring network, mesh network, star-bus network, tree or hierarchical network, but is not limited thereto.

[0033] Each of the servers (150, 160) may be implemented as a computer device or multiple computer devices that communicate with multiple electronic devices (110, 120, 130, 140) through a network (170) to provide commands, code, files, content, services, etc. For example, the server (150) may be a system that provides services to multiple electronic devices (110, 120, 130, 140) connected through the network (170).

[0034] FIG. 2 is a block diagram illustrating an example of a computer device according to an embodiment of the present invention. Each of the plurality of electronic devices (110, 120, 130, 140) or servers (150, 160) described above can be implemented by the computer device (200) illustrated in FIG. 2.

[0035] As illustrated in FIG. 2, such a computer device (200) may include memory (210), a processor (220), a communication interface (230), and an input / output interface (240). The memory (210) is a computer-readable recording medium and may include a non-perishable mass storage device such as RAM (random access memory), ROM (read only memory), and a disk drive. Here, a non-perishable mass storage device such as a ROM and a disk drive may be included in the computer device (200) as a separate permanent storage device distinct from the memory (210). Additionally, an operating system and at least one program code may be stored in the memory (210). These software components may be loaded into the memory (210) from a computer-readable recording medium separate from the memory (210). This separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, or a memory card. In another embodiment, software components may be loaded into memory (210) via a communication interface (230) rather than a computer-readable recording medium. For example, software components may be loaded into memory (210) of a computer device (200) based on a computer program installed by files received through a network (170).

[0036] The processor (220) may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to the processor (220) via memory (210) or a communication interface (230). For example, the processor (220) may be configured to execute instructions received according to program code stored in a recording device such as memory (210).

[0037] The communication interface (230) may provide a function for the computer device (200) to communicate with other devices (e.g., storage devices described above) through the network (170). For example, requests, commands, data, files, etc. generated by the processor (220) of the computer device (200) according to program code stored in a recording device such as memory (210) may be transmitted to other devices through the network (170) under the control of the communication interface (230). Conversely, signals, commands, data, files, etc. from other devices may be received by the computer device (200) through the communication interface (230) of the computer device (200) via the network (170). Signals, commands, data, etc. received through the communication interface (230) may be transmitted to the processor (220) or memory (210), and files, etc. may be stored in a storage medium (the permanent storage device described above) that the computer device (200) may further include.

[0038] The input / output interface (240) may be a means for interfacing with an input / output device (250). For example, the input device may include a device such as a microphone, keyboard, or mouse, and the output device may include a device such as a display or speaker. As another example, the input / output interface (240) may be a means for interfacing with a device in which the functions for input and output are integrated into one, such as a touchscreen. At least one of the input / output devices (250) may be configured as a single device with the computer device (200). For example, it may be implemented in a form in which a touchscreen, microphone, speaker, etc., are included in the computer device (200), such as in a smartphone.

[0039] Additionally, in other embodiments, the computer device (200) may include fewer or more components than the components of FIG. 2. However, it is not necessary to clearly illustrate most of the prior art components. For example, the computer device (200) may be implemented to include at least some of the input / output devices (250) described above, or may include other components such as a transceiver, a database, etc.

[0040] FIG. 3 is a diagram illustrating an example of an overall process for managing advertising performance in an embodiment of the present invention. An advertising performance management system (300) according to the embodiment of FIG. 3 may be implemented by at least one computer device. Such an advertising performance management system (300) may pattern the flow of user behavior after an advertisement exposure, specify detailed intentions that can be estimated therefrom, and collect pattern frequencies corresponding to each intention to form an indicator. In addition, the advertising performance management system (300) may select targets among the acquired indicators that have semantic similarity and numerical correlation with existing performance indicators and use them as new indicators, and may generate and provide performance improvement plans by utilizing clues that can be derived from each indicator and related patterns.

[0041] In step (310), the advertising performance management system (300) can collect user behavior logs after an ad is displayed. For example, a system that displays an ad or a system that publishes an ad can collect logs recording the process of a user clicking on an ad or browsing a website after viewing an ad. Additionally, subsequent behavior data such as purchases, adding items to a cart, and page dwell time after an ad is displayed can be stored in the form of logs. These user behavior logs can be collected and stored by an ad server that displays an ad or a platform that publishes an ad (e.g., a website, an app), and in some cases, the publisher system and the ad delivery system may be linked to record them in a common log storage. The advertising performance management system (300) can obtain user behavior logs that are included in or linked to such a publisher system or ad delivery system.

[0042] In step (320), the advertising performance management system (300) can generate user behavior patterns by analyzing collected user behavior logs and patterning user behavior flows. For example, the advertising performance management system (300) can generate user behavior patterns by listing user behaviors in the order of occurrence, constructing a transition graph consisting of major behaviors and states, and generalizing and patterning each user behavior flow.

[0043] In step (330), the advertising performance management system (300) can estimate user intent based on user behavior patterns. For example, the advertising performance management system (300) can estimate the user's detailed intent by considering the final state of each user behavior pattern and the frequency, time, and meaning of specific behaviors within the pattern. The estimated intent can be classified into items such as, for example, 'strong refusal', 'refusal', 'indifference', 'interest', 'strong interest', 'concern', 'strong concern', 'purchase plan', 'additional need', 'purchase', etc.

[0044] In step (340), the advertising performance management system (300) can quantify the estimated user intent. For example, the advertising performance management system (300) can collect pattern frequencies corresponding to each user intent and calculate a simple frequency, ratio, and a cumulative figure of intents of a similar nature. At this time, the advertising performance management system (300) can create groups based on advertising space, creative category, user gender, user age, etc., and report the rate of change over a preset period such as hourly, daily, or weekly.

[0045] In step (350), the advertising performance management system (300) can discover new indicators that affect existing performance indicators. First, the advertising performance management system (300) may assume an expectation for the indicator by considering the correlation between each indicator (e.g., frequency of 'strong rejection', frequency of 'indifference', etc.) and existing performance indicators (e.g., CTR (Click-Through Rate), CVR (Conversion Rate), ROAS (Return On Advertising Spend), etc.) or may not specify an expectation in order to discover unexpected correlations.

[0046] First, when expectations are assumed, the advertising performance management system (300) can measure similarity, trends, and correlations between each indicator and existing performance indicators. In this case, the advertising performance management system (300) can select an indicator that shows results relatively consistent with expectations as a new indicator. Expectations may represent assumptions about the trends expected to be possessed by each indicator and existing performance indicators.

[0047] For example, expectations can be defined for each existing performance indicator using similarity, trends, and correlations. Here, similarity may be based on cases where the numerical averages or standard deviations of indicators over a specific period are similar. In this context, 'similarity' may refer to cases where the numerical average or standard deviation of two indicators collected over a certain period is below a preset threshold. As another example, similarity between two indicators may be determined through similarity calculation techniques, such as cosine similarity or Euclidean distance, regarding the numerical values ​​of the indicators over a specific period.

[0048] Additionally, the trend can be based on cases where the rise and fall of numerical values ​​between indicators over a specific period match with similar frequency. For example, an advertising performance management system (300) can determine the trend between two indicators by simply comparing the rise and fall of numerical values ​​of the indicators or by using a moving average line.

[0049] Additionally, the correlation can be determined by calculating the correlation coefficient between indicators over a specific period to check for positive or negative correlation and measuring the strength (magnitude of the correlation coefficient). For example, an advertising performance management system (300) can determine the correlation between indicators through the Pearson correlation coefficient or the Spearman correlation coefficient.

[0050] Thus, if it is possible to predict whether performance indicators to be compared will have high similarity or whether they will exhibit a positive or negative correlation, the assumptions regarding this can be defined as 'expectations.' For instance, since CTR is highly correlated with user interest in advertisements, it can be expected to have a negative correlation with the frequency of 'strong rejection' and 'rejection,' which are user intentions appearing in behavioral patterns, and thus will not satisfy the criteria for trend or similarity. On the other hand, the frequency of 'strong interest,' another user intention appearing in behavioral patterns, can be expected to have a similar trend and positive correlation with CTR. By defining these assumptions as expectations, if the target indicator satisfies these expectations, it can be selected as a new indicator for that channel.

[0051] In addition, if expectations are not specified, the advertising performance management system (300) can measure the correlation between each indicator and existing performance indicators. In this case, the advertising performance management system (300) can select an indicator that shows a clear correlation as a new indicator. For example, the advertising performance management system (300) can select a target indicator as a new indicator if the correlation coefficient strength exceeds a predefined value without considering expectations. As a more specific example, if the frequency of 'strong deliberation,' which is a user intention appearing in user behavior patterns, does not exceed a correlation coefficient of 0.8 with CTR on page A but exceeds a correlation coefficient of 0.8 on page B, the advertising performance management system (300) can select it as a new indicator with a strong positive correlation applicable only to page B. Here, pages A and B may be examples of channels where advertisements are displayed. Similarity and tendency may also be utilized along with the correlation coefficient under the pretext of "discovering unexpected associations."

[0052] In step (360), the advertising performance management system (300) can collect new metrics and user behavior patterns. Here, collecting new metrics and user behavior patterns may mean generating data that links the metrics discovered as new metrics with the user behavior patterns corresponding to those metrics.

[0053] In step (370), the advertising performance management system (300) can generate a list of clues using data in which new metrics and user behavior patterns are aggregated. The list of clues may be a list of information that can be explicitly obtained through new metrics and user behavior patterns. For example, a list of clues such as "the frequency of 'strong rejection' in Place 1 is higher than in other Places" or "the re-search rate within the user behavior flow in Place 1 is higher than in other Places" may be generated. This list of clues may be generated for each channel (the Place in the example above) that is the subject of analysis. The generation of this list of clues may be obtained through a process of comparing the value of a new metric or user behavior pattern for each channel with the value of a new metric or user behavior pattern of another channel.

[0054] In step (380), the advertising performance management system (300) may present a performance improvement plan based on a list of new indicators and clues. For example, the advertising performance management system (300) may generate and provide a performance improvement plan by applying a rule methodology, a Large Language Model (LM) methodology, and / or a semi-automatic methodology by utilizing the meaning of the list of clues and each new indicator. The review items for the performance improvement plan may include, for example, the appropriateness of the ad creative, the appropriateness of the exposure space, the convenience of the purchasing process, and / or price competitiveness.

[0055] In the rule methodology, a rule refers to using a predefined type of clue and a corresponding performance improvement plan template. For example, regarding the clue "higher frequency of strong rejection than other placements," the advertising performance management system (300) can provide the template "Since the frequency of [intention] regarding the advertisement itself is high, the appropriateness of the ad material and the appropriateness of the exposure placement must be reviewed" by replacing the [intention] part with "strong rejection." In this case, the performance improvement plan provided can be "Since the frequency of strong rejection regarding the advertisement itself is high, the appropriateness of the ad material and the appropriateness of the exposure placement must be reviewed." Additionally, regarding the clue "higher frequency of indifference than other placements," the advertising performance management system (300) can provide the template by replacing the [intention] part with "indifference" using the same template. In this case, the performance improvement plan provided can be "Since the frequency of indifference regarding the advertisement itself is high, the appropriateness of the ad material and the appropriateness of the exposure placement must be reviewed."

[0056] The LLM methodology is a method that utilizes prompts, including the meaning of performance indicators, the relationships between basic indicators, and a diverse list of clues, to provide analysis results through an LLM model or to identify the basis for a phenomenon and measures for improvement.

[0057] For example, the prompt can be configured as shown in Table 1 below.

[0058] You are a professional planner specializing in analyzing advertising performance and identifying areas for improvement. Please familiarize yourself with the meanings of the performance metrics below and the relationships between them, and review the various factors that must be considered in the advertising field. Furthermore, utilizing the characteristics of the given ad placement, identify the basis for the phenomenon and propose a solution for improvement. - Performance Metrics: CTR: Click-through rate, a figure calculated by dividing ad clicks by ad impressions; highly correlated with user interest in the ad. CVR: ... - Characteristics of the ad placement to analyze: ~[Record list of clues]~ - Basis for the phenomenon and improvement plan: (Start generating LLM sentences)

[0059] In the semi-automatic methodology, "semi-automatic" refers to human intervention, and it is a method of presenting or emphasizing only the improvement plan selected by a human among the improvement plans created through rules and the improvement plans generated through two or more types of LLM as the final result. At this time, the advertising performance management system (300) can deliver the aggregated and summarized performance improvement plans to the manager by utilizing a function that aggregates and summarizes the selected results through LLM. FIG. 4 is a diagram illustrating an example of patterning user behavior flow in an embodiment of the present invention. As previously explained, the advertising performance management system (300) can generate user behavior patterns by listing user behaviors in the order of occurrence, constructing a transition graph composed of major behaviors and states, and generalizing and patterning each user behavior flow. At this time, FIG. 4 shows an example of a transition graph. In the transition graph, "Ad Mute" may refer to a case where a user presses the "Yes" button in response to the prompt "Would you like to stop seeing this ad?" during ad exposure, and "Ad Quickback" may refer to an action where a user clicks an ad and returns to the original page within a very short period of time. In this case, the sequence of user actions in the transition graph can become a user behavior pattern. For example, if a user acts in the order of "Start → (Image) Ad Impression → Ad Mute," "Start → (Image) Ad Impression → Ad Mute" can become a user behavior pattern.

[0060] FIG. 5 is a diagram illustrating an example of estimating user intent in an embodiment of the present invention. The embodiment of FIG. 5 illustrates an example of classifying user intent into categories such as 'strong refusal', 'refusal', 'indifference', 'interest', 'strong interest', 'concern', 'strong concern', 'purchase plan', 'additional need', and 'purchase' according to user behavior patterns. For each user behavior pattern derivable from a transition graph, a user intent can be predefined, and depending on what user behavior pattern a specific user exhibits, that user behavior pattern can be defined as a predefined user intent.

[0061] FIG. 6 is a diagram illustrating an example of quantifying user intent in an embodiment of the present invention. The embodiment of FIG. 6 illustrates an example of quantifying user intent by generating an indicator for user intent. More specifically, the embodiment of FIG. 6 illustrates an example of quantifying the frequency of "strong rejection" in a specific channel (main ad space) for a user intent of "strong rejection" in a preset period unit (day unit), and an example of quantifying the frequency of "indifference" in the same channel for a user intent of "indifference" in the same period unit.

[0062] FIG. 7 is a diagram illustrating an example of discovering a new indicator that affects performance indicators in an embodiment of the present invention. The embodiment of FIG. 7 illustrates an example of discovering a new indicator by comparing an indicator of user intention with an existing performance indicator. More specifically, the embodiment of FIG. 7 illustrates an example of comparing the indicatorized user intention of FIG. 6 with CTR as an existing performance indicator. Since the CTR and "strong rejection" frequency for a specific channel (main ad space) show results that meet expectations, the indicator "strong rejection" frequency is selected as a "new indicator showing negative performance," whereas the CTR and "indifference" frequency do not meet expectations, so the indicator "indifference" frequency is excluded from the new indicator candidates.

[0063] FIG. 8 is a diagram illustrating an example of combining new indicators and user behavior patterns in an embodiment of the present invention. The embodiment of FIG. 8 shows an example of data combining new indicators and user behavior patterns. In the embodiment of FIG. 8, for a specific channel (main advertising space), the frequency of user intention "strong rejection," the frequency of "strong deliberation," and the frequency of "purchase plan" are selected as new indicators, and user behavior patterns associated with the new indicators are combined to display user behavior patterns by frequency ranking.

[0064] FIG. 9 is a diagram illustrating an example of a clue list in an embodiment of the present invention. As previously explained, the clue list may be a list of information that can be explicitly obtained through new indicators and user behavior patterns. The embodiment of FIG. 9 illustrates an example in which explicit clues are obtained by comparing the frequency of 'strong rejection', the frequency of 'strong deliberation', the re-search rate, and the frequency of 'purchase plan', each selected as a new indicator in a specific channel (main ad space), with other channels.

[0065] FIG. 10 is a diagram illustrating an example of a performance improvement plan in an embodiment of the present invention. The embodiment of FIG. 10 shows an example of a performance improvement plan generated by utilizing a list of clues and the meaning of new indicators. As previously explained, the performance improvement plan can be generated based on a rule methodology, an LLM methodology, and / or a semi-automatic methodology, and items such as the appropriateness of advertising materials, the appropriateness of exposure placement, the convenience of the purchasing process, and / or price competitiveness may be reviewed.

[0066] FIG. 11 is a flowchart illustrating an example of an advertising performance management method according to an embodiment of the present invention. The advertising performance management method according to the present embodiment may be performed by an advertising performance management system (300) implemented through at least one computer device. At this time, each of the at least one computer device may correspond to the computer device (200) described above through FIG. 2. At least one processor included in the at least one computer device may be implemented to execute a control instruction according to the code of an operating system included in memory or the code of at least one computer program. Here, the at least one processor may operate according to the control instruction provided by the code to control the advertising performance management system (300) so that the advertising performance management system (300) performs steps (1110 to 1150) included in the method of FIG. 11.

[0067] In step (1110), the advertising performance management system (300) can generate user behavior patterns by analyzing user behavior logs after an ad exposure. For example, the advertising performance management system (300) can collect user behavior logs and analyze the collected user behavior logs to pattern the flow of user behavior. At this time, the advertising performance management system (300) can list user behaviors in the order of occurrence based on the user behavior logs and construct a transition graph composed of the main behaviors and predefined states of the listed user behaviors. Subsequently, the advertising performance management system (300) can generalize and pattern each flow of user behavior in the transition graph.

[0068] In step (1120), the advertising performance management system (300) can estimate user intent for each user behavior pattern and convert it into an indicator. For example, the advertising performance management system (300) can estimate user intent for each user behavior pattern based on at least one of the final state of the user behavior pattern, the frequency of a specific behavior within the user behavior pattern, the time taken for the user behavior, and the meaning of the user behavior. Subsequently, the advertising performance management system (300) can generate an indicator based on the estimated user intent by calculating at least one of the frequency, ratio, and cumulative number for the estimated user intent. Here, frequency may refer to the frequency at which the user behavior corresponding to the user intent occurred, ratio may refer to the ratio of the user behavior corresponding to the user intent, and cumulative number may refer to the cumulative number of times the user behavior corresponding to the user intent occurred.

[0069] In step (1130), the advertising performance management system (300) can select new indicators based on the indicatorized user intent. For example, the advertising performance management system (300) can determine whether to select new indicators based on the user intent based on at least one of similarity, trend, and correlation between the indicator based on the user intent and a predefined existing performance indicator.

[0070] Here, similarity may be determined based on at least one of the mean, standard deviation, and similarity calculated using a similarity calculation technique for the values ​​of the indicator based on user intent and the predefined existing performance indicator over a preset period. The predefined existing performance indicator may include at least some of the already well-known performance indicators, such as CTR, CVR, and ROAS, as previously described. Additionally, the trend may be determined based on at least one of the frequency of increase and the frequency of decrease for the values ​​of the indicator based on user intent and the predefined existing performance indicator over a preset period. Furthermore, the correlation may be determined based on the strength of the correlation coefficient between the indicator based on user intent and the predefined existing performance indicator. For example, the correlation coefficient may be calculated using the Pearson correlation coefficient or the Spearman correlation coefficient.

[0071] Additionally, the advertising performance management system (300) may assume an expected relationship between an indicator based on user intent and a predefined existing performance indicator as an expectation. In this case, the advertising performance management system (300) may select an indicator based on user intent where at least one of similarity, trend, and correlation satisfies the expectation as a new indicator. If no expectation is assumed, the advertising performance management system (300) may select an indicator where at least one of similarity, trend, and correlation exceeds a predefined threshold as a new indicator.

[0072] In step (1140), the advertising performance management system (300) can generate a list of clues based on data that combines new metrics and user behavior patterns. For example, the advertising performance management system (300) can generate data that links new metrics and user behavior patterns corresponding to the new metrics. In this case, the advertising performance management system (300) can generate a list of clues that includes the result of comparing the new metrics of a specific channel with the new metrics of another channel based on the generated data.

[0073] In step (1150), the advertising performance management system (300) can generate and provide performance improvement plans based on new indicators and a list of clues. For example, the advertising performance management system (300) can generate and provide performance improvement plans for new indicators by reflecting new indicators in a template of predefined performance improvement plans according to a list of clues. This method was previously explained in detail through the rule methodology. As another example, the advertising performance management system (300) can generate and provide performance improvement plans based on the output of a macro-language model by inputting a prompt containing the meaning of the new indicator, the relationship between the new indicator and the predefined basic performance indicator, and a list of clues into a macro-language model. This method was previously explained in detail through the LLM methodology. Additionally, the advertising performance management system (300) can provide the manager with at least one performance improvement plan generated by reflecting new indicators in a template of predefined performance improvement plans according to a list of clues, and two or more performance improvement plans generated using two or more different macro-language models, and provide the performance improvement plan selected by the manager. This method was previously explained in detail through the semi-automatic methodology.

[0074] As such, according to the embodiments of the present invention, an advertising performance management method and system for estimating intent regarding user behavior after ad exposure and linking it to performance indicators can be provided. This advertising performance management method and system can pattern the flow of user behavior after ad exposure to specify detailed intents that can be estimated therefrom, and collect the frequency of patterns corresponding to each intent to create an indicator. Furthermore, among the acquired indicators, targets that have semantic similarity and numerical correlation with existing performance indicators are selected and used as new indicators, and performance improvement measures can be generated and provided by utilizing clues that can be derived from each indicator and related patterns.

[0075] The system or device described above may be implemented as a hardware component, or a combination of a hardware component and a software component. For example, the device and component described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0076] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or instruct the processing unit independently or collectively. Software and / or data may be embodied in any type of machine, component, physical device, virtual equipment, computer storage medium, or device so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0077] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either individually or in combination. The medium may continuously store a program executable by a computer, or temporarily store it for execution or download. Furthermore, the medium may be various recording or storage means in the form of a single or multiple hardware components, and is not limited to a medium directly connected to a computer system, but may also exist distributed over a network. Examples of media may include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and media configured to store program instructions, including ROM, RAM, and flash memory. Additionally, other examples of media may include recording or storage media managed by app stores that distribute applications or sites and servers that supply or distribute various other software. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0078] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0079] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

Claim 1 An advertising performance management method of an advertising performance management system implemented by at least one computer device, wherein the at least one computer device includes at least one processor, and the advertising performance management method comprises: a step of generating a user behavior pattern by analyzing a user behavior log after an advertisement exposure by the at least one processor; a step of generating an indicator according to the user intention by estimating the user intention for each user behavior pattern by the at least one processor; a step of selecting the indicator according to the user intention as a new indicator based on the result of comparing the indicator according to the user intention and the value of each of the previously defined existing performance indicators during a preset period by the at least one processor; a step of generating data linked to the new indicator and the user behavior pattern corresponding to the new indicator by the at least one processor, and generating a clue list including the result of comparing the new indicator of a specific channel with the new indicator of another channel based on the generated data; and a step of generating and providing a performance improvement plan based on the new indicator and the clue list by the at least one processor. Claim 2 An advertising performance management method according to claim 1, wherein the step of generating the user behavior pattern comprises: a step of collecting the user behavior log; and a step of analyzing the collected user behavior log to pattern the user behavior flow. Claim 3 An advertising performance management method according to claim 2, wherein the step of patterning the user behavior flow comprises: a step of listing user behaviors in the order of occurrence based on the user behavior log; a step of constructing a transition graph composed of major behaviors and predefined states of the listed user behaviors; and a step of generalizing and patterning each user behavior flow in the transition graph. Claim 4 An advertising performance management method according to claim 1, wherein the step of generating an indicator according to the user intention comprises: a step of estimating the user intention based on at least one of the final state of the user behavior pattern, the frequency of a specific behavior within the user behavior pattern, the time taken for the user behavior, and the meaning of the user behavior for each user behavior pattern; and a step of generating an indicator according to the estimated user intention by calculating at least one of the frequency, ratio, and cumulative number for the estimated user intention. Claim 5 An advertising performance management method according to claim 1, wherein the step of selecting an indicator based on the user's intention as a new indicator includes the step of determining whether to select the indicator based on the user's intention as a new indicator based on at least one of similarity, trend, and correlation between the indicator based on the user's intention and a predefined existing performance indicator. Claim 6 An advertising performance management method characterized in that, in paragraph 5, the similarity is determined based on at least one of the average, standard deviation, and similarity calculated through a similarity calculation technique for numerical values ​​over a preset period of the indicator according to the user's intention and the predefined existing performance indicator. Claim 7 An advertising performance management method characterized in that, in paragraph 5, the above trend is determined based on at least one of the frequency of increase and the frequency of decrease of the numerical value of the indicator according to the user intention and the predefined existing performance indicator during a pre-set period. Claim 8 An advertising performance management method characterized in that, in paragraph 5, the correlation is determined based on the strength of the correlation coefficient between the indicator according to the user's intention and the predefined existing performance indicator. Claim 9 An advertising performance management method according to claim 5, wherein the step of determining whether to select a new indicator based on the user intent comprises: a step of assuming as an expectation the expected relationship between the indicator based on the user intent and a predefined existing performance indicator; and a step of selecting as the new indicator an indicator in which at least one of the similarity, trend, and correlation satisfies the expectation. Claim 10 In claim 5, the step of determining whether to select a new indicator based on the user's intent is characterized by selecting an indicator as the new indicator in which at least one of the similarity, trend, and correlation exceeds a predefined threshold. Claim 11 An advertising performance management method according to claim 1, wherein the above list of provisos further includes the result of comparing the user behavior pattern corresponding to the new indicator of a specific channel with the user behavior pattern of another channel. Claim 12 An advertising performance management method according to claim 1, wherein the step of generating and providing the performance improvement plan is characterized by generating and providing the performance improvement plan for the new indicator by reflecting the new indicator in a template of the performance improvement plan defined according to the list of clues. Claim 13 An advertising performance management method according to claim 1, wherein the step of generating and providing the performance improvement plan comprises inputting a prompt including the meaning of the new indicator, the relationship between the new indicator and a predefined basic performance indicator, and the list of clues into a large language model, and generating and providing a performance improvement plan based on the output of the large language model. Claim 14 An advertising performance management method according to claim 1, wherein the step of generating and providing the performance improvement plan comprises: providing to a manager at least one performance improvement plan generated by reflecting the new indicator in a template of a performance improvement plan defined according to the clue list, and two or more performance improvement plans generated using two or more different big language models; and providing a performance improvement plan selected by the manager. Claim 15 A computer program stored on a non-transient computer-readable recording medium that is combined with a computer device and executes the method of any one of claims 1 to 14 on the computer device. Claim 16 An advertising performance management system implemented by at least one computer device, wherein the at least one computer device includes at least one processor, and the at least one processor analyzes a user behavior log after an ad exposure to generate a user behavior pattern, estimates a user intention for each user behavior pattern to generate an indicator according to the user intention, selects the indicator according to the user intention as a new indicator based on the result of comparing the indicator according to the user intention with a value over a preset period for each of the predefined existing performance indicators, generates data linked to the new indicator and the user behavior pattern corresponding to the new indicator, generates a clue list including the result of comparing the new indicator of a specific channel with the new indicator of another channel based on the generated data, and generates and provides a performance improvement plan based on the new indicator and the clue list. Claim 17 An advertising performance management system according to claim 16, characterized in that, in order to generate the above-mentioned user behavior pattern, the above-mentioned user behavior log is collected by the above-mentioned at least one processor, and the collected user behavior log is analyzed to pattern the user behavior flow. Claim 18 An advertising performance management system according to claim 16, characterized in that, in order to generate an indicator according to the above user intention, the above at least one processor estimates the user intention for each user behavior pattern based on at least one of the final state of the user behavior pattern, the frequency of a specific behavior within the user behavior pattern, the time taken for the user behavior, and the meaning of the user behavior, and calculates at least one of the frequency, ratio, and cumulative number for the estimated user intention to generate an indicator according to the estimated user intention. Claim 19 An advertising performance management system according to claim 16, characterized in that, in order to select the indicator according to the user's intention as a new indicator, the at least one processor determines whether to select the indicator according to the user's intention as a new indicator based on at least one of similarity, trend, and correlation between the indicator according to the user's intention and a predefined existing performance indicator. Claim 20 An advertising performance management system characterized in that, in Clause 16, the above-mentioned list of provisos further includes the result of comparing user behavior patterns corresponding to the above-mentioned new indicators of a specific channel with user behavior patterns of another channel.

Images