Method and apparatus for evaluating data delivery, device, and storage medium

By obtaining the indicators and exception types in user configuration information, automatically identifying data delivery abnormalities and alarming, the problem of inability to identify key causes and low efficiency in the existing technology is solved, and the flexibility and efficiency improvement of data delivery evaluation is achieved.

WO2024234969A1PCT designated stage expired Publication Date: 2024-11-21DOUYIN VISION CO LTD

Patent Information

Application Number
PCT/CN2024/089917
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-12
Filing Date
2024-04-25
Publication Date
2024-11-21

AI Technical Summary

Technical Problem

The existing data delivery evaluation methods rely on the delivery platform, cannot give the key reasons for the occurrence of abnormalities, cannot meet the personalized evaluation needs of resource suppliers, and the traditional methods require manual intervention and are inefficient.

Method used

By obtaining the metrics to be evaluated in user configuration information, and comparing the evaluation results of the metrics with thresholds, we automatically identify the abnormal situations of data delivery and alarms, improving the flexibility and efficiency of evaluation.

Benefits of technology

It realizes rapid identification and processing of data delivery abnormalities, improves evaluation effect and user experience, reduces calculation amount, and improves evaluation efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024089917_21112024_PF_FP_ABST
    Figure CN2024089917_21112024_PF_FP_ABST
Patent Text Reader

Abstract

Embodiments of the present disclosure provide a method and an apparatus for evaluating data delivery, a device, and a storage medium. The method comprises: receiving a first selection, by a user, of at least one indicator for evaluating the data delivery and a second selection of at least one exception type for evaluating the data delivery; obtaining user configuration information for evaluating the data delivery on the basis of the first selection and the second selection by the user, wherein the user configuration information at least comprises at least one indicator to be evaluated and at least one exception type; and on the basis of comparison of an evaluation result for the at least one indicator with a corresponding threshold, giving an alarm to the exception of the data delivery, wherein a corresponding threshold of the at least one indicator is associated with the at least one exception type. Therefore, the exception condition of the data delivery can be conveniently and quickly evaluated, an alarm can be given, and the evaluation effect is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Method, device, equipment and storage medium for evaluating data delivery

[0001] This application claims priority to the Chinese invention patent application entitled “Methods, devices, equipment and storage media for evaluating data delivery” and application number 202310539944.5, filed on May 12, 2023. The entire contents of that application are incorporated herein by reference. Technical Field

[0002] Example embodiments of the present disclosure generally relate to the field of computers, and more particularly, to methods, apparatuses, devices, and computer-readable storage media for evaluating data delivery. Background Art

[0003] The internet provides access to a wide variety of resources. For example, applications, products, audio and video content, and more can be accessed online. Furthermore, accessible content includes specific recommended content items related to various objects / resources, such as advertisements. Resource providers with resources can provide content distributors with information about recommended content items. Resource providers (e.g., advertisers) expect their resources to attract a large number of customers.

[0004] Summary of the Invention

[0005] In a first aspect of the present disclosure, a method for evaluating data delivery is provided. The method comprises: obtaining user configuration information for evaluating data delivery, the user configuration information comprising at least one metric to be evaluated and at least one anomaly type; and generating an alarm for an anomaly in the data delivery based on a comparison of an evaluation result for the at least one metric with a corresponding threshold, wherein the corresponding threshold for the at least one metric is associated with the at least one anomaly type.

[0006] In a second aspect of the present disclosure, a device for evaluating data delivery is provided. The device includes: an information acquisition module configured to acquire user configuration information for evaluating data delivery, the user configuration information including at least one indicator to be evaluated and at least one anomaly type; and an anomaly alarm module configured to generate an alarm for an anomaly in the data delivery based on a comparison of an evaluation result for the at least one indicator with a corresponding threshold, wherein the corresponding threshold for the at least one indicator is associated with the at least one anomaly type.

[0007] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.

[0008] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and the computer program can be executed by a processor to implement the method of the first aspect.

[0009] It should be understood that the content described in this summary section is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0011] FIG1 shows a schematic diagram of an example environment in which embodiments of the present disclosure can be implemented;

[0012] FIG2 illustrates a flow chart of a process for evaluating data delivery according to some embodiments of the present disclosure;

[0013] FIG3 is a schematic diagram showing a process of data delivery evaluation according to some embodiments of the present disclosure;

[0014] FIG4 shows a schematic diagram of a process for evaluating data delivery according to some embodiments of the present disclosure;

[0015] FIG5 shows a schematic structural block diagram of an apparatus for evaluating data delivery according to certain embodiments of the present disclosure; and

[0016] FIG6 illustrates a block diagram of an electronic device in which one or more embodiments of the present disclosure may be implemented. DETAILED DESCRIPTION

[0017] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0018] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may be included below. The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may be included below.

[0019] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) must comply with the requirements of relevant laws, regulations and relevant provisions.

[0020] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0021] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

[0022] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0023] It is understandable that the above notification and user authorization process are merely illustrative and do not limit the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0024] As briefly mentioned above, resource providers expect their resources to attract widespread customer attention. To ensure the effectiveness of data delivery, they need to evaluate the data delivery of their resources and promptly address any anomalies. Traditional evaluation methods often rely on delivery platforms or systems, often requiring manual intervention to identify anomalies and providing vague feedback at the operational level. For example, in the case of ad delivery, an ad platform might provide feedback at the "creative" level indicating "high creative duplication." However, traditional evaluation methods fail to identify the key causes of anomalies and fail to identify key issues for resource providers. When optimizing delivery results, users are more focused on resolving these "key issues." Furthermore, delivery platforms often evaluate data delivery from a platform perspective, determining evaluation results and recommendations based on platform data, while resource providers often consider more personalized delivery factors. Therefore, traditional platform-dependent data delivery evaluation methods fail to meet the evaluation needs of resource providers.

[0025] The embodiments of the present disclosure propose a solution for evaluating data delivery. According to this solution, user-configured information, including the metrics to be evaluated and the types of anomalies, is obtained. Data delivery anomalies are determined based on a comparison of the metric evaluation results with corresponding thresholds. If an anomaly is determined, an alarm is generated. In this way, data delivery anomalies can be evaluated based on the user-configured metrics to be evaluated, improving the flexibility and efficiency of the evaluation, enhancing the evaluation results, and enhancing the user experience.

[0026] FIG1 shows a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. As shown in FIG1 , the example environment 100 may include an electronic device 110 .

[0027] In this example environment 100, electronic device 110 may run application 115. Application 115 may be any suitable type of application for evaluating data delivery, such as a simulation application, a data statistics application, or the like. User 120 may be a resource provider requiring data delivery evaluation. In the context of advertising delivery, user 120 may also be referred to as an advertiser. User 120 may interact with application 115 via electronic device 110 and / or its attached devices.

[0028] In some embodiments, in the environment 100 of FIG. 1 , if the application 115 is active, the electronic device 110 may present an interface 140 of the application 115. Through the interface 140, the application 115 may provide one or more services related to data delivery evaluation to the user 120. For example, if the interface 140 is a user configuration interface, the electronic device 110 may determine indicators and anomaly types associated with data detection in response to receiving settings set by the user 120 in the user configuration interface.

[0029] The electronic device 110 can obtain data associated with the delivery data 160 in the delivery platform 150, which may include, for example, the number of customers, customer retention, customer consumption data, etc. The delivery platform 150 may include any appropriate type of application for data delivery, examples of which may include but are not limited to: content sharing applications, social applications, chat applications, shopping applications, etc. The delivery data 160 may include, for example, various objects, examples of which may include applications, physical goods, virtual goods, audio and video content, etc. In some embodiments, the delivery data 160 refers to content presented in order to recommend corresponding resources. Examples of delivery data 160 may include advertisements. A customer group may include one or more customer members, and a customer member may be any potential consumer of a resource, such as a user, group, organization, entity, etc.

[0030] The electronic device 110 may include any computing system with computing capabilities, such as various computing devices / systems, terminal devices, server devices, etc. The terminal device may be any type of mobile terminal, fixed terminal or portable terminal, etc., including a mobile phone, desktop computer, laptop computer, notebook computer, netbook computer, tablet computer, media computer, multimedia tablet, personal communication system (PCS) device, personal navigation device, personal digital assistant (PDA), audio / video player, digital camera / camcorder, positioning device, television receiver, radio broadcast receiver, e-book device, virtual reality (VR) all-in-one machine, game console, game book or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. The server device may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, and big data and artificial intelligence platforms. The server device may include, for example, a computing system / server, such as a mainframe, an edge computing node, a computing device in a cloud environment, and the like.

[0031] It should be understood that the structure and function of the various elements in the environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of the present disclosure.

[0032] Some example embodiments of the present disclosure will be described below with continued reference to FIG. 2 to FIG. 4 .

[0033] 2 shows a flow chart of a process 200 for evaluating data delivery according to some embodiments of the present disclosure. The process 200 may be implemented at the electronic device 110. For ease of discussion, the process 200 will be described with reference to the environment 100 of FIG. 1 .

[0034] At block 205, electronic device 110 receives a user's first selection of at least one metric for evaluating data delivery and a second selection of at least one anomaly type for evaluating data delivery. At block 210, electronic device 110 obtains user configuration information for evaluating data delivery based on the user's first and second selections. The user configuration information includes at least the at least one metric to be evaluated and the at least one anomaly type.

[0035] In some embodiments, electronic device 110 may present a user interface for obtaining user configuration information, such as interface 140 in FIG. 1 , and determine the user configuration information based on user input received in the user interface. In addition to indicators and anomaly types, user configuration information may also include other configuration information for detecting data delivery, such as standards, reference values, or thresholds for determining whether an anomaly has occurred.

[0036] In some embodiments, the electronic device 110 may present multiple indicators and multiple exception types as options in a user interface for user selection. The electronic device 110 may receive a first selection of at least one indicator by the user, and a second selection of at least one exception type by the user, and determine user configuration information based on the first selection and the second selection. For example, the electronic device 110 may present four indicators, namely, indicator A, indicator B, indicator C, and indicator D, and three exception types, namely, exception type A, exception type B, and exception type C, in the user interface. The electronic device 110 may determine that user configuration information including indicator A and exception type A has been received in response to receiving a user selection operation for indicator A and exception type A in the user interface.

[0037] Alternatively or additionally, the user may input user configuration information about indicators and exception types through other means other than user interface operations. For example, the user may input user configuration information through voice instructions (e.g., "evaluation indicator A, exception type is determined to be exception type A"). Accordingly, the electronic device 110 may determine that the user configuration information is received in response to receiving a voice instruction associated with the user configuration. For another example, the user may input user configuration information through an instruction or program (e.g., a string of code). In response to receiving an instruction or program associated with the user configuration, the electronic device 110 may determine that the user configuration information is received.

[0038] In some embodiments, the configuration of indicators and / or exception categories may also be determined by the electronic device 110 based on the desired delivery effect of the delivery data. For example, if the delivery data is an advertisement, the electronic device 110 may use the return on investment (ROI) as an indicator.

[0039] The indicators to be evaluated may include any appropriate indicators that can reflect abnormal data delivery. In some embodiments, at least one indicator in the user configuration information obtained by the electronic device 110 may be associated with at least one of the number of new customers, customer retention, input-output ratio, and cold start. Customers are, for example, the audience for data delivery. The number of new customers may include daily new users (DNU). The input-output ratio is, for example, the ratio between the benefits and costs when a user delivers data. Cold start is, for example, a way of starting, which means starting from zero. In the process of data delivery, "cold start" can be an initial stage in which data has not yet been fully accumulated.

[0040] The indicators to be evaluated may include one or more of the four items of the number of new customers, customer retention, input-output ratio, and cold start, and may also include indicators related to one or more of these four items. For example, the indicators to be evaluated may include the success rate of cold start, the speed of cold start, the survival rate of cold start (which, for example, indicates that the cold start is successful, stable, or even growing or decaying), etc., which are associated with cold start. Alternatively or additionally, the indicators to be evaluated may include the number of retained customers and the customer retention rate associated with customer retention, etc. Alternatively or additionally, the indicators associated with the number of new users may include the proportion of new users to active users. Alternatively or additionally, ROI may include the ROI difference rate.

[0041] The anomaly type may reflect the category or classification of various abnormal situations of data delivery. In some embodiments, the anomaly type may include at least one of absolute anomaly, relative anomaly, and trend anomaly.

[0042] An absolute anomaly can indicate that a certain indicator has not reached its target. For example, the target level for the number of new customers is 1000, but the actual level achieved is 800, which falls short of the target. Therefore, electronic device 110 can determine that an absolute anomaly has occurred with respect to this indicator.

[0043] Relative anomalies can reflect the anomalies of a certain data delivery method relative to other data delivery methods. Data delivery can be achieved through a variety of delivery methods, for example, through different delivery platforms, or using different delivery methods, such as through web pages, links, programs, etc. The delivery results obtained for the same delivery data through different delivery methods may be different. For example, the number of customers attracted by the same delivery data on different delivery platforms may be different. If the delivery results obtained by different delivery methods for the same indicator vary greatly, for example, the corresponding value of the indicator obtained by one method is much lower than the corresponding value of the indicator obtained by other methods, then it can be determined that a relative anomaly has occurred.

[0044] For example, when the target value of indicator A is 500, the specific value of indicator A obtained by delivery method A is 600, while the specific value of indicator A obtained by delivery method B is 100. Since the value of indicator A obtained by delivery method B is much lower than the value of indicator A obtained by delivery method A, electronic device 110 can determine that the indicator A obtained by delivery method B is relatively abnormal.

[0045] A trend anomaly may refer to a situation where the difference in change between two values ​​corresponding to the same indicator in different periods is greater than a large difference (e.g., greater than a threshold). The value of the specific threshold may be configured by the user or automatically set by the electronic device 110. For example, the value corresponding to indicator A in the first time period (e.g., the first week) is 1000, and the value corresponding to indicator A in the subsequent second time period (e.g., the second week) is 800. The difference in the values ​​of indicator A corresponding to the first week and the second week is 200, which is greater than the threshold set to 100. Thus, the electronic device 110 may determine that indicator A has a trend anomaly.

[0046] In some embodiments, the exception type may include an exception determined by the delivery platform, which may be an abnormal situation evaluated by the delivery platform based on its own policies and logic. For example, the user can configure the electronic device 110 to evaluate and alert an anomaly determined and reported by the delivery platform.

[0047] In box 220, the electronic device 110 issues an alarm for an anomaly in data delivery based on a comparison of the evaluation result for at least one indicator with a corresponding threshold value, wherein the corresponding threshold value of at least one indicator is associated with at least one anomaly type. For example, the electronic device 110 may evaluate each indicator based on the indicator to be evaluated and the anomaly type included in the user configuration information to determine whether an anomaly corresponding to the anomaly type occurs in the data delivery. For example, the user configuration information indicates evaluation indicator A and an absolute anomaly, and the electronic device 110 monitors indicator A to determine whether an absolute anomaly occurs in indicator A. In this way, the user-configured indicators and anomaly types can be evaluated, thereby performing a targeted evaluation of the data delivery, which helps to reduce the amount of calculation and improve the evaluation efficiency.

[0048] In some embodiments, the electronic device 110 may perform an evaluation on the delivery data in response to receiving user configuration information that includes at least at least one indicator to be evaluated and at least one abnormal type, thereby obtaining an evaluation result. In some embodiments, the electronic device 110 may continuously or periodically perform data delivery evaluation and obtain an evaluation result of at least one indicator in the user configuration information. For example, the electronic device 110 may perform data delivery evaluation and obtain evaluation results on a monthly basis. The electronic device 110 may also independently determine and change the evaluation cycle based on user needs or data delivery conditions. For example, for the first delivery of certain data, the electronic device 110 may determine a shorter evaluation cycle (e.g., one week) in order to obtain evaluation results in a timely manner, determine abnormal conditions, and make corresponding adjustments, which helps to ensure the delivery effect of the data delivery.

[0049] In some embodiments, the electronic device 110 may pre-acquire a threshold value corresponding to a certain indicator and associated with an abnormality type, for example, a threshold value A for an absolute abnormality of indicator A, a threshold value B for a relative abnormality of indicator A, and so on. These thresholds may be statically configured or dynamically configured (e.g., by a user). The electronic device 110 may compare the evaluation result of at least one indicator with the corresponding threshold value, and if the comparison result indicates that an abnormality has occurred, issue an alarm for the abnormality of data delivery.

[0050] Electronic device 110 can issue an alarm in a variety of ways, including but not limited to sending a pop-up window containing information related to the anomaly, voice, or text messages. For example, electronic device 110 can issue an alarm by displaying a message such as "XX indicator has XX anomaly" on the user interface. Alternatively or additionally, in some embodiments, to enhance the effectiveness of the alarm, electronic device 110 can also issue an alarm by vibrating, flashing the screen, or displaying a special color.

[0051] In some embodiments, the electronic device 110 may also provide the user with relevant information about the anomaly, such as the specific dimension of the anomaly, specific indicators, specific type of the anomaly, etc. Based on such information, the user may optimize the anomaly.

[0052] Data delivery can be performed from top to bottom according to multiple dimensions. Multiple dimensions may include at least one of the resources, delivery platforms, delivery accounts, delivery plans, delivery targets, and delivery methods used for recommendation. For example, data delivery can be performed on different platforms for different resources to be recommended, and data delivery can be performed on the same platform through different delivery accounts for the same resource to be recommended, or different delivery plans or delivery targets can be used for data delivery. A low-level dimension can be called a "sub-item" or "sub-level" of a high-level dimension. For example, a delivery platform can contain multiple delivery accounts, and delivery accounts can be called sub-items or sub-levels of the delivery platform.

[0053] Data delivery for recommended resources may include applications associated with the delivered data. For example, data delivery may be targeted at a particular application, and that application may be considered a resource for which data delivery is used for recommendation. A delivery platform may include a platform or system for data delivery, which may include various application platforms, such as content sharing applications, social applications, chat applications, shopping applications, and the like. Data delivery on the delivery platform may be performed through links, web pages, audio and video content, and the like. A delivery account may include a user account used to log in to or use the delivery platform.

[0054] A delivery plan may refer to a specific plan for data delivery, for example, it may include materials for data delivery. For example, in an advertising scenario, a delivery plan may be an advertising plan, which may include multiple advertising materials. A delivery target may indicate a user's desired goal for data delivery, for example, the expected number of new customers, or the expected customer retention rate, etc. Different desired goals may correspond to different bid types. For example, in an advertising delivery scenario, two bid types may be set, the first bid type corresponding to the desired goal of the number of new customers, and the second bid type corresponding to the desired goals of both the number of new customers and customer retention.

[0055] In some embodiments, the effectiveness of data delivery across multiple dimensions can be evaluated, and anomalies can be subsequently determined based on the evaluation results of these dimensions. For example, electronic device 110 can evaluate specific frequency metrics based on user configuration information and issue an alarm if an anomaly corresponding to the anomaly type specified in the user configuration information is detected. This process can be referred to as anomaly detection. During the anomaly detection process, an alarm can be issued if the evaluation result reaches a threshold.

[0056] The evaluation result of a certain indicator can be determined based on the following formula:

[0057] Taking the ROI indicator as an example, the evaluation result of this indicator can be determined based on the following formula:

[0058] The analysis group indicates the results within the analysis scope (e.g., one or more dimensions), while the benchmark group indicates the comparison benchmark, which can:

[0059] 1) Within the same analysis scope as the analysis group, but in a different time period: commonly used in month-on-month fluctuation analysis;

[0060] 2) In a different analysis range than the analysis group, but in the same time period: commonly used in two-group comparative analysis;

[0061] 3) Absolute value: commonly used in absolute goal achievement analysis.

[0062] If ΔROI% is lower than the threshold, it can be determined that the ROI indicator is abnormal.

[0063] The absolute value of the indicator can be determined by the following formula:

[0064] Wherein, j=1,…,n represents the jth evaluation cycle, and n is a positive integer.

[0065] In some embodiments, the electronic device 110 can determine the corresponding contribution of the data in each dimension of the multiple dimensions to the anomaly (also called "contribution rate"). Based on the comparison of the corresponding contribution of each dimension with the corresponding threshold contribution, the key dimension can be determined from the multiple dimensions. This comparison may include a comparison of the contribution (or contribution rate) with the threshold. For example, a dimension with a higher contribution may be determined as a key dimension. The comparison may also include a comparison of the contribution change with the threshold. For example, if the contribution of a dimension changes from 60% to 40%, a decrease of 20%, which exceeds the decrease threshold ratio of 10%, then the dimension can be determined to be a key dimension.

[0066] The electronic device 110 can determine the corresponding contribution of data in multiple dimensions for each indicator in the user configuration information. The contribution includes, but is not limited to, the specific value corresponding to the indicator, the specific value corresponding to the change in the indicator, the proportion of the change in the specific value corresponding to the indicator (the greater the proportion of the change, the greater the contribution), the proportion of the specific value corresponding to the indicator, and the like.

[0067] In some embodiments, the electronic device 110 can determine the contribution of a dimension of an indicator based on a first evaluation value obtained in one dimension of multiple dimensions for one indicator of at least one indicator and a second evaluation value obtained in multiple dimensions. For example, for a certain indicator, the ratio of the first evaluation value in a certain dimension to the second evaluation value in multiple dimensions can be determined as the contribution of the dimension. As an example, data can be delivered on four dimensions A, B, C and D, and the indicator of the number of new customers can be evaluated. If the number of new customers corresponding to dimension A is 1500, the number of new customers corresponding to dimension B is 3500, and the number of new customers corresponding to dimension C and dimension D are both 2500, then the corresponding contribution of the four dimensions A to D can be 15%, 35%, 25%, and 25%.

[0068] As another example, for the customer retention indicator, if the first evaluation value obtained in the dimension of a certain delivery account corresponding to a certain delivery platform is 2000, and the second evaluation value obtained in multiple dimensions corresponding to multiple accounts on the delivery platform is 10000, then for the customer retention indicator, the contribution of the delivery account is 20%.

[0069] In some embodiments, an anomaly in a dimension can be associated with a change in the overall evaluation value of all dimensions (also referred to as the "big picture"). For example, the contribution of a dimension can be determined as the ratio of the change in a first evaluation value in a dimension to the change in second evaluation values ​​in multiple dimensions.

[0070] As an example, the contribution of item i can be determined as:

[0071] The comparison period can refer to the current evaluation period, and the base period can refer to the previous evaluation period. Xs with a subscript i represent the sub-item's evaluation value, while Xs without a subscript i represent the overall market's evaluation value. The greater the percentage change in sub-item i, the higher its contribution, and vice versa. By comparing data from a particular period with the previous period, we can assess increases and decreases and calculate the percentage change in trend. This allows for anomaly analysis through month-over-month fluctuation analysis.

[0072] For example, in the previous cycle, the first evaluation value obtained for a customer retention metric in a certain delivery account on a certain delivery platform was 2000, and the second evaluation value obtained on the same delivery platform was 10000. In the current cycle, the first evaluation value obtained for a customer retention metric in a certain delivery account on the same delivery platform was 1000, and the second evaluation value obtained on the same delivery platform was 8000. Between the two cycles, the change in the first evaluation value was -1000, and the change in the second evaluation value was -2000. In this case, the contribution of the change in the first evaluation value to the change in the second evaluation value can be 50%.

[0073] In some embodiments, the anomaly may be associated with a change in the ratio of the first evaluation value to the second evaluation value. In this case, the electronic device 110 may determine the contribution of the change in the first evaluation value to the change in the ratio, and the contribution of the change in the second evaluation value to the change in the ratio, respectively.

[0074] As an example, the electronic device 110 may determine the proportion index of the sub-item (eg, DNU proportion) by the following formula:

[0075] Where X represents the evaluation value.

[0076] The change in the ratio of the first evaluation value to the second evaluation value, for example, the change in the ratio of the sub-item to the overall market, can be expressed as:

[0077] The contribution of the change of the first evaluation value, for example, the contribution of the change of the sub-item can be determined as:

[0078] The contribution of the change in the second evaluation value, for example, the contribution of the change in the market can be determined as:

[0079] In some embodiments, the anomaly may be associated with changes in the second evaluation values ​​across multiple dimensions. In this case, the contribution of a dimension may be determined by comprehensively considering both the change in the first evaluation value of a dimension and the change in the proportion of the first evaluation value relative to the second evaluation value.

[0080] For example, the electronic device 110 may determine a ratio indicator (such as retention or ROI) using the following formula:

[0081] Among them, p represents the weight.

[0082] The contribution of item i can be determined as:

[0083] Among them, the contribution rate iX差异 Indicates the contribution of the change of sub-item i, contribution rate i新增占比差异 Indicates the contribution of the new proportion of sub-item i.

[0084] In some embodiments, the electronic device 110 can, based on the selected control group, decompose the difference rate between the analysis group and the benchmark group into sub-level contributions in the form of a decision tree. For example, the difference rate of a delivery platform can be decomposed into the contribution of each delivery account. Based on the threshold contribution determined in advance, the decision tree can be pruned, for example, by removing dimensions that have been determined to have no abnormalities after comparison. The pruned path can be regarded as the core problem path (i.e., the key dimension). This process performed by the electronic device 110 can also be called dimensional drill-down. After performing dimensional drill-down, the electronic device 110 can only evaluate the indicators of the key dimensions, which helps to reduce the amount of calculation of the electronic device 110 and improve the evaluation efficiency of the electronic device 110.

[0085] In some embodiments, the sum of the contributions of multiple dimensions corresponding to the same metric is 100%. The sum of the contributions of a sub-level is consistent with the contribution of the level above it. For example, for the metric of number of new customers, the contribution of a delivery platform can be equal to the sum of the contributions of each delivery account in the delivery platform.

[0086] The contribution of a sub-level can be decomposed into the sum of the change rates of a single indicator: Contribution rate = Control impact indicator A * Change rate of impact indicator B + Control impact indicator B * Change rate of impact indicator A (11)

[0087] The contribution of each sub-level can reflect its impact on the overall indicator. The decomposed contribution can guide subsequent optimization.

[0088] The calculation method of the contribution of a dimension is as follows: Contribution rate i = control proportion * sub-dimension indicator difference amplitude + control sub-dimension indicator level * sub-dimension proportion difference amplitude = analysis group i proportion * analysis group i excess difference rate to the benchmark group + benchmark group i difference rate to the benchmark group * analysis group i excess contribution (12)

[0089] Among them, factors with i are associated with the sub-level, and factors without i are associated with the level above the sub-level.

[0090] In some embodiments, the user configuration information may include multiple indicators. The electronic device 110 may determine the corresponding contribution of each of the multiple indicators to the anomaly and determine the key indicator from the multiple indicators based on a comparison of the corresponding contribution of each indicator with the corresponding threshold contribution. This process performed by the electronic device 110 may be referred to as indicator drill-down. Indicator drill-down may be performed at each dimension (or sub-item or sub-level).

[0091] As an example, the electronic device 110 may use an indicator with a corresponding contribution greater than a corresponding threshold contribution as a key indicator. The electronic device 110 may then evaluate only the key indicators, further reducing the amount of computation and power consumption of the electronic device 110 and improving evaluation efficiency.

[0092] In some embodiments, after determining the key indicator, the indicator can be further decomposed into multiple lower-level indicators for abnormal analysis. This process is also called abnormal drill-down. For example, for a ratio indicator (e.g., ROI, retention, etc.), it can be further decomposed into the contribution of the numerator and denominator. As an example, the electronic device 110 can determine the contribution of the ratio indicator (e.g., ROI, retention, etc.) of sub-item i using the following formula:

[0093] Taking the ROI metric as an example, after determining that an anomaly exists in the ROI metric, the electronic device 110 may break the ROI down into two lower-level metrics: revenue and output. Furthermore, based on comparing the contribution of revenue and / or output to the anomaly with corresponding threshold contribution levels, the electronic device 110 may determine whether an anomaly exists in the revenue and / or output. Because revenue is associated with customer retention and revenue, in some embodiments, after determining that an anomaly exists in the revenue, the electronic device 110 may further break the revenue down into two lower-level metrics: customer retention and revenue. For example, in response to determining that an anomaly exists in the revenue, the electronic device 110 may determine whether an anomaly exists in customer retention based on a comparison of the customer retention variance rate with a first threshold variance rate, and determine whether an anomaly exists in revenue based on a comparison of the revenue variance rate with a second threshold variance rate. Because output is associated with cost, in some embodiments, the electronic device 110 may further break the output down into at least one cost item. For example, in response to determining that an anomaly exists in the output, the electronic device 110 may determine that an anomaly exists in the at least one cost item based on a comparison of the variance rate of the at least one cost item with a third threshold variance rate.

[0094] FIG3 shows a schematic diagram of a process 300 of data delivery evaluation according to some embodiments of the present disclosure.

[0095] In process 300, at 305, after electronic device 110 determines that the input-output ratio is abnormal, it further analyzes revenue contribution and cost contribution. As shown in Figure 3, at 310, it is determined whether revenue contribution > X% holds true. If not, the diagnosis of this branch ends at 312. If true, electronic device 110 further determines to analyze customer retention at 315.

[0096] Because customer retention is significantly affected by time, electronic device 110 can first analyze the short-term retention variance rate (i.e., the short-term customer retention variance rate) and then analyze the long-term retention variance rate (i.e., the long-term customer retention variance rate). As shown in FIG3 , at 320 , a determination is made as to whether the short-term retention variance rate < -X% holds true. If so, a diagnosis of short-term retention is performed at 325 . If not, a determination is made as to whether the long-term retention variance rate < -X% holds true at 330 . If so, a long-term retention diagnosis is performed at 335 . If not, the diagnosis of the customer retention branch is terminated at 337 .

[0097] If the revenue contribution > X% is determined at 310, a profit analysis may be performed at 340. At 345, a determination is made as to whether the profit variance < -X% is true. If so, a profit diagnosis is performed at 350. If not, the profit branch diagnosis is terminated at 352.

[0098] Regarding the analysis of cost contribution, as shown in FIG3 , at 355 , it is determined whether the cost contribution > X% is true. If not, the diagnosis of the branch is terminated at 357 . If true, the electronic device 110 further analyzes the cost item. At 360 , it is determined whether the difference rate of cost item M1 > X% is true. If true, the diagnosis of cost item M1 is performed at 365 . If not, the diagnosis of the branch regarding cost item M1 is terminated at 367 . At 370 , it is determined whether the difference rate of cost item M2 > X% is true. If true, the diagnosis of cost item M2 is performed at 375 . If not, the diagnosis of the branch regarding cost item M2 is terminated at 377 .

[0099] It should be understood that each threshold value in FIG3 is expressed as x% for illustrative purposes only and is not intended to be limiting. In some embodiments, the threshold values ​​for each factor may be different. In this way, electronic device 110 can gradually refine the indicators in a top-down manner, clarifying the specific factors that lead to the anomaly, thereby helping the user to subsequently resolve the anomaly.

[0100] As described above, the electronic device 110 can determine the key factors causing the abnormality by breaking down the dimensions and indicators, which helps to provide the user with more detailed alarm information and subsequently make abnormality diagnosis based on the alarm information.

[0101] FIG4 shows a schematic diagram of an architecture 400 for evaluating data delivery according to some embodiments of the present disclosure.

[0102] As shown in FIG. 4 , the electronic device 110 may perform the data delivery evaluation task at least through a data module 410 , an early warning module 420 , and an anomaly location module 430 .

[0103] The data module 410 can be configured to perform the indicator decomposition 405 and can use the core data to decompose the indicator to be evaluated into the number of new customers, customer retention, input-output ratio, cold start, etc. The data module 410 can also use the information visualization module to present a user configuration interface to obtain user configuration information for indicators and anomaly types.

[0104] The early warning module 420 can be configured to perform anomaly discovery 415, and the types of anomalies discovered include, for example, absolute anomalies, relative anomalies, trend anomalies, and anomalies determined by the delivery platform, etc. The early warning module 420 may include an anomaly definition module, which can define different anomaly types. The early warning module 420 may also include an anomaly capture module, which can perform anomaly capture based on the user's evaluation configuration in the user configuration information, such as evaluating data anomalies and determining the type of anomaly. The early warning module 420 may also include an information visualization module for presenting the captured anomaly indicators to the user (for example, through a relevant interface). In some embodiments, the early warning module 420 may also include a statistical analysis module, which can be configured to perform statistical analysis on the anomalies that occur.

[0105] The anomaly location module 430 can be configured to perform dimension drill-down 425. As an example, the dimensions can be, from high to low, application (as an example of a resource to be recommended), bid type (e.g., corresponding to a delivery target), channel (e.g., delivery platform), account (e.g., delivery account), channel package (e.g., data corresponding to different delivery platforms), and plan (e.g., delivery plan). The anomaly location module 430 can calculate the contribution of each dimension through the drill-down calculation module based on the dimension drill-down logic, and determine the key dimension where the anomaly occurs through the drill-down conclusion judgment module.

[0106] The anomaly location module 430 can also be configured to perform indicator drilldown 435 based on the determined key dimensions. Taking ROI (return on investment) as an example, the anomaly location module 430 can break it down into two lower-level indicators: revenue and cost. Furthermore, revenue can be broken down into two lower-level indicators: retention and user revenue. User revenue can be broken down into specific revenue items as lower-level indicators. The anomaly location module 430 can also break down costs into specific cost items as lower-level indicators.

[0107] Based on the indicator drill-down logic, the anomaly location module 430 can calculate the contribution of each indicator through the drill-down calculation module, and determine the key indicator causing the anomaly through the drill-down conclusion judgment module. The anomaly location module 430 can then use the information visualization module to present a user interface containing the anomaly indicator, the contribution rate of the lower-level indicator, and the lower-level contribution rate to the user, so as to present the specific factors that caused the anomaly to the user. In some embodiments, the anomaly location module 430 may also include a statistical analysis module, which can be configured to perform statistical analysis on the results of dimension drill-down and indicator drill-down.

[0108] In this way, the electronic device 110 can evaluate the delivery data simply and efficiently, and can intuitively display the specific factors that lead to the abnormality to the user through an interface, etc., which helps to improve the evaluation effect and efficiency.

[0109] In some embodiments, the architecture 400 may further include a problem diagnosis module 440 , a strategy adjustment module 450 , and an effect feedback module 460 , which respectively perform delivery factor diagnosis 445 , automatic tuning 455 , and effect feedback 465 .

[0110] In this manner, electronic device 110 can perform data delivery evaluation tasks through data module 410, warning module 420, and anomaly location module 430 in architecture 400, and perform anomaly handling tasks through problem diagnosis module 440, strategy invocation module 450, and effect feedback module 460. Electronic device 110 can simply and efficiently diagnose and adjust the effectiveness of advertising delivery based on the evaluation results, helping to improve the efficiency of handling anomalies and further enhance data delivery effectiveness.

[0111] 5 shows a schematic structural block diagram of an apparatus 500 for evaluating data delivery according to certain embodiments of the present disclosure. Apparatus 500 may be implemented as or included in electronic device 110. Each module / component in apparatus 500 may be implemented by hardware, software, firmware, or any combination thereof.

[0112] As shown in Figure 5, the device 500 includes a receiving module 505, which is configured to receive a user's first selection of at least one indicator for evaluating data delivery; and receive a user's second selection of at least one anomaly type for evaluating data delivery. The device 500 includes an information acquisition module 510, which is configured to obtain user configuration information for evaluating data delivery based on the user's first selection and second selection, and the user configuration information includes at least at least one indicator to be evaluated and at least one anomaly type. The device 500 also includes an anomaly alarm module 520, which is configured to alarm for anomalies in data delivery based on a comparison of the evaluation result for at least one indicator with a corresponding threshold value, wherein the corresponding threshold value of at least one indicator is associated with at least one anomaly type.

[0113] In some embodiments, the evaluation result of the at least one indicator is obtained based on a periodic evaluation of the data delivery.

[0114] In some embodiments, the device 500 also includes: a dimension contribution determination module, configured to determine the corresponding contribution of the data delivery in each dimension of multiple dimensions to the anomaly; and a key dimension determination module, configured to determine the key dimension from multiple dimensions based on a comparison of the corresponding contribution of each dimension with the corresponding threshold contribution.

[0115] In some embodiments, the multiple dimensions include at least one of the following: a resource used for recommendation, a delivery platform, a delivery account, a delivery plan, and a delivery target.

[0116] In some embodiments, a respective contribution of each dimension is determined for each of the at least one indicator.

[0117] In some embodiments, the dimension contribution determination module is further configured to determine the contribution of a dimension for an indicator based on a first evaluation value obtained for one indicator in one of multiple dimensions and a second evaluation value obtained in multiple dimensions for one indicator of at least one indicator.

[0118] In some embodiments, the anomaly is associated with a change in the ratio of the first evaluation value to the second evaluation value, and the dimension contribution determination module is further configured to: determine the contribution of the change in the first evaluation value to the change in the comparison value; and determine the contribution of the change in the second evaluation value to the change in the comparison value.

[0119] In some embodiments, the anomaly is associated with a change in the second evaluation value, wherein the dimension contribution determination module is further configured to: determine the contribution of the change in the first evaluation value to the change in the second evaluation value; and determine the contribution of the change in the ratio of the first evaluation value to the second evaluation value to the change in the second evaluation value.

[0120] In some embodiments, at least one indicator includes multiple indicators, and the device 500 also includes: an indicator contribution determination module, configured to determine the corresponding contribution of each indicator in the multiple indicators to the anomaly; and a key indicator determination module, configured to determine the key indicator from the multiple indicators based on a comparison of the corresponding contribution of each indicator with the corresponding threshold contribution.

[0121] In some embodiments, the multiple indicators include an input-output ratio, and the abnormality alarm module 520 is further configured to determine that an abnormality occurs in income and / or output based on a comparison of the contribution of income and / or output to the abnormality with a corresponding threshold contribution.

[0122] In some embodiments, the abnormality alarm module 520 is further configured to: in response to determining that an abnormality has occurred in revenue, determine that an abnormality has occurred in customer retention based on a comparison of the customer retention difference rate with a first threshold difference rate; and / or determine that an abnormality has occurred in revenue based on a comparison of the revenue difference rate with a second threshold difference rate.

[0123] In some embodiments, the abnormality alarm module 520 is further configured to: in response to determining that the output is abnormal, determine that the at least one cost item is abnormal based on a comparison of a difference rate of the at least one cost item with a third threshold difference rate.

[0124] In some embodiments, the at least one indicator is associated with at least one of the following: number of new customers, customer retention, input-output ratio, and cold start.

[0125] In some embodiments, the at least one anomaly type includes at least one of the following: absolute anomaly, relative anomaly, and trend anomaly.

[0126] It should be understood that the features and operations of the electronic device 110 and the corresponding effects involved in the method 200 and the process 300 discussed above with reference to FIG. 1 to FIG. 4 are also applicable to the apparatus 500 and will not be described in detail here.

[0127] The units included in the device 500 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units can be implemented using software and / or firmware, such as machine executable instructions stored on a storage medium. In addition to or as an alternative to machine executable instructions, some or all of the units in the device 500 can be implemented at least in part by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0128] FIG6 shows a block diagram of an electronic device 600 in which one or more embodiments of the present disclosure may be implemented. It should be understood that the electronic device 600 shown in FIG6 is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. The electronic device 600 shown in FIG6 can be used to implement the electronic device 110 of FIG1 .

[0129] As shown in FIG6 , electronic device 600 is a general-purpose electronic device. Components of electronic device 600 may include, but are not limited to, one or more processors or processing units 610, memory 620, storage device 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660. Processing unit 610 may be a real or virtual processor and is capable of performing various processes according to programs stored in memory 620. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to enhance the parallel processing capabilities of electronic device 600.

[0130] The electronic device 600 typically includes a plurality of computer storage media. Such media can be any accessible media that can be obtained by the electronic device 600, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 620 can be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 630 can be a removable or non-removable medium and can include a machine-readable medium, such as a flash drive, a disk, or any other medium that can be used to store information and / or data (e.g., training data for training) and can be accessed within the electronic device 600.

[0131] The electronic device 600 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG6 , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a “floppy disk”) and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to a bus (not shown) by one or more data media interfaces. The memory 620 may include a computer program product 625 having one or more program modules configured to perform various methods or actions of various embodiments of the present disclosure.

[0132] The communication unit 640 enables communication with other electronic devices via a communication medium. Additionally, the functions of the components of the electronic device 600 can be implemented in a single computing cluster or multiple computing machines that can communicate via a communication connection. Thus, the electronic device 600 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0133] The input device 650 may be one or more input devices, such as a mouse, keyboard, or trackball. The output device 660 may be one or more output devices, such as a display, a speaker, or a printer. The electronic device 600 may also communicate with one or more external devices (not shown) through the communication unit 640 as needed, such as a storage device, a display device, or the like, with one or more devices that allow a user to interact with the electronic device 600, or with any device that allows the electronic device 600 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0134] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0135] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0136] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, such that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0137] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0138] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple implementations of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part for a module, program segment or instruction, and a part for a module, program segment or instruction comprises one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0139] While various implementations of the present disclosure have been described above, the foregoing description is intended to be illustrative, not exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is selected to best explain the principles of the implementations, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for evaluating data delivery, comprising: receiving a user's first selection of at least one metric for evaluating the data delivery and a second selection of at least one anomaly type for evaluating the data delivery; Based on the first selection and the second selection of the user, obtaining user configuration information for evaluating the data delivery, the user configuration information at least including at least one indicator to be evaluated and at least one abnormality type; as well as Based on the comparison of the evaluation result of the at least one indicator with the corresponding threshold, an alarm is issued for the abnormality of the data delivery, wherein the corresponding threshold of the at least one indicator is associated with the at least one abnormality type. 2 . The method according to claim 1 , wherein the evaluation result of the at least one indicator is obtained based on a periodic evaluation of the data delivery.

3. The method according to claim 1, further comprising: Determining the respective contribution of the delivery of data in each of the multiple dimensions to the anomaly; as well as Based on the comparison between the corresponding contribution of each dimension and the corresponding threshold contribution, a key dimension is determined from the multiple dimensions.

4. The method according to claim 3, wherein the multiple dimensions include at least one of the following: the delivery of the data is used for recommended resources, delivery platform, delivery account, delivery plan, and delivery target. 5 . The method according to claim 3 , wherein the corresponding contribution of the respective dimensions is determined for each of the at least one indicator.

6. The method according to claim 5, wherein determining the corresponding contribution of the placement of the data in each of the multiple dimensions to the anomaly comprises: The contribution degree of the one dimension for one indicator among the at least one indicator is determined based on a first evaluation value obtained in one dimension of the multiple dimensions and a second evaluation value obtained in the multiple dimensions.

7. The method according to claim 6, wherein the anomaly is associated with a change in a ratio of the first evaluation value to the second evaluation value, and wherein determining the contribution of the one dimension to the one indicator comprises: Determining a contribution of a change in the first evaluation value to a change in the ratio; as well as The contribution of the change in the second evaluation value to the change in the ratio is determined.

8. The method according to claim 6, wherein the anomaly is associated with a change in the second evaluation value, wherein determining the contribution of the one dimension to the one indicator comprises: determining a contribution of a change in the first evaluation value to a change in the second evaluation value; as well as A contribution of a change in the ratio of the first evaluation value to the second evaluation value to a change in the second evaluation value is determined.

9. The method of claim 1, wherein the at least one indicator comprises a plurality of indicators, and the method further comprises: Determining a corresponding contribution of each of the plurality of indicators to the anomaly; as well as Based on the comparison between the corresponding contribution of each indicator and the corresponding threshold contribution, a key indicator is determined from the multiple indicators.

10. The method according to claim 9, wherein the plurality of indicators include input-output ratio, and wherein alarming the anomaly comprises: Based on the comparison of the contribution of the determined income and / or output to the anomaly and the corresponding threshold contribution, it is determined that the income and / or output is abnormal.

11. The method according to claim 10, wherein the alarming of the abnormality further comprises: In response to determining that the income is abnormal, Determining that customer retention is abnormal based on a comparison between the customer retention difference rate and a first threshold difference rate; and / or Based on a comparison between the earnings difference rate and a second threshold difference rate, it is determined that earnings are abnormal.

12. The method according to claim 10, wherein the alarming of the abnormality further comprises: In response to determining that the output is abnormal, the at least one cost item is determined to be abnormal based on a comparison of a difference rate of the at least one cost item with a third threshold difference rate.

13. The method according to claim 1, wherein the at least one indicator is associated with at least one of the following: number of new customers, customer retention, input-output ratio, and cold start. The method according to claim 1 , wherein the at least one anomaly type comprises at least one of the following: absolute anomaly, relative anomaly, trend anomaly.

15. An apparatus for evaluating data delivery, comprising: A receiving module configured to receive a user's first selection of at least one indicator for evaluating the data delivery and a second selection of at least one anomaly type for evaluating the data delivery; An information acquisition module, configured to acquire user configuration information for evaluating the data delivery based on the first selection and the second selection of the user, wherein the user configuration information at least includes at least one indicator to be evaluated and at least one abnormality type; as well as The abnormality alarm module is configured to alarm the abnormality of the data delivery based on the comparison of the evaluation result of the at least one indicator with the corresponding threshold, wherein the corresponding threshold of the at least one indicator is associated with the at least one abnormality type.

16. An electronic device, comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 14 when executed by the at least one processing unit.

17. A computer-readable storage medium having a computer program stored thereon, wherein the computer program can be executed by a processor to implement the method according to any one of claims 1 to 14.

Citation Information

Patent Citations

  • Method and device used for evaluating injection effect of injected information automatically

    CN106296292A

  • Advertising quality detection method and device

    CN106469383A

  • Index transaction analysis method and equipment, computer storage medium and computer equipment

    CN109697203A

  • Anomaly detection method and device, server and storage medium

    CN112953904A

  • Advertisement putting effect evaluation method based on multiple dimensions and storage medium

    CN114266604A

Cited By

  • Data processing method and device, equipment, storage medium and program product

    CN119226747A

  • Method and apparatus for information distribution, electronic device, and computer program product

    WO2026123156A1