Detection method and device for sample proportion mismatching problem, medium and related equipment

Abstract

According to the sample proportion mismatching problem detection method and device, the medium and the related equipment provided by the invention, when SRM detection is carried out on an A / B experiment, a user only needs to input A / B experiment information, such as an experiment ID, an experiment background and an experiment target, required by SRM detection, the user does not need to master professional data analysis knowledge, and does not need to carry out data processing personally, so that the SRM detection efficiency is greatly improved. Therefore, the target user features influencing the A / B experiment can be intelligently matched through the method, and whether the A / B experiment has the SRM problem or not is detected after the feature data is obtained according to the target user features, so that the requirement for professional skills of the user is greatly reduced, and the analysis efficiency and accuracy are effectively improved.

Description

Technical Field

[0001] The present application relates to the technical field of A / B testing, and in particular to a detection method, device, medium and related equipment for sample ratio mismatch problem. Background Art

[0002] In the field of A / B testing, i.e., controlled experiments, a common challenge is how to ensure the consistency of user feature distribution between the two versions or groups of the experiment. Currently, in traditional A / B testing practices, when the observed data between the experimental group and the control group show significant differences, researchers often worry that this difference may not be caused by the experimental changes themselves, but by the influence of sample ratio mismatch (SRM).

[0003] In many cases, the SRM problem is caused by inconsistent user feature allocation. Traditional processing methods usually rely on professional data analysts to conduct in-depth analysis of different specific scenarios and use a series of statistical tests to determine the existence and impact of SRM problems.

[0004] However, as the data scale and experimental complexity increase, this method that relies on manual analysis increasingly shows its limitations, including low analysis efficiency, inability to respond in real time, and over-reliance on analysts' subjective judgment. Therefore, there is an urgent need for a detection method that can automatically and quickly analyze and judge SRM problems. Summary of the invention

[0005] The purpose of this application is to solve at least one of the above-mentioned technical defects, especially the technical defects in the prior art that determine the existence and impact of SRM problems rely on manual analysis, resulting in low analysis efficiency, long time consumption, and inability to respond in real time.

[0006] The present application provides a method for detecting a sample ratio mismatch problem, the method comprising:

[0007] When performing SRM testing on an A / B experiment, obtaining A / B experiment information required for the SRM testing input by the user;

[0008] Evaluate the target user characteristics that cause the SRM problem in the A / B experiment based on the A / B experiment information, and obtain a user ID dataset that participated in the A / B experiment within a preset historical period;

[0009] Extracting feature data of each user ID in the user ID data set according to the target user feature, and performing experimental grouping and feature aggregation on each feature data to obtain first aggregated data formed in the experimental group and second aggregated data formed in the control group according to the target user feature;

[0010] The correlation between the first aggregated data and the second aggregated data is analyzed, and it is determined whether an SRM problem occurs in the A / B experiment according to the analysis result.

[0011] Optionally, the obtaining of A / B experiment information required for SRM testing input by the user includes:

[0012] Get the content information entered by the user;

[0013] Performing intent recognition on the content information, and determining whether the content information involves SRM inspection according to the recognition result;

[0014] If so, determining whether the content information contains key information for SRM inspection;

[0015] If included, the content information is used as the A / B experiment information required for SRM testing;

[0016] If not, the user is prompted to supplement until the supplemented information contains the key information for SRM inspection;

[0017] If not, the user is guided to enter key information for SRM inspection.

[0018] Optionally, the A / B experiment information includes at least an experiment background and an experiment goal;

[0019] The evaluating, according to the A / B experiment information, the target user characteristics that cause the SRM problem in the A / B experiment includes:

[0020] Determine the preset number of user features;

[0021] Evaluate multiple user characteristics that cause the A / B experiment to have SRM problems based on the experiment context and the experiment goal;

[0022] A target user feature corresponding to the number of user features is selected from the plurality of user features.

[0023] Optionally, the acquiring a data set of user IDs participating in the A / B experiment within a preset historical period includes:

[0024] Obtain multiple user IDs that participated in the A / B experiment within a preset historical period;

[0025] Determine whether the number of users of the plurality of user IDs exceeds a preset number threshold;

[0026] If it exceeds, data sampling is performed on the multiple user IDs, and a user ID data set is constructed based on the sampled user IDs;

[0027] If not, directly construct a user ID data set based on the multiple user IDs.

[0028] Optionally, after performing experimental grouping and feature aggregation on each feature data, obtaining first aggregated data formed in the experimental group and second aggregated data formed in the control group according to the target user feature includes:

[0029] According to the user ID corresponding to each feature data, each feature data is experimentally grouped to obtain feature data in the experimental group and feature data in the control group;

[0030] Feature aggregation is performed on the feature data in the experimental group and the feature data in the control group according to the target user feature to obtain first aggregated data formed in the experimental group and second aggregated data formed in the control group according to the target user feature.

[0031] Optionally, the analyzing the correlation between the first aggregated data and the second aggregated data includes:

[0032] determining the data volume of the first aggregated data and the second aggregated data;

[0033] Determining, according to the data volume, a statistical method used when performing correlation analysis on the first aggregated data and the second aggregated data;

[0034] The correlation between the first aggregated data and the second aggregated data is analyzed based on the statistical method.

[0035] Optionally, the determining, according to the data volume, a statistical method used for performing correlation analysis on the first aggregated data and the second aggregated data includes:

[0036] Determining whether the data volume is greater than a preset data volume threshold;

[0037] If it is greater than, determining to use a chi-square test to perform a correlation analysis on the first aggregated data and the second aggregated data;

[0038] If not, it is determined to use Fisher's exact test to perform a correlation analysis on the first aggregated data and the second aggregated data.

[0039] The present application also provides a detection device for sample ratio mismatch problem, comprising:

[0040] The experiment information acquisition module is used to obtain the A / B experiment information required for the SRM test input by the user when the A / B experiment is subjected to the SRM test;

[0041] A user data acquisition module is used to evaluate the target user characteristics that cause the SRM problem in the A / B experiment according to the A / B experiment information, and obtain a user ID data set that participated in the A / B experiment in a preset historical period;

[0042] A feature aggregation module, used to extract feature data of each user ID in the user ID data set according to the target user feature, and perform experimental grouping and feature aggregation on each feature data to obtain first aggregated data formed in the experimental group and second aggregated data formed in the control group according to the target user feature;

[0043] The correlation analysis module is used to analyze the correlation between the first aggregated data and the second aggregated data, and determine whether the A / B experiment has an SRM problem according to the analysis result.

[0044] The present application also provides a computer-readable storage medium, in which computer-readable instructions are stored. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the method for detecting the sample ratio mismatch problem as described in any of the above embodiments.

[0045] The present application also provides a computer device, comprising: one or more processors, and a memory;

[0046] The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the method for detecting the sample ratio mismatch problem as described in any one of the above embodiments are performed.

[0047] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0048] The detection method, device, medium and related equipment for the sample ratio mismatch problem provided in the present application can obtain the A / B experiment information required for the SRM test input by the user when performing the SRM test on the A / B experiment, and then evaluate the target user characteristics that cause the SRM problem in the A / B experiment based on the A / B experiment information, and obtain the user ID data set that participated in the A / B experiment in a preset historical period. In this way, the feature data of each user ID in the user ID data set can be extracted according to the target user characteristics, and after experimental grouping and feature aggregation of each feature data, first aggregated data formed in the experimental group according to the target user characteristics and second aggregated data formed in the control group are obtained. Finally, the correlation between the first aggregated data and the second aggregated data is analyzed, and it is determined whether the A / B experiment has an SRM problem based on the analysis result. During this process, users only need to input the A / B experiment information required for SRM testing, such as experiment ID, experiment background, experiment objectives, etc. They do not need to master professional data analysis knowledge, nor do they need to process the data themselves. They can use this application to intelligently match the target user characteristics that affect the A / B experiment, and obtain feature data based on the target user characteristics to test whether there is an SRM problem in the A / B experiment. This not only greatly reduces the requirements for users' professional skills, but also effectively improves analysis efficiency and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0050] Figure 1 A flow chart of a method for detecting a sample ratio mismatch problem provided in an embodiment of the present application;

[0051] Figure 2 A schematic diagram of the interaction process of each module of the system provided in the embodiment of the present application;

[0052] Figure 3 A schematic diagram of the process of obtaining A / B experimental information required for SRM testing provided in an embodiment of the present application;

[0053] Figure 4 A schematic diagram of a process for obtaining a user ID data set participating in an A / B experiment within a preset historical period provided in an embodiment of the present application;

[0054] Figure 5 A schematic diagram of the structure of a detection device for sample ratio mismatch problem provided in an embodiment of the present application;

[0055] Figure 6 A schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0057] In one embodiment, Figure 1 As shown, Figure 1 A flow chart of a method for detecting a sample ratio mismatch problem provided in an embodiment of the present application; the present application provides a method for detecting a sample ratio mismatch problem, the method may include:

[0058] S110: When performing SRM test on the A / B experiment, obtain A / B experiment information required for the SRM test input by the user.

[0059] In this step, in the current A / B test practice, when the observed data between the experimental group and the control group show significant differences, researchers often worry that this difference may not be caused by the experimental change itself, but by the influence of sample ratio mismatch (Sample Ratio Mismatch, referred to as SRM). Therefore, for this difference problem, the detection method of this application can be used to detect whether there is a sample ratio mismatch problem during the A / B experiment, and then timely discover and solve the sample ratio mismatch problem, and select appropriate analysis methods and increase the sample size to improve the accuracy and reliability of the experimental results.

[0060] Specifically, when the present application performs SRM test on the A / B experiment, the A / B experiment information required for the SRM test input by the user can be obtained first, and the A / B experiment information includes but is not limited to the experiment ID, the experiment background and the experiment goal. Among them, the experiment ID here refers to the experiment identification corresponding to different A / B experiments; the experiment background here refers to the comprehensive description of the specific environment, conditions and related circumstances of this A / B experiment, and the experiment background includes the scene of the experiment, the target population of the experiment, the strategy of the experiment, etc. For example, for an algorithm model experiment in a black room, the target population can be all users who enter the black room, and the experiment strategy can be different algorithm modeling methods (features of user's real-time interests and hobbies / separate modeling / multi-objective modeling); the experiment goal here refers to the expected benefits of the experiment. For example, when different algorithm modeling methods are given in the experiment strategy, the experiment goal can be to compare whether different modeling methods can improve the user's exposure retention, the user activity retained the next day after entering the room, and the room black experience. The specific experiment background and experiment goal can be set according to the actual situation, and there is no restriction here.

[0061] Furthermore, when the system receives the A / B experimental information input by the user, it can be handed over to the relevant modules for analysis and identification to confirm that the information obtained is the A / B experimental information required for SRM inspection, which can provide data guarantee for subsequent testing work and thus improve the accuracy of the test results.

[0062] In a specific implementation, Figure 2 As shown, Figure 2 A schematic diagram of the interaction process of each module of the system provided in the embodiment of the present application; Figure 2 In this application, after obtaining the A / B experiment information input by the user, the Plan Agent module of the system can perform intent recognition analysis. If the information provided is insufficient for SRM inspection, the Plan Agent module will provide real-time feedback to the user and ask him to supplement the missing relevant information; once it is confirmed that the user has entered sufficient information, the PlanAgent module can perform task planning. This module is not only responsible for analyzing the completeness and accuracy of the input information, but also for detailed planning of task execution, thereby effectively improving detection efficiency and detection accuracy.

[0063] S120: Evaluate target user characteristics that cause SRM problems in the A / B experiment based on the A / B experiment information, and obtain a user ID dataset that participated in the A / B experiment within a preset historical period.

[0064] In this step, after obtaining the A / B experiment information required for SRM verification input by the user through S110, the present application can evaluate the target user characteristics that may cause SRM problems in the A / B experiment based on the A / B experiment information, and obtain the user ID data set that participated in the A / B experiment within a preset historical period. In this way, the feature data corresponding to each user ID in the user ID data set can be collected, and the A / B experiment can be verified whether there is an SRM problem based on the feature data.

[0065] Among them, since the A / B experiment information of this application includes but is not limited to the experiment ID, experiment background and experiment objectives, etc. This information not only records the comprehensive description of the specific environment, conditions and related circumstances of this A / B experiment, but also records the expected benefits of this A / B experiment. Therefore, this application can analyze the target user characteristics that may cause SRM problems during the A / B experiment through A / B experiment information. The target user characteristics here can be one or more, including but not limited to user gender, user age, user activity, etc., which can be set according to the actual situation and are not limited here.

[0066] After evaluating the target user characteristics that cause the SRM problem in the A / B experiment, the application can obtain a dataset of user IDs that participated in the A / B experiment within a preset historical period. Figure 2 As shown, the present application can first determine the preset historical period of the acquired user ID, so that Figure 2 The tool agent module in calls a specific tool to obtain the user ID dataset that participated in the A / B experiment within a preset historical period, so as to improve the data acquisition efficiency and thus improve the detection efficiency.

[0067] It is understandable that when the present application performs SRM testing on an A / B experiment, the Plan Agent module can be used to analyze whether the current data is insufficient. When the data is insufficient, the Plan Agent module can pass the context information to the ToolAgent module, and the Tool Agent module will decide whether it is necessary to call a specific tool to assist in problem solving. If a tool needs to be called, the Tool Agent module will perform the operation and return the acquired data to the Plan Agent module for the next task execution.

[0068] Among them, a skill list is pre-stored in the Tool Agent module of the present application, which describes the preset tools and their tool capabilities in advance. When the Tool Agent module obtains the skill list, it can decide which tool to use currently based on the missing data type. The specific code can use the function call capability of open AI to call the corresponding tool so as to use the corresponding tool to obtain the corresponding data, which includes but is not limited to the user data participating in the experiment (mainly user ID), user feature data and feature aggregation data.

[0069] S130: Extracting feature data of each user ID in the user ID data set according to the target user feature, and performing experimental grouping and feature aggregation on each feature data to obtain first aggregated data formed in the experimental group and second aggregated data formed in the control group according to the target user feature.

[0070] S140: Analyze the correlation between the first aggregated data and the second aggregated data, and determine whether an SRM problem occurs in the A / B experiment according to the analysis result.

[0071] In the above steps, after obtaining the user ID data set participating in the A / B experiment within a preset historical period through S120, the present application can extract the feature data of each user ID in the user ID data set according to the target user characteristics, and after performing experimental grouping and feature aggregation on each feature data, obtain the first aggregated data formed in the experimental group according to the target user characteristics and the second aggregated data formed in the control group. In this way, after analyzing the correlation between the first aggregated data and the second aggregated data, it is possible to determine whether an SRM problem occurs in the A / B experiment based on the analysis results.

[0072] To elaborate, after determining the target user characteristics that may cause SRM problems in the A / B experiment, and the user ID data set that participated in the A / B experiment within a preset historical period, the present application can use the Tool Agent module to batch extract the feature data corresponding to each user ID in the user ID data set according to the target user characteristics. Moreover, since the present application performs SRM inspection on the A / B experiment, after extracting the feature data of each user ID, the present application can also perform experimental grouping and feature aggregation on each feature data, so as to divide each feature data into feature data of the experimental group and feature data of the control group, and then perform feature aggregation on the feature data in the experimental group and the control group respectively, so as to obtain the first aggregated data and the second aggregated data of the experimental group and the control group under the target user characteristics.

[0073] Next, the present application can use appropriate statistical methods to analyze the correlation between the first aggregated data and the second aggregated data, and determine whether the A / B experiment has an SRM problem based on the analysis results, thereby providing a basis for subsequent decision-making of the experiment. In this process, the present application can also call a drawing tool through the Tool Agent module to generate statistical charts, and display the generated statistical charts to the user to intuitively show the differences between user experiences.

[0074] It can be understood that when the target user feature of the present application is one, the above method can be used to obtain the first aggregated data and the second aggregated data of the experimental group and the control group under the target user feature, and after analyzing the correlation between the first aggregated data and the second aggregated data under the target user feature, it is determined based on the analysis result whether the sample distribution of the A / B experiment on the target user feature has an SRM problem; when the target user feature of the present application is multiple, the above method can be used to respectively obtain the first aggregated data and the second aggregated data of the experimental group and the control group under different target user features, and after analyzing the correlation between the first aggregated data and the second aggregated data under different target user features, it is determined based on the analysis result whether the sample distribution of the A / B experiment on different target user features has an SRM problem.

[0075] In the above embodiment, when performing SRM inspection on the A / B experiment, the A / B experiment information required for the SRM inspection input by the user can be obtained, and then the target user characteristics that cause the SRM problem in the A / B experiment are evaluated based on the A / B experiment information, and the user ID data set participating in the A / B experiment in a preset historical period is obtained. In this way, the feature data of each user ID in the user ID data set can be extracted according to the target user characteristics, and after experimental grouping and feature aggregation of each feature data, first aggregated data formed in the experimental group according to the target user characteristics and second aggregated data formed in the control group are obtained. Finally, the correlation between the first aggregated data and the second aggregated data is analyzed, and it is determined whether the A / B experiment has an SRM problem based on the analysis result. During this process, users only need to input the A / B experiment information required for SRM testing, such as experiment ID, experiment background, experiment objectives, etc. They do not need to master professional data analysis knowledge, nor do they need to process the data themselves. They can use this application to intelligently match the target user characteristics that affect the A / B experiment, and obtain feature data based on the target user characteristics to test whether there is an SRM problem in the A / B experiment. This not only greatly reduces the requirements for users' professional skills, but also effectively improves analysis efficiency and accuracy.

[0076] In one embodiment, Figure 3 As shown, Figure 3A schematic diagram of a process for obtaining A / B experimental information required for SRM testing provided in an embodiment of the present application; obtaining the A / B experimental information required for SRM testing input by the user in S110 may include:

[0077] S111: Obtain content information input by the user.

[0078] S112: Perform intent recognition on the content information, and determine whether the content information involves SRM verification based on the recognition result; if so, execute S113; if not, execute S116.

[0079] S113: Determine whether the content information contains key information for SRM verification; if so, execute S114; if not, execute S115.

[0080] S114: Use the content information as A / B experiment information required for SRM testing.

[0081] S115: Prompt the user to supplement until the supplemented information contains key information for SRM inspection.

[0082] S116: Guide the user to input key information for SRM inspection.

[0083] In this embodiment, Figure 2 , 3 As shown, the user can input the A / B experiment information required for SRM inspection in the control terminal through the text chat interface, including but not limited to the experiment ID, experiment background and experiment goal. After the system receives the input, it can be handed over to the Plan Agent module for intention recognition analysis to determine whether the content input by the user involves SRM inspection. If it involves SRM inspection, it will further determine whether the user input content is comprehensive to ensure whether it contains key information such as experiment ID, experiment background, and experiment goal, so as to facilitate the subsequent accurate matching of user features. If the information provided is insufficient for SRM inspection, the Plan Agent module will provide real-time feedback to the user, requiring him to supplement the missing relevant information until the supplemented information contains the key information for SRM inspection; if the content information contains the key information for SRM inspection, the content information is directly used as the A / B experiment information required for SRM inspection; if it does not involve SRM inspection, the user can be guided to enter the key information for SRM inspection.

[0084] In a specific implementation, the intent recognition of the present application can be achieved by inputting a prompt into an LLM (large language model). For example, the present application can define what an SRM question is in the prompt, and tell it to reject irrelevant questions, and then use some examples to strengthen the LLM's understanding of this content (i.e., the few shot mechanism).

[0085] For example, the application may input the following example to llm:

[0086] Snippet 1:

[0087] User: Can you cook?

[0088] Assistant: The current user problem is not related to the SRM problem.

[0089] Snippet 2:

[0090] user: I want to know if there is any problem with the experiment.

[0091] Assistant: OK, I can help you analyze the SRM issue. If you need it, please provide the following information: experiment ID, experiment name, experiment purpose, and experiment strategy.

[0092] Through the above examples, LLM can strengthen its understanding of user input content, so as to identify the intent of the input content information and then determine whether the user has entered the A / B experiment information required for SRM testing.

[0093] In one embodiment, the A / B experiment information includes at least an experiment background and an experiment goal.

[0094] In S120, evaluating the target user characteristics causing the SRM problem in the A / B experiment according to the A / B experiment information may include:

[0095] S121: Determine a preset number of user characteristics.

[0096] S122: Evaluate multiple user characteristics that cause SRM problems in the A / B experiment based on the experiment background and the experiment goal.

[0097] S123: Selecting a target user feature corresponding to the number of user features from the plurality of user features.

[0098] In this embodiment, when evaluating the target user features that cause SRM problems in an A / B experiment, the present application can first determine a preset number of user features, and then evaluate multiple user features that cause SRM problems in the A / B experiment based on the experimental background and experimental objectives in the A / B experiment information, and finally select the target user features corresponding to the number of user features from the multiple user features.

[0099] It is understandable that the existing method of performing SRM inspection by presetting user features is not flexible enough and may not be suitable for most experimental scenarios. The user features that may be affected in different experiments may be different. In addition, if there are too few preset features, other important influencing features may be missed, resulting in inaccurate results. If there are too many preset features, many invalid calculations may be included, resulting in high calculation costs and waste of resources.

[0100] Based on this, the application can first obtain the A / B experiment information, because the information not only records the comprehensive description of the specific environment, conditions and related situations of this A / B experiment, but also records the expected benefits of this A / B experiment. Therefore, the application can make a brief description through prompt, so that the large language model can evaluate the multiple user features that cause the SRM problem in the A / B experiment based on the brief description, and then the large language model can select the target user features with a higher matching degree from the multiple user features based on the preset number of user features.

[0101] When briefly describing the problem through prompt, the application can use the format of dimension: reason to describe how to evaluate the target user characteristics that may cause SRM problems in the A / B experiment based on the experimental background and experimental goals input by the user. For example, the application can input the following description to llm:

[0102] Gender: This experiment implements different strategies based on user gender. The control group may cause different degrees of loss or retention of users of different genders, resulting in inconsistent gender ratios between experimental groups.

[0103] Of course, the present application can also train a large language model through other descriptions so that the large language model can analyze the n dimensions in which SRM problems may occur based on the input experimental background and experimental objectives. Of course, the dimensions here can also be a range, such as 3 to 5, etc., which can be set according to the actual situation and are not limited here.

[0104] In one embodiment, Figure 4 As shown, Figure 4 A flow chart of obtaining a user ID data set participating in an A / B experiment within a preset historical period provided in an embodiment of the present application; obtaining a user ID data set participating in the A / B experiment within a preset historical period in S120 may include:

[0105] S124: Acquire multiple user IDs that participated in the A / B experiment within a preset historical period.

[0106] S125: Determine whether the number of users of the multiple user IDs exceeds a preset number threshold; if so, execute S126; if not, execute S127.

[0107] S126: Sampling data for multiple user IDs, and constructing a user ID data set based on the sampled user IDs.

[0108] S127: directly construct a user ID data set based on multiple user IDs.

[0109] In this embodiment, when obtaining a user ID data set that participates in an A / B experiment within a preset historical period, the preset historical period can be determined first, and then multiple user IDs that participate in the A / B experiment within the preset historical period can be obtained, and it can be determined whether the number of users of the multiple user IDs exceeds a preset number threshold. If it exceeds, it indicates that the number of users currently obtained is large. At this time, in order to reduce computational complexity and increase analysis speed, data sampling can be performed on multiple user IDs, and a user ID data set can be constructed based on the sampled user IDs; if it does not exceed, it indicates that the number of users currently obtained is relatively appropriate, and there is no need to perform data sampling, and the user ID data set can be directly constructed based on multiple user IDs.

[0110] Among them, when the application performs data sampling on multiple user IDs, any data sampling method can be used, and the specific method can be selected according to the actual situation, and there is no restriction here. For example, the application can choose to use the Monte Carlo sampling method for data sampling to reduce the computational complexity and increase the analysis speed; when the Monte Carlo sampling method is used for data sampling, the sampling rate "p" can be dynamically adjusted according to different user orders of magnitude, and the specific formula is as follows:

[0111]

[0112] in, refers to the i-th user ID, and p refers to the sampling rate.

[0113] In one embodiment, after performing experimental grouping and feature aggregation on each feature data in S130, first aggregated data formed in the experimental group and second aggregated data formed in the control group according to the target user feature are obtained, which may include:

[0114] S131: performing experimental grouping on each feature data according to the user ID corresponding to each feature data, and obtaining feature data in the experimental group and feature data in the control group.

[0115] S132: performing feature aggregation on the feature data in the experimental group and the feature data in the control group according to the target user feature, to obtain first aggregated data formed in the experimental group and second aggregated data formed in the control group according to the target user feature.

[0116] In this embodiment, when performing experimental grouping and feature aggregation on each feature data, each feature data can be first divided into feature data of the experimental group and feature data of the control group according to the experimental group type set in the A / B experiment and each user ID. Then, this application can perform feature aggregation on the feature data in the experimental group and the feature data in the control group respectively according to the target user characteristics, so as to obtain the final first aggregated data formed in the experimental group according to the target user characteristics and the second aggregated data formed in the control group.

[0117] For example, the present application may first establish a map mapping, that is, which experimental group the user ID belongs to, such as uid111->group 1, uid222->group 2. Then, the present application may aggregate according to the feature data and establish a map mapping to record the number of group->feature->feature values, such as group 1->male->number 300, group 1->female->number 289, group 2->male->number 302, group 2->female->number 288.

[0118] Subsequently, statistical analysis can be performed directly based on the data, thereby effectively improving detection efficiency and accuracy.

[0119] Furthermore, the present application can also construct a histogram based on the first aggregated data formed by the target user characteristics in the experimental group and the second aggregated data formed in the control group. For example, the horizontal axis of the histogram constructed by the present application can be the characteristic value of the target user characteristic, such as male and female; the vertical axis can be the quantity. For example, when the first aggregated data is group 1->male->quantity 300, group 1->female->quantity 289, and the second aggregated data is group 2->male->quantity 302, group 2->female->quantity 288, four histograms can be formed, which are four quantities, group 1 male 300, group 2 male 302, group 1 female 289, group 2 female 288. After displaying the histogram to the user, the distribution of the target user characteristics in different groups can be intuitively seen.

[0120] In one embodiment, analyzing the correlation between the first aggregated data and the second aggregated data in S140 may include:

[0121] S141: Determine the data volume of the first aggregated data and the second aggregated data.

[0122] S142: Determine, according to the data volume, a statistical method used when performing correlation analysis on the first aggregated data and the second aggregated data.

[0123] S143: Analyze the correlation between the first aggregated data and the second aggregated data based on the statistical method.

[0124] In this embodiment, when performing correlation analysis on the first aggregated data and the second aggregated data, the data volume of the first aggregated data and the second aggregated data can be determined first, and the statistical method for performing the correlation analysis can be determined based on the data volume. Then, the correlation between the first aggregated data and the second aggregated data can be analyzed based on the statistical method. In this way, the data analysis efficiency and analysis accuracy can be further improved through appropriate statistical methods.

[0125] In one embodiment, determining, according to the data volume, a statistical method used for performing correlation analysis on the first aggregated data and the second aggregated data in S142 may include:

[0126] S1421: Determine whether the data volume is greater than a preset data volume threshold.

[0127] S1422: If it is greater than, determine to use chi-square test to perform correlation analysis on the first aggregated data and the second aggregated data.

[0128] S1423: If not greater than, determine to use Fisher's exact test to perform a correlation analysis on the first aggregated data and the second aggregated data.

[0129] In this embodiment, when determining the statistical method to be used for correlation analysis of the first aggregated data and the second aggregated data based on the data volume, the present application may first determine whether the data volume is greater than a preset data volume threshold; if it is greater than, it indicates that the current data volume is large, and therefore a chi-square test may be used to perform a correlation analysis on the first aggregated data and the second aggregated data, and determine whether there is a significant difference in the distribution of the two groups of aggregated data on the target user characteristics based on the p-value in the chi-square test, that is, whether there is an SRM problem; if it is not greater than, it indicates that the current data volume is small, and therefore a Fisher's exact test may be used to perform a correlation analysis on the first aggregated data and the second aggregated data, and determine whether there is a significant difference in the distribution of the two groups of aggregated data on the target user characteristics based on the p-value of the Fisher's exact test, and then determine whether there is an SRM problem.

[0130] It is understandable that Fisher's exact test is a statistical method for analyzing the association between two categorical variables, especially for situations where the sample size is small or the data does not meet the chi-square test assumptions. Therefore, this application can combine it with the chi-square test. When the amount of data is small, Fisher's exact test is used for data statistics, and when the amount of data is large, the chi-square test is used for data statistics, so as to improve the detection accuracy.

[0131] The following is a description of a detection device for the sample ratio mismatch problem provided in an embodiment of the present application. The detection device for the sample ratio mismatch problem described below and the detection method for the sample ratio mismatch problem described above can refer to each other.

[0132] In one embodiment, Figure 5 As shown, Figure 5 A schematic diagram of a detection device for a sample ratio mismatch problem provided in an embodiment of the present application; the present application also provides a detection device for a sample ratio mismatch problem, including:

[0133] The experiment information acquisition module 210 is used to acquire the A / B experiment information required for the SRM test input by the user when the SRM test is performed on the A / B experiment.

[0134] The user data acquisition module 220 is used to evaluate the target user characteristics that cause the SRM problem in the A / B experiment according to the A / B experiment information, and obtain a user ID data set that participated in the A / B experiment within a preset historical period.

[0135] The feature aggregation module 230 is used to extract feature data of each user ID in the user ID data set according to the target user feature, and after performing experimental grouping and feature aggregation on each feature data, obtain first aggregated data formed in the experimental group according to the target user feature and second aggregated data formed in the control group according to the target user feature.

[0136] The correlation analysis module 240 is used to analyze the correlation between the first aggregated data and the second aggregated data, and determine whether the A / B experiment has an SRM problem based on the analysis result.

[0137] In the above embodiment, when performing SRM inspection on the A / B experiment, the A / B experiment information required for the SRM inspection input by the user can be obtained, and then the target user characteristics that cause the SRM problem in the A / B experiment are evaluated based on the A / B experiment information, and the user ID data set participating in the A / B experiment in a preset historical period is obtained. In this way, the feature data of each user ID in the user ID data set can be extracted according to the target user characteristics, and after experimental grouping and feature aggregation of each feature data, first aggregated data formed in the experimental group according to the target user characteristics and second aggregated data formed in the control group are obtained. Finally, the correlation between the first aggregated data and the second aggregated data is analyzed, and it is determined whether the A / B experiment has an SRM problem based on the analysis result. During this process, users only need to input the A / B experiment information required for SRM testing, such as experiment ID, experiment background, experiment objectives, etc. They do not need to master professional data analysis knowledge, nor do they need to process the data themselves. They can use this application to intelligently match the target user characteristics that affect the A / B experiment, and obtain feature data based on the target user characteristics to test whether there is an SRM problem in the A / B experiment. This not only greatly reduces the requirements for users' professional skills, but also effectively improves analysis efficiency and accuracy.

[0138] In one embodiment, the present application also provides a computer-readable storage medium, which stores computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the method for detecting the sample ratio mismatch problem as described in any of the above embodiments.

[0139] In one embodiment, the present application also provides a computer device, including: one or more processors, and a memory.

[0140] The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the method for detecting the sample ratio mismatch problem as described in any one of the above embodiments are performed.

[0141] Indicatively, Figure 6 As shown, Figure 6 This is a schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. The computer device 300 may be provided as a server. Figure 6The computer device 300 includes a processing component 302, which further includes one or more processors, and a memory resource represented by a memory 301, for storing instructions executable by the processing component 302, such as an application. The application stored in the memory 301 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 302 is configured to execute instructions to perform the detection method of the sample ratio mismatch problem of any of the above embodiments.

[0142] The computer device 300 may further include a power supply component 303 configured to perform power management of the computer device 300, a wired or wireless network interface 304 configured to connect the computer device 300 to a network, and an input / output (I / O) interface 305. The computer device 300 may operate based on an operating system stored in the memory 301, such as Windows Server TM, Mac OS X TM, Unix TM, Linux TM, Free BSD TM, or the like.

[0143] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0144] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0145] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

[0146] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for detecting sample ratio mismatch problem, characterized in that: The method comprises: When performing SRM testing on an A / B experiment, obtaining A / B experiment information required for the SRM testing input by the user; Evaluate the target user characteristics that cause the SRM problem in the A / B experiment based on the A / B experiment information, and obtain a user ID dataset that participated in the A / B experiment within a preset historical period; Extracting feature data of each user ID in the user ID data set according to the target user feature, and performing experimental grouping and feature aggregation on each feature data to obtain first aggregated data formed in the experimental group and second aggregated data formed in the control group according to the target user feature; The correlation between the first aggregated data and the second aggregated data is analyzed, and it is determined whether an SRM problem occurs in the A / B experiment according to the analysis result.

2. The method for detecting sample ratio mismatch problem according to claim 1, characterized in that: The obtaining of the A / B experiment information required for the SRM test input by the user includes: Get the content information entered by the user; Performing intent recognition on the content information, and determining whether the content information involves SRM inspection according to the recognition result; If so, determining whether the content information contains key information for SRM inspection; If included, the content information is used as the A / B experiment information required for SRM testing; If not, the user is prompted to supplement until the supplemented information contains the key information for SRM inspection; If not, the user is guided to enter key information for SRM inspection.

3. The method for detecting sample ratio mismatch problem according to claim 1, characterized in that: The A / B experiment information at least includes the experiment background and the experiment goal; The evaluating, according to the A / B experiment information, the target user characteristics causing the SRM problem in the A / B experiment includes: Determine the preset number of user features; Evaluate multiple user characteristics that cause the A / B experiment to have SRM problems based on the experiment context and the experiment goal; A target user feature corresponding to the number of user features is selected from the plurality of user features.

4. The method for detecting sample ratio mismatch problem according to claim 1, characterized in that: The step of obtaining a user ID dataset that participated in the A / B experiment within a preset historical period includes: Obtain multiple user IDs that participated in the A / B experiment within a preset historical period; Determine whether the number of users of the plurality of user IDs exceeds a preset number threshold; If it exceeds, data sampling is performed on the multiple user IDs, and a user ID data set is constructed based on the sampled user IDs; If not, directly construct a user ID data set based on the multiple user IDs.

5. The method for detecting sample ratio mismatch problem according to claim 1, characterized in that: After performing experimental grouping and feature aggregation on each feature data, first aggregated data formed in the experimental group and second aggregated data formed in the control group according to the target user feature are obtained, including: According to the user ID corresponding to each feature data, each feature data is experimentally grouped to obtain feature data in the experimental group and feature data in the control group; Feature aggregation is performed on the feature data in the experimental group and the feature data in the control group according to the target user feature to obtain first aggregated data formed in the experimental group and second aggregated data formed in the control group according to the target user feature.

6. The method for detecting sample ratio mismatch problem according to claim 1, characterized in that: The analyzing the correlation between the first aggregated data and the second aggregated data includes: determining the data volume of the first aggregated data and the second aggregated data; Determining, according to the data volume, a statistical method used when performing correlation analysis on the first aggregated data and the second aggregated data; The correlation between the first aggregated data and the second aggregated data is analyzed based on the statistical method.

7. The method for detecting sample ratio mismatch problem according to claim 6, characterized in that: The determining, according to the data volume, a statistical method used when performing correlation analysis on the first aggregated data and the second aggregated data includes: Determining whether the data volume is greater than a preset data volume threshold; If it is greater than, determining to use a chi-square test to perform a correlation analysis on the first aggregated data and the second aggregated data; If not, it is determined to use Fisher's exact test to perform a correlation analysis on the first aggregated data and the second aggregated data.

8. A detection device for sample ratio mismatch problem, characterized in that: include: The experiment information acquisition module is used to obtain the A / B experiment information required for the SRM test input by the user when the A / B experiment is subjected to the SRM test; A user data acquisition module is used to evaluate the target user characteristics that cause the SRM problem in the A / B experiment according to the A / B experiment information, and obtain a user ID data set that participated in the A / B experiment in a preset historical period; A feature aggregation module, used to extract feature data of each user ID in the user ID data set according to the target user feature, and perform experimental grouping and feature aggregation on each feature data to obtain first aggregated data formed in the experimental group and second aggregated data formed in the control group according to the target user feature; The correlation analysis module is used to analyze the correlation between the first aggregated data and the second aggregated data, and determine whether the A / B experiment has an SRM problem according to the analysis result.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the method for detecting the sample ratio mismatch problem as described in any one of claims 1 to 7.

10. A computer device, characterized in that: include: one or more processors, and memory; The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the method for detecting the sample ratio mismatch problem as described in any one of claims 1 to 7 are performed.

Images