AB experiment generation and optimization method based on AI model and local vector database

Abstract

The invention discloses an AB experiment generation and optimization method based on an AI model and a local vector database. The AB experiment generation and optimization method comprises the following steps: constructing a local experiment database comprising a business knowledge base, a data mart and historical experiment data; carrying out vectorization processing on the business knowledge base, the data marts and the historical experimental data, and storing generated vectors in a local vector database; a large-scale language AI model is utilized to carry out a prompt project, full-process automation of AB experiments is realized, and the full-process automation comprises automatic experiment creation, automatic effect estimation and automatic result analysis. According to the method, the experimental design scheme can be automatically analyzed and generated through training and fine tuning by utilizing the large language AI model, and the fine tuning process enables the large language AI model to adapt to a specific business scene and accurately identify key factors influencing user behaviors, so that intelligent suggestions are provided for experimental design.

Description

Technical Field

[0001] The present invention relates to the field of AI model technology, and in particular to an AB experiment generation and optimization method based on an AI model and a local vector database. Background Art

[0002] In traditional AB testing practices, companies usually rely on manual data processing and basic statistical methods to analyze and evaluate experimental results. This method is often limited by the efficiency and accuracy of data processing, especially when faced with large-scale data sets and complex experimental designs. In addition, traditional methods often lack in-depth analysis of data and real-time feedback mechanisms, resulting in limitations in the interpretation and application of experimental results. With the advent of the big data era and the development of artificial intelligence technology, there is an increasing demand for a more efficient, automated, and intelligent AB testing method.

[0003] Among existing technologies, although there have been some attempts to apply AI technology to data analysis, these attempts are often limited to specific analysis tasks and lack a comprehensive, automated framework to cover the entire life cycle of AB testing. In addition, these solutions usually require a lot of manual configuration and expertise, limiting their applicability in a rapidly changing market environment. Summary of the invention

[0004] In view of the problems existing in the prior art, the present invention provides an AB experiment generation and optimization method based on an AI model and a local vector database. The core of the present invention is to use an external or self-developed large-scale language AI model (LLM), which can automatically analyze and generate experimental design plans through training and fine-tuning. The fine-tuning process enables LLM to adapt to specific business scenarios and accurately identify the key factors that affect user behavior, thereby providing intelligent suggestions for experimental design.

[0005] In addition, the present invention combines the use of local vector databases, which show great advantages in high-dimensional data storage and retrieval. Vector database types include open source solutions such as Faiss, Milvus, Pinecone and Weaviate, as well as commercial products such as Amazon OpenSearch and Google Bigtable. These databases support efficient storage and retrieval of high-dimensional vectors, and can quickly process large-scale user behavior data, thereby accelerating the feedback cycle of experimental results. The introduction of vector databases enables the system to perform data analysis with lower latency and higher accuracy, effectively supporting the dynamic adjustment of A / B testing.

[0006] In terms of supported large language models, the current open source head large models include GPT-2, BERT and T5 in Hugging Face's Transformers library. These models have powerful natural language processing capabilities and can adapt to specific application scenarios through prompt engineering and fine-tuning. In addition, commercial API interfaces such as OpenAI and Baidu Wenxin Yiyan platforms all provide fine-tuning and data vectorization interfaces, which can provide support for the present invention.

[0007] Related technologies include training and fine-tuning of large models, prompt engineering, and the application of vector databases. By fine-tuning LLM, the system can use historical experimental data and user feedback to continuously optimize the model's predictive capabilities. By integrating these advanced technologies and tools, the present invention aims to build an intelligent A / B testing platform to achieve automated generation and optimization of experiments, improve experimental efficiency and reliability of results, and thus help companies achieve better performance in a highly competitive market.

[0008] In order to achieve the above object, the present invention provides an AB experiment generation and optimization method based on an AI model and a local vector database, comprising the following steps:

[0009] Step S1: Build a local experimental database including business knowledge base, data mart and historical experimental data;

[0010] Step S2: vectorize the business knowledge base, data mart and historical experimental data, and store the generated vectors in the local vector database;

[0011] Step S3: Use a large language AI model to perform prompt engineering to automate the entire process of AB experiments, including automatic experiment creation, automatic effect estimation, and automatic result analysis;

[0012] The automatic creation of experiments includes business scenario matching, grouping strategy recommendation, observation indicator recommendation, and experimental plan formulation;

[0013] Automatic effect estimation includes grouping result evaluation and distribution test evaluation, using statistical and machine learning techniques to perform data distribution tests and effect estimation;

[0014] Automatic result analysis includes result evaluation, business strategy proposals, and experiment optimization proposals. It uses a large AI model to comprehensively analyze the experimental results and evaluate the overall performance and business value of the experiment.

[0015] Preferably, in step S1, the data mart obtains business data by connecting to a traditional database or other data sources, the traditional database includes an advertising data system ADS, and other data sources include a service layer API.

[0016] Preferably, the specific steps of the vectorization processing in step S2 are as follows:

[0017] Step S201: selecting a vectorization technology and using a pre-trained deep learning model to capture semantic information in text data;

[0018] Step S202: data preprocessing, including data cleaning, format standardization, and feature extraction; data cleaning is used to remove duplicate, redundant, or irrelevant data to ensure the purity of the data set; format standardization is used to unify data from different sources into the same format for batch processing; feature extraction is used to identify features related to the experiment so as to better vectorize;

[0019] Step S203: Select a vectorization model and convert the processed data: input the text content of the business knowledge base, data mart and historical experimental data into the model, the model generates the corresponding vector representation, and combines the context information to ensure that the vector can reflect the actual meaning of the data.

[0020] Preferably, in step S2, a large language AI model is used to vectorize the information of multiple files for the data in the vectorization processing stage. The specific process is as follows:

[0021] Step S204: Data preparation: collect business knowledge base, data mart and historical experimental data into folders and integrate them into input for vectorization processing;

[0022] Step S205: Read the folder contents: use Python to read all the file contents in the folder;

[0023] Step S206: text input and model call: use OpenAI's API to call the selected large language AI model for vectorization processing, input the file content into the model, and generate a vector representation;

[0024] Step S207: vector generation: the large language AI model generates a high-dimensional vector based on the input text, reflecting the semantics and contextual information of the text;

[0025] Step S208: storing the vector: storing the generated vector in a local vector database for subsequent retrieval and use.

[0026] Preferably, in the early stage of experiment creation, the system matches the business scenario in step S3. Based on the experiment name input by the user, the system performs semantic analysis and intent recognition through a large language AI model to automatically match the most suitable experimental business scenario, and recommends corresponding experimental template parameters to the user based on industry experience and historical data. These parameters include basic experimental settings, grouping strategies, and observation indicators; and the selected experimental template will be used as a priority option for subsequent experimental strategy recommendations to ensure that the grouping design, observation indicators, etc. are highly matched with the experimental intent.

[0027] Preferably, the grouping strategy recommendation in step S3: In the experimental grouping stage, the system intelligently highlights the key parameter settings in the grouping based on the large-scale language AI model, including the number of groups, group names, and multi-dimensional screening schemes, and provides a brief analysis of the parameter settings to help users understand the pros and cons of different schemes.

[0028] Preferably, in step S3, the sub-observation indicator recommendation: the system combines historical experimental data and large-scale language AI model decisions to automatically screen and recommend the most suitable observation indicators, including core indicators and secondary indicators, to ensure the comprehensiveness and scientificity of data analysis;

[0029] Experimental plan formulation in step S3: The system automatically recommends the start and end time of the experiment based on the entered experimental information and historical experimental data, taking into account grouping conflicts and historical performance factors of similar experiments, and provides detailed recommendation reasons and AI analysis reports.

[0030] Preferably, the automatic effect estimation in step S3 includes the following specific steps:

[0031] Group result evaluation: Before the experiment begins, the system uses Python scripts or other automated tools to splice and process data according to the settings of each experimental group, and generates real data for each group in real time to ensure that the group meets the experimental requirements;

[0032] Distribution test evaluation: After completing the grouping result evaluation, the system will simulate and estimate the experimental data based on the existing grouping scheme and observation indicators, combined with the experimental plan.

[0033] Preferably, during the automatic effect estimation in step S3, the specific process of using statistics and machine learning techniques to perform data distribution inspection and effect estimation is as follows:

[0034] S301, data extraction and distribution verification: The system first extracts sample data from the database and log system data sources through the ETL process to ensure the representativeness of the data;

[0035] Use stratified sampling or random sampling techniques, combined with the Shapiro-Wilk test or the Kolmogorov-Smirnov test, to check whether the data conform to normal distribution;

[0036] If the data do not meet the assumptions, the system will automatically apply normalization techniques or nonparametric statistical methods, such as logarithmic transformation or rank sum test;

[0037] S302, Variance Analysis and Effect Prediction: The system uses a variety of statistical techniques to estimate the potential effects of the experiment, including:

[0038] T test: used to analyze the significant difference between the experimental group and the control group, and calculate the t value and p value;

[0039] ANOVA: used to analyze mean differences among multiple groups;

[0040] Monte Carlo simulation: simulate the experimental process through a large number of random sampling to estimate the potential experimental effect and confidence interval;

[0041] Use machine learning algorithms to learn and analyze historical data to predict group differences and change trends;

[0042] S303, significance test and P value calculation: the system calculates the P value based on the grouped data to measure the significance of the experimental results;

[0043] If the P value is too large, the system will provide optimization suggestions, including adjusting the experimental duration, sample size or grouping strategy;

[0044] Large language AI models will provide adjustment suggestions or experimental reconstruction options based on historical data analysis to ensure the scientificity and effectiveness of experimental design;

[0045] S304, data reporting and visualization: The system presents the evaluation results through visual reports to help users intuitively understand the various data indicators of the experiment. These reports include group distribution diagrams, significant difference analysis diagrams, simulation effect estimation diagrams and risk assessment reports.

[0046] Preferably, the result analysis in step S3 includes the following steps:

[0047] Result evaluation: The system will use a large-scale language AI model to conduct a comprehensive analysis of experimental observation indicators and evaluate the overall performance of the experiment. Through historical data and AI algorithms, the significance level of each observation indicator is calculated, and the impact of the experiment on the core business indicators is analyzed. The system will conduct an in-depth evaluation of the feasibility and business value of the experiment, and predict the long-term effect of the experimental results in actual scenarios.

[0048] Business strategy proposals: Based on the experimental results and in-depth analysis of large-scale language AI models, the system will make suggestions on business strategies; by analyzing the execution effect of the current strategy, the large-scale language AI model can identify potential optimization opportunities and propose feasible adjustment plans; the system will predict the potential impact of these strategies on business operations to ensure that the decisions are commercially viable.

[0049] Experiment optimization suggestions: The system analyzes the specific execution of the experiment and uses a large language AI model to provide further experimental optimization suggestions, including:

[0050] Suggestions on adjusting the experimental grouping strategy, optimizing the number of groups or redesigning the grouping dimensions;

[0051] Provide suggestions for optimizing screening conditions to ensure the representativeness of samples and the accuracy of experimental results;

[0052] And suggestions for introducing more observation indicators are provided to expand the depth of experimental evaluation and improve the comprehensiveness and reference value of experimental results.

[0053] The technical solution of the present invention has the following beneficial effects:

[0054] The present invention utilizes a large-scale language AI model through training and fine-tuning to automatically analyze and generate experimental design plans. The fine-tuning process enables the large-scale language AI model to adapt to specific business scenarios and accurately identify key factors that affect user behavior, thereby providing intelligent suggestions for experimental design.

[0055] In addition, the present invention combines the use of local vector databases, which support efficient storage and retrieval of high-dimensional vectors, and can quickly process large-scale user behavior data, thereby accelerating the feedback cycle of experimental results. The introduction of vector databases enables the system to perform data analysis with lower latency and higher accuracy, effectively supporting the dynamic adjustment of AB testing.

[0056] In terms of supported large language models, the current open source head large models include GPT-2, BERT and T5 in Hugging Face's Transformers library. These models have powerful natural language processing capabilities and can adapt to specific application scenarios through prompt engineering and fine-tuning. In addition, commercial API interfaces such as OpenAI and Baidu Wenxin Yiyan platforms all provide fine-tuning and data vectorization interfaces, which can provide support for the present invention.

[0057] Improve the level of experimental automation: This invention realizes the full process automation of AB experiments by combining the AI ​​big model and the local vector database, reduces manual operations, and improves the efficiency and accuracy of the experiments.

[0058] Optimize experimental design: Using AI models to match business scenarios and recommend grouping strategies, the system can intelligently generate experimental templates to ensure that experimental design is closely linked to business objectives.

[0059] Improve the depth and breadth of data analysis: The system uses statistical and machine learning techniques to test data distribution and estimate effects, providing comprehensive data analysis, including significance testing and long-term effect prediction.

[0060] Real-time effect prediction and risk assessment: The automatic effect prediction function allows potential results to be predicted before the experiment begins, providing real-time risk analysis and optimization suggestions to help users adjust experimental strategies in a timely manner.

[0061] Enhance the interpretability of experimental results: Through visual reports and data reports, the system converts complex data analysis results into intuitive graphics, allowing non-technical users to easily understand the key indicators of the experiment.

[0062] Improve the scientificity and operability of experiments: The detailed recommendation reasons and AI analysis reports provided by the system make the experimental design more scientific, while improving the operability of the experiments and making the experimental results easier for the business team to understand and apply.

[0063] Adaptability to complex experimental scenarios: The method of the present invention is applicable to various complex experimental scenarios, including multi-dimensional screening and multi-group comparison, can process large-scale data sets, and is suitable for enterprises of different sizes and types. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 Flowchart for constructing a local experimental database for the present invention

[0065] Figure 2 AB experiment flow chart of the present invention using a large language AI model DETAILED DESCRIPTION

[0066] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0067] Reference Figure 1 to Figure 2 The present invention provides an AB experiment generation and optimization method based on an AI model and a local vector database, comprising the following steps:

[0068] Step S1: Build a local experimental database including business knowledge base, data mart and historical experimental data; centrally store and manage business knowledge base, data mart and historical experimental data to provide necessary data support for AB testing, ensure data accessibility and consistency, facilitate experimental data integration and analysis, and improve data utilization;

[0069] Business knowledge base provides a common industry knowledge reserve, including basic questions and precautions in various experimental scenarios. This part of the data is presented in the form of question-and-answer and dictionary, which is convenient for the system to quickly consult when formulating strategies and configuring experiments. The addition of the business knowledge base ensures that the system can refer to conventional industry standards and rules when configuring experiments or optimizing strategies, helping AI make decisions with greater business understanding and scenario relevance. Problem points in the business knowledge base may include common precautions in experimental design, such as grouping strategies, definitions of key observation indicators, etc.

[0070] Data marts obtain business data by connecting to traditional databases (such as advertising data systems ADS, etc.) or other data sources (such as service layer APIs). The system scans and reads the format, data type, field definition, etc. of these data tables to fully understand the data assets of the business. In this process, the quality of the data mart is crucial and will affect the subsequent AI strategy output. The AI ​​capabilities of large models can quickly understand the company's data situation by reading the data warehouse table structure and notes.

[0071] Historical experimental data: Historical experimental data includes various A / B tests that have been conducted before. These experiments have been manually intervened and evaluated, and have clear results (such as success or failure). These data not only record the specific configuration of the experiment (such as grouping method, indicator selection, experimental duration, etc.), but also include the experimental effect and business value feedback; historical experimental data is an important basis for AI strategy optimization. By introducing these data, the AI ​​system can continuously iterate and optimize its output;

[0072] Step S2: vectorize the business knowledge base, data mart and historical experimental data, and store the generated vectors in the local vector database; convert the unstructured text data into high-dimensional vectors so that the machine learning model can process it, so that the text data can be quantified, which is convenient for similarity comparison and pattern recognition, and provides numerical feature representation for subsequent experimental design and analysis;

[0073] Step S3: Use a large language AI model (LLM) to perform prompt engineering to achieve full process automation of AB experiments, including automatic creation of experiments, automatic effect estimation, and automatic result analysis; among them, automatic creation of experiments includes business scenario matching, grouping strategy recommendation, observation indicator recommendation, and experimental plan formulation; Automatic creation of experiments: reduce manual operations, improve the efficiency and accuracy of experiment creation, and ensure that the experimental design meets the needs of business scenarios; Automatic effect estimation includes grouping result evaluation, distribution test evaluation, and use statistical and machine learning techniques to perform data distribution test and effect estimation; Automatic effect estimation: Use statistical and machine learning techniques to estimate the experimental results, help predict the possible effects of the experiment, and reduce experimental risks. Automatic result analysis includes result evaluation, business strategy solution recommendations, and experimental optimization recommendations. The AI ​​large model is used to comprehensively analyze the experimental results to evaluate the overall performance and business value of the experiment; Automatic result analysis: The AI ​​model comprehensively analyzes the experimental results, provides in-depth business strategy recommendations, and optimizes the experimental design and business decisions. In step S1, the data mart obtains business data by connecting to traditional databases or other data sources. Traditional databases include the advertising data system ADS, and other data sources include the service layer API.

[0074] Furthermore, the specific steps of the vectorization processing in step S2 are as follows:

[0075] Step S201: selecting a vectorization technology and using a pre-trained deep learning model to capture semantic information in text data; the deep learning model includes BERT and GPT, which can effectively capture semantic information in text data, thereby generating a high-dimensional vector representation;

[0076] Step S202: data preprocessing, including data cleaning, format standardization, and feature extraction; data cleaning: used to remove duplicate, redundant or irrelevant data to ensure the purity of the data set; format standardization: used to unify data from different sources into the same format for batch processing; feature extraction: used to identify features related to the experiment for better vectorization; before vectorization, the original data is preprocessed as necessary to ensure data quality and consistency;

[0077] Step S203: Select a vectorization model and convert the processed data: input the text content of the business knowledge base, data mart and historical experimental data into the model, the model generates the corresponding vector representation, and combines the context information to ensure that the vector can reflect the actual meaning of the data.

[0078] Furthermore, in step S2, the data in the vectorization processing stage is vectorized using a large language AI model (LLM) to vectorize the information of multiple files. The specific process is as follows:

[0079] Step S204: Data preparation: collect business knowledge base, data mart and historical experimental data into folders and integrate them into input for vector processing; ensure that all relevant data are centrally managed to facilitate subsequent processing and improve the efficiency and consistency of data processing.

[0080] Step S205: Read the folder contents: Use Python to read the contents of all files in the folder. You can use the os module to traverse the folder and read the contents of each file. Automatically read files in the folder through programming to reduce manual operations and increase processing speed. Quickly access and process a large number of files to ensure the timely availability of data and provide basic data for vectorized processing.

[0081] Step S206: Text input and model call: Use OpenAI's API to call the selected large language AI model for vectorization processing, input the file content into the model, and generate a vector representation; according to OpenAI's latest API documentation, you can use the openai.Embedding.create method to generate embeddings, and use the powerful capabilities of the AI ​​model to convert text data into a machine-understandable numerical form, providing a quantitative basis for data analysis.

[0082] Step S207: Vector generation: The large language AI model generates high-dimensional vectors based on the input text, reflecting the semantics and contextual information of the text. These vector representations will contain the characteristics of each input text, facilitating subsequent analysis and retrieval; the vector representation can reveal the similarities and differences between texts, providing support for subsequent similarity search, classification or clustering analysis.

[0083] Step S208: Store vectors: Store the generated vectors in a local vector database and create corresponding records to ensure that they can be retrieved and used as needed in the future. For example, SQLite or other database systems can be used for storage. The persistent storage of vector data allows the data to be accessed and analyzed at different times and locations, improving the reusability of the data and the flexibility of the experiment.

[0084] The complete demonstration code is as follows:

[0085] import os

[0086] import openai

[0087] import numpy as np

[0088] import sqlite3

[0089] #Set OpenAI API key

[0090] openai.api_key='YOUR_API_KEY' #Please replace with your API key

[0091] #Data preparation: Read all text files in the folder

[0092] folder_path='path / to / your / folder'#Please replace with your folder path

[0093] document_contents=[]

[0094] for filename in os.listdir(folder_path):

[0095] if filename.endswith('.txt'): #Read only text files

[0096] with open(os.path.join(folder_path, filename), 'r', encoding = 'utf-8') as file:

[0097] document_contents.append(file.read())#Add file contents to the list

[0098] #Vectorization process: Call OpenAI's API to generate text embedding

[0099] responses = []

[0100] for content in document_contents:

[0101] response=openai.Embedding.create(

[0102] input=content,

[0103] model = "text-embedding-ada-002" # Use the latest embedding model )

[0105] responses.append(response['data'][0]['embedding'])#Get vector representation

[0106] #Store vectors in local database

[0107] conn=sqlite3.connect('local_vector_db.db')

[0108] cursor = conn.cursor()

[0109] cursor.execute('''CREATE TABLE IF NOT EXISTS experiment_vectors(idINTEGER PRIMARY KEY, vector BLOB)'')

[0110] for vec in responses:

[0111] cursor.execute('INSERT INTO experiment_vectors(vector)VALUES(?)', (np.array(vec).tobytes(),))

[0112] conn.commit()

[0113] conn.close()

[0114] print("Vectorization is complete, and all vectors have been stored in the database.")

[0115] Furthermore, in the business scenario matching in step S3, at the early stage of experiment creation, the system automatically matches the most appropriate experimental business scenario based on the experiment name input by the user through a large language AI model for semantic analysis and intent recognition, and recommends corresponding experiment template parameters to the user based on industry experience and historical data. These parameters include basic experiment settings, grouping strategies, and observation indicators; and the selected experiment template will be used as the priority option for subsequent experimental strategy recommendations to ensure that the grouping design, observation indicators, etc. are highly matched with the experimental intent; for example, if the user's experiment involves the optimization of an e-commerce platform, the system may recommend templates related to the user's purchase path, giving priority to matching user needs;

[0116] Semantic analysis and intent recognition: Through a large-scale language AI model, the experiment name entered by the user is analyzed to understand the user's experimental purpose and needs, accurately capture the user's experimental intention, provide accurate contextual information for subsequent steps, and ensure the relevance and effectiveness of the experimental design.

[0117] Automatically match experimental business scenarios: Based on the results of semantic analysis and intent recognition, the system automatically matches the most appropriate experimental business scenarios from the business knowledge base, quickly locates the business scenarios that match the user's experimental intent, and improves the efficiency and quality of experimental design.

[0118] Recommended experiment template parameters: The system recommends experiment template parameters to users based on the matched business scenarios, combined with industry experience and historical data, and provides standardized experiment design parameters to reduce users' guesswork and errors when designing experiments and improve the standardization and repeatability of experiments.

[0119] Ensure that the experimental template is highly matched with the experimental intent. The selected experimental template will be used as the priority option for subsequent experimental strategy recommendations. Ensure that the grouping design, observation indicators, etc. are highly matched with the experimental intent. By ensuring the consistency and relevance of the experimental design, the reliability and effectiveness of the experimental results are improved.

[0120] Prioritize matching user needs. The system recommends relevant templates based on the user's experimental field (such as e-commerce platform optimization), prioritizes meeting the user's specific needs, provides customized experimental design suggestions, enhances user experience, and improves the pertinence and effectiveness of experimental design.

[0121] Through this process, the method of the present invention can ensure that the design of AB experiments closely fits the specific business needs of users, thereby improving the success rate of experiments and the support for business decisions. This method is particularly suitable for scenarios that require precise experimental design to guide business development, such as e-commerce platforms, social media platforms, etc., and can significantly improve the effect of experiments and business value.

[0122] Further, the grouping strategy recommendation in step S3: In the experimental grouping stage, the system intelligently highlights the key parameter settings in the grouping based on the large language AI model, including the number of groups, group names, and multi-dimensional screening schemes, and provides a brief analysis of the parameter settings to help users understand the pros and cons of different schemes and support scientific decision-making. For example, the system can prompt the characteristic differences of certain groups to help optimize the rationality and operability of the experimental design. In this embodiment, the key parameter settings are intelligently highlighted: the system uses a large language AI model to analyze the experimental requirements, intelligently identify and highlight the key parameters in the grouping strategy, and ensure that users can focus on the most important parameters to improve the efficiency and accuracy of the experimental design. Number of groups, group names, and multi-dimensional screening schemes, the system recommends the appropriate number of groups to ensure that the experiment is statistically significant; provides group name suggestions to enhance the organization and readability of the experiment; proposes a multi-dimensional screening scheme to help users refine the grouping, and improve the reliability of the experiment and the effectiveness of the results through a reasonable number and name of groups, as well as precise multi-dimensional screening.

[0123] Provide a brief analysis of parameter settings. The system provides a brief analysis of each recommended parameter setting to help users understand the pros and cons of different solutions. Users can quickly grasp the impact of each parameter setting, make more informed decisions, and reduce uncertainty and risk when designing experiments. Help users understand the pros and cons of different solutions. Through analysis, the system helps users understand the possible impact of different grouping strategies on experimental results. Users can make more reasonable experimental design choices based on data and analysis, improve the quality of experimental design and the reliability of the final results. Through this process, the method of the present invention can ensure that users can obtain intelligent recommendations and analysis of AI models in the experimental grouping stage, thereby designing a more scientific and reasonable experimental grouping strategy. This method is particularly suitable for scenarios that require fine grouping to explore different user behaviors or product characteristics, and can significantly improve the accuracy and business value of the experiment.

[0124] Furthermore, in step S3, the sub-observation indicator recommendation: the system combines historical experimental data and large-scale language AI model decisions to automatically screen and recommend the most suitable observation indicators, including core indicators and secondary indicators, to ensure the comprehensiveness and scientificity of data analysis;

[0125] For observation indicator recommendations, the system uses historical experimental data and the decision-making capabilities of large-scale language AI models to automatically screen out the most suitable indicators for observation. By combining the analysis of historical data and AI models, the system can recommend a comprehensive indicator system that includes both core indicators and secondary indicators, ensuring that the experiment can comprehensively evaluate different aspects of business goals.

[0126] Experimental plan formulation in step S3: The system automatically recommends the start and end time of the experiment based on the entered experimental information and historical experimental data, taking into account grouping conflicts and historical performance factors of similar experiments, and provides detailed recommendation reasons and AI analysis reports;

[0127] Experimental plan formulation: The system automatically recommends the start and end time of the experiment based on the entered experimental information and historical experimental data; reasonable experimental time arrangement can ensure that the experiment is carried out at the best time, improving the responsiveness and effectiveness of the experiment. Considering grouping conflicts and historical performance factors of similar experiments, when formulating experimental plans, the system will consider potential grouping conflicts and interference from other experiments, as well as the historical performance of similar experiments. By comprehensively considering these factors, the system can avoid or reduce deviations and errors in the experiment and improve the reliability and effectiveness of the experimental results.

[0128] Provide detailed recommendation reasons and AI analysis reports: The system provides detailed recommendation reasons and AI analysis reports for the recommended experimental plans to help users understand the logic behind the recommendations. The detailed analysis reports can help users better understand the choice of experimental plans and increase users' trust in the experimental results.

[0129] Through this process, the method of the present invention can ensure that the design and execution of AB experiments can be based on data and intelligent analysis, thereby improving the scientificity and effectiveness of the experiment. This method is particularly suitable for scenarios that require accurate data support to guide business decisions, and can significantly improve the value of the experiment and its impact on the business.

[0130] Furthermore, the automatic effect estimation in step S3 includes the following specific steps:

[0131] Group result evaluation: Before the experiment begins, the system uses Python scripts or other automated tools to splice and process data according to the settings of each experimental group, and generates real data for each group in real time to ensure that the group meets the experimental requirements; ensure that the data of each group is accurate and meets the expectations of the experimental design, and provide a reliable data basis for subsequent experimental analysis. Real-time data generation improves the efficiency of experimental preparation, reduces human errors, and ensures the consistency and comparability of experimental data.

[0132] Distribution test evaluation: After completing the grouping result evaluation, the system will simulate and estimate the experimental data based on the existing grouping scheme and observation indicators, and combined with the experimental plan (such as experimental duration, start time, etc.). Through simulation estimation, the system can evaluate the data distribution before the experiment starts, predict the potential results of the experiment, and help users adjust the experimental strategy. The system considers factors such as experimental duration and start time to conduct a comprehensive simulation estimate of the experimental data. Comprehensive data simulation estimation can help users understand the possible performance of the experiment under different conditions and provide predictive support for the smooth progress of the experiment.

[0133] Through this process, the method of the present invention can ensure that the experimental results can be effectively estimated before the AB experiment begins, helping users to make more informed experimental designs and decisions. This method is particularly suitable for scenarios that require accurate prediction and evaluation of experimental results, and can significantly improve the efficiency and effectiveness of the experiment.

[0134] Furthermore, during the automatic effect estimation in step S3, the specific process of using statistical and machine learning techniques to perform data distribution inspection and effect estimation is as follows:

[0135] S301, data extraction and distribution verification: The system first extracts sample data from the database and log system data sources through the ETL (Extract, Transform, Load) process to ensure the representativeness of the data;

[0136] Use stratified sampling or random sampling techniques, combined with the Shapiro-Wilk test or the Kolmogorov-Smirnov test, to check whether the data conform to normal distribution;

[0137] If the data do not meet the assumptions, the system will automatically apply normalization techniques or nonparametric statistical methods, such as logarithmic transformation or rank sum test;

[0138] S302, Variance Analysis and Effect Prediction: The system uses a variety of statistical techniques to estimate the potential effects of the experiment, including:

[0139] T test: used to analyze the significant difference between the experimental group and the control group, and calculate the t value and p value;

[0140] Analysis of variance (ANOVA): used to analyze mean differences among multiple groups, especially suitable for complex multi-factor experiments;

[0141] Monte Carlo simulation: simulate the experimental process through a large number of random sampling to estimate the potential experimental effect and confidence interval;

[0142] Learn and analyze historical data through machine learning algorithms (such as regression models and time series prediction models) to predict group differences and change trends;

[0143] S303, significance test and P value calculation: the system calculates the P value based on the grouped data to measure the significance of the experimental results;

[0144] If the P value is too large, the system will provide optimization suggestions, including adjusting the experimental duration, sample size or grouping strategy;

[0145] Large language AI models will provide adjustment suggestions or experimental reconstruction options based on historical data analysis to ensure the scientificity and effectiveness of experimental design;

[0146] S304, data reporting and visualization: The system presents the evaluation results through visual reports to help users intuitively understand the various data indicators of the experiment. These reports include group distribution diagrams, significant difference analysis diagrams, simulation effect estimation diagrams, and risk assessment reports;

[0147] By integrating multiple technologies such as ETL process, distribution test, statistical and machine learning models, Monte Carlo simulation, etc., this system can not only accurately predict the experimental results, but also provide real-time risk analysis and dynamic optimization suggestions to ensure the scientific nature and operability of the experiment.

[0148] Furthermore, in the result analysis in step S3, the following steps are included:

[0149] Result evaluation: The system will use a large-scale language AI model to conduct a comprehensive analysis of the experimental observation indicators and evaluate the overall performance of the experiment; through historical data and AI algorithms, the significance level of each observation indicator (such as p-value) is calculated, and the impact of the experiment on core business indicators (such as GMV, user retention, etc.) is analyzed; the system will conduct an in-depth evaluation of the feasibility and business value of the experiment, and predict the long-term effects of the experimental results in actual scenarios; through in-depth analysis, the system can provide a comprehensive evaluation of the experimental results, including predictions of significance levels and long-term effects, thereby providing solid data support for decision makers.

[0150] Business strategy proposal recommendations: Based on the experimental results and in-depth analysis of large-scale language AI models, the system will make recommendations on business strategies; by analyzing the execution effect of the current strategy, the large-scale language AI model can identify potential optimization opportunities and propose feasible adjustment plans, such as optimizing marketing promotion strategies, improving user experience, etc.; the system will predict the potential impact of these strategies on business operations to ensure that the decisions are commercially viable. These recommendations will help improve existing marketing strategies and user experience, improve the effectiveness of business operations, and ensure that the decisions are commercially viable.

[0151] Experiment optimization suggestions: The system analyzes the specific execution of the experiment and uses a large language AI model to provide further experimental optimization suggestions, including:

[0152] Suggestions on adjusting the experimental grouping strategy, optimizing the number of groups or redesigning the grouping dimensions;

[0153] Provide suggestions for optimizing screening conditions to ensure the representativeness of samples and the accuracy of experimental results;

[0154] And suggestions for introducing more observation indicators are provided to expand the depth of experimental evaluation and improve the comprehensiveness and reference value of experimental results.

[0155] Through this process, the method of the present invention can ensure that AB experiments can not only provide immediate effect evaluation, but also propose specific business strategies and experiment optimization suggestions based on in-depth analysis. This method is particularly suitable for scenarios that require accurate data support to guide business decisions and optimize strategies, and can significantly improve the value of experiments and their impact on business.

[0156] It can be seen from the above embodiments that the present invention utilizes a large-scale language AI model through training and fine-tuning to automatically analyze and generate experimental design plans. The fine-tuning process enables the large-scale language AI model to adapt to specific business scenarios and accurately identify key factors that affect user behavior, thereby providing intelligent suggestions for experimental design.

[0157] In addition, the present invention combines the use of local vector databases, which support efficient storage and retrieval of high-dimensional vectors, and can quickly process large-scale user behavior data, thereby accelerating the feedback cycle of experimental results. The introduction of vector databases enables the system to perform data analysis with lower latency and higher accuracy, effectively supporting the dynamic adjustment of AB testing.

[0158] In terms of supported large language models, the current open source head large models include GPT-2, BERT and T5 in Hugging Face's Transformers library. These models have powerful natural language processing capabilities and can adapt to specific application scenarios through prompt engineering and fine-tuning. In addition, commercial API interfaces such as OpenAI and Baidu Wenxin Yiyan platforms all provide fine-tuning and data vectorization interfaces, which can provide support for the present invention.

[0159] Improve the level of experimental automation: This invention realizes the full process automation of AB experiments by combining the AI ​​big model and the local vector database, reduces manual operations, and improves the efficiency and accuracy of the experiments.

[0160] Optimize experimental design: Using AI models to match business scenarios and recommend grouping strategies, the system can intelligently generate experimental templates to ensure that experimental design is closely linked to business objectives.

[0161] Improve the depth and breadth of data analysis: The system uses statistical and machine learning techniques to test data distribution and estimate effects, providing comprehensive data analysis, including significance testing and long-term effect prediction.

[0162] Real-time effect prediction and risk assessment: The automatic effect prediction function allows potential results to be predicted before the experiment begins, providing real-time risk analysis and optimization suggestions to help users adjust experimental strategies in a timely manner.

[0163] Enhance the interpretability of experimental results: Through visual reports and data reports, the system converts complex data analysis results into intuitive graphics, allowing non-technical users to easily understand the key indicators of the experiment.

[0164] Improve the scientificity and operability of experiments: The detailed recommendation reasons and AI analysis reports provided by the system make the experimental design more scientific, while improving the operability of the experiments and making the experimental results easier for the business team to understand and apply.

[0165] Adaptability to complex experimental scenarios: The method of the present invention is applicable to various complex experimental scenarios, including multi-dimensional screening and multi-group comparison, can process large-scale data sets, and is suitable for enterprises of different sizes and types.

[0166] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. All equivalent structural changes made by using the contents of the present invention specification and drawings under the inventive concept of the present invention, or directly / indirectly applied in other related technical fields are included in the patent protection scope of the present invention.

Claims

1. A method for generating and optimizing AB experiments based on an AI model and a local vector database, characterized in that: The following steps are involved: Step S1: Build a local experimental database including business knowledge base, data mart and historical experimental data; Step S2: vectorize the business knowledge base, data mart and historical experimental data, and store the generated vectors in the local vector database; Step S3: Use a large language AI model to perform prompt engineering to automate the entire process of AB experiments, including automatic experiment creation, automatic effect estimation, and automatic result analysis; The automatic creation of experiments includes business scenario matching, grouping strategy recommendation, observation indicator recommendation, and experimental plan formulation; Automatic effect estimation includes grouping result evaluation and distribution test evaluation, using statistical and machine learning techniques to perform data distribution tests and effect estimation; Automatic result analysis includes result evaluation, business strategy proposals, and experiment optimization proposals. It uses a large AI model to comprehensively analyze the experimental results and evaluate the overall performance and business value of the experiment.

2. The AB experiment generation and optimization method based on the AI ​​model and the local vector database according to claim 1 is characterized in that: In step S1, the data mart obtains business data by connecting to traditional databases or other data sources. Traditional databases include the advertising data system ADS, and other data sources include the service layer API.

3. The AB experiment generation and optimization method based on AI model and local vector database according to claim 1 is characterized in that: The specific steps of the vectorization process in step S2 are as follows: Step S201: selecting a vectorization technology and using a pre-trained deep learning model to capture semantic information in text data; Step S202: data preprocessing, including data cleaning, format standardization, and feature extraction; in Data cleaning: used to remove duplicate, redundant or irrelevant data to ensure the purity of the data set; Format standardization: used to unify data from different sources into the same format for batch processing; Feature extraction: used to identify features relevant to the experiment for better vectorization; Step S203: Select a vectorization model and convert the processed data: input the text content of the business knowledge base, data mart and historical experimental data into the model, the model generates the corresponding vector representation, and combines the context information to ensure that the vector can reflect the actual meaning of the data.

4. The AB experiment generation and optimization method based on the AI ​​model and the local vector database according to claim 3 is characterized in that: In step S2, the data in the vectorization processing stage is vectorized using a large language AI model to vectorize the information of multiple files. The specific process is as follows: Step S204: Data preparation: collect business knowledge base, data mart and historical experimental data into folders and integrate them into input for vectorization processing; Step S205: Read the folder contents: use Python to read all the file contents in the folder; Step S206: text input and model call: use OpenAI's API to call the selected large language AI model for vectorization processing, input the file content into the model, and generate a vector representation; Step S207: vector generation: the large language AI model generates a high-dimensional vector based on the input text, reflecting the semantics and contextual information of the text; Step S208: storing the vector: storing the generated vector in a local vector database for subsequent retrieval and use.

5. The AB experiment generation and optimization method based on AI model and local vector database according to claim 1 is characterized in that: In the early stage of experiment creation, the system automatically matches the most suitable experimental business scenario based on the experiment name entered by the user through semantic analysis and intent recognition by a large language AI model in step S3. It also recommends corresponding experimental template parameters to the user based on industry experience and historical data. These parameters include basic experimental settings, grouping strategies, and observation indicators. The selected experimental template will be used as a priority option for subsequent experimental strategy recommendations to ensure that the grouping design, observation indicators, etc. are highly matched with the experimental intent.

6. The AB experiment generation and optimization method based on AI model and local vector database according to claim 5 is characterized in that: Grouping strategy recommendation in step S3: During the experimental grouping stage, the system intelligently highlights the key parameter settings in the grouping based on the large-scale language AI model, including the number of groups, group names, and multi-dimensional screening schemes, and provides a brief analysis of the parameter settings to help users understand the pros and cons of different schemes.

7. The AB experiment generation and optimization method based on AI model and local vector database according to claim 6 is characterized in that: Recommendation of sub-observation indicators in step S3: The system combines historical experimental data and large-scale language AI model decisions to automatically screen and recommend the most suitable observation indicators, including core indicators and secondary indicators, to ensure the comprehensiveness and scientificity of data analysis; Experimental plan formulation in step S3: The system automatically recommends the start and end time of the experiment based on the entered experimental information and historical experimental data, taking into account grouping conflicts and historical performance factors of similar experiments, and provides detailed recommendation reasons and AI analysis reports.

8. The AB experiment generation and optimization method based on AI model and local vector database according to claim 7 is characterized in that: The automatic effect estimation in step S3 includes the following specific steps: Group result evaluation: Before the experiment begins, the system uses Python scripts or other automated tools to splice and process data according to the settings of each experimental group, and generates real data for each group in real time to ensure that the group meets the experimental requirements; Distribution test evaluation: After completing the grouping result evaluation, the system will simulate and estimate the experimental data based on the existing grouping scheme and observation indicators, combined with the experimental plan.

9. The AB experiment generation and optimization method based on AI model and local vector database according to claim 8, characterized in that: During the automatic effect estimation in step S3, the specific process of using statistical and machine learning techniques to perform data distribution inspection and effect estimation is as follows: S301, data extraction and distribution verification: The system first extracts sample data from the database and log system data sources through the ETL process to ensure the representativeness of the data; Use stratified sampling or random sampling techniques, combined with the Shapiro-Wilk test or the Kolmogorov-Smirnov test, to check whether the data conform to normal distribution; If the data do not meet the assumptions, the system will automatically apply normalization techniques or nonparametric statistical methods, such as logarithmic transformation or rank sum test; S302, Variance Analysis and Effect Prediction: The system uses a variety of statistical techniques to estimate the potential effects of the experiment, including: T test: used to analyze the significant difference between the experimental group and the control group, and calculate the t value and p value; ANOVA: used to analyze mean differences among multiple groups; Monte Carlo simulation: simulate the experimental process through a large number of random sampling to estimate the potential experimental effect and confidence interval; Use machine learning algorithms to learn and analyze historical data to predict group differences and change trends; S303, significance test and P value calculation: the system calculates the P value based on the grouped data to measure the significance of the experimental results; If the P value is too large, the system will provide optimization suggestions, including adjusting the experimental duration, sample size or grouping strategy; Large language AI models will provide adjustment suggestions or experimental reconstruction options based on historical data analysis to ensure the scientificity and effectiveness of experimental design; S304, data reporting and visualization: The system presents the evaluation results through visual reports to help users intuitively understand the various data indicators of the experiment. These reports include group distribution diagrams, significant difference analysis diagrams, simulation effect estimation diagrams and risk assessment reports.

10. The AB experiment generation and optimization method based on AI model and local vector database according to claim 9 is characterized in that: The result analysis in step S3 includes the following steps: Result evaluation: The system will use a large-scale language AI model to conduct a comprehensive analysis of experimental observation indicators and evaluate the overall performance of the experiment. Through historical data and AI algorithms, the significance level of each observation indicator is calculated, and the impact of the experiment on the core business indicators is analyzed. The system will conduct an in-depth evaluation of the feasibility and business value of the experiment, and predict the long-term effect of the experimental results in actual scenarios. Business strategy proposals: Based on the experimental results and in-depth analysis of large-scale language AI models, the system will make suggestions on business strategies; by analyzing the execution effect of the current strategy, the large-scale language AI model can identify potential optimization opportunities and propose feasible adjustment plans; the system will predict the potential impact of these strategies on business operations to ensure that the decisions are commercially viable. Experiment optimization suggestions: The system analyzes the specific execution of the experiment and uses a large language AI model to provide further experimental optimization suggestions, including: Suggestions on adjusting the experimental grouping strategy, optimizing the number of groups or redesigning the grouping dimensions; Provide suggestions for optimizing screening conditions to ensure the representativeness of samples and the accuracy of experimental results; And suggestions for introducing more observation indicators are provided to expand the depth of experimental evaluation and improve the comprehensiveness and reference value of experimental results.

Images