AB experiment report automatic generation method and device, equipment and medium

Abstract

The invention provides an AB experiment report automatic generation method and apparatus, a device and a medium. The method comprises the steps of obtaining an unstructured document bearing a user demand; carrying out analysis and named entity identification on the unstructured document to obtain a standardized screening condition and attention index; generating a database query statement according to the standardized screening condition, the attention index, the business knowledge base and the index definition knowledge base; executing the database query statement, obtaining a query result from the AB experiment database, and performing homogeneity verification and significance analysis on the query result to determine the statistical reliability of the query result; and injecting the query result and the statistical reliability thereof into a template engine to obtain an experimental report. According to the method, end-to-end automation of AB experiment reports is achieved, SQL is generated through intelligent analysis and knowledge base constraint, statistical verification is automatically completed, a report containing insight is generated by fusing a template and LLM, the generation period is compressed to the minute level from several days, and efficiency and accuracy are greatly improved.

Description

Technical Field

[0001] This invention relates to the field of computer application technology, and in particular to a method, apparatus, device, and medium for automatically generating AB experiment reports. Background Technology

[0002] With the rapid iteration of internet businesses, A / B testing has become a core tool for enterprises to evaluate product features, operational strategies, and the effectiveness of algorithm models. A complete A / B testing process ultimately requires the generation of detailed experimental reports to support business decisions. However, with the increasing complexity of experiments and the surge in demand for customized analysis, the traditional report generation model, which relies on fixed indicators and manual operation, can no longer meet the efficiency and flexibility requirements of modern business's rapid response.

[0003] Currently, several A / B testing platforms and automation solutions exist in the industry. For example, the Chinese patent "A / B Test Implementation Method and Device" (Publication No.: CN110990270A) discloses a typical implementation: generating test reports by acquiring test configurations and data and relying on fixed indicator templates built into the platform. While this type of solution achieves automated report output to some extent, its core logic relies on a predefined general indicator library, and the report generation path is fixed. Furthermore, existing technical solutions typically process structured test configuration information directly, without addressing the automated parsing and understanding of unstructured requirement documents (such as Word documents or online collaborative documents) submitted by business personnel.

[0004] These existing technologies have the following main drawbacks: First, they lack the ability to handle unstructured requirements and cannot directly parse business documents, still requiring manual intervention to extract and communicate key screening conditions and customized indicators, resulting in high communication costs; second, the indicator system has poor scalability and cannot support the personalized indicator analysis needs flexibly proposed by business parties that are not pre-built in the platform; finally, the overall level of automation is insufficient, with multiple steps from data querying and SQL writing to statistical verification and report interpretation requiring manual participation, resulting in a lengthy report generation cycle, usually requiring 5 to 7 working days, which seriously affects the timeliness of business decisions. Summary of the Invention

[0005] In view of the above-mentioned deficiencies of the prior art, the present invention provides a method, apparatus, device, and medium for automatically generating AB test reports, so as to solve the technical problem of low automation and intelligence in the generation of AB test reports.

[0006] To achieve the above and other related objectives, this invention provides a method for automatically generating A / B experiment reports, comprising: acquiring an unstructured document carrying user requirements; parsing and named entity recognition of the unstructured document to obtain standardized filtering conditions and key metrics; generating a database query statement based on the standardized filtering conditions and key metrics, as well as a preset business knowledge base and metric definition knowledge base; executing the database query statement to obtain query results from the A / B experiment database, and performing homogeneity verification and significance analysis on the query results to determine their statistical reliability; and injecting the query results and their statistical reliability into a preset template engine to obtain the experiment report.

[0007] In one embodiment of the present invention, parsing and named entity recognition of the unstructured document to obtain standardized screening conditions and attention indicators includes: parsing the unstructured document to obtain screening conditions and attention indicators; and using named entity recognition technology to map the screening conditions and attention indicators to a preset screening condition knowledge base to obtain the standardized screening conditions and attention indicators.

[0008] In one embodiment of the present invention, parsing the unstructured document to obtain filtering conditions and attention indicators includes: determining whether the unstructured document is a dedicated document hosted on an online collaboration platform; if so, calling the API provided by the online collaboration platform to extract the document content to obtain the filtering conditions and attention indicators; if not, using OCR technology to extract the text content of the unstructured document to obtain the filtering conditions and attention indicators.

[0009] In one embodiment of the present invention, a database query statement is generated based on the standardized filtering conditions and attention indicators, as well as a preset business knowledge base and indicator definition knowledge base. This includes: mapping the extracted standardized filtering conditions to the business knowledge base to obtain the corresponding business knowledge definitions, and generating the query conditions for the database query statement; mapping the extracted standardized attention indicators to the indicator definition knowledge base to obtain the corresponding indicator definitions and indicator source tables, and generating the query fields and field source tables for the database query statement; and generating the database query statement based on the query conditions, query fields, and field source tables.

[0010] In one embodiment of the present invention, the database query statement is syntax-validated by an SQL parser, and the database query statement is regenerated if the validation fails.

[0011] In one embodiment of the present invention, executing the database query statement to obtain query results from the AB experiment database, and performing homogeneity verification and significance analysis on the query results to determine their statistical reliability, includes: executing the database query statement using a distributed computing cluster to obtain query results from the AB experiment database, and applying dynamic partitioning and pruning technology to optimize query speed during execution; obtaining homogeneity verification results of AB experiment groups based on AB backtracking calculation; performing significance analysis on the indicator data in the query results, calculating the p-value through T-test or Bayesian inference, and obtaining the significance analysis results.

[0012] In one embodiment of the present invention, the query results and their statistical reliability are injected into a preset template engine to obtain an experimental report, including: rendering the query results and their statistical reliability through an HTML template engine to generate report content containing structured data; inputting the query results and their statistical reliability into a large language model to generate corresponding natural language conclusions and suggestions; and combining the report content containing structured data with the natural language conclusions and suggestions to generate the final experimental report.

[0013] To achieve the above and other related objectives, the present invention also provides an automatic A / B experiment report generation device, comprising: a data acquisition unit for acquiring unstructured documents carrying user requirements; a document parsing unit for parsing and named entity recognition of the unstructured documents to obtain standardized filtering conditions and attention indicators; a statement construction unit for generating database query statements based on the standardized filtering conditions and attention indicators, as well as a preset business knowledge base and indicator definition knowledge base; a statement execution unit for executing the database query statements, obtaining query results from the A / B experiment database, and performing homogeneity verification and significance analysis on the query results to determine their statistical reliability; and a report generation unit for injecting the query results and their statistical reliability into a preset template engine to obtain an experiment report.

[0014] To achieve the above and other related objectives, the present invention also provides an electronic device, including a processor, a memory, and a communication bus; the communication bus is used to connect the processor and the memory; the processor is used to execute a computer program stored in the memory to implement the method provided in any of the above embodiments.

[0015] To achieve the above and other related objectives, the present invention also provides a computer-readable storage medium having a computer program stored thereon, the computer program being used to cause a computer to perform the method provided in any of the above embodiments.

[0016] The beneficial effects of this invention are as follows: This invention proposes an automatic A / B experiment report generation method, apparatus, device, and medium. This method achieves end-to-end full-process automation from unstructured document input to experiment report output, completely changing the traditional work mode that relies on manual communication, manual writing of query statements, and report compilation. Through intelligent parsing and named entity recognition technology, the system can automatically and accurately extract standardized experimental requirements from business documents, effectively avoiding ambiguity and errors that may arise from manual interpretation. On this basis, it dynamically generates database query statements in conjunction with a preset business and indicator knowledge base, and automatically performs statistical homogeneity verification and significance analysis, ensuring the standardization of the data analysis process and the scientific reliability of the conclusions. Finally, through the collaborative work of a template engine and a large language model, the system directly generates a professional report containing data results and natural language conclusions, compressing the traditional report generation cycle of several days to the hour or even minute level, greatly improving efficiency while significantly reducing labor costs and technical barriers. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. The accompanying drawings are incorporated in and constitute a part of this specification, illustrating embodiments consistent with this application, and are used together with the description to explain the principles of this application. Obviously, the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating an automatic AB experiment report generation method according to an embodiment of the present invention; Figure 2 A detailed flowchart of step S102 provided in an embodiment of the present invention; Figure 3 A detailed flowchart of step S103 provided in an embodiment of the present invention; Figure 4 A detailed flowchart of step S104 provided in an embodiment of the present invention; Figure 5 A detailed flowchart of step S105 provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of an automatic AB experiment report generation device provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0019] Explanation of reference numerals in the attached figures: 601, data acquisition unit; 602, document parsing unit; 603, statement construction unit; 604, statement execution unit; 605, report generation unit; 701, processor; 702, memory. Detailed Implementation

[0020] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. It should be noted that, unless otherwise specified, the following embodiments and features can be combined with each other. In addition to the specific methods, equipment, and materials used in the embodiments, based on the knowledge of the prior art and the description of the present invention by those skilled in the art, any prior art methods, equipment, and materials similar to or equivalent to the methods, equipment, and materials in the embodiments of the present invention can be used to implement the present invention.

[0021] It should be understood that the terminology used in the embodiments of this invention is for describing specific implementations and not for limiting the scope of protection of this invention. Unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art.

[0022] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In some embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0023] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functions, and operations that may be implemented in the methods and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0024] Please see Figure 1 , Figure 1An embodiment of the present invention provides a method for automatically generating AB experiment reports, including steps S101 to S105.

[0025] Step S101: Obtain unstructured documents containing user requirements. This step is the starting point of the end-to-end automated process. Its core lies in directly receiving and identifying the original requirement materials provided by the business party, such as Word documents, Excel spreadsheets, or online collaborative documents like Lark. These documents contain experimental backgrounds, screening dimensions (such as specific cities or user groups), and customized indicators (such as new compound growth rates) described in natural language and semi-structured table formats. This design eliminates the requirement form-based and data preprocessing steps that are necessary for system adaptation in traditional processes, achieving direct connection with "raw" requirements in real business scenarios. This lays a solid foundation for subsequent automated parsing and processing, avoiding the risk of information distortion during multiple relays from the source.

[0026] Step S102: Parse and perform named entity recognition on the unstructured documents to obtain standardized filtering criteria and key metrics. This step is the core of intelligently understanding user needs.

[0027] Please see Figure 2 In a specific embodiment of the present invention, step S102 includes steps S201 and S202.

[0028] Step S201: Parse the unstructured document to obtain the filtering criteria and key metrics. In this step, the unstructured document is parsed to accurately extract the content of interest, such as the filtering criteria and key metrics.

[0029] In a specific embodiment of the present invention, step S201 includes: determining whether the unstructured document is a dedicated document hosted on an online collaboration platform; if yes, then calling the API provided by the online collaboration platform to extract the document content and obtain the filtering conditions and attention indicators; if no, then using OCR technology to extract the text content of the unstructured document and obtain the filtering conditions and attention indicators.

[0030] In this embodiment, different processing methods are applied based on the document type. Dedicated documents hosted on online collaboration platforms refer to online documents created, stored, and managed through platforms such as Lark, DingTalk, and WeChat Work. Their formats include, but are not limited to, online documents, online spreadsheets, or online slideshows. The content and structured information (such as cell data and table styles) of these documents are maintained by the platform server and can be securely and properly accessed and read through the standardized application programming interfaces (APIs) provided by the platform. Unlike locally stored static files, these dedicated documents support real-time collaboration and dynamic updates. This invention, by calling its official API, can directly and accurately obtain the latest version of the document structure and text content, thereby avoiding information loss and structural errors that may occur due to file format conversion or offline downloads.

[0031] For dedicated documents not hosted on online collaboration platforms, typically local static files (such as text files, spreadsheets, image files, etc.), OCR technology can be used to extract the text content. Understandably, for certain common document formats (such as Word and Excel), specialized tools or technologies can be used to read their content.

[0032] Step S202: Using named entity recognition technology, the filtering conditions and key metrics are mapped to a pre-defined filtering condition knowledge base to obtain standardized filtering conditions and key metrics. In this step, named entity recognition technology is used to automatically identify and extract key business entities, such as filtering conditions like "city" and "vehicle type," and metric names like "total transaction amount" and "conversion rate." By mapping these to the pre-defined filtering condition knowledge base, diverse business terms are uniformly transformed into standardized tags that the system can recognize. This achieves the conversion from fuzzy, unstructured natural language to precise, structured business instructions, eliminating the subjectivity and error of manual interpretation and providing reliable input for the subsequent automatic generation of accurate data query statements.

[0033] The filtering criteria knowledge base is similar to a thesaurus or an entity standardization mapping table. Step S201 is used to identify text fragments in the original document that may be poorly worded, while step S202 is used to align the identified content to standardized business terms that the system can understand. For example, the content identified in step S201 may be "Beijing", "Beijing Municipality", "Capital", etc. Through the processing in step S202, it is processed into standardized filtering criteria {"field":"city","value":"Beijing"}.

[0034] Step S103: Generate database query statements based on standardized screening criteria and key indicators, as well as a pre-defined business knowledge base and indicator definition knowledge base. This step is responsible for converting standardized business requirements into executable data query instructions. It achieves accurate and automatic translation from business language to machine language, completely replacing the work of data analysts manually writing and debugging SQL statements in the traditional model. This not only greatly improves efficiency but also avoids human error from a mechanism perspective, ensuring the accuracy and consistency of the query.

[0035] Please see Figure 3 In a specific embodiment of the present invention, step S103 includes steps S301 to S303.

[0036] Step S301: Map the extracted standardized filtering conditions to the business knowledge base, obtain the corresponding business knowledge definitions, and generate the query conditions for the database query statement.

[0037] In this step, the business knowledge base stores the value range and mapping table fields of business dimensions (such as city and business line) in a structured manner; then, the standardized filtering conditions are mapped to the business knowledge base to obtain their corresponding fields and value ranges in the database, thereby generating the WHERE conditions for the query.

[0038] For example, the standardized filtering criteria obtained after step S102 are {"field":"city","value":"Beijing"}. The business knowledge base pre-stores the definition of the business dimension "city," including its corresponding database field name "city_name," its data table name "dim_user," and its value "Beijing" stored as 'beijing' in the database. Through this mapping, the system can automatically generate an accurate SQL query fragment: WHERE dim_user.city_name='beijing'. This process ensures that business terms are unambiguously converted into precise database query instructions.

[0039] Step S302: Map the extracted standardized metrics of interest to the metric definition knowledge base to obtain the corresponding metric definitions and metric source tables, and generate the query fields and field source tables for the database query statement. In this step, the standardized metrics of interest are matched with the metric definition knowledge base to obtain their corresponding data source tables, fields, and calculation formulas, thereby generating the SELECT fields and FROM clauses for the query.

[0040] For example, the standardized metric obtained after step S102 is {"metric":"GMV"}. The metric definition knowledge base pre-stores the technical definition of this metric, including its data source table as hive.ods.order_master, the calculated field as payment_amount, and the calculation formula as SUM(payment_amount). Through this mapping, the system can automatically generate a precise SQL query fragment: SELECT SUM(payment_amount) AS GMV FROM hive.ods.order_master. This process achieves a precise and automated conversion from business metrics to database query elements.

[0041] Step S303: Generate a database query statement based on the query conditions, query fields, and field source tables of the database query statement. This step intelligently integrates the query fields, field source tables, and query conditions generated in the previous steps, and uses the code generation capabilities of a large language model to automatically assemble a grammatically correct and logically complete SQL query statement. It also automatically supplements necessary association logic and grouping conditions to form a standard query instruction that can be submitted to the distributed database for execution.

[0042] In a specific embodiment of the present invention, step S103 further includes step S304: performing syntax validation on the database query statement using an SQL parser, and regenerating the database query statement if validation fails. The SQL parser can be, for example, Apache Calcite, etc., these professional SQL parsers can perform syntax parsing and validation on the generated statement. If validation fails, the system uses the original statement, error message, and corrected example to form a new prompt context, triggering the large language model to perform secondary generation and correction. This fallback mechanism forms a self-improving closed loop, significantly improving the first-time success rate of SQL generation and the accuracy of the final output, ensuring the robustness of the automated process.

[0043] Step S104: Execute the database query statement, obtain the query results from the AB experimental database, and perform homogeneity verification and significance analysis on the query results to determine their statistical reliability.

[0044] Please see Figure 4 In a specific embodiment of the present invention, step S104 includes steps S401 to S403.

[0045] Step S401: Execute database query statements using a distributed computing cluster, retrieve query results from the AB experimental database, and apply dynamic partition pruning technology to optimize query speed during execution. In this embodiment, the database query statements (which may be multiple) generated in step S103 are submitted as a single query task to a distributed computing cluster managed by Kubernetes (k8s). k8s is responsible for scheduling and managing computing resources to ensure that a large number of queries can run efficiently and stably in parallel. The system intelligently applies technologies such as dynamic partition pruning. For example, if the experimental time range is from 2024-01-01 to 2024-01-07, the generated SQL will automatically add a condition such as WHERE dt BETWEEN '2024-01-01' AND '2024-01-07'. In this way, Hive will only read partition data for these 7 days when scanning data, greatly reducing the amount of data scanned and improving query speed.

[0046] Step S402: Based on AB backtracking calculation, obtain the homogeneity verification results of the AB experimental groups. This step ensures that the users / traffic of the experimental group and the control group are essentially similar, thus guaranteeing that the differences observed later are indeed caused by experimental modifications, rather than initial biases. For example, if a feature is shown to "new users" and "old users," their behaviors are inherently different, and comparing their conversion rates is meaningless. Homogeneity verification aims to avoid this situation.

[0047] In this step, the system can automatically query the distribution of experimental groups across key dimensions (such as user gender, age, and region) and perform statistical tests to see if the distributions are consistent; alternatively, it can directly integrate the logic of a mature A / B testing framework (such as Facebook PlanOut) to complete this verification. Step S403: Perform significance analysis on the indicator data in the query results, calculate the p-value using a t-test or Bayesian inference, and obtain the significance analysis results. Significance analysis is responsible for performing statistical inference on the retrieved data to answer the core business question: "Is the experimental effect significant?"

[0048] In this step, the T-test is often used to compare whether there is a significant difference between the means of two continuous values ​​(such as GMV or average duration per person), and it calculates a p-value. Bayesian inference is another statistical framework that, in addition to calculating the p-value, can provide intuitive conclusions such as "the experimental group has an 85% probability of being better than the control group." A small p-value indicates that the difference between the two groups is not caused by random fluctuations and is statistically significant. A large p-value indicates that the observed difference cannot be ruled out as random coincidence, and the conclusion is not significant.

[0049] Step S401 retrieves the query results, providing an accurate data foundation for subsequent analysis. Steps S402 and S403, on the other hand, involve deeper analysis, rigorously validating the data from two dimensions: the rationality of the experimental setup and the validity of the results. Homogeneity verification ensures the comparability of the experimental and control groups before the experiment, eliminating biases in conclusions caused by initial differences. Significance analysis quantitatively assesses whether the differences in effects brought about by the experimental strategy far exceed the range of random fluctuations. These two steps together constitute a double guarantee of the scientific validity of the experimental conclusions, ensuring that the final report not only presents the data but also reveals the reliable statistical inferences behind it.

[0050] Step S105: Inject the query results and their statistical reliability into the preset template engine to obtain the experiment report.

[0051] Please see Figure 5 In a specific embodiment of the present invention, step S105 includes step S501.

[0052] Step S501: Render the query results and their statistical reliability using an HTML template engine to generate a report containing structured data. In this step, the system pre-designs the overall format and layout of the report (HTML template), reserving placeholders for dynamically filling in data. The results obtained in previous steps are automatically injected into the corresponding placeholders in the template, thus quickly and accurately generating a standardized report draft containing all key data. For example, the placeholder in the template: The conversion rate of experimental group A is: {{conversion_rate_A}}; after filling: The conversion rate of experimental group A is: 15.2%.

[0053] In step S105, the standardized report draft can be directly output as the final experimental report. To further enhance the persuasiveness of the experimental report, in another specific embodiment of the present invention, step S105 also includes steps S502 and S503.

[0054] Step S502: Input the query results and their statistical reliability into the large language model to generate corresponding natural language conclusions and suggestions. In this step, the large language model can be regarded as a senior data analyst, interpreting and summarizing the data and providing business suggestions.

[0055] For example, the draft report obtained after step S501 states, "The GMV of Experimental Group B increased by 12.3%, and the P-value is 0.02 (less than 0.05), indicating that the increase is statistically significant." After processing with the large language model, the conclusion can be, "This significant increase may indicate that the new recommendation algorithm better matches user preferences, leading to higher transaction amounts." The report also suggests, "It is recommended to fully implement the strategy for Experimental Group B. Furthermore, the differentiated effects of this strategy in different user segments (such as new and existing users) can be further analyzed to optimize iterations." This step ensures the report's "intelligence" and "excellence," providing expert-level interpretation so that the report is no longer just a collection of cold numbers, but rather provides insights directly serving decision-making.

[0056] Step S503: Combine the report content containing structured data with natural language conclusions and recommendations to generate the final experiment report. In this step, all content obtained in steps S501 and S502 is integrated to generate the final experiment report. The experiment report can be rendered in multiple formats to suit different usage scenarios. For example, HTML format: suitable for direct viewing and interaction in a browser, and can be integrated into an internal management backend; PDF format: suitable for archiving, sending via email, and formal reporting, with a fixed format and good compatibility; Markdown format: suitable for internal communication within the technical team or for subsequent document processing.

[0057] It should be noted that the steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they contain the same logical relationship, they are all within the scope of protection of this application. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this patent.

[0058] Please see Figure 6 , Figure 6An embodiment of the present invention provides an automatic A / B experiment report generation device, comprising a data acquisition unit 601, a document parsing unit 602, a statement construction unit 603, a statement execution unit 604, and a report generation unit 605. The data acquisition unit 601 acquires unstructured documents carrying user requirements; the document parsing unit 602 parses the unstructured documents and performs named entity recognition to obtain standardized filtering conditions and key indicators; the statement construction unit 603 generates database query statements based on the standardized filtering conditions and key indicators, as well as a preset business knowledge base and indicator definition knowledge base; the statement execution unit 604 executes the database query statements, obtains query results from the A / B experiment database, and performs homogeneity verification and significance analysis on the query results to determine their statistical reliability; the report generation unit 605 injects the query results and their statistical reliability into a preset template engine to obtain the experiment report.

[0059] It should be noted that the AB experiment report automatic generation device of this embodiment is a device corresponding to the AB experiment report automatic generation method described above. The functional modules in the AB experiment report automatic generation device may correspond to the corresponding steps in the AB experiment report automatic generation method. The AB experiment report automatic generation device of this embodiment can be implemented in conjunction with the AB experiment report automatic generation method. That is, where there is no conflict, the relevant technical details mentioned in the AB experiment report automatic generation method of the above embodiment can also be applied to the AB experiment report automatic generation device of this embodiment.

[0060] Please see Figure 7 , Figure 7 An electronic device provided in one embodiment of the present invention includes a processor 701, a memory 702, and a communication bus; the communication bus is used to connect the processor 701 and the memory 702; the processor 701 is used to execute the computer program stored in the memory 702 to realize the above-mentioned automatic generation method of AB experiment report.

[0061] The aforementioned electronic device is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0062] The aforementioned electronic devices can be any electronic product capable of human-computer interaction with users, such as personal computers, tablets, smartphones, personal digital assistants (PDAs), game consoles, interactive network television (IPTV), smart wearable devices, etc.

[0063] The aforementioned electronic devices may also include network devices and / or user devices. Among them, network devices include, but are not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0064] The networks in which the aforementioned electronic devices are located include, but are not limited to, the Internet, wide area networks, metropolitan area networks, local area networks, and virtual private networks (VPNs).

[0065] The processors mentioned above can be, for example, general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components; the memory mentioned above may include random access memory (RAM), and may also include non-volatile memory, such as at least one disk storage device.

[0066] An embodiment of the present invention also provides a computer-readable storage medium having a computer program stored thereon, the computer program being used to cause a computer to execute the above-described method for automatically generating AB experiment reports.

[0067] In summary, the automatic generation method for AB experiment reports in this invention has many advantages: (1) Efficiency leap: The report generation cycle is compressed from 5-7 days to hours or even minutes, and the manual input is reduced by more than 90%; (2) Accuracy optimization: Through the NL2SQL conversion with knowledge base constraints, the success rate of SQL generation is increased to 95% (compared to 70% for unconstrained models); (3) Enhanced business scalability: It supports the analysis needs of non-general indicators across industries and multiple scenarios.

[0068] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. An AB experiment report automatic generation method, characterized in that, The method comprises the following steps: obtaining an unstructured document carrying user requirements; parsing and named entity recognition are performed on the unstructured document to obtain standardized screening conditions and attention indicators; generating a database query statement according to the standardized screening conditions and attention indicators, and a preset business knowledge base and an indicator definition knowledge base; executing the database query statement to obtain query results from an AB experiment database, and performing homogeneity verification and significance analysis on the query results to determine their statistical reliability; injecting the query results and their statistical reliability into a preset template engine to obtain an experiment report.

2. The AB experimental report automatic generation method according to claim 1, characterized in that, The parsing and named entity recognition are performed on the unstructured document to obtain standardized screening conditions and attention indicators, which comprises the following steps: parsing the unstructured document to obtain screening conditions and attention indicators; using named entity recognition technology, mapping the screening conditions and the attention indicators to a preset screening condition knowledge base to obtain the standardized screening conditions and attention indicators.

3. The AB experimental report automatic generation method according to claim 2, characterized in that, The parsing of the unstructured document to obtain screening conditions and attention indicators comprises the following steps: determining whether the unstructured document is a special document hosted on an online collaboration platform: if yes, calling an API provided by the online collaboration platform to extract document content to obtain the screening conditions and the attention indicators; if no, using OCR technology to extract the text content of the unstructured document to obtain the screening conditions and the attention indicators.

4. The AB experimental report automatic generation method according to claim 1, characterized in that, The generation of the database query statement according to the standardized screening conditions and attention indicators, and a preset business knowledge base and an indicator definition knowledge base comprises the following steps: mapping the extracted standardized screening conditions to the business knowledge base to obtain the business knowledge definition of the corresponding conditions, and generating the query conditions of the database query statement; mapping the extracted standardized attention indicators to the indicator definition knowledge base to obtain the corresponding indicator definition and indicator source table, and generating the query fields and field source table of the database query statement; generating the database query statement according to the query conditions, query fields and field source table of the database query statement.

5. The AB experimental report automatic generation method according to claim 4, characterized in that, It also comprises the following steps: performing syntax verification on the database query statement by a SQL parser, and regenerating the database query statement when the verification fails.

6. The AB experimental report automatic generation method according to claim 1, wherein, The execution of the database query statement to obtain query results from an AB experiment database, and the performance of homogeneity verification and significance analysis on the query results to determine their statistical reliability comprises the following steps: using a distributed computing cluster to execute the database query statement to obtain query results from an AB experiment database, and applying dynamic partition pruning technology to optimize query speed during execution; based on AB backtracking calculation, obtaining homogeneity verification results of AB experiment groups; performing significance analysis on the indicator data in the query results, calculating P value through T test or Bayesian inference to obtain significance analysis results.

7. The AB experimental report automatic generation method according to claim 1, characterized in that, The injection of the query results and their statistical reliability into a preset template engine to obtain an experiment report comprises the following steps: rendering the query results and their statistical reliability through an HTML template engine to generate report content containing structured data; The query results and their statistical reliability are input into the large language model to generate corresponding natural language conclusions and suggestions. The final experimental report is generated by combining the report content containing structured data with the natural language conclusions and recommendations.

8. An AB experiment report automatic generation device characterized by comprising: include: The data acquisition unit is used to acquire unstructured documents that meet user requirements. The document parsing unit is used to parse the unstructured document and perform named entity recognition to obtain standardized screening conditions and attention indicators; The statement construction unit is used to generate database query statements based on the standardized filtering conditions and focus indicators, as well as the preset business knowledge base and indicator definition knowledge base. The statement execution unit is used to execute the database query statement, obtain the query results from the AB experimental database, and perform homogeneity verification and significance analysis on the query results to determine their statistical reliability. as well as The report generation unit is used to inject the query results and their statistical reliability into a preset template engine to obtain an experiment report.

9. An electronic device, comprising: It includes a processor, a memory, and a communication bus; the communication bus is used to connect the processor and the memory; the processor is used to execute a computer program stored in the memory to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program thereon, the computer program being used to cause the computer to perform the method as described in any one of claims 1 to 7.

Images