AB experiment method, device, system, equipment, storage medium and product

Abstract

The invention relates to the technical field of software testing, and discloses an AB experiment method, device, system and equipment, a storage medium and a product. According to the scheme, the AB experiment rule can be obtained based on the current call application program, the AB experiment rule comprises the shunting rule and the experiment configuration, the communication connection with the interaction object is established, shunting is carried out based on the shunting rule, the shunting result is obtained, and the experiment data is reported according to the experiment configuration and the shunting result after the communication is finished, so that the AB experiment is completed. According to the scheme provided by the embodiment of the invention, the AB experiment rule can be issued to the client in advance, the AB experiment rule is executed on site when the service meets the condition, the shunting result is quickly obtained, the AB experiment shunting efficiency and the subsequent experiment accuracy are remarkably improved, in addition, the shunting rule needing to be used does not need to be supported by an experiment platform and only needs to be supported by the client, and the workload of the client is reduced. And the method also has better expansibility.

Description

Technical Field

[0001] This application relates to the field of software testing technology, specifically to an A / B testing method, apparatus, system, equipment, storage medium, and product. Background Technology

[0002] When a new feature is introduced to a client, A / B testing (comparative experiment) is needed to measure the differences between the new feature and existing features in terms of benefits and impact. Common A / B testing methods involve configuring rules on an A / B testing platform. When the client retrieves the test configuration, the platform uses preset traffic distribution attributes (such as UID, DeviceID, etc.) to determine whether the test rules are met. If the rules are met, an A / B flag is sent to the client, indicating that the test feature has been applied. Then, the test data is reported, aggregated, and compared with existing features to draw conclusions. This experimental method is widely used in internet products for product optimization.

[0003] In practical use, the applicant found that in certain special call scenarios, such as calls where the called client involves multiple devices or group calls, A / B testing often involves pairing the calling and called clients after the call is established, pulling the test switch from the A / B testing platform, and then activating the corresponding test feature. However, this results in excessively long call setup times, affecting the final accuracy of the experiment. Furthermore, since the test function can only be activated after pulling the switch, the loading of the corresponding functional modules and resource allocation can only be delayed, extending the call processing time and potentially further affecting the accuracy of the experimental results. Summary of the Invention

[0004] This application provides an A / B testing method, apparatus, system, device, storage medium, and product that can pre-deploy A / B testing rules to the client and execute them locally when the business conditions are met, quickly obtaining the traffic splitting results and significantly improving the efficiency of A / B testing traffic splitting and the accuracy of subsequent experiments.

[0005] This application provides an A / B testing method applied to a client, including: The A / B experiment rules are obtained based on the current call application, and the A / B experiment rules include traffic splitting rules and experiment configurations. Establish a communication connection with the interactive object and perform traffic splitting based on the aforementioned traffic splitting rules to obtain the traffic splitting result; After the communication ends, experimental data is reported according to the experimental configuration and the splitting results to complete the AB experiment.

[0006] In one embodiment, before performing traffic splitting based on the splitting rules, the method further includes: Receive a communication identifier sent by the server and determine whether the communication identifier meets a preset value; If the conditions are met, then the step of traffic splitting based on the aforementioned splitting rules will be executed.

[0007] In one embodiment, before performing traffic splitting based on the splitting rules, the method further includes: Obtain the client version information of the interaction object; Match the current client version information with the client version information of the interaction object; If a match is found, the step of traffic splitting based on the aforementioned splitting rules will be executed.

[0008] In one embodiment, matching the current client version information with the client version information of the interaction object includes: Analyze the version support list in the experiment configuration; Determine whether the version information of the current client and the client version information of the interaction object are both in the list; If both are present, then the match is successful.

[0009] In one embodiment, obtaining the A / B test rules based on the current call application includes: Determine whether the application has started or switched to the foreground; If so, actively retrieve the AB experiment rule script from the AB experiment platform and store it locally.

[0010] This application also provides an A / B testing method applied to a server, including: When a call is detected to be initiated by a calling client, a communication identifier is generated and a call request carrying the communication identifier is sent to the called client; Receive the call response information from the called client, and establish a communication connection between the calling client and the called client based on the response information; Send a return message carrying the communication identifier to the calling client; The communication identifier is configured to be used as input parameters by the calling client and the called client respectively, and substituted into the same AB experiment rules stored locally for calculation, so as to independently obtain consistent traffic splitting results, thereby executing the AB experiment method described in any one of the embodiments of this application.

[0011] In one embodiment, the method further includes: After detecting that the calling client has initiated a call, the version information of the calling client is extracted and forwarded to the called client; After establishing a communication connection between the calling client and the called client, the version information of the called client is extracted and forwarded to the calling client.

[0012] In one embodiment, the called client includes multiple devices, and the method further includes: After the interaction object establishes a communication connection with the calling client, the interaction object is identified as the target device, and the other devices among the multiple devices of the called client, excluding the target device, are controlled to ignore the call request.

[0013] This application also provides an AB experiment apparatus for use on a client side, including: The acquisition unit is used to acquire AB experiment rules based on the current call application, the AB experiment rules including traffic splitting rules and experiment configuration; The traffic splitting unit is used to establish a communication connection with the interactive object and perform traffic splitting based on the traffic splitting rules to obtain the traffic splitting result. The reporting unit is used to report experimental data according to the experimental configuration and the splitting results after the communication ends, so as to complete the AB experiment.

[0014] This application also provides an AB experimental apparatus, applied to a server, including: The generation unit is used to generate a communication identifier and send a call request carrying the communication identifier to the called client when a call is detected to be initiated by the calling client; A communication unit is used to receive call response information from the called client and establish a communication connection between the calling client and the called client based on the response information. The sending unit is used to send return information carrying the communication identifier to the calling client, wherein the communication identifier is configured to be used as input parameters by the calling client and the called client respectively, and calculated by substituting them into the same AB experiment rules stored locally, so as to independently obtain consistent traffic splitting results, thereby executing the AB experiment method described in any one of the embodiments of this application.

[0015] This application also provides an A / B experiment system, including a client, an A / B experiment platform, a server, and a data analysis platform; The client is used to obtain AB experiment rules from the AB experiment platform based on the current call application. The AB experiment rules include flow splitting rules and experiment configuration. After establishing a communication connection with the interaction object, the client performs flow splitting based on the flow splitting rules to obtain the flow splitting result. After the communication ends, the client reports the experiment data to the data analysis platform according to the experiment configuration and the flow splitting result to complete the AB experiment. The server is configured to generate a communication identifier when a call is detected from a calling client and send a call request carrying the communication identifier to the called client. After receiving the call response information from the called client, the server establishes a communication connection between the calling client and the called client based on the response information and sends a return message carrying the communication identifier to the calling client. The communication identifier is configured to be used as input parameters by the calling client and the called client respectively, and calculated by substituting them into the same A / B test rules stored locally, so as to independently obtain consistent traffic splitting results, thereby executing the A / B test method described in any one of the embodiments of this application. The data analysis platform is used to receive experimental data reported by the client and perform analysis and processing.

[0016] This application embodiment also provides a terminal device, the terminal device including: a memory and a processor, wherein the memory stores an application processing program, and when the application processing program is executed by the processor, it implements the steps of any of the AB test methods provided in this application embodiment.

[0017] This application also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute the steps of any of the AB experimental methods provided in this application.

[0018] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the AB test methods provided in this application.

[0019] The A / B testing method provided in this application can obtain A / B testing rules based on the current call application. These rules include traffic splitting rules and test configurations. A communication connection is established with the interaction object, and traffic is split based on the rules to obtain the splitting results. After the communication ends, test data is reported according to the test configuration and the splitting results to complete the A / B test. The solution provided in this application can pre-deploy A / B testing rules to the client and execute them locally when the business conditions are met, quickly obtaining splitting results. This significantly improves the efficiency of A / B test traffic splitting and the accuracy of subsequent tests. Furthermore, the splitting rules used do not require platform support; only client support is needed, resulting in better scalability. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the first AB test method provided in the embodiments of this application; Figure 2 This is a schematic diagram of the second AB test method provided in the embodiments of this application; Figure 3 This is a schematic diagram of the third AB test method provided in the embodiments of this application; Figure 4 This is a schematic diagram of the architecture of the AB experimental system provided in the embodiments of this application; Figure 5 This is a schematic diagram of the first structure of the AB experimental apparatus provided in the embodiments of this application; Figure 6 This is a schematic diagram of a second structure of the AB experimental apparatus provided in the embodiments of this application; Figure 7 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application. Detailed Implementation

[0022] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0023] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, components, features, and elements with the same names in different embodiments of this application may have the same meaning or different meanings, the specific meaning of which must be determined by its interpretation in that specific embodiment or further in conjunction with the context of that specific embodiment.

[0024] It should be understood that although the steps in the flowcharts of this application's embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0025] It should be noted that step designations such as S101 and S102 are used in this document for the purpose of more clearly and concisely describing the corresponding content, and do not constitute a substantial limitation on the order. In specific implementation, those skilled in the art may execute S102 first and then S101, etc., but these should all be within the protection scope of this application.

[0026] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0027] In existing technologies, when a client introduces a new feature, it often needs to conduct A / B testing to verify the benefits and impacts of the new feature, thereby ensuring the stability and user experience after the new feature is launched. The typical A / B testing process is as follows: the A / B platform pre-configures the test traffic distribution rules, the client pulls the test configuration at startup or at a specific time, the A / B platform determines whether the client has been selected for the test based on preset traffic distribution attributes such as UID and DeviceID, and then sends an A / B flag to the client. The client then activates the test feature and reports the data accordingly, ultimately completing the comparative analysis of the test results. However, in VoIP call scenarios, there is a special type of A / B test that requires both the calling and called parties to meet specific preconditions before the test can be started. For example, in an experiment introducing H.265 encoding, both the calling and called client versions must support H.265 encoding and decoding capabilities. If only one end starts the test, it will cause failures such as no video output because the other end cannot decode. This type of test cannot be determined by the client pre-pulling the configuration; it requires real-time matching based on the current callers' devices and version information.

[0028] For this type of special A / B test in VoIP call scenarios, there are currently two mainstream implementation solutions in the industry. One is the backend version matching solution, in which the VoIP backend calculates the version information of the calling and called parties during the call request processing stage to determine whether both parties are eligible for the test. If they are, the test start command is directly sent to the calling and called parties. While this solution enables real-time detection of experimental on / off states, it suffers from two major drawbacks: First, the backend requires precise knowledge of the complete support list for each functional module on different platforms and versions, resulting in high coupling between the backend code and the functional list of the client version. As client versions and functions continue to iterate, the experimental judgment logic in the backend becomes increasingly complex, significantly increasing maintenance costs. Furthermore, modifications to the backend code are highly prone to causing compatibility issues, leading to online service failures, which is unsustainable from a software engineering and long-term maintenance perspective. Second, since the same user account can log in to multiple devices (such as PC and mobile), the backend cannot determine the actual receiving device when a call is not answered. It typically selects the most active device version among multiple devices as the matching criterion. However, in real-world scenarios, users may use inactive devices to answer calls. If the caller matches the highly active device version and the experiment is successful, but the actual receiving device does not support the experimental function, it will severely affect the accuracy of the experimental data and the user's call experience.

[0029] The second approach is to retrieve the experimental switch after the called party answers. This approach aims to address the issue of misjudgment of devices in the background version matching. It requires the called party to proactively request the AB platform to obtain the experimental hit status after answering the call, and then activate the corresponding experimental function based on the returned result. However, this approach also has significant drawbacks: First, retrieving the experimental switch after the called party answers significantly increases the overall call setup time, directly reducing the accuracy of the experimental data; second, waiting for the experimental switch to be retrieved before loading functional modules and allocating resources further prolongs the call processing latency, negatively impacting the reliability of the experimental results.

[0030] In summary, existing technologies struggle to achieve a good balance between real-time performance, accuracy, maintainability, and system scalability when addressing the need for multi-party collaborative A / B testing in VoIP scenarios. Therefore, there is an urgent need for a VoIP call A / B testing solution that can achieve low latency, high accuracy, and easy scalability to support complex, real-time, multi-party collaborative testing scenarios.

[0031] This application provides an AB experiment method. The subject executing the AB experiment method can be the AB experiment device provided in this application, or a terminal device (smart terminal, etc.) that integrates the AB experiment device. The AB experiment device can be implemented in hardware or software.

[0032] Specifically, please refer to Figure 1 , Figure 1This is a schematic diagram of the first AB experiment method provided in this application embodiment. The AB experiment method is applied to the client, and the specific process can be as follows: S101. Obtain the AB experiment rules based on the current call application. The AB experiment rules include traffic splitting rules and experiment configuration.

[0033] In one embodiment, the aforementioned clients include a calling client and a called client. A / B testing rules can be uniformly configured, managed, and distributed by an A / B testing platform deployed in the cloud. These rules can be in the form of script strings that the client can directly parse and execute, such as Lua scripts, Shell scripts, or custom structured scripts, ensuring that the calling and called clients can execute locally without relying on additional parsing plugins. During system initialization or application runtime, the client will asynchronously and proactively retrieve currently active A / B testing rules from the A / B testing platform based on the current call application. It should be noted that the called client may also include multiple potentially logged-in devices, such as mobile phones, tablets, smartwatches, and PCs that the user may use to answer calls.

[0034] In one embodiment, the step of obtaining A / B testing rules based on the current call application may include: determining whether the application has started or switched to the foreground; if so, actively obtaining the A / B testing rule script from the A / B testing platform and storing it locally. For example, before the client participates in a VoIP call, the system first completes the preloading of A / B testing rules, that is, when the calling client starts the application or switches to the foreground, it can actively initiate a rule retrieval request to the A / B testing platform. The A / B testing platform returns the configured testing rules to the calling client in the form of an executable script. Similarly, the called client also completes the preloading of testing rules at the corresponding time.

[0035] In one embodiment, the timing for obtaining the AB experiment rules in advance may include the following: When the application starts: After the terminal starts the VoIP call-related application (such as an audio and video call APP), it will automatically initiate a rule retrieval request within 3 seconds after completing basic initialization (such as network connection and account login).

[0036] When the application switches to the foreground: If more than 30 minutes have passed since the last rule fetch after the application switches from the background to the foreground, an incremental fetch is triggered, which only retrieves newly added or updated rules.

[0037] When rules are updated: After the AB experiment platform updates the rules, it will push an update notification to all online terminals. After receiving the notification, the client will immediately pull the latest rules.

[0038] Pre-check: The calling client can check the local rules at preset time intervals. If the local rules are found to be missing or expired (the expiration threshold is set to 24 hours), the client will pull the rules from the A / B test platform again. After the rules are pulled, they will be stored locally to avoid the experiment being unable to be executed due to missing rules.

[0039] In one embodiment, the A / B experiment rules include traffic splitting rules and experiment configurations. For example, the A / B experiment platform allows administrators to configure these two core rules through a visual interface. Specifically, traffic splitting rules are used to determine whether a client is included in the experiment. These rules can be flexibly associated with call-related fields, including but not limited to unique communication identifiers (callid), terminal version numbers, device types (PC / mobile phone / tablet), network types (5G / Wi-Fi / 4G), and user account attributes (UID). For example, the traffic splitting rules can be configured as "callid is even and the terminal supports H265 encoding / decoding" or "network type is Wi-Fi and device type is mobile phone," etc. This embodiment will not elaborate further on these details.

[0040] In one embodiment, the experiment configuration is used to specify the functional parameters corresponding to the experiment. These parameters may include an experiment identifier, a function activation threshold, data reporting fields (such as call setup latency, audio / video stuttering rate, decoding success rate, etc.), and data reporting frequency. For example, the configuration for an H265 encoding experiment might be "Experiment ID: EXP_H265_2024; Activation condition: hit result; Reporting fields: call setup time, video frame rate, packet loss rate," etc.

[0041] In one embodiment, the aforementioned A / B experiment rules can be stored in a distributed database of the A / B experiment platform. The database supports rule version management and canary release, facilitating subsequent rule iteration and rollback. After the client pulls the rules, it can store them in a local encrypted cache. Furthermore, the client can periodically verify the integrity and validity of the locally stored A / B experiment rules. If a rule file is detected as corrupted or inconsistent with the rule version of the A / B experiment platform, a re-fetching can be automatically triggered.

[0042] S102. Establish a communication connection with the interactive object and perform traffic splitting based on the traffic splitting rules to obtain the traffic splitting result.

[0043] In one embodiment, the communication connection can be a VoIP call, and the interaction object is the VoIP call object, i.e., the calling client or the called client. Specifically, if the current client is the calling client, the interaction object is the called client; conversely, if the current client is the called client, the interaction object is the calling client. Further, when initiating a call, the calling client can collect basic call-related information, such as the calling client's version number, device model, current network type, and the called client's account. Then, it sends a call request to the VoIP server, carrying the aforementioned basic information and the rule version number cached locally by the calling client, facilitating the server's verification of rule validity.

[0044] Furthermore, after receiving a call request from a calling client, the VoIP server can first verify the calling client's account status and network connectivity, such as whether it is logged in, has call permissions, and network quality. Once the verification is successful, a unique communication identifier, i.e., a callid, can be generated, for example, callid=123456. Specifically, this unique communication identifier can also be generated using a combination of "timestamp + random number + calling account hash value" to ensure that each call's callid is globally unique and supports uniqueness verification in a distributed environment.

[0045] Next, after generating the callid, the server can initiate call requests to all online devices under that account (i.e., multiple devices of the called client) based on the called client's account login status. The core information carried in the request can include a unique communication identifier, as well as basic information about the calling client and call-related parameters, such as the calling client's version number, device type, network type, call priority, and media stream transmission protocol type. Please refer to [link / reference] for details. Figure 2 Before performing traffic splitting based on the aforementioned splitting rules, the method further includes: S1021. Receive the communication identifier sent by the server and determine whether the communication identifier meets the preset value; S1022. If satisfied, obtain the client version information of the interactive object; S1023. Match the current client version information with the client version information of the interaction object; S1024. If the match is successful, proceed with the step of traffic splitting based on the splitting rules; S1025. If the communication identifier does not meet the preset value or the version information fails to match, the experiment will be terminated.

[0046] In one embodiment, after receiving a call request from the server, the called client displays a call notification on the application interface, such as a ringtone, vibration, or pop-up window. When the user of the called client clicks the "Answer" button, the called client sends an answer response to the server, which may include information such as its version number, device type, and network status. After receiving the answer response from the called client, the server can immediately send a "Called party has answered" notification to the calling client and establish a media stream transmission channel between the calling and called clients, such as based on the SIP or WebRTC protocol, to complete the call setup. At this point, both the calling and called clients have obtained complete call-related information, including call ID, version numbers of both parties, device information, and network information.

[0047] In one embodiment, after a call is established, both the calling and called clients execute cached routing rules locally. Specifically, the client can automatically collect all the field data required by the routing rules, including callid, version numbers of both parties, device type, network type, and a list of supported functions (such as whether H.265 encoding / decoding is supported, and whether HD voice is supported). The version numbers of both parties can be obtained from the call request or answer response forwarded by the server, ensuring data accuracy. Then, the client can call the local rule execution engine to input the collected field data into the routing rule script for logical operations. For example, if the routing rule is "callid%2==0 and both the calling and called clients support H.265 encoding / decoding," the client will first calculate the modulo result of callid, then verify whether the function list corresponding to both versions includes H.265 encoding / decoding capability, ultimately determining whether the routing result is "experiment hit" or "experiment miss." Finally, the routing result is bound to callid and stored locally for subsequent experimental function activation and data reporting.

[0048] In one embodiment, if both the calling and called clients achieve a "hit experiment" result, both ends will automatically activate the corresponding experimental functions according to the functional parameters in the experimental configuration. Specifically, the client can load the modules required for the experimental functions locally, such as an H265 encoding module or a high-definition voice processing module. The loading process can be executed asynchronously in the background without affecting the audio and video transmission of the current call. Then, the functional parameters, such as the bitrate and frame rate of H265 encoding, are set according to the thresholds in the experimental configuration to ensure that the experimental functions run according to the preset scheme. In addition, if the loading of the experimental functions fails during the execution of the experiment, such as due to missing modules or insufficient resources, the system will automatically revert to the original function and record the reason for the failure. This reason will be reported along with the experimental data to ensure that the user's call experience is not affected.

[0049] In one embodiment, the client can also obtain and match the client version information of the interaction object. For example, when the calling client initiates a VoIP call, the server can simultaneously extract the calling client's version information and encapsulate the communication identifier and the calling client version information in a call request, then initiate a call to all online called clients. Considering that a user may log in to the same account on multiple devices simultaneously, the server can initiate call requests to all online devices of the called account. Each called device receives a call request containing the same communication identifier and calling client version information.

[0050] In one embodiment, when a called client device answers a call, it can perform experimental function matching verification. The called client device can extract the version support list from the experimental configuration and determine whether its own client version and the calling client version meet the experimental requirements. The specific matching process includes: the called client device parsing the version support list in the experimental configuration, which specifies the range of all client versions that support the experimental function. Then, it verifies whether its own version and the calling client version are within the support list. Only when both versions meet the requirements is the experimental function matching considered successful.

[0051] Correspondingly, after the called client device answers the call, the server can extract the client version information of the called client device and forward it to the calling client. Upon receiving the response, the calling client executes the same experimental function matching process: extracting the version support list from the experimental configuration, and verifying whether its own version and the called client device version meet the experimental requirements. This involves matching the current client's version information with the client version information of the interaction object. This may include parsing the version support list in the experimental configuration, determining whether both the current client's version information and the interaction object's client version information are in the list, and if both are, then the match is successful. This two-way verification mechanism provided in this embodiment ensures that the preconditions for starting the experiment are fully met, avoiding experimental failures or user experience issues caused by single-end function mismatches.

[0052] In one embodiment, after both the calling client and the called client have completed function matching and passed verification, both parties execute the pre-acquired traffic splitting rules locally. The execution of the traffic splitting rules is calculated based on the communication identifier generated by the server, for example, determining the experimental group through the judgment logic of "callid%2==0". The advantage of this distributed execution mechanism provided in this embodiment is that it avoids the network latency caused by the need to query the A / B experimental platform in real time in traditional solutions, and at the same time ensures that the calling and called parties obtain consistent traffic splitting results through the same communication identifier.

[0053] S103. After the communication ends, report the experimental data according to the experimental configuration and the splitting results to complete the AB experiment.

[0054] In one embodiment, experimental data reporting can adopt a "one-time reporting after call ends" approach, thereby avoiding the consumption of network bandwidth and impact on call quality during data reporting. For example, after the calling or called client clicks the "hang up" button to end the call, the client can initiate the data reporting process within 10 seconds. If the call is terminated due to abnormal circumstances such as network interruption or equipment failure, the client can also automatically report the collected experimental data within 30 seconds after the network connection is restored.

[0055] In one embodiment, the calling and called clients collect and organize experimental data according to the reporting fields specified in the experimental configuration. This experimental data may include basic experimental information such as callid, experiment ID, traffic splitting result (hit / miss), calling and called client version numbers, device type, call start time, and call duration. The experimental data may also include runtime data, such as call setup latency, video frame rate, packet loss rate, and decoding success rate reported for H265 encoding experiments; and voice clarity scores, echo cancellation effects, and noise suppression effects reported for high-definition voice experiments. Furthermore, if abnormal situations such as function loading failure, audio / video stuttering, or disconnection occur during the experiment, the experimental data can also record the time of occurrence, type of abnormality, and error code. The reported data can be encapsulated in JSON format and transmitted to the data reporting platform via HTTPS protocol encryption.

[0056] In one embodiment, after receiving experimental data from the calling and called clients, the data analysis platform can first associate the reported data of the calling and called clients based on the callid to form a complete call experiment record. It can also remove invalid and abnormal data to ensure data accuracy, such as records with format errors or missing data exceeding 30%, and values ​​significantly exceeding the normal range. Finally, the data analysis platform obtains the cleaned experimental data and performs comparative analysis based on preset experimental indicators. For example, it compares the call setup latency, video clarity, and network bandwidth utilization between the hit experimental group (with H265 encoding) and the non-hit control group (using H264 encoding) to draw conclusions about the experimental results. After the analysis is completed, the data analysis platform generates a detailed experimental report, including data statistics charts, indicator comparison results, and experimental conclusion suggestions, for the experiment administrator to view and make decisions, thus completing the entire A / B experiment process.

[0057] The solution provided in this embodiment can also be applied to multi-terminal call scenarios such as multi-person conferences or multi-person live streaming. For example, when the calling client initiates a multi-person group call, the calling client pre-fetches rules, and the server generates a unified callid and distributes it to all group member devices. After each group member answers, they perform version and function matching with the calling client. The experimental function is only enabled in the group call if all members support the experimental function and the traffic splitting rule is hit, ensuring consistency of the experiment in the group call. In addition, when conducting cross-platform calls, the calling client and the called client execute local rules, which can flexibly adapt to the functional differences of different platforms. For example, if the PC supports a certain high-definition transmission protocol but the mobile phone does not, the traffic splitting rule will determine "experiment not hit" and automatically fall back to the compatible function, avoiding cross-platform compatibility issues.

[0058] As described above, the A / B testing method proposed in this application can obtain A / B testing rules based on the current call application. These rules include traffic splitting rules and test configurations. A communication connection is established with the interaction object, and traffic is split based on the rules to obtain the splitting results. After the communication ends, test data is reported according to the test configuration and the splitting results to complete the A / B test. The solution provided in this application can pre-deploy A / B testing rules to the client and execute them locally when the business conditions are met, quickly obtaining the splitting results. This significantly improves the efficiency of A / B test traffic splitting and the accuracy of subsequent tests. Furthermore, the splitting rules used do not require experimental platform support; only client support is needed, resulting in better scalability.

[0059] Please see Figure 3 , Figure 3 This is a schematic diagram of the second type of AB experiment method provided in this application embodiment. The method is applied to the server side and specifically includes: S201. When a call is detected to be initiated by the calling client, a communication identifier is generated and a call request carrying the communication identifier is sent to the called client.

[0060] In one embodiment, if the called client comprises multiple devices, when a call is detected from the calling client, the server can generate a unique communication identifier, i.e., a callid, and initiate call requests carrying the communication identifier to the multiple devices of the called client. For example, based on the account login status of the called client, call requests can be initiated to all online devices under that account, i.e., the multiple devices of the called client. The core information carried in the request may include the unique communication identifier, as well as basic information of the calling client and call-related parameters, such as the calling client's version number, device type, network type, call priority, and media stream transmission protocol type. For example, after detecting that the calling client has initiated a call, the version information of the calling client is extracted and forwarded to the called client.

[0061] S202. Receive the call response information from the called client and establish a communication connection between the calling client and the called client based on the response information.

[0062] In one embodiment, the server receives call response information from the called client, which may include the called client's version number, device type, network status, and other information. The server then sends a "call answered" notification to the calling client and establishes a media stream transmission channel between the calling and called clients, thus completing the call setup.

[0063] S203. Send a return message carrying the communication identifier to the calling client.

[0064] In one embodiment, the communication identifier is configured to be used as an input parameter by both the calling and called clients, and calculated using the same locally stored A / B test rules to independently obtain consistent traffic splitting results, thereby executing the A / B test method. Specifically, after the call is established, the server can send a return message carrying the communication identifier to the calling client, so that the calling and called clients can sequentially verify the communication identifier and the version information at both ends, and after verification, execute the cached traffic splitting rules and conduct the A / B test locally.

[0065] It should be noted that if the called client includes multiple devices, the called client device that establishes the call can be defined as the target device. After the target device establishes a communication connection with the calling client, the server can also control the other devices among the multiple devices of the called client, excluding the target device, to ignore the call request.

[0066] As described above, the A / B testing method proposed in this application can generate a communication identifier and send a call request carrying the communication identifier to the called client when a call is detected from the calling client. It then receives the call response information from the called client, establishes a communication connection between the calling and called clients based on the response information, and sends return information carrying the communication identifier to the calling client. This allows the calling and called clients to verify the communication identifier and complete the A / B test. The solution provided in this application can send communication identifiers to both the calling and called clients and verify the communication identifiers when establishing a communication connection. After successful verification, the client can execute the experimental rules locally, quickly obtaining the traffic splitting results, significantly improving the efficiency of A / B test traffic splitting and the accuracy of subsequent experiments.

[0067] This application also provides an A / B experiment system, including a client, an A / B experiment platform, a server, and a data analysis platform; The client is used to obtain AB experiment rules from the AB experiment platform based on the current call application. The AB experiment rules include flow splitting rules and experiment configuration. After establishing a communication connection with the interaction object, the client performs flow splitting based on the flow splitting rules to obtain the flow splitting results. After the communication ends, the client reports the experiment data to the data analysis platform according to the experiment configuration and flow splitting results to complete the AB experiment. The server is used to generate a communication identifier when a call is detected from a calling client and to send a call request carrying the communication identifier to the called client. After receiving the call response information from the called client, the server establishes a communication connection between the calling client and the called client based on the response information and sends a return message carrying the communication identifier to the calling client. The communication identifier is configured to be used as input parameters by the calling client and the called client respectively, and is calculated by substituting them into the same A / B test rules stored locally, so as to independently obtain consistent traffic splitting results, thereby executing the A / B test method described in any one of the embodiments of this application. The data analysis platform is used to receive experimental data reported by clients and perform analysis and processing.

[0068] For example, the above AB experiment system framework is as follows: Figure 4 As shown, the system includes a client, an A / B testing platform, a server, and a data analysis platform. The client includes calling and called clients, with the called client comprising multiple devices. The calling client is the terminal device that initiates the call request, such as a user's mobile phone or computer. Before the call, it retrieves the testing rules from the A / B testing platform; during the call, it executes the traffic splitting rules; and after the call, it reports the testing data to the data analysis platform. The called client consists of multiple devices (called client 1, 2... called client N) that receive the call request; the number of these devices can be multiple. Similar to the calling client, it needs to retrieve the testing rules in advance, participate in version matching and traffic splitting rule execution during the call, and report the testing data after the call. The A / B testing platform is responsible for configuring, managing, and distributing A / B testing rules, including traffic splitting rules and testing configurations. Providing the pre-retrieval service of rules to the multiple devices of both the calling and called clients is the core configuration management unit for implementing A / B testing. The server-side component generates a unique communication identifier (callid) when the calling client initiates a call and sends call requests carrying this identifier to multiple devices on the called client. It is also responsible for extracting and forwarding version information between the calling and called clients, serving as the core control and information relay unit for the call flow. The data analysis platform receives experimental data reported by the calling and called clients after the call ends, summarizes, cleans, and analyzes the data, generates experimental reports, and provides data support for A / B testing effectiveness evaluation and decision-making. It is the core unit for evaluating experimental effectiveness.

[0069] This application's embodiments effectively solve the real-time and accuracy issues of A / B testing in VoIP call scenarios by offloading the A / B test traffic splitting decision process to the client's local execution. Compared to existing technologies, this invention avoids additional network requests during call establishment through pre-acquisition rules and local execution strategies. Multi-device version collaborative verification ensures that experimental conditions are met before enabling experimental functions. Adding new experimental dimensions only requires client support, without modifying the A / B test platform or server. Function matching logic is implemented on the client, reducing server complexity and maintenance costs.

[0070] To implement the above methods, this application also provides an AB test method apparatus, which can be integrated into a terminal device such as a mobile phone or tablet computer.

[0071] For example, such as Figure 5 The diagram shown is a structural schematic of an AB experimental apparatus provided in an embodiment of this application. The AB experimental apparatus includes: The acquisition unit 301 is used to acquire AB experiment rules based on the current call application, the AB experiment rules including traffic splitting rules and experiment configuration; The traffic splitting unit 302 is used to establish a communication connection with the interactive object and perform traffic splitting based on the traffic splitting rules to obtain the traffic splitting result. The reporting unit 303 is used to report experimental data according to the experimental configuration and the splitting result after the communication ends, so as to complete the AB experiment.

[0072] The A / B testing device proposed in this application can obtain A / B testing rules based on the current call application. These rules include traffic splitting rules and test configurations. A communication connection is established with the interaction object, and traffic is split based on the rules to obtain the splitting results. After the communication ends, test data is reported according to the test configuration and the splitting results to complete the A / B test. The solution provided in this application can pre-deploy A / B testing rules to the client and execute them locally when the service conditions are met, quickly obtaining splitting results. This significantly improves the efficiency of A / B test traffic splitting and the accuracy of subsequent tests. Furthermore, the splitting rules used do not require experimental platform support; only client support is needed, resulting in better scalability.

[0073] Furthermore, embodiments of this application also provide an AB experimental method apparatus, which can be specifically integrated into a server, for example, such as... Figure 6 The diagram shown is another structural schematic of the AB experimental apparatus provided in this application embodiment. The AB experimental apparatus includes: The generation unit 304 is used to generate a communication identifier and send a call request carrying the communication identifier to the called client when a call is detected to be initiated by the calling client; The communication unit 305 is used to receive the call response information of the called client and establish a communication connection between the calling client and the called client based on the response information. The sending unit 306 is used to send return information carrying the communication identifier to the calling client, wherein the communication identifier is configured to be used as input parameters by the calling client and the called client respectively, and calculated by substituting them into the same AB experiment rules stored locally, so as to independently obtain consistent traffic splitting results, thereby executing the AB experiment method described in any one of the embodiments of this application.

[0074] This application also provides a terminal device, such as... Figure 7 As shown, the terminal device 400 may include a memory 401 and a processor 402. The memory 401 stores an application program, which, when executed by the processor 402, implements the steps of the AB experiment method described above in this embodiment.

[0075] Specifically, in this embodiment, the processor 402 in the terminal device 400 loads the executable files corresponding to the processes of one or more applications into the memory 401 according to the following instructions, and the processor 402 runs the applications stored in the memory 401 to achieve various functions: The A / B experiment rules are obtained based on the current call application, and the A / B experiment rules include traffic splitting rules and experiment configurations. Establish a communication connection with the interactive object and perform traffic splitting based on the aforementioned traffic splitting rules to obtain the traffic splitting result; After the communication ends, experimental data is reported according to the experimental configuration and the splitting results to complete the AB experiment.

[0076] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the detailed description of the AB test method above, which will not be repeated here.

[0077] As can be seen from the above, the terminal device in this embodiment can obtain A / B test rules based on the current call application. The A / B test rules include traffic splitting rules and test configurations. It establishes a communication connection with the interaction object and performs traffic splitting based on the traffic splitting rules to obtain the traffic splitting results. After the communication ends, it reports the test data according to the test configuration and traffic splitting results to complete the A / B test. The solution provided in this embodiment can pre-deploy A / B test rules to the client and execute them locally when the service conditions are met, quickly obtaining traffic splitting results. This significantly improves the efficiency of A / B test traffic splitting and the accuracy of subsequent tests. In addition, the traffic splitting rules used do not require support from the test platform, only client support, and also have better scalability.

[0078] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0079] Therefore, embodiments of this application provide a computer-readable storage medium storing multiple instructions that can be loaded by a processor to execute the steps of any of the A / B testing methods provided in these embodiments. The storage medium may include: read-only memory (ROM), random access memory (RAM), a disk, or an optical disk, etc. Since the instructions stored in this storage medium can execute the steps of any of the A / B testing methods provided in these embodiments, the beneficial effects achievable by any of the A / B testing methods provided in these embodiments can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0080] In some possible implementations, various aspects of the methods provided in this application can also be implemented as a program product comprising program code that, when run on a computer device, is configured to cause the computer device to perform the steps of the methods according to the various exemplary embodiments of this application described above. For example, the computer device may perform the AB test method described in the embodiments of this application. The program product may be implemented using any combination of one or more readable media.

[0081] The foregoing has provided a detailed description of an AB test method, apparatus, system, device, storage medium, and product provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An A / B testing method applied to a client, characterized in that, include: The A / B experiment rules are obtained based on the current call application, and the A / B experiment rules include traffic splitting rules and experiment configurations. Establish a communication connection with the interactive object and perform traffic splitting based on the aforementioned traffic splitting rules to obtain the traffic splitting result; After the communication ends, experimental data is reported according to the experimental configuration and the splitting results to complete the AB experiment.

2. The AB test method as described in claim 1, characterized in that, Before performing traffic splitting based on the aforementioned splitting rules, the method further includes: Receive a communication identifier sent by the server and determine whether the communication identifier meets a preset value; If the conditions are met, then the step of traffic splitting based on the aforementioned splitting rules will be executed.

3. The AB test method as described in claim 1, characterized in that, Before performing traffic splitting based on the aforementioned splitting rules, the method further includes: Obtain the client version information of the interaction object; Match the current client version information with the client version information of the interaction object; If a match is found, the step of traffic splitting based on the aforementioned splitting rules will be executed.

4. The AB test method as described in claim 3, characterized in that, The step of matching the current client version information with the client version information of the interaction object includes: Analyze the version support list in the experiment configuration; Determine whether the version information of the current client and the client version information of the interaction object are both in the list; If both are present, then the match is successful.

5. The AB test method as described in claim 1, characterized in that, The method of obtaining A / B test rules based on the current call application includes: Determine whether the application has started or switched to the foreground; If so, actively retrieve the AB experiment rule script from the AB experiment platform and store it locally.

6. An A / B testing method applied to the server side, characterized in that, include: When a call is detected to be initiated by a calling client, a communication identifier is generated and a call request carrying the communication identifier is sent to the called client; Receive the call response information from the called client, and establish a communication connection between the calling client and the called client based on the response information; Send a return message carrying the communication identifier to the calling client; The communication identifier is configured to be used as input parameters by the calling client and the called client respectively, and substituted into the same AB experiment rules stored locally for calculation, so as to independently obtain consistent traffic splitting results, thereby executing the AB experiment method as described in any one of claims 1 to 5.

7. The AB test method as described in claim 6, characterized in that, The method further includes: After detecting that the calling client has initiated a call, the version information of the calling client is extracted and forwarded to the called client; After establishing a communication connection between the calling client and the called client, the version information of the called client is extracted and forwarded to the calling client.

8. The AB test method as described in claim 6, characterized in that, The called client includes multiple devices, and the method further includes: After the interaction object establishes a communication connection with the calling client, the interaction object is identified as the target device, and the other devices among the multiple devices of the called client, excluding the target device, are controlled to ignore the call request.

9. An AB experimental setup, applied to a client, characterized in that, include: The acquisition unit is used to acquire AB experiment rules based on the current call application, the AB experiment rules including traffic splitting rules and experiment configuration; The traffic splitting unit is used to establish a communication connection with the interactive object and perform traffic splitting based on the traffic splitting rules to obtain the traffic splitting result. The reporting unit is used to report experimental data according to the experimental configuration and the splitting results after the communication ends, so as to complete the AB experiment.

10. An AB experimental setup, applied to a server, characterized in that, include: The generation unit is used to generate a communication identifier and send a call request carrying the communication identifier to the called client when a call is detected to be initiated by the calling client; A communication unit is used to receive call response information from the called client and establish a communication connection between the calling client and the called client based on the response information. The sending unit is configured to send return information carrying the communication identifier to the calling client, wherein the communication identifier is configured to be used as input parameters by the calling client and the called client respectively, and calculated by substituting them into the same AB experiment rules stored locally, so as to independently obtain consistent traffic splitting results, thereby executing the AB experiment method as described in any one of claims 1 to 5.

11. An AB experimental system, characterized in that, This includes the client, A / B testing platform, server, and data analysis platform; The client is used to obtain AB experiment rules from the AB experiment platform based on the current call application. The AB experiment rules include flow splitting rules and experiment configuration. After establishing a communication connection with the interaction object, the client performs flow splitting based on the flow splitting rules to obtain the flow splitting result. After the communication ends, the client reports the experiment data to the data analysis platform according to the experiment configuration and the flow splitting result to complete the AB experiment. The server is configured to generate a communication identifier when a call is detected from a calling client and send a call request carrying the communication identifier to the called client. After receiving the call response information from the called client, the server establishes a communication connection between the calling client and the called client based on the response information and sends a return message carrying the communication identifier to the calling client. The communication identifier is configured to be used as input parameters by the calling client and the called client respectively, and calculated by substituting them into the same A / B test rules stored locally, so as to independently obtain consistent traffic splitting results, thereby executing the A / B test method as described in any one of claims 1 to 5. The data analysis platform is used to receive experimental data reported by the client and perform analysis and processing.

12. A terminal device, characterized in that, The terminal device includes: a memory and a processor, wherein the memory stores an application program, and when the application program is executed by the processor, it implements the steps of the AB experimental method as described in any one of claims 1 to 8.

13. A computer-readable storage medium, characterized in that, The storage medium stores a plurality of instructions adapted for loading by a processor to execute the steps of the AB experimental method according to any one of claims 1 to 8.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the AB experimental method as described in any one of claims 1 to 8.

Images