Network application detection method and device, electronic equipment and storage medium

Abstract

The invention provides a network application detection method and device, electronic equipment and a storage medium. The method comprises the steps that experiment configuration data of a network application are acquired, and the experiment configuration data comprise scene nodes used for an experiment and flow distribution rules of the scene nodes; receiving access traffic passing through the scene node; based on the traffic distribution rule of the scene node, distributing the access traffic to different experiments; and executing an experiment process of the experiment based on the distributed access traffic to obtain at least one of intermediate data and an execution result of the experiment. According to the invention, the experiment efficiency of the network application can be improved.

Description

Technical Field

[0001] The present application relates to computer technology, and in particular to a method, device, electronic device and storage medium for detecting network applications. Background Art

[0002] There are many kinds of network applications at present, such as online chat, instant messaging, intelligent customer service, etc. When network applications are put into use, experiments are usually carried out to evaluate the advantages and disadvantages of different versions. Taking intelligent customer service as an example, in the relevant technology, for multiple versions of front-end interaction experiments, if you want to assign two different robots to a channel of a business party and conduct multiple rounds of dialogue, knowledge, algorithms or robot intelligent conversations, etc., you can only configure the experiment separately according to a fixed layout, for example, set up independent experiments for each scenario layout.

[0003] Related Art The experimental method of network applications is fixed and the experimental efficiency is low. Summary of the invention

[0004] The embodiments of the present application provide a method, device, electronic device and computer-readable storage medium for detecting a network application, which can improve the efficiency of experiments on network applications.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides a method for detecting a network application, the method comprising:

[0007] Acquire experimental configuration data of the network application, wherein the experimental configuration data includes scene nodes used for the experiment and traffic distribution rules of the scene nodes;

[0008] Receiving access traffic passing through the scene node;

[0009] Based on the traffic distribution rule of the scenario node, the access traffic is distributed to different experiments;

[0010] The experimental process of the experiment is executed based on the allocated access traffic to obtain at least one of intermediate data and execution results of the experiment.

[0011] The present application provides a network application detection device, including:

[0012] An experiment configuration module, used to obtain experiment configuration data of a network application, wherein the experiment configuration data includes a scene node used for the experiment and a traffic distribution rule of the scene node;

[0013] An experiment execution module, used for receiving access traffic passing through the scene node, and distributing the access traffic to different experiments based on the traffic distribution rule of the scene node;

[0014] The experiment execution module is further used to execute the experiment process of the experiment based on the allocated access traffic to obtain at least one of the intermediate data and execution results of the experiment.

[0015] An embodiment of the present application provides an electronic device, the electronic device comprising:

[0016] A memory for storing computer executable instructions;

[0017] The processor is used to implement the network application detection method provided in the embodiment of the present application when executing the computer executable instructions stored in the memory.

[0018] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions for implementing the network application detection method provided in the embodiment of the present application when executed by a processor.

[0019] An embodiment of the present application provides a computer program product, including a computer program or a computer executable instruction. When the computer program or the computer executable instruction is executed by a processor, the network application detection method provided in the embodiment of the present application is implemented.

[0020] The embodiments of the present application have the following beneficial effects:

[0021] By obtaining the experimental configuration data of multiple scenario nodes and executing the experimental process of access traffic based on the experimental configuration data, different scenario nodes can be tested during the experiment, and the access traffic distribution method can be distinguished through traffic distribution rules. The access traffic can be divided into different experiments, which can realize experiments on multiple different scenario nodes at the same time, thereby improving the efficiency of network application experiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1A It is a schematic diagram of an application mode of a method for detecting a network application provided in an embodiment of the present application;

[0023] Figure 1B It is a schematic diagram of an application mode of a method for detecting a network application provided in an embodiment of the present application;

[0024] Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0025] FIG. 3A to FIG. 3C It is a flowchart of a method for detecting a network application provided in an embodiment of the present application;

[0026] Figure 4A It is a schematic diagram of the principle of the intelligent customer service provided in the embodiment of the present application;

[0027] Figure 4B This is an optional flow chart of a method for detecting a network application provided in an embodiment of the present application;

[0028] Figure 4C It is a schematic diagram of the relationship between scene nodes provided in an embodiment of the present application;

[0029] Figure 5 This is an optional flow chart of a method for detecting a network application provided in an embodiment of the present application;

[0030] Figure 6 It is a schematic diagram of the principle of flow orthogonality provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0032] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0033] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0034] It should be pointed out that the collection and processing of relevant data in this application (for example, user access traffic) should be strictly in accordance with the requirements of relevant national laws and regulations when applied in examples, and the informed consent or separate consent of the personal information subject should be obtained. Subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0035] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0036] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0037] 1) AB experiment, also known as control experiment, is used to determine which of two elements or versions (A and B) is better. During the control experiment, at least two elements or two versions are tested at the same time, and the best version is selected for use based on the experimental results.

[0038] 2) Intelligent customer service is an online customer service system based on a conversational robot, which is used to solve most of the user's consultation requests and guide users to complete functional processing. Intelligent customer service can be classified into intelligent customer service for the B-end (for enterprises or merchants) and for the C-end (for users).

[0039] 3) Natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between humans and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics, and usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graphs and other technologies.

[0040] 4) Conversation refers to a conversation between two or more people. The two parties in the conversation can be human to human, human to artificial intelligence, or artificial intelligence to artificial intelligence. For example, when a user chats with a robot, a conversation will be initiated first. This conversation runs through the entire process of the user's chat, but the conversation will be automatically closed two hours after the user stops asking questions or answering questions, and after the user is transferred to manual customer service.

[0041] Intelligent customer service is an online customer service system based on a conversational robot, with the primary purpose of solving most of the user's consulting demands and guiding the user to complete the function processing. Most existing intelligent customer service experimental schemes tend to provide a fixed-format AB test configuration. If you want to assign two different robots to a business channel and conduct multiple rounds of dialogue, knowledge, algorithms or robots and other multiple control experiments, you can only configure the experiment separately according to the fixed format. In the embodiment of the present application, multiple scene nodes are configured during the network application experiment, and experiments can be conducted simultaneously on multiple scene nodes. By allocating the access traffic in the experiment according to the pre-configured traffic allocation rules, the comprehensiveness of the network application experiment is improved, and experiments are conducted simultaneously on multiple different scene nodes, which improves the efficiency of the network application experiment.

[0042] Embodiments of the present application provide a method for detecting a network application, a device for detecting a network application, an electronic device, a computer-readable storage medium, and a computer program product.

[0043] The following describes an exemplary application of an electronic device provided by an embodiment of the present application. The electronic device provided by an embodiment of the present application can be implemented as a laptop, a tablet computer, a desktop computer, a set-top box, a mobile device (for example, a mobile phone, a portable music player, a personal digital assistant, a dedicated messaging device, a portable gaming device), a vehicle terminal, a virtual reality (VR) device, an augmented reality (AR) device, and other types of user terminals), and can also be implemented as a server. Below, an exemplary application when the device is implemented as a terminal device or a server will be described.

[0044] refer to Figure 1A , Figure 1A Schematic diagram of the application mode of the network application detection method provided in the embodiment of the present application; for example, Figure 1A The server 200, network 300, terminal devices 400-1 and 400-2, and database 500 are involved. Terminal devices 400-1 and 400-2 are connected to server 200 via network 300. Network 300 can be a wide area network or a local area network, or a combination of the two.

[0045] In some embodiments, the server 200 is a server for performing experiments on network applications, and the terminal device 400-1 is a mobile phone or computer used by the user, which is installed with a network application or applet to be experimented, or the user uses the web version of the network application. The terminal device 400-1 sends the instructions or questions input by the user to the server 200 through the network 300, and the server 200 obtains the answer to the question from the database 500 according to the question raised by the user. The server 200 is also used to call the detection method of the network application provided in the embodiment of the present application to configure the experimental configuration data, record the intermediate data and execution results during the experiment, and send the intermediate data and execution results to the experiment initiator holding the terminal device 400-2. The experiment initiator can be a technician who develops network applications.

[0046] In some embodiments, the network application detection method of the embodiment of the present application can also be applied in the following implementation environment: Figure 1B , Figure 1B It is a schematic diagram of an application mode of the detection method of the network application provided in the embodiment of the present application, and is explained by taking the network application being intelligent customer service as an example.

[0047] For example, in actual applications, there may be multiple different users, and multiple different users use different terminal devices 400-1 to send questions raised by the users to the server 200. The server 200 includes an intelligent customer service system 204, an experiment configuration module 201, an experiment execution module 202, and an experiment data recording module 203. Each module (system) can be implemented by an independent server. The intelligent customer service system 204 extracts the answer to the question from the sub-database question and answer database 502 in the database 500, and feeds the answer to the question back to the user's terminal device 400-1. When the experiment configuration module 201 detects that the access traffic of the user accessing the intelligent customer service system 204 passes through the scene node of the preset experiment, the corresponding experiment configuration data is extracted from the experiment database 501, and the experiment configuration data is sent to the experiment execution module 202, so that the experiment execution module 202 executes the experiment process according to the access traffic, and sends the data generated during the experiment to the experiment data recording module 203. The experiment data recording module 203 sends the generated experiment results to the experiment database 501 of the database 500 for storage, and can also send the generated experiment results to the terminal device 400-2, so that the experiment initiator holding the terminal device 400-2 can view the experiment results, so as to evaluate the network application of the tested intelligent customer service.

[0048] The embodiments of the present application can also be implemented through blockchain technology. The experimental results and intermediate data of the network application obtained by the embodiments of the present application can be uploaded to the blockchain for storage, and the reliability of the experimental results and intermediate data can be guaranteed by the consensus algorithm. Blockchain is a new application mode of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods, each of which contains information about a batch of experimental results, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. The blockchain can include the underlying blockchain platform, the platform product service layer, and the application service layer.

[0049] The embodiments of the present application can also be implemented through database technology. In short, a database can be regarded as an electronic file cabinet where electronic files are stored. Users can add, query, update, delete, etc. data in the files. The so-called "database" is a collection of data that is stored together in a certain way, can be shared with multiple users, has as little redundancy as possible, and is independent of the application program.

[0050] A database management system (DBMS) is a computer software system designed for managing databases. It generally has basic functions such as storage, retrieval, security, and backup. Database management systems can be classified according to the database model they support, such as relational, XML (Extensible Markup Language); or according to the type of computer they support, such as server clusters, mobile phones; or according to the query language used, such as Structured Query Language (SQL), XQuery; or according to performance focus, such as maximum scale, maximum operating speed; or other classification methods. Regardless of the classification method used, some DBMS can cross categories, for example, supporting multiple query languages ​​at the same time.

[0051] The embodiments of the present application can also be implemented through cloud technology. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model application. It can form a resource pool, which is used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the high development and application of the Internet industry, as well as the promotion of search services, social networks, mobile commerce and open collaboration, each item may have its own hash code identification mark in the future, and all need to be transmitted to the background system for logical processing. Data of different levels will be processed separately, and all kinds of industry data require strong system backing support, which can be achieved through cloud computing.

[0052] In some embodiments, the server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The electronic device may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal device and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present invention.

[0053] See also Figure 2 , Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may be Figure 1A or Figure 1BServer 200 in Figure 2 The server 200 shown includes: at least one processor 410, a memory 450, and at least one network interface 420. The various components in the server 200 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 440 is not described in detail. Figure 2 Various buses are labeled as bus system 440 .

[0054] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0055] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0056] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0057] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0058] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0059] A network communication module 452, used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.;

[0060] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 The detection device 455 of the network application stored in the memory 450 is shown, which can be software in the form of programs and plug-ins, including the following software modules: an experiment configuration module 4551, an experiment execution module 4552, and an experiment data recording module 4553. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0061] In some embodiments, the terminal or server can implement the detection method of the network application provided in the embodiment of the present application by running a computer program. For example, the computer program can be a native program or software module in the operating system; it can be a local (Native) application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as a shopping APP or an instant messaging APP; it can also be a small program, that is, a program that can be run only by downloading it to a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be an application, module or plug-in in any form.

[0062] The detection method of the network application provided in the embodiment of the present application will be explained in combination with the exemplary application and implementation of the terminal provided in the embodiment of the present application.

[0063] The following describes the network application detection method provided by the embodiment of the present application. As mentioned above, the electronic device implementing the network application detection method of the embodiment of the present application can be a terminal, a server, or a combination of the two. Therefore, the execution subject of each step will not be repeatedly described below.

[0064] It should be noted that in the examples below, intelligent customer service is used as an example. Based on their understanding of the following, those skilled in the art can apply the network application detection method provided in the embodiments of the present application to experiments on other types of network applications, such as instant messaging software, chat software, etc.

[0065] See also Figure 3A , Figure 3A is a flow chart of the network application detection method provided in the embodiment of the present application, which will be combined with Figure 3A The steps shown are explained.

[0066] In step 301, experimental configuration data of a network application is obtained. Here, the experimental configuration data includes multiple scene nodes for the experiment, the access flows of scene nodes at different levels in the multiple scene nodes are orthogonal to each other, and the access flows of scene nodes at the same level in the multiple scene nodes are mutually exclusive.

[0067] For example: the scene nodes include N, N is a positive integer, the N scene nodes are distributed in M ​​levels, the access traffic of the i-th level is merged into the i+1-th level, and dispersed into the scene nodes of the i+1-th level, wherein 1≤i≤M-1.

[0068] In some embodiments, the access traffic of scene nodes at different levels among the multiple scene nodes is orthogonal to each other, which can be achieved in the following way: there is no dependency between scene nodes at different levels among the multiple scene nodes, except for the scene nodes at the last level among the different levels, the access traffic of each level is merged before entering the next level, and the access traffic entering the scene nodes at the next level is dispersed; the access traffic of scene nodes at the same level among the multiple scene nodes is mutually exclusive, which can be achieved in the following way: the intersection of the access traffic entering the scene nodes at the same level among the multiple scene nodes is an empty set.

[0069] refer to Figure 6 , Figure 6 It is a schematic diagram of the principle of traffic orthogonality provided in an embodiment of the present application. The experimental groups are divided into Group A and Group B, wherein there are differences in the versions of intelligent customer services corresponding to Group A and Group B. For example, Group A is an improved intelligent customer service based on Group B. In the case of non-orthogonal traffic (mutually exclusive traffic), when the access traffic used up in each experimental layer enters the next layer, it is maintained in the corresponding experimental group. In the case of orthogonal traffic, when the access traffic used up in each layer enters the next layer, it is evenly redistributed to different experimental groups.

[0070] In some embodiments, the multiple scene nodes may include: channel nodes, opening words recommendation nodes, fixed recommendation position recommendation nodes, fuzzy question recommendation nodes, similar question recommendation nodes, front-end style display nodes (human-computer interaction interface display nodes), and robot nodes; wherein the types of robot nodes include: knowledge base answer nodes, multi-round dialogue answer nodes, and intent recognition nodes.

[0071] refer to Figure 4C , Figure 4C It is a relationship diagram of the scenario nodes provided in the embodiment of the present application. Among them, the conversation intelligent allocation strategy scenario node is used to characterize the node represented by any process involving traffic allocation in the intelligent conversation scenario. Therefore, the conversation intelligent allocation strategy scenario node is the parent scenario node of all scenarios. Channels, opening words recommendation, fuzzy rhetorical question recommendation, similar question recommendation, fixed recommendation position recommendation, robot and front-end style display are all sub-scenario nodes under the conversation intelligent allocation strategy scenario node, and knowledge base answer, multi-round dialogue answer and intention recognition algorithm are sub-nodes under the robot scenario node.

[0072] Taking the channel scenario node as an example, before providing services to users, the intelligent customer service will find the corresponding session intelligent allocation plan to initialize the session connection based on the current incoming channel (the incoming channel refers to the business channel through which the user connects to the intelligent customer service, including web pages, applications, mini-programs, etc.). The channel diversion scenario node performs diversion verification by configuring different control experiments.

[0073] In some embodiments, when the scene node is a channel node, the access traffic corresponding to each session of each visiting user within a preset period is allocated to the same experiment, and the session of each visiting user is implemented by a robot corresponding to each visiting user.

[0074] For example, in order to ensure that a user can get the service of the same robot within a certain period of time under a certain business channel, when the experiment is configured as a session-level traffic allocation rule (the traffic of the same user in the same session is divided into one experiment), the same user can get the service of the same robot within a certain period of time (configurable).

[0075] In the embodiment of the present application, by enabling the same user to obtain the service of the same robot within a certain period of time, the user's continuity experience of intelligent customer service is improved, and the efficiency of user-level experiments can be improved.

[0076] In some embodiments, reference Figure 3B , Figure 3B It is a flow chart of the detection method of network application provided in the embodiment of the present application; before step 302, steps 3021 to 3023 are executed, which are described in detail below.

[0077] In step 3021, the ratio between the access traffic of the executed experimental process and the total access traffic is configured.

[0078] For example, the total access traffic is all the traffic passing through the scene nodes within the effective time of an experiment. For example, the total duration (effective time) of the experimental process of the experiment is from 8 am to 8 pm on April 23, 2023. Then, the total access traffic passing through the scene nodes corresponding to the experiment within this duration is the total access traffic of the experiment. The ratio between the access traffic of the executed experimental process and the total access traffic, that is, the proportion of experimental traffic, can be configured according to the needs of the experiment.

[0079] For example, step 3022 is performed before step 3023. In the embodiment of the present application, the order of step 3021, step 3022 and step 3023 is not limited, and step 3021 can be performed between, before or after step 3022 and step 3023.

[0080] In step 3022, user information of the accessing user is obtained.

[0081] For example, in the field of intelligent customer service, the current visiting user is also the current online user, and there can be multiple online users at the same time.

[0082] In step 3023, the user type of the accessing user is determined according to the user information.

[0083] For example, the user types include anonymous users, tagged users, real-name users, and regular users, wherein the access frequency of the regular users is greater than the frequency threshold. An anonymous user refers to a user who does not have a unique identifier for identification. Real-name users, in contrast to anonymous users, have a unique identifier for identification. Tagged users are specific customers who are marked by the system of the network application during the use of the network application, such as non-member users, member users, whitelist users, or blacklist users.

[0084] For example, there are crossovers between user type attributes, for example, a user can be a real-name user and a regular user. Therefore, there are priorities between user types, regular users have a higher priority than other user types, and tagged users have a higher priority than real-name users and anonymous users. For example, when a user has the attributes of a real-name user and a regular user, the regular user has a higher priority than the real-name user, and is classified as a regular user.

[0085] For example, the access frequency refers to the number of times a scene node is accessed within a preset duration, and the frequency threshold can be set based on the actual scenario of the experiment. For example, in an experiment on the intelligent customer service of a shopping software, if the number of conversations a user has with the intelligent customer service is greater than 5 times (frequency threshold) within a month (preset duration), then the user is a familiar customer (regular user).

[0086] Continue to refer Figure 3A , in step 302, access traffic passing through the scene node is received.

[0087] For example, in the process of receiving access traffic passing through the scene node, the access traffic can be diverted according to the above user type so as to execute the corresponding experimental process.

[0088] In step 303, based on the traffic distribution rules of the scenario nodes, the access traffic is distributed to different experiments.

[0089] For example, the traffic distribution rule is used to distinguish the distribution mode of the access traffic, that is, the traffic distribution rule is used to distinguish multiple modes of dividing the traffic into different experiments. The traffic distribution rule can be set according to specific experimental requirements.

[0090] In some embodiments, the experiment configuration data includes traffic allocation rules corresponding to preset scene nodes, and the access traffic is allocated to different experiments based on the traffic allocation rules of the scene nodes.

[0091] In some embodiments, the types of traffic distribution rules include: session level, user level and problem level; in step 303, this can be achieved by:

[0092] 1. In response to the traffic allocation rule being at the session level, the access traffic is allocated to an experiment corresponding to the session to which the access traffic belongs.

[0093] For example, the session-level traffic allocation rule means that all traffic of the same session is allocated to the same experiment. For example, all access traffic of user A during the same session of intelligent customer service is allocated to one experiment. If user A stops this session and starts the next session, the access traffic of the next session is allocated to another experiment.

[0094] 2. In response to the traffic distribution rule being at the user level, the access traffic is distributed to an experiment corresponding to the user to which the access traffic belongs.

[0095] For example, the user-level traffic allocation rule means that all traffic belonging to the same user within a period of time is allocated to the same experiment. For example: all access traffic of user A to smart customer service within a period of time is allocated to experiment A. All access traffic of user B to smart customer service within a period of time is allocated to experiment B. Experiment A and experiment B are different experiments, and they can be run at the same time.

[0096] 3. In response to the traffic allocation rule being at the problem level, the access traffic is allocated to an experiment corresponding to the problem to which the access traffic belongs.

[0097] For example, the question-level traffic allocation rule means that the traffic of each question asked by the user is reallocated to the experiment (that is, the traffic of each question is allocated to one experiment). For example: each question asked by user A and its answer are allocated to each experiment. User A asked 3 questions, and the access traffic of each question and its corresponding answer are allocated to 3 different experiments.

[0098] In the embodiment of the present application, access traffic is allocated to different experiments according to the characteristics of traffic adhesion (problem adhesion, session adhesion, user adhesion), taking into account the fine-grained requirements of different experiments. Through different traffic adhesion methods, the access traffic in the experimental process is diverted, which facilitates the statistics of experimental data, improves the experimental efficiency of AB experiments, and improves the accuracy of experimental results. For example: at the problem level, experiments can be performed on different problems, the granularity of the experiment is higher than other types, and the relevant response content of different problems can be tested; at the user level, the access traffic of a user is allocated to the same experiment, and the corresponding intelligent customer service strategy can be customized for the user according to the experimental results to improve user satisfaction; at the session level, the response efficiency of the intelligent customer service session can be improved according to the experimental results.

[0099] In some embodiments, before step 303, experimental rules are allocated to the access traffic of the access user according to the user type of the access user, wherein the experimental rules are configured to allocate the access traffic to different experiments based on the user type and the traffic allocation rules.

[0100] For example, since different types of access users have different characteristics, the same access traffic allocation method may not be suitable for every type of user, and corresponding experimental rules can be allocated according to the user type.

[0101] In some embodiments, the user types include anonymous users, tagged users, real-name users, and regular users. According to the user types of the access users, the experimental rules for allocating the access traffic to the access users can be implemented in the following ways:

[0102] 1. When the user type of the access user is an anonymous user, the access traffic of each session corresponding to the anonymous user is allocated to different experiments.

[0103] For example, anonymous users do not have a unique user ID, and can configure session or question stickiness levels, but cannot configure user stickiness levels. The specific configuration of session or question stickiness levels is determined according to specific experimental requirements in actual applications.

[0104] 2. When the user type of the access user is a real-name user, the access traffic of the real-name user is allocated to different experiments based on the traffic allocation rule.

[0105] For example, a real-name user has a unique user ID, which is applicable to various types of traffic allocation rules. The access traffic of the real-name user can be configured according to the experimental requirements of the current experiment. The traffic adhesion of the current experiment is the traffic allocation rule configured for the current experiment in actual application.

[0106] 3. When the user type of the access user is a tag user, the access traffic of the tag user is allocated to different experiments based on the experimental rules corresponding to the tag user.

[0107] For example, the intelligent customer service system provides many labels for real-name users, such as members, high-risk users, partners' users, etc. Under a certain label rule, a configuration parameter for whether to participate in the experiment and a configuration parameter for the proportion of participating in the experiment will be provided. The default parameter is to participate in the experiment. This configuration is defined in the experiment extension node interface, and each experiment node needs to implement the definition. For example, real-name users with the high-risk user label do not participate in any experiment, while real-name users with the member label are configured to have 30% of users participate in the experiment.

[0108] 4. When the user type of the access user is a regular user, the access traffic corresponding to the regular user is configured to not execute the experimental process.

[0109] For example, regular users are familiar customers. Since their access traffic has been recorded, the reference value of their access traffic is lower than that of other types of customers. Therefore, the experimental process may not be performed on their access traffic.

[0110] In the embodiment of the present application, experimental rules corresponding to access traffic configuration are configured according to user types, thereby improving experimental efficiency, effectively utilizing the characteristics of various user types, and avoiding experimental errors caused by user types. For example, anonymous users cannot perform experimental processes through user-level traffic allocation rules, resulting in experimental failure, thereby improving experimental efficiency.

[0111] In step 304, the experimental process of the experiment is executed based on the allocated access traffic to obtain at least one of intermediate data and execution results of the experiment.

[0112] For example, the experiment process includes obtaining the data generated in the experiment, executing the experiment within a preconfigured effective time, and related experimental content, which are described in detail below.

[0113] In some embodiments, the experimental configuration data also includes the effective time of the experiment; the step 304 can be implemented in the following manner: within the effective time of the experiment of the scene node, the experimental process of the access traffic is executed based on the experimental configuration data to obtain intermediate data and execution results during the execution process.

[0114] For example, the specified experiment process takes effect during the experiment validity period. For example, if you configure 2023-03-06 12:00:00 to 2023-03-07 12:00:00, it means that the experiment is only valid from 12:00 on March 6, 2023 to 12:00 on March 7, 2023.

[0115] In some embodiments, the experiment configuration data further includes an experiment fuse rule; when executing step 304, the following processing is synchronously performed:

[0116] 1. In response to the experimental process satisfying a first fuse condition, terminating the experimental process, wherein the first fuse condition includes at least one of the following: the experimental process fails during the first execution, and the number of failures of the experimental process reaches a threshold number.

[0117] For example, the failure of the experimental process execution means that an error occurs during the experiment, and the cause of the error is caused by the execution of the experimental process. For example: for the answering experiment of the intelligent customer service, there is an error in the program code of the answering system configured by the intelligent customer service, which causes a failure during operation, and the experimental process execution fails. If the experimental process fails to execute for the first time, the experimental process is terminated. The technicians can promptly repair the program code of the answering system.

[0118] 2. In response to the experimental process satisfying a second fuse condition, terminate the current experimental process and configure the access traffic to other experimental processes, wherein the second fuse condition includes the number of failures of the experimental process reaching a threshold number.

[0119] For example, the "first" and "second" in the second fuse condition and the first fuse condition are to distinguish different fuse conditions, and do not represent the order of execution of the two. Assume that the experimental traffic of experiment one accounts for 15% of the total access traffic. When the number of failures of experiment one reaches the number threshold, the experimental process of experiment one is abandoned, and the access traffic corresponding to experiment one is transferred to experiment two (other experiments besides experiment one) for execution by experiment two.

[0120] In the embodiment of the present application, a fuse mechanism is used to avoid wasting time and failing to obtain experimental results due to repeated execution of erroneous experiments. By transferring the traffic of failed experiments to other experiments, access traffic is effectively utilized and experimental efficiency is improved.

[0121] In some embodiments, the experimental configuration data also includes an experimental strategy corresponding to the scene node, and the experimental strategy specifies the experimental content and experimental indicators of the current experiment; for example, before step 304, the following processing is performed to configure the experimental configuration data corresponding to the scene node: in response to the scene node being a knowledge base answer node, the knowledge content to be verified and the answer to the question in the experimental configuration data are configured according to the experimental strategy; in response to the scene node being an intent recognition node, the algorithm type to be verified and the indicator corresponding to the algorithm type to be verified in the experimental configuration data are configured according to the experimental strategy.

[0122] For example, the knowledge base answering node needs to be verified as samples of knowledge content and question answers to determine the answering ability of the intelligent customer service in the experiment.

[0123] In the embodiment of the present application, for different scene nodes, corresponding indicators are configured according to the actual needs of the scene nodes, which can effectively evaluate the effect of the network application on each scene node, making it convenient for the experiment initiator to update and evaluate the network application.

[0124] In some embodiments, reference Figure 3C , Figure 3C It is a flow chart of the detection method of the network application provided in the embodiment of the present application; after step 304, step 305 is executed, and in step 305, at least one of the intermediate data and the execution result is recorded.

[0125] For example, the record includes: obtaining corresponding indicators according to the intermediate data, storing the intermediate data and the execution result, and updating the user type according to the execution result.

[0126] In some embodiments, Figure 3A Step 305 in Figure 3C Steps 3051 to 3054 in are implemented as described in detail below.

[0127] In step 3051, user information of the accessing user corresponding to the access traffic and intermediate data corresponding to each scene node are stored.

[0128] For example, user information and intermediate data corresponding to each scene node are stored in the experimental database, or the experimental results are uploaded to the blockchain.

[0129] In step 3052, the user type of each accessing user is updated according to the execution result, and in response to the updated user type being a regular user, the valid duration corresponding to the regular user is updated.

[0130] For example, the effective duration of a regular user refers to the duration during which the access traffic corresponding to the regular user is not included in the experiment. Assuming that the user type is updated to a regular user, the effective duration corresponding to the user is configured from 0 to the preset effective duration. The length of the effective duration is set according to the experimental requirements.

[0131] For example, after the experiment is over, since the number of user visits has increased compared to before the experiment, the user type can be updated according to the experimental situation. For example: in this experiment, an anonymous user registered as a real-name user, then the type of the anonymous user is updated to a real-name user, and the user's access traffic is processed in the same way as a real-name user in the next experiment. Another example: after this experiment, the user is converted to a regular user, then the experimental process may not be executed for the user's access traffic in the next experiment.

[0132] In step 3053, the scene index of each scene node is determined based on the intermediate data and the execution result.

[0133] Here, the in-scene indicator includes at least one of the following: arrival rate, in-scene conversion rate.

[0134] For example, the arrival rate is the ratio of the access traffic passing through the scenario node to the total access traffic. For scenarios such as opening word recommendation, similar question recommendation, or fixed recommendation position recommendation, the conversion rate refers to the ratio between the number of questions and answers that users click on the recommendation and the total number of questions and answers recommended to users. In the self-service multi-round dialogue, the conversion rate is characterized by the probability of completing the self-service.

[0135] In step 3054, the inter-scene index between each scene node is determined based on the intermediate data and the execution result.

[0136] Here, the inter-scenario index includes at least one of the following: conversion rate and difference rate under different access traffic ratios. The difference rate is an index used to compare different access traffic ratios.

[0137] In the embodiment of the present application, by calculating the obtained intermediate data and execution results to obtain corresponding indicators that more accurately express the experimental effect, the execution efficiency of the experiment is improved, so that the experimental results can more clearly reflect the effect of the network application, which is convenient for the experiment initiator to update and modify the network application.

[0138] In some embodiments, in response to a newly added scene node setting, an extended interface is called to obtain node configuration parameters of the newly added scene node, wherein the node configuration parameters belong to experimental configuration data, and the node configuration parameters include at least one of the following: the role of the scene node, the experimental indicators corresponding to the scene node, the preconfigured experimental rules corresponding to the scene node, and the code corresponding to the preconfigured experimental rules; in response to the node configuration parameters of an existing scene node to be updated, the extended interface is called to obtain the updated node configuration parameters corresponding to the scene node.

[0139] For example, the role of a scenario node refers to which process or function of the intelligent customer service system the scenario acts on. For example, the role of a channel scenario node is to divide different business channels in the intelligent customer service system.

[0140] For example, assuming that the newly added scenario node is a channel scenario node, then obtain the relevant information of the business channel of the channel scenario node, and the corresponding experimental indicators can be the user's reach rate and forwarding rate. If the experimental strategy newly uploaded to the server by the experiment initiator is received, the extended interface can be called to obtain the newly uploaded experimental strategy, and the old experimental strategy of the existing scenario node can be updated according to the newly uploaded experimental strategy.

[0141] In the embodiment of the present application, by obtaining the experimental configuration data of multiple scene nodes, the access traffic of scene nodes at different levels in the multiple scene nodes is orthogonal to each other, and the access traffic of scene nodes at the same level in the multiple scene nodes is mutually exclusive, so that different scene nodes can be tested during the experiment. During the experiment, the user's access traffic is mutually exclusive in the same experimental layer to maintain the control effect of the experimental data of different experimental groups. The access traffic in different experimental layers is orthogonal, which ensures the accuracy of the experimental data. By executing the experimental process of the access traffic based on the experimental configuration data, recording at least one of the intermediate data and the execution result during the execution process, the comprehensiveness of the network application experiment is improved, and it is possible to implement experiments for multiple different scene nodes at the same time, thereby improving the efficiency of the network application experiment.

[0142] Next, an exemplary application of the network application detection method according to an embodiment of the present application in an actual implementation environment will be described.

[0143] Intelligent customer service is an online customer service system based on a conversational robot. Its primary purpose is to solve most of the user's consultation requests and guide users to complete the function processing. When a user enters the intelligent customer service, it first finds a conversation intelligent allocation strategy based on the user's channel, then finds the associated robot based on the strategy, and finally initializes a conversation to provide services to the user. The service provided by the robot is composed of modules such as knowledge base, algorithm, multi-round conversation, and regular matching in intent recognition. Any slight adjustment to any of these modules may have a significant impact on the user experience.

[0144] Most existing intelligent customer service experimental schemes tend to provide a fixed-format AB test configuration. If you want to assign two different robots to a business channel and conduct multiple control experiments such as multi-round dialogue, knowledge, algorithm or robot, you can only configure the experiment separately according to the fixed format. Therefore, the relevant technology has the following problems: 1. It cannot meet the needs of various algorithms, rules and front-end interaction diversity in the field of intelligent customer service; 2. It cannot provide multiple versions of front-end interaction experiments in the multi-round dialogue module in the field of intelligent customer service; 3. It lacks the association relationship (user stickiness) generated by users when using the service, and the inter-layer and intra-layer rules of different experiments that are unique to the field of intelligent customer service.

[0145] The network application detection method provided in the embodiment of the present application is used to solve the above problems. The network application detection method provided in the embodiment of the present application can be applied to various network applications. In the embodiment of the present application, intelligent customer service is used as an example for explanation. For ease of understanding, the principle of intelligent customer service is explained below.

[0146] refer to Figure 4A , Figure 4AIt is a schematic diagram of the principle of intelligent customer service provided by the embodiment of the present application; the user uses the application related to intelligent customer service through the terminal device, and the user accesses the intelligent customer service through the current business channel (for example: application, web page or applet), and the current business channel finds the corresponding session intelligent allocation strategy to select a robot and initialize a session connection to provide the user with intelligent customer service services. Taking the web page as an example, the user asks a question to the web front-end robot, and then the robot will hand the question to the answer processing system; after the answer processing system gets the question raised by the user, it will hand it over to natural language processing (Natural Language Processing, NLP) to identify the intention of the user's question and divide its intention into one of question-answering type, task type and chat type; when the intention of the user's question is question-answering type, the answer is obtained from the knowledge database; when the intention of the user's question is task type, the answer is obtained from the multi-round dialogue processing program; when the intention of the user's question is chat type, the robot gives a chat answer. If the user is not satisfied with the question recommendation given by the robot, the corresponding option can be triggered to transfer to manual service. After the three steps of whether to transfer to manual judgment, answer processing and natural language generation, the answer is returned to the conversational robot and displayed to the user.

[0147] The network application detection method provided in the embodiment of the present application can be used for initializing session connections, the natural language processing stage in the DM response system, task-based multi-round dialogue scenarios, question-and-answer key management service (KMS) answer scenarios, and the stage of returning answers to the front-end robot, etc., which require embedded experiments.

[0148] refer to Figure 4B , Figure 4B It is an optional flow chart of the detection method of network application provided by the embodiment of the present application; the embodiment of the present application is implemented by three modules: experiment configuration module 401, experiment execution module 402 and experiment data recording module 403. The experiment configuration module is used for various experimental parameters and indicator configurations of the previous sequence, the experiment execution module (executor) is mainly responsible for executing various experiments configured in the previous sequence, and the experiment data recording module is responsible for recording various data and execution results generated during the experiment execution by the executor, and classifies and summarizes various data transmitted from the experiment execution module for storage.

[0149] Among them, after the intelligent customer service system receives the user's question, when the user's corresponding access traffic passes through the scene node, the corresponding experimental rule data is read and sent to the experimental execution module. After obtaining the traffic and experimental rule data, the experimental execution module will execute the current process according to the rules configured by the experiment, and pass the various data and execution results generated during the experimental execution to the experimental data recording module. After receiving the data from the experimental execution module, the experimental data recording module will identify and clean the data, where identification refers to obtaining specific data as experimental results (for example: an experiment wants to calculate the ratio of users transferring to the manual hotline after the end of the session, and identifies the total number of ended sessions and the number of transfers to the manual hotline after the end of the session in the experimental data.), cleaning refers to calculating the indicator value based on the obtained data (for example: the total number of ended sessions and the number of transfers to the manual hotline after the end of the session are substituted into a specific formula to obtain the corresponding ratio as the indicator value), and persisting the cleaned and identified data to the persistence layer.

[0150] For example, the various data and execution results generated during the experiment execution process mentioned above include: various intermediate data generated during the experiment execution process and various indicator data generated after the experiment is completed.

[0151] Among them, the intermediate data generated during the experiment execution refers to the experimental indicator data generated during the experiment execution and used to drive the experiment to continue and finally calculate and generate. For example: the purpose of a certain experiment is to test "the success rate of robot A and robot B's services in solving problems respectively", then during the experiment execution, the number of times users are transferred to manual after logging in, the rate of transfer to manual after 24 hours, and the number of user likes and dislikes will be generated. These data are intermediate data, and the intermediate data are used to generate indicators for evaluating the experimental effect after the experiment is completed.

[0152] The various types of indicator data generated after the experiment is completed include: general indicators and experiment-defined indicators. General indicators are indicator data that will be calculated and generated regardless of the experiment, such as: traffic turnover rate, real-name user ratio and anonymous user ratio, etc.; while experiment-defined indicator data focuses on the indicators expected by the current experiment, such as: the ratio of conversations transferred to manual and the ratio of self-service scenario handling in each robot under different traffic ratios in the "robot diversion" experiment, etc.

[0153] For example, the data generated by different test purposes are different. For example, the test goal of a certain experiment is to "clear the proof of multiple rounds of dialogue" and whether the user's self-service processing rate can be provided after modifying a certain judgment node. Then the data generated after the end of the experiment is the proportion of users who successfully processed in the multiple rounds of dialogue and the year-on-year comparison before and after the experiment. Therefore, what is recorded here is the unified indicator data after the end of different experiments and the customized data that the current experiment specifies as the focus.

[0154] The following, Figure 4B The specific functions of each module in the

[0155] The AB experiment configuration module defines the data structure of the experiment rule configuration and the scene nodes that allow the experiment to be inserted, and provides unified configuration and management to facilitate subsequent maintenance, reading and execution.

[0156] The configuration of the scene node for inserting the experiment also specifies that the experiment acts on a certain processing link in the answering process. The following scene nodes are included in the embodiment of the present application: intelligent conversation allocation strategy, robot, multi-round dialogue version diversion (where different dialogue versions use different algorithms or different databases, and the generated reply content is different), intention recognition algorithm diversion, knowledge base answer diversion, opening words recommendation, fixed recommendation position recommendation, fuzzy question recommendation, similar question recommendation and front-end page display diversion

[0157] refer to Figure 4C , Figure 4C It is a schematic diagram of the relationship between the scenario nodes provided in the embodiment of the present application. Among them, the conversation intelligent allocation strategy scenario node is the parent scenario node of all scenarios, the opening words recommendation, fuzzy rhetorical question recommendation, similar question recommendation, fixed recommendation position recommendation, robot and front-end style display are all sub-scenario nodes under the conversation intelligent allocation strategy scenario node, and the knowledge base answer, multi-round dialogue answer and intention recognition algorithm are sub-nodes under the robot scenario node.

[0158] Taking the channel diversion scenario node as an example, before providing services to users, the intelligent customer service will find the corresponding session intelligent allocation plan to initialize the session connection based on the current incoming channel (the incoming channel refers to the business channel through which the user connects to the intelligent customer service, including web pages, applications, mini programs, etc.). The channel diversion scenario node performs diversion verification by configuring different control experiments.

[0159] For example, the intelligent customer service system provides a manually coded interface for experiments. When there is a new scene node suitable for executing an experiment, the interface is called to implant the new scene node and obtain the experimental configuration data of the new scene node. For existing scene nodes, the interface can also be called to update the experimental configuration file of the existing scene node.

[0160] For example, the experiment configuration module has an extended experiment scenario node interface, which is used to obtain the following parameters: experiment scenario node, which process and function the scenario node acts on in the intelligent customer service system, scenario preset experiment indicators, scenario preset experiment rule parameters, and scenario-specific experiment rule codes, etc. When adding an experiment scenario node or adding an experiment strategy in a certain scenario, you can use this interface to configure or encode it according to its definition specifications.

[0161] In some embodiments, all scenario nodes are running under the session intelligent allocation strategy scenario node, so they can be horizontally layered according to it, and the traffic of each layer (different scenario sub-nodes) of the same level is independent of each other (that is, the traffic of different scenario sub-nodes does not cross, which is equivalent to the vertical division of the traffic of different scenario sub-nodes, that is, orthogonal between layers and mutually exclusive within layers). For example, the experiment added to the session intelligent allocation strategy scenario node will not affect the traffic distribution of the sub-scenario under it, that is, the traffic between layers is orthogonal. Reference Figure 6 , Figure 6 It is a schematic diagram of the principle of flow orthogonality provided in an embodiment of the present application.

[0162] The experimental groups are divided into Group A and Group B. There are differences in the versions of intelligent customer service corresponding to Group A and Group B. For example, Group A is an improved intelligent customer service based on Group B. In the case of non-orthogonal traffic, the access traffic used up in each experimental layer will be maintained in the corresponding experimental group when entering the next layer. In the case of orthogonal traffic, the access traffic used up in each layer will be evenly redistributed to different experimental groups when entering the next layer.

[0163] In some embodiments, the experimental configuration data includes experimental rules in addition to the experimental scenario nodes. The types of experimental rules in the embodiments of the present application include: traffic allocation rules, experimental effective time, experimental traffic proportion, experimental fuses and experimental strategies.

[0164] 1. Traffic stickiness level (traffic allocation rules mentioned above): In the field of intelligent customer service, traffic stickiness is the correlation between traffic generated by users asking multiple questions to intelligent customer service. Traffic stickiness levels include the following levels: user level, question level, and conversation level. Among them, the session level means that the traffic of the same session is allocated to the same experiment, the user level means that the traffic of the same user is allocated to the same experiment within a period of time, and the question level means that the experiment is reallocated every time the user asks a question (that is, the traffic of one question is allocated to one experiment).

[0165] For example, when the intelligent customer service system handles user questions, it will create a session ID for each session. Therefore, as long as session-level traffic adhesion is configured, it is possible to directly determine whether the session ID has been assigned before. If it has been assigned, the historical assignment will be used directly without rerouting. The user-level processing logic is the same.

[0166] 2. Experiment validity period: Specifies that the experiment process is effective in a specific time period. For example, if you configure 2023-03-06 12:00:00 to 2023-03-07 12:00:00, it means that the experiment is only valid from 12:00 on March 6, 2023 to 12:00 on March 7, 2023.

[0167] 3. Experimental traffic proportion: defines the proportion of the access traffic for executing a certain experiment in the total access traffic. For example, if it is configured as 30%, it means that the access traffic passing through the experiment must account for 30% of the total access traffic.

[0168] 4. Experimental circuit breaker mechanism: a mechanism that terminates an experiment when certain conditions are met. The experimental circuit breaker mechanism is a solution used to ensure business continuity. Since not all experiments can run normally, you can configure the experimental circuit breaker mechanism to define how to handle errors in the experiment. The types of experimental circuit breakers include: abandonment on first failure, abandonment after {n} experiment failures, and traffic transfer.

[0169] Among them, the experiment is abandoned when it fails for the first time, which can be achieved in the following ways: if the current experiment fails in the first run, the experiment is abandoned directly and the traffic is returned; the experiment is abandoned after failure {n} times, which can be achieved in the following ways: when an experiment fails for n times, it is abandoned again and the traffic is returned, where n is a positive integer; the traffic transfer can be achieved in the following ways: when the experiment fails for n times, the experiment is abandoned first, and then the traffic configured for the experiment is transferred to other experiments. For example: the access traffic configured for experiment A is 15%, and the fuse mechanism for transferring the traffic to experiment B after failure is configured, then as long as experiment A fails, the 15% of the access traffic will be transferred to experiment B.

[0170] For example, an error occurs during the experiment. If the current error is caused by the execution of the experiment, then the experiment is considered to have failed.

[0171] 5. Experimental strategy: Specify the experimental content of the current experiment according to the scenario node. For example, if the experimental scenario node is a knowledge base answer scenario node, the experimental strategy includes: specifying the knowledge content and answers that need to be verified in the current experiment; if the experimental scenario node is an intent recognition algorithm scenario node, the experimental strategy includes: specifying a certain algorithm type or an algorithm indicator that needs to be verified.

[0172] To facilitate understanding, the data structure of the experimental scenario node and experimental rule configuration is illustrated by taking the conversation intelligent allocation strategy scenario node and the multi-round dialogue response scenario node as examples.

[0173] Refer to Table 1 below, where the configuration converted into JSON structure is [{scenario node: "strategy", name: "conversation intelligent allocation strategy scenario"}, {scenario node: "robot", name: "multi-round dialogue response scenario"}]. From the JSON structure, the scenario node configuration exists in the form of an array list, that is, multiple scenarios can be configured, and multiple scenario nodes are at the same level. The orthogonal and non-orthogonal (mutually exclusive) traffic between scenario nodes can refer to the above. Figure 6 .

[0174]

[0175] Table (1)

[0176] Refer to the following table (2), which is the experimental rule configuration.

[0177]

[0178] Table (2)

[0179] Among them, the experimental rule configuration in Table (2) is connected to the experimental scenario node configuration, that is, one experimental scenario node configuration corresponds to one experimental rule configuration, as shown in Table (2) Session Intelligent Allocation Strategy Scenario {traffic: "Scenario Node" Session, Valid Time: "2023030612:00:00-2023030712:00:00", Experimental Traffic Percentage: "30", Hystrix: "Stop", Rule: "Strategy_1"}, which defines the traffic adhesion of the experiment as session level, the valid time is 2023030612:00:00-2023030712:00:00, the experimental traffic percentage is 30%, and the experimental fuse is stopped at the first failure. The experimental strategy specifies a policy rule named strategy_1 under the current scenario node; while in the multi-round dialogue response scenario, only one experimental strategy is different. Here, a policy rule named repayment is specified, that is, the experiment will be carried out according to the experimental rule implemented by repayment. Among them, repayment refers to the English code of the multi-round dialogue of self-service processing for early settlement in the intelligent customer service. When the rule is configured with "repayment", it means that the experiment will be applied to the multi-round dialogue of self-service processing for "early settlement".

[0180] In some embodiments, in the field of intelligent customer service, there are different types of users, such as anonymous users, real-name users, familiar customers, and various tagged users, where familiar customers refer to the regular users mentioned above. In the process of executing the experiment, the session information and user information of the current incoming line are obtained, and the user information is used to identify whether the current incoming user is an anonymous user, a real-name user, a tagged user, or a familiar customer. The following experimental strategies can be executed for each type of user:

[0181] 1. When the user is anonymous, the anonymous user's access traffic is processed at the session level or question level.

[0182] For example, anonymous users do not have a unique user ID, so they can only configure the session or question stickiness level.

[0183] 2. When the user is a real-name user, the access traffic of the real-name user is processed strictly according to the session stickiness level configured in the experiment.

[0184] For example, there is a unique representation for identifying real-name users, and real-name users are applicable to different types of stickiness levels. For the session stickiness level, for anonymous users and real-name users with session stickiness levels configured, no matter how many rounds of questions and answers the user and the robot have, the experiment will not be triggered again in the current session as long as the experimental rules are selected when it is first triggered, that is, the same experiment will only trigger the experimental rules once in the same session. For example: in the "multi-round dialogue experiment scenario", if 30% of the traffic is configured to trigger the early settlement of self-service multi-round dialogues, then the experiment is selected when it is first triggered in the session to trigger the early settlement of self-service multi-round dialogues. No matter how many rounds of questions and answers the user and the robot have, the experiment will not execute the experimental rules again, and will only be conducted according to the first time. That is, the access traffic of real-name users and anonymous users with session stickiness levels configured is divided into corresponding experiments based on sessions.

[0185] For example, real-name users are also applicable to the user stickiness level. For the user stickiness level, the execution of the experimental rules is managed through the unique identifier used to identify the real-name user, that is, the experiment will only be triggered for the first time for the same user during the validity period of the experiment. Similarly, taking the example of the session stickiness level as an example, during the validity period of the experiment, no matter how many sessions and how many rounds of answers the user has, the experiment will only execute the experimental rules once, that is, as long as the user's first question can trigger the early settlement of multiple rounds of self-service dialogues, it can be triggered in the future.

[0186] 3. When the user is a tagged user, the tag rules of the tagged user will be followed, and the rules of the experimental configuration will be completely abandoned. For example: the intelligent customer service system provides many tags for real-name users, such as members, high-risk users, partners' users, and so on. Under a certain tag rule, a configuration parameter for whether to participate in the experiment and a configuration parameter for the proportion of participating in the experiment will be provided. The default parameter is to participate in the experiment. This configuration is defined in the experiment extension node interface, and each experimental node needs to implement the definition. For example: real-name users with the high-risk user tag do not participate in any experiments, while real-name users with the session tag are configured to have 30% of users participate in the experiment.

[0187] 4. When the user is a regular customer, it means that the regular customer mode has been entered, then all traffic of the current user will not be included in the experiment, that is, the access traffic of regular customers will not participate in any experiment.

[0188] In some embodiments, the level of adhesion used in the experiment can be freely set by the experiment initiator according to his own experimental scenario. For example, if a multi-round dialogue scenario experiment is required, then the question adhesion level may not be appropriate, because the question adhesion level is triggered according to the answer round, and it is very likely that the context output by the robot is not related. The corresponding adhesion level can be selected according to the experimental requirements.

[0189] In some embodiments, when the experiment is applied to the session intelligent allocation scenario, in order to ensure that the user can obtain the service of the same robot within a certain period of time under a certain business channel, a user adhesion mechanism is superimposed on the user-level session adhesion configured in the experiment (the traffic of the same user in the same session is divided into one experiment) (this mechanism is only allowed in the session intelligent allocation scenario node). The so-called user adhesion mechanism means that the same user can obtain the service of the same robot within a certain period of time (configurable), giving the user a business continuity experience.

[0190] In some embodiments, the experimental scene nodes are horizontally layered through a hash code and vertically bucketed to ensure orthogonal traffic between layers and mutual exclusion of traffic within a layer. When the access traffic passes through a certain experimental scene node, the experimental scene of the same layer as the current scene is obtained, and then the traffic is diverted according to the session-level traffic adhesion mechanism (session-level traffic allocation rule) configured in the experiment to ensure that the traffic of the same layer is mutually exclusive, and the bucket of the current scene is obtained according to the relationship between the scene nodes. The traffic of the scene nodes in the same bucket can coexist orthogonally.

[0191] In some embodiments, during the experiment execution process, the experiment execution module will package all the generated data and experimental results and hand them over to the experiment data recording module for cleaning and persistence. The experiment data recording module is used to clean the experimental data and store the cleaned results. Cleaning refers to obtaining corresponding indicators based on the experimental data and processing the experimental data.

[0192] For example, the experiment execution module is used to perform the following processing:

[0193] 1. Store the user information, traffic distribution information and other basic information corresponding to the current traffic during the experiment.

[0194] 2. When the user is a real-name user or a tag user, determine whether the current user meets the conditions for entering the familiar customer mode. If the conditions for the familiar customer mode are met, the current user type is updated to a familiar customer and the familiar customer validity period is updated. For example: determine whether the number of times the current user uses the intelligent customer service within the preset period is greater than the set value. If so, the current user type is converted to a familiar customer.

[0195] 3. Calculate indicators such as the arrival rate and conversion rate of the same bucket scenario during the traffic operation process; the arrival rate is the ratio of the access traffic passing through the scenario node to the total access traffic. For scenarios such as opening word recommendation, similar question recommendation, or fixed recommendation position recommendation, the conversion rate refers to the ratio between the number of questions and answers that users click on the recommendation and the total number of questions and answers recommended to users. In the self-service multi-round dialogue, the conversion rate is characterized by the probability of completing the self-service.

[0196] 4. Calculate the performance and comparison indicators such as conversion rate and difference rate of each experiment in the same layer scenario in different traffic proportions during the traffic operation process.

[0197] In some embodiments, reference Figure 5 , Figure 5 is an optional flow chart of the network application detection method provided in the embodiment of the present application; for example, based on Figure 4B The experimental configuration module 401, the experimental execution module 402 and the experimental data recording module 403 described in the above respectively correspond to the functions, which can achieve the following: Figure 5 The experiment of network application is described in detail below.

[0198] In step S501, a question raised by a user is received.

[0199] In step S502, it is determined whether the question raised by the user is a scene node.

[0200] If the result of step S502 is yes, step S503 is executed to read the experimental data. If the result of step S502 is no, step S501 is continued.

[0201] The user asks a question on the intelligent customer service robot, which determines whether the node through which the traffic passes is an experimental scenario node. If not, the experimental process is not executed on the traffic, and the robot continues to receive questions raised by other users and determines whether it is a scenario node; if so, the steps of reading experimental data are executed.

[0202] Step S503 is performed by the experiment configuration module 511 from the database 511 ( Figure 1A The experimental data is called from the database 500 in the execution.

[0203] For example, Figure 5 The experimental configuration module 511 is Figure 4B The experiment configuration module 401.

[0204] In step S504 , the experiment configuration module 511 performs user type detection.

[0205] If the experimental scenario node is configured with experimental data, the session information and user information of the current traffic are read to determine the user type and whether it is necessary to enter the familiar customer mode based on the read information, so as to determine the diversion mode.

[0206] In step S505 , the experiment module 512 is executed to perform traffic diversion.

[0207] In step S506 , the experiment execution module 512 executes the experiment strategy.

[0208] Figure 5 The execution experiment module 512 is Figure 4B The experiment execution module 402 delivers the experiment data, user information and diversion mode to the experiment execution module, diverts the execution experiment, and uniformly packages the generated data to the experiment data recording module 513.

[0209] After step S506 , the experimental data recording module 513 executes steps S507 to S509 . Figure 5 The recording experimental data module 513 is Figure 4B A module 403 for recording experimental data.

[0210] In step S507, various types of data are saved.

[0211] In step S508, the experimental process data is cleaned.

[0212] In step S509, the index is calculated.

[0213] For example, steps S507 to S509 may be performed synchronously. In some embodiments, the experimental results of each indicator in the form of various charts (pie charts, line charts, etc.) are sent to the experiment initiator (for example: a technician who develops intelligent customer service), and the experiment initiator can analyze whether the experiment meets expectations based on the experimental results.

[0214] The embodiment of the present application defines the configuration structure and diversion scheme of the AB experiment, and implements a special diversion strategy for independent operation, and records the process data, result data and some indicator data of the AB experiment. The detection method of the network application provided by the embodiment of the present application can quickly start an experiment that accommodates multiple versions of the network application and obtain its experimental results, thereby improving the experimental efficiency of the network application and being widely applicable to different network application experiments, thereby combining the experimental results to help business improvement optimization and enhance the front-end experience.

[0215] The following continues to describe the exemplary structure of the network application detection device 455 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2As shown, the software modules in the detection device 455 of the network application stored in the memory 450 may include: an experimental configuration module 4551, used to obtain the experimental configuration data of the network application, wherein the experimental configuration data includes the scene nodes used for the experiment and the traffic distribution rules of the scene nodes; an experimental execution module 4552, used to receive the access traffic passing through the scene nodes; the experimental execution module 4552, used to distribute the access traffic to different experiments based on the traffic distribution rules of the scene nodes; the experimental execution module 4552, used to execute the experimental process of the experiment based on the distributed access traffic, and obtain at least one of the intermediate data and execution results of the experiment.

[0216] In some embodiments, the scene nodes include N, the N scene nodes are distributed in M ​​levels, the access traffic of the i-th level is merged into the i+1-th level, and dispersed into the scene nodes of the i+1-th level, where 1≤i≤M-1.

[0217] In some embodiments, the types of traffic distribution rules include: session level, user level, and problem level; the experiment execution module 4552 is used to distribute the access traffic to the experiment corresponding to the session to which the access traffic belongs when the traffic distribution rule is at the session level; distribute the access traffic to the experiment corresponding to the user to which the access traffic belongs when the traffic distribution rule is at the user level; distribute the access traffic to the experiment corresponding to the problem to which the access traffic belongs when the traffic distribution rule is at the problem level.

[0218] In some embodiments, the experiment execution module 4552 is used to allocate experimental rules to the access traffic of the access users according to the user type of the access users before allocating the access traffic to different experiments based on the traffic allocation rules of the scene nodes, wherein the experimental rules are configured to allocate the access traffic to different experiments based on the user type and the traffic allocation rules.

[0219] In some embodiments, the experiment execution module 4552 is used to obtain user information of the visiting user before allocating the experiment rules to the access traffic of the visiting user according to the user type of the visiting user; and determine the user type of the visiting user according to the user information.

[0220] In some embodiments, the user types include anonymous users, tagged users, real-name users, and regular users, wherein the access frequency of regular users is greater than the frequency threshold; the experiment execution module 4552 is used to allocate the experimental rules to the access traffic of the access users according to the user types of the access users, including at least one of the following:

[0221] Allocate the access traffic of each session of anonymous users to different experiments; allocate the access traffic of real-name users to different experiments based on traffic allocation rules; allocate the access traffic of tagged users to different experiments based on the experimental rules corresponding to the tagged users; configure the access traffic of regular users to not execute the experimental process.

[0222] In some embodiments, the experimental configuration data also includes experimental fuse rules; the experimental execution module 4552 is used to terminate the experimental process in response to the experimental process satisfying a first fuse condition when executing the experimental process of the experiment based on the allocated access traffic, wherein the first fuse condition includes at least one of the following: the experimental process fails during the initial execution and the number of failures of the experimental process reaches a number threshold; in response to the experimental process satisfying a second fuse condition, terminate the current experimental process and configure the access traffic to other experimental processes, wherein the second fuse condition includes the number of failures of the experimental process reaching a number threshold.

[0223] In some embodiments, the experimental configuration data also includes an experimental strategy; an experimental execution module 4552 is used to, before executing the experimental process of the experiment based on the allocated access traffic, in response to the scenario node being a knowledge base response node, configure the knowledge content to be verified and the answer to the question in the experimental configuration data according to the experimental strategy; in response to the scenario node being an intent recognition node, configure the algorithm type to be verified in the experimental configuration data and the indicator corresponding to the algorithm type to be verified according to the experimental strategy.

[0224] In some embodiments, the experiment execution module 4552 is used to call the extension interface to obtain the node configuration parameters of the newly added scene node in response to the newly added scene node, wherein the node configuration parameters belong to the experiment configuration data, and the node configuration parameters include at least one of the following: the role of the scene node, the experiment indicators of the scene node, the pre-configured experiment rules of the scene node, and the code corresponding to the pre-configured experiment rules; in response to the experiment content of the existing scene node to be updated, call the extension interface to obtain the updated experiment content corresponding to the scene node.

[0225] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the electronic device executes the network application detection method provided in the embodiment of the present application.

[0226] The embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the network application detection method provided by the embodiment of the present application, for example, FIG. 3A to FIG. 3C A method for detecting a network application is shown.

[0227] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0228] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0229] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file storing other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0230] As an example, the executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0231] In summary, through the experimental configuration data of multiple scene nodes obtained by the embodiment of the present application, the access traffic of scene nodes at different levels in the multiple scene nodes is orthogonal to each other, and the access traffic of scene nodes at the same level in the multiple scene nodes is mutually exclusive, so that different scene nodes can be tested during the experiment. During the experiment, the user's access traffic is mutually exclusive in the same experimental layer to maintain the control effect of the experimental data of different experimental groups. The access traffic in different experimental layers is orthogonal, which ensures the accuracy of the experimental data. By executing the experimental process of the access traffic based on the experimental configuration data, recording at least one of the intermediate data and the execution result during the execution process, the comprehensiveness of the network application experiment is improved, and it is possible to implement experiments on multiple different scene nodes at the same time, thereby improving the efficiency of the network application experiment.

[0232] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A method for detecting a network application, characterized in that: The method comprises: Acquire experimental configuration data of the network application, wherein the experimental configuration data includes scene nodes used for the experiment and traffic distribution rules of the scene nodes; Receiving access traffic passing through the scene node; Based on the traffic distribution rule of the scenario node, the access traffic is distributed to different experiments; The experimental process of the experiment is executed based on the allocated access traffic to obtain at least one of intermediate data and execution results of the experiment.

2. The method according to claim 1, characterized in that: The scene nodes include N, where N is a positive integer. The N scene nodes are distributed in M ​​levels. The access traffic passing through the i-th level is merged into the i+1-th level and dispersed into the scene nodes of the i+1-th level, wherein 1≤i≤M-1.

3. The method according to claim 1, characterized in that The traffic distribution rules include: session level, user level, and problem level; The traffic distribution rule based on the scene node distributes the access traffic to different experiments, including: When the traffic allocation rule is at the session level, the access traffic is allocated to the experiment corresponding to the session to which the access traffic belongs; When the traffic distribution rule is user-level, the access traffic is distributed to the experiment corresponding to the user to which the access traffic belongs; When the traffic allocation rule is at the problem level, the access traffic is allocated to the experiment corresponding to the problem to which the access traffic belongs.

4. The method according to claim 1, characterized in that: Before allocating the access traffic to different experiments based on the traffic allocation rule of the scenario node, the method further includes: According to the user type of the access user, the access traffic of the access user is allocated an experimental rule, wherein the experimental rule is configured to allocate the access traffic to different experiments based on the user type and the traffic allocation rule.

5. The method according to claim 4, characterized in that Before allocating the experimental rule to the access traffic of the access user according to the user type of the access user, the method further includes: Obtaining user information of the accessing user; The user type of the accessing user is determined according to the user information.

6. The method according to claim 4, characterized in that The user types include: anonymous users, tagged users, real-name users, and regular users, wherein the access frequency of the regular users is greater than the frequency threshold; The experimental rule for allocating the access traffic of the access user according to the user type of the access user includes at least one of the following: Allocate the access traffic of each session of the anonymous user to different experiments; Allocating the access traffic of the real-name user to different experiments based on the traffic allocation rule; Allocate the access traffic of the tagged user to different experiments based on the experimental rules corresponding to the tagged user; The access traffic of the regular users is configured not to execute the experimental process.

7. The method according to any one of claims 1 to 4, characterized in that: The experimental configuration data also includes experimental fuse rules; When the experimental process of performing the experiment based on the allocated access traffic is performed, the method further includes: In response to the experimental process satisfying a first fuse condition, terminating the experimental process, wherein the first fuse condition includes at least one of the following: the experimental process fails when it is first executed, and the number of failures of the experimental process reaches a number threshold; In response to the experimental process satisfying a second fuse condition, the current experimental process is terminated and the access traffic is configured to other experiments, wherein the second fuse condition includes that the number of failures of the experimental process reaches a threshold number.

8. The method according to any one of claims 1 to 4, characterized in that: The experiment configuration data also includes an experiment strategy; Before executing the experimental process of the experiment based on the allocated access traffic, the method further includes: configuring the experimental content and experimental indicators corresponding to the scene node according to the experimental strategy.

9. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: In response to the newly added scene node setting, calling the extension interface to obtain the node configuration parameters of the newly added scene node, wherein the node configuration parameters include at least one of the following: the role of the scene node, the experimental index of the scene node, the preconfigured experimental rule of the scene node, and the code corresponding to the preconfigured experimental rule; In response to the node configuration parameters of an existing scene node being to be updated, the extended interface is called to obtain updated node configuration parameters corresponding to the scene node.

10. A network application detection device, characterized in that: The device comprises: An experiment configuration module, used to obtain experiment configuration data of a network application, wherein the experiment configuration data includes a scene node used for the experiment and a traffic distribution rule of the scene node; An experiment execution module, used for receiving access traffic passing through the scene node, and distributing the access traffic to different experiments based on the traffic distribution rule of the scene node; The experiment execution module is further used to execute the experiment process of the experiment based on the allocated access traffic to obtain at least one of the intermediate data and execution results of the experiment.

11. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions; The processor is used to implement the network application detection method according to any one of claims 1 to 9 when executing the computer executable instructions or computer programs stored in the memory.

12. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the method for detecting network applications according to any one of claims 1 to 9 is implemented.

Images