Intelligent-based business scenario identification method and device, and gray scale diversion method and device
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ALIPAY COM CO LTD
- Filing Date
- 2023-04-23
- Publication Date
- 2026-08-07
AI Technical Summary
[0008]由于目前没有形成通用业务场景的自动识别过程,因此,在一些大型网站服务系统中,需要根据业务场景来进行灰度引流等时,往往只能通过强人工专家经验来实现,因此会出现效率低、成本高、统计不全面的问题
[0007] This specification provides one or more embodiments of an intelligent business scenario identification method, apparatus, device, and storage medium to solve the following technical problems:
Smart Images

Figure CN116467520B_ABST
Abstract
Description
Technical Field
[0001] This specification relates to the technical field of artificial intelligence, and in particular to a business scenario recognition method and device based on intelligence, as well as a grayscale traffic redirection method and device. Background Technology
[0002] Some large website service systems encompass a vast array of business scenarios. During development, for example, when a new version of a service is about to be released, a small portion of traffic is redirected to the new version for testing. This allows for the timely detection of program issues and effectively prevents widespread failures. This release method is called canary release. How to conduct a canary release depends on the business scenario. It requires filling in canary traffic redirection rules based on the business scenario, then redirecting traffic to a subset of machines within the system. These machines are designated as the canary environment, where testing is then conducted.
[0003] Furthermore, some services involve not only a single business scenario, but also a collection of call chains generated by the combination of multiple business-related scenarios. For example, the scenario of adding a shopping cart, submitting an order, and making a payment on a shopping platform. Therefore, full-chain testing of call chains involving multiple business-related scenarios is essential for business scenarios.
[0004] Identifying the business scenarios within a large website's service system has always been a crucial but unresolved issue. Due to the inability to guarantee the quality of business scenario allocation, many questions arise during actual testing. For example, are the test cases sufficient to cover all business scenarios? Are the traffic acquisition scenarios during the gray-scale rollout adequate? What low-traffic scenarios are missing? Is automated testing comprehensive? Is 100% code coverage sufficient? Do the business scenarios included in online business management cover all business scenarios? And so on.
[0005] All of these issues rely on a relatively objective denominator for the business scenario. Currently, the denominator for this business scenario is typically manually determined by the business owner based on their expert experience. This process is often a one-time processing based on specific conditions, business logs, or offline tables, resulting only in a denominator specific to that business scenario. There is no automated analysis and derivation process for a general business scenario denominator. Furthermore, the solution for the specific business scenario denominator is not obtained through a systematic approach that minimizes human intervention. This compromises the timeliness, accuracy, and statistical objectivity of the data. Moreover, relying heavily on intensive human expert experience can lead to biases in terms of time frame and individual scope.
[0006] Since there is currently no automated analysis and derivation process for the denominator of general business scenarios, some large website service systems often have to rely on the experience of human experts to carry out tasks such as gray-scale traffic acquisition based on business scenarios. This often results in low efficiency, high cost, and incomplete statistics. Summary of the Invention
[0007] This specification provides one or more embodiments of an intelligent business scenario identification method, apparatus, device, and storage medium to solve the following technical problems:
[0008] Since there is currently no automated identification process for general business scenarios, some large website service systems often have to rely on the experience of human experts to carry out tasks such as gray-scale traffic generation based on business scenarios. This often results in low efficiency, high cost, and incomplete statistics.
[0009] The intelligent business scenario recognition method provided in this embodiment of the invention includes the following steps:
[0010] Acquire business data and parse the business data to obtain business parameters and the corresponding values of the business parameters;
[0011] The business parameters are pre-filtered to obtain preliminary screening parameters containing business semantics. The preliminary screening parameters include continuous parameters and enumerated parameters.
[0012] Based on the distribution fitting results of the continuous parameters according to the preset statistical distribution model, an enumerated type of quantized parameters is generated;
[0013] A pre-trained target semantic recognition model is used to identify the effective parameters in the enumerated parameters and the quantized parameters, wherein the effective parameters are parameters containing business semantics;
[0014] Clustering operations are performed on the effective parameters to identify the business scenarios corresponding to the business data.
[0015] This invention provides an intelligent business scenario recognition device, comprising:
[0016] An acquisition unit is used to acquire business data and parse the business data to obtain business parameters and the corresponding values of the business parameters;
[0017] The pre-filtering unit is used to perform data pre-filtering processing on the business parameters to obtain preliminary screening parameters containing business semantics. The preliminary screening parameters include continuous parameters and enumerated parameters.
[0018] The generation unit produces the distribution fitting results of the continuous parameters according to the preset statistical distribution model, and generates enumerated quantized parameters.
[0019] The parameter recognition unit is used to identify valid parameters containing business semantics in the enumerated parameters and the quantized parameters by using a pre-trained target semantic recognition model.
[0020] The scene recognition unit is used to perform clustering operations on the effective parameters to identify the business scene corresponding to the business data.
[0021] This invention also provides a method for gray-scale traffic redirection based on business scenarios, including:
[0022] Obtain grayscale traffic redirection requests, wherein the grayscale traffic redirection requests include business request data;
[0023] Based on the business request data, the above-mentioned intelligent business scenario identification method is used to identify the business scenario corresponding to the business request data.
[0024] The gray-scale traffic redirection rules are determined based on the business scenario, and traffic is redirected according to the gray-scale traffic redirection rules.
[0025] This invention also provides a gray-scale traffic redirection device based on business scenarios, comprising:
[0026] The acquisition unit is used to acquire grayscale traffic redirection requests, wherein the grayscale traffic redirection requests include business request data;
[0027] The identification unit is used to identify the business scenario corresponding to the business request data by adopting the intelligent business scenario method described above.
[0028] The traffic redirection unit is used to determine the gray-scale traffic redirection rules based on the business scenario, and to redirect traffic according to the gray-scale traffic redirection rules. Attached Figure Description
[0029] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0030] Figure 1 This is a schematic diagram illustrating an application scenario of the intelligent business scenario recognition method provided in one or more embodiments of this specification;
[0031] Figure 2a This is a flowchart illustrating one or more embodiments of the intelligent business scenario recognition method provided in this specification.
[0032] Figure 2b This is a flowchart illustrating a gray-scale traffic redirection method based on a business scenario provided in one or more embodiments of this specification.
[0033] Figure 3a This is a schematic diagram of a smart business scenario recognition device provided in one or more embodiments of this specification;
[0034] Figure 3b This is a schematic diagram of a grayscale traffic redirection device based on a business scenario provided in one or more embodiments of this specification;
[0035] Figure 4 This is a schematic diagram of the architecture of a terminal based on intelligent business scenario recognition provided in one or more embodiments of this specification;
[0036] Figure 5a This is an interactive schematic diagram of one or more embodiments of the intelligent business scenario recognition method provided in this specification;
[0037] Figure 5b This is a schematic diagram of the dictionary setup provided in one or more embodiments of this specification;
[0038] Figure 5c This is another flowchart illustrating the intelligent business scenario recognition method provided in one or more embodiments of this specification. Detailed Implementation
[0039] This specification provides an intelligent grayscale diversion method, apparatus, device, and storage medium.
[0040] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.
[0041] For example, refer to Figure 1A business scenario recognition system based on intelligence can include servers and terminals. Besides the terminals shown in the diagram, the system may also include multiple other terminals; the specific number of terminals is not limited here. Servers and terminals can be connected via a communication network, which can include wireless and wired networks. The wireless network includes one or more combinations of wireless wide area networks (WANs), wireless local area networks (LANs), wireless metropolitan area networks (MANs), and wireless personal networks (PANs). Network entities such as routers and gateways are included in the network, but are not shown in the diagram. Terminals can interact with the server and perform applications through the communication network. For example, a terminal can send a request for gray-scale traffic redirection to the server via the communication network, and the server can return a machine ID that matches the gray-scale traffic redirection business scenario to the terminal.
[0042] The intelligent business scenario recognition system may include a business scenario denominator mining device, which may be integrated into a server. The server may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.
[0043] like Figure 1 As shown, the server can be used to acquire business data and parse the business data to obtain business parameters and their corresponding values; perform data pre-filtering on the business parameters to obtain preliminary screening parameters containing business semantics, the preliminary screening parameters including continuous parameters and enumerated parameters; generate enumerated quantitative parameters based on the distribution fitting result of the continuous parameters according to a preset statistical distribution model; use a pre-trained target semantic recognition model to identify effective parameters containing business semantics in the enumerated parameters and the quantitative parameters; perform clustering operations on the effective parameters to identify the business scenario corresponding to the business data.
[0044] In this invention, business scenario denominator mining can be understood as identifying all business scenarios within the service system. The solution of this invention can form an automated identification process for general business scenarios. An automatic data collection program can be deployed in the service system to continuously collect real-time business data. The automated identification process is implemented through a business scenario denominator mining device integrated in the server. New business scenarios are continuously identified through a trained semantic recognition model. The business scenarios of this invention are identified in a systematic way that minimizes human intervention. Real-time business data can be collected for identification, thus ensuring the timeliness, accuracy, and statistical objectivity of the data. Using a trained model for identification reduces reliance on human expert experience and minimizes deviations in timeframes and individual scopes.
[0045] The terminal can be a mobile phone, tablet, laptop, or other terminal device, or a wearable device, smart speaker, smart home appliance, or other smart terminal, but is not limited to these. The terminal and server can be connected directly or indirectly via wired or wireless communication, which is not limited herein. The terminal can install various applications needed by the user to achieve interaction with the user. The terminal can collect user-input grayscale traffic redirection requests and send them to the server for grayscale traffic redirection. The terminal can display some processing results, allow for manual intervention and adjustment, or allow input of parameters at certain steps in the process to improve accuracy.
[0046] The above Figure 1 The example given is merely a system architecture instance for implementing embodiments of the present invention, and embodiments of the present invention are not limited to those described above. Figure 1 The system structure shown is illustrated, and various embodiments of the present invention are proposed based on this system architecture.
[0047] The following sections provide detailed descriptions of each example. It should be noted that the sequence numbers of the following embodiments are not intended to limit the preferred order of the embodiments.
[0048] This embodiment will be described from the perspective of an intelligent business scenario denominator mining device, which can be integrated into a server with storage units and a microprocessor for computing power.
[0049] Figure 2a This document provides a flowchart illustrating an intelligent business scenario identification method for one or more embodiments. This method can guide various steps in the R&D process and influence decision-making. For example, it can guide the establishment of a grayscale environment in the grayscale traffic redirection step and influence the direction of traffic redirection.
[0050] Figure 2a The process may specifically include the following steps:
[0051] This specification provides one or more embodiments of an intelligent business scenario denominator mining method, apparatus, device, and storage medium to solve the following technical problems:
[0052] 101. Obtain business data and parse the business data to obtain business parameters and the corresponding values of the business parameters.
[0053] The business data originates from the request and return data generated during the processing of business requests by the service system of this invention. For example, during payment, the terminal sends a request to the server, and the server returns data to the terminal. Specifically, business data from real online applications can be collected using a data acquisition engine. This business data is then downsampled and imported into the system. Downsampling, also known as extraction, involves taking samples from a sequence of samples at intervals of several samples, resulting in a new sequence that is a downsampled version of the original sequence. Since the amount of real online business data is enormous, only a portion of the business data needs to be extracted for calculation. Different data interfaces can be set up to continuously collect business data from multiple interfaces.
[0054] refer to Figure 5c In one embodiment, the collected business data can be imported into an offline ODPS (Open Data Processing Service) for storage and subsequent computation.
[0055] ODPS is a distributed processing service designed for TB / PB level data with low real-time requirements. It primarily serves the storage and computation of batch structured data, providing solutions for massive data warehouses and analytical modeling services for big data.
[0056] The following will describe in detail the parsing of the collected business data in this embodiment. The specific steps are as follows:
[0057] Establish a unified data underlying intervention standard to unify the business data source format of the gray component into the data source format of the doom component, and standardize it into standard business data.
[0058] Flatten all input parameters (including main call and sub-call) into one dimension, where "[number]" (e.g., [1], [2]) represents which input parameter it is;
[0059] Filter input parameter position markers: Replace "[number]" with "[*]", fields with the same name can actually represent the same input parameter.
[0060] Since the collected business data is in binary format, it first needs to be parsed. The parsed data contains all the key business data parameters.
[0061] refer to Figure 5c The business data is decomposed into key-value pairs according to the path, resulting in multiple different types of parameters, as follows:
[0062] 1. Continuous values without business meaning can be discarded: such as timestamps, random numbers from external requests, etc.
[0063] 2. Enumeration values without business meaning can be discarded: such as Chinese and English characters that generally represent names or titles.
[0064] 3. Continuous values containing key business implications: such as discount amounts. These types of values generally require solving for extreme values using a PDF or determining their existence.
[0065] 4. Enumerated values containing key business meanings: such as business status (WAIT_FOR_RATE, SUCCESS), key business type ("product":"FP_SENIOR_PARTNER", "bizSubType":"104"), etc. These types of values are generally used for initial screening after traffic data has accumulated to a certain amount, based on statistical distribution patterns and a reasonable ratio. These parameters are often key input data for calculating business characteristics.
[0066] 5. IDs containing key business meanings: such as user UID, merchant PID, application APPID, etc. These IDs are often associated with other key business attributes, such as user type, merchant type, etc. This type of ID often has a certain standardized format and needs to be specially marked according to the standardized format during processing.
[0067] Here, Key refers to the parameter, and Value refers to the value of the parameter. For example, "product":"FP_SENIOR_PARTNER", "bizSubType":"104", where key="product" or "bizSubType", and value="FP_SENIOR_PARTNER" or "104".
[0068] 102. Perform pre-filtering on the business data to obtain initial screening parameters containing business semantics. The initial screening parameters include continuous parameters and enumerated parameters.
[0069] Due to the large number and complexity of business data parameters, there are a large number of invalid parameters. Approximately 90% or more of the data in the business data are continuous values without business meaning, such as timestamps, random numbers for external requests, and enumeration values without business meaning, such as Chinese and English characters that generally represent names or titles.
[0070] Before training the model, these invalid parameters need to be filtered out to reduce the computational load of the model and improve its recognition accuracy.
[0071] The pre-filtration process of this invention mainly includes:
[0072] 1) Static code analysis refers to the analysis of source code without executing the computer program to find invalid parameters.
[0073] 2) User-defined data interception: Users have accumulated a large number of test cases based on their historical experience. We store this human expert experience on the server to filter out invalid parameters.
[0074] 3) Feature-based data interception specifically refers to filtering out invalid parameters from the input parameters of the entry method based on data statistical distribution through three methods: incremental judgment, historical traffic patterns, and expert experience. The details are as follows:
[0075] The incremental judgment method is used to filter the enumerated parameters among many unknown parameters: radio will not increase with the accumulation of business data traffic (in this embodiment, the incremental judgment method is selected between "incremental judgment" and "threshold judgment", of course, those skilled in the art can choose other methods according to actual needs).
[0076] Filtering based on historical traffic patterns: 0 <radio<20%,val_cnt> 2, tot_cnt>10, supports user modification.
[0077] Filtering is performed based on expert experience: invalid parameter fields and valid parameter fields manually marked by the user.
[0078] It's worth noting that when identifying business scenarios, to avoid an excessive number of scenarios, we use an incremental judgment method to filter out enumerated parameters from the numerous unknown parameters. The incremental judgment method refers to the principle that as the volume of traffic increases, the number of possible values tends to stabilize. It's conceivable that the incremental judgment method is only applicable when the volume of traffic is large. Therefore, in practical applications, parameters with a small volume of traffic and non-enumerated parameters are often filtered out.
[0079] Specifically, we define traffic of less than 100 transactions per hour across the entire cluster as low traffic, and in offline data, we define traffic occurring less than 8 times as low traffic.
[0080]
[0081] The default minimum number of traffic entries is 100, and the recommended recording period is over 30 days. The online sampling ratio is 10,000:1. Therefore, we define traffic that appears less than 8 times in offline data as small traffic. Small traffic faces two main challenges: first, it is characterized by high volatility (strong noise), high proportion, and low frequency and volume; second, some small traffic transactions are mixed in with large traffic transactions, such as niche channels within payment services, making it difficult to distinguish between small traffic and noise from large traffic transactions.
[0082] To address the issue of low traffic volume, for high-traffic interfaces, we still use the general incremental judgment method to filter and obtain enumerated parameters. To avoid leaving low traffic volume within high-traffic interfaces, we use a target semantic model to intelligently judge the filtered traffic. For low-traffic interfaces, we only retain parameters where the number of traffic containing the key is less than the threshold, and the number of values corresponding to the key is less than the threshold.
[0083] 103. Generate enumerated quantization parameters based on the distribution fitting results of the continuous parameters according to the preset statistical distribution model.
[0084] refer to Figure 5c Currently, all business parameters can be categorized into the following types:
[0085] 1) Pure whitelist: The important parameters of the application system are accumulated by human experts in the form of a structure. They generally have strong business attributes and are effective parameters. We extract them and take their commonalities, such as bizIdentity and scene.
[0086] 2) Pure blacklist: The blacklist is a structure that is accumulated in the application system by human experts. Generally, people will not assign it as a business attribute, so it is an invalid parameter. We extract it to take its common characteristics, such as amount, cent, email, host-name.
[0087] 3) Pure numeric type: such as '[0-9]', which can be further divided into monetary type, fixed length type such as pid / uid / cid type, short value, and others.
[0088] 4) Pure English characters: such as '[a-zA-Z]'. If the feature value is finite, it belongs to the enumeration class. Generally, the enumeration class has the strongest business attributes. We need to use a blacklist and whitelist intelligent recognition model with expert experience to distinguish whether the enumeration parameter contains business semantics among all enumeration parameters.
[0089] 5) Characters containing Chinese characters, such as u'[\u4e00-\u9fa5]', can be discarded.
[0090] 6) Contains email addresses, MAC addresses, IP addresses, or other types of random strings: can be discarded.
[0091] Experiments revealed that all continuous parameters exhibit a long tail characteristic. By fitting over 20 statistical distribution models to the continuous parameters, the norm and gamma distributions showed the best fit and conformed to natural laws.
[0092] In one embodiment, an enumerated quantization parameter is generated based on the distribution fitting result of the continuous parameter using a preset statistical distribution model. This may specifically include the following steps:
[0093] The continuous parameters are fitted to a distribution based on a preset statistical distribution model to obtain the distribution fitting result;
[0094] Based on the distribution fitting results, determine the head range, normal range, and tail range of the continuous parameter;
[0095] The head value range, the normal value range, and the tail value range are used as the quantization parameters corresponding to the continuous parameter.
[0096] The following example uses the parameter avg_frequency_d to illustrate the fitting process:
[0097] The reasonable range for avg_frequency_d is x=[0.07,0.79], with x<0.07 at the beginning and x>0.79 at the end, and median med=0.158; the reasonable range for per_terminal_trd_cnt_d is x=[3.17e-17,11222], with x<3.17e-17 at the beginning and x>11222 at the end, and median med=9.2.
[0098] For parameters conforming to the norm distribution, we know the standard "3σ" rule:
[0099] 1σ principle: The probability that the value is distributed in (μ-σ, μ+σ) is 0.6526;
[0100] 2σ rule: The probability that the value is distributed in (μ-2σ, μ+2σ) is 0.9544;
[0101] The 3σ rule states that the probability of a value falling within the range of (μ-3σ, μ+3σ) is 0.9974.
[0102] Our outlier selection principle follows the basic idea of "low-probability events" in hypothesis testing. Low-probability events typically refer to events with a probability of occurrence less than 5%, considered almost impossible to occur in a single trial. Therefore, when the probability of a value falling outside (μ-3σ, μ+3σ) is less than 0.3%, the corresponding event is often considered unlikely to occur in practical problems. Thus, the interval (μ-3σ, μ+3σ) can generally be considered the reasonable range of values for this parameter.
[0103] For parameters that follow a Gamma distribution, assuming the parameter values are a set of continuous random variables x, if its probability density function... It conforms to the following formula:
[0104]
[0105] This probability density function is called the gamma (Г, Gamma) distribution, denoted as . In a Gamma distribution, the parameter α is called the shape parameter (α>0), and β is called the inverse scaling parameter (β>0). The variance and mean of a continuous variable then satisfy:
[0106] , =
[0107] α and β are calculated from the mean and variance as follows:
[0108] ,
[0109] Based on the given probability values, we solve for the quantiles corresponding to ICDF, which are the threshold points for outlier values.
[0110] Therefore, we fit the original data to a statistical distribution model and calculate the data density based on the statistical distribution model. We take the first 5% of the data as the head range, the middle 90% as the normal range, and the last 5% as the tail range.
[0111] 104. Using a pre-trained target semantic recognition model, identify the effective parameters containing business semantics among the enumerated parameters and the quantized parameters.
[0112] In embodiments of the present invention, the target semantic model is a combination of the word2vec algorithm and the xgboost model, and may specifically include the following steps:
[0113] According to the preset characters, the enumerated parameters, the quantization parameters, and the terminal parameters are respectively divided into atomic units and molecular units, wherein the molecular unit includes at least two atomic units, and the terminal parameter is the last parameter of the business parameters;
[0114] A preset mapping sub-model is used to map the atomic units, molecular units, and the values of each parameter into atomic multidimensional vectors and molecular multidimensional vectors of different dimensions, respectively.
[0115] By concatenating the atomic multidimensional vectors and molecular multidimensional vectors corresponding to each parameter, a multidimensional semantic vector corresponding to each parameter is obtained.
[0116] A preset classification tree sub-model is used to identify the parameter labels corresponding to the multidimensional semantic vector, and the parameter labels include valid parameter labels and invalid parameter labels;
[0117] The effective parameters in the enumerated parameters and the quantization parameters are determined based on the parameter labels.
[0118] The last parameter refers to the last parameter of the path, i.e., last_key.
[0119] Before practical application, sample data can be collected for training. The training steps for the XGBoost model are as follows:
[0120] Collect business data, and parse, pre-filter, and perform distribution fitting on the business data to obtain sample parameters;
[0121] Based on whether they contain business semantics, the sample parameters are classified into valid parameters and invalid parameters and labeled to obtain a training sample set. The sample parameters in the training sample set carry preset labels.
[0122] The initial classification tree sub-model is trained using the training sample set to obtain the preset classification tree model.
[0123] The specific steps for training the initial classification tree sub-model using the training sample set to obtain the preset classification tree model include the following:
[0124] The training sample set is input into the first tree in the initial classification tree sub-model for training to obtain the first score value;
[0125] Obtain the first residual value between the first score and the preset label;
[0126] The first residual value is used as the preset label of the second tree and iterative training is performed until each tree is trained, resulting in the trained preset classification tree sub-model.
[0127] refer to Figure 5c and Figure 5a The main idea of the XgBoost model is to integrate a large number of weak classifiers (based on a tree structure) into a strong classifier with high accuracy. The XgBoost model primarily uses feature dimensions to partition the sample space, adding trees through continuous feature splitting, and then using this model to predict samples. Based on the sample's features, it falls into the corresponding leaf node in each tree. The scores corresponding to the leaf nodes are summed to obtain the predicted value for the sample. Thus, the test can be input into each tree in the XgBoost tree model to obtain k scores. These k scores are then summed to obtain the first test score, which is used as the probability value of the effective parameters.
[0128] In practical applications, whitelist data and corresponding valid parameter labels, along with black sample data and corresponding black sample labels, are combined to form the training sample dataset.
[0129] In the field where this model is applied, black sample data can be considered invalid parameters, while white sample data can be considered valid parameters. Both black and white sample data can be obtained based on expert labeling results.
[0130] Before training, a parameter lexicon can be built within the service website of this invention based on the expert's labeling results, and the frequency of parameters can be counted. The specific steps are as follows:
[0131] According to preset fields, the sample parameters are divided into atomic units and molecular units, and the atomic units and molecular units are used as parameter lexicon;
[0132] The frequency of each atomic unit and each molecular unit in the parameter lexicon being marked as a valid parameter or an invalid parameter is counted to obtain the parameter frequency;
[0133] The initial classification tree sub-model is trained based on the frequency of the sample parameters.
[0134] Among them, the classification tree sub-model, namely the xgboost model, constructs a decision tree based on classification features, and uses the classification features as split points to obtain child nodes.
[0135] In this solution, the attribute of "word" is special; it is a "parameter with a path," which is significantly different from existing Chinese-English thesauruses on the market and cannot be directly quantified. Therefore, this invention establishes a specialized thesaurus. The specific solution is as follows:
[0136] In the technical solution of this invention, most parameters follow the camelCase naming convention. We break down the parameters into the smallest different particle structures by using the first capital letter (or underscore) and separators. The smallest atom is called an atomic structure, such as order, chashier, etc. The structure formed by different combinations of atoms is called a molecular structure, such as orderTermMap, chashierAssetInfos, etc. The values of the parameters are then processed into a val dictionary using the same processing method.
[0137] Since the last character of most feature fields, i.e. the terminal parameter, often has strong business attributes, we have also built a dedicated molecular lexicon and atomic lexicon for last_key.
[0138] refer to Figure 5b We define the sample parameter as a combination of multiple atoms and molecules. The sample parameter in the figure can be broken down into the following four molecules, corresponding to different molecules in the molecular vocabulary. We have already defined the number of times each molecule appears in the blacklist and whitelist in the molecular vocabulary (see the explanation below for specific feature definitions). After assembly, we can obtain the multidimensional feature representation of each molecule result. Then, we further break down the molecular structure into the smallest atomic results, corresponding to different atoms in the atomic vocabulary. Based on the number of times each atom appears in the blacklist and whitelist in the atomic vocabulary, we can obtain the multidimensional feature representation of each atomic result after assembly. Subsequently, we use similar methods to assemble the multidimensional features of the val vocabulary, as well as the molecular multidimensional feature representation and atomic multidimensional feature representation of the last_key dimension, and concatenate all features into a large wide table to jointly represent the business semantics of the parameter.
[0139] In the vocabulary, features are stored in the form of {key:value}. In the molecular vocabulary, {"product": 54320432}, where key is the parameter and value is the corresponding vector size. In the VAL vocabulary, {"FP_SENIOR_PARTNER": 23232354}, where key is the value of the parameter and value is the corresponding vector size.
[0140] refer to Figure 5b This scheme determines features across multiple dimensions based on parameter frequency and parameter lexicon. The definitions and explanations of some features are shown in the table below:
[0141]
[0142] 105. Perform clustering operations on the effective parameters to identify the business scenarios corresponding to the business data.
[0143] In one embodiment, the specific steps may include:
[0144] Calculate the parameter distance between the effective parameter and the initial scene category center, and update the scene category center based on the parameter distance;
[0145] If the distance between the initial scene category center and the updated scene category center is not less than a preset threshold, repeat the above steps until the distance between the initial scene category center and the updated scene category center is less than the preset threshold.
[0146] If the distance between the initial scene category center and the updated scene category center is less than a preset threshold, the business scene corresponding to the business data is determined based on the scene category center.
[0147] The initial scenario category center is calculated based on the sample parameters, which are divided into multiple categories, each representing a business scenario.
[0148] Among them, the most classic and widely used clustering algorithm is K-Means unsupervised clustering. Its principle is to pre-divide n objects into k clusters so that the clusters have a high degree of similarity, and the center point is the average value of all objects in a cluster.
[0149] However, there are two problems when using the K-Means algorithm, as follows:
[0150] 1) How to choose the objective function? Generally, Euclidean distance or cosine distance of vectors is used. Then, how do we evaluate the distance between two feature values? Assuming there's only a single-dimensional feature, transaction state (TRADE_SUCCESS, WAIT_BUYER_PAY), the values should be infinite, representing two completely different scenario types. Therefore, they cannot be directly quantified as 0 or 1 for Euclidean distance calculation. Furthermore, when the dimension expands to N dimensions, each different combination of values should be completely orthogonal in the N-dimensional vector space. In addition, there's a more complex issue: how to combine enumerated variables and numerical variables for calculation?
[0151] 2) How to select the K value for clustering. K-class clustering algorithms require the final K value to be specified in advance by the user, and then the inter-cluster performance of the final clusters is evaluated. In order to obtain a better K value, it is necessary to continuously iterate and recursively solve for a better K value, which is often extremely time-consuming.
[0152] For question 1), we adopted the k-prototype algorithm for clustering mixed attribute datasets, which is a combination of k-means and k-modes algorithms. k-means is used for clustering numerical data, while k-modes is used for clustering categorical data. Let X and Y be two objects described by m-dimensional categorical attributes, and Z be an object described by n-dimensional numerical attributes. The dissimilarity measure between X, Y, and Z can be defined by the combined distance of the corresponding attribute categories of the three objects:
[0153]
[0154] in:
[0155]
[0156] k-prototype divides n objects into k clusters, ensuring low dissimilarity (or distance) within each cluster. The dissimilarity is obtained by a weighted average of k-means and k-mode algorithms, i.e.:
[0157]
[0158] in, Distance representing the mixed-attribute dataset, Represents the distance of numerical data. Represents the distance between categorical data. This represents assigning appropriate weights to categorical data. The ultimate goal of clustering algorithms is to minimize the result of the following formula:
[0159]
[0160] The steps for calculating kprototype are as follows:
[0161] (1) Select k initial parameters from the sample parameters as initial class centers;
[0162] (2) Calculate the dissimilarity between each remaining sample parameter and the cluster center, and assign this sample parameter to the cluster with the smallest dissimilarity;
[0163] (3) Update the center of each category;
[0164] (4) Repeat steps (2) and (3) until the criterion function converges.
[0165] Regarding question 2), extensive clustering experiments have shown that the best clustering results are achieved when the specified clustering target value K is equal to N obtained from the joint distinct data. Actual data validation results are consistent with this inference. When the specified clustering target K is less than N of the joint distinct data, the clustering algorithm will first remove dimensions with lower frequency of occurrence during the solution process; that is, smaller scenes will be clustered into nearby regular scenes.
[0166] However, in real-world business scenarios, smaller scenarios are crucial components of the overall business process. These smaller scenarios may be overlooked by human experts and should not be ignored at the data level. In other words, when the clustering algorithm achieves maximum separation and minimizes the loss function, it infinitely approaches the result of the mixed data distinct clustering. Finally, engineering verification confirmed that the clustering result is the complete set of the mixed data distinct clustering, but the mixed data distinct clustering is more accurate and efficient.
[0167] Based on the same idea, one or more embodiments of this specification also provide apparatus and devices corresponding to the above methods, such as... Figure 3a , Figure 4 As shown.
[0168] Figure 3a This is a schematic diagram of a risk data mining apparatus provided in one or more embodiments of this specification. The dashed boxes in the diagram represent optional modules. The apparatus includes:
[0169] The acquisition unit 301 is used to acquire business data and parse the business data to obtain business parameters and the corresponding values of the business parameters;
[0170] The pre-filtering unit 302 is used to pre-filter the business parameters to obtain preliminary screening parameters containing business semantics. The preliminary screening parameters include continuous parameters and enumerated parameters.
[0171] The generation unit 303 is used to generate the distribution fitting result of the continuous parameter according to the preset statistical distribution model, and generate the enumerated quantization parameter.
[0172] The parameter recognition unit 304 is used to identify effective parameters containing business semantics in the enumerated parameters and the quantized parameters by using a pre-trained target semantic recognition model.
[0173] The scene recognition unit 304 is used to perform clustering operations on the effective parameters to identify the business scene corresponding to the business data.
[0174] Figure 4 This specification provides a schematic diagram of the structure of a business scenario recognition device according to one or more embodiments. The device includes:
[0175] At least one processor; and,
[0176] A memory communicatively connected to the at least one processor; wherein,
[0177] The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to:
[0178] Acquire business data and parse the business data to obtain business parameters and the corresponding values of the business parameters;
[0179] The business parameters are pre-filtered to obtain preliminary screening parameters containing business semantics. The preliminary screening parameters include continuous parameters and enumerated parameters.
[0180] The production process generates enumerated quantized parameters based on the distribution fitting results of the continuous parameters according to a preset statistical distribution model.
[0181] A pre-trained target semantic recognition model is used to identify valid parameters containing business semantics among the enumerated parameters and the quantized parameters.
[0182] Clustering operations are performed on the effective parameters to identify the business scenarios corresponding to the business data.
[0183] In some embodiments of the present invention, to facilitate expert annotation operations for users, they can click "Add Task" in the task list on the terminal page, enter the application name for accessing business data collection, and after the task is submitted, they can see the execution status and process details of the modeling task in the task list. Once the task is completed, the scene details will jump to the detailed content of the analysis results. The detailed content includes a list of analyzed interfaces, the total number of business scenarios, the total number of valid parameters, and the total number of features contained in each interface.
[0184] Understandably, in practical applications, the accuracy of the entire solution can be improved through continuous iteration and algorithm updates. To facilitate expert annotation and algorithm optimization, a valid parameter annotation page can be displayed on the terminal. Parameters can be removed or restored directly from the operation queue in the parameter list or blacklist. Multiple selections can also be made directly within the table to add or remove parameters.
[0185] refer to Figure 2b The present invention also provides a method for gray-scale traffic redirection based on business scenarios, the specific process of which is as follows:
[0186] 201. Obtain the grayscale traffic redirection request, wherein the grayscale traffic redirection request includes business request data.
[0187] An application can be installed in the terminal, and users can operate within the application page to publish canary traffic redirection requests. For example, by clicking "Add Task" in the task list, entering the name of the application for traffic collection, and submitting the task, the execution status and process details of the modeling task will be displayed in the task list. Once the task is completed, the scene details will redirect to the detailed analysis results. The details include a list of analyzed interfaces, the total number of scenarios and features for each interface. Clicking "Details" allows further viewing of the scenario list, feature list, blacklist feature list, and related traffic redirection rules for each interface. The blacklist feature list contains non-business feature information determined by human expert experience or system recommendations.
[0188] The derived scenario rules and features are automatically generated and pushed as gray-scale rules. In the pilot process based on the order center within the service website, the automatically derived traffic-driving rules are adopted:
[0189] Based on single-feature traffic generation, the core code (grayscale code coverage / online code coverage) coverage reached 81% within 1 hour and 92% within 12 hours.
[0190] Based on precise rule-based traffic generation, 610 precise traffic generation rules were automatically derived during the traffic generation cycle, with 411 rules being hit, resulting in a rule hit rate of 73%. The core code coverage (code coverage during the gray-scale phase / code coverage in the online phase) was approximately 88.331%, which indirectly proves the sufficiency of the scenario rules from the perspective of code coverage.
[0191] 202. Based on the business request data, the above-mentioned intelligent business scenario identification method is used to identify the business scenario corresponding to the business request data;
[0192] 203. Determine the gray-scale traffic redirection rules based on the business scenario, and redirect traffic according to the gray-scale traffic redirection rules.
[0193] Based on the same idea, one or more embodiments of this specification also provide apparatus and devices corresponding to the above methods, such as... Figure 3b , Figure 4 As shown.
[0194] Figure 3b This is a schematic diagram of a risk data mining apparatus provided in one or more embodiments of this specification. The dashed boxes in the diagram represent optional modules. The apparatus includes:
[0195] Acquisition unit 401 is used to acquire grayscale traffic redirection requests, wherein the grayscale traffic redirection requests include business request data;
[0196] The identification unit 402 is used to identify the business scenario corresponding to the business request data using the intelligent business scenario identification method based on the business request data.
[0197] Traffic redirection unit 403 is used to determine grayscale traffic redirection rules based on the business scenario and to redirect traffic according to the grayscale traffic redirection rules.
[0198] Figure 5 is a structural schematic diagram of a grayscale drainage device provided in one or more embodiments of this specification. The device includes:
[0199] At least one processor; and,
[0200] A memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to:
[0201] Obtain grayscale traffic redirection requests, wherein the grayscale traffic redirection requests include business request data;
[0202] Based on the business request data, the above-mentioned intelligent business scenario identification method is used to identify the business scenario corresponding to the business request data.
[0203] The gray-scale traffic redirection rules are determined based on the business scenario, and traffic is redirected according to the gray-scale traffic redirection rules.
[0204] Those skilled in the art should also understand that by simply performing some logic programming on the method flow using an existing hardware description language and programming it into an integrated circuit, the hardware circuit that implements the logic method flow can be easily obtained.
[0205] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.
[0206] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.
[0207] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.
[0208] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the embodiments of this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0209] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0210] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0211] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0212] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0213] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0214] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0215] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.
[0216] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, devices, and non-volatile computer storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.
[0217] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.
Claims
1. A business scenario recognition method based on intelligence, comprising: Obtain business data generated by the service system during the processing of business requests, and parse the business data to obtain business parameters and the corresponding values of the business parameters; The business parameters are pre-filtered to obtain preliminary screening parameters containing business semantics. The preliminary screening parameters include continuous parameters and enumerated parameters. Based on the distribution fitting results of the continuous parameters according to the preset statistical distribution model, an enumerated type of quantized parameters is generated; A pre-trained target semantic recognition model is used to identify the effective parameters in the enumerated parameters and the quantized parameters, wherein the effective parameters are parameters containing business semantics; Clustering operations are performed on the effective parameters to identify the business scenarios corresponding to the business data.
2. The intelligent business scenario recognition method as described in claim 1, wherein generating enumerated quantized parameters based on the distribution fitting result of the continuous parameters according to a preset statistical distribution model specifically includes: The continuous parameters are fitted to a distribution based on a preset statistical distribution model to obtain the distribution fitting result; Based on the distribution fitting results, determine the head range, normal range, and tail range of the continuous parameter; The head value range, the normal value range, and the tail value range are used as the quantization parameters corresponding to the continuous parameter.
3. The intelligent business scenario recognition method as described in claim 2, wherein the step of using a pre-trained target semantic recognition model to identify valid parameters containing business semantics among the enumerated parameters and the quantized parameters further includes: Based on the preset characters, the enumerated parameters and the terminal parameters are divided into atomic units and molecular units respectively, and the quantization parameters are used as molecular units, while the terminal parameters are the last parameters in the business parameters; Using a pre-defined mapping sub-model, the atomic units and molecular units corresponding to each parameter, as well as the values corresponding to each parameter, are mapped to atomic multidimensional vectors and molecular multidimensional vectors of different dimensions, respectively. By concatenating the atomic multidimensional vectors and molecular multidimensional vectors corresponding to each parameter, a multidimensional semantic vector corresponding to each parameter is obtained. A preset classification tree sub-model is used to identify the parameter labels corresponding to the multidimensional semantic vector, and the parameter labels include valid parameter labels and invalid parameter labels; The effective parameters in the enumerated parameters and the quantization parameters are determined based on the parameter labels.
4. The intelligent business scenario recognition method as described in claim 3, before identifying the valid parameters among the enumerated parameters and the quantized parameters using the pre-trained target semantic recognition model, further includes: Collect business data samples, and perform parsing, pre-filtering and distribution fitting on the business data to obtain sample parameters; The sample parameters are categorized and labeled according to whether they contain business semantics to obtain a training sample set. The sample parameters in the training sample set carry preset labels, which include valid parameters and invalid parameters. The initial classification tree sub-model is trained using the training sample set to obtain the preset classification tree model.
5. The intelligent business scenario recognition method as described in claim 4, wherein training the initial classification tree sub-model using the training sample set to obtain the preset classification tree model includes: The training sample set is input into the first tree in the initial classification tree sub-model for training to obtain the first score value; Obtain the first residual value between the first score and the preset label; The first residual value is used as the preset label of the second tree and iterative training is performed until each tree is trained, resulting in the trained preset classification tree sub-model.
6. The intelligent business scenario recognition method as described in claim 5, wherein inputting the training sample set into the first tree of the initial classification tree sub-model for training includes: According to preset fields, the sample parameters are divided into atomic units and molecular units, and the atomic units and molecular units are used as parameter lexicon; The frequency of each atomic unit and each molecular unit in the parameter lexicon being marked as a valid parameter or an invalid parameter is counted to obtain the sample parameter frequency; The initial classification tree sub-model is trained based on the frequency of the sample parameters.
7. The intelligent business scenario identification method as described in claim 1, wherein performing clustering operations on the effective parameters to identify the business scenario corresponding to the business data includes: Calculate the parameter distance between the effective parameter and the initial scene category center, and update the scene category center based on the parameter distance; If the distance between the initial scene category center and the updated scene category center is not less than a preset threshold, repeat the above steps until the distance between the initial scene category center and the updated scene category center is less than the preset threshold. If the distance between the initial scene category center and the updated scene category center is less than a preset threshold, the business scene corresponding to the business data is determined based on the scene category center.
8. The intelligent business scenario recognition method as described in claim 4, comprising acquiring business data and parsing the business data to obtain business parameters and the corresponding values of the business parameters, including: Select initial parameters from the sample parameters, and calculate the initial distance between each remaining sample parameter and each initial parameter; Based on the initial distance, the sample parameters are assigned to the categories corresponding to the initial parameters; Update the scene center based on the existing sample parameters in the category, and calculate the sample distance between the updated scene center and the sample parameters. Repeat the above steps until the sample distance is less than a preset threshold, and determine the business scenario corresponding to the sample parameters based on the updated scene center.
9. The intelligent business scenario recognition method as described in claim 8, wherein selecting initial parameters from the sample parameters and calculating the initial distance between each remaining sample parameter and each initial parameter includes: Based on the sample parameters and their values, obtain the multidimensional classification vector and the multidimensional data vector; Based on the multidimensional classification vector, calculate the classification distance between the remaining sample parameters and each initial sample parameter; Based on the multidimensional data vector, calculate the data distance between the remaining sample parameters and each initial sample parameter; The initial distance is determined based on the classification distance and the data distance.
10. A business scenario recognition device based on intelligence, comprising: The acquisition unit is used to acquire business data generated by the service system during the processing of business requests, and to parse the business data to obtain business parameters and the corresponding values of the business parameters. The pre-filtering unit is used to perform data pre-filtering processing on the business parameters to obtain preliminary screening parameters containing business semantics. The preliminary screening parameters include continuous parameters and enumerated parameters. The generation unit generates enumerated quantized parameters based on the distribution fitting results of the continuous parameters according to a preset statistical distribution model. The parameter recognition unit is used to identify valid parameters containing business semantics in the enumerated parameters and the quantized parameters by using a pre-trained target semantic recognition model. The scene recognition unit is used to perform clustering operations on the effective parameters to identify the business scene corresponding to the business data.
11. A method for gradual traffic acquisition based on business scenarios, comprising: Obtain grayscale traffic redirection requests, wherein the grayscale traffic redirection requests include business request data; Based on the business request data, the business scenario corresponding to the business request data is identified using the intelligent business scenario identification method as described in any one of claims 1 to 9; The gray-scale traffic redirection rules are determined based on the business scenario, and traffic is redirected according to the gray-scale traffic redirection rules.
12. A gray-scale traffic redirection device based on business scenarios, comprising: The acquisition unit is used to acquire grayscale traffic redirection requests, wherein the grayscale traffic redirection requests include business request data; The identification unit is configured to identify the business scenario corresponding to the business request data using the intelligent business scenario identification method as described in any one of claims 1 to 9, based on the business request data. The traffic redirection unit is used to determine the gray-scale traffic redirection rules based on the business scenario, and to redirect traffic according to the gray-scale traffic redirection rules.
Citation Information
Patent Citations
Method and device for shunt of network flow
CN104601486A
Method and apparatus for identifying abnormal clicking of advertising space
CN107330731A