Traffic collection and identification method based on massive App

Abstract

The invention discloses a method for collecting and identifying traffic based on massive App. The method specifically comprises the following steps: acquiring App traffic by utilizing a packet capture program, forming a traffic data file, preprocessing the traffic data file, extracting key traffic characteristics, respectively arranging the key traffic characteristics into a training set and a test set, and storing the training set and the test set in a database; an AI analysis program reads the training set from the database, further processes key traffic features in the training set into vectors suitable for an artificial intelligence model, and inputs the vectors into the artificial intelligence model for model training and adjustment; and classifying and identifying the key traffic characteristics in the test set by using a trained artificial intelligence model, thereby realizing acquisition and identification of App traffic.

Description

Technical Field

[0001] The present invention relates to the field of automated traffic collection and analysis, and in particular, mainly to a method for collecting and identifying massive App traffic. Background Art

[0002] With the development of mobile Internet, the number and types of mobile applications (mobile application or mobile app), often referred to as applications (app, which is referred to as app in this article), have increased rapidly, resulting in the increasing complexity and diversity of network traffic. How to effectively identify and classify these traffic has become an important research direction in the fields of network security, traffic management and data analysis. Traditional traffic identification methods mainly rely on feature matching and statistical analysis. Faced with increasingly complex traffic patterns, it is difficult to meet the needs of efficient and accurate identification. Therefore, traffic identification technology based on artificial intelligence has emerged. Using machine learning and deep learning algorithms, it can more accurately analyze and identify complex App traffic patterns.

[0003] In the prior art, traffic identification is mainly carried out through deep packet inspection (DPI), protocol analysis, domain names, port numbers, and IP addresses. However, with the popularization of encryption technology and the diversification of network traffic, the accuracy and efficiency of traditional methods have encountered bottlenecks. In particular, the traffic of mobile applications, its feature confusion and dynamic variability make it difficult for traditional methods to handle effectively. In addition, the method of manually configuring features and rules is not only cumbersome but also difficult to cope with the rapid iteration of new apps. For example, the invention patent with announcement number CN118337717A discloses an encrypted network traffic classification technology, but the classification granularity of the traffic is not detailed enough, and the source method of the traffic to be classified is not detailed enough; the patent with announcement number CN118138306A discloses an encrypted traffic classification method based on machine learning, but it also lacks an accurate description of the source of training samples.

[0004] Therefore, the present invention introduces automated control of mobile applications, automatic extraction and analysis of traffic features, and artificial intelligence technology, which becomes an effective way to solve this problem through machine learning and deep learning algorithms. The use of artificial intelligence technology can not only improve the accuracy and efficiency of recognition, but also automatically adapt to changes in new traffic patterns. Summary of the invention

[0005] In order to overcome the problems of labor-intensive, serious interference from miscellaneous traffic, and limited identification accuracy in the process of identifying app traffic in the prior art, the present invention provides a system that uses automation technology to perform batch operations on apps (including but not limited to installation, startup, click, exit and uninstall, etc.) and capture and save their corresponding traffic. After cleaning and further formatting the traffic data, the data is modeled through artificial intelligence technology and applied to subsequent prediction and identification.

[0006] According to one aspect of the present invention, a method for collecting and identifying massive App traffic is proposed, comprising:

[0007] Use a packet capture program to obtain App traffic and form a traffic data file, pre-process the traffic data file and extract key traffic features, organize the key traffic features into a training set and a test set, and save them together in a database;

[0008] The AI ​​parsing program reads the training set from the database, further processes the key traffic features in the training set into vectors suitable for the artificial intelligence model, and then inputs them into the artificial intelligence model for model training and adjustment;

[0009] The trained artificial intelligence model is used to classify and identify the key traffic features in the test set, thereby realizing the collection and identification of App traffic.

[0010] Furthermore, the packet capture program is an automated Python program processing flow, and using the packet capture program to obtain App traffic specifically includes the following steps:

[0011] Run the packet capture program, load the configuration file, and enter the application market mode or installation package mode. If you enter the application market mode, the device automatically opens the application market, selects a category, and automatically clicks on the first application to download and install. If you enter the installation package mode, the device uses different commands to install the installation package file to the device.

[0012] After the application is installed, traffic capture automatically starts. The traffic comes from the interaction with the application. After the preset interaction time is reached, the application is uninstalled, the traffic capture ends, and the captured traffic data is saved as a pcap file.

[0013] After cleaning the application cache files, if it is in application market mode, the application market will be automatically opened to download and install the next application. If it is in installation package mode, the next installation package file will be automatically installed. The Android side can use tools such as tcpdump / iptables to capture traffic. The iOS side can use the virtual interface created by the rvictl command and capture packets through Tshark. Among them, tcpdump is a command line tool that can capture network data packets on Unix / Linux systems.

[0014] Furthermore, the automatic clicking on the first application to download and install and the uninstalling of the application both come from an automation framework, which uses the open source framework Appium to achieve automated operations on mobile devices through specific APIs, including application installation, startup, clicking, closing and uninstalling.

[0015] Furthermore, the preprocessing is mainly used to perform data cleaning and formatting on the collected traffic; the main background of data cleaning is that different apps may call the same code set (usually called a third-party SDK) for development, and these codes may trigger the same traffic characteristics, thereby interfering with the subsequent model training; for this, it is necessary to perform preliminary filtering on the collected traffic through Python code to retain its unique characteristics; the main function of formatting is that in the subsequent model training process, the data must be converted into a vector form that the model can accept (including but not limited to tokenizing domain name information such as host / SNI).

[0016] Furthermore, the key traffic features specifically represent key data combinations extracted from traffic data files, including domain names, ports, IP addresses, data packet contents, protocol types, and session timing relationships. Different feature combinations will affect the effect of subsequent model training.

[0017] It is recommended to save the traffic data file as a pcap format file. The pcap format is a storage format for traffic data supported by the famous packet capture tool Wireshark. Saving the traffic in the pcap format is convenient for manual opening in the early stage to check the correctness of the program capturing the traffic, and it is also convenient to call the code library that supports parsing the pcap format in the Python program to parse the file.

[0018] Furthermore, the artificial intelligence model adopts the SVM model, and the specific formula is as follows:

[0019]

[0020] y i (w T x i +b)≥1

[0021] Among them, w represents the weight vector, w T represents the transpose of w, b represents the bias term, x represents the input vector, x i Represents the i-th vector in the input vector, i∈R, R represents a natural number starting from 1, y i Represents the label of the i-th vector, which takes a value of +1 or -1.

[0022] The specific steps of model training and adjustment of the artificial intelligence model after inputting the vector are as follows:

[0023] Loading a configuration file, and selecting a program operation mode according to the configuration file, including: feature training mode, model training mode, and prediction training mode;

[0024] Loading the characteristic parameters or model corresponding to the operation mode according to the configuration file, wherein if it is a characteristic training mode, the model is loaded; if it is a model training mode, the characteristic parameters are loaded; if it is a prediction judgment mode, the model and characteristic parameters are loaded;

[0025] Reading the training set in the database, and adjusting it into a vector form after preprocessing;

[0026] The feature training mode adjusts the model by loading the feature parameter range defined in the configuration file; the model training mode trains the model by combining the model type and options specified in the configuration file with the feature parameters, and obtains the optimal model by comparing the prediction effects of different models; the prediction judgment mode processes the predicted traffic and makes a prediction classification by using the feature parameters and models defined in the configuration file;

[0027] The feature training mode saves the optimal feature parameters; the model training mode saves the model with the best prediction accuracy; and the prediction judgment mode outputs the judgment result corresponding to the predicted flow.

[0028] The specific steps of App traffic identification after the artificial intelligence model has been trained in different modes include: the test set is used as the input data of the model, the optimal feature parameters saved in the feature training mode are used as input parameters, and are input into the model with the best prediction accuracy saved in the model training mode, the prediction judgment mode in the model is called, and the corresponding prediction results are output.

[0029] In addition to the above-mentioned SVM model, the artificial intelligence model can also use Random Forest, Decision Tree, and can also include common neural network models in deep learning such as Convolutional Neural Network CNN, Recurrent Neural Network RNN ​​and Feedforward Neural Network FNN, etc. By inputting the processed traffic feature vectors into different artificial intelligence models for training, a batch of available traffic prediction models can be obtained.

[0030] According to a second aspect of the present invention, a computer-readable storage medium is provided, on which one or more computer programs are stored. When the one or more computer programs are executed by a computer processor, the above method is implemented.

[0031] The above one or more technical solutions in the embodiments of the present application have at least one of the following technical effects:

[0032] The present invention completely controls the use cycle of the App through an automated framework, effectively eliminating the problems of incomplete traffic capture and traffic mixing in traditional traffic analysis methods, which is beneficial to the subsequent artificial intelligence model training. By preprocessing the captured pcap traffic data, interfering traffic such as third-party SDK libraries that may be contained is eliminated, effectively improving the accuracy of the recognition model. By introducing artificial intelligence models including machine learning and deep learning to learn and train traffic, the defects of traditional traffic analysis methods that the traffic characteristics are complicated and difficult to identify manually are eliminated. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated into and constitute a part of this specification. The accompanying drawings illustrate the embodiments and are used together with the description to explain the principles of the present invention. It will be easy to recognize other embodiments and many expected advantages of the embodiments because they become better understood by reference to the following detailed description. The elements of the drawings are not necessarily to scale with each other. The same reference numerals refer to corresponding similar parts.

[0034] Figure 1 A schematic diagram of a process of collecting and identifying a method based on massive App traffic according to an embodiment of the present invention is shown.

[0035] Figure 2 A schematic flow chart of a method for traffic capture according to an embodiment of the present invention is shown.

[0036] Figure 3 A schematic flow chart of a model training method according to an embodiment of the present invention is shown.

[0037] Figure 4A schematic diagram of a flow chart of traffic prediction and identification based on a trained model according to an embodiment of the present invention is shown.

[0038] Figure 5 A schematic diagram of a process for collecting and identifying massive App traffic according to an embodiment of the present invention is shown.

[0039] Figure 6 It is a structural diagram of a computer system suitable for implementing an electronic device of an embodiment of the present application. DETAILED DESCRIPTION

[0040] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0041] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0042] Figure 1 FIG. 1 shows a schematic diagram of a process based on massive App traffic collection and identification according to an embodiment of the present invention. Figure 1 As shown:

[0043] Use a packet capture program to obtain App traffic and form a traffic data file, pre-process the traffic data file and extract key traffic features, organize the key traffic features into a training set and a test set, and save them together in a database;

[0044] Further, such as Figure 2 As shown in the figure, using the packet capture program to obtain App traffic specifically includes the following steps:

[0045] Run the packet capture program, load the configuration file, and enter the application market mode or installation package mode; the user selects the running mode and selects different device connection methods according to different device systems. In this embodiment, the application market supports AppStore, Google play, Wandoujia, etc., and the installation package mode supports .apk, .xapk, and .ipa files.

[0046] If you enter the application market mode, the device automatically opens the application market and selects a category, and automatically clicks on the first application to download and install. If you enter the installation package mode, the device uses different commands to install the installation package file to the device.

[0047] After the application is installed, traffic capture starts automatically. The traffic comes from the interaction with the application. For example, when a user opens the application and clicks on the application pop-up window, interactive traffic is generated.

[0048] After the preset interaction time is reached, the application is uninstalled, traffic capture ends, and the captured traffic data is saved as a pcap file;

[0049] After cleaning the application cache files, if it is in application market mode, the application market will be automatically opened to download and install the next application. If it is in installation package mode, the next installation package file will be automatically installed.

[0050] The purpose of cleaning up application cache files is to prevent applications from automatically installing too many cache files and consuming a large amount of device storage.

[0051] The automatic click on the first application to download and install and the uninstallation of the application are both from the automation framework. The automation framework adopts the open source framework Appium and realizes the automation operation of the mobile device through a specific API, including the installation, startup, click, closing and uninstallation of the application.

[0052] The AI ​​parsing program reads the training set from the database, further processes the key traffic features in the training set into vectors suitable for the artificial intelligence model, and then inputs them into the artificial intelligence model for model training and adjustment;

[0053] The key traffic features specifically represent key data combinations extracted from the traffic data file, including domain name, port, IP, data packet content, protocol type and session timing relationship.

[0054] The trained artificial intelligence model is used to classify and identify the key traffic features in the test set, thereby realizing the collection and identification of App traffic.

[0055] The artificial intelligence model adopts the SVN model, and the specific formula is as follows:

[0056]

[0057] y i (w T x i +b)≥1

[0058] Among them, w represents the weight vector, w T represents the transpose of w, b represents the bias term, x represents the input vector, x i Represents the i-th vector in the input vector, i∈R, R represents a natural number starting from 1, y i Represents the label of the i-th vector, which takes a value of +1 or -1.

[0059] in, is the objective function, min_(w, b) means to find the optimal values ​​of w and b so that Minimum.

[0060] And y i (w T x i +b)≥1 is a constraint condition to ensure that all sample features are correctly classified. Therefore, in model training, there is still a step to label and classify the sample features.

[0061] Further, such as Figure 3 As shown, the specific steps of model training and adjustment of the artificial intelligence model after inputting the vector are as follows:

[0062] Loading a configuration file, and selecting a program operation mode according to the configuration file, including: feature training mode, model training mode, and prediction training mode;

[0063] Loading the characteristic parameters or model corresponding to the operation mode according to the configuration file, wherein if it is a characteristic training mode, the model is loaded; if it is a model training mode, the characteristic parameters are loaded; if it is a prediction judgment mode, the model and characteristic parameters are loaded;

[0064] Reading the training set in the database, and adjusting it into a vector form after preprocessing;

[0065] The feature training mode adjusts the model by loading the feature parameter range defined in the configuration file; the model training mode trains the model by combining the model type and options specified in the configuration file with the feature parameters, and obtains the optimal model by comparing the prediction effects of different models; the prediction judgment mode processes the predicted traffic and makes a prediction classification by using the feature parameters and models defined in the configuration file;

[0066] The feature training mode saves the optimal feature parameters; the model training mode saves the model with the best prediction accuracy; and the prediction judgment mode outputs the judgment result corresponding to the predicted flow.

[0067] Further, such as Figure 4 As shown, Figure 4 A schematic diagram of a flow chart of traffic prediction and identification based on a trained model according to an embodiment of the present invention is shown:

[0068] The test set is used as the input data of the model, and the optimal feature parameters saved in the feature training mode are used as input parameters. It is input into the model with the best prediction accuracy saved in the model training mode, and the prediction judgment mode in the model is called to output the corresponding prediction results.

[0069] According to another embodiment of the present invention, Figure 5 As shown, Figure 5 A schematic diagram of a framework based on massive App traffic collection and identification according to an embodiment of the present invention is shown, and the specific steps of the embodiment are as follows:

[0070] The host device needs to meet certain hardware and software requirements, including a Python environment that can execute automation programs and AI parsing programs, can identify and connect to mobile devices through adb programs, and install graphics cards and corresponding GPU drivers. The host device is connected to the mobile device via a USB data cable. The host device is configured with necessary drivers and software settings to identify and operate the mobile device. The automation program is executed on the host device to achieve automated installation and operation of the target App. During the running of the App, the automation program captures the network traffic of the corresponding process in real time through the packet capture program (tcpdump) and saves it as a data file (pcap format). After obtaining the pcap file, the parsing process is performed to extract key features in the traffic, such as the target IP address, target port number, protocol type, packet length, timestamp, session timing relationship, etc. These features are processed and saved in the database, which can be used as input data for subsequent machine learning models. The processed traffic features are input into the pre-designed artificial intelligence model for training. The model can use Random Forest, Decision Tree in machine learning, or Convolutional Neural Network (CNN), Recurrent Neural Network (RNN) and other structures in deep learning to learn and identify complex traffic patterns. After the training process is completed, the optimal model will be saved as the final model. The parsing program will subsequently call the model to classify and identify the newly captured traffic features and determine the type of App to which it belongs; it can be deployed in the actual network environment to monitor and identify real-time traffic. By integrating with network devices (such as firewalls and routers), it can achieve management and control of specific App traffic.

[0071] Furthermore, after obtaining the optimal characteristic parameters, the optimal model, and the corresponding predicted traffic judgment results, this information is used to iterate and optimize the process of App traffic collection and identification. In some embodiments, the iterative process that can be taken is as follows:

[0072] Step 1: Determine the optimization goal, such as improving prediction accuracy, reducing latency, and reducing resource consumption.

[0073] Step 2: Apply the optimal features and models. Apply the verified optimal feature parameters and models to the actual traffic collection system to ensure that the model deployment is stable and reliable and can efficiently process real-time or batch data.

[0074] Step 3: Implement A / B testing. Conduct A / B testing in a real environment to verify the effect of the optimal model compared with the old model. Select a part of users as the experimental group and apply the optimal model. Select another part of users as the control group and continue to use the old model. Collect experimental data and analyze it to determine whether the optimal model has achieved the expected effect.

[0075] Step 4: Continuous monitoring: Monitor the performance of the model in real time to ensure that it performs as expected in the actual environment, and regularly check key performance indicators to ensure that the model remains effective.

[0076] Step 5: Data feedback and retraining. According to the performance of the model in the actual environment, new data points are collected. The data points may include the model's prediction results, user behavior feedback, etc. These new data points are used to retrain the model to further improve its performance.

[0077] Step 6: Dynamic adjustment: dynamically adjust feature parameters or the model itself according to the performance of the model. For example, if you find that the importance of certain features has changed, you can adjust the feature weights or choose different feature combinations. If the model performs poorly, you may need to consider changing the model architecture or adjusting the hyperparameters.

[0078] Step 7: Model update. Update the model regularly to adapt to changes in data distribution. The update cycle can be determined based on the frequency of data updates and the speed at which model performance deteriorates.

[0079] Step 8: Expand the scale. If the model performs well, you can gradually expand the scope of application until it covers the entire system. During the expansion process, pay attention to monitoring the overall performance of the system to ensure stable operation of the system.

[0080] Step 9: Continuous optimization requires constant iteration, regular evaluation of model performance, and adjustments based on business needs and technological advances.

[0081] Furthermore, the optimal feature parameters and models are used to optimize App traffic collection. The specific steps are as follows:

[0082] Step 1: Deploy the model: Deploy the optimal model to the online environment.

[0083] Step 2: Implement A / B testing, select a part of users as the experimental group, use the optimal model to predict traffic, and select another part of users as the control group, use the old model.

[0084] Step 3: Collect feedback, collect data from the experimental group and the control group, and analyze the differences between the two groups.

[0085] Step 4: Analyze the results. If the optimal model performs better, consider promoting it to all users. If the optimal model performs poorly, further analyze the reasons.

[0086] Step 5: Model iteration, adjust model parameters or features according to experimental results, retrain the model, and conduct A / B testing again.

[0087] Step 6: Continuous monitoring, regularly monitor the performance of the model and make adjustments as needed. Through repeated iterations of the process of the present invention, the process of App traffic collection is continuously improved to ensure that the system can be continuously optimized with time and changes in user behavior.

[0088] In summary, the present invention introduces artificial intelligence models including machine learning and deep learning to learn and train traffic, eliminating the problem of complex traffic characteristics existing in traditional traffic analysis methods. In the iterative process of practical applications, the model and feature vectors are continuously optimized to improve the accuracy of traffic identification.

[0089] Reference below Figure 6 , which shows a schematic diagram of the structure of a computer system 600 suitable for implementing an electronic device of an embodiment of the present application. Figure 6 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0090] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0091] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0092] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above functions defined in the method of the present application are executed. It should be noted that the computer-readable storage medium of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wireline, optical cable, RF, etc., or any suitable combination of the foregoing.

[0093] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0094] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0095] The modules involved in the embodiments of the present application may be implemented by software or by hardware.

[0096] As another aspect, the present application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or it may exist independently without being assembled into the electronic device. The above computer-readable storage medium carries one or more programs. When the above one or more programs are executed by the electronic device, the electronic device: uses a packet capture program to obtain App traffic, forms a traffic data file, pre-processes the traffic data file and extracts key traffic features, organizes the key traffic features into training sets and test sets respectively, and saves them to a database together; the AI ​​parsing program reads the training set from the database, further processes the key traffic features in the training set into vectors suitable for the artificial intelligence model, and then inputs them into the artificial intelligence model for model training and adjustment; uses the trained artificial intelligence model to classify and identify the key traffic features in the test set, thereby realizing the collection and identification of App traffic.

[0097] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A method based on massive App traffic collection and identification, characterized in that: The following steps are involved: Use a packet capture program to obtain App traffic and form a traffic data file, pre-process the traffic data file and extract key traffic features, organize the key traffic features into a training set and a test set, and save them together in a database; The AI ​​parsing program reads the training set from the database, further processes the key traffic features in the training set into vectors suitable for the artificial intelligence model, and then inputs them into the artificial intelligence model for model training and adjustment; The trained artificial intelligence model is used to classify and identify the key traffic features in the test set, thereby realizing the collection and identification of App traffic.

2. The method for flow collection and identification according to claim 1, characterized in that: The method of using the packet capture program to obtain App traffic specifically includes the following steps: Run the packet capture program, load the configuration file, and enter the application market mode or installation package mode. If you enter the application market mode, the device automatically opens the application market, selects a category, and automatically clicks on the first application to download and install. If you enter the installation package mode, the device uses different commands to install the installation package file to the device. After the application is installed, traffic capture automatically starts. The traffic comes from the interaction with the application. After the preset interaction time is reached, the application is uninstalled, the traffic capture ends, and the captured traffic data is saved as a pcap file. After cleaning the application cache files, if it is in application market mode, the application market will be automatically opened to download and install the next application. If it is in installation package mode, the next installation package file will be automatically installed.

3. The method for flow collection and identification according to claim 2, characterized in that: The automatic click on the first application to download and install and the uninstallation of the application are both from the automation framework. The automation framework adopts the open source framework Appium and realizes the automation operation of the mobile device through a specific API, including the installation, startup, click, closing and uninstallation of the application.

4. The method for flow collection and identification according to claim 1, characterized in that: The key traffic features specifically represent key data combinations extracted from the traffic data file, including domain name, port, IP, data packet content, protocol type and session timing relationship.

5. The method for flow collection and identification according to claim 1, characterized in that: The artificial intelligence model adopts the SVM model, and the specific formula is as follows: y i (w T x i +b)≥1 Among them, w represents the weight vector, w T represents the transpose of w, b represents the bias term, x represents the input vector, x i Represents the i-th vector in the input vector, i∈R, R represents a natural number starting from 1, y i Represents the label of the i-th vector, which takes a value of +1 or -1.

6. The method for flow collection and identification according to claim 1, characterized in that: The specific steps of the artificial intelligence model for model training after inputting the vector are: Loading a configuration file, and selecting a program operation mode according to the configuration file, including: feature training mode, model training mode, and prediction training mode; Loading the characteristic parameters or model corresponding to the operation mode according to the configuration file, wherein if it is a characteristic training mode, the model is loaded; if it is a model training mode, the characteristic parameters are loaded; if it is a prediction judgment mode, the model and characteristic parameters are loaded; Reading the training set in the database, and adjusting it into a vector form after preprocessing; The feature training mode adjusts the model by loading the feature parameter range defined in the configuration file; the model training mode trains the model by combining the model type and options specified in the configuration file with the feature parameters, and obtains the optimal model by comparing the prediction effects of different models; the prediction judgment mode processes the predicted traffic and makes a prediction classification by using the feature parameters and models defined in the configuration file; The feature training mode saves the optimal feature parameters; the model training mode saves the model with the best prediction accuracy; and the prediction judgment mode outputs the judgment result corresponding to the predicted flow.

7. The method for flow collection and identification according to claim 1, 5 or 6, characterized in that: The specific steps of App traffic identification after the artificial intelligence model has been trained in different modes include: the test set is used as the input data of the model, the optimal feature parameters saved in the feature training mode are used as input parameters, and are input into the model with the best prediction accuracy saved in the model training mode, the prediction judgment mode in the model is called, and the corresponding prediction results are output.

8. A computer program product, characterized in that A computer program is stored thereon, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

9. A computing system, characterized in that: The method comprises a processor and a memory, wherein the processor is configured to execute the method according to any one of claims 1 to 7.

Images