Method and device for training integrity identification model and electronic equipment

Abstract

The embodiment of the invention discloses a training method of an integrity identification model, a traffic acquisition integrity identification method and device and electronic equipment, and the training method of the integrity identification model comprises the steps: randomly screening out a part of data from a plurality of pieces of acquired traffic data; according to whether the screened data is complete to generate a label, expanding the screened data and the label thereof by adopting a mask mode; converting the expanded data and labels into corresponding vector representation through a natural language processing model; performing feature extraction on the vector representation based on a self-attention mechanism to obtain low-dimensional features; and training a neural network model to identify whether the data represented by the low-dimensional features are complete or not by taking the low-dimensional features as input and a dichotomy result as output so as to obtain a completeness identification model, and the dichotomy result is used for identifying that the data are complete or incomplete.

Description

Technical Field

[0001] The present invention relates to the field of traffic collection technology, and in particular to a training method for an integrity recognition model, a method for recognizing traffic collection integrity, a device and an electronic device. Background Art

[0002] Traffic collection refers to the collection of data traffic transmitted in the network through specific technical means, and the analysis and mining of data traffic to obtain valuable information. The core purpose of traffic collection is to obtain information such as network usage, performance status, and security threats, so as to provide a basis for control behaviors such as network management, fault location, optimization of network structure, and security protection.

[0003] When collecting traffic, some data may be lost due to network fluctuations and hardware reasons, resulting in incomplete data packets. Therefore, in order to ensure the quality of process collection, it is crucial to judge the integrity of traffic collection data. At present, the integrity of traffic collection data is usually judged using a neural network model. The judgment process often relies on a large amount of manually labeled data, which is costly and inefficient. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a training method for an integrity recognition model, a method, a device and an electronic device for recognizing the integrity of flow collection, so as to solve the problems in the prior art.

[0005] To solve the above technical problems, the embodiments of the present invention are implemented as follows: In a first aspect, an embodiment of the present invention provides a method for training a completeness recognition model, comprising: Randomly select some data from the multiple collected traffic data; Based on whether the labels of the filtered data are completely generated, the filtered data and its labels are expanded by using a mask method; The expanded data and labels are converted into corresponding vector representations through a natural language processing model; Extracting features from the vector representation based on a self-attention mechanism to obtain low-dimensional features; Taking the low-dimensional features as input and the binary classification results as output, a neural network model is trained to identify whether the data represented by the low-dimensional features is complete, thereby obtaining a completeness recognition model, wherein the binary classification results are used to identify whether the data is complete or incomplete.

[0006] In a second aspect, an embodiment of the present invention provides a method for identifying flow collection integrity, including: Obtain traffic collection data; Convert the data into corresponding vector representations through a natural language processing model; Extracting features from the vector representation based on a self-attention mechanism to obtain low-dimensional features; Inputting the low-dimensional features into a completeness recognition model to identify whether the data corresponding to the low-dimensional features are complete; The integrity recognition model is a integrity recognition model trained using the training method of the integrity recognition model.

[0007] In a third aspect, an embodiment of the present invention provides a training device for a completeness recognition model, comprising: A screening module is used to randomly screen out some data from multiple collected traffic data; The expansion module is used to generate labels according to whether the screening data is complete, and to expand the screening data and its labels in a masking manner; The conversion module is used to convert the expanded data and labels into corresponding vector representations through the natural language processing model; An extraction module, used for extracting features from the vector representation based on a self-attention mechanism to obtain low-dimensional features; A training module is used to take the low-dimensional features as input and the binary classification results as output, train a neural network model to identify whether the data represented by the low-dimensional features is complete, and obtain a completeness recognition model, wherein the binary classification results are used to identify whether the data is complete or incomplete.

[0008] In a fourth aspect, an embodiment of the present invention provides a device for identifying flow collection integrity, including: An acquisition module is used to acquire traffic collection data; A conversion module, used for converting the data into corresponding vector representations through a natural language processing model; An extraction module, used for extracting features from the vector representation based on a self-attention mechanism to obtain low-dimensional features; An identification module, used for inputting the low-dimensional features into a completeness identification model to identify whether the data corresponding to the low-dimensional features are complete; The integrity recognition model is a integrity recognition model trained using the training method of the integrity recognition model.

[0009] In a fifth aspect, an embodiment of the present invention provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the training method of the above-mentioned integrity recognition model, or the steps of the above-mentioned method for identifying the integrity of traffic collection.

[0010] In a sixth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the training method of the above-mentioned integrity recognition model or the steps of the above-mentioned method for identifying the integrity of traffic collection are implemented.

[0011] In a seventh aspect, an embodiment of the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the training method of the above-mentioned integrity recognition model, or the steps of the above-mentioned method for identifying the integrity of traffic collection.

[0012] It can be seen from the technical solution provided by the above embodiments of the present invention that the embodiments of the present invention randomly screen out part of the data from the multiple collected traffic data, generate labels according to whether the screened data is complete, expand the screened data and its labels by masking, convert the expanded data and labels into corresponding vector representations through a natural language processing model, extract features from the vector representations based on the self-attention mechanism to obtain low-dimensional features, use the low-dimensional features as input and the binary classification results as output, train a neural network model to identify whether the data represented by the low-dimensional features is complete, and obtain a completeness recognition model, wherein the binary classification results are used to identify whether the data is complete or incomplete. Therefore, expanding the data set for training based on the mask method effectively reduces the dependence on label data, greatly reduces labor costs, and improves the efficiency of model training. Moreover, the processing based on the self-attention mechanism improves the ability of feature recognition and extraction, thereby improving the accuracy of model training.

[0013] In addition, the embodiment of the present invention uses a trained integrity recognition model to identify traffic data, which can obtain high-quality recognition results without human intervention, realize automatic recognition, can quickly identify incomplete data, improve the accuracy of traffic collection integrity recognition, and greatly improve the quality and reliability of traffic collection. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0015] Figure 1 A flowchart of a training method for an integrity recognition model provided by an embodiment of the present invention; Figure 2 A flowchart of another method for training a completeness recognition model provided by an embodiment of the present invention; Figure 3 A schematic diagram of the control flow of a flow collection task provided by an embodiment of the present invention; Figure 4 A schematic diagram of the processing flow of the self-supervised learning model provided by an embodiment of the present invention; Figure 5 A schematic diagram of the principle of the integrity recognition model provided by an embodiment of the present invention; Figure 6 A schematic diagram of a process for expanding a data set provided by an embodiment of the present invention; Figure 7 A flow chart of a method for identifying flow collection integrity provided by an embodiment of the present invention; Figure 8 A schematic diagram of the structure of a training device for an integrity recognition model provided by an embodiment of the present invention; Fig. 9 A schematic diagram of the structure of a device for identifying flow collection integrity provided by an embodiment of the present invention; Fig.10 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0016] The embodiments of the present invention provide a training method for an integrity recognition model, a method and device for recognizing the integrity of flow collection, and an electronic device.

[0017] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0018] The training method and device of the integrity recognition model and the identification method and device of the flow collection integrity provided in the embodiments of the present invention can be applied to an electronic device, which can be a terminal device or a server. The server can be an independent server or a server cluster composed of multiple servers, which is not specifically limited. The terminal device can be a device such as a personal computer, or a mobile terminal device such as a mobile phone or a tablet computer, which is not specifically limited. The application scenarios of the above methods and devices include: training scenarios of integrity recognition models on terminal devices or servers, or scenarios of flow collection, etc.

[0019] Through the training method and device of the completeness recognition model, the data set is expanded for training based on the mask method, which effectively reduces the dependence on label data, greatly reduces labor costs, and improves the efficiency of model training. Moreover, based on the processing of the self-attention mechanism, the ability of feature recognition and extraction is improved, thereby improving the accuracy of model training.

[0020] Through the above-mentioned flow collection integrity identification method and device, the trained integrity identification model is used to identify the flow data, which can obtain high-quality identification results without human intervention, realize automatic identification, and can quickly identify incomplete data, thereby improving the accuracy of flow collection integrity identification and greatly improving the quality and reliability of flow collection.

[0021] like Figure 1 As shown, an embodiment of the present invention provides a training method for a completeness recognition model, which may include the following steps: S102: Randomly select some data from the multiple pieces of collected traffic data.

[0022] In the embodiment of the present invention, the collected traffic data is usually multiple, and can be massive data, such as collecting 10,000 or 20,000 traffic data, etc., which is not specifically limited.

[0023] The randomly screened data may be screened using a random algorithm, and the number of screened data items may be set as needed, such as screening out 1,000 or 5,000 items, etc., without specific limitation.

[0024] S104: Generate labels based on whether the filtered data is complete, and expand the filtered data and its labels in a masking manner.

[0025] The above-mentioned tag can be set to a specified first value or a second value, the first value is used to identify that the corresponding data is complete, and the second value is used to identify that the corresponding data is incomplete.

[0026] For example, setting the tag value to 1 indicates that the corresponding data is complete, and setting the tag value to 0 indicates that the corresponding data is incomplete.

[0027] In the embodiment of the present invention, the above step S104 may specifically include: Classify the filtered data according to the data type to obtain a data classification table with M rows and K columns, where M is the number of filtered data and K is the number of data types; Generate M rows and K columns of labels to obtain a completeness table, where the labels are used to identify whether the data at the same position in the data classification table is complete; The method of duplicating the existing data in the data classification table and performing random masking on the duplicated data based on random numbers and random vectors is adopted to expand the data classification table and the integrity table.

[0028] Wherein, the above M and K are both positive integers, and the specific values ​​are not limited. For example, M=1000, K=5, then the data classification table includes 1000 data, and is divided into five types, such as source address, destination address, protocol type, port number and data length.

[0029] In one implementation manner, the method of copying the existing data in the data classification table, and randomly masking the copied data based on random numbers and random vectors, and expanding the data classification table and the integrity table may specifically include: Follow these steps to expand a row in the data classification table and the completeness table respectively, and repeat until the specified number of rows is reached: Select a random integer m between 1 and M and set a random vector. The random vector includes K random labels used to identify whether the data is complete. Add a new row in the data classification table, and copy the data of the mth row in the data classification table to the new row, determine the position of the random label in the random vector that identifies the incomplete data, and randomly mask the data at the same position in the new row of the data classification table; Add a new row to the completeness table, copy the random label in the mth row of the completeness table to the new row, and copy the random label in the random vector that indicates incomplete data to the same position in the new row of the completeness table.

[0030] S106: Convert the expanded data and labels into corresponding vector representations through a natural language processing model.

[0031] In the embodiment of the present invention, after the above step S106, the following steps may also be included: The Transformer model is used to positionally encode the vector representation and embed the position information of the elements in the expanded data into the vector representation.

[0032] The Transformer model mainly includes self-attention mechanism, parallel computing, encoder-decoder architecture, multi-head attention mechanism, position encoding, and non-convolution and non-loop structure. The self-attention mechanism is the core of the Transformer model, which allows the model to pay attention to all other elements in the sequence when processing a certain element in the sequence. This mechanism can capture long-distance dependencies, which has significant advantages when processing data with complex dependencies.

[0033] S108: Extract features from the vector representation based on the self-attention mechanism to obtain low-dimensional features.

[0034] In the embodiment of the present invention, the above step S108 may specifically include: The vector representation is input into the self-supervised learning model based on the self-attention mechanism to extract features to obtain high-dimensional features, and the high-dimensional features are subjected to feature dimensionality reduction to obtain low-dimensional features.

[0035] S110: Using low-dimensional features as input and binary classification results as output, a neural network model is trained to identify whether the data represented by the low-dimensional features is complete, thereby obtaining a completeness recognition model.

[0036] Among them, the binary classification results are used to identify whether the data is complete or incomplete.

[0037] In the embodiment of the present invention, before the above step S102, the following steps may also be included: When the system resource usage rate is lower than the preset threshold, start or resume the flow collection task to collect multiple process data; When the system resource usage is greater than or equal to a preset threshold, the traffic collection task is not started or is stopped.

[0038] The above method provided by the embodiment of the present invention randomly screens out part of the data from the multiple collected traffic data, generates labels according to whether the screened data is complete, expands the screened data and its labels in a masked manner, converts the expanded data and labels into corresponding vector representations through a natural language processing model, extracts features from the vector representations based on a self-attention mechanism to obtain low-dimensional features, uses the low-dimensional features as input and the binary classification results as output, trains a neural network model to identify whether the data represented by the low-dimensional features is complete, and obtains a completeness recognition model, wherein the binary classification results are used to identify whether the data is complete or incomplete. Therefore, expanding the data set for training based on a masked manner effectively reduces the dependence on labeled data, greatly reduces labor costs, and improves the efficiency of model training. Moreover, the processing based on the self-attention mechanism improves the ability of feature recognition and extraction, thereby improving the accuracy of model training.

[0039] like Figure 2 As shown, the embodiment of the present invention provides another training method for the integrity recognition model, which may include the following steps: S202: Randomly select some data from the multiple pieces of collected traffic data.

[0040] In the embodiment of the present invention, before the above step S202, the following steps may also be included: When the system resource usage rate is lower than the preset threshold, start or resume the flow collection task to collect multiple process data; When the system resource usage is greater than or equal to a preset threshold, the traffic collection task is not started or is stopped.

[0041] The system resource usage includes but is not limited to at least one of the following: memory, disk, CPU, etc. The system resource usage can reflect the current operating status of the system and help to find potential problems in a timely manner. In particular, when the amount of traffic data collected is large, if other tasks are running on the same server, collecting traffic while resources are highly occupied may cause thread deadlock, thereby affecting the stability and reliability of the system.

[0042] In the embodiment of the present invention, the above-mentioned preset threshold can be flexibly set. For example, a lower threshold can be set during peak hours to ensure that other services of the system are not affected; a higher threshold can be set during off-peak hours to fully utilize system resources. Such flexible configuration can be adjusted according to different business needs to ensure that system resources are optimally utilized.

[0043] Figure 3 A schematic diagram of the control flow of the flow collection task provided by an embodiment of the present invention. Figure 3 , dynamically judge the system resource usage rate and the preset threshold. If the system resource usage rate is greater than or equal to the preset threshold, the traffic collection task is suspended (or the traffic collection task is not started) and the system resource usage is waited for to decrease. If the system resource usage rate is less than the preset threshold, the traffic collection task is automatically restarted (or the traffic collection task is automatically started). This automatic restart mechanism can effectively prevent thread deadlock problems caused by resource overload, which not only improves the self-healing ability of the system, but also reduces the need for manual intervention, making the traffic collection process more efficient and reliable, reducing the impact of the traffic data collection task on other system services, and improving the stability and reliability of the system.

[0044] S204: Classify the filtered data according to the data type to obtain a data classification table with M rows and K columns, where M is the number of filtered data and K is the number of data types.

[0045] Wherein, M and K are both positive integers, and the specific values ​​can be set as needed, such as M is 3000 or 5000, K is 5, 6 or 8, etc., and the specific values ​​are not limited.

[0046] In one implementation, data may be divided into five types, including source address, destination address, protocol type, port number, and data length, and therefore, K=5 is set.

[0047] S206: Generate M rows and K columns of labels to obtain a completeness table, where the labels are used to identify whether the data at the same position in the data classification table is complete.

[0048] Among them, for any label in the completeness table, if the label is the first value, it indicates that the data at the same position in the data classification table is complete; if the label is the second value, it indicates that the data at the same position in the data classification table is incomplete.

[0049] For example, collect 1 million pieces of traffic data, randomly select 1,000 pieces of data, and generate a data classification table with 1,000 rows and 5 columns according to the data type source address, destination address, protocol type, port number and data length. Manually set labels for the 1,000 pieces of data to identify whether the source address, destination address, protocol type, port number and data length are complete, and obtain a completeness table with 1,000 rows and 5 columns. The cells in the data classification table correspond to the cells in the completeness table one by one. The label in the cell of the completeness table is 0, indicating that the data in the corresponding cell of the data classification table is incomplete, and the label in the cell of the completeness table is 1, indicating that the data in the corresponding cell of the data classification table is complete.

[0050] If the label of a row in the integrity table is 1,1,1,1,1, it means that the data in the same row in the corresponding data classification table are all complete. If the label of a row in the integrity table is 0,1,0,1,1, it means that the data in the same row in the corresponding data classification table are: source address incomplete, destination address complete, protocol type incomplete, port number complete, data length complete. I will not give examples one by one here.

[0051] S208: The data classification table and the integrity table are expanded by duplicating the existing data in the data classification table and performing random masking on the duplicated data based on random numbers and random vectors.

[0052] In the embodiment of the present invention, a masked language model (MLM) can be used to complete masking. Specifically, some words can be randomly selected from the input sentence of the model according to a preset masking rate, and they can be masked. Then the model is trained to eventually learn to predict these masked words, such as the Bert model. This technology can help the model learn useful word context representations, and then fine-tune it for specific natural language processing (NLP) tasks.

[0053] S210: Convert the expanded data and labels into corresponding vector representations through a natural language processing model.

[0054] In the embodiment of the present invention, the traffic data may be text data of a character string type. To facilitate subsequent processing, the traffic data usually needs to be quantified, such as converted into quantitative information such as numbers, vectors or matrices.

[0055] In the embodiment of the present invention, the above natural language processing model can specifically use the Word2vec model to quantify the traffic data. Of course, other natural language processing modules can also be used, which are not limited. Among them, the Word2vec model is a technology that maps words to vector space. It converts words in the text into vector representations with semantic similarity by training a neural network model.

[0056] For example, the Word2vec model can treat each element in each traffic data (such as a word, a letter, or a symbol) as a vocabulary, convert these elements into corresponding vector representations, and thus convert the original string data into numerical data that can be processed by a computer.

[0057] S212: Use the Transformer model to positionally encode the vector representation and embed the position information of the elements in the expanded data into the vector representation.

[0058] In an embodiment of the present invention, since traffic sending and transmission is a process with time series characteristics, in order to retain the position information in the traffic data, the position encoding technology in the Transformer model can be used to implement it. Position encoding is a method of embedding the position information of each element in a sequence into its vector representation. For example, sine position encoding can encode the position information through sine and cosine functions to generate a vector with the same dimension as the element vector. After these position vectors are added to the original element vector, the model can not only capture the semantic relationship between elements, but also understand the relative position of the elements in the sequence.

[0059] The above-mentioned mechanism of combining the Word2vec model and the Transformer model position encoding can convert traffic data into a numerical vector representation. This vector representation not only retains the original information of the traffic data, but also captures the complex relationship and position dependency between the data, thereby achieving normalized processing of the data and providing high-quality input data for subsequent deep learning models.

[0060] S214: Input the vector representation into a self-supervised learning model based on a self-attention mechanism to perform feature extraction to obtain high-dimensional features, and perform feature dimensionality reduction processing on the high-dimensional features to obtain low-dimensional features.

[0061] In an embodiment of the present invention, a self-supervised learning model based on a self-attention mechanism can complete effective learning and feature extraction by utilizing the structure and features of the data itself without relying on external labels, thereby realizing self-supervised learning without the need for data labels.

[0062] Figure 4A schematic diagram of the processing flow of the self-supervised learning model provided by an embodiment of the present invention. Figure 4 , the self-supervised learning module includes an encoding layer and a decoding layer. The encoding layer includes multiple modules connected in series, each module includes a self-attention layer, an activation layer and a normalization layer. The decoding layer also includes multiple modules connected in series, each module includes a normalization layer, an activation layer and a self-attention layer. The vector representation is input into the self-supervised learning model as input data, firstly randomly masked (i.e. pre-training), then encoded by the encoding layer, and decoded by the decoding layer, and finally the output data is obtained. Through the training of the self-supervised learning model, the output data and the data at the random mask position are learned. After the training is completed, the self-supervised learning model can learn the corresponding relationship and feature information between the data at different positions of the traffic data. The final output data can be high-dimensional features obtained by feature extraction, or low-dimensional features obtained by feature dimensionality reduction processing of high-dimensional features.

[0063] The above self-attention mechanism can capture complex dependencies in the data, such as capturing the relationship between distant elements in the data. This feature gives the self-attention mechanism a significant advantage in processing data with complex dependencies. Incorporating the above dependencies into the representation of elements can retain as much information as possible from the original data, thereby generating high-dimensional features containing rich semantic information. Through feature dimensionality reduction processing, high-dimensional data can be mapped to a low-dimensional space while retaining the key semantic information and features of the original data as much as possible. The low-dimensional features after dimensionality reduction not only reduce the redundancy of the data, but also reduce the computational complexity, thereby improving the efficiency and effectiveness of subsequent traffic collection integrity recognition tasks.

[0064] S216: Using low-dimensional features as input and binary classification results as output, a neural network model is trained to identify whether the data represented by the low-dimensional features is complete, thereby obtaining a completeness recognition model.

[0065] Among them, the binary classification result is used to identify whether the data is complete or incomplete, such as the binary classification result 1 indicates that the data is complete, and the binary classification result 0 indicates that the data is incomplete.

[0066] When the number of data types is K, the above binary classification result is specifically a K-dimensional binary classification result, that is, it includes the binary classification result of each type.

[0067] Figure 5 This is a schematic diagram of the principle of the integrity recognition model provided by the embodiment of the present invention. Figure 5, the neural network model can include multiple neural units, each of which includes an attention layer, a fully connected layer, an activation layer, and a normalization layer. The input of the neural network model is the above low-dimensional features, and the output is a binary classification result. The figure uses the data types including source address, destination address, port number, protocol type, and data length as an example for explanation. The output of the model includes 5 binary classification results, corresponding to: whether the source address is complete, whether the destination address is complete, whether the port number is complete, whether the protocol type is complete, and whether the data length is complete. Through repeated training and optimization, an accurate and efficient completeness recognition model can be obtained. The model can identify the completeness of traffic data, which helps to solve the incompleteness problem that occurs during data collection.

[0068] Figure 6 A schematic diagram of a process for expanding a data set provided by an embodiment of the present invention. Figure 6 , the above step S208 can specifically follow the following steps to expand one row in the data classification table and the integrity table respectively, and repeat the steps until the number of rows is expanded to the specified number: S602: Select a random integer m between 1 and M, and set a random vector, where the random vector includes K random labels for identifying whether the data is complete.

[0069] For example, if M is 1000, then a random integer m=105 or m=381 can be selected from 1 to 1000, without any specific limitation.

[0070] The value in the random vector can be a number between 0 and 1, such as the random vector vector=[1, 0, 0, 1, 1], where there are 5 random labels, the second and third identification data are incomplete, and the first, fourth and fifth identification data are complete.

[0071] S604: Add a new row to the data classification table, and copy the data of the mth row in the data classification table to the new row, determine the position of the random label that identifies the incomplete data in the random vector, and randomly mask the data at the same position in the new row of the data classification table.

[0072] Take the random vector vector=[1, 0, 0, 1, 1] as an example. After copying the data of the mth row in the data classification table to the newly added row, the second and third data in the newly added row are randomly masked. The random mask will make the data of the corresponding cell incomplete, that is, the expansion will obtain incomplete data.

[0073] S606: Add a new row to the completeness table, copy the random label in the mth row of the completeness table to the new row, and copy the random label in the random vector that indicates incomplete data to the same position in the new row of the completeness table.

[0074] Still taking the above random vector vector=[1, 0, 0, 1, 1] as an example, copy the random label of the mth row in the completeness table to the newly added row, and copy the second and third labels in the random vector to the second and third cells in the newly added row, that is, set the values ​​of the second and third columns of the newly added row to 0.

[0075] In the embodiment of the present invention, the above-mentioned designated number of rows can be preset, such as being set to 50,000, etc., and the specific value is not limited.

[0076] After the above process, we can get the expanded data classification table and completeness table. Each row of data in the data classification table can be restored to a complete text data, so that a specified number of traffic data with different completeness can be obtained, which provides a rich data set for model training, greatly saves the time of manual labeling, and reduces costs.

[0077] The above method provided by the embodiment of the present invention randomly selects part of the data from the collected multiple traffic data, generates labels according to whether the screened data is complete, expands the screened data and its labels by masking, converts the expanded data and labels into corresponding vector representations through a natural language processing model, extracts features from the vector representations based on a self-attention mechanism to obtain low-dimensional features, uses the low-dimensional features as input, and uses the binary classification results as output to train a neural network model to identify whether the data represented by the low-dimensional features is complete, thereby obtaining a completeness recognition model, wherein the binary classification results are used to identify whether the data is complete or incomplete. Therefore, training based on the masked method to expand the data set effectively reduces the dependence on label data, greatly reduces labor costs, and improves the efficiency of model training. Moreover, the processing based on the self-attention mechanism improves the ability of feature recognition and extraction, thereby improving the accuracy of model training. In addition, by continuously optimizing and upgrading the model, it can adapt to various complex and changeable application scenarios, improve the function of the completeness recognition model, and improve accuracy and reliability.

[0078] like Figure 7 As shown, the embodiment of the present invention also provides a method for identifying flow collection integrity, which may include the following steps: S702: Obtain traffic collection data.

[0079] There are usually multiple data items collected for traffic flow, and traffic flow data within a specified time range can be collected as needed, without specific limitation.

[0080] S704: Convert the data into corresponding vector representations through a natural language processing model.

[0081] S706: Extract features from the vector representation based on the self-attention mechanism to obtain low-dimensional features.

[0082] In this embodiment, feature extraction is performed on the vector representation based on the self-attention mechanism, and high-dimensional features can be obtained first, and then dimensionality reduction processing is performed to obtain low-dimensional features. Please refer to the description in the above embodiment, which will not be repeated here.

[0083] Among them, based on the self-attention mechanism, important semantic information can be extracted and redundant features can be removed. The obtained low-dimensional features have greatly reduced the data scale, which not only reduces the redundant information of the data, simplifies the dimension of the data, effectively reduces the complexity of the data, but also significantly reduces the computational complexity and storage requirements. It helps the completeness recognition model to identify incomplete data more quickly, making subsequent processing more efficient, and further improving the quality and reliability of traffic collection.

[0084] S708: Input the low-dimensional features into the integrity recognition model to identify whether the data corresponding to the low-dimensional features are complete.

[0085] The above-mentioned integrity recognition model is an integrity recognition model obtained by training using the method for training an integrity recognition model provided by any of the above-mentioned embodiments.

[0086] In this embodiment, the collected traffic data may include multiple types, and the result of the integrity recognition model identification includes whether each type of data is complete.

[0087] For example, the collected traffic data includes: source address, destination address, port number, protocol type and data length. The output results of the integrity recognition model include 5 binary classification results, corresponding to: whether the source address is complete, whether the destination address is complete, whether the port number is complete, whether the protocol type is complete and whether the data length is complete.

[0088] In the embodiment of the present invention, after the above step S708, the following may also be included: If incomplete data is identified, a pre-packaged script is called to start re-collection of the incomplete data based on the position information of the elements in the data.

[0089] For example, if it is identified that the port number is incomplete, a prepackaged script may be called to initiate re-collection of the data with the incomplete port number.

[0090] By calling the script to automatically start re-collection, the continuity and efficiency of the data collection process can be ensured, the integrity of the flow packet capture can be greatly improved, and the quality of the collected data can be greatly improved, laying a data foundation for subsequent performance analysis such as flow playback. Moreover, this automated processing method can reduce labor costs, improve the efficiency of flow collection, and significantly improve data quality and system stability.

[0091] The above-mentioned method of feature dimensionality reduction and model iterative updating based on self-supervised learning not only improves the adaptability and generalization ability of the model, but also significantly improves the processing speed and efficiency, providing strong technical support for the automatic monitoring and optimization of traffic integrity.

[0092] The above method provided by the embodiment of the present invention obtains the data of traffic collection, converts the data into corresponding vector representation through a natural language processing model, extracts features of the vector representation based on the self-attention mechanism to obtain low-dimensional features, inputs the low-dimensional features into the integrity recognition model, and identifies whether the data corresponding to the low-dimensional features are complete. It can obtain high-quality recognition results without human intervention, realizes automatic recognition, can quickly identify incomplete data, improves the accuracy of traffic collection integrity recognition, and greatly improves the quality and reliability of traffic collection.

[0093] Based on the same idea as the training method of the integrity recognition model provided in the embodiment of the present invention, the embodiment of the present invention also provides a training device for the integrity recognition model, such as Figure 8 As shown, the device may include: The screening module 801 is used to randomly screen out some data from the multiple pieces of collected traffic data.

[0094] The expansion module 802 is used to generate labels according to whether the screening data is complete, and expand the screening data and its labels in a masking manner.

[0095] The conversion module 803 is used to convert the expanded data and labels into corresponding vector representations through a natural language processing model.

[0096] The extraction module 804 is used to extract features from the vector representation based on the self-attention mechanism to obtain low-dimensional features.

[0097] The training module 805 is used to train a neural network model to identify whether the data represented by the low-dimensional features is complete using low-dimensional features as input and binary classification results as output, thereby obtaining a completeness recognition model, wherein the binary classification results are used to identify whether the data is complete or incomplete.

[0098] In the embodiment of the present invention, the expansion module 802 can be used for: Classify the filtered data according to the data type to obtain a data classification table with M rows and K columns, where M is the number of filtered data and K is the number of data types; Generate M rows and K columns of labels to obtain a completeness table, where the labels are used to identify whether the data at the same position in the data classification table is complete; The method of duplicating the existing data in the data classification table and performing random masking on the duplicated data based on random numbers and random vectors is adopted to expand the data classification table and the integrity table.

[0099] In one implementation manner, the method of copying the existing data in the data classification table, and randomly masking the copied data based on random numbers and random vectors, and expanding the data classification table and the integrity table may include: Follow these steps to expand a row in the data classification table and the completeness table respectively, and repeat until the specified number of rows is reached: Select a random integer m between 1 and M and set a random vector. The random vector includes K random labels used to identify whether the data is complete. Add a new row in the data classification table, and copy the data of the mth row in the data classification table to the new row, determine the position of the random label in the random vector that identifies the incomplete data, and randomly mask the data at the same position in the new row of the data classification table; Add a new row to the completeness table, copy the random label in the mth row of the completeness table to the new row, and copy the random label in the random vector that indicates incomplete data to the same position in the new row of the completeness table.

[0100] In the embodiment of the present invention, the conversion module 803 is further used for: After the expanded data and labels are converted into corresponding vector representations through the natural language processing model, the Transformer model is used to positionally encode the vector representation and embed the position information of the elements in the expanded data into the vector representation.

[0101] In the embodiment of the present invention, the extraction module 804 may be used to: The vector representation is input into the self-supervised learning model based on the self-attention mechanism to extract features to obtain high-dimensional features, and the high-dimensional features are subjected to feature dimensionality reduction to obtain low-dimensional features.

[0102] In the embodiment of the present invention, the above device can also be used for: Among the multiple flow data collected, before randomly screening out some data, when the system resource usage is lower than the preset threshold, the flow collection task is started or resumed to collect multiple process data; when the system resource usage is greater than or equal to the preset threshold, the flow collection task is not started or is stopped.

[0103] The above-mentioned device provided by the embodiment of the present invention can execute the method in any of the above-mentioned method embodiments. For detailed processes, please refer to the description of each method embodiment, which will not be repeated here.

[0104] The above-mentioned device provided by the embodiment of the present invention randomly screens out part of the data from the multiple collected traffic data, generates labels according to whether the screened data is complete, expands the screened data and its labels in a masked manner, converts the expanded data and labels into corresponding vector representations through a natural language processing model, extracts features from the vector representations based on a self-attention mechanism to obtain low-dimensional features, uses the low-dimensional features as input and the binary classification results as output, trains a neural network model to identify whether the data represented by the low-dimensional features is complete, and obtains a completeness recognition model, wherein the binary classification results are used to identify whether the data is complete or incomplete. Therefore, expanding the data set for training based on a masked manner effectively reduces the dependence on label data, greatly reduces labor costs, and improves the efficiency of model training. Moreover, the processing based on the self-attention mechanism improves the ability of feature recognition and extraction, thereby improving the accuracy of model training.

[0105] Based on the same idea as the method for identifying the integrity of flow collection provided in the embodiment of the present invention, the embodiment of the present invention also provides a device for identifying the integrity of flow collection, such as Fig. 9 As shown, the device may include: The acquisition module 901 is used to acquire traffic collection data.

[0106] The conversion module 902 is used to convert the data into corresponding vector representations through a natural language processing model.

[0107] The extraction module 903 is used to extract features from the vector representation based on the self-attention mechanism to obtain low-dimensional features.

[0108] The identification module 904 is used to input the low-dimensional features into the integrity identification model to identify whether the data corresponding to the low-dimensional features are complete.

[0109] The above-mentioned integrity recognition model is a integrity recognition model trained using the training method of the integrity recognition model provided in the above-mentioned embodiment.

[0110] In the embodiment of the present invention, the identification module 904 may also be used for: If incomplete data is identified, a pre-packaged script is called to start re-collection of the incomplete data based on the position information of the elements in the data.

[0111] The above-mentioned device provided by the embodiment of the present invention can execute the method in any of the above-mentioned method embodiments. For detailed processes, please refer to the description of each method embodiment, which will not be repeated here.

[0112] The above-mentioned device provided by the embodiment of the present invention obtains the data of traffic collection, converts the data into corresponding vector representation through a natural language processing model, extracts features of the vector representation based on the self-attention mechanism to obtain low-dimensional features, inputs the low-dimensional features into the integrity recognition model, and identifies whether the data corresponding to the low-dimensional features are complete. It can obtain high-quality recognition results without human intervention, realizes automatic recognition, can quickly identify incomplete data, improves the accuracy of traffic collection integrity recognition, and greatly improves the quality and reliability of traffic collection.

[0113] It is worth mentioning that in the embodiment of the present invention, the training device of the integrity recognition model and the flow collection integrity recognition device can be deployed in the same electronic device or in different electronic devices, and the embodiment of the present invention does not make specific limitations on this.

[0114] For example, the integrity identification model is trained and used for traffic identification in the same server; or, the integrity identification model is trained in one server and used for traffic identification in another server. It can be deployed according to actual needs, and the embodiments of the present invention do not make specific limitations on this.

[0115] Fig.10 The hardware structure diagram of an electronic device for implementing various embodiments of the present invention is shown below. The electronic device 1000 includes but is not limited to: a radio frequency unit 1001, a network module 1002, an audio output unit 1003, an input unit 1004, a sensor 1005, a display unit 1006, a user input unit 1007, an interface unit 1008, a memory 1009, a processor 1010, and a power supply 1011. Those skilled in the art will appreciate that Fig.10 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and the electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. In the embodiments of the present invention, the electronic device includes but is not limited to a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted terminal, a wearable device, and a pedometer.

[0116] Among them, the processor 1010 is used to randomly filter out some data from multiple collected traffic data; generate labels according to whether the filtered data is complete, and expand the filtered data and its labels by masking; convert the expanded data and labels into corresponding vector representations through a natural language processing model; extract features of the vector representation based on the self-attention mechanism to obtain low-dimensional features; use the low-dimensional features as input and the binary classification results as output to train a neural network model to identify whether the data represented by the low-dimensional features is complete, and obtain a completeness recognition model, wherein the binary classification results are used to identify whether the data is complete or incomplete.

[0117] Alternatively, processor 1010 is used to obtain traffic collection data; convert the data into corresponding vector representations through a natural language processing model; extract features from the vector representations based on a self-attention mechanism to obtain low-dimensional features; input the low-dimensional features into an integrity recognition model to identify whether the data corresponding to the low-dimensional features is complete; wherein the integrity recognition model is an integrity recognition model trained using the training method of the integrity recognition model as described above.

[0118] An embodiment of the present invention provides an electronic device, which randomly screens out part of the data from multiple collected traffic data, generates labels according to whether the screened data is complete, expands the screened data and its labels in a masking manner, converts the expanded data and labels into corresponding vector representations through a natural language processing model, extracts features from the vector representations based on a self-attention mechanism to obtain low-dimensional features, uses the low-dimensional features as input and a binary classification result as output, trains a neural network model to identify whether the data represented by the low-dimensional features is complete, and obtains a completeness recognition model, wherein the binary classification result is used to identify whether the data is complete or incomplete. Therefore, expanding the data set for training based on a masking method effectively reduces the dependence on label data, greatly reduces labor costs, and improves the efficiency of model training. Moreover, the processing based on the self-attention mechanism improves the ability of feature recognition and extraction, thereby improving the accuracy of model training. Alternatively, by acquiring traffic collection data, converting the data into corresponding vector representations through a natural language processing model, performing feature extraction on the vector representation based on a self-attention mechanism to obtain low-dimensional features, and inputting the low-dimensional features into an integrity recognition model to identify whether the data corresponding to the low-dimensional features is complete. High-quality recognition results can be obtained without human intervention, and automated recognition is achieved. Incomplete data can be quickly identified, thereby improving the accuracy of traffic collection integrity recognition and greatly improving the quality and reliability of traffic collection.

[0119] It should be understood that in the embodiment of the present invention, the RF unit 1001 can be used for receiving and sending signals during information transmission or calls. Specifically, after receiving downlink data from the base station, it is sent to the processor 1010 for processing; in addition, uplink data is sent to the base station. Generally, the RF unit 1001 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, etc. In addition, the RF unit 1001 can also communicate with the network and other electronic devices through a wireless communication system.

[0120] The electronic device provides users with wireless broadband Internet access through the network module 1002, such as helping users to send and receive emails, browse web pages, and access streaming media.

[0121] The audio output unit 1003 can convert the audio data received by the RF unit 1001 or the network module 1002 or stored in the memory 1009 into an audio signal and output it as sound. Moreover, the audio output unit 1003 can also provide audio output related to a specific function performed by the electronic device 1000 (for example, a call signal reception sound, a message reception sound, etc.). The audio output unit 1003 includes a speaker, a buzzer, a receiver, etc.

[0122] The input unit 1004 is used to receive audio or video signals. The input unit 1004 may include a graphics processor (GPU) 10041 and a microphone 10042. The graphics processor 10041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The processed image frame can be displayed on the display unit 1006. The image frame processed by the graphics processor 10041 can be stored in the memory 1009 (or other storage medium) or sent via the radio frequency unit 1001 or the network module 1002. The microphone 10042 can receive sound and can process such sound into audio data. The processed audio data can be converted into a format output that can be sent to a mobile communication base station via the radio frequency unit 1001 in the case of a telephone call mode.

[0123] The electronic device 1000 also includes at least one sensor 1005, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor includes an ambient light sensor and a proximity sensor, wherein the ambient light sensor can adjust the brightness of the display panel 10061 according to the brightness of the ambient light, and the proximity sensor can turn off the display panel 10061 and / or the backlight when the electronic device 1000 is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary, which can be used to identify the posture of the electronic device (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; the sensor 1005 can also include a fingerprint sensor, a pressure sensor, an iris sensor, a molecular sensor, a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, etc., which will not be repeated here.

[0124] The display unit 1006 is used to display information input by the user or information provided to the user. The display unit 1006 may include a display panel 10061, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0125] The user input unit 1007 can be used to receive input digital or character information, and generate key signal input related to user settings and function control of the electronic device. Specifically, the user input unit 1007 includes a touch panel 10071 and other input devices 10072. The touch panel 10071, also known as a touch screen, can collect user touch operations on or near it (such as operations performed by users using fingers, styluses, or any other suitable objects or accessories on or near the touch panel 10071). The touch panel 10071 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact point coordinates, and then sends it to the processor 1010, receives the command sent by the processor 1010 and executes it. In addition, the touch panel 10071 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch panel 10071, the user input unit 1007 may also include other input devices 10072. Specifically, other input devices 10072 may include but are not limited to physical keyboards, function keys (such as volume control keys, switch keys, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0126] Furthermore, the touch panel 10071 may be overlaid on the display panel 10061. When the touch panel 10071 detects a touch operation on or near it, it is transmitted to the processor 1010 to determine the type of touch event. Then, the processor 1010 provides corresponding visual output on the display panel 10061 according to the type of touch event. Fig.10 In the figure, the touch panel 10071 and the display panel 10061 are two independent components to realize the input and output functions of the electronic device. However, in some embodiments, the touch panel 10071 and the display panel 10061 can be integrated to realize the input and output functions of the electronic device, which is not limited here.

[0127] The interface unit 1008 is an interface for connecting an external device to the electronic device 1000. For example, the external device may include a wired or wireless headset port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device with an identification module, an audio input / output (I / O) port, a video I / O port, a headphone port, etc. The interface unit 1008 may be used to receive input (e.g., data information, power, etc.) from an external device and transmit the received input to one or more elements within the electronic device 1000 or may be used to transmit data between the electronic device 1000 and an external device.

[0128] The memory 1009 can be used to store software programs and various data. The memory 1009 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 1009 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0129] The processor 1010 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 1009 and calling data stored in the memory 1009, it performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. The processor 1010 may include one or more processing units; preferably, the processor 1010 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1010.

[0130] The electronic device 1000 may also include a power source 1011 (such as a battery) for supplying power to various components. Preferably, the power source 1011 may be logically connected to the processor 1010 via a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system.

[0131] Preferably, an embodiment of the present invention further provides an electronic device, comprising a processor 1010, a memory 1009, and a computer program stored in the memory 1009 and executable on the processor 1010. When the computer program is executed by the processor 1010, the various processes of the above-mentioned method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0132] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the various processes of the above-mentioned method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0133] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each process of the above method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0134] The above-mentioned computer-readable storage medium provided by the embodiment of the present invention randomly screens out part of the data from the multiple collected traffic data, generates labels according to whether the screened data is complete, expands the screened data and its labels in a masked manner, converts the expanded data and labels into corresponding vector representations through a natural language processing model, extracts features from the vector representations based on a self-attention mechanism to obtain low-dimensional features, uses the low-dimensional features as input and the binary classification results as output, trains a neural network model to identify whether the data represented by the low-dimensional features is complete, and obtains a completeness recognition model, wherein the binary classification results are used to identify whether the data is complete or incomplete. Therefore, expanding the data set for training based on a masked manner effectively reduces the dependence on label data, greatly reduces labor costs, and improves the efficiency of model training. Moreover, the processing based on the self-attention mechanism improves the ability of feature recognition and extraction, thereby improving the accuracy of model training. Alternatively, by acquiring traffic collection data, converting the data into corresponding vector representations through a natural language processing model, performing feature extraction on the vector representation based on a self-attention mechanism to obtain low-dimensional features, and inputting the low-dimensional features into an integrity recognition model to identify whether the data corresponding to the low-dimensional features is complete. High-quality recognition results can be obtained without human intervention, and automated recognition is achieved. Incomplete data can be quickly identified, thereby improving the accuracy of traffic collection integrity recognition and greatly improving the quality and reliability of traffic collection.

[0135] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0136] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0137] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0138] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0139] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0140] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0141] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. According to the definition in this article, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0142] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0143] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A training method for an integrity recognition model, characterized in that: include: Randomly select some data from the multiple collected traffic data; Based on whether the labels of the filtered data are completely generated, the filtered data and its labels are expanded by using a mask method; The expanded data and labels are converted into corresponding vector representations through a natural language processing model; Extracting features from the vector representation based on a self-attention mechanism to obtain low-dimensional features; Taking the low-dimensional features as input and the binary classification results as output, a neural network model is trained to identify whether the data represented by the low-dimensional features is complete, thereby obtaining a completeness recognition model, wherein the binary classification results are used to identify whether the data is complete or incomplete.

2. The method according to claim 1, characterized in that: The step of generating labels based on whether the screening data is complete and expanding the screening data and its labels in a mask manner includes: Classify the filtered data according to the data type to obtain a data classification table with M rows and K columns, where M is the number of filtered data and K is the number of data types; Generate M rows and K columns of labels to obtain a completeness table, where the labels are used to identify whether the data at the same position in the data classification table is complete; The method of duplicating the existing data in the data classification table and performing random masking on the duplicated data based on random numbers and random vectors is adopted to expand the data classification table and the integrity table.

3. The method according to claim 2, characterized in that The method of duplicating the existing data in the data classification table and performing random masking on the duplicated data based on random numbers and random vectors, and expanding the data classification table and the integrity table includes: Follow these steps to expand a row in the data classification table and the completeness table respectively, and repeat until the specified number of rows is reached: A random integer m is selected between 1 and M, and a random vector is set, where the random vector includes K random labels for identifying whether the data is complete; Add a new row in the data classification table, and copy the data of the mth row in the data classification table to the new row, determine the position of the random label in the random vector that identifies the incomplete data, and randomly mask the data at the same position in the new row of the data classification table; Add a new row to the completeness table, copy the random label in the mth row of the completeness table to the new row, and copy the random label in the random vector that indicates incomplete data to the same position in the new row of the completeness table.

4. The method according to claim 1, characterized in that After the expanded data and labels are converted into corresponding vector representations through a natural language processing model, the following further includes: The Transformer model is used to perform position encoding on the vector representation, and the position information of the elements in the expanded data is embedded into the vector representation.

5. The method according to claim 1, characterized in that The step of extracting features from the vector representation based on the self-attention mechanism to obtain low-dimensional features includes: The vector representation is input into a self-supervised learning model based on a self-attention mechanism to perform feature extraction to obtain high-dimensional features, and feature dimensionality reduction processing is performed on the high-dimensional features to obtain low-dimensional features.

6. The method according to any one of claims 1 to 5, characterized in that Before randomly screening out some of the collected multiple pieces of traffic data, the method further includes: When the system resource usage rate is lower than the preset threshold, start or resume the flow collection task to collect multiple process data; When the system resource usage is greater than or equal to a preset threshold, the traffic collection task is not started or is stopped.

7. A method for identifying flow collection integrity, characterized in that: include: Obtain traffic collection data; Convert the data into corresponding vector representations through a natural language processing model; Extracting features from the vector representation based on a self-attention mechanism to obtain low-dimensional features; Inputting the low-dimensional features into a completeness recognition model to identify whether the data corresponding to the low-dimensional features are complete; The integrity recognition model is a integrity recognition model trained using the method described in any one of claims 1 to 6.

8. The method according to claim 7, characterized in that After identifying whether the data corresponding to the low-dimensional feature is complete, the method further includes: If incomplete data is identified, a pre-packaged script is called based on the position information of the elements in the data to start re-collection of the incomplete data.

9. A training device for an integrity recognition model, characterized in that: include: A screening module is used to randomly screen out some data from multiple collected traffic data; The expansion module is used to generate labels according to whether the screening data is complete, and to expand the screening data and its labels in a masking manner; The conversion module is used to convert the expanded data and labels into corresponding vector representations through the natural language processing model; An extraction module, used for extracting features from the vector representation based on a self-attention mechanism to obtain low-dimensional features; A training module is used to take the low-dimensional features as input and the binary classification results as output, train a neural network model to identify whether the data represented by the low-dimensional features is complete, and obtain a completeness recognition model, wherein the binary classification results are used to identify whether the data is complete or incomplete.

10. A flow collection integrity identification device, characterized in that: include: An acquisition module is used to acquire traffic collection data; A conversion module, used for converting the data into corresponding vector representations through a natural language processing model; An extraction module, used for extracting features from the vector representation based on a self-attention mechanism to obtain low-dimensional features; An identification module, used for inputting the low-dimensional features into a completeness identification model to identify whether the data corresponding to the low-dimensional features are complete; The integrity recognition model is a integrity recognition model trained using the method described in any one of claims 1 to 6.

11. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the method according to any one of claims 1 to 8 when executed by the processor.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

13. A computer program product, characterized in that The invention comprises a computer program, which implements the steps of the method according to any one of claims 1 to 8 when being executed by a processor.

Images