Message data lagging detection method, system and device and storage medium

Abstract

The embodiment of the invention relates to the technical field of communication, and discloses a message data lag detection method, system and device and a storage medium, and the method comprises the steps: carrying out the preprocessing of message data; taking a convolutional neural network as a baseline model, and constructing a feature engineering dimensionality reduction model; carrying out dimensionality reduction on the preprocessed message data through a feature engineering dimensionality reduction model, and outputting a message feature vector after dimensionality reduction; and inputting the message feature vector into an anomaly detection algorithm for lagging judgment. According to the embodiment of the invention, in the actual use process, the objectives of acquiring the message data in real time, preprocessing the message data and reducing the dimension of the feature engineering can be realized, and meanwhile, the processed data can be directly input into the existing anomaly judgment model, and various application lagging events can be monitored in real time.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of communication technology, and in particular to an intelligent network physical freeze detection method, system, device and storage medium for large data messages. Background Art

[0002] The perceived lag is an important indicator for judging network quality and a key technology for operators to ensure user experience. The implementation of this technology involves multiple fields, including network protocols and data communications, system performance monitoring, packet capture and analysis, distributed system management, and lag abnormality judgment. The current mainstream method for lag abnormality judgment is to manually collect and analyze system and application logs for each application, create an expert system, and identify abnormal events and performance issues through keyword and pattern recognition.

[0003] When using a deep learning model for physical lag detection, the message data has the characteristics of high dimensionality and large quantity. Directly inputting the message data into the existing abnormal judgment model will lead to problems such as excessive noise caused by high dimensionality, slow processing speed, and increased hardware requirements, which will affect the accuracy of subsequent lag algorithms. Summary of the invention

[0004] The embodiments of the present disclosure provide a message data jam detection method, system, device and storage medium to solve or alleviate one or more of the above technical problems in the prior art.

[0005] According to one aspect of the present disclosure, a method for detecting message data jamming is provided, comprising:

[0006] Preprocess the message data;

[0007] Using convolutional neural network as the baseline model, we built a feature engineering dimensionality reduction model.

[0008] The pre-processed message data is reduced in dimension through the feature engineering dimension reduction model, and the reduced-dimensional message feature vector is output;

[0009] The message feature vector is input into the anomaly detection algorithm for jamming judgment.

[0010] In a possible implementation, preprocessing the message data includes:

[0011] Detect and process missing values, outliers, and duplicate values ​​in message data;

[0012] The detected and processed message data is normalized to map the numerical ranges of different features to the same scale.

[0013] In a possible implementation, the preprocessing of the message data includes:

[0014] Based on the preprocessed message data, sequence data is created and the head-to-tail coupling relationship of the data is captured.

[0015] In a possible implementation, creating sequence data and capturing the head-to-tail coupling relationship of the data based on the preprocessed message data includes:

[0016] Define a sliding window of set size and step length on the message data;

[0017] Generate subsequences by sliding the sliding window over the message data;

[0018] The subsequence is used as input feature and the subsequent value of the subsequence is used as target value and input into the convolutional neural network.

[0019] In one possible implementation, using a convolutional neural network as the baseline model, building a feature engineering dimensionality reduction model includes:

[0020] Adjust the convolution kernel width of the convolutional neural network;

[0021] Adding deformable convolutional layers to convolutional neural networks.

[0022] In one possible implementation, adjusting the convolution kernel width of the convolutional neural network includes:

[0023] The convolution kernel width is adjusted to 1 so that the convolutional neural network performs convolution operation on a single message data each time.

[0024] In one possible implementation, adding a deformable convolutional layer to a convolutional neural network includes:

[0025] Pass the preprocessed message data to the offset to generate the convolution kernel;

[0026] Generating a convolution kernel through the offset and performing a convolution operation to generate a set of offsets;

[0027] The offset and message data feature map are input into the added deformable convolution layer for convolution operation.

[0028] According to one aspect of the present disclosure, a message data jam detection system is provided, comprising:

[0029] A preprocessing unit, used for preprocessing message data;

[0030] A construction unit for building a feature engineering dimensionality reduction model using a convolutional neural network as a baseline model;

[0031] The feature engineering dimensionality reduction model is used to reduce the dimension of the preprocessed message data and output the message feature vector after dimensionality reduction;

[0032] A judgment unit is used to input the message feature vector into an anomaly detection algorithm for jamming judgment.

[0033] According to one aspect of the present disclosure, there is provided a device, comprising:

[0034] Processor and memory;

[0035] The memory is used to store a computer program, and the processor calls the computer program stored in the memory to execute any of the above-mentioned message data jamming detection methods.

[0036] According to one aspect of the present disclosure, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the processor is enabled to execute any of the above-mentioned message data jamming detection methods.

[0037] The exemplary embodiments of the present disclosure have the following beneficial effects: The exemplary embodiments of the present disclosure, for message big data, propose a preprocessing and feature engineering dimensionality reduction method specifically for massive message data, and apply it to the jam detection system. Specifically, through data preprocessing, the outliers in the data are cleaned, and necessary information such as delay is calculated. At the same time, the convolutional neural network is used as the baseline model, and various improvements are made to it for massive messages to construct a feature engineering dimensionality reduction model. In actual use, the goals of real-time acquisition of message data, preprocessing and feature engineering dimensionality reduction can be achieved. At the same time, the processed data can be directly input into the existing abnormal judgment model to monitor various application jam events in real time.

[0038] The details of one or more embodiments of the present application are presented in the following drawings and descriptions. Other features and advantages of the present application will become apparent from the accompanying drawings. It should be understood that the above general description and the detailed descriptions below are only exemplary and explanatory and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0040] Figure 1 is one of the flow charts of a method for detecting message data jamming of the present exemplary embodiment;

[0041] Figure 2This is the second flow chart of a method for detecting message data jamming in this exemplary embodiment;

[0042] Figure 3 is a data preprocessing flow chart of the present exemplary embodiment;

[0043] Figure 4 is a convolution model flow chart of this exemplary embodiment;

[0044] Figure 5 is a schematic diagram of sequence data of this exemplary embodiment;

[0045] Figure 6 is a schematic diagram of the convolution effect of a convolution kernel with a width of 1 in this exemplary embodiment;

[0046] Figure 7 is a deformable convolution calculation flow chart of this exemplary embodiment;

[0047] Figure 8 is a schematic diagram of the training and deployment process of this exemplary embodiment;

[0048] Fig. 9 is a block diagram of a message data jam detection system of the present exemplary embodiment;

[0049] Fig.10 It is a schematic structural diagram of a device of this exemplary embodiment. DETAILED DESCRIPTION

[0050] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0051] In addition, the accompanying drawings are only schematic illustrations of the present disclosure. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities, which do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware units or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0052] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the steps. For example, some steps may be decomposed, while some steps may be combined or partially combined, so the actual execution order may change according to the actual situation.

[0053] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example.

[0054] In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or sub-modules is not necessarily limited to those steps or sub-modules explicitly listed, but may include other steps or sub-modules not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0055] Figure 1 is one of the flow charts of a method for detecting message data jamming in this exemplary embodiment. Figure 1 As shown, an exemplary embodiment of the present disclosure provides a method for detecting message data jamming, including:

[0056] S1 pre-processes the message data;

[0057] S2 uses convolutional neural network as the baseline model to build a feature engineering dimensionality reduction model;

[0058] S3 reduces the dimension of the preprocessed message data through the feature engineering dimensionality reduction model and outputs the message feature vector after dimensionality reduction;

[0059] S4 inputs the message feature vector into an anomaly detection algorithm for jamming judgment.

[0060] Figure 2 This is a second flow chart of a method for detecting message data jamming in this exemplary embodiment. Figure 2As shown, this embodiment first pre-processes the massive message information, preliminarily calculates the delay, packet loss and other information, and performs pre-processing operations such as missing value filling; then the message data is input into the improved convolutional neural network. The specific improvement method is: improve the data reading form of the convolutional neural network, optimize the convolution kernel width, and add a deformable convolution kernel; finally, the extracted message features are input into the anomaly detection algorithm for jamming judgment. In summary, this embodiment improves the accuracy of jamming monitoring by pre-processing the data and constructing feature engineering for massive message data to reduce the data dimension.

[0061] Message data dimensionality reduction is to use convolutional neural network (CNN) to reduce the dimension of message data. Convolutional neural network is composed of multiple convolution structures. The calculation process of a single convolution structure is as follows: Figure 4 As shown in the figure, the message data is first convolved through the convolution kernel to extract the local features of the message information and reduce the feature dimension. Then the pooling layer further extracts features to reduce the feature dimension.

[0062] The convolution kernel extracts features from the input matrix E and uses multiple convolution kernels for feature mapping. The size of the convolution kernel is generally k×d, where k is the size of the convolution kernel and d is the dimension of the data. Each convolution kernel slides on E at a certain step length to extract the output features corresponding to the convolution kernel to form a feature matrix C. If there are m convolution kernels, the feature matrix C can be expressed as:

[0063] C=[c1,c2,c3,...,c m ];

[0064] The pooling layer reduces the dimension of the feature matrix C. It downsamples through the pooling layer and only saves the important parts of the feature matrix C. The pooling layer can effectively reduce the number of model parameters, thereby improving the training speed of the model. After the feature matrix C passes through the pooling layer, m one-dimensional vectors are obtained. All the vectors are concatenated to obtain a one-dimensional vector, which can be expressed as:

[0065]

[0066] Among them, concat(·) represents the concatenation operation.

[0067] Specifically, S1 preprocesses the message data including:

[0068] Detect and process missing values, outliers, and duplicate values ​​in message data;

[0069] The detected and processed message data is normalized to map the numerical ranges of different features to the same scale.

[0070] like Figure 3As shown in the figure, in the data preprocessing stage, the original message data is first quality checked and cleaned. This includes detecting and processing missing values, outliers, and duplicate values ​​in the data to ensure the integrity and consistency of the data. Next, the message data is normalized to map the numerical ranges of different features to the same scale to eliminate the impact of dimensional differences between features on the model.

[0071] Specifically, after S1 preprocesses the message data, it includes:

[0072] Based on the preprocessed message data, sequence data is created and the head-to-tail coupling relationship of the data is captured.

[0073] Specifically, based on the preprocessed message data, creating sequence data and capturing the head-to-tail coupling relationship of the data includes:

[0074] Define a sliding window of set size and step length on the message data;

[0075] Generate subsequences by sliding the sliding window over the message data;

[0076] The subsequence is used as input feature and the subsequent value of the subsequence is used as target value and input into the convolutional neural network.

[0077] Specifically, S2 uses a convolutional neural network as the baseline model to build a feature engineering dimensionality reduction model including:

[0078] Adjust the convolution kernel width of the convolutional neural network;

[0079] Adding deformable convolutional layers to convolutional neural networks.

[0080] This embodiment captures the coupling relationship between the head and tail data by creating sequence data. Figure 5 As shown in the figure, taking step size 2 as an example, a window with a step size of 2 is defined on the original data to generate multiple overlapping subsequences, each of which can be used as an input feature of the model, and a value after it is used as the target value. Based on the above operation, the end of each window overlaps with the beginning of the next window, solving the problem that it is difficult to capture coupling relationships between data that are far apart.

[0081] Specifically, adjusting the convolution kernel width of the convolutional neural network includes:

[0082] The convolution kernel width is adjusted to 1 so that the convolutional neural network performs convolution operation on a single message data each time.

[0083] like Figure 6As shown in the figure, considering that the timing information is extracted by the subsequent timing model, the convolution kernel width is adjusted to 1, so that the CNN (convolutional neural network) performs convolution operation on a single message information each time, discards useless timing information between multiple messages, and focuses on extracting features between multiple dimensions of a single message, reducing invalid information in dimensionality reduction and improving the efficiency of dimensionality reduction.

[0084] Specifically, adding a deformable convolutional layer to a convolutional neural network involves:

[0085] Pass the preprocessed message data to the offset to generate the convolution kernel;

[0086] Generating a convolution kernel through the offset and performing a convolution operation to generate a set of offsets;

[0087] The offset and message data feature map are input into the added deformable convolution layer for convolution operation.

[0088] This embodiment introduces a learnable offset so that the sampling position during each convolution operation is dynamically adjusted, thereby capturing richer spatial information. Figure 7 As shown in the figure, in the improved convolutional neural network, the preprocessed message data is first passed to the offset to generate the convolution kernel. The convolution layer generates a set of offsets through standard convolution operations, and its output shape is (Batch, Height, Width, 2*K), where K is the size of the convolution kernel, and the generated offset is used to dynamically adjust the sampling position of the convolution kernel. In the next deformable convolution layer, the input feature map and the offset are used together for the convolution operation. Specifically, for each convolution kernel position, the corresponding offset is used to adjust the sampling position so that the convolution kernel can flexibly select sampling points on the input feature map. This step handles non-integer sampling coordinates through bilinear interpolation to ensure the continuity and accuracy of the convolution calculation.

[0089] Furthermore, if Figure 8 As shown, this embodiment uses existing data to train the constructed model, and deploys the trained model to the target hardware. Exemplarily including the steps: S10 model initialization; S20 model training; S30 determines whether the model training has reached the training round or accuracy, if not, returns to S20; if reached, enters S40 to verify the model through the test set, if the verification passes, returns to S30; if the verification passes, enters S50 training is completed and fixed weights are output. In this embodiment, the preprocessed data is first input into the improved convolutional neural network, and then the improved convolutional neural network is coupled with the jamming detection algorithm, and the variable weights are trained using the gradient descent method. The gradient descent method involves forward propagation and back propagation. Specifically, the forward propagation calculation formula of the lth layer in the deep learning model is:

[0090] a l =σ(w l a l-1 +b l );

[0091] where a l with a l-1 are the outputs of the lth layer and the l-1th layer respectively, w l is the weight of the lth layer, b l is the offset of the lth layer.

[0092] After the forward propagation is completed, the total error is obtained, and then the back propagation algorithm is used to update the weights in the deep learning model. The specific calculation formula is:

[0093]

[0094] where w ′ l is the updated weight, n is the learning rate, is the partial derivative of the layer weight with respect to the error.

[0095] When the iteration reaches the target accuracy or the maximum number of iterations, the training is stopped and the test set data is applied to the model. If the test set prediction results reach the target accuracy, the training weights are fixed. Otherwise, the training set is continued for iterative training to optimize the model parameters. After the training is completed, a model with the same structure is created on the target hardware. After importing the fixed weights, real-time application somatosensory jam monitoring can be performed.

[0096] In summary, this embodiment first performs preprocessing, such as outlier processing, on the characteristics of huge message information data volume and the presence of abnormal data. Then, in view of its high dimensional features, a convolutional neural network is used as the baseline network for dimensionality reduction. Secondly, in view of the unique characteristics of the message data in the communication field, the message data is reorganized into sequence data so that the subsequent convolutional network can perform feature extraction and dimensionality reduction. Considering that the main function of the convolutional network is dimensionality reduction, the convolution kernel width is set to 1, and the fixed-length convolution kernel is optimized to a deformable convolution kernel to reduce information loss and optimize the dimensionality reduction effect. Finally, the data after dimensionality reduction is used as the input of the subsequent jamming detection algorithm to realize real-time monitoring of jamming events of various applications.

[0097] Fig. 9 is a block diagram of a message data jam detection system of the present exemplary embodiment. Fig. 9 As shown, an exemplary embodiment of the present disclosure provides a message data jam detection system, including:

[0098] A preprocessing unit 10, used for preprocessing message data;

[0099] A construction unit 20 is used to construct a feature engineering dimensionality reduction model using a convolutional neural network as a baseline model;

[0100] The feature engineering dimension reduction model 30 is used to reduce the dimension of the preprocessed message data and output the message feature vector after the dimension reduction;

[0101] The judgment unit 40 is used to input the message feature vector into an anomaly detection algorithm to perform a jamming judgment.

[0102] Fig.10 Schematic diagram of the structure of a device of this exemplary embodiment. Fig.10 As shown, corresponding to the message data jam detection method provided above, the present disclosure also provides a device. Since the embodiment of the device is similar to the above method embodiment, the description is relatively simple. Please refer to the description of the above method embodiment for relevant details. The device described below is only schematic. The device may include: a processor (processor) 1, a memory (memory) 2 and a communication bus (i.e., the above device bus) and a search engine, wherein the processor 1 and the memory 2 communicate with each other through the communication bus and communicate with the outside through the communication interface. The processor 1 can call the logic instructions in the memory 2 to execute the message data jam detection method.

[0103] In addition, the logic instructions in the above-mentioned memory 2 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present disclosure. The aforementioned storage medium includes: a storage chip, a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, and other media that can store program codes.

[0104] On the other hand, an embodiment of the present disclosure further provides a processor-readable storage medium, on which a computer program 3 is stored. When the computer program 3 is executed by the processor 1, the method for detecting message data jamming provided in the above embodiments is implemented.

[0105] The processor-readable storage medium can be any available medium or data storage device that can be accessed by the processor 1, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NANDFLASH), solid-state drive (SSD)), etc.

[0106] The above are only preferred implementations of the present disclosure, and the protection scope of the present disclosure is not limited to the above embodiments. All technical solutions under the concept of the present disclosure belong to the protection scope of the present disclosure. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present disclosure should be regarded as the protection scope of the present disclosure.

Claims

1. A method for detecting message data jamming, characterized in that: include: Preprocess the message data; Using convolutional neural network as the baseline model, we built a feature engineering dimensionality reduction model. The pre-processed message data is reduced in dimension through the feature engineering dimension reduction model, and the reduced-dimensional message feature vector is output; The message feature vector is input into the anomaly detection algorithm for jamming judgment.

2. The method for detecting message data jamming according to claim 1, characterized in that: Preprocessing of message data includes: Detect and process missing values, outliers, and duplicate values ​​in message data; The detected and processed message data is normalized to map the numerical ranges of different features to the same scale.

3. The method for detecting message data jamming according to claim 1, characterized in that: The preprocessing of the message data includes: Based on the preprocessed message data, sequence data is created and the head-to-tail coupling relationship of the data is captured.

4. The method for detecting message data jamming according to claim 3, characterized in that: Based on the preprocessed message data, creating sequence data and capturing the head-to-tail coupling relationship of the data includes: Define a sliding window of set size and step length on the message data; Generate subsequences by sliding the sliding window over the message data; The subsequence is used as input feature and the subsequent value of the subsequence is used as target value and input into the convolutional neural network.

5. The method for detecting message data jamming according to claim 1, characterized in that: Using the convolutional neural network as the baseline model, constructing a feature engineering dimensionality reduction model includes: Adjust the convolution kernel width of the convolutional neural network; Adding deformable convolutional layers to convolutional neural networks.

6. The method for detecting message data jamming according to claim 5, characterized in that: Adjusting the convolution kernel width of a convolutional neural network includes: The convolution kernel width is adjusted to 1 so that the convolutional neural network performs convolution operation on a single message data each time.

7. The method for detecting message data jamming according to claim 5, characterized in that: Adding a deformable convolutional layer to a convolutional neural network involves: Pass the preprocessed message data to the offset to generate the convolution kernel; Generating a convolution kernel through the offset and performing a convolution operation to generate a set of offsets; The offset and message data feature map are input into the added deformable convolution layer for convolution operation.

8. A message data jam detection system, characterized in that: include: A preprocessing unit, used for preprocessing message data; A construction unit for building a feature engineering dimensionality reduction model using a convolutional neural network as a baseline model; The feature engineering dimensionality reduction model is used to reduce the dimension of the preprocessed message data and output the message feature vector after dimensionality reduction; A judgment unit is used to input the message feature vector into an anomaly detection algorithm for jamming judgment.

9. A device, characterized in that: include: Processor and memory; The memory is used to store a computer program, and the processor calls the computer program stored in the memory to execute the message data jamming detection method described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is enabled to execute the message data jamming detection method according to any one of claims 1 to 7.

Images