A communication information segmentation analysis method based on big data and artificial intelligence

Abstract

The present application discloses a communication information segmentation and parsing method based on big data and artificial intelligence, which belongs to the field of communication information technology. The method includes: receiving and processing communication data in real time; parsing the communication data; applying corresponding segmentation strategies according to the different formats and structures of the communication data, and parsing the segmented data into structured headers and content; using adaptive enhanced variational autoencoders to detect anomalies on the preprocessed and parsed data to identify potential anomaly data; matching the identified anomaly data with a knowledge graph based on big data to determine the anomaly type; for known anomalies, reporting the anomaly type and its contextual information; for unknown anomalies, generating an alarm and submitting it to experts for analysis and confirmation; if it is confirmed to be a new anomaly pattern, adding it to the knowledge graph and dynamically updating the anomaly detection model.

Description

Technical Field

[0001] The present application relates to the field of communication information technology, and more specifically, to a communication information segmentation parsing method based on big data and artificial intelligence. Background Art

[0002] In today's information age, the transmission and processing of communication data are becoming increasingly important. With the development of big data technology and advancements in artificial intelligence, the management of communication information is facing new challenges. Communication refers to the exchange and transmission of information between people or between people and nature through some behavior or medium. Broadly speaking, it involves two or more parties requiring information, using any method and medium to accurately and securely transmit information from the sender to the receiver, without violating their respective wills.

[0003] Big data, or massive amounts of information, refers to information so large that it cannot be captured, managed, and processed within a reasonable timeframe using traditional software tools to support business decision-making. Artificial intelligence (AI) is an emerging scientific discipline that studies and develops theories, methods, techniques, and application systems that simulate, extend, and expand human intelligence. Combining big data and AI technologies can significantly improve the efficiency and intelligence of communication data processing.

[0004] Existing technologies have made some progress in the segmented parsing of communication information. For example, existing technologies incorporate a FIFO (first-in-first-out) algorithm for data caching and employ synchronous mirroring technology for synchronous data backup to ensure data integrity and security. Furthermore, by employing a multi-threaded pool and a BFS (breadth-first search) algorithm for recursive traversal and parsing of data, concurrent processing is achieved, significantly improving parsing efficiency and processing capabilities. Other technologies utilize methods such as z-score normalization, KMP (Knuth-Morris-Pratt) string matching, and time-delayed neural networks, combined with big data analysis, to detect and address anomalies in communication data. However, these existing technologies still have some shortcomings when dealing with the complex communication data found in modern big data environments.

[0005] Specifically, existing technologies cannot fully adapt to the real-time demands of modern big data environments and struggle to effectively parse communication data of varying formats and structures. Furthermore, existing technologies also have shortcomings in anomaly detection, struggling to accurately detect unknown anomaly types and unable to update anomaly types and processing rules in real time. These issues have limited the further development and application of communication information processing technology.

[0006] In summary, how to achieve efficient and accurate segmented analysis of communication information with the support of big data and artificial intelligence technologies, and to cope with the challenges in real-time data processing and anomaly detection, has become a technical problem that needs to be solved urgently. Summary of the Invention

[0007] In order to overcome a series of defects in the existing technology, the purpose of this application is to provide a communication information segmentation parsing method based on big data and artificial intelligence to address the above problems, including the following steps.

[0008] Step S1: receiving and processing communication data in real time.

[0009] Step S2: parse the communication data.

[0010] Step S3: applying corresponding segmentation strategies according to different formats and structures of the communication data, and parsing the segmented data into structured headers and contents.

[0011] In step S4, an adaptive enhanced variational autoencoder is used to perform anomaly detection on the preprocessed and parsed data to identify potential abnormal data.

[0012] Step S5: Match the identified abnormal data with the knowledge graph based on big data to determine the abnormality type.

[0013] Step S6: For known anomalies, report the anomaly type and its context information; for unknown anomalies, generate an alarm and submit it to experts for analysis and confirmation.

[0014] Step S7: If it is confirmed to be a new abnormal pattern, it will be added to the knowledge graph and the anomaly detection model will be dynamically updated; if it is confirmed to be abnormal data, a shielding operation will be performed; if it is confirmed to be normal data, a receiving operation will be performed to allow the data to continue to flow and be processed.

[0015] Furthermore, step S1 includes the following steps.

[0016] Step S101: receiving structured or unstructured data in real time through a communication device.

[0017] Step S102: Select a distributed stream processing framework to provide real-time data stream processing capabilities with high throughput and low latency.

[0018] Step S103: data preprocessing, including removing noise and processing missing values, and converting the data into a unified format to ensure data consistency.

[0019] Step S104: compress the pre-processed data using the LZ4 algorithm to effectively reduce the amount of data transmission.

[0020] Step S105: Use the distributed cache system to efficiently store and read data.

[0021] Step S106: storing the pre-processed and compressed data in a distributed storage system for subsequent processing.

[0022] Furthermore, in step S103, the Z-score normalized data is converted into a unified format, and the specific formula is: z=(X-μ) / σ, where X is the data value, μ is the mean of the data, σ is the standard deviation, and z is the corresponding normalized score.

[0023] In step S104, the compression and decompression processes are expressed as: C(t)=LZ4_Compress(D(t)), D(t)=LZ4_Decompress(C(t)), where D(t) is the amount of preprocessed data D(t), C(t) represents the compressed data, and D(t) is the decompressed data; LZ4_Compress(·) is a function used to compress input data into a smaller format; LZ4_Decompress(·) is used to decompress data compressed by LZ4_Compress back to the original format.

[0024] In step S106, data storage is represented as: Storage_Write(key, C(t)), and indexing and query optimization techniques are used to quickly retrieve data. The retrieval process is represented as: Storage_Read(key)→Retrieved_Data; Storage_Write(key,·) means storing data in a distributed storage system, using key as an identifier; key is a unique identifier of data and is used to find data in the storage system; Storage_Read(key) means reading data from the storage system according to key; Retrieved_Data represents the data read from the storage system.

[0025] Furthermore, step S2 includes the following steps.

[0026] Step S201 , using a multi-level parsing thread pool architecture to process different types of tasks, defining thread pool levels and configuring thread pool parameters.

[0027] Step S202: According to the specific requirements of each parsing task, the priority, name and other attributes of the thread are configured to meet the task requirements.

[0028] Step S203: Select a suitable file converter based on the data format and set parameters, including encoding format, separator, and date format.

[0029] Step S204: Implement the task scheduling strategy to reasonably allocate tasks and ensure that the priorities and execution order of the tasks meet the requirements.

[0030] Step S205 , reading data from the distributed storage system using streaming reading to ensure data consistency and integrity.

[0031] Furthermore, step S3 implements different segmentation strategies for communication data of different formats and structures, and parses them into structured data, which specifically includes the following steps.

[0032] The processing of structured data includes: using conventional parsing tools to parse data and extract fields and content; segmenting based on the row and column relationships of the data table and the logical association of the fields. For a table T with a primary key PK and a foreign key FK, the segmentation logic is expressed by the following pseudocode: Segment1={t∈T:ForeignKey(t)∈KeySet1}, Segment2={t∈T:ForeignKey(t)∈KeySet2}, where KeySet1 and KeySet2 are primary key sets defined based on logical conditions, Segment1 and Segment2 are data sets assigned to KeySet1 and KeySet2 based on the values of ForeignKey, t is a data item in T, and ForeignKey(t) is the foreign key value extracted from the data item t for data association and segmentation; considering the hierarchical structure and referential integrity factors of the data, the data is divided into multiple paragraphs or sub-tables.

[0033] Unstructured data processing involves cleaning and formatting the data to remove noise; inputting the processed text into a pre-trained T5 model. The input data format for the T5 model is: X = Preprocess(D), where X represents the result of preprocessing data D, D is the original unstructured data, and Preprocess(·) is a data cleaning and normalization function. The T5 model uses a self-attention mechanism to understand the contextual information in the data and generate a segmentation strategy. The generated segmentation strategy is then applied to divide the text into meaningful paragraphs or sub-sections and convert them into structured data.

[0034] The processing of semi-structured data includes: parsing the existing structured part of the data and extracting clear fields and content, where the formula for extracting the structured part is expressed as: Fields={Extract(D,Path):Path∈Schema}, where Extract(D,Path) is a function used to extract data, Schema is a set of field paths defined, Fields represents a set of fields extracted from data D, and Path is a path or key used to locate or access a specific part of data D; for free text or irregular content, natural language processing tools are used to generate dynamic segmentation strategies; the parsed structured part is merged with the unstructured part, the mapping relationship is aligned, and a complete structured data header and content are generated, where the segmentation strategy S is expressed as: S={(s i ,e i )}, where s i and e i Respectively represent the start and end position of each segment, the segmented data D i Expressed as: D i =D[s i :e i ], the formula for combining the parsed structured part and the unstructured part is: D merged =D structured ∪D unstructured , where D structured Represents structured data, that is, data that has been organized according to a specific structure or pattern; D unstructured Represents unstructured data; D merged It's D structured and D unstructured The combined result of the two represents comprehensive data that combines structured data and unstructured data.

[0035] Furthermore, step S4 includes the following steps.

[0036] Step S401: Standardize all structured data to reduce dimensionality effects.

[0037] Step S402: Reduce the computational complexity by dimensionality reduction while retaining the main feature information of the data.

[0038] Step S403: reconstruct the test set data using the trained adversarial variational autoencoder, calculate the reconstruction error of each sample, and set the anomaly detection threshold τ based on the reconstruction error distribution.

[0039] Step S404: perform an abnormality determination on each sample. The determination rule is: if the reconstruction error of the sample is greater than τ, it is determined to be abnormal.

[0040] Furthermore, in S403 , the model architecture of the adversarial variational autoencoder includes an encoder network and a decoder network, and its structure and training process are as follows.

[0041] For the encoder network, the input feature X is mapped to the distribution q(z|X) in the latent space. The output of the encoder is the mean and variance of the latent variable z, q(z|X)=N(μ(X),Σ(X)), q(z|X) represents the conditional probability distribution of the latent variable z given the input data X; N(μ(X),Σ(X)) represents a normal distribution whose mean and covariance are defined by μ(X) and Σ(X); μ(X) is the mean function, which represents the mean of the latent variable z given the input X; Σ(X) is the covariance matrix function, which represents the covariance matrix of the latent variable z given the input X; N represents the normal distribution.

[0042] For the decoder network, the latent variable z is remapped back to the data space to generate an output X^X^=fdecoder(z) with the same scale as the input data X, where fdecoder is the neural network function of the decoder.

[0043] The loss function of the adversarial variational autoencoder includes reconstruction loss and KL divergence loss, where the reconstruction loss is used to measure the difference between the original input X and the reconstructed output X^, using the mean square error (MSE): L reconstruction =||XX^|| 2 , where L reconstruction Represents the reconstruction loss, which is used to measure the difference between the input data and the reconstructed data; the KL divergence loss is used to measure the difference between the output distribution of the encoder and the prior distribution: L KL =D KL (q(z|X)||p(z)), where L KL represents the KL divergence loss, which measures the difference between the potential distribution and the prior distribution; D KL represents the KL divergence, which represents the distance between two probability distributions; p(z) represents the prior distribution, which defines the underlying probability distribution of the latent variable z; the total loss function L AVAE Expressed as: L AVAE =L reconstruction +βL KL , where β is the weight coefficient used to balance the reconstruction loss and KL divergence.

[0044] An adaptive mechanism is introduced to dynamically adjust the structure of the encoder network and decoder network according to the complexity of the data, and an attention mechanism is used to increase the focus on important features in the data.

[0045] Use the stochastic gradient descent optimization algorithm to minimize the total loss function of the adversarial variational autoencoder and gradually adjust the model parameters: θ^=θ-αm t / (v t 1 / 2 +ε), where θ is the model parameter, α is the learning rate, and m t represents the value of the gradient momentum at time step t; v t represents the variance of the gradient at time step t; θ^ represents the updated model parameters; ε is a small constant used to avoid division by zero errors.

[0046] Furthermore, step S5 includes the following steps.

[0047] Based on big data, we construct an abnormal knowledge graph G=(V,E), where V represents the node set of abnormal patterns, E represents the relationship set between abnormal patterns, and each node v i ∈V represents an abnormal pattern, which includes the following information: abnormal characteristics, types, handling measures and other related metadata.

[0048] For each detected abnormal sample, its feature vector X i Mapped to the corresponding node in the knowledge graph.

[0049] The similarity measurement method is used to calculate the similarity between abnormal samples and each abnormal pattern. The formula is expressed as: Sim(X i ,v j )=X i v j / ||X i ||||v j ||,v j Is the feature vector of the node in the knowledge graph; Sim(X i ,v j ) represents vector X i and vector v j The similarity between them.

[0050] According to the calculated similarity score, the similarity threshold strategy τ is used for matching, which is expressed as: ifSim(X i ,v j )>τ,then X i ~v j , X i ~v j Represents X i and v j There is some kind of relationship or match between them.

[0051] If the match is successful, call the exception pattern v that matched jThe exception is handled by the associated predefined exception handling rules; otherwise, it is marked as an unknown exception.

[0052] The anomaly knowledge graph is updated regularly based on newly discovered anomaly patterns and processing results.

[0053] Furthermore, in step S7, for unknown anomalies that do not match existing anomaly patterns in the knowledge graph, a further analysis and optimization process is performed, and the specific steps include:

[0054] This includes dynamically updating anomaly detection rules and models, and using integrated scheduling algorithms to automatically schedule expert analysis, model training, and knowledge graph update tasks.

[0055] When a new abnormal pattern is detected and confirmed by experts, it is added to the knowledge graph G, which is expressed as: G'=G∪{X new}, where X new represents the new abnormal pattern, and G' represents the updated abnormal knowledge graph.

[0056] Use the new data and updated abnormal patterns to retrain the model and re-optimize the parameters of the adaptive enhanced variational autoencoder model to improve the detection ability of new abnormal patterns. The specific formula is: γ'=γ-η▽γL AVAE (X new ), where η represents the learning rate, γ represents the model parameters, and γ' represents the updated adaptive enhanced variational autoencoder model parameters; ▽γ represents the loss function L AVAE For the gradient of γ; L AVAE represents the loss function of the adaptive boosted variational autoencoder model.

[0057] A communication information segmentation parsing system based on big data and artificial intelligence is used to implement the above-mentioned communication information segmentation parsing method based on big data and artificial intelligence, and includes the following modules.

[0058] The communication data receiving module is responsible for receiving the communication data from the communication equipment in real time and performing preliminary format verification and data integrity check.

[0059] The distributed stream processing module adopts a distributed stream processing framework to support real-time processing and preprocessing of large-scale data streams; at the same time, it uses the LZ4 compression algorithm to reduce data transmission volume and uses a distributed cache system to efficiently store and read data.

[0060] The task scheduling module is responsible for creating a thread pool, configuring parsing threads and file converters, and starting the thread pool to read data from the distributed storage system to achieve efficient data parsing and processing.

[0061] The segmentation strategy generation module generates corresponding segmentation strategies according to communication data of different formats and structures, and parses the data into structured headers and contents.

[0062] The anomaly detection and processing module uses an adaptive enhanced variational autoencoder to perform anomaly detection on preprocessed and parsed data, identify potential abnormal data, and infer and predict it in a big data knowledge graph to determine the type of anomaly.

[0063] The knowledge graph update module reports the anomaly type and its contextual information for detected abnormal data. If it is a known anomaly, corresponding processing measures are taken. For unknown anomalies, an alarm is generated and submitted to experts for analysis and confirmation. If it is confirmed to be a new abnormal pattern, it is added to the knowledge graph, thereby dynamically updating the system's anomaly detection model.

[0064] The data processing control module takes corresponding processing measures based on the abnormality detection results, including shielding abnormal data and performing receiving operations on data determined to be normal, allowing the data to continue to flow and be processed.

[0065] Compared with the prior art, this application has the following beneficial effects.

[0066] This application achieves efficient segmented parsing and anomaly detection of communication information through a series of steps: first, using a distributed stream processing framework to receive and preprocess data in real time, then using a multi-level parsing thread pool and an adaptive enhanced variational autoencoder to detect anomalies in the data, and finally matching the anomaly information with the knowledge graph, and dynamically updating the anomaly detection model to improve system performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 A flowchart of a communication information segmentation parsing method based on big data and artificial intelligence is provided in an embodiment of the present application.

[0068] Figure 2 A structural diagram of a communication information segmentation parsing system based on big data and artificial intelligence provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] To make the objectives, technical solutions, and advantages of the present invention more apparent, the technical solutions in the embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Throughout the drawings, identical or similar reference numerals represent identical or similar elements or elements having identical or similar functions. The described embodiments are only some, not all, of the embodiments of the present invention.

[0070] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative work shall fall within the scope of protection of the present invention.

[0071] The embodiments and directional terms described below with reference to the accompanying drawings are exemplary and intended to be used to explain the present invention, but should not be construed as limiting the present invention.

[0072] like Figure 1 As shown, a communication information segmentation parsing method based on big data and artificial intelligence includes the following steps.

[0073] Step S1, real-time data reception and processing stage: receive communication data in real time through communication equipment, use distributed stream processing framework to perform real-time data processing and preprocessing, use LZ4 algorithm to compress data to reduce transmission volume, and use distributed cache system to efficiently store and read data.

[0074] Step S2, data parsing and processing phase: creating and configuring a thread pool; setting parsing threads and a file converter; starting the thread pool and recursively traversing the data storage to achieve efficient parallel data parsing.

[0075] Step S3, data segmentation and structured parsing stage: applying corresponding segmentation strategies according to different formats and structures of communication data; parsing the segmented data into structured headers and content.

[0076] Step S4, anomaly detection stage: Adaptive enhanced variational autoencoder (AVAE) is used to perform anomaly detection on the preprocessed and parsed data to identify potential abnormal data.

[0077] Step S5: Match the identified abnormal data with the knowledge graph based on big data to determine the abnormality type.

[0078] Step S6: For known anomalies, report the anomaly type and its context information; for unknown anomalies, generate an alarm and submit it to experts for analysis and confirmation.

[0079] Step S7: If it is confirmed to be a new abnormal pattern, it will be added to the knowledge graph and the anomaly detection model will be dynamically updated; if it is confirmed to be abnormal data, a shielding operation will be performed; if it is confirmed to be normal data, a receiving operation will be performed to allow the data to continue to flow and be processed.

[0080] Furthermore, step S1 includes the following steps.

[0081] Step S101: receiving structured (eg, JSON, XML) or unstructured (eg, raw byte stream) data in real time via a communication device.

[0082] Step S102: Select a suitable distributed stream processing framework, such as Apache Flink, to provide real-time data stream processing capabilities with high throughput and low latency.

[0083] Step S103: data preprocessing, including removing noise and processing missing values, and converting the data into a unified format to ensure data consistency.

[0084] Step S104: compress the pre-processed data using the LZ4 algorithm to effectively reduce the amount of data transmission.

[0085] Step S105: Use a distributed cache system (such as Cassandra) to efficiently store and read data.

[0086] Step S106: storing the pre-processed and compressed data in a distributed storage system (such as HDFS) for subsequent processing.

[0087] In step S103, the Z-score is used to standardize the data and convert it into a unified format. The specific calculation formula is: z=(X-μ) / σ, where X is the data value, μ is the mean of the data, σ is the standard deviation, and z is the corresponding standardized score.

[0088] In step S104, LZ4 is a high-speed compression algorithm suitable for real-time applications. The compression and decompression process can be expressed as: C(t) = LZ4_Compress(D'(t)), D"(t) = LZ4_Decompress(C(t)), where D'(t) is the amount of preprocessed data D(t), C(t) represents the compressed data, and D"(t) is the decompressed data. LZ4_Compress(·) is a function used to compress input data into a smaller format; LZ4_Decompress(·) is used to decompress data compressed by LZ4_Compress back to its original format.

[0089] In step S106, data storage is represented by: Storage_Write(key, C(t)), using indexes and query optimization techniques (such as secondary indexes and hash tables) to quickly retrieve data. The retrieval process can be represented as: Storage_Read(key) → Retrieved_Data; Storage_Write(key, ·) means storing data in a distributed storage system, using key as an identifier; key is a unique identifier for data and is used to find data in the storage system; Storage_Read(key) means reading data from the storage system based on key; Retrieved_Data represents the data read from the storage system.

[0090] Furthermore, step S2 includes the following steps.

[0091] Step S201 , using a multi-level parsing thread pool architecture to process different types of tasks, defining thread pool levels and configuring thread pool parameters.

[0092] Step S202: According to the specific requirements of each parsing task, the priority, name and other attributes of the thread are configured to meet the task requirements.

[0093] Step S203: Select a suitable file converter based on the data format (such as JSON, XML), and set necessary parameters, including encoding format, separator, and date format.

[0094] Step S204: Implement the task scheduling strategy (priority queue) to reasonably allocate tasks and ensure that the priority and execution order of the tasks meet the requirements.

[0095] Step S205 , reading data from the distributed storage system using streaming reading to ensure data consistency and integrity.

[0096] In step S201, the multi-level parsing thread pool architecture includes: a core thread pool, which handles basic parsing tasks; an extended thread pool, which handles tasks requiring additional computing resources; and an I / O thread pool, which specifically handles file read and write operations. Thread pool parameters include: the number of core threads, which sets the number of threads in the core thread pool to handle routine tasks; the maximum number of threads, which sets the maximum number of threads in the extended thread pool to handle high-load tasks; the queue capacity, which configures the capacity of the task queue to manage task queuing and scheduling; and the thread pool scheduling policy, which selects an appropriate scheduling policy (e.g., fixed thread pool or cached thread pool).

[0097] In step 202, parsing tasks include data conversion, formatting, and cleaning.

[0098] Furthermore, step S3 implements different segmentation strategies for communication data of different formats and structures, and parses them into structured data, which specifically includes the following steps.

[0099] The processing of structured data includes: using conventional parsing tools (such as Python's json, pandas, xml.etree.ElementTree, etc.) to parse data and extract fields (table headers) and content; segmenting based on the row and column relationships of the data table and the logical associations of the fields (such as primary keys and foreign keys). For a table T with a primary key PK and a foreign key FK, the segmentation logic is expressed in the following pseudocode: Segment1={t∈T:ForeignKey(t)∈KeySet1}, Segment2={t∈T:ForeignKey(t)∈KeySet2}, where KeySet1 and KeySet2 are primary key sets defined based on logical conditions; Segment1 and Segment2 are data sets assigned to KeySet1 and KeySet2 based on the values of ForeignKey; t is a data item in T; ForeignKey(t) is the foreign key value extracted from data item t, used for data association and segmentation; considering factors such as the hierarchical structure of the data (such as parent-child relationships) and referential integrity, the data is divided into multiple segments or sub-tables.

[0100] Unstructured data processing involves cleaning and formatting the data to remove noise (such as special characters and HTML tags). The processed text is fed into a pre-trained T5 model. The input format for the T5 model is: X = Preprocess(D), where X represents the preprocessed data D; D is the raw unstructured data, and Preprocess(·) is a data cleaning and normalization function. The T5 model uses a self-attention mechanism to understand the contextual information in the data and generate a segmentation strategy. The segmentation strategy S contains the start and end positions and key attributes of each paragraph. The generated segmentation strategy is then applied to divide the text into meaningful paragraphs or subsections and convert them into structured data (such as headers and content).

[0101] The processing of semi-structured data includes: parsing the existing structured parts in the data (such as JSON key-value pairs and XML tags) and extracting clear fields and content. The formula for extracting the structured part is expressed as: Fields={Extract(D,Path):Path∈Schema}, where Extract(D,Path) is a function used to extract data, Schema is a collection of field paths; Fields represents a set of fields extracted from data D; Path is a path or key used to locate or access a specific part of data D; for free text or irregular content, natural language processing tools such as the T5 model are used to generate dynamic segmentation strategies; the parsed structured part is merged with the unstructured part, the mapping relationship is aligned, and a complete structured data header and content are generated. The segmentation strategy S is expressed as: S={(s i ,e i )}, where s i and e i Respectively represent the start and end position of each segment, the segmented data D i Expressed as: D i =D[s i :e i ], the formula for combining the parsed structured part and the unstructured part is: D merged =D structured ∪D unstructured , where D structured Represents structured data, that is, data that has been organized according to a specific structure or pattern; D unstructured Represents unstructured data; D merged It's D structured and D unstructured The combined result represents a comprehensive data set that combines structured and unstructured data. Throughout the processing process, if parsing failures or exceptions are encountered, the unparseable portion is skipped or a warning message is generated. For large datasets, batch processing and parallel computing strategies are employed to improve parsing and conversion efficiency.

[0102] Furthermore, step S4 includes the following steps.

[0103] In step S401 , all structured data (including originally structured data, as well as semi-structured and unstructured data processed by the T5 model) are standardized to reduce dimensionality effects.

[0104] Step S402 : Dimensionality reduction processing (such as PCA or t-SNE) is used to reduce computational complexity while retaining the main feature information of the data.

[0105] In step S403, the trained adversarial variational autoencoder (AVAE) is used to reconstruct the test set data, and the reconstruction error of each sample is calculated. At the same time, the anomaly detection threshold τ is set according to the distribution of the reconstruction error.

[0106] Step S404: perform an abnormality determination on each sample. The determination rule is: if the reconstruction error of the sample is greater than τ, it is determined to be abnormal.

[0107] Among them, in S403, the model architecture of the Adversarial Variational Autoencoder (AVAE) includes an encoder network and a decoder network. The detailed structure and training process are as follows.

[0108] For the encoder network, the input feature X is mapped to the distribution q(z|X) in the latent space. The output of the encoder is the mean and variance of the latent variable z, q(z|X)=N(μ(X),Σ(X)), q(z|X) represents the conditional probability distribution of the latent variable z given the input data X; N(μ(X),Σ(X)) represents a normal distribution whose mean and covariance are defined by μ(X) and Σ(X); μ(X) is the mean function, which represents the mean of the latent variable z given the input X; Σ(X) is the covariance matrix function, which represents the covariance matrix of the latent variable z given the input X; N represents the normal distribution.

[0109] For the decoder network, the latent variable z is remapped back to the data space to generate an output X^ with the same scale as the input data X: X^=fdecoder(z), where fdecoder is the neural network function of the decoder.

[0110] The loss function of the Adversarial Variational Autoencoder (AVAE) includes reconstruction loss and KL divergence loss, where the reconstruction loss is used to measure the difference between the original input X and the reconstructed output X^, using the mean square error (MSE): L reconstruction =||XX^|| 2 , where L reconstruction Represents the reconstruction loss, which is used to measure the difference between the input data and the reconstructed data; the KL divergence loss is used to measure the difference between the output distribution of the encoder and the prior distribution: L KL =D KL (q(z|X)||p(z)), where L KL represents the KL divergence loss, which measures the difference between the potential distribution and the prior distribution; D KL represents the KL divergence, which represents the distance between two probability distributions; p(z) represents the prior distribution, which defines the underlying probability distribution of the latent variable z; the total loss function L AVAE Expressed as: L AVAE =L reconstruction +βL KL, where β is a weight coefficient used to balance the reconstruction loss and KL divergence. The Adversarial Variational Autoencoder (AVAE) introduces an adaptive mechanism that dynamically adjusts the structure of the encoder network and decoder network according to the complexity of the data, and uses an attention mechanism to increase the focus on important features in the data. Using the stochastic gradient descent (SGD) optimization algorithm, the total loss function of the Adversarial Variational Autoencoder (AVAE) is minimized, and the model parameters are gradually adjusted: θ^=θ-αm t / (v t 1 / 2 +ε), where θ is the model parameter, α is the learning rate, and m t represents the value of the gradient momentum at time step t; v t represents the variance of the gradient at time step t; θ^ represents the updated model parameters; ε is a small constant used to avoid division by zero errors.

[0111] Furthermore, step S5 includes the following steps.

[0112] Based on big data, we construct an abnormal knowledge graph G=(V,E), where V represents the node set of abnormal patterns, E represents the relationship set between abnormal patterns, and each node v i ∈V represents an abnormal pattern, which includes the following information: abnormal characteristics, types, handling measures and other related metadata.

[0113] For each detected abnormal sample, its feature vector X i Map to the corresponding node in the knowledge graph through hash mapping or vectorization method.

[0114] The similarity between abnormal samples and each abnormal pattern is calculated using similarity measurement methods including cosine similarity, Euclidean distance or Mahalanobis distance. The formula is expressed as: Sim(X i ,v j )=X i v j / ||X i ||||v j ||,v j Is the feature vector of the node in the knowledge graph; Sim(X i ,v j ) represents vector X i and vector v j The similarity between them.

[0115] According to the calculated similarity score, the K-nearest neighbor (KNN) algorithm or the similarity threshold strategy τ is used for matching, which can be expressed as follows: if Sim(X i ,v j )>τ,then X i ~vj , X i ~v j Represents X i and v j There is some kind of relationship or match between them.

[0116] If the match is successful, the predefined exception handling rule associated with the matched exception pattern vj is called for processing; otherwise, it is marked as an unknown exception.

[0117] The anomaly knowledge graph is updated regularly based on newly discovered anomaly patterns and processing results.

[0118] Furthermore, in step S7, for unknown anomalies that do not match the existing anomaly patterns in the knowledge graph, further analysis and optimization processes need to be performed. The specific steps include: dynamically updating anomaly detection rules and models, and automatically scheduling expert analysis, model training, and knowledge graph update tasks using an integrated scheduling algorithm; when a new anomaly pattern is detected and confirmed by experts, it is added to the knowledge graph G, expressed as: G'=G∪{X new}, where X new Represents the new abnormal pattern, and G' represents the updated abnormal knowledge graph; the model is retrained using the new data and the updated abnormal pattern, and the parameters of the adaptive enhanced variational autoencoder (AVAE) model are re-optimized to improve the detection ability of the new abnormal pattern. This process is achieved by the gradient descent method, that is, γ'=γ-η▽γL AVAE (X new ), where η represents the learning rate, γ represents the model parameters, and γ' represents the updated adaptive enhanced variational autoencoder model parameters; ▽γ represents the loss function L AVAE For the gradient of γ; L AVAE represents the loss function of the adaptive boosted variational autoencoder model.

[0119] like Figure 2 As shown, a communication information segmentation parsing system based on big data and artificial intelligence includes the following modules.

[0120] Communication data receiving module: responsible for receiving communication data from communication equipment in real time and performing preliminary format verification and data integrity check.

[0121] Distributed stream processing module: adopts a distributed stream processing framework to support real-time processing and preprocessing of large-scale data streams; uses the LZ4 compression algorithm to reduce data transmission volume, and uses a distributed cache system to efficiently store and read data.

[0122] Task scheduling module: responsible for creating thread pools, configuring parsing threads and file converters, and starting thread pools to read data from distributed storage systems to achieve efficient data parsing and processing.

[0123] Segmentation strategy generation module: Generates corresponding segmentation strategies based on communication data of different formats and structures, and parses the data into structured headers and content.

[0124] Anomaly detection and processing module: Use adaptive enhanced variational autoencoder (AVAE) to perform anomaly detection on preprocessed and parsed data, identify potential abnormal data and perform reasoning and prediction based on big data knowledge graph to determine the type of anomaly.

[0125] Knowledge Graph Update Module: For detected anomalies, the module reports the anomaly type and context. If it is a known anomaly, appropriate action is taken. For unknown anomalies, an alert is generated and submitted to experts for analysis and confirmation. If it is confirmed to be a new anomaly pattern, it is added to the knowledge graph, dynamically updating the system's anomaly detection model.

[0126] Data processing control module: Takes corresponding processing measures based on the anomaly detection results, including shielding abnormal data and performing receiving operations on data determined to be normal, allowing the data to continue to flow and be processed.

[0127] Finally, it should be noted that the above embodiments are intended only to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art will appreciate that modifications may be made to the technical solutions described in the above embodiments, or that some of the technical features may be replaced with equivalents; such modifications or replacements do not deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A communication information segmentation analysis method based on big data and artificial intelligence, characterized in that: The following steps are involved: Step S1, receiving and processing communication data in real time; Step S2, performing data parsing on the communication data, including: using a multi-level parsing thread pool architecture to process different types of tasks, defining thread pool levels and configuring thread pool parameters; Step S3, applying corresponding segmentation strategies according to different formats and structures of the communication data, and parsing the segmented data into structured headers and contents; Step S4, using an adaptive enhanced variational autoencoder to perform anomaly detection on the processed and parsed data to identify potential abnormal data; Step S5: Match the identified abnormal data with the knowledge graph based on big data to determine the abnormality type; Step S6: For known anomalies, report the anomaly type and its context information; for unknown anomalies, generate an alarm and submit it to experts for analysis and confirmation; Step S7: If it is confirmed to be a new abnormal pattern, it is added to the knowledge graph and the anomaly detection model is dynamically updated; if it is confirmed to be abnormal data, a shielding operation is performed; if it is confirmed to be normal data, a receiving operation is performed to allow the data to continue to flow and be processed; Step S3 specifically includes the following steps: The processing of structured data includes: using conventional parsing tools to parse data and extract fields and content; segmenting based on the row-column relationship of the data table and the logical association of the fields. For a table T with a primary key PK and a foreign key FK, the segmentation logic is expressed in the following pseudocode: Segment1={t∈T:ForeignKey(t)∈KeySet1}, Segment2={t∈T:ForeignKey(t)∈KeySet2}, where KeySet1 and KeySet2 are primary key sets defined based on logical conditions, Segment1 and Segment2 are data sets assigned to KeySet1 and KeySet2 based on the values of ForeignKey, t is a data item in T, and ForeignKey(t) is the foreign key value extracted from data item t for data association and segmentation; considering the hierarchical structure and referential integrity of the data, the data is divided into multiple segments or subtables; Unstructured data processing involves: cleaning and formatting the data to remove noise; inputting the processed text into a pre-trained T5 model; the T5 model uses a self-attention mechanism to understand the contextual information in the data and generate a segmentation strategy; applying the generated segmentation strategy to divide the text into meaningful paragraphs or sub-sections and converting it into structured data; The processing of semi-structured data includes: parsing the existing structured part of the data and extracting clear fields and content; using natural language processing tools to generate dynamic segmentation strategies for free text or irregular content; merging the parsed structured part with the unstructured part, aligning the mapping relationship, and generating a complete structured data header and content, where the segmentation strategy S is expressed as: S={(s i ,e i )}, where s i and e i Respectively represent the start and end position of each segment, the segmented data D i Expressed as: D i =D[s i :e i ].

2. The communication information segmentation parsing method based on big data and artificial intelligence according to claim 1 is characterized in that: Step S1 includes the following steps: Step S101, receiving structured or unstructured data in real time through a communication device; Step S102: Select a distributed stream processing framework to provide high throughput and low latency real-time data stream processing capabilities; Step S103: data preprocessing, including removing noise and processing missing values, and converting the data into a unified format to ensure data consistency; Step S104, compressing the pre-processed data using the LZ4 algorithm to effectively reduce the amount of data transmission; Step S105, using a distributed cache system to efficiently store and read data; Step S106: storing the pre-processed and compressed data in a distributed storage system for subsequent processing.

3. The communication information segmentation parsing method based on big data and artificial intelligence according to claim 2 is characterized in that: In step S103, the Z-score is used to standardize the data and convert it into a unified format. The specific formula is: z = (X - μ) / σ, where X is the data value, μ is the mean of the data, σ is the standard deviation, and z is the corresponding standardized score; In step S104, the compression and decompression processes are expressed as: C(t)=LZ4_Compress(D(t)), D(t)=LZ4_Decompress(C(t)), where D(t) is the amount of preprocessed data D(t), C(t) represents the compressed data, and D(t) is the decompressed data; LZ4_Compress(·) is a function used to compress input data into a smaller format; LZ4_Decompress(·) is used to decompress the data compressed by LZ4_Compress back to the original format; In step S106, data storage is represented as: Storage_Write(key, C(t)), and indexing and query optimization techniques are used to quickly retrieve data. The retrieval process is represented as: Storage_Read(key)→Retrieved_Data; Storage_Write(key,·) means storing data in a distributed storage system, using key as an identifier; key is a unique identifier of data and is used to find data in the storage system; Storage_Read(key) means reading data from the storage system according to key; Retrieved_Data represents the data read from the storage system.

4. The communication information segmentation parsing method based on big data and artificial intelligence according to claim 1 is characterized in that: Step S5 includes the following steps: Based on big data, we construct an abnormal knowledge graph G=(V,E), where V represents the node set of abnormal patterns, E represents the relationship set between abnormal patterns, and each node v i ∈V represents an anomaly pattern, which includes the following information: anomaly characteristics, types, treatment measures and related metadata; For each detected abnormal sample, its feature vector X i Map to the corresponding node in the knowledge graph; The similarity measurement method is used to calculate the similarity between abnormal samples and each abnormal pattern. The formula is expressed as: Sim(X i ,v j )=X i v j / ||X i ||||v j ||,v j Is the feature vector of the node in the knowledge graph; Sim(X i ,v j ) represents vector X i and vector v j similarity between According to the calculated similarity score, the similarity threshold strategy τ is used for matching, which can be expressed as follows: if Sim(X i ,v j )>τ,then X i ~v j , X i ~v j Represents X i and v j There is a relationship or match between them; If the match is successful, call the exception pattern v that matched j The associated predefined exception handling rules are used for processing; otherwise, it is marked as an unknown exception; The anomaly knowledge graph is updated regularly based on newly discovered anomaly patterns and processing results.

5. The communication information segmentation parsing method based on big data and artificial intelligence according to claim 1 is characterized in that: In step S7, for unknown anomalies that do not match existing anomaly patterns in the knowledge graph, further analysis and optimization processes are performed. The specific steps include: Dynamically update anomaly detection rules and models, and use integrated scheduling algorithms to automatically schedule expert analysis, model training, and knowledge graph update tasks; When a new abnormal pattern is detected and confirmed by experts, it is added to the knowledge graph G, which is expressed as: G'=G∪{X new }, where X new represents the new abnormal pattern, and G' represents the updated abnormal knowledge graph; Use the new data and updated abnormal patterns to retrain the model and re-optimize the parameters of the adaptive enhanced variational autoencoder model to improve the detection ability of new abnormal patterns. The specific formula is: γ'=γ-η▽γL AVAE (X new ), where η represents the learning rate, γ represents the model parameters, and γ' represents the updated adaptive enhanced variational autoencoder model parameters; ▽γ represents the loss function L AVAE For the gradient of γ; L AVAE represents the loss function of the adaptive boosted variational autoencoder model.

6. A communication information segmentation parsing system based on big data and artificial intelligence, used to implement a communication information segmentation parsing method based on big data and artificial intelligence as described in any one of claims 1 to 5, characterized in that: Includes the following modules: The communication data receiving module is responsible for receiving the communication data from the communication equipment in real time and performing preliminary format verification and data integrity check; The distributed stream processing module uses a distributed stream processing framework to support real-time processing and pre-processing of large-scale data streams. It also uses the LZ4 compression algorithm to reduce data transmission volume and uses a distributed cache system to efficiently store and read data. The task scheduling module is responsible for creating a thread pool, configuring parsing threads and file converters, and starting the thread pool to read data from the distributed storage system to achieve efficient data parsing and processing; The segmentation strategy generation module generates corresponding segmentation strategies based on communication data of different formats and structures, and parses the data into structured headers and content; The anomaly detection and processing module uses an adaptive enhanced variational autoencoder to detect anomalies in preprocessed and parsed data, identify potential anomaly data, and perform reasoning and prediction based on the big data knowledge graph to determine the anomaly type. The knowledge graph update module reports the anomaly type and context for detected anomaly data. If it is a known anomaly, appropriate treatment measures are taken. For unknown anomalies, an alert is generated and submitted to experts for analysis and confirmation. If it is confirmed to be a new anomaly pattern, it is added to the knowledge graph, thereby dynamically updating the system's anomaly detection model. The data processing control module takes corresponding processing measures based on the abnormality detection results, including shielding abnormal data and performing receiving operations on data determined to be normal, allowing the data to continue to flow and be processed.

Images