Communication information segmentation analysis method based on big data and artificial intelligence

Abstract

The invention discloses a communication information segmentation analysis method based on big data and artificial intelligence, and belongs to the technical field of communication information. The method comprises the following steps: receiving and processing communication data in real time; performing data analysis on the communication data; corresponding segmentation strategies are applied according to different formats and structures of the communication data, and the segmented data are analyzed into structured headers and contents; carrying out anomaly detection on the preprocessed and analyzed data by adopting a self-adaptive enhanced variational auto-encoder, and identifying potential abnormal data; matching the identified abnormal data with a knowledge graph based on big data, and determining an abnormal type; for known exceptions, exception types and context information thereof are reported; for unknown abnormity, generating an alarm and submitting the alarm to an expert for analysis and confirmation; and if the abnormal mode is determined to be a new abnormal mode, adding the abnormal mode to the knowledge graph and dynamically updating the abnormal detection model.

Description

Technical Field

[0001] The present application relates to the field of communication information technology, and more specifically, to a communication information segmentation parsing method based on big data and artificial intelligence. Background Art

[0002] In today's information age, the transmission and processing of communication data has become increasingly important. With the development of big data technology and the advancement of artificial intelligence, the management of communication information is also facing new challenges. Communication refers to the exchange and transmission of information between people or between people and nature through certain behaviors or media. In a broad sense, it involves two or more parties who need information using any method and media to accurately and securely transmit information from the sender to the receiver without violating their respective wishes.

[0003] Big data, or massive data, refers to the amount of data involved that is so large that it cannot be acquired, managed, and processed by traditional software tools within a reasonable time to support business decision-making. Artificial intelligence is an emerging technical science that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. Combining big data and artificial intelligence technology can significantly improve the efficiency and intelligence level of communication data processing.

[0004] The existing technology has made some progress in the segmented parsing process of communication information. For example, the existing technology introduces the FIFO (first-in-first-out) algorithm for data caching and adopts the synchronous mirroring technology to achieve synchronous backup of data to ensure the integrity and security of the data. In addition, by setting up a multi-thread pool and a BFS (breadth-first search) algorithm, recursively traverse and parse the data, and achieve concurrent processing, thereby significantly improving the parsing efficiency and processing capabilities. There are also technologies that use methods such as z-score standardization, KMP (Knuth-Morris-Pratt) string matching, and delayed neural networks, combined with big data analysis to achieve anomaly detection and processing of communication data. However, these existing technologies still have some shortcomings when dealing with complex communication data in modern big data environments.

[0005] Specifically, existing technologies cannot fully adapt to the real-time needs in modern big data environments, and it is difficult to effectively parse communication data of different formats and structures. In addition, existing technologies also have deficiencies in anomaly detection, making it difficult to accurately detect unknown anomaly types, and unable to update anomaly types and processing rules in real time. These problems limit the further development and application of communication information processing technology.

[0006] To sum up, how to achieve efficient and accurate segmentation analysis of communication information with the support of big data and artificial intelligence technologies, and to cope with the challenges in real-time data processing and anomaly detection has become a technical problem that needs to be solved urgently. Summary of the invention

[0007] In order to overcome a series of defects existing in the prior art, the purpose of this application is to provide a communication information segmentation parsing method based on big data and artificial intelligence to address the above problems, including the following steps.

[0008] Step S1, receiving and processing communication data in real time.

[0009] Step S2, analyzing the communication data.

[0010] Step S3, applying corresponding segmentation strategies according to different formats and structures of the communication data, and parsing the segmented data into structured headers and contents.

[0011] Step S4, using an adaptive enhanced variational autoencoder to perform anomaly detection on the preprocessed and parsed data to identify potential abnormal data.

[0012] Step S5, matching the identified abnormal data with the knowledge graph based on big data to determine the abnormal type.

[0013] Step S6: For known anomalies, report the anomaly type and its context information; for unknown anomalies, generate an alarm and submit it to experts for analysis and confirmation.

[0014] Step S7: If it is confirmed as a new abnormal pattern, it is added to the knowledge graph and the anomaly detection model is dynamically updated; if it is confirmed as abnormal data, a shielding operation is performed; if it is confirmed as normal data, a receiving operation is performed to allow the data to continue to flow and be processed.

[0015] Furthermore, step S1 includes the following steps.

[0016] Step S101: receiving structured or unstructured data in real time through a communication device.

[0017] Step S102: Select a distributed stream processing framework to provide real-time data stream processing capabilities with high throughput and low latency.

[0018] Step S103, data preprocessing, including removing noise and processing missing values, and converting the data into a unified format to ensure data consistency.

[0019] Step S104, using the LZ4 algorithm to compress the preprocessed data, effectively reducing the amount of data transmission.

[0020] Step S105: Use the distributed cache system to efficiently store and read data.

[0021] Step S106, storing the preprocessed and compressed data in a distributed storage system for subsequent processing.

[0022] Furthermore, in step S103, the Z-score standardized data is converted into a unified format, and the specific formula is: z=(X-μ) / σ, where X is the data value, μ is the mean of the data, σ is the standard deviation, and z is the corresponding standardized score.

[0023] In step S104, the compression and decompression process is expressed as: C(t)=LZ4_Compress(D(t)), D(t)=LZ4_Decompress(C(t)), where D(t) is the amount of preprocessed data D(t), C(t) represents the compressed data, and D(t) is the decompressed data; LZ4_Compress(·) is a function used to compress input data into a smaller format; LZ4_Decompress(·) is used to decompress data compressed by LZ4_Compress back to the original format.

[0024] In step S106, data storage is represented as: Storage_Write(key, C(t)), and index and query optimization techniques are used to quickly retrieve data. The retrieval process is represented as: Storage_Read(key)→Retrieved_Data; Storage_Write(key,·) means storing data in a distributed storage system, using key as an identifier; key is a unique identifier of data and is used to find data in the storage system; Storage_Read(key) means reading data from the storage system according to key; Retrieved_Data represents the data read from the storage system.

[0025] Furthermore, step S2 includes the following steps.

[0026] Step S201, using a multi-level parsing thread pool architecture to process different types of tasks, defining thread pool levels and configuring thread pool parameters.

[0027] Step S202, according to the specific requirements of each parsing task, configure the priority, name and other attributes of the thread to meet the task requirements.

[0028] Step S203, selecting a suitable file converter based on the data format and setting parameters, including encoding format, separator and date format.

[0029] Step S204, implement the task scheduling strategy to reasonably allocate tasks to ensure that the priority and execution order of the tasks meet the requirements.

[0030] Step S205, reading data from the distributed storage system using streaming reading to ensure data consistency and integrity.

[0031] Furthermore, step S3 implements different segmentation strategies for communication data of different formats and structures, and parses them into structured data, which specifically includes the following steps.

[0032] The processing of structured data includes: using conventional parsing tools to parse data, extract fields and content; segmenting based on the row-column relationship of the data table and the logical association of the fields. For a table T with a primary key PK and a foreign key FK, the segmentation logic is represented by the following pseudo code: Segment 1 ={t∈T:ForeignKey(t)∈KeySet 1}, Segment 2 ={t∈T:ForeignKey(t)∈KeySet 2}, where KeySet 1 and KeySet 2 It is a primary key set defined based on logical conditions. 1 and Segment 2 The value of the ForeignKey is assigned to the KeySet 1 and KeySet 2 A data set in T, t is a data item in T, and ForeignKey(t) is the foreign key value extracted from data item t, which is used for data association and segmentation; considering the hierarchical structure and referential integrity factors of the data, the data is divided into multiple paragraphs or sub-tables.

[0033] The processing of unstructured data includes: cleaning and formatting data to remove noise; inputting the processed text into the pre-trained T5 model. The input data format of the T5 model is: X=Preprocess(D), where X represents the result of preprocessing the data D, D is the original unstructured data, and Preprocess(·) is a data cleaning and normalization function; the T5 model understands the contextual information in the data based on the self-attention mechanism and generates a segmentation strategy; applying the generated segmentation strategy, the text is divided into meaningful paragraphs or sub-parts and converted into structured data form.

[0034] The processing of semi-structured data includes: parsing the existing structured part of the data and extracting clear fields and content, where the formula for extracting the structured part is expressed as: Fields={Extract(D,Path):Path∈Schema}, where Extract(D,Path) is a function used to extract data, Schema is a set of defined field paths, Fields represents a set of fields extracted from data D, and Path is a path or key used to locate or access a specific part of data D; for free text or irregular content parts, natural language processing tools are used to generate dynamic segmentation strategies; the parsed structured part is merged with the unstructured part, the mapping relationship is aligned, and a complete structured data header and content are generated, where the segmentation strategy S is expressed as: S={(s i ,e i )}, where s i and e i Respectively represent the start and end position of each segment, and the segmented data D i Expressed as: D i =D[s i :e i ], the formula for combining the parsed structured part and the unstructured part is: D merged =D structured ∪D unstructured , where D structured Represents structured data, that is, data that has been organized according to a specific structure or pattern; unstructured Represents unstructured data; D merged Yes D structured and D unstructured The combined result of the two represents comprehensive data that combines structured data and unstructured data.

[0035] Furthermore, step S4 includes the following steps.

[0036] Step S401, standardize all structured data to reduce the impact of dimensions.

[0037] Step S402, reducing the computational complexity by dimensionality reduction processing while retaining the main feature information of the data.

[0038] Step S403, reconstruct the test set data using the trained adversarial variational autoencoder, calculate the reconstruction error of each sample, and set the anomaly detection threshold τ according to the reconstruction error distribution.

[0039] Step S404: perform an abnormality determination on each sample, and the determination rule is: if the reconstruction error of the sample is greater than τ, it is determined to be abnormal.

[0040] Furthermore, in S403, the model architecture of the adversarial variational autoencoder includes an encoder network and a decoder network, and its structure and training process are as follows.

[0041] For the encoder network, the input feature X is mapped to the distribution q(z|X) in the latent space. The output of the encoder is the mean and variance of the latent variable z, q(z|X)=N(μ(X),Σ(X)), where q(z|X) represents the conditional probability distribution of the latent variable z given the input data X; N(μ(X),Σ(X)) represents a normal distribution whose mean and covariance are defined by μ(X) and Σ(X); μ(X) is the mean function, which represents the mean of the latent variable z given the input X; Σ(X) is the covariance matrix function, which represents the covariance matrix of the latent variable z given the input X; N represents the normal distribution.

[0042] For the decoder network, the latent variable z is remapped back to the data space to generate an output X^X^=fdecoder(z) with the same scale as the input data X, where fdecoder is the neural network function of the decoder.

[0043] The loss function of the adversarial variational autoencoder includes reconstruction loss and KL divergence loss, where the reconstruction loss is used to measure the difference between the original input X and the reconstructed output X^, using the mean square error (MSE): L reconstruction =||XX^|| 2 , where L reconstruction Represents the reconstruction loss, which is used to measure the difference between the input data and the reconstructed data; the KL divergence loss is used to measure the difference between the output distribution of the encoder and the prior distribution: L KL =D KL (q(z|X)||p(z)), where L KL represents the KL divergence loss, which measures the difference between the potential distribution and the prior distribution; D KL represents the KL divergence, which represents the distance between two probability distributions; p(z) represents the prior distribution, which defines the underlying probability distribution of the latent variable z; the total loss function L AVAE Expressed as: L AVAE =L reconstruction +βL KL , where β is the weight coefficient used to balance the reconstruction loss and KL divergence.

[0044] An adaptive mechanism is introduced to dynamically adjust the structure of the encoder network and the decoder network according to the complexity of the data, and an attention mechanism is used to increase the focus on important features in the data.

[0045] Use the stochastic gradient descent optimization algorithm to minimize the total loss function of the adversarial variational autoencoder and gradually adjust the model parameters: θ^=θ-αm t / (v t 1 / 2 +ε), where θ is the model parameter, α is the learning rate, and m t represents the value of the gradient momentum at time step t; v t represents the variance of the gradient at time step t; θ^ represents the updated model parameters; ε is a small constant used to avoid division by zero errors.

[0046] Further, step S5 includes the following steps.

[0047] Based on big data, we construct an abnormal knowledge graph G=(V,E), where V represents the node set of abnormal patterns, E represents the relationship set between abnormal patterns, and each node v i ∈V represents an abnormal pattern, including the following information: abnormal characteristics, types, handling measures and other related metadata.

[0048] For each detected abnormal sample, its feature vector X i Mapped to the corresponding node in the knowledge graph.

[0049] The similarity measurement method is used to calculate the similarity between abnormal samples and each abnormal pattern. The formula is expressed as: Sim(X i ,v j )=X i v j / ||X i ||||v j ||,v j is the feature vector of a node in the knowledge graph; Sim(X i ,v j ) represents the vector X i and vector v j The similarity between .

[0050] According to the calculated similarity score, the similarity threshold strategy τ is used for matching, which is expressed as: ifSim(X i ,v j )>τ, then X i ~v j , X i ~v j Represents X i and v j There is some kind of relationship or match between them.

[0051] If the match is successful, the exception pattern v that matches is called. jThe exception is processed by the associated predefined exception handling rules; otherwise, it is marked as an unknown exception.

[0052] The anomaly knowledge graph is updated regularly based on newly discovered anomaly patterns and processing results.

[0053] Furthermore, in step S7, for unknown anomalies that do not match existing anomaly patterns in the knowledge graph, a further analysis and optimization process is performed, and the specific steps include:

[0054] This includes dynamically updating anomaly detection rules and models, and using integrated scheduling algorithms to automatically schedule expert analysis, model training, and knowledge graph update tasks.

[0055] When a new abnormal pattern is detected and confirmed by experts, it is added to the knowledge graph G, expressed as: G'=G∪{X new}, where X new represents the new abnormal pattern, and G' represents the updated abnormal knowledge graph.

[0056] Use the new data and updated abnormal patterns to retrain the model and re-optimize the parameters of the adaptive enhanced variational autoencoder model to improve the detection ability of new abnormal patterns. The specific formula is: γ'=γ-η▽γL AVAE (X new ), where η represents the learning rate, γ represents the model parameters, γ' represents the updated adaptive enhanced variational autoencoder model parameters; ▽γ represents the loss function L AVAE For the gradient of γ; L AVAE Represents the loss function of the adaptive boosted variational autoencoder model.

[0057] A communication information segmentation parsing system based on big data and artificial intelligence is used to implement the above-mentioned communication information segmentation parsing method based on big data and artificial intelligence, and includes the following modules.

[0058] The communication data receiving module is responsible for receiving the communication data from the communication equipment in real time and performing preliminary format verification and data integrity check.

[0059] The distributed stream processing module adopts a distributed stream processing framework to support real-time processing and preprocessing of large-scale data streams; at the same time, it uses the LZ4 compression algorithm to reduce the amount of data transmission, and uses a distributed cache system to efficiently store and read data.

[0060] The task scheduling module is responsible for creating a thread pool, configuring parsing threads and file converters, and starting the thread pool to read data from the distributed storage system to achieve efficient data parsing and processing.

[0061] The segmentation strategy generation module generates corresponding segmentation strategies according to communication data of different formats and structures, and parses the data into structured headers and contents.

[0062] The anomaly detection and processing module uses an adaptive enhanced variational autoencoder to perform anomaly detection on preprocessed and parsed data, identify potential abnormal data and infer and predict it in a big data knowledge graph to determine the type of anomaly.

[0063] The knowledge graph update module reports the anomaly type and its context information for detected abnormal data. If it is a known anomaly, corresponding processing measures are taken. For unknown anomalies, an alarm is generated and submitted to experts for analysis and confirmation. If it is confirmed to be a new abnormal pattern, it is added to the knowledge graph, thereby dynamically updating the system's anomaly detection model.

[0064] The data processing control module takes corresponding processing measures according to the abnormal detection results, including shielding abnormal data and performing receiving operations on data determined to be normal, allowing the data to continue to flow and be processed.

[0065] Compared with the prior art, the present application has the following beneficial effects.

[0066] This application achieves efficient segmented parsing and anomaly detection of communication information through a series of steps: first, using a distributed stream processing framework to receive and preprocess data in real time, then using a multi-level parsing thread pool and an adaptive enhanced variational autoencoder to detect anomalies in the data, and finally matching the anomaly information with the knowledge graph, and dynamically updating the anomaly detection model to improve system performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 A flowchart of a communication information segmentation parsing method based on big data and artificial intelligence is provided in an embodiment of the present application.

[0068] Figure 2 A structural diagram of a communication information segmentation parsing system based on big data and artificial intelligence provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical scheme and advantages of the implementation of the present invention clearer, the technical scheme in the embodiment of the present invention will be described in more detail below in conjunction with the drawings in the embodiment of the present invention. In the drawings, the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The described embodiments are part of the embodiments of the present invention, not all of the embodiments.

[0070] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making any creative work shall fall within the scope of protection of the present invention.

[0071] The embodiments and directional terms described below with reference to the accompanying drawings are exemplary and intended to be used to explain the present invention, but should not be construed as limiting the present invention.

[0072] like Figure 1 As shown, a communication information segmentation parsing method based on big data and artificial intelligence includes the following steps.

[0073] Step S1, real-time data reception and processing stage: receive communication data in real time through communication equipment, use distributed stream processing framework to perform real-time data processing and preprocessing, use LZ4 algorithm to compress data to reduce transmission volume, and use distributed cache system to efficiently store and read data.

[0074] Step S2, data parsing processing stage: create and configure a thread pool; set parsing threads and file converters; start the thread pool, recursively traverse the data storage device, and achieve efficient parallel data parsing.

[0075] Step S3, data segmentation and structured parsing stage: applying corresponding segmentation strategies according to different formats and structures of communication data; parsing the segmented data into structured headers and contents.

[0076] Step S4, anomaly detection stage: Adaptive enhanced variational autoencoder (AVAE) is used to perform anomaly detection on the preprocessed and parsed data to identify potential abnormal data.

[0077] Step S5, matching the identified abnormal data with the knowledge graph based on big data to determine the abnormal type.

[0078] Step S6: For known anomalies, report the anomaly type and its context information; for unknown anomalies, generate an alarm and submit it to experts for analysis and confirmation.

[0079] Step S7: If it is confirmed as a new abnormal pattern, it is added to the knowledge graph and the anomaly detection model is dynamically updated; if it is confirmed as abnormal data, a shielding operation is performed; if it is confirmed as normal data, a receiving operation is performed to allow the data to continue to flow and be processed.

[0080] Furthermore, step S1 includes the following steps.

[0081] Step S101: receiving structured (eg, JSON, XML) or unstructured (eg, raw byte stream) data in real time via a communication device.

[0082] Step S102: Select a suitable distributed stream processing framework, such as Apache Flink, to provide real-time data stream processing capabilities with high throughput and low latency.

[0083] Step S103, data preprocessing, including removing noise and processing missing values, and converting the data into a unified format to ensure data consistency.

[0084] Step S104, using the LZ4 algorithm to compress the preprocessed data, effectively reducing the amount of data transmission.

[0085] Step S105: Use a distributed cache system (such as Cassandra) to efficiently store and read data.

[0086] Step S106, storing the preprocessed and compressed data in a distributed storage system (such as HDFS) for subsequent processing.

[0087] Among them, in step S103, the Z-score is used to standardize the data and convert it into a unified format. The specific calculation formula is: z=(X-μ) / σ, where X is the data value, μ is the mean of the data, σ is the standard deviation, and z is the corresponding standardized score.

[0088] In step S104, LZ4 is an algorithm with high compression speed and is suitable for real-time applications. The compression and decompression process can be expressed as: C(t)=LZ4_Compress(D'(t)), D"(t)=LZ4_Decompress(C(t)), where D'(t) is the amount of preprocessed data D(t), C(t) represents compressed data, and D"(t) is decompressed data; LZ4_Compress(·) is a function for compressing input data into a smaller format; LZ4_Decompress(·) is used to decompress data compressed by LZ4_Compress back to the original format.

[0089] In step S106, data storage is represented as: Storage_Write(key, C(t)), using index and query optimization technology (such as secondary index, hash table) to quickly retrieve data. The retrieval process can be represented as, Storage_Read(key)→Retrieved_Data; Storage_Write(key,·) means storing data in a distributed storage system, using key as an identifier; key is a unique identifier of data, used to find data in the storage system; Storage_Read(key) means reading data from the storage system according to key; Retrieved_Data means data read from the storage system.

[0090] Furthermore, step S2 includes the following steps.

[0091] Step S201, using a multi-level parsing thread pool architecture to process different types of tasks, defining thread pool levels and configuring thread pool parameters.

[0092] Step S202, according to the specific requirements of each parsing task, configure the priority, name and other attributes of the thread to meet the task requirements.

[0093] Step S203, select a suitable file converter based on the data format (such as JSON, XML), and set necessary parameters, including encoding format, separator, date format, etc.

[0094] Step S204, implement the task scheduling strategy (priority queue) to reasonably allocate tasks and ensure that the priority and execution order of the tasks meet the requirements.

[0095] Step S205, reading data from the distributed storage system using streaming reading to ensure data consistency and integrity.

[0096] Among them, in step S201, the multi-level parsing thread pool architecture includes: a core thread pool, which handles basic parsing tasks; an extended thread pool, which handles tasks that require additional computing resources; and an IO thread pool, which specifically handles file read and write operations. The thread pool parameters include: the number of core threads, which sets the number of threads in the core thread pool to handle regular tasks; the maximum number of threads, which sets the maximum number of threads in the extended thread pool to handle high-load tasks; the queue capacity, which configures the capacity of the task queue to manage the queuing and scheduling of tasks; and the thread pool scheduling strategy, which selects an appropriate scheduling strategy (such as a fixed thread pool or a cache thread pool).

[0097] In step 202, the parsing tasks include data conversion, formatting and cleaning.

[0098] Furthermore, step S3 implements different segmentation strategies for communication data of different formats and structures, and parses them into structured data, which specifically includes the following steps.

[0099] The processing of structured data includes: using conventional parsing tools (such as Python's json, pandas, xml.etree.ElementTree, etc.) to parse data and extract fields (table headers) and content; segmenting based on the row-column relationship of the data table and the logical association of the fields (such as primary key, foreign key). For a table T with a primary key PK and a foreign key FK, the segmentation logic is represented by the following pseudo code: Segment 1 ={t∈T:ForeignKey(t)∈KeySet 1}, Segment 2={t∈T:ForeignKey(t)∈KeySet 2}, where KeySet 1 and KeySet 2 It is a primary key set defined based on logical conditions; Segment 1 and Segment 2 The value of the ForeignKey is assigned to the KeySet 1 and KeySet 2 A data set; t is a data item in T; ForeignKey(t) is the foreign key value extracted from data item t, which is used for data association and segmentation; considering factors such as the hierarchical structure of the data (such as parent-child relationship) and referential integrity, the data is divided into multiple paragraphs or sub-tables.

[0100] The processing of unstructured data includes: cleaning and formatting the data to remove noise (such as special characters, HTML tags, etc.); inputting the processed text into the pre-trained T5 model. The input data format of the T5 model is: X=Preprocess(D), where X represents the result of preprocessing the data D; D is the original unstructured data, and Preprocess(·) is the data cleaning and normalization function; the T5 model understands the contextual information in the data based on the self-attention mechanism and generates a segmentation strategy. The segmentation strategy S contains the start position, end position and key attributes of each paragraph; applying the generated segmentation strategy, the text is divided into meaningful paragraphs or sub-parts and converted into structured data forms (such as headers and contents).

[0101] The processing of semi-structured data includes: parsing the existing structured parts in the data (such as JSON key-value pairs and XML tags) and extracting clear fields and content. The formula for extracting the structured part is expressed as: Fields={Extract(D,Path):Path∈Schema}, where Extract(D,Path) is a function used to extract data, Schema is a collection of field paths; Fields represents a collection of fields extracted from data D; Path is a path or key used to locate or access a specific part of data D; for free text or irregular content parts, natural language processing tools such as the T5 model are used to generate dynamic segmentation strategies; the parsed structured part is merged with the unstructured part, the mapping relationship is aligned, and a complete structured data header and content are generated. The segmentation strategy S is expressed as: S={(s i ,e i )}, where s i and e i Respectively represent the start and end position of each segment, and the segmented data Di Expressed as: D i =D[s i :e i ], the formula for combining the parsed structured part and the unstructured part is: D merged =D structured ∪D unstructured , where D structured Represents structured data, that is, data that has been organized according to a specific structure or pattern; unstructured Represents unstructured data; D merged Yes D structured and D unstructured The combined result of the two represents the comprehensive data that combines structured data and unstructured data. During the entire processing process, if parsing fails or anomalies are encountered, the unparseable part is skipped or a warning message is generated. For large-scale data sets, batch processing and parallel computing strategies are used to improve the efficiency of parsing and conversion.

[0102] Furthermore, step S4 includes the following steps.

[0103] Step S401 , standardize all structured data (including originally structured data, and semi-structured and unstructured data processed by the T5 model) to reduce the impact of dimensions.

[0104] Step S402, reducing the computational complexity by dimensionality reduction processing (such as PCA or t-SNE) while retaining the main feature information of the data.

[0105] Step S403, use the trained adversarial variational autoencoder (AVAE) to reconstruct the test set data, calculate the reconstruction error of each sample, and set the threshold τ of anomaly detection according to the reconstruction error distribution.

[0106] Step S404, perform an abnormality determination on each sample, and the determination rule is: if the reconstruction error of the sample>τ, it is determined to be abnormal.

[0107] Among them, in S403, the model architecture of the adversarial variational autoencoder (AVAE) includes an encoder network and a decoder network, and the detailed structure and training process are as follows.

[0108] For the encoder network, the input feature X is mapped to the distribution q(z|X) in the latent space. The output of the encoder is the mean and variance of the latent variable z, q(z|X)=N(μ(X),Σ(X)), where q(z|X) represents the conditional probability distribution of the latent variable z given the input data X; N(μ(X),Σ(X)) represents a normal distribution whose mean and covariance are defined by μ(X) and Σ(X); μ(X) is the mean function, which represents the mean of the latent variable z given the input X; Σ(X) is the covariance matrix function, which represents the covariance matrix of the latent variable z given the input X; N represents the normal distribution.

[0109] For the decoder network, the latent variable z is remapped back to the data space to generate an output X^ with the same scale as the input data X: X^=fdecoder(z), where fdecoder is the neural network function of the decoder.

[0110] The loss function of the Adversarial Variational Autoencoder (AVAE) includes reconstruction loss and KL divergence loss, where the reconstruction loss is used to measure the difference between the original input X and the reconstructed output X^, using the mean square error (MSE): L reconstruction =||XX^|| 2 , where L reconstruction Represents the reconstruction loss, which is used to measure the difference between the input data and the reconstructed data; the KL divergence loss is used to measure the difference between the output distribution of the encoder and the prior distribution: L KL =D KL (q(z|X)||p(z)), where L KL represents the KL divergence loss, which measures the difference between the potential distribution and the prior distribution; D KL represents the KL divergence, which represents the distance between two probability distributions; p(z) represents the prior distribution, which defines the underlying probability distribution of the latent variable z; the total loss function L AVAE Expressed as: L AVAE =L reconstruction +βL KL , where β is the weight coefficient used to balance the reconstruction loss and KL divergence. The Adversarial Variational Autoencoder (AVAE) introduces an adaptive mechanism that can dynamically adjust the structure of the encoder network and the decoder network according to the complexity of the data, and uses an attention mechanism to increase the focus on important features in the data. Use the stochastic gradient descent (SGD) optimization algorithm to minimize the total loss function of the Adversarial Variational Autoencoder (AVAE) and gradually adjust the model parameters: θ^=θ-αm t / (v t 1 / 2 +ε), where θ is the model parameter, α is the learning rate, and m trepresents the value of the gradient momentum at time step t; v t represents the variance of the gradient at time step t; θ^ represents the updated model parameters; ε is a small constant used to avoid division by zero errors.

[0111] Further, step S5 includes the following steps.

[0112] Based on big data, we construct an abnormal knowledge graph G=(V,E), where V represents the node set of abnormal patterns, E represents the relationship set between abnormal patterns, and each node v i ∈V represents an abnormal pattern, including the following information: abnormal characteristics, types, handling measures and other related metadata.

[0113] For each detected abnormal sample, its feature vector X i Mapped to the corresponding nodes in the knowledge graph through hash mapping or vectorization methods.

[0114] The similarity between abnormal samples and each abnormal pattern is calculated using similarity measurement methods including cosine similarity, Euclidean distance or Mahalanobis distance. The formula is: Sim(X i ,v j )=X i v j / ||X i ||||v j ||,v j is the feature vector of a node in the knowledge graph; Sim(X i ,v j ) represents the vector X i and vector v j The similarity between .

[0115] According to the calculated similarity score, the K-nearest neighbor (KNN) algorithm or the similarity threshold strategy τ is used for matching, which can be expressed as: if Sim(X i ,v j )>τ, then X i ~v j , X i ~v j Represents X i and v j There is some kind of relationship or match between them.

[0116] If the match is successful, the predefined exception handling rule associated with the matched exception pattern vj is called for processing; otherwise, it is marked as an unknown exception.

[0117] The anomaly knowledge graph is updated regularly based on newly discovered anomaly patterns and processing results.

[0118] Furthermore, in step S7, for unknown anomalies that do not match the existing anomaly patterns in the knowledge graph, further analysis and optimization processes need to be performed. The specific steps include: dynamically updating anomaly detection rules and models, and automatically scheduling expert analysis, model training, and knowledge graph update tasks using an integrated scheduling algorithm; when a new anomaly pattern is detected and confirmed by an expert, it is added to the knowledge graph G, expressed as: G'=G∪{X new}, where X new Represents the new abnormal pattern, G' represents the updated abnormal knowledge graph; the model is retrained using the new data and the updated abnormal pattern, and the parameters of the adaptive enhanced variational autoencoder (AVAE) model are re-optimized to improve the detection ability of the new abnormal pattern. This process is achieved through the gradient descent method, that is, γ'=γ-η▽γL AVAE (X new ), where η represents the learning rate, γ represents the model parameters, γ' represents the updated adaptive enhanced variational autoencoder model parameters; ▽γ represents the loss function L AVAE For the gradient of γ; L AVAE Represents the loss function of the adaptive boosted variational autoencoder model.

[0119] like Figure 2 As shown, a communication information segmentation parsing system based on big data and artificial intelligence includes the following modules.

[0120] Communication data receiving module: responsible for receiving communication data from communication equipment in real time and performing preliminary format verification and data integrity check.

[0121] Distributed stream processing module: It adopts a distributed stream processing framework to support real-time processing and preprocessing of large-scale data streams; it uses the LZ4 compression algorithm to reduce the amount of data transmission, and uses a distributed cache system to efficiently store and read data.

[0122] Task scheduling module: responsible for creating thread pools, configuring parsing threads and file converters, and starting thread pools to read data from distributed storage systems to achieve efficient data parsing and processing.

[0123] Segmentation strategy generation module: Generates corresponding segmentation strategies according to communication data of different formats and structures, and parses the data into structured headers and contents.

[0124] Anomaly detection and processing module: Use adaptive enhanced variational autoencoder (AVAE) to perform anomaly detection on preprocessed and parsed data, identify potential abnormal data and infer and predict it in the big data knowledge graph to determine the type of anomaly.

[0125] Knowledge graph update module: For detected abnormal data, report the abnormal type and its context information. If it is a known abnormality, take corresponding measures. For unknown abnormalities, generate alarms and submit them to experts for analysis and confirmation. If it is confirmed to be a new abnormal pattern, add it to the knowledge graph, thereby dynamically updating the system's anomaly detection model.

[0126] Data processing control module: Take corresponding processing measures according to the abnormal detection results, including shielding abnormal data, and performing receiving operations on data judged to be normal, allowing the data to continue to flow and be processed.

[0127] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Although the present invention has been described in detail with reference to the above embodiments, a person skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A communication information segmentation analysis method based on big data and artificial intelligence, characterized in that: The following steps are involved: Step S1, receiving and processing communication data in real time; Step S2, performing data analysis on the communication data; Step S3, applying corresponding segmentation strategies according to different formats and structures of the communication data, and parsing the segmented data into structured headers and contents; Step S4, using an adaptive enhanced variational autoencoder to perform anomaly detection on the preprocessed and parsed data to identify potential abnormal data; Step S5, matching the identified abnormal data with the knowledge graph based on big data to determine the abnormal type; Step S6: For known anomalies, report the anomaly type and its context information; for unknown anomalies, generate an alarm and submit it to experts for analysis and confirmation; Step S7: If it is confirmed as a new abnormal pattern, it is added to the knowledge graph and the anomaly detection model is dynamically updated; if it is confirmed as abnormal data, a shielding operation is performed; if it is confirmed as normal data, a receiving operation is performed to allow the data to continue to flow and be processed.

2. According to the method for segmented analysis of communication information based on big data and artificial intelligence in claim 1, it is characterized in that: Step S1 includes the following steps: Step S101, receiving structured or unstructured data in real time through a communication device; Step S102, selecting a distributed stream processing framework to provide real-time data stream processing capabilities with high throughput and low latency; Step S103, data preprocessing, including removing noise and processing missing values, and converting the data into a unified format to ensure data consistency; Step S104, compressing the pre-processed data using the LZ4 algorithm to effectively reduce the amount of data transmission; Step S105, using a distributed cache system to efficiently store and read data; Step S106, storing the preprocessed and compressed data in a distributed storage system for subsequent processing.

3. According to claim 2, a communication information segmentation analysis method based on big data and artificial intelligence is characterized in that: In step S103, the Z-score is used to standardize the data and convert it into a unified format. The specific formula is: z=(X-μ) / σ, where X is the data value, μ is the mean of the data, σ is the standard deviation, and z is the corresponding standardized score; In step S104, the compression and decompression process is represented as: C(t)=LZ4_Compress(D(t)), D(t)=LZ4_Decompress(C(t)), where D(t) is the amount of preprocessed data D(t), C(t) represents compressed data, and D(t) is decompressed data; LZ4_Compress(·) is a function for compressing input data into a smaller format; LZ4_Decompress(·) is used to decompress data compressed by LZ4_Compress back to the original format; In step S106, data storage is represented as: Storage_Write(key, C(t)), and index and query optimization techniques are used to quickly retrieve data. The retrieval process is represented as: Storage_Read(key)→Retrieved_Data; Storage_Write(key,·) means storing data in a distributed storage system, using key as an identifier; key is a unique identifier of data and is used to find data in the storage system; Storage_Read(key) means reading data from the storage system according to key; Retrieved_Data represents the data read from the storage system.

4. According to the method for segmented analysis of communication information based on big data and artificial intelligence in claim 1, it is characterized in that: Step S2 includes the following steps: Step S201, using a multi-level parsing thread pool architecture to process different types of tasks, defining the thread pool hierarchy and configuring thread pool parameters; Step S202, according to the specific requirements of each parsing task, configure the priority, name and other attributes of the thread to meet the task requirements; Step S203, selecting a suitable file converter based on the data format and setting parameters, including encoding format, separator and date format; Step S204, implement the task scheduling strategy to reasonably allocate tasks to ensure that the priority and execution order of tasks meet the requirements; Step S205, reading data from the distributed storage system using streaming reading to ensure data consistency and integrity.

5. According to claim 4, a communication information segmentation analysis method based on big data and artificial intelligence is characterized in that: Step S3 implements different segmentation strategies for communication data of different formats and structures, and parses them into structured data. It includes the following steps: The processing of structured data includes: parsing data using conventional parsing tools to extract fields and content; segmenting based on the row-column relationship of the data table and the logical association of the fields. For a table T with a primary key PK and a foreign key FK, the segmentation logic is represented by the following pseudo code: Segment1={t∈T:ForeignKey(t)∈KeySet1}, Segment2={t∈T:ForeignKey(t)∈KeySet2}, where KeySet1 and KeySet2 are primary key sets defined based on logical conditions, Segment1 and Segment2 are data sets assigned to KeySet1 and KeySet2 based on the value of ForeignKey, t is a data item in T, and ForeignKey(t) is a foreign key value extracted from data item t for data association and segmentation; considering the hierarchical structure and referential integrity factors of the data, the data is divided into multiple paragraphs or subtables; The processing of unstructured data includes: cleaning and formatting data to remove noise; inputting the processed text into the pre-trained T5 model. The input data format of the T5 model is: X=Preprocess(D), where X represents the result of preprocessing the data D, D is the original unstructured data, and Preprocess(·) is the data cleaning and normalization function; the T5 model understands the contextual information in the data based on the self-attention mechanism and generates a segmentation strategy; applying the generated segmentation strategy, the text is divided into meaningful paragraphs or sub-parts and converted into structured data form; The processing of semi-structured data includes: parsing the existing structured part of the data and extracting clear fields and content, where the formula for extracting the structured part is expressed as: Fields={Extract(D,Path):Path∈Schema}, where Extract(D,Path) is a function used to extract data, Schema is a set of defined field paths, Fields represents a set of fields extracted from data D, and Path is a path or key used to locate or access a specific part of data D; for free text or irregular content parts, natural language processing tools are used to generate dynamic segmentation strategies; the parsed structured part is merged with the unstructured part, the mapping relationship is aligned, and a complete structured data header and content are generated, where the segmentation strategy S is expressed as: S={(s i ,e i )}, where s i and e i Respectively represent the start and end position of each segment, and the segmented data D i Expressed as: D i =D[s i :e i ], the formula for combining the parsed structured part and the unstructured part is: D merged =D structured ∪D unstructured , where D structured Represents structured data, that is, data that has been organized according to a specific structure or pattern; unstructured Represents unstructured data; D merged Yes D structured and D unstructured The combined result of the two represents comprehensive data that combines structured data and unstructured data.

6. The communication information segmentation parsing method based on big data and artificial intelligence according to claim 1 is characterized in that: Step S4 includes the following steps: Step S401, standardize all structured data to reduce the impact of dimensions; Step S402, reducing the computational complexity by dimensionality reduction processing while retaining the main feature information of the data; Step S403, reconstruct the test set data using the trained adversarial variational autoencoder, calculate the reconstruction error of each sample, and set the threshold τ for anomaly detection according to the distribution of the reconstruction error; Step S404: perform an abnormality determination on each sample, and the determination rule is: if the reconstruction error of the sample is greater than τ, it is determined to be abnormal.

7. The communication information segmentation parsing method based on big data and artificial intelligence according to claim 6 is characterized in that: In S403, the model architecture of the adversarial variational autoencoder includes an encoder network and a decoder network, and its structure and training process are as follows: For the encoder network, the input feature X is mapped to the distribution q(z|X) in the latent space. The output of the encoder is the mean and variance of the latent variable z, q(z|X)=N(μ(X),Σ(X)), q(z|X) represents the conditional probability distribution of the latent variable z given the input data X; N(μ(X),Σ(X)) represents a normal distribution whose mean and covariance are defined by μ(X) and Σ(X); μ(X) is the mean function, which represents the mean of the latent variable z given the input X; Σ(X) is the covariance matrix function, which represents the covariance matrix of the latent variable z given the input X; N represents the normal distribution; For the decoder network, the latent variable z is remapped back to the data space to generate an output X^X^=fdecoder(z) with the same scale as the input data X, where fdecoder is the neural network function of the decoder; The loss function of the adversarial variational autoencoder includes reconstruction loss and KL divergence loss, where the reconstruction loss is used to measure the difference between the original input X and the reconstructed output X^, using the mean square error (MSE): L reconstruction =||XX^|| 2 , where L reconstruction Represents the reconstruction loss, which is used to measure the difference between the input data and the reconstructed data; the KL divergence loss is used to measure the difference between the output distribution of the encoder and the prior distribution: L KL =D KL (q(z|X)||p(z)), where L KL represents the KL divergence loss, which measures the difference between the potential distribution and the prior distribution; D KL represents the KL divergence, which represents the distance between two probability distributions; p(z) represents the prior distribution, which defines the underlying probability distribution of the latent variable z; the total loss function L AVAE Expressed as: L AVAE =L reconstruction +βL KL , where β is the weight coefficient used to balance the reconstruction loss and KL divergence; Introducing an adaptive mechanism to dynamically adjust the structure of the encoder network and decoder network according to the complexity of the data, and using an attention mechanism to increase the focus on important features in the data; Use the stochastic gradient descent optimization algorithm to minimize the total loss function of the adversarial variational autoencoder and gradually adjust the model parameters: θ^=θ-αm t / (v t 1 / 2 +ε), where θ is the model parameter, α is the learning rate, and m t represents the value of the gradient momentum at time step t; v t represents the variance of the gradient at time step t; θ^ represents the updated model parameters; ε is a small constant used to avoid division by zero errors.

8. The communication information segmentation parsing method based on big data and artificial intelligence according to claim 1 is characterized in that: Step S5 includes the following steps: Based on big data, we construct an abnormal knowledge graph G=(V,E), where V represents the node set of abnormal patterns, E represents the relationship set between abnormal patterns, and each node v i ∈V represents an anomaly pattern, including the following information: anomaly characteristics, types, treatment measures, and other related metadata; For each detected abnormal sample, its feature vector X i Map to the corresponding node in the knowledge graph; The similarity measurement method is used to calculate the similarity between abnormal samples and each abnormal pattern. The formula is expressed as: Sim(X i ,v j )=X i v j / ||X i ||||v j ||,v j is the feature vector of a node in the knowledge graph; Sim(X i ,v j ) represents the vector X i and vector v j The similarity between According to the calculated similarity score, the similarity threshold strategy τ is used for matching, which is expressed as: if Sim(X i ,v j )>τ, then X i ~v j , X i ~v j Represents X i and v j There is a relationship or match between them; If the match is successful, the exception pattern v that matches is called. j The associated predefined exception handling rules are used for processing; otherwise, it is marked as an unknown exception; The anomaly knowledge graph is updated regularly based on newly discovered anomaly patterns and processing results.

9. The communication information segmentation parsing method based on big data and artificial intelligence according to claim 1 is characterized in that: In step S7, for unknown anomalies that do not match the existing anomaly patterns in the knowledge graph, further analysis and optimization processes are performed, and the specific steps include: This includes dynamically updating anomaly detection rules and models, and using integrated scheduling algorithms to automatically schedule expert analysis, model training, and knowledge graph update tasks; When a new abnormal pattern is detected and confirmed by experts, it is added to the knowledge graph G, expressed as: G'=G∪{X new }, where X new represents the new abnormal pattern, and G' represents the updated abnormal knowledge graph; Use the new data and updated abnormal patterns to retrain the model and re-optimize the parameters of the adaptive enhanced variational autoencoder model to improve the detection ability of new abnormal patterns. The specific formula is: γ'=γ-η▽γL AVAE (X new ), where η represents the learning rate, γ represents the model parameters, γ' represents the updated adaptive enhanced variational autoencoder model parameters; ▽γ represents the loss function L AVAE For the gradient of γ; L AVAE Represents the loss function of the adaptive boosted variational autoencoder model.

10. A communication information segmentation analysis system based on big data and artificial intelligence, implemented based on a communication information segmentation analysis method based on big data and artificial intelligence according to any one of claims 1 to 9, characterized in that: Includes the following modules: The communication data receiving module is responsible for receiving the communication data from the communication equipment in real time and performing preliminary format verification and data integrity check; The distributed stream processing module adopts a distributed stream processing framework to support real-time processing and preprocessing of large-scale data streams. It also uses the LZ4 compression algorithm to reduce the amount of data transmission and uses a distributed cache system to efficiently store and read data. The task scheduling module is responsible for creating a thread pool, configuring parsing threads and file converters, and starting the thread pool to read data from the distributed storage system to achieve efficient data parsing and processing; The segmentation strategy generation module generates corresponding segmentation strategies according to communication data of different formats and structures, and parses the data into structured headers and contents; The anomaly detection and processing module uses an adaptive enhanced variational autoencoder to detect anomalies in preprocessed and parsed data, identify potential abnormal data, and perform reasoning and prediction based on the big data knowledge graph to determine the type of anomaly. The knowledge graph update module reports the anomaly type and its context information for detected abnormal data. If it is a known anomaly, it takes corresponding processing measures. For unknown anomalies, it generates an alarm and submits it to experts for analysis and confirmation. If it is confirmed to be a new abnormal pattern, it is added to the knowledge graph, thereby dynamically updating the system's anomaly detection model. The data processing control module takes corresponding processing measures according to the abnormal detection results, including shielding abnormal data and performing receiving operations on data determined to be normal, allowing the data to continue to flow and be processed.

Images