Mobile signaling data processing method and system based on streaming computing

Abstract

The invention belongs to the technical field of mobile communication network big data processing, and discloses a mobile signaling data processing method and system based on streaming computing, and the method comprises the steps: receiving a mobile signaling real-time data stream; constructing a composite key based on the base station identifier and the user identifier, and grouping the data streams; a session window based on event time is applied to each group of data, and continuous signaling events are divided into the same processing unit; in each processing unit, incremental aggregation is carried out on the signaling data, and a Snappy compression algorithm is adopted to carry out compression processing on the signaling data; the compressed data stream is input into a complex event processing module based on an Flink CEP library; performing mode matching on the compressed data stream based on a service rule mode, and when an event sequence conforming to the service rule mode is matched, retaining one signaling event in the event sequence and discarding other repeated events; and outputting the de-duplicated data. According to the invention, the mobile signaling data of the mobile phone is efficiently compressed, and the data volume is greatly reduced.

Description

Technical Field

[0001] This invention belongs to the field of mobile communication network big data processing technology, specifically relating to a mobile signaling data processing method and system based on streaming computing. Background Technology

[0002] With the large-scale deployment of 5G networks and the arrival of the era of the Internet of Things, the signaling data generated in mobile communication networks (such as user location updates, session establishment, handover, paging events, etc.) is experiencing explosive growth. This type of data is a core asset for operators to optimize networks, analyze user behavior, and gain business insights, but its inherent data characteristics pose significant challenges to traditional data processing technologies.

[0003] Specifically, mobile signaling data has four typical characteristics: (1) Huge data volume. The daily signaling data volume generated in a single province can reach TB level, and with the surge in the number of 5G access devices, the data volume is expected to increase by more than 10 times. (2) Strong spatiotemporal continuity: Data records strictly follow the time sequence and have strong spatial correlation, such as the continuous location update trajectory of user equipment; (3) Extremely high redundancy: Due to the periodic reporting mechanism of the network and the mobility of user equipment, the same user will generate a large amount of data with repeated or nearly repeated fields in the same location area in a short period of time; (4) Low business value density: The original signaling data contains a large amount of underlying signaling used for network connection maintenance, and only about 20%-50% of the data has direct value for upper-layer business analysis.

[0004] Currently, the industry commonly uses big data technologies based on Hadoop MapReduce or Apache Spark batch processing frameworks to process such data. However, the inherent high latency of batch processing (typically tens of minutes to several hours) cannot meet the business needs of real-time network monitoring, anomaly detection, and other applications that require response times in seconds or even milliseconds. Summary of the Invention

[0005] The purpose of this invention is to provide a mobile signaling data processing method based on streaming computing. By using Flink's window operations and incremental aggregation technology, combined with the Snappy compression algorithm, mobile signaling data of mobile phones can be compressed efficiently, significantly reducing the amount of data.

[0006] In a first aspect, embodiments of the present invention provide a mobile signaling data processing method based on streaming computing, the method being implemented based on the Apache Flink streaming computing framework, the method comprising: Receive a real-time mobile signaling data stream, wherein the real-time mobile signaling data stream contains signaling events with spatiotemporal continuity; A composite key is constructed based on the base station identifier and user identifier in the signaling event, and the data stream is grouped using the composite key so that signaling events of the same user under the same base station are assigned to the same parallel processing instance. Apply an event-time-based session window to each group of data to divide consecutively generated signaling events into the same processing unit; Within each processing unit, the signaling data is incrementally aggregated, and before the incrementally aggregated data is stored in the status data, it is compressed using the Snappy compression algorithm to obtain compressed data. The compressed data stream is input into the complex event processing module based on the Flink CEP library; In the complex event processing module, pattern matching is performed on the compressed data stream based on a predefined business rule pattern. When an event sequence that matches the business rule pattern is matched, one signaling event in the event sequence is retained and the remaining duplicate events are discarded to achieve business-level deduplication. The business rule pattern is used to identify duplicate signaling events generated by the same user within a preset time threshold and a geographical displacement threshold. Output the deduplicated data to the downstream storage or analysis system.

[0007] Optionally, the incremental aggregation of signaling data includes: For numerical data in the signaling data, the sum or average value is calculated using the ReduceFunction or AggregateFunction; the numerical data includes signal strength or traffic data. For the text data in the signaling data, perform classification and statistical counting, and the text data includes signaling types.

[0008] Optionally, after compressing the data using the Snappy compression algorithm to obtain the compressed data, the method further includes: The compressed data is stored in the RocksDB state backend, and the state backend is used to persist the state data to the local disk. Enable Flink's distributed snapshot mechanism to periodically create checkpoints for the state data in the state backend and persist them to a remote persistent storage system; the distributed snapshot mechanism is configured as asynchronous snapshot mode and / or incremental snapshot mode. A time-to-live (TTL) is set for the state data in the state backend to automatically clean up expired and unupdated state data based on key granularity. The key includes user identifier and base station identifier.

[0009] Optionally, in the complex event processing module, pattern matching is performed on the compressed data stream based on predefined business rule patterns. When an event sequence that matches the business rule pattern is found, one signaling event in the event sequence is retained and the remaining duplicate events are discarded, thus achieving business-level deduplication, including: Define a rule pattern to identify repeated location update signaling generated by the same user identifier within a specific geographical range within a preset time window; When an event sequence that matches the rule pattern is found, the first signaling data in the sequence is retained and the remaining signaling data is discarded; The rule pattern is defined using Flink CEP's Pattern API, and the definition process includes: Define the signaling event of type location update as the start event; The number of times the starting event occurs within the preset time window is limited by the quantifier API and reaches a preset threshold. The conditional API is used to limit the location coordinate deviation of consecutively occurring starting events within a specific geographical range.

[0010] Optionally, the process of outputting the deduplicated data to a downstream storage system or analysis system includes: The deduplicated data is sent to downstream systems through the output interface of the Apache Flink streaming computing framework; wherein, the downstream systems include at least one of the following: a message middleware, a distributed file system, and a relational database, wherein the message middleware is used to provide data to a real-time data analysis system; the distributed file system is used to store data in a columnar storage format for a long time; and the relational database is used to support online querying and analysis.

[0011] Optionally, the output interface is the addSink operation provided by the DataStream API of the Apache Flink stream computing framework.

[0012] Secondly, embodiments of the present invention provide a mobile signaling data processing system based on streaming computing, implemented using the Apache Flink streaming computing framework, the system comprising: A data stream receiving module is used to receive a real-time mobile signaling data stream, wherein the real-time mobile signaling data stream contains signaling events with spatiotemporal continuity; The data stream grouping module is used to construct a composite key based on the base station identifier and user identifier in the signaling event, and to group the data stream with the composite key so that the signaling events of the same user under the same base station are assigned to the same parallel processing instance. The processing unit partitioning module is used to apply an event time-based session window to each group of data, dividing continuously generated signaling events into the same processing unit; The data compression module is used to perform incremental aggregation on signaling data in each processing unit, and to compress the incrementally aggregated data using the Snappy compression algorithm before storing the data in the status data, so as to obtain compressed data. The compressed data sending module is used to input the compressed data stream into the complex event processing module based on the Flink CEP library; The data deduplication module is used in the complex event processing module to perform pattern matching on the compressed data stream based on a predefined business rule pattern. When an event sequence that matches the business rule pattern is matched, one signaling event in the event sequence is retained and the remaining duplicate events are discarded to achieve business-level deduplication. The business rule pattern is used to identify duplicate signaling events generated by the same user within a preset time threshold and a geographical displacement threshold. The deduplication data output module is used to output deduplicated data to downstream storage or analysis systems.

[0013] Thirdly, embodiments of the present invention provide an electronic device, including: At least one processor; Memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method described in the first aspect.

[0014] Fourthly, embodiments of the present invention provide a computer-readable storage medium that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the method described in the first aspect.

[0015] Fifthly, embodiments of the present invention provide a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect.

[0016] The mobile signaling data processing method based on streaming computing provided in this invention utilizes Flink's windowing operations and incremental aggregation technology, combined with the Snappy compression algorithm, to efficiently compress mobile signaling data, significantly reducing the data volume. During data transmission, it effectively reduces the required bandwidth resources. For example, when processing mobile signaling data transmission in large-scale urban areas, it can reduce processing bandwidth by approximately 40% compared to traditional methods, alleviating network transmission pressure. Furthermore, leveraging Flink CEP's powerful pattern matching capabilities, it can accurately deduplicate mobile signaling data according to complex business rules, effectively removing redundant information caused by data duplication and improving data quality. In practical applications, it can accurately identify and remove approximately 80% of duplicate signaling data, providing a more reliable data foundation for subsequent data analysis and decision-making.

[0017] Furthermore, the technical solution of this invention is based on Flink's distributed architecture, which can easily cope with the continuous growth of mobile signaling data volume. As the network scale expands and the number of users increases, the system's processing capacity can be expanded by adding computing nodes; at the same time, business rule patterns can be easily modified and adjusted to adapt to the needs of mobile signaling data processing under different operators and different business scenarios. Attached Figure Description

[0018] Figure 1 A flowchart illustrating a mobile signaling data processing method based on streaming computing, provided as an embodiment of the present invention; Figure 2 A schematic diagram of a mobile signaling data processing system based on streaming computing provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0019] The present invention will be described in detail below through embodiments.

[0020] With the large-scale deployment of 5G networks and the arrival of the era of the Internet of Things, the signaling data generated in mobile communication networks (such as user location updates, session establishment, handover, paging events, etc.) is experiencing explosive growth. This type of data is a core asset for operators to optimize networks, analyze user behavior, and gain business insights, but its inherent data characteristics pose significant challenges to traditional data processing technologies.

[0021] Specifically, mobile signaling data has four typical characteristics: (1) Huge data volume. The daily signaling data volume generated in a single province can reach TB level, and with the surge in the number of 5G access devices, the data volume is expected to increase by more than 10 times. (2) Strong spatiotemporal continuity: Data records strictly follow the time sequence and have strong spatial correlation, such as the continuous location update trajectory of user equipment; (3) Extremely high redundancy: Due to the periodic reporting mechanism of the network and the mobility of user equipment, the same user will generate a large amount of data with repeated or nearly repeated fields in the same location area in a short period of time; (4) Low business value density: The original signaling data contains a large amount of underlying signaling used for network connection maintenance, and only about 20%-50% of the data has direct value for upper-layer business analysis.

[0022] Currently, the industry commonly uses big data technologies based on Hadoop MapReduce or Apache Spark batch processing frameworks to process such data. However, the inherent high latency of batch processing (typically tens of minutes to several hours) cannot meet the business needs of real-time network monitoring, anomaly detection, and other applications that require response times in seconds or even milliseconds.

[0023] Furthermore, in the process of implementing this invention, the following obvious defects were found in the prior art when processing signaling data: 1. Batch processing mode has high latency. It cannot meet the real-time requirements of signaling data (usually requiring end-to-end latency <100ms).

[0024] 2. Centralized compression bottleneck. Traditional methods often rely on single-node compression algorithms (such as GZIP and Snappy), which only perform byte-level redundancy elimination and cannot utilize distributed computing resources for parallel compression. This results in single-point performance bottlenecks and data skew issues, with compression latency reaching the second level.

[0025] 3. Insufficient identification of semantic redundancy. General hash-based deduplication methods can only identify completely identical data copies, and cannot identify data that is completely redundant in business logic due to slight deviations in field values ​​caused by spatiotemporal continuity (such as multiple location updates by the same user within a small displacement range). This type of semantic deduplication usually requires the use of third-party components, introducing additional system complexity and performance overhead.

[0026] 4. Low resource utilization and high costs. The structured characteristics of signaling data were not fully considered, and there was a lack of specific compression optimization for high-entropy fields such as user ID, location area code (LAC), and cell identifier (CID), resulting in an overall compression rate of less than 30%. Storage costs already account for more than 40% of the operator's total expenditure, and inefficient compression further exacerbates the cost pressure.

[0027] 5. Lack of end-to-end optimization. Data compression and business deduplication processes are often isolated and executed as independent processing steps, failing to form a collaborative optimization mechanism, resulting in a lengthy overall processing chain and high resource consumption.

[0028] In summary, facing the multiple challenges of data volume, real-time performance, and cost in the 5G era, existing big data batch processing architectures and traditional data compression and deduplication technologies are no longer sufficient to effectively address these issues. Therefore, there is an urgent need in this field for an integrated end-to-end solution that deeply integrates streaming processing, distributed compression, and business semantic understanding to achieve efficient governance of mobile signaling data across the entire chain from collection to storage.

[0029] To address the aforementioned technical problems in existing technologies, this invention provides a mobile signaling data processing method based on streaming computing. This method is implemented using the Apache Flink streaming computing framework, as follows: Figure 1 As shown, the method includes: S110 receives real-time mobile signaling data streams.

[0030] Among them, the real-time data stream of mobile signaling includes signaling events with spatiotemporal continuity.

[0031] Specifically, mobile signaling data is collected and buffered through a distributed message middleware (such as Apache Kafka). Flink jobs subscribe to data in real time from a specified Kafka topic by implementing the SourceFunction interface or directly using the FlinkKafkaConsumer connector. The real-time mobile signaling data stream contains signaling events with spatiotemporal continuity. Each signaling event is a structured data record containing the following key fields: unique event identifier, timestamp, user identifier (such as user's mobile phone number), base station identifier (such as cell identifier or cell identity identifier), signaling type (such as location update, handover request, session establishment, etc.), location coordinates, signal strength, and other service-related attributes.

[0032] S120: Construct a composite key based on the base station identifier and user identifier in the signaling event, and group the data stream with the composite key so that the signaling events of the same user under the same base station are assigned to the same parallel processing instance.

[0033] Specifically, the base station identifier and user identifier are extracted from each signaling event, and these two identifiers are concatenated or combined into a composite key. Then, using Flink's `keyBy()` operation, the data stream is re-partitioned using this composite key as the partitioning key. This ensures that all signaling events with the same base station identifier and user identifier are routed to the same parallel processing instance on the same Flink task manager in a distributed environment, laying the foundation for subsequent windowing processing based on spatiotemporal locality.

[0034] S130 applies an event-time-based session window to each group of data, dividing continuously generated signaling events into the same processing unit.

[0035] Specifically, a session window based on event time is applied to the data for each group. The session window is defined using the Windowfunction method, and its session interval (e.g., 5 minutes) can be configured according to business characteristics. This windowing strategy can effectively capture the signaling sequence generated by a user during a continuous activity. Flink's watermark mechanism is responsible for handling out-of-order events, ensuring the correctness of window calculation.

[0036] S140: Within each processing unit, the signaling data is incrementally aggregated, and before the incrementally aggregated data is stored in the status data, it is compressed using the Snappy compression algorithm to obtain compressed data.

[0037] Specifically, incremental aggregation is performed on the signaling data within each window. The incremental aggregation method differs for numerical and textual data within the signaling data. To clarify the solution, the specific implementation of incremental aggregation will be described in detail in the following examples.

[0038] After aggregation, the data is compressed using the Snappy compression algorithm before being stored in the state data. Specifically, a custom Snappy-based serializer is used to compress the data before storing it in the state data, significantly reducing the storage space occupied by the state data and alleviating storage pressure.

[0039] S150 inputs the compressed data stream into the complex event processing module based on the Flink CEP library.

[0040] The data stream, after window compression, is input into the complex event processing module built on the Flink CEP library via connect() or direct forwarding. This module, as the next stage of the entire process, is responsible for semantic redundancy identification.

[0041] S160, in the complex event processing module, performs pattern matching on the compressed data stream based on predefined business rule patterns. When an event sequence that matches the business rule pattern is matched, one signaling event in the event sequence is retained and the remaining duplicate events are discarded, thus achieving business-level deduplication.

[0042] Among them, the business rule mode is used to identify repeated signaling events generated by the same user within preset time thresholds and geographical displacement thresholds.

[0043] Specifically, in the CEP module, pattern matching is performed based on predefined business rule patterns.

[0044] First, define the pattern. Using Flink CEP's Pattern API, for example, define a rule to identify more than 3 location update signaling events generated by the same user within a 50-meter geographic range within 5 minutes.

[0045] Secondly, pattern matching and deduplication are performed. The CEP engine's internal NFA (Non-deterministic Finite Automaton) matches the data stream. When a sequence of events that matches the rules is found, a decision is made based on business semantics (such as whether to keep the first or last valid signaling message) to discard the remaining duplicate events, thereby achieving business-level deduplication rather than just byte-level deduplication.

[0046] S170 outputs the deduplicated data to the downstream storage or analysis system.

[0047] The cleaned data stream, after deduplication, is output to downstream systems via Flink's Sink operation. Different Sink connectors can be flexibly configured according to different business needs.

[0048] Specifically, output can be sent to message middleware (such as Kafka) for consumption by systems such as real-time monitoring and risk control. Alternatively, it can be sent to a distributed file system (such as HDFS) and stored in a columnar storage format (such as Parquet) for offline big data analysis. It can also be sent to relational databases (such as MySQL) or OLAP databases, supporting online interactive queries and report generation.

[0049] The mobile signaling data processing method based on streaming computing provided in this invention utilizes Flink's windowing operations and incremental aggregation technology, combined with the Snappy compression algorithm, to efficiently compress mobile signaling data, significantly reducing the data volume. During data transmission, it effectively reduces the required bandwidth resources. For example, when processing mobile signaling data transmission in large-scale urban areas, it can reduce processing bandwidth by approximately 40% compared to traditional methods, alleviating network transmission pressure. Furthermore, leveraging Flink CEP's powerful pattern matching capabilities, it can accurately deduplicate mobile signaling data according to complex business rules, effectively removing redundant information caused by data duplication and improving data quality. In practical applications, it can accurately identify and remove approximately 80% of duplicate signaling data, providing a more reliable data foundation for subsequent data analysis and decision-making.

[0050] Furthermore, the technical solution of this invention is based on Flink's distributed architecture, which can easily cope with the continuous growth of mobile signaling data volume. As the network scale expands and the number of users increases, the system's processing capacity can be expanded by adding computing nodes; at the same time, business rule patterns can be easily modified and adjusted to adapt to the needs of mobile signaling data processing under different operators and different business scenarios.

[0051] exist Figure 1 Based on the illustrated embodiment, as one implementation of this invention, incremental aggregation of signaling data may include the following steps, namely steps a1 and a2: Step a1: For numerical data in the signaling data, calculate its sum or average value using the ReduceFunction or AggregateFunction; numerical data includes signal strength or traffic data. Step a2: For text data in the signaling data, perform classification, statistical counting, and the text data includes signaling types.

[0052] In this implementation, differentiated incremental aggregation strategies are adopted for numerical and textual data in signaling data, taking into account their different characteristics. Compared with the uniform processing method in existing technologies, this approach brings at least significant benefits: 1. Significantly improves data processing efficiency and system throughput.

[0053] (1) Achieve accurate calculation and reduce redundancy. For numerical data (such as signal strength and traffic consumption), the sum, average, maximum and other aggregated values ​​are directly calculated through ReduceFunction or AggregateFunction, avoiding the need to pass all the original values ​​down and store them. For example, 100 signal strength reports from a user in a window are aggregated into one "average strength" value, reducing the data volume by 99%.

[0054] (2) Achieve efficient statistics. For text-based data (such as signaling data), the large number of repetitive text enumeration values ​​are converted into lightweight key-value pairs through classification and statistical counting. For example, "Location update, location update, location update, handover request" can be directly aggregated into {LOCATION_UPDATE:3, HANDOVER_REQUEST:1}. This greatly reduces the amount of data that needs to be serialized and transmitted, especially for highly repetitive text fields such as signaling data, where the compression effect is extremely significant.

[0055] This refined processing strategy effectively reduces data volume at the forefront of data processing, directly alleviating the computational load of all subsequent processing stages (such as compression, network transmission, state storage, and CEP matching), thereby significantly improving the throughput and processing efficiency of the entire stream processing pipeline.

[0056] 2. Significantly optimizes the utilization of memory and CPU computing resources.

[0057] (1) Incremental aggregation, resource-friendly. ReduceFunction and AggregateFunction adopt incremental aggregation mode. The system does not need to cache all the original data in the window in memory, but only needs to maintain a very small aggregation state (such as an accumulator or a counter). This greatly reduces the state size of Flink jobs, reduces garbage collection pressure, avoids the risk of memory overflow caused by excessive state data, and enables the system to stably process large-scale data streams with lower resource overhead.

[0058] (2) This is in stark contrast to the method of using ProcessWindowFunction, which requires collecting all data before calculation. The latter consumes a lot of memory and creates a performance bottleneck when the window is large.

[0059] 3. Lays an efficient foundation for subsequent compression and storage.

[0060] After aggregation, the data is transformed from "line-by-line, highly redundant logs" into "highly structured, information-dense statistical results." These aggregation results (such as numerical values ​​and counts) themselves have excellent compressibility, laying an optimal foundation for subsequent efficient compression using compression algorithms such as Snappy, thereby further significantly reducing storage and input / output costs.

[0061] 4. Retain key business information to increase data value density.

[0062] This approach doesn't simply discard data; instead, it intelligently extracts and retains the most valuable statistical information while reducing the data volume. Averages reflect the overall level of user experience, while counts demonstrate the frequency of user or web behaviors—these are the core metrics that subsequent data analysis will focus on.

[0063] Ultimately, what is transmitted and stored is no longer a large number of low-value raw bytes, but high-value-density business information, which maximizes the cost-effectiveness of storage and data value.

[0064] Through refined data governance, a significant reduction in data volume was achieved at the beginning of the data processing chain, thereby achieving higher processing throughput and more valuable data output with lower resource consumption and computing costs, laying a solid foundation for the entire end-to-end optimization solution.

[0065] exist Figure 1 Based on the illustrated embodiment, as an implementation of this invention, after compressing the data using the Snappy compression algorithm to obtain the compressed data, the method may further include the following steps, namely steps b1 to b3: Step b1: Store the compressed data in the RocksDB state backend, and use the state backend to persist the state data to the local disk.

[0066] The Snappy compressed data is stored on the RocksDB state backend. RocksDB, a high-performance, embedded, persistent key-value store, is deeply integrated into Flink as the specific implementation of the state backend. Its working principle utilizes a log-structured merge tree data structure. First, state data is written to a mutable in-memory table (MemTable). When the MemTable is full, it is converted to an immutable read-only structure and flushed to the local disk to form an SSTable file. This design ensures that the vast majority of state data is persisted to disk, while memory is only used as a high-speed cache (BlockCache) to store frequently accessed hot data. This shifts the cost of state data storage from expensive memory to lower-cost disk, allowing the system to handle nearly unlimited state data with limited memory resources. This effectively avoids the Out of Memory (OOM) problem caused by state data bloat, achieving optimal resource utilization.

[0067] Step b2 enables Flink's distributed snapshot mechanism, which periodically creates checkpoints for state data in the state backend and persists them to a remote persistent storage system. The distributed snapshot mechanism is configured as asynchronous snapshot mode and / or incremental snapshot mode.

[0068] Specifically, Flink's distributed snapshot mechanism, or checkpoint mechanism, is enabled. This mechanism triggers a global operation periodically (e.g., every minute) to coordinate all parallel task nodes to create checkpoints for their state data in RocksDB. This checkpoint data is not stored locally but is persisted to a remote persistent storage system (such as HDFS, S3, or OSS), ensuring that even in the event of a catastrophic failure of the entire computing cluster, recovery can be achieved from the most recent successful checkpoint, guaranteeing the exact-once semantics of the data. To further optimize performance and avoid blocking normal data processing during checkpoint creation, the distributed snapshot mechanism is configured in asynchronous snapshot mode, allowing the snapshot process to run asynchronously in a background thread without pausing the main data processing thread. Additionally, an incremental snapshot mode can be configured, persisting only the state data that has changed since the last snapshot, rather than the entire state data, significantly reducing network transmission overhead and I / O pressure on remote storage, minimizing the impact of the fault tolerance mechanism on data processing performance.

[0069] Step b3 sets a time-to-live (TTL) for the state data in the state backend to automatically clean up expired and unupdated state data based on key granularity. The key includes the user identifier and the base station identifier.

[0070] Specifically, a TTL (Time-To-Live) is set for the state data in the state backend. Using Flink's State TTL API, a timeout (e.g., 1 hour) can be configured for each state item (based on key granularity). The system automatically tracks the time when the state data corresponding to each key (composed of user and base station identifiers) was last updated. For keys where no new signaling data flows in within the timeout period (e.g., a user has left the coverage area of ​​a base station or has turned off), their corresponding state data will be automatically and asynchronously cleaned up by the system. This fine-grained cleanup strategy based on key granularity avoids invalid state data occupying valuable disk and memory resources indefinitely, achieves automatic storage space reclamation, and ensures that the system's processing capacity and resource utilization will not degrade due to the infinite growth of state data during long-term operation, thus achieving system self-maintenance and high resource utilization.

[0071] This embodiment, by integrating steps b1 to b3, further enhances the system's production-grade reliability, stability, and economy based on efficient data processing. Its beneficial effects include: (1) By sinking the state data to the local disk through the RocksDB state backend, the bottleneck of the state data volume being limited by the physical memory capacity of the cluster is broken, enabling the system to process TB or even PB level state data at a lower hardware cost, and achieving excellent economy and horizontal scalability.

[0072] (2) Significantly reduces storage pressure. Utilizing Flink's distributed storage and state management mechanism, the compressed data is stored in the RocksDB state backend. Redundant state data is cleaned up by appropriately setting state TTLs, reducing memory and disk storage space usage. Compared to storage solutions that do not employ this invention, storing the same amount of mobile signaling data can save approximately 50% of storage resources, significantly reducing storage pressure.

[0073] (3) It provides strong fault tolerance through a distributed checkpoint mechanism based on remote storage. Any job interruption caused by software or hardware failure can be automatically recovered from the nearest checkpoint within minutes, without manual intervention and with zero data loss, meeting the high availability requirements of continuous and stable operation 24 / 7.

[0074] (4) Optimized performance and extremely low operation and maintenance overhead. Asynchronous and incremental snapshot modes minimize the performance overhead of fault tolerance mechanisms, avoid data processing pauses caused by snapshot creation, and ensure high throughput and low latency data processing capabilities. At the same time, the state TTL mechanism realizes automatic cleanup and reclamation of storage resources without manual intervention by operation and maintenance personnel, which greatly reduces the long-term operation and maintenance costs of the system.

[0075] (5) Complete end-to-end solution. This embodiment, combined with the aforementioned data compression and CEP deduplication steps, forms a complete end-to-end data governance solution covering efficient computing, intelligent governance, reliable storage and automated operation and maintenance. It systematically solves various technical problems in mobile signaling data processing and has high industrial implementation value.

[0076] exist Figure 1 Based on the illustrated embodiment, as one implementation of this invention, in the complex event processing module, pattern matching is performed on the compressed data stream based on a predefined business rule pattern. When an event sequence that conforms to the business rule pattern is matched, one signaling event in the event sequence is retained and the remaining duplicate events are discarded, thereby achieving business-level deduplication. This may include the following steps c1 and c2: Step c1: Define a rule pattern to identify repeated location update signaling generated by the same user identifier within a specific geographical range within a preset time window.

[0077] The rule pattern is defined through Flink CEP's Pattern API. The definition process of the rule pattern includes: Define the signaling event of type location update as the start event; The quantifier API is used to limit the occurrence of the starting event to a preset threshold within a preset time window. Use conditional APIs to limit the location coordinate deviation of consecutively occurring starting events to a specific geographical range.

[0078] The core of step c1 lies in transforming business knowledge into pattern rules that can be recognized by the Flink CEP engine. The specific implementation is as follows: First, define the pattern's start event. By calling the Flink CEP's `Pattern.begin()` method and using a conditional filter, the pattern's start is explicitly defined as a signaling event of type "LOCATION_UPDATE". This condition ensures that the pattern matching engine only focuses on signaling types relevant to the deduplication target, efficiently filtering out a large amount of irrelevant signaling data (such as handover completion, call end, etc.), significantly improving matching efficiency.

[0079] Secondly, the frequency and time range of events are defined using the quantifier API. Using the `.timesOrMore()` or `.times()` quantifier methods, it is specified that the "location update" event must occur at or above a preset threshold (e.g., 3 times) within a preset time window (e.g., 5 minutes). This condition is subject to a time constraint using the `.within()` method. This definition accurately captures the frequent and repetitive reporting behavior of user devices within a short period due to signal fluctuations, network reselection, etc., which is a typical characteristic of service redundancy.

[0080] Finally, spatial constraints are defined using conditional APIs. This is crucial for achieving "spatiotemporal continuity" deduplication. By appending `.where()` conditions to subsequent event patterns and writing custom filtering functions, the geographic location deviation between two consecutive location update events is calculated (e.g., by calculating the Euclidean distance of latitude and longitude or calculating the surface distance). By determining whether this deviation is within a specific geographic range (e.g., 50 meters), it is confirmed whether multiple reports occurred in the same small geographic area. This ensures that the system can intelligently identify truly redundant data that is "repeatedly reported at the same location," while retaining signaling from different locations generated by normal user movement, thus avoiding accidental deletion.

[0081] Step c2: When an event sequence that matches the rule pattern is found, retain the first signaling data in the sequence and discard the remaining signaling data.

[0082] The pattern defined in step c1 is compiled into a nondeterministic finite automaton (NFA) and run in the Flink CEP engine. The engine scans the input compressed data stream, tracking the event sequence for each user (using the user identifier as the key) in real time.

[0083] When a user's event sequence fully satisfies all the conditions defined in step c1 (i.e., within 5 minutes, within a 50-meter radius, there are 3 or more "location update" signaling events), the engine determines that it matches an event sequence that conforms to the rules.

[0084] At this point, the system will trigger a pre-defined selection strategy. In this implementation, the strategy is to retain the first signaling data in the event sequence and explicitly discard the remaining signaling data in the sequence. The first data is retained because it typically represents the first valid event occurring within that time window and geographical scope, and has the highest business value. This directly achieves deduplication at the business logic level, rather than simply deleting duplicate data at the byte level.

[0085] This embodiment, through the implementation of steps c1 and c2 above, brings the following significant beneficial effects: (1) High deduplication accuracy and maximized data value. By introducing a geographic displacement threshold as one of the core judgment conditions for deduplication, this invention achieves intelligent deduplication based on business semantics. The system can accurately distinguish between "redundant reports from the same location" and "valid signaling generated by normal movement", thereby eliminating high redundancy while retaining valuable data reflecting the user's real movement trajectory to the maximum extent, significantly improving the accuracy and value of subsequent data analysis.

[0086] (2) Excellent processing efficiency. Pattern matching based on the NFA state machine of Flink CEP enables single-pass scanning and real-time processing of streaming data, avoiding the huge overhead of multiple full table scans or join queries required in traditional solutions. At the same time, the aforementioned data compression steps greatly reduce the amount of data that the CEP engine needs to process, jointly ensuring low latency and high throughput of deduplication operations.

[0087] (3) The rules are flexible and configurable, and highly adaptable. The deduplication rules (time window, frequency threshold, geographical range) are defined through a high-level API, rather than being hard-coded into the program. This allows business personnel to quickly adjust and optimize the deduplication strategy according to different network characteristics, regional features, or analysis needs, making the solution highly adaptable and scalable.

[0088] In summary, this implementation provides an efficient, accurate, and flexible business-level data deduplication solution, effectively solving the problem of business logic redundancy caused by the spatiotemporal continuity of mobile signaling data, and providing downstream systems with a cleaner and higher value density data source.

[0089] exist Figure 1 Based on the illustrated embodiment, as one implementation of this invention, outputting the deduplicated data to a downstream storage system or analysis system may include: The deduplicated data is sent to the downstream system through the output interface of the Apache Flink stream computing framework.

[0090] The downstream system includes at least one of the following: message middleware, distributed file system, and relational database. The message middleware is used to provide data to the real-time data analysis system; the distributed file system is used to store data in a columnar storage format for a long time; and the relational database is used to support online querying and analysis.

[0091] Specifically, the deduplicated data is sent out by calling the general output interface of the Apache Flink streaming computing framework. This interface is a core component of the Flink DataStream API, providing a unified programming model for sending data to external systems. Developers can achieve seamless data output by calling this method and passing in the connector instance required to connect to a specific downstream system as a parameter. This design decouples the data processing logic from the output target, resulting in a clear system architecture and convenient maintenance.

[0092] Furthermore, this invention fully leverages Flink's rich ecosystem, supporting the flexible output of data to various types of downstream systems to adapt to different business scenario requirements: 1. Message Middleware. Utilizing components such as Kafka Connector provided by Flink, the deduplicated data stream is sent to the message topic in real time. This data can be immediately consumed by subsequent real-time data analysis systems (such as real-time dashboards, risk control engines, and real-time recommendation systems), providing instant data support for businesses requiring latency in the second or even millisecond range.

[0093] 2. Distributed File System. Using the StreamingFileSink connector, data is written to the distributed file system in a columnar storage format. These formats, due to their high compression ratio and query optimization features, are ideal for long-term storage of massive amounts of historical data and subsequent batch offline analysis (such as user behavior analysis and network quality report generation), significantly reducing storage costs and improving query efficiency.

[0094] 3. Relational databases. Data is written to database tables. This is primarily used to support online querying and analysis, providing operators with real-time interactive query capabilities, or providing data interfaces for user-facing applications (such as billing queries and network coverage maps).

[0095] As one implementation of this invention, the output interface is the addSink operation provided by the DataStream API of the Apache Flink streaming computing framework.

[0096] The `addSink` operation is a method of the `DataStream` object, responsible for receiving an object that implements the `SinkFunction` interface. `SinkFunction` encapsulates all the complex logic for interacting with external systems, including: (1) Connection Management. Establishing, maintaining, and closing connections with downstream systems.

[0097] (2) Transaction coordination (if needed). Participate in Flink's checkpoint mechanism to achieve end-to-end exact one-time semantic output, ensuring that data is not lost or duplicated in the event of a failure.

[0098] (3) Data serialization. Convert the data objects inside Flink into a format suitable for the target system.

[0099] (4) Batch writing. To optimize I / O performance, SinkFunction usually implements a batch writing strategy, which caches multiple data entries and writes them to the target system at once, significantly improving throughput.

[0100] This implementation method, through the above-described output design and implementation, brings the following significant beneficial effects: (1) Through Flink's unified Sink API, this invention can seamlessly connect to various mainstream external systems. When business requirements change and new output targets need to be replaced or added, only the corresponding connector configuration needs to be changed, without modifying the core data processing logic, making the system architecture highly flexible and able to respond quickly to business changes.

[0101] (2) End-to-end data reliability guarantee. Through deep integration with Flink's distributed snapshot mechanism, it supports exactly-once output semantics. Even if a system failure occurs during the data output process, it can ensure that each deduplicated data is output exactly once, avoiding data duplication or loss, and providing extremely reliable data protection for downstream businesses.

[0102] (3) By distributing data to different storage systems on demand, hierarchical storage and value maximization of data are achieved. High-frequency real-time data enters the message queue to support immediate decision-making, all historical data is stored in a low-cost object storage in an efficient columnar format to support in-depth data mining, and key indicator data is stored in the database to support online querying. This model achieves optimal control of overall storage costs while meeting diverse business needs.

[0103] (4) High performance and high throughput output. The addSink() operation and the connector implementation behind it have been highly optimized, supporting parallelism settings and batch writing. It can make full use of network and I / O resources, ensuring that the processed data stream can be written to the downstream system with extremely high throughput, avoiding the output stage becoming the performance bottleneck of the entire pipeline.

[0104] In summary, this output implementation scheme provides a reliable, flexible, efficient, and enterprise-level scalable data outlet for the present invention, ensuring that high-value signaling data after complex processing can be stably and efficiently consumed and utilized by various downstream systems.

[0105] This invention also provides a mobile signaling data processing system based on streaming computing, implemented using the Apache Flink streaming computing framework, such as... Figure 2 As shown, the system includes: Data stream receiving module 210 is used to receive mobile signaling real-time data stream, wherein the mobile signaling real-time data stream contains signaling events with spatiotemporal continuity; The data stream grouping module 220 is used to construct a composite key based on the base station identifier and user identifier in the signaling event, and to group the data stream with the composite key so that the signaling events of the same user under the same base station are assigned to the same parallel processing instance. The processing unit partitioning module 230 is used to apply an event time-based session window to each group of data, dividing continuously generated signaling events into the same processing unit; The data compression module 240 is used to perform incremental aggregation on the signaling data in each processing unit, and to compress the incrementally aggregated data using the Snappy compression algorithm before storing the data in the status data to obtain the compressed data. The compressed data sending module 250 is used to input the compressed data stream into the complex event processing module based on the Flink CEP library; The data deduplication module 260 is used in the complex event processing module to perform pattern matching on the compressed data stream based on a predefined business rule pattern. When an event sequence that conforms to the business rule pattern is matched, one signaling event in the event sequence is retained and the remaining duplicate events are discarded to achieve business-level deduplication. The business rule pattern is used to identify duplicate signaling events generated by the same user within a preset time threshold and a geographical displacement threshold. The deduplication data output module 270 is used to output the deduplicated data to the downstream storage system or analysis system.

[0106] Thirdly, embodiments of the present invention provide an electronic device, such as... Figure 3 As shown, it includes: At least one processor 301; Memory 302 for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method described in the first aspect.

[0107] Fourthly, embodiments of the present invention provide a computer-readable storage medium that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the method described in the first aspect.

[0108] Fifthly, embodiments of the present invention provide a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect.

[0109] The mobile signaling data processing method based on streaming computing provided in this invention utilizes Flink's windowing operations and incremental aggregation technology, combined with the Snappy compression algorithm, to efficiently compress mobile signaling data, significantly reducing the data volume. During data transmission, it effectively reduces the required bandwidth resources. For example, when processing mobile signaling data transmission in large-scale urban areas, it can reduce processing bandwidth by approximately 40% compared to traditional methods, alleviating network transmission pressure. Furthermore, leveraging Flink CEP's powerful pattern matching capabilities, it can accurately deduplicate mobile signaling data according to complex business rules, effectively removing redundant information caused by data duplication and improving data quality. In practical applications, it can accurately identify and remove approximately 80% of duplicate signaling data, providing a more reliable data foundation for subsequent data analysis and decision-making.

[0110] Furthermore, the technical solution of this invention is based on Flink's distributed architecture, which can easily cope with the continuous growth of mobile signaling data volume. As the network scale expands and the number of users increases, the system's processing capacity can be expanded by adding computing nodes; at the same time, business rule patterns can be easily modified and adjusted to adapt to the needs of mobile signaling data processing under different operators and different business scenarios.

[0111] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention without departing from the principles and spirit of the present invention.

Claims

1. A mobile signaling data processing method based on streaming computing, characterized in that, The method is implemented based on the Apache Flink stream computing framework, and the method includes: Receive a real-time mobile signaling data stream, wherein the real-time mobile signaling data stream contains signaling events with spatiotemporal continuity; A composite key is constructed based on the base station identifier and user identifier in the signaling event, and the data stream is grouped using the composite key so that signaling events of the same user under the same base station are assigned to the same parallel processing instance. Apply an event-time-based session window to each group of data to divide consecutively generated signaling events into the same processing unit; Within each processing unit, signaling data is incrementally aggregated, and before the incrementally aggregated data is stored in the status data, it is compressed using the Snappy compression algorithm to obtain compressed data. The compressed data stream is input into the complex event processing module based on the Flink CEP library; In the complex event processing module, pattern matching is performed on the compressed data stream based on a predefined business rule pattern. When an event sequence that matches the business rule pattern is matched, one signaling event in the event sequence is retained and the remaining duplicate events are discarded to achieve business-level deduplication. The business rule pattern is used to identify duplicate signaling events generated by the same user within a preset time threshold and a geographical displacement threshold. Output the deduplicated data to the downstream storage or analysis system.

2. The method according to claim 1, characterized in that, The incremental aggregation of signaling data includes: For numerical data in the signaling data, the sum or average value is calculated using the ReduceFunction or AggregateFunction; the numerical data includes signal strength or traffic data. For the text data in the signaling data, perform classification and statistical counting, and the text data includes signaling types.

3. The method according to claim 1, characterized in that, After compressing the data using the Snappy compression algorithm to obtain the compressed data, the method further includes: The compressed data is stored in the RocksDB state backend, and the state backend is used to persist the state data to the local disk. Enable Flink's distributed snapshot mechanism to periodically create checkpoints for the state data in the state backend and persist them to a remote persistent storage system; the distributed snapshot mechanism is configured as asynchronous snapshot mode and / or incremental snapshot mode. A time-to-live (TTL) is set for the state data in the state backend to automatically clean up expired and unupdated state data based on key granularity. The key includes user identifier and base station identifier.

4. The method according to claim 1, characterized in that, In the complex event processing module, pattern matching is performed on the compressed data stream based on predefined business rule patterns. When an event sequence that matches the business rule pattern is found, one signaling event in the event sequence is retained and the remaining duplicate events are discarded, thus achieving business-level deduplication, including: Define a rule pattern to identify repeated location update signaling generated by the same user identifier within a specific geographical range within a preset time window; When an event sequence that matches the rule pattern is found, the first signaling data in the sequence is retained and the remaining signaling data is discarded; The rule pattern is defined using Flink CEP's Pattern API, and the definition process includes: Define the signaling event of type location update as the start event; The number of times the starting event occurs within the preset time window is limited by the quantifier API and reaches a preset threshold. The conditional API is used to limit the location coordinate deviation of consecutively occurring starting events within a specific geographical range.

5. The method according to claim 1, characterized in that, The process of outputting deduplicated data to a downstream storage or analysis system includes: The deduplicated data is sent to downstream systems through the output interface of the Apache Flink streaming computing framework; wherein, the downstream systems include at least one of the following: a message middleware, a distributed file system, and a relational database, wherein the message middleware is used to provide data to a real-time data analysis system; the distributed file system is used to store data in a columnar storage format for a long time; and the relational database is used to support online querying and analysis.

6. The method according to claim 5, characterized in that, The output interface is the addSink operation provided by the DataStream API of the Apache Flink stream computing framework.

7. A mobile signaling data processing system based on streaming computing, characterized in that, The system, implemented based on the Apache Flink stream computing framework, includes: A data stream receiving module is used to receive a real-time mobile signaling data stream, wherein the real-time mobile signaling data stream contains signaling events with spatiotemporal continuity; The data stream grouping module is used to construct a composite key based on the base station identifier and user identifier in the signaling event, and to group the data stream with the composite key so that the signaling events of the same user under the same base station are assigned to the same parallel processing instance. The processing unit partitioning module is used to apply an event-time-based session window to each group of data, dividing continuously generated signaling events into the same processing unit; The data compression module is used to perform incremental aggregation on signaling data in each processing unit, and to compress the incrementally aggregated data using the Snappy compression algorithm before storing the data in the status data, so as to obtain compressed data. The compressed data sending module is used to input the compressed data stream into the complex event processing module based on the Flink CEP library; The data deduplication module is used in the complex event processing module to perform pattern matching on the compressed data stream based on a predefined business rule pattern. When an event sequence that matches the business rule pattern is matched, one signaling event in the event sequence is retained and the remaining duplicate events are discarded to achieve business-level deduplication. The business rule pattern is used to identify duplicate signaling events generated by the same user within a preset time threshold and a geographical displacement threshold. The deduplication data output module is used to output deduplicated data to downstream storage or analysis systems.

8. An electronic device, characterized in that, include: At least one processor; Memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the method as described in any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.

Images