Communication bill real-time analysis and abnormality detection system and method

Abstract

This invention discloses a real-time call detail record (CDR) parsing and anomaly detection system and method, belonging to the field of communication data processing technology. The system includes: real-time acquisition of raw CDR data streams from network elements; generation of standardized CDR sequences based on a dynamic parsing template library combined with a sliding time window and sequence number verification; extraction of four-dimensional feature vectors (user, location, behavior, and tariff) from these sequences; inputting these vectors into a pre-trained lightweight anomaly detection model to calculate anomaly risk scores; and constructing a correlation graph between users and base stations. When the score exceeds a first threshold, the system identifies abnormal group activity based on the correlation graph and generates an alarm signal with a chain of evidence. If the score does not exceed the first threshold, the system automatically fine-tunes the model weights based on historical verification labels. Simultaneously, the system monitors the performance of streaming data entry and visualization, triggering degradation strategies to ensure stable operation and improving anomaly detection accuracy, real-time performance, and system robustness. This system is suitable for efficient CDR parsing and risk control.

Description

Technical Field

[0001] This invention belongs to the field of communication data processing technology, specifically a system and method for real-time parsing and anomaly detection of communication call detail records. Background Technology

[0002] With the rapid development of mobile communication networks and the continuous increase in communication network elements, raw call detail record (CDR) data is characterized by massive volume, high concurrency, and multiple formats. It commonly suffers from problems such as out-of-order processing, fragmentation, and inconsistent formats. Traditional fixed parsing methods are difficult to adapt to various types of network elements, resulting in low efficiency and accuracy in CDR parsing, failing to meet real-time processing requirements. Meanwhile, risks such as communication fraud, organized fraudulent calls, and abnormal traffic are becoming increasingly covert. Single-dimensional CDR detection is prone to missed or false detections, making it difficult to identify organized or related abnormal behaviors. Existing anomaly detection methods largely rely on offline analysis and static rules, lacking real-time performance. Furthermore, the models cannot be dynamically iterated and optimized based on manual review results, making it difficult to continuously improve detection accuracy. In high-concurrency CDR stream processing scenarios, the system is prone to high write latency and stuttering in visualization rendering. The lack of effective traffic smoothing and performance degradation mechanisms fails to guarantee the stable operation of core services. In addition, the inability of parsing templates to adaptively correct, incomplete anomaly evidence chains, and the absence of handling feedback mechanisms result in a long-term inefficient closed loop for CDR parsing and risk detection, making it difficult to meet the actual needs of current communication network security and efficient operation. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention proposes a real-time communication call detail record (CDR) parsing and anomaly detection system and method. It collects raw CDR data streams from network elements in real time, generates standardized CDR sequences using a dynamic parsing template library combined with a sliding time window and sequence number verification. Four-dimensional feature vectors (user, location, behavior, and tariff) are then extracted from these sequences and input into a pre-trained lightweight anomaly detection model to calculate an anomaly risk score, constructing a correlation graph between users and base stations. When the score exceeds a first threshold, it identifies group anomalies based on the correlation graph and generates an alarm signal with a chain of evidence. If the score does not exceed the first threshold, it automatically fine-tunes the model weights based on historical verification tags. Simultaneously, it monitors the performance of streaming data entry and visualization, triggering degradation strategies to ensure stable operation, improving anomaly detection accuracy, real-time performance, and system robustness. This system is suitable for efficient CDR parsing and risk control.

[0004] To achieve the above objectives, the present invention provides the following technical solution:

[0005] Real-time call detail record (CDR) parsing and anomaly detection methods include:

[0006] The system collects raw call detail record (CDR) data streams from different network element devices in real time, and simultaneously obtains preset dynamic parsing template library configuration data. Based on the corresponding rules of the dynamic parsing template library, combined with sliding time window and sequence number verification, the system performs timing correction and session merging on out-of-order and fragmented CDRs in the raw CDR data stream to generate a standardized CDR sequence.

[0007] The four-dimensional feature vectors of user, location, behavior and tariff are extracted in real time from the standardized call detail record (CDR) sequence and input into a pre-trained lightweight anomaly detection model to calculate the anomaly risk score of each CDR entity and construct the association graph between users and base stations.

[0008] If the abnormal risk score exceeds a preset first threshold, the abnormal behavior of the gang is identified based on the association graph, and a first risk alarm signal containing an abnormal evidence chain is generated; if it does not exceed the first threshold, the rule weights of the lightweight anomaly detection model are automatically fine-tuned based on the acquired historical review tags.

[0009] Based on the first risk alarm signal or the optimized rule weight, the standardized call detail record (CDR) sequence is streamed into the database and visualized at a preset real-time streaming frame rate, and the processing status identification data of each CDR entity is collected during the visualization process.

[0010] Specifically, the process of generating the standardized call detail record (CDR) sequence includes:

[0011] The original call detail record (CDR) data stream is captured in real time from the data output port of the network element device using a zero-copy memory mapping method, and the network element device identifier field, timestamp field, and sequence number field are extracted from the original CDR data stream.

[0012] Based on the network element device identifier field in the original call detail record (CDR) data stream, the corresponding target parsing template is queried from the dynamic parsing template library configuration data; the target parsing template includes a set of field offsets, a set of field lengths, and a field type mapping table.

[0013] Based on the target parsing template, the original call detail record (CDR) data stream is parsed to obtain an initial set of CDR records;

[0014] For the initial call detail record set, a sliding time window is constructed with terminal identification information and base station identification as the key. Within the sliding time window, the initial call detail record set is sorted according to the sequence number field, out-of-order call detail record records with missing sequence numbers are identified, and the out-of-order call detail record records are corrected by time interpolation according to the timestamp field in the call detail record records with adjacent sequence numbers, so as to generate corrected call detail record records.

[0015] For fragmented call detail records (CDRs), the fragmentation identifier field and the total fragmentation number field are extracted from the fragmented CDRs. The fragmented CDRs belonging to the same original message are grouped according to the fragmentation identifier field, and then reassembled according to the fragmentation sequence number field to generate reassembled CDR records.

[0016] The corrected and reassembled call detail records are merged according to the terminal identification information and session identification field to generate a standardized call detail record sequence.

[0017] Specifically, the process of constructing the association graph between users and base stations includes:

[0018] Extract user-dimensional features, location-dimensional features, behavior-dimensional features, and pricing-dimensional features from standardized call detail record (CDR) sequences;

[0019] The user dimension features, location dimension features, behavior dimension features, and pricing dimension features are concatenated to generate a four-dimensional feature vector, which is then input into a lightweight anomaly detection model. The lightweight anomaly detection model adopts an ensemble learning structure based on gradient boosting decision trees, which contains multiple decision tree base learners. Each decision tree base learner splits nodes according to the feature values ​​in the four-dimensional feature vector, calculates the weight of the leaf node reached by each call single entity on each decision tree base learner, and sums the weights of the leaf nodes of all decision tree base learners in a weighted manner to generate an anomaly risk score.

[0020] Based on the terminal identification information and base station identification in the standardized call detail record (CDR) sequence, an initial association graph is constructed with the terminal identification information as nodes and the base station identification as associated edges. The abnormal risk score is added as a node attribute value to the initial association graph to generate an association graph between users and base stations.

[0021] Specifically, if the abnormal risk score exceeds a preset first threshold, then based on the association graph, abnormal group behavior is identified, and a first risk alarm signal containing an abnormal evidence chain is generated, including:

[0022] Call detail records (CDRs) entities in the standardized call detail record (CDR) sequence whose abnormal risk scores exceed a preset first threshold are marked as risky CDR entities, and the terminal identity information of the risky CDR entities is extracted as risk node identifiers.

[0023] In the user-base station association graph, the corresponding risk node is located according to the risk node identifier. The breadth-first search algorithm is used to start from the risk node and traverse the neighbor nodes of the risk node within the preset hop count range to generate a risk subgraph.

[0024] Obtain the terminal identification information of all neighboring nodes in the risk subgraph, query historical call detail records based on the terminal identification information, and extract multidimensional similarity feature vectors.

[0025] Specifically, if the abnormal risk score exceeds a preset first threshold, then based on the association graph, abnormal group behavior is identified, and a first risk alarm signal containing an abnormal evidence chain is generated, further including:

[0026] The multidimensional similarity feature vector is input into the preset hierarchical clustering model. The hierarchical clustering model calculates the Euclidean distance between risk nodes and neighboring nodes based on the multidimensional similarity feature vector, and divides risk nodes and neighboring nodes into different group clusters according to the Euclidean distance.

[0027] Clusters containing more than a preset group size threshold are marked as suspected groups. Terminal identity information, common base station identifier set, and overlapping call periods of all nodes in the suspected groups are extracted to generate an abnormal evidence chain.

[0028] Based on the abnormal evidence chain, a first risk alarm signal is generated, which includes a list of terminal identification information of suspected gangs, active base station areas, and description fields of abnormal behavior characteristics.

[0029] Specifically, if the first threshold is not exceeded, the rule weights of the lightweight anomaly detection model are automatically fine-tuned based on the acquired historical review labels, including:

[0030] When the abnormal risk score does not exceed the first threshold, the handling status identification data is obtained; the handling status identification data includes a normal call identification, an abnormal warning identification, and a manual review identification.

[0031] From the processing status identifier data, select the call detail record (CDR) entities carrying the manual review identifier, extract the terminal identity identifier information, review result label and review timestamp of the CDR entities carrying the manual review identifier, and generate manual review label data;

[0032] Call detail records (CDRs) entities whose review results are labeled as abnormal and whose review timestamps are within the time window are marked as abnormal review samples, and CDR entities whose review results are labeled as normal and whose review timestamps are within the time window are marked as normal review samples.

[0033] Extract the first historical four-dimensional feature vector corresponding to the reviewed abnormal sample, extract the second historical four-dimensional feature vector corresponding to the reviewed normal sample, input the first historical four-dimensional feature vector and the second historical four-dimensional feature vector into the lightweight anomaly detection model, calculate the hit distribution of the first leaf node of the reviewed abnormal sample on each decision tree base learner in the lightweight anomaly detection model, and calculate the hit distribution of the second leaf node of the reviewed normal sample on each decision tree base learner in the lightweight anomaly detection model.

[0034] Based on the hit distribution of the first leaf node and the hit distribution of the second leaf node, the gradient descent algorithm is used to update the leaf node weight matrix of each decision tree base learner in the lightweight anomaly detection model, thereby completing the automatic fine-tuning of the rule weights.

[0035] Specifically, after constructing the association graph between users and base stations, the method further includes:

[0036] A community detection algorithm is used to calculate the modularity of the association graph and identify the tightly connected subgraphs in the association graph.

[0037] Extract the terminal identity information set and base station identity set contained in the tightly connected subgraph, generate potential user groups based on the terminal identity information set, generate hotspot area sets based on the base station identity set, and associate and store the potential user groups and the hotspot area sets to generate a group area association mapping table;

[0038] When the abnormal risk score in each batch of call detail record (CDR) data stream exceeds a preset first threshold, the group area association mapping table is queried based on the terminal identity information corresponding to the abnormal risk score. If the query is successful, the set of potential user groups and hotspot areas that are matched is added to the first risk alarm signal as context information.

[0039] Specifically, after generating the first risk alarm signal, the method further includes:

[0040] Obtain the historical activity base station sequence of all terminal identification information in the suspected gang;

[0041] Time series pattern analysis was performed on the historical activity base station sequences to identify the base stations and time windows in which the suspected gangs periodically appeared, and to generate the gang's spatiotemporal clustering characteristics.

[0042] The spatiotemporal clustering features of the gang are input into a pre-trained long short-term memory network model in chronological order. The long short-term memory network model is used to predict the hot spots and activity intensity of the suspected gang within a preset time period, thereby generating gang behavior prediction information.

[0043] The gang behavior prediction information is attached to the first risk alarm signal.

[0044] Specifically, after performing streaming data entry and visualization of the standardized call detail record (CDR) sequence at a preset real-time streaming frame rate based on the first risk alarm signal or the optimized rule weight, and collecting the processing status identifier data of each CDR entity during the visualization process, the method further includes:

[0045] Obtain the set of review call detail records (CDRs) entities that contain manual review identifiers in the handling status identifier data, and extract the first CDR entity whose review result label is marked as abnormal and the second CDR entity whose review result label is marked as normal from the set of review CDR entities;

[0046] The four-dimensional feature vector corresponding to the first verified call detail record entity is used as a positive sample, and the four-dimensional feature vector corresponding to the second verified call detail record entity is used as a negative sample to generate an incremental training dataset.

[0047] According to the preset incremental learning cycle, the incremental training dataset is input into the lightweight anomaly detection model. The decision tree base learner of the lightweight anomaly detection model is incrementally updated using an online gradient boosting algorithm to generate an updated lightweight anomaly detection model. The updated lightweight anomaly detection model then replaces the original lightweight anomaly detection model.

[0048] Specifically, the method further includes:

[0049] Calculate the percentage concentration of the first risk abnormal call records corresponding to the first risk alarm signal in the current batch of call record data stream, and determine whether the percentage concentration is greater than a preset second threshold.

[0050] If the percentage concentration is greater than or equal to the second threshold, a traffic shaping control strategy is triggered to dynamically downgrade the detection intensity of non-core business dimensions, while increasing the inference priority of the lightweight anomaly detection model; if the percentage concentration is less than the second threshold, the current detection intensity and rule weights are maintained, and the handling status identification data is synchronously fed back to the dynamic parsing template library.

[0051] A real-time call detail record (CDR) analysis and anomaly detection system, including:

[0052] The standardized parsing module is used to collect raw call detail record (CDR) data streams from different network element devices in real time, obtain configuration data of the dynamic parsing template library, and generate standardized CDR sequences by combining sliding time windows and sequence number verification.

[0053] The anomaly scoring module is used to extract four-dimensional feature vectors from standardized call detail record (CDR) sequences in real time, input them into a pre-trained lightweight anomaly detection model, and obtain an anomaly risk score for each CDR entity.

[0054] The anomaly detection module constructs a user-base station association graph based on terminal identity information and base station identifiers, extracts multi-dimensional similarity feature vectors, divides gang clusters through hierarchical clustering, and generates a first risk alarm signal containing anomaly evidence chains.

[0055] The visualization module uses a distributed message queue to divide the standardized call detail record (CDR) sequence into streaming data batches according to a preset real-time streaming frame rate. It executes streaming data import and front-end visualization rendering in parallel, and monitors write latency and rendering frame rate in real time.

[0056] Compared with the prior art, the beneficial effects of the present invention are:

[0057] 1. This invention proposes a real-time communication call detail record (CDR) parsing and anomaly detection system, and optimizes and improves its architecture, operation steps, and processes. The system has the advantages of simple process, low investment and operating costs, and low production costs.

[0058] 2. This invention proposes a real-time call detail record (CDR) parsing and anomaly detection method. By using a dynamic parsing template library, sliding time window, and sequence number verification, it can efficiently complete the timing correction and session merging of out-of-order and fragmented CDRs, quickly generate standardized CDR sequences, and improve the accuracy and compatibility of CDR parsing. Combining four-dimensional feature vectors and a lightweight anomaly detection model, it achieves real-time scoring of CDR anomaly risks. Relying on the association graph between users and base stations and clustering algorithms, it accurately identifies group anomalies, forms a complete evidence chain, improves the accuracy and response speed of anomaly detection, and effectively supports communication risk prevention and control.

[0059] 3. This invention proposes a real-time parsing and anomaly detection method for communication call detail records (CDRs). This invention supports automatic fine-tuning of model weights, incremental online learning, and self-correction of parsing templates to improve detection adaptability. Through streaming data import, visual interaction, and manual review of data feedback, it balances system real-time performance with ease of processing. It also features strategies such as write latency monitoring, rendering degradation, and traffic shaping to ensure priority processing of core services in high-concurrency scenarios, reduce system load, and improve overall operational stability and robustness. Attached Figure Description

[0060] Figure 1 This is a schematic diagram of the real-time parsing and anomaly detection method for communication call detail records of the present invention;

[0061] Figure 2 This is a flowchart illustrating the principle of the real-time parsing and anomaly detection method for communication call detail records of the present invention.

[0062] Figure 3 This is a diagram of the communication call detail record (CDR) real-time parsing and anomaly detection system of the present invention. Detailed Implementation

[0063] Example 1:

[0064] Please see Figure 1 and Figure 2 The present invention provides an embodiment of a method for real-time parsing and anomaly detection of communication call detail records (CDRs), the method comprising S1 to S4, including the following steps:

[0065] S1: Real-time acquisition of raw call detail record (CDR) data streams from different network element devices, and acquisition of preset dynamic parsing template library configuration data. Based on the corresponding rules of the dynamic parsing template library, combined with sliding time window and sequence number verification, timing correction and session merging are performed on out-of-order and fragmented CDRs in the raw CDR data stream to generate a standardized CDR sequence.

[0066] Furthermore, after performing time-series correction and session merging on out-of-order and fragmented call detail records (CDRs) in the original CDR data stream based on the rules corresponding to the dynamic parsing template library, combined with sliding time windows and sequence number verification, and generating a standardized CDR sequence, the terminal identity information and base station identifiers in the standardized CDR sequence are extracted to construct a time-space mapping matrix. The time-space mapping matrix uses timestamps as row indices and base station identifiers as column indices, and the matrix elements are lists of terminal identity information associated with the corresponding base station identifier under the corresponding timestamp. The time-space mapping matrix is ​​then subjected to frequent pattern mining using a time pattern mining algorithm to extract periodically occurring base station handover sequences. The periodically occurring base station handover sequences are stored as user behavior baselines in a behavior baseline library. When the base station handover sequence in the standardized CDR sequence deviates from the user behavior baseline by more than a preset deviation threshold, a behavior deviation warning signal is generated and attached to the first risk alarm signal. In this embodiment, the deviation threshold is set to 70%. The time pattern mining algorithm is existing technology in this field and is not an inventive solution of this application, so it will not be described in detail here.

[0067] Furthermore, this embodiment takes the core network element equipment of a provincial mobile communication operator as the application scenario. The operator's network simultaneously deploys 4G and 5G network element equipment from multiple vendors, generating more than 5 billion raw call detail records (CDRs) daily, including various types such as voice, SMS, data traffic, VoLTE video calls, and 5G private network services. The raw CDR data is continuously output from each network element equipment in the form of a binary stream, resulting in serious problems such as out-of-order delivery, fragmentation, and duplication. In this embodiment, the system deploys 12 high-performance acquisition servers on key network elements such as the Mobility Management Entity, Service Gateway, Packet Data Gateway, and Online Billing System. Each server runs a zero-copy memory-mapped acquisition program, directly connecting to the data output ports of the network elements, achieving an acquisition bandwidth of 100Gbps, enabling the capture of all raw call detail record (CDR) data streams without loss. Simultaneously, the system loads a dynamic parsing template library from the configuration center. This library pre-configures parsing templates for 36 different network elements, covering all types of network elements used by the operator. The acquired raw CDR data streams are first matched with the corresponding parsing template based on the network element identifier. After parsing, an initial CDR record set containing 28 standard fields is generated. Subsequently, the system constructs a 5-second sliding time window using the user's IMSI code and the base station's ECI code as a key, and records the initial CDRs within the window. The system sorted the records by sequence number and found that about 3.2% of the call detail records (CDRs) had out-of-order issues. For example, a user's call end CDR arrived 2 seconds earlier than the call start CDR. The system corrected this by interpolating the timestamps of adjacent CDRs to correct the call end time. Simultaneously, the system detected that about 1.8% of the CDRs were fragmented CDRs. These CDRs were split into 2-5 fragments due to their large data volume. The system reassembled them based on the fragment identifier and fragment sequence number to reconstruct the complete large-data-volume CDRs. Finally, the system merged the corrected and reassembled CDRs according to the IMSI code and session ID, eliminating about 0.5% of duplicate CDRs, supplementing fields lost during transmission, and ultimately generating a standardized CDR sequence. This standardized CDR sequence has a uniform data format, complete fields, and accurate timing, with data quality improved by more than 95% compared to the original CDRs, fully meeting the requirements for real-time anomaly detection.

[0068] S2: Extract four-dimensional feature vectors of user, location, behavior and tariff from the standardized call detail record (CDR) sequence in real time and input them into a pre-trained lightweight anomaly detection model to calculate the anomaly risk score of each CDR entity and construct the association graph between users and base stations.

[0069] In this embodiment, the four-dimensional feature vector extracted by the system from the standardized call detail record (CDR) sequence contains 64 specific feature items, including 12 user-dimensional features, 16 location-dimensional features, 22 behavior-dimensional features, and 14 tariff-dimensional features. For example, for a specific user CDR entity, the user-dimensional features extracted show that the user's IMSI code corresponds to a historical average monthly active duration of 280 hours, the number belongs to a local mobile number, the package type is a 5G unlimited package, the network duration is 36 months, and there are no historical abnormal records. The location-dimensional features extracted show that the ECI code of the base station connected to the user corresponds to the geographic grid code. The base station number is 320106001056, the base station type is an urban macro base station, the adjacent base station handover frequency is 2 times per hour, and the base station is located in the city center business district. Behavioral features extracted show that this user made 12 calls in one hour, with an average call duration of 8 seconds, a 100% outgoing call rate, peak data usage at 2 AM, and connected to 3 different base stations within one hour. Pricing features extracted show that this user's average cost per call is 0.15 yuan, data usage cost is 0 yuan in one hour, data allowance is 85% remaining, and account balance remains unchanged. The system concatenates these 64 features into a four-dimensional feature vector, which is then input into a light... The model employs a lightweight anomaly detection system, consisting of 50 lightweight decision trees of depth 8. Offline training utilized nearly 1 billion labeled call detail records (CDRs) from the operator over the past six months, including 920 million normal samples and 80 million abnormal samples. These anomalies cover various types such as telecommunications fraud, unauthorized voice calls, data theft, fake IoT cards, and mass-sent fraudulent text messages. The model performs inference calculations on the CDR entities, with each decision tree making node judgments based on feature values. For example, decision tree 1 judges a node as high-risk based on call frequency and caller ID ratio, while decision tree 2 judges a node as high-risk based on location switching frequency. The risk level was determined to be medium. After weighted summation of the tree nodes of all decision trees, the abnormal risk score of the call detail record entity was obtained as 82 points. At the same time, the system used the user's IMSI code as a node, the ECI codes of the three base stations connected to it within one hour as the associated edges, and the abnormal risk score of 82 points as the node attribute to construct a subgraph of association between the user and the base stations. This subgraph was then integrated into the global user-base station association graph. At this point, the global user-base station association graph contained the real-time association relationships of 23 million users and 180,000 base stations across the provincial operator's network, and the node attributes and associated edge information were updated every 5 seconds.

[0070] S3: If the abnormal risk score exceeds the preset first threshold, the abnormal behavior of the gang is identified based on the association graph, and a first risk alarm signal containing the abnormal evidence chain is generated; if it does not exceed the first threshold, the rule weights of the lightweight anomaly detection model are automatically fine-tuned according to the obtained historical review labels.

[0071] In this embodiment, if the system detects that the abnormal risk score of any call detail record (CDR) entity is 82 points, exceeding the preset first threshold of 70 points, the user's IMSI code is immediately marked as a risk node identifier. After locating the risk node in the global user-base station association graph, a breadth-first search algorithm is used to traverse neighbor nodes within a 3-hop range, finding a total of 42 user nodes with association relationships, forming a risk subgraph containing 43 user nodes and 126 base station association edges. The system retrieves the historical CDR records of these 43 users, extracts 8-dimensional similarity feature vectors such as common base station access sequences, overlapping call periods, and similar tariff changes, and inputs them into a hierarchical clustering model. After model calculation, it is found that 12 users have highly similar behavioral characteristics: all of them made high-frequency short-duration calls between 1:00 AM and 3:00 AM, all of them connected to 3 identical base stations in the city center business district, all of them had a 100% outgoing call ratio, all of them had call durations between 5 and 10 seconds, and all of them had no significant changes in their account balances. The typical characteristics of a telecommunications fraud gang led to these 12 users being grouped into the same cluster. Since this cluster contained 12 users, exceeding the preset threshold of a 5-person gang, the system flagged it as a suspected telecommunications fraud gang. Subsequently, the system extracted information from the 12 users' IMSI codes, the ECI codes of the three base stations they accessed, overlapping call times between 1 AM and 3 AM, high-frequency short-duration call records, and no changes in tariffs, compiling a complete chain of abnormal evidence. This chain clearly demonstrated the gang's behavior of making abnormal calls in a fixed area and at fixed times for the past three days. Finally, the system generated a first risk alarm signal, stating, "A suspected telecommunications fraud gang has been discovered, involving 12 users, operating in the city center commercial area near three base stations, exhibiting abnormal behavior of high-frequency short-duration outgoing calls in the early morning, with an abnormal risk score of 82 points, and a complete chain of abnormal evidence attached," and pushed it in real-time to the operator's fraud monitoring platform and anti-fraud center system.

[0072] Furthermore, in this embodiment, if the system detects that the abnormal risk scores of most call detail record (CDR) entities are below 70 points, it is considered normal communication behavior. At this time, the system retrieves manual review tag data from the handling status database for the past 24 hours, filtering out 1260 CDR entities carrying manual review tags. Among these, 89 samples have abnormal review results, and 1171 samples have normal review results. The system extracts the first historical four-dimensional feature vectors of these 89 abnormal review samples and the second historical four-dimensional feature vectors of the 1171 normal review samples, and inputs them into a lightweight anomaly detection model. The model calculates the hit distribution of the abnormal review samples in the first leaf nodes of 50 decision trees. It was found that these samples had a low probability of hitting high-risk leaf nodes in the decision trees, resulting in a low risk score from the model and a risk of false negatives. At the same time, it was calculated that the probability of hitting low-risk leaf nodes for normal samples was high, and the model score was basically accurate. Subsequently, the system used the gradient descent algorithm to update the weights of the leaf nodes of 50 decision trees, increasing the weight of high-risk leaf nodes that are frequently hit by abnormal samples and decreasing the weight of low-risk leaf nodes that are frequently hit by normal samples. The entire fine-tuning process took only 1.2 seconds. After completion, the model's accuracy in detecting hidden abnormal behavior improved by 3.5%, the false negative rate decreased by 2.1%, and the false positive rate remained below 0.3%, with continuous optimization of model performance.

[0073] S4: Based on the first risk alarm signal or the optimized rule weight, the standardized call detail record (CDR) sequence is streamed into the database and visualized at a preset real-time streaming frame rate, and the processing status identification data of each CDR entity is collected during the visualization process.

[0074] Furthermore, based on the first risk alarm signal or the optimized rule weights, the standardized call detail record (CDR) sequence is streamed into the database and visualized at a preset real-time streaming frame rate. The specific process of collecting the processing status identifier data for each CDR entity during visualization includes: the system presets a real-time streaming frame rate of 1 million CDRs per second, which matches the CDR generation speed of the operator's provincial network, ensuring no delay or backlog in data processing; after generating the first risk alarm signal or completing the fine-tuning of the model rule weights, the system processes the standardized CDR sequence in real-time through a distributed streaming framework, synchronously writing standardized CDR data, abnormal risk scores, correlation graph information, alarm information, and other data into a distributed time-series database and a relational database to achieve persistent data storage. The database adopts a sharded and partitioned storage strategy, separating hot and cold data. Hot data is stored in an in-memory database for real-time querying, while cold data is stored in a distributed disk database for historical tracing. The data retention period is 6 months, meeting the operator's data compliance and auditing requirements. Simultaneously, the system pushes the processed call detail records (CDRs), anomaly detection results, correlation maps, and alarm information to a large visualization screen. This screen uses a geographic information system (GIS) as its base map, displaying real-time information such as the distribution of base stations across the province, user communication hotspots, anomaly risk score distribution, locations of abnormal group behavior, and alarm signal statistics. The display interface is divided into four modules: real-time monitoring view, anomaly statistics view, group analysis view, and evidence chain view. Operations and maintenance personnel and security management personnel can intuitively grasp the real-time status and abnormal behavior distribution of the entire network's call detail records through the visualization screen. During the visualization process, the system collects the handling status identifier data for each CDR entity in real time. This handling status identifier data is jointly generated by the business handling system, the manual review platform, and the automatic blocking system. It includes the system-automatically determined normal call identifier, the system-warning anomaly warning identifier, the normal / abnormal identifier after manual review, and the handling identifier after automatic blocking. The system associates and stores the collected handling status identifier data with the corresponding CDR entity.

[0075] Furthermore, in this embodiment, the system processes data such as standardized call detail records (CDR) sequences, anomaly risk scores, user-base station correlation graphs, and suspected fraud gang alarm information at a real-time streaming frame rate of 1 million frames per second, writing them into the ClickHouse distributed time-series database and the MySQL relational database. The data write latency is controlled within 200 milliseconds, supporting millions of data writes and tens of thousands of data queries per second. Simultaneously, a large visual display screen shows the real-time communication status of 180,000 base stations across the province, marking high-risk base stations in red, medium-risk base stations in yellow, and normal base stations in green, clearly marking them on the map. The system displays the activity locations of two suspected fraud gang members, along with their abnormal risk scores, abnormal behavior characteristics, and abnormal evidence chains. Maintenance personnel can directly view the gang's detailed information and evidence chain content on a large screen. During the visualization process, the system collects real-time data on the handling status of each call detail record (CDR) entity. For the aforementioned 12 suspected fraud gang members, the system automatically generates an abnormal warning label. After manual review by personnel on the review platform, the handling status label is updated to the manual review abnormal label. The system then collects this label and immediately adds it to the review label database for the next model weight fine-tuning, forming a complete processing loop.

[0076] Furthermore, during the process of streaming the standardized call detail record (CDR) sequence into the database and displaying it at a preset real-time streaming frame rate, the write latency index for streaming into the database and the rendering frame rate index for visualization are monitored in real time. It is determined whether the write latency index exceeds a preset latency threshold or whether the rendering frame rate index is lower than a preset frame rate threshold. If the write latency index exceeds the latency threshold or the rendering frame rate index is lower than the frame rate threshold, a display degradation strategy is triggered. Specifically, the display degradation strategy involves switching the rendering mode of the visualization from real-time dynamic rendering to static snapshot rendering. The static snapshot rendering generates a visualization snapshot image at a preset time interval and pushes the visualization snapshot image to the front-end visualization interface for display. Simultaneously, the database entry method for the standardized CDR sequence is switched from synchronous writing to asynchronous batch writing. The asynchronous batch writing accumulates multiple streaming data batches into batch data blocks and writes them to the distributed file system all at once. In this embodiment, the latency threshold is set to 200ms, the frame rate threshold is set to 20fps, and the time interval is set to 5s.

[0077] The process of generating the standardized call detail record (CDR) sequence includes:

[0078] S1.1: The original call detail record (CDR) data stream is captured in real time from the data output port of the network element device using a zero-copy memory mapping method, and the network element device identifier field, timestamp field, and sequence number field are extracted from the original CDR data stream;

[0079] Furthermore, call detail record (CDR) collection agents are deployed on network elements at different levels of the communication network, including the core network, access network, and service support system. These network elements include, but are not limited to, mobility management entities, service gateways, packet data gateways, service control points, online billing systems, and offline billing systems. Different network elements are responsible for generating different types of raw CDR data, covering all categories of communication services such as voice calls, SMS sending and receiving, mobile data access, and value-added services. Each CDR collection agent establishes a connection with the data output port of the network element through a standard communication interface, and uses zero-copy memory mapping technology to directly read the CDR data buffer in the network element's memory, avoiding the disk I / O overhead and data latency caused by traditional file reading methods, ensuring that the raw CDR data stream can be captured in real time at the microsecond level. Simultaneously with the initial call detail record (CDR) data stream, the system pulls the latest configuration data from the dynamic parsing template library in real time from the configuration center. This dynamic parsing template library is pre-built and stored, initially pre-configured with standard binary parsing templates corresponding to mainstream manufacturers and various network element device models. The templates are structured and stored according to a four-dimensional index of network element device identifier, manufacturer, model, and software version. Each template contains key configuration information such as the unique identifier of the corresponding network element device, a set of CDR field offsets, a set of field lengths, a field type mapping table, field encoding format, and data verification rules. The dynamic parsing template library supports a dynamic update mechanism that includes manual addition, automatic learning and correction, and version rollback. It can adapt to heterogeneous network element devices from multiple manufacturers, models, and versions in real time, solving the pain point of traditional CDR parsing systems that can only adapt to fixed devices and cannot be compatible with heterogeneous devices from multiple manufacturers. S1.2: Based on the network element device identifier field in the original CDR data stream, the system queries the corresponding target parsing template from the configuration data of the dynamic parsing template library; the target parsing template contains a set of field offsets, a set of field lengths, and a field type mapping table.

[0080] Furthermore, the dynamic parsing template library adopts a key-value pair storage structure. The key is the network element device identifier field, and the value is the corresponding complete parsing rule. The system can complete the matching of network element device identifier and parsing template in microseconds through a hash query algorithm. The target parsing template is a parsing rule customized for a specific manufacturer and model of network element device, which is fully adapted to the call detail record (CDR) data format of the device. The field offset set defines the starting position of each standard CDR field in the original CDR data stream, the field length set defines the number of bytes occupied by each field, and the field type mapping table defines the data type and encoding format of each field. Through these three core configurations, the system can accurately parse each standard field from the unstructured original binary data stream, solving the problem of inconsistent CDR formats for multiple manufacturers and models of network element devices, and realizing standardized parsing of heterogeneous CDR data. The original CDR data stream is in binary format.

[0081] S1.3: Based on the target parsing template, parse the original call detail record (CDR) data stream to obtain the initial CDR record set;

[0082] Furthermore, after completing the acquisition of raw call detail record (CDR) data stream and dynamic parsing template library configuration data, the system uses the network element device identifier as an index to quickly match the corresponding target parsing template. According to the field offset and length defined in the target parsing template, the system parses the raw CDR data stream field by field, converting the originally unstructured binary data stream into an initial CDR record set containing key information such as user number, base station information, call time, call duration, service type, traffic volume, tariff information, sequence number, timestamp, fragment identifier, and total number of fragments.

[0083] Furthermore, the specific process of parsing the original call detail record (CDR) data stream based on the target parsing template is as follows:

[0084] (1) The system first locks the original call detail record (CDR) data stream to be parsed, reads the overall CDR length value recorded in the target parsing template, and determines the total length of the data to be parsed. At the same time, it clarifies the start and end positions of the original CDR data stream to avoid the parsing range from exceeding the valid data range and to prevent omission of valid data content. The overall CDR length of the target parsing template for the current network element device is set to a fixed value of 384 bytes. The system uses this as the boundary and only performs parsing operations on the data stream within the 384-byte length range. Content exceeding this length is judged as invalid padding data and is ignored directly. Content less than this length is marked as incomplete data and temporarily stored in the supplementary queue.

[0085] (2) According to the set of field offsets in the target parsing template, locate the starting reading position of each standard call detail record field in the original call detail record data stream in turn. The set of field offsets is arranged in the order of the generation of the call detail record fields. Each offset represents the byte distance of the corresponding field from the starting position of the data stream. The core field offsets involved in this parsing are set to 0 bytes, 4 bytes, 12 bytes, 20 bytes, 24 bytes, 36 bytes, 44 bytes, 52 bytes, 60 bytes, and 68 bytes in turn. Starting from the first offset, the system moves the reading pointer to the corresponding position in turn to prepare to read the complete content of the current field.

[0086] (3) Read the field length value corresponding to the current field in the target parsing template, determine the number of bytes occupied by the current field according to the field length value, and read the data according to the field length. In this parsing, the length of each field is set to 4 bytes, 8 bytes, 8 bytes, 4 bytes, 12 bytes, 8 bytes, 8 bytes, 8 bytes, 8 bytes, 16 bytes in sequence according to the template. The system reads the corresponding length of binary data continuously from the starting position determined by the field offset as the starting point, which is used as the original data content of the current field.

[0087] (4) Call the field type mapping table in the target parsing template, select the matching data parsing method according to the type identifier corresponding to the current field, and convert the original call detail record data stream in binary form into recognizable and usable structured data. The preset types in the field type mapping table include unsigned integer, string, timestamp, encoded, and floating-point. Perform the corresponding conversion logic for different types. For unsigned integer fields, the system converts binary data into decimal integers according to the big-endian rule. For string fields, the system uses UTF8 encoding format to complete the conversion from binary to text characters. For timestamp fields, the system converts binary data into time values ​​accurate to milliseconds. For encoded fields, the system directly retains the numerical format corresponding to the encoding. For floating-point fields, the system completes the conversion according to the single-precision floating-point rule.

[0088] (5) After the type conversion is completed, the system binds the parsed field content with the corresponding field name and stores it in the temporary storage structure of the current call detail record in a fixed order to form the complete key-value correspondence of the current field. Each time the parsing of a field is completed, the field is written into the temporary storage structure without disrupting the field arrangement order, ensuring that it is consistent with the call detail record field order defined by the communication network.

[0089] (6) Repeat (2)-(5) to process all fields defined in the target parsing template in turn until all standard fields have been parsed and written to the temporary storage structure;

[0090] (7) After all fields are parsed, perform integrity verification on all fields in the current temporary storage structure, check whether the number and total length of the parsed fields meet the requirements of the target parsing template, check whether there are obvious abnormalities in the field content, and after the verification is passed, combine all fields in the temporary storage structure into a complete structured call detail record. The structured call detail record contains all standard fields and corresponding parsed content.

[0091] (8) Add the currently generated structured call detail record to the initial call detail record set, and at the same time clear the current temporary storage structure, release memory space, and prepare to receive the parsing task of the next raw call detail record data stream;

[0092] (9) Complete the structured conversion of all raw call detail record data streams in sequence until all raw call detail record data streams in the current batch are parsed, and finally form an initial call detail record set containing all valid structured call detail record records.

[0093] S1.4: For the initial call detail record set, a sliding time window is constructed with the terminal identification information and the base station identification as the key. Within the sliding time window, the initial call detail record set is sorted according to the sequence number field, out-of-order call detail records with missing sequence numbers are identified, and the out-of-order call detail records are corrected by time interpolation according to the timestamp field in the call detail records with adjacent sequence numbers, and the corrected call detail records are generated.

[0094] Furthermore, the specific logic for constructing the sliding time window and correcting out-of-order call detail records (CDRs) is as follows: the combination key consists of user terminal identification information and base station identification, ensuring that CDRs generated by the same user at the same base station are classified into the same time window, avoiding interference between CDR data from different users and different base stations; the sliding time window adopts a fixed-size, continuously sliding mode, with the window size dynamically adjusted according to network quality. The window expands when network quality is poor and out-of-order calls are severe, and shrinks when network quality is good and out-of-order calls are less frequent. The default 5-second window size balances correction accuracy and processing efficiency, and the window continues in 1-second increments. Slide forward; within the sliding time window, the system sorts the records in ascending order based on the sequence number field. After sorting, it checks whether the sequence numbers are consecutive. If there are skipped, missing, or reversed sequence numbers, the records are identified as out-of-order call detail records (CDRs). For out-of-order CDRs, the system finds the preceding and following CDRs with normal sequence numbers, reads the timestamps of these two records, performs linear time interpolation based on the time difference and sequence number interval, calculates the correct time for the out-of-order CDR, and replaces the original incorrect time, thus completing the timing correction. The corrected CDR time series completely matches the actual communication behavior sequence, solving the out-of-order problem caused by network transmission.

[0095] S1.5: For fragmented call detail records (CDRs), extract the fragment identifier field and the total fragment number field from the fragmented CDRs. Based on the fragment identifier field, group the fragmented CDRs belonging to the same original message, and reassemble them according to the fragment sequence number field to generate reassembled CDR records.

[0096] Furthermore, the specific operation of fragmented call detail record (CDR) reassembly is as follows: When the original communication message data volume is too large, the network element device will split it into multiple fragmented CDRs for transmission. Each fragmented CDR contains the same fragment identifier field, different fragment sequence number fields, and the same total fragment number field. The system first groups the fragmented CDRs according to the fragment identifier field, with CDRs with the same fragment identifier grouped together, representing that they belong to the same original message. Then, the fragmented CDRs within the group are sorted in ascending order of the fragment sequence number field. After sorting, the data of all fragments are spliced ​​together to restore the complete original message data and convert it into a complete CDR record. The reassembly process can completely recover the CDR information lost due to fragmented transmission, ensuring the integrity of the CDR data.

[0097] S1.6: Merge the corrected call detail records and the reassembled call detail records according to the terminal identification information and session identification field to generate a standardized call detail record sequence.

[0098] Furthermore, the specific process of call detail record merging is as follows: The session identifier field is a unique identifier assigned by the network element to the same communication session. The same call, the same data usage, and the same SMS sending and receiving by the same user will all correspond to the same session identifier. Based on the terminal identity information and the session identifier field, the system merges the corrected call detail records and the reconstructed call detail records belonging to the same user and the same communication session. During the merging process, duplicate fields are removed, missing fields are filled in, the data format is unified, and invalid data is removed. After merging, each standardized call detail record sequence corresponds to a complete user communication session, containing all necessary field information, with a unified format, complete data, correct timing, no duplication, and no missing data, meeting the data quality requirements for real-time anomaly detection.

[0099] The process of constructing the user-base station association map includes:

[0100] S2.1: Extract user-dimensional features, location-dimensional features, behavioral-dimensional features, and pricing-dimensional features from the standardized call detail record (CDR) sequence; the user-dimensional features include the historical active duration, number location type, and package type code corresponding to the terminal identification information; the location-dimensional features include the geographic location grid code, base station type, and adjacent base station handover frequency corresponding to the base station identification; the behavioral-dimensional features include call frequency, call duration distribution, peak data usage periods, and the ratio of outgoing to incoming calls; the pricing-dimensional features include the cost per call, remaining data allowance, and account balance fluctuation.

[0101] Furthermore, the extraction rules for the four-dimensional features include: User-dimensional features focus on the user's own attributes; historical active duration is obtained by statistically analyzing the average daily communication duration of the user over the past three months, reflecting the user's communication activity; the number's location type is determined by the first seven digits of the number, categorized into four types: local, out-of-province, overseas, and virtual operator; the package type code is obtained through querying the operator's business system, corresponding to different package levels and service permissions; Location-dimensional features focus on communication location; the geographic location grid code divides the entire province into 100m × 100m grids for precise user location positioning; and the base station type is categorized based on the base station deployment environment and function, with different types... The risks of abnormal behavior differ depending on the type of base station. The frequency of handover between adjacent base stations reflects the user's movement speed; excessively rapid handover may indicate an abnormal mobile terminal. Behavioral characteristics focus on communication operations: call frequency is the number of calls per unit time; call duration distribution is the proportion of calls in different duration intervals; peak data usage periods are the time periods when the user consumes the most data; and the caller-call ratio is the ratio of the number of calls made to the number of calls received. Pricing characteristics focus on the status of charges: single call charge is the charge incurred for a single call; remaining data allowance is the proportion of remaining data to the total data allowance; and account balance change is the change in balance per unit time, with changes indicating abnormal charges.

[0102] S2.2: The user dimension features, location dimension features, behavior dimension features, and pricing dimension features are concatenated to generate a four-dimensional feature vector, and the four-dimensional feature vector is input into a lightweight anomaly detection model; the lightweight anomaly detection model adopts an ensemble learning structure based on gradient boosting decision trees, which contains multiple decision tree base learners. Each decision tree base learner splits nodes according to the feature values ​​in the four-dimensional feature vector, calculates the weight of the leaf node reached by each call single entity on each decision tree base learner, and sums the weights of the leaf nodes of all decision tree base learners in a weighted manner to generate an anomaly risk score;

[0103] Furthermore, the generation process of the four-dimensional feature vector includes: For user-dimensional features, the system performs linear normalization on continuous numerical features, mapping all continuous feature values ​​to the interval between zero and one; for discrete classification features, it performs label mapping, converting different categories into continuous ordered integer identifiers, resulting in 12 effective features for the user dimension; for location-dimensional features, the system performs linear normalization on continuous numerical features and label mapping on discrete features, resulting in 16 effective features for the location dimension; for behavior-dimensional features, the system performs normalization on numerical features and standardization transformation on classification features, resulting in 22 effective features for the behavior dimension; for pricing-dimensional features, the system performs normalization on numerical features and stabilization transformation on state-type features, resulting in 14 effective features for the pricing dimension; after the above processing, the system obtains the preprocessed user-dimensional features. The system consists of preprocessed location dimension features, preprocessed behavior dimension features, and preprocessed pricing dimension features. Using these preprocessed user dimension features, location dimension features, behavior dimension features, and pricing dimension features as input, the system performs feature concatenation in the following order: user dimension features first, location dimension features in the middle, behavior dimension features next, and pricing dimension features last. This concatenation of 12 preprocessed user dimension features, 16 preprocessed location dimension features, 22 preprocessed behavior dimension features, and 14 preprocessed pricing dimension features forms a four-dimensional feature vector with a fixed dimension and uniform structure. This four-dimensional feature vector has a total length of 64 dimensions, with each dimension corresponding to a standardized valid feature. The features are non-overlapping, non-omitted, and maintain a fixed order.

[0104] Furthermore, the lightweight anomaly detection model consists of fifty decision tree-based learners. All decision tree-based learners are constructed in a serial structure. The maximum tree depth of each decision tree-based learner is set to 8 layers, the minimum number of sample splits in the decision tree is set to 20, the minimum number of leaf node samples in the decision tree is set to 10, the learning rate of the model is set to 0.1, and L2 regularization constraints are added during training with a regularization coefficient of 0.01. The weights of the leaf nodes are initialized with a uniform distribution, with the initial weight range being 0 to 1. The input dimension is consistent with the four-dimensional feature vector, and the output is a single continuous numerical value used for anomaly risk score calculation.

[0105] Furthermore, the generation and output process of the anomaly risk score is as follows: The concatenated four-dimensional feature vector is input into the lightweight anomaly detection model. The lightweight anomaly detection model executes the inference calculation of the decision tree-based learners sequentially, including: For each decision tree-based learner, the lightweight anomaly detection model performs node splitting layer by layer starting from the root node. The node splitting is based on the comparison result of the feature value of the corresponding dimension in the four-dimensional feature vector with the node's preset splitting threshold, dividing the feature data into the left child node or the right child node. In this embodiment, the splitting threshold is set to 0.25. The node splitting process strictly follows the limitation of a maximum tree depth of 8 layers. After reaching the 8th layer, the splitting stops and enters the leaf node. Each decision tree-based learner maps the currently input four-dimensional feature vector to a unique leaf node. After reaching the leaf node, the model reads the... The fixed weight values ​​corresponding to the leaf nodes after training are used to obtain the single weight result output by the current decision tree base learner. The inference calculation of 50 decision tree base learners is completed in sequence, and the corresponding leaf node weight values ​​are obtained for each tree. Finally, 50 independent leaf node weight results are obtained. Using the 50 leaf node weight results as input, a weighted summation calculation is performed. The weight coefficients corresponding to each decision tree base learner have been fixed during the model training phase, and the sum of all weight coefficients is 1. The weight of the leaf node output by each decision tree base learner is multiplied by the corresponding weight coefficient, and all multiplication results are accumulated in sequence to obtain a comprehensive score value. The obtained comprehensive score value is linearly scaled to map the comprehensive score value to a value range of 0 to 100, resulting in an abnormal risk score that can be directly used for business judgment.

[0106] S2.3: Based on the terminal identification information and base station identifier in the standardized call detail record sequence, construct an initial association graph with the terminal identification information as nodes and the base station identifier as associated edges, and add the abnormal risk score as a node attribute value to the initial association graph to generate the association graph between users and base stations.

[0107] Furthermore, the specific steps for constructing the association graph include: the initial association graph is stored in a graph database, with nodes serving as user terminal identifiers and each node storing basic user information; the association edges represent the communication connection relationships between users and base stations, with each association edge storing information such as connection time, connection duration, and service type; anomaly risk scores are used as the core attributes of nodes, while key features from the four-dimensional feature vector are added as auxiliary attributes to form complete user node information; the association graph adopts a real-time update mode, immediately updating the corresponding user node attributes and association edge information after each new standardized call detail record (CDR) sequence is generated, ensuring that the association graph data is always consistent with the actual communication status; the global association graph can intuitively display the association relationships between all users and base stations in the network, transforming isolated user data into interconnected graph data.

[0108] If the abnormal risk score exceeds a preset first threshold, then based on the association graph, abnormal group behavior is identified, and a first risk alarm signal containing an abnormal evidence chain is generated, including:

[0109] S3.1: Mark the call detail record (CDR) entities in the standardized call detail record (CDR) sequence whose abnormal risk scores exceed the preset first threshold as risky CDR entities, and extract the terminal identity information of the risky CDR entities as risk node identifiers;

[0110] Furthermore, the system presets a first threshold of 70 points, which is determined jointly by the operator's historical abnormal data statistics, business scenario requirements, and expert experience, thus balancing detection accuracy and false alarm rate. After calculating the abnormal risk score of the call detail record entity in real time, the system immediately compares the abnormal risk score with the first threshold. If the abnormal risk score of any call detail record entity is greater than or equal to 70 points, the call detail record entity is marked as a high-risk call detail record entity, and the user terminal identity information corresponding to the call detail record entity is extracted as a risk node identifier.

[0111] S3.2: In the user-base station association graph, the corresponding risk node is located according to the risk node identifier. The breadth-first search algorithm is used to start from the risk node and traverse all neighbor nodes of the risk node within a preset 3-hop range to generate a risk subgraph. The breadth-first search algorithm is the prior art in this field and is not an inventive solution of this application. It will not be described in detail here.

[0112] Furthermore, the 3-hop range refers to all user nodes that are directly connected to the risk node, indirectly connected once, or indirectly connected twice. This range can cover most of the relationships in the group's communication behavior, avoiding the omission of group members due to an overly small range or the introduction of irrelevant data due to an overly large range.

[0113] Furthermore, the specific logic for generating the risk subgraph is as follows: the breadth-first search algorithm is a graph search algorithm that traverses layer by layer. Starting from the risk node, it first traverses all directly connected neighbor nodes, and then traverses the neighbor nodes of the neighbor nodes, until a preset hop count range is reached. This range can cover the vast majority of communication gang relationships, while avoiding an increase in computational load due to an excessively large traversal range. After the traversal is completed, the risk node, all neighbor nodes, and associated edges form a risk subgraph, which contains all potential gang members associated with high-risk users.

[0114] S3.3: Obtain the terminal identification information of all neighboring nodes in the risk subgraph, retrieve the historical call detail records of these users through the big data query interface, compare the communication behavior characteristics between the risk node and the neighboring nodes, and extract multi-dimensional similarity feature vectors such as common base station access sequence, overlapping call periods, similar tariff change patterns, the same SMS recipients, similar call duration distribution, the same calling and called ratio, and consistent traffic usage periods.

[0115] Furthermore, a shared base station access sequence refers to a list of the same base stations accessed by a risky node and its neighboring nodes within the same time period. Members of a criminal gang often operate in the same area and connect to the same base stations. Overlapping call periods refer to the time periods during which a risky node and its neighboring nodes communicate simultaneously. Fraud gangs often commit crimes collectively during fixed time periods. Similar tariff change patterns refer to the consistent trends in the balance and remaining data allowance of a risky node and its neighboring nodes. Gang operations often do not incur individual tariff consumption. The same SMS recipients refer to risky nodes and their neighboring nodes sending or receiving SMS messages to or from completely identical or highly overlapping phone numbers within the same statistical period. This manifests as a high degree of overlap in the target recipients of SMS messages. Gang members typically use a unified set of SMS outbound or group-messaging targets to create highly similar SMS behavior characteristics. Similar call duration distribution refers to the distribution of call durations between risky nodes and their neighboring nodes within a single statistical period. The proportion of call duration intervals is highly consistent with the overall distribution pattern. The intervals in which call durations are concentrated largely overlap, and the ratio of short to long call durations is basically the same. When gangs commit crimes, they usually use fixed scripts and fixed communication durations, so the distribution of call durations shows obvious convergence characteristics. The same caller-receiver ratio means that the ratio of the number of calls initiated by the risk node to the number of calls received by the neighbor node is consistent within the same statistical period. Both are mainly single callers or single recipients, and the structure ratio of callers and recipients is exactly the same. Gang activities such as telecommunications fraud usually only make one-way outbound calls, so they will show a highly consistent caller-receiver ratio. Consistent traffic usage time periods mean that the time intervals in which the risk node and the neighbor node consume data traffic completely overlap within 24 hours a day. The peak and off-peak traffic usage periods are synchronized. Both start data services at the same time and stop data services at the same time, so the traffic usage time periods show obvious consistency characteristics.

[0116] S3.4: Input the multidimensional similarity feature vector into the preset hierarchical clustering model. The hierarchical clustering model calculates the Euclidean distance between the risk node and its neighboring nodes based on the multidimensional similarity feature vector, and divides the risk node and its neighboring nodes into different group clusters according to the Euclidean distance. The Euclidean distance calculation formula is the prior art in this field and is not an inventive solution of this application, so it will not be described in detail here.

[0117] Furthermore, the hierarchical clustering model calculates the similarity of behavioral characteristics between risk nodes and their neighboring nodes, grouping users with highly similar behavioral characteristics and close relationships into the same cluster, and users with large differences in behavioral characteristics and loose relationships into different clusters.

[0118] Furthermore, the specific steps in S3.4 include:

[0119] (1) Load and initialize the hierarchical clustering model. The hierarchical clustering model adopts a bottom-up aggregated clustering architecture. All parameters are fixed before deployment and are not adjusted during online operation. The hierarchical clustering model uses Euclidean distance as the similarity measure between nodes and the average distance method as the basis for merging clusters. At the same time, a fixed clustering distance threshold of 0.3 is preset. When the average distance between clusters is less than or equal to the clustering distance threshold, merging is performed. When it is greater than the clustering distance threshold, they remain independent. The input of the hierarchical clustering model is a multidimensional similarity feature vector, and the output is a set of divided gang clusters. Each gang cluster contains a set of user node identifiers with highly similar behavior. The average distance method is the prior art in this field and is not an inventive solution of this application. It will not be described in detail here.

[0120] (2) After the hierarchical clustering model is initialized, the set of nodes to be clustered, consisting of risk nodes and all neighbor nodes, is loaded into the hierarchical clustering model. At the same time, a corresponding multidimensional similarity feature vector is bound to each user node to ensure that all nodes have corresponding feature data. The system performs unified standardization processing on all loaded multidimensional similarity feature vectors, mapping all feature values ​​to the range of zero to one, eliminating the dimensional differences between different features.

[0121] (3) After feature standardization is completed, the hierarchical clustering model uses the standardized multidimensional similarity feature vector as a basis to traverse all node combinations in the nodes to be clustered one by one, calculates the Euclidean distance between each pair of nodes in turn, compares the feature differences between nodes dimension by dimension, accumulates them step by step to obtain the complete Euclidean distance value, and stores the Euclidean distance values ​​of all node pairs in a unified manner to form a complete set of distance calculation results.

[0122] (4) After obtaining the Euclidean distance values ​​between all nodes, the initial cluster is constructed. Each user node is divided into an independent cluster, so that the number of initial clusters is consistent with the number of nodes to be clustered. The hierarchical clustering model assigns a unique cluster identifier to each initial cluster, and records the user node identifier and multidimensional similarity feature vector corresponding to the cluster to complete the construction of the initial cluster structure.

[0123] (5) After the initial clusters are constructed, the hierarchical clustering model calculates the average distance between any two clusters in turn based on the obtained Euclidean distance values ​​between nodes and the average distance method. During the calculation, the hierarchical clustering model extracts the Euclidean distance values ​​between all nodes in the two clusters, calculates the arithmetic mean of all Euclidean distance values, and uses the average value as the average distance between the two clusters, thus forming the set of average distances between all clusters.

[0124] (6) The hierarchical clustering model finds the pair of clusters with the smallest average distance in the set of average distances between clusters, and compares the smallest average distance with the preset clustering distance threshold of 0.3. If the smallest average distance is less than or equal to 0.3, the hierarchical clustering model merges this pair of clusters into a new cluster and updates the user nodes and feature information contained in the new cluster. If the smallest average distance is greater than 0.3, the hierarchical clustering model determines that the merging condition is not met and stops the iteration directly. After each merging, the hierarchical clustering model recalculates the average distance between the new cluster and other clusters, updates the set of average distances between clusters, finds the smallest average distance again and performs the threshold comparison, and repeats the iterative merging process until the smallest average distance between all clusters is greater than 0.3, and the iteration process automatically terminates.

[0125] (7) After the iteration and merging terminates, the hierarchical clustering model takes all the currently retained clusters as the final gang clusters. The Euclidean distance between user nodes in each gang cluster does not exceed 0.3, and the communication behavior characteristics remain highly similar, which meets the behavior judgment conditions of abnormal gangs. The hierarchical clustering model outputs all gang clusters and records the user node identifier and the number of nodes in each gang cluster at the same time.

[0126] S3.5: Mark clusters containing more than a preset group size threshold as suspected groups, extract terminal identity information, common base station identifier set and overlapping call periods of all nodes in the suspected group, and generate an abnormal evidence chain. In this embodiment, the preset group size threshold is 5 users.

[0127] Furthermore, the process of generating the abnormal chain of evidence includes:

[0128] (1) Read all the already divided gang clusters, count the number of user nodes in each gang cluster, call the preset gang size threshold, specifically set to 5 user nodes, and compare the number of user nodes in each gang cluster with the gang size threshold one by one. When the number of user nodes in a gang cluster is greater than 5, the gang cluster is marked as a suspected gang. When the number of user nodes in a gang cluster is less than or equal to 5, it is determined that the current cluster does not meet the gang abnormality judgment condition, and no further operation is performed on the cluster.

[0129] (2) For suspected gangs that have been marked, traverse all user nodes within the suspected gang, read the terminal identity information corresponding to each user node in turn, summarize all the terminal identity information read, and perform deduplication on the summarized information to ensure that each terminal identity information is unique and valid, and finally form a list of suspected gang terminal identity information.

[0130] (3) Based on the generated list of suspected gang terminal identifiers, retrieve all base station identifiers that all user nodes have accessed within the same observation period from historical call detail records and user base station association data. Compare and filter all retrieved base station identifiers, retain only the base station identifiers that all user nodes have accessed together, and retain only the base station identifiers that have been accessed together for more than 3 consecutive time windows. Remove the base station identifiers that have been accessed by only some nodes, and finally form a set of common base station identifiers of suspected gangs.

[0131] (4) Based on the list of suspected gang terminal identifiers, retrieve the call time records of all user nodes from the standardized call detail record sequence, compare the call time records of all user nodes segment by segment, find the time intervals in which all nodes are in communication at the same time, retain only the time intervals with an overlap of more than ten minutes, integrate all the time intervals that meet the conditions, and arrange them in chronological order to finally form the overlapping call time periods of suspected gangs.

[0132] (5) The obtained list of suspected gang terminal identifiers, common base station identifiers and overlapping call periods are associated and organized in sequence. First, the overall number of the suspected gang is marked, and then the list of terminal identifiers, common base station identifiers and overlapping call periods are presented in sequence. Clear type labels are added to each type of information so that the correspondence between various types of data is clear and identifiable.

[0133] (6) Integrate all the data that have been organized and labeled to form an abnormal evidence chain that is structurally complete, logically clear and data traceable. Bind and store the generated abnormal evidence chain with the corresponding suspected gang so that the abnormal evidence chain can be directly used for the generation of the first risk alarm signal and manual review and verification.

[0134] S3.6: Based on the abnormal evidence chain, generate a first risk alarm signal containing a list of terminal identification information of suspected gangs, active base station areas, and description fields of abnormal behavior characteristics.

[0135] Furthermore, the first risk alarm signal includes fields such as a list of user terminal identification information suspected to be from the gang, the area of ​​the gang's base station, the type of abnormal behavior, the abnormal risk score, the time of the abnormal occurrence, and the details of the abnormal evidence chain. It will be pushed to multiple business systems such as the operator's security management platform, fraud monitoring platform, and customer service early warning system in real time to realize real-time early warning of abnormal behavior.

[0136] If the first threshold is not exceeded, the rule weights of the lightweight anomaly detection model are automatically fine-tuned based on the obtained historical review labels, including:

[0137] S3.7: When the abnormal risk score does not exceed the first threshold, the handling status identifier data is obtained; the handling status identifier data includes a normal call identifier, an abnormal warning identifier, and a manual review identifier;

[0138] Furthermore, the sources of the status identification data include: normal call identification is automatically generated by the system; abnormal warning identification is generated by a lightweight abnormality detection model; and manual review identification is generated after manual verification by professionals, which is the most reliable labeling data.

[0139] S3.8: Select the call detail record (CDR) entity carrying the manual review identifier from the processing status identifier data, extract the terminal identity identifier information, review result label and review timestamp of the CDR entity carrying the manual review identifier, and generate manual review label data;

[0140] Furthermore, the specific steps in S3.8 include:

[0141] (1) Read all the processing status identifier data collected in the current period, perform integrity verification on all the processing status identifier data read, and confirm that each call detail record entity is accompanied by the corresponding status identifier and does not contain invalid data with missing status identifiers;

[0142] (2) Taking the status identifier data of the completed integrity verification as the processing object, traverse all call detail record entities and identify the corresponding status identifiers one by one, retain only the call detail record entities with manual review identifiers, remove the call detail record entities with normal call identifiers and abnormal warning identifiers, and form an effective data set containing only the call detail record entities related to manual review.

[0143] (3) Taking the effective data set as the processing object, the key fields are extracted sequentially for each call detail record entity. First, the terminal identity information corresponding to each call detail record entity is extracted, then the review result label corresponding to each call detail record entity is extracted, and finally the review timestamp corresponding to each call detail record entity is extracted.

[0144] (4) Synchronously bind the terminal identity information, review result label and review timestamp corresponding to the same call detail record entity to form a one-to-one corresponding data unit, and then summarize all the data units to obtain the initial set of manual review labels.

[0145] (5) Taking the initial set of manual review labels as the processing object, sort all data units according to the order of review timestamps. The sorting step size is based on a complete natural day. At the same time, deduplication verification is performed on all data units. Only one valid record is retained for the same terminal identity under the same review timestamp, and the normalized manual review label data units are obtained.

[0146] (6) All the standardized manual review label data units are uniformly packaged to form manual review label data with complete structure, standardized fields and accurate time sequence.

[0147] S3.9: Mark call detail records (CDRs) with an abnormal review result and a review timestamp within the time window as abnormal review samples, and mark CDRs with a normal review result and a review timestamp within the time window as normal review samples.

[0148] S3.10: Extract the first historical four-dimensional feature vector corresponding to the reviewed abnormal sample, extract the second historical four-dimensional feature vector corresponding to the reviewed normal sample, input the first historical four-dimensional feature vector and the second historical four-dimensional feature vector into the lightweight anomaly detection model, calculate the hit distribution of the first leaf node of the reviewed abnormal sample on each decision tree base learner in the lightweight anomaly detection model, and calculate the hit distribution of the second leaf node of the reviewed normal sample on each decision tree base learner in the lightweight anomaly detection model.

[0149] Furthermore, the specific steps of S3.10 include:

[0150] (1) Read the generated manual review label data. Based on the review result label in the manual review label data, divide all reviewed call detail records entities into review abnormal samples and review normal samples. Call detail records entities with review result labels of abnormal are classified as review abnormal samples, and call detail records entities with review result labels of normal are classified as review normal samples.

[0151] (2) Based on the completed review of abnormal samples, extract the corresponding stored four-dimensional feature vector from the historical call detail record feature library, and determine the four-dimensional feature vector as the first historical four-dimensional feature vector. The four-dimensional feature vector has a fixed length of 64 dimensions. Based on the completed review of normal samples, extract the corresponding stored four-dimensional feature vector from the historical call detail record feature library, and determine the four-dimensional feature vector as the second historical four-dimensional feature vector. The feature vector also has a fixed length of 64 dimensions.

[0152] Furthermore, the historical call detail record (CDR) feature library is a pre-built structured feature library. By parsing standardized CDR sequences in real time, it extracts 64 features across four dimensions—user, location, behavior, and tariff—from each CDR and stores them persistently after normalization. It establishes an index based on user identifier and timestamp, supports millisecond-level historical feature retrieval, and automatically updates incrementally daily and archives and cleans up monthly to ensure that the feature data is complete, consistent, and traceable.

[0153] (3) Load the pre-trained lightweight anomaly detection model, input the first historical four-dimensional feature vector into the lightweight anomaly detection model, perform a complete forward inference calculation on the first historical four-dimensional feature vector, the lightweight anomaly detection model traverses all 50 decision tree-based learners in sequence, and propagates layer by layer from the root node in each decision tree-based learner until it reaches the corresponding leaf node; the lightweight anomaly detection model records the leaf node number of the verification anomaly sample hit on each decision tree-based learner, and combines the leaf node numbers corresponding to all 50 decision tree-based learners in sequence to form the first leaf node hit distribution corresponding to the verification anomaly sample;

[0154] (4) Input the second historical four-dimensional feature vector into the lightweight anomaly detection model. The lightweight anomaly detection model performs forward inference calculation on the second historical four-dimensional feature vector in complete consistency with the verification anomaly sample. It traverses all 50 decision tree-based learners in order, and propagates layer by layer from the root node in each decision tree-based learner until it reaches the corresponding leaf node. The lightweight anomaly detection model records the leaf node number hit by the verification normal sample on each decision tree-based learner. It combines the leaf node numbers corresponding to all 50 decision tree-based learners in order to form the second leaf node hit distribution corresponding to the verification normal sample.

[0155] (5) Store the first leaf node hit distribution and the second leaf node hit distribution independently, and keep the order of the first leaf node hit distribution and the second leaf node hit distribution completely consistent with the order of the decision tree base learners in the lightweight anomaly detection model.

[0156] S3.11: Based on the hit distribution of the first leaf node and the hit distribution of the second leaf node, the gradient descent algorithm is used to update the weight matrix of the leaf nodes of each decision tree base learner in the lightweight anomaly detection model, so that the anomaly risk score of the reviewed anomaly sample increases and the anomaly risk score of the reviewed normal sample decreases, thus completing the automatic fine-tuning of the rule weights. The gradient descent algorithm is the prior art in this field and is not an inventive solution of this application, so it will not be described in detail here.

[0157] Furthermore, the specific steps of S3.11 include:

[0158] (1) Read the generated first leaf node hit distribution and second leaf node hit distribution. The first leaf node hit distribution is the leaf node hit result of the verified abnormal sample on all decision tree base learners. The second leaf node hit distribution is the leaf node hit result of the verified normal sample on all decision tree base learners. Both are consistent with the order of the 50 decision tree base learners in the lightweight anomaly detection model.

[0159] (2) Load the lightweight anomaly detection model that requires weight updates;

[0160] (3) Initialize the gradient descent algorithm parameters, set the algorithm to a fixed learning rate of 0.05, a maximum number of iterations of 50, a weight update convergence threshold of 0.001, use L2 regularization constraint during the algorithm operation, keep the regularization coefficient at 0.01, and complete the fixed configuration of all parameters before the algorithm starts and do not change during the operation.

[0161] (4) Based on the first leaf node hit distribution, locate the leaf node hit by the abnormal sample on each decision tree base learner and extract the current weight value of the node. At the same time, based on the second leaf node hit distribution, locate the leaf node hit by the normal sample on each decision tree base learner and extract the current weight value of the node.

[0162] (5) Calculate the weight update gradient for each decision tree base learner. Calculate the gradient component for increasing the weight based on the hit nodes of the verified abnormal samples and the gradient component for decreasing the weight based on the hit nodes of the verified normal samples. Combine the two types of gradient components to obtain the comprehensive gradient value of the current decision tree base learner. The gradient update formula is the prior art in this field and is not an inventive solution of this application. It will not be elaborated here.

[0163] (6) Multiply the combined gradient value with the set learning rate to obtain the update increment of the weight of each leaf node. Then, add the update increment to the current weight of the corresponding leaf node to complete the weight update of a single iteration. During the update process, regularization constraints are applied simultaneously to avoid abnormal fluctuations in weight.

[0164] (7) After completing a single full iteration, calculate the abnormal risk score of the reviewed abnormal sample and the reviewed normal sample under the current weight, and at the same time calculate the weight update magnitude and compare the weight update magnitude with the weight update convergence threshold.

[0165] (8) Determine the current iteration state. If the weight update magnitude is greater than the weight update convergence threshold and the number of iterations has not reached the maximum number of iterations, continue to execute the next round of iterations. If the weight update magnitude is less than or equal to the weight update convergence threshold or the number of iterations has reached the maximum number of iterations, stop the iteration and complete the update of the leaf node weight matrix of the current decision tree base learner.

[0166] (9) Following the same calculation and update process, update the leaf node weight matrix of all 50 decision tree base learners in sequence to ensure that all decision tree base learners have been optimized;

[0167] (10) Replace the original leaf node weight matrix in the lightweight anomaly detection model with the updated leaf node weight matrix to complete the overall weight update. The updated lightweight anomaly detection model can be directly used for real-time call detail record anomaly detection.

[0168] Following the construction of the user-base station association map, the following is also included:

[0169] A1: The community detection algorithm is used to calculate the modularity of the association graph and identify the tightly connected subgraphs in the association graph;

[0170] Furthermore, the specific steps of A1 include:

[0171] (1) Read the completed user and base station association graph; the association graph uses terminal identity information as nodes, base station identifiers as association edges, and abnormal risk scores as node attribute values, and contains a complete set of nodes and a set of association edges.

[0172] (2) Based on the read association graph, construct computable graph structure data, clarify the adjacent nodes corresponding to each node and the association edge connection relationship between adjacent nodes, and form a standard graph structure that can be directly processed by the community discovery algorithm. The graph structure construction is the existing technology in this field and is not an inventive solution of this application, so it will not be described in detail here.

[0173] (3) Load the community detection algorithm of the label propagation type and perform unified parameter initialization on the community detection algorithm. Assign a unique initial community label to each independent node so that the number of initial communities is consistent with the total number of nodes. Set the maximum number of iterations of the algorithm to 100 times, set the modularity convergence threshold to 0.001, and set the minimum number of community nodes threshold to 5. All parameters remain fixed after initialization and are not dynamically adjusted during operation. The community detection algorithm is the prior art in this field and is not the inventive solution of this application. It will not be described in detail here.

[0174] (4) Start executing the label propagation iteration process. In each iteration, traverse all nodes and update the community label of the current node to the community label that appears most frequently among the adjacent nodes. If multiple identical highest frequency labels appear, randomly select one to complete the label update of all nodes in this round.

[0175] (5) After each round of label propagation, the modularity of the current community division is calculated. The modularity is used to characterize the difference between the tightness of the internal connections and the sparseness of the external connections of the community. The system counts the number of associated edges inside each community and the number of associated edges outside each community to obtain the overall modularity value corresponding to the current community division.

[0176] (6) Calculate the difference between the modularity value obtained in this round and the modularity value in the previous round to obtain the modularity change range, and compare the modularity change range with the preset modularity convergence threshold of 0.001.

[0177] (7) Determine the conditions for continuing the iteration. If the change in modularity is greater than the modularity convergence threshold and the current iteration count has not reached 100, then continue to execute the next round of label propagation and modularity calculation. If the change in modularity is less than or equal to the modularity convergence threshold or the iteration count has reached 100, then stop the iteration and output the final community division result.

[0178] (8) Based on the final community division results, traverse all generated communities, select communities with more than or equal to 5 internal nodes, and determine the communities that meet the conditions as tightly connected subgraphs in the association graph.

[0179] (9) Perform information processing on all tightly connected subgraphs and record the terminal identity information set and base station identity set corresponding to each tightly connected subgraph.

[0180] A2: Extract the terminal identity information set and base station identity set contained in the tightly connected subgraph, generate potential user groups based on the terminal identity information set, generate hotspot area set based on the base station identity set, and associate and store the potential user groups and the hotspot area set to generate a group area association mapping table;

[0181] Furthermore, the specific steps of A2 include:

[0182] (1) Read all tightly connected subgraphs that have been identified. Each tightly connected subgraph consists of a corresponding node and an associated edge. The node carries the terminal identification information, and the associated edge carries the corresponding base station identification information.

[0183] (2) For each tightly connected subgraph, traverse all internal nodes, extract the terminal identity information corresponding to each node one by one, summarize all terminal identity information in the same tightly connected subgraph, and obtain the set of terminal identity information corresponding to the tightly connected subgraph.

[0184] (3) Perform deduplication on the terminal identity information set to ensure that each terminal identity information in the set is unique and valid. The processed terminal identity information set is the potential user group corresponding to the current closely connected subgraph.

[0185] (4) For each tightly connected subgraph, traverse all internal associated edges, extract the base station identifiers corresponding to each associated edge one by one, summarize all base station identifiers in the same tightly connected subgraph, and obtain the base station identifier set corresponding to the tightly connected subgraph.

[0186] (5) Perform deduplication and invalid identifier removal on the base station identifier set to ensure that each base station identifier in the set is unique and valid. The processed base station identifier set is the hot spot area set corresponding to the current closely connected subgraph.

[0187] (6) Bind the potential user groups corresponding to the same tightly connected subgraph to the hot spot area set one by one, so that each potential user group uniquely corresponds to a hot spot area set;

[0188] (7) Organize all the potential user groups and hotspot areas that have completed the binding in a structured manner in order, and establish a stable corresponding relationship record for each binding relationship;

[0189] (8) Store all structured correspondences in a unified manner to generate a group-region association mapping table. Each record in the group-region association mapping table contains a potential user group and a corresponding set of hotspot areas.

[0190] (9) Perform integrity verification on the generated group area association mapping table to ensure that all potential user groups match the corresponding hot spot area set without missing or disordered data. After the verification is passed, store the group area association mapping table into the system.

[0191] A3: When the abnormal risk score in each batch of call detail record (CDR) data stream exceeds the preset first threshold, the group area association mapping table is queried according to the terminal identity information corresponding to the abnormal risk score. If the query is successful, the set of potential user groups and hotspot areas that are matched is added to the first risk alarm signal as context information.

[0192] Furthermore, the specific steps in A3 include:

[0193] (1) Receive the current batch of call detail record (CDR) data streams, perform a complete abnormal risk score calculation on each CDR entity, and form complete CDR entity data containing terminal identity information and abnormal risk score;

[0194] (2) Read the preset first threshold, which is fixed before the system is deployed. The specific value is 70 points, which is used to quickly determine the risk level of the call detail record entity;

[0195] (3) Compare the abnormal risk score of the current call detail record entity with the preset first threshold of 70 points to determine whether the abnormal risk score of the call detail record entity exceeds the first threshold. If the abnormal risk score of the current call detail record entity exceeds 70 points, the system confirms that the call detail record entity meets the high risk triggering condition, and then extracts the terminal identity information corresponding to the call detail record entity as the retrieval basis for querying the group area association mapping table.

[0196] (4) Read the generated and stored group area association mapping table. The group area association mapping table contains multiple records. Each record corresponds to a unique potential user group and a unique set of hotspot areas. The group area association mapping table structure supports retrieval operations based on terminal identity information.

[0197] (5) Using the extracted terminal identity information as the search keyword, perform a matching query in the group area association mapping table, compare the set of terminal identity information contained in each potential user group in the group area association mapping table one by one, and find out if there is a record that completely matches the search keyword.

[0198] (6) Determine the query result. If a potential user group containing the terminal's identity information is found in the group area association mapping table, the query is determined to be a hit. Extract all information of the hit potential user group and simultaneously extract all content of the corresponding hotspot area set bound to the potential user group.

[0199] (7) Organize the extracted potential user group information and hotspot area set information to form a structured context information, which includes the associated groups and activity area characteristics of high-risk users;

[0200] (8) Read the generated first risk alarm signal. The first risk alarm signal contains the original abnormal risk score and the corresponding terminal identification information. The system will add the sorted context information to the first risk alarm signal so that the first risk alarm signal can be supplemented with a complete group and area association background, forming a richer enhanced risk alarm signal.

[0201] (9) If the query results show that the terminal's identity information does not exist in any potential user group, the system determines that the query is a miss and does not perform any information addition operation, but directly keeps the original content of the first risk alarm signal unchanged;

[0202] (10) Output the final enhanced risk alarm signal or the unchanged first risk alarm signal.

[0203] After generating the first risk alarm signal, the process also includes:

[0204] B1: Obtain the historical activity base station sequence of all terminal identification information in the suspected gang;

[0205] Furthermore, the specific steps for B1 include:

[0206] (1) Read the marked suspected gangs, identify all terminal identity information contained in the suspected gangs, and load the preset abnormal gang identification configuration. The configuration contains fixed parameter values, of which the preset number of gang nodes is 50 and the preset feature matching threshold is 0.75. This is used to perform preliminary screening and verification of the terminal identity information of the suspected gangs to ensure that all terminal identity information included in the processing belongs to the suspected gang.

[0207] (2) The system uses the preset number of gang nodes as a benchmark, traverses all terminal identity information related to the suspected gang, compares each terminal identity information with the core feature set of the suspected gang, and calculates the matching value between the two. If the matching value between any terminal identity information and the core feature set reaches or exceeds 0.75, the system confirms that the terminal identity information belongs to the suspected gang; if the matching value is lower than 0.75, the terminal identity information is removed and not included in the processing. After the traversal and comparison are completed, the system summarizes all terminal identity information that meets the conditions and forms a preliminary list of all terminal identity information in the suspected gang.

[0208] (3) Call the preset historical active base station sequence query module. The historical active base station sequence query module has completed the fixed parameter configuration in advance. The specific parameter values ​​are as follows: the single query timeout is set to 5 seconds, the data return batch size is set to 100 records, the historical data time span is set to the last 300 days, the system reads the terminal identification information of each terminal in the preliminary list, sends a query request to the historical active base station sequence query module one by one, and queries the historical active base station data of the terminal identification information in the last 300 days.

[0209] (4) The system judges the response result of each query request. If the query response time is less than or equal to 5 seconds and the number of returned data records does not exceed 100, the system extracts the historical active base station sequence data corresponding to the terminal identity information and includes it in the basic dataset. If the query response time exceeds 5 seconds or the number of returned data records exceeds 100, the system records the query as abnormal and does not extract the historical active base station sequence data corresponding to the terminal identity information to avoid invalid data affecting subsequent processing. After all terminal identity information has been queried, all valid data is summarized to form a basic dataset containing the historical active base station sequence corresponding to all valid terminal identity information.

[0210] (5) The basic dataset is structured and integrated, and fixed parameter values ​​are set to standardize the integration logic. The data deduplication threshold is set to 0.1, the sequence sorting field is set to the base station access timestamp, and the sorting direction is set to ascending order. The system traverses the basic dataset and compares the difference between the access timestamps of two adjacent base stations for each historical active base station sequence corresponding to each terminal identity information. If the difference is less than the deduplication threshold of 0.1, the system determines that the two sequences are duplicate sequences and retains only the base station information that appears for the first time, and removes the subsequent duplicates. If the difference is greater than or equal to the deduplication threshold of 0.1, the system retains the integrity of the two sequences. After integration, the historical active base station sequences corresponding to all terminal identity information are uniformly arranged in the ascending order of the base station access timestamp to form a structured set of historical active base station sequences of all terminal identity information in the suspected gang.

[0211] (6) Perform validity verification on the total set of historical active base station sequences. Set fixed parameter values ​​to standardize the verification logic. The verification batch size is set to 20 terminal identity information, the verification interval is set to 10 seconds, and the maximum interval between timestamps of adjacent base stations is set to 1 hour. The system reads the historical active base station sequence corresponding to each batch one by one, with a verification interval of 10 seconds between each batch, and verifies the continuity of timestamps of base station information in each historical active base station sequence. If the timestamp interval between two adjacent base stations in any historical active base station sequence does not exceed 1 hour, the system determines that the sequence is valid. If the interval exceeds 1 hour, the system determines that the historical active base station sequence is invalid, marks it as an invalid sequence, and removes it. After all verification batches have been verified, the system retains all valid sequences to form the final historical active base station sequence of all terminal identity information in the suspected gang.

[0212] B2: Perform time series pattern analysis on the historical activity base station sequence to identify the base stations and time windows in which the suspected gang periodically appears, and generate the gang's spatiotemporal clustering characteristics.

[0213] Furthermore, the specific steps of B2 include:

[0214] (1) Read the historical activity base station sequence of all terminal identification information of the suspected gang. At the same time, load the preset time series pattern mining algorithm. The time series pattern mining algorithm adopts fixed parameter configuration and does not make any dynamic adjustment during operation. The specific parameter values ​​are as follows: the time series division granularity is set to 15 minutes to divide the continuous timestamps into time windows of fixed duration; the minimum period of periodic mining is set to 1 day and the maximum period is set to 7 days to limit the detection range of terminal periodic features; the common base station occurrence frequency threshold is set to 30 times to determine whether the base station is an associated base station of the gang's common activities; the time window overlap threshold is set to 0.8 to filter the effective time window of the gang's common activities.

[0215] (2) All historical active base station sequences read are preprocessed in a unified manner. According to the preset 15-minute time granularity, the historical active base station sequence corresponding to each terminal is divided into several fixed-duration time windows. Each time window corresponds to a 15-minute time interval. At the same time, the identifiers of all base stations accessed by the terminal in each time window and the corresponding access times are recorded. After the preprocessing is completed, a base station access sequence exclusive to each terminal and divided by time window is formed to ensure that the time sequence division standard of all terminals is completely consistent.

[0216] (3) Start the time series pattern mining algorithm, take the preprocessed terminal base station access sequence as input, and perform periodic analysis on the base station access sequence of each terminal one by one. With 1 day as the minimum period and 7 days as the maximum period, iterate through all possible period values ​​within this range, calculate the repetition of the terminal base station access sequence in each period, and if the repetition of the sequence in any period reaches the preset repetition threshold of 70%, it is determined that the base station access behavior of the terminal has corresponding periodic characteristics. Simultaneously record the periodic information of the terminal and the base station access pattern in the corresponding period. After all terminals have completed the periodic analysis, summarize the periodic characteristics of all terminals to form a terminal periodic set suspected of being a gang. This terminal periodic set contains the periodic information and corresponding access pattern of each terminal.

[0217] (4) Using the generated terminal period set as the processing object, compare the period characteristics of all terminals one by one, filter out the period range where the period characteristics of all terminals completely overlap, determine the overlapping period as the common activity period of the suspected group, after determining the common activity period, traverse all time windows within the common activity period, count the base station identifiers that all terminals jointly access in each time window, and record the frequency of each base station being jointly accessed by all terminals in the corresponding time window, forming a preliminary list of candidate common base stations and corresponding time windows.

[0218] (5) Compare the common access frequency of each base station in the preliminary list with the preset common base station occurrence frequency threshold of 30 times. If the common access frequency of any base station in any time window reaches or exceeds 30 times, then mark the base station and the corresponding time window as candidate common base station and candidate time window; if the common access frequency is less than 30 times, then directly remove the base station and the corresponding time window and do not include them in the screening.

[0219] (6) Further screen all candidate common base stations and candidate time windows, calculate the overlap of all terminals accessing the corresponding candidate common base station within each candidate time window. The overlap calculation standard is the ratio of the number of terminals accessing the candidate common base station within the time window to the total number of suspected gang terminals. The system compares the calculated overlap with the preset time window overlap threshold of 0.8. If the overlap in a candidate time window reaches or exceeds 0.8, the time window is confirmed as a common time window of a suspected gang, and the corresponding candidate common base station is a base station that is suspected to appear periodically in the gang. If the overlap is less than 0.8, the candidate common base station and the corresponding candidate time window are removed to ensure that the screening results meet the characteristics of gang joint activities.

[0220] (7) Summarize all confirmed periodically co-occurring base stations and their corresponding common time windows, organize these data in a structured manner, clarify all common time windows corresponding to each common base station, and all common base stations corresponding to each common time window, to form complete spatiotemporal association data of the gang. Then, encapsulate the spatiotemporal association data of the gang in a unified manner to generate spatiotemporal aggregation features of the gang containing three core contents: common base station identifier, common time window, and common activity cycle features. Among them, the common activity cycle features clearly mark the period range of suspected gang common activities, fully reflecting the spatiotemporal aggregation pattern of the gang.

[0221] B3: Input the spatiotemporal aggregation features of the gang into a pre-trained long short-term memory network model in chronological order, and use the long short-term memory network model to predict the hot spots and activity intensity of the suspected gang within a preset time period, thereby generating gang behavior prediction information;

[0222] Furthermore, the specific steps for B3 include:

[0223] (1) Read the generated spatiotemporal clustering features of the gang, perform normalization preprocessing on the read spatiotemporal clustering features of the gang, sort all common time windows and corresponding common base station identifiers and common activity cycle features in chronological order to ensure that the input data is arranged continuously in the time dimension, and then convert the spatiotemporal clustering features of the gang into a sequence data format that can be directly recognized by the long short-term memory network model. At the same time, verify the data dimension to ensure that the dimension of the converted sequence data is completely matched with the dimension of the model input layer to avoid prediction and inference failure due to dimension mismatch. After the preprocessing is completed, a standardized model input sequence is formed.

[0224] (2) Load the pre-trained Long Short-Term Memory (LSTM) network model. The LTM network model is specifically designed for the prediction of suspected gang behavior. It adopts a fixed three-layer network structure of input layer, hidden layer, and output layer. The connection relationship of each layer is fixed. The model training uses spatiotemporal aggregation feature data of historical abnormal gangs in real communication networks. The parameter configuration during the training process is fixed and cannot be dynamically adjusted. The specific parameter values ​​and structure configuration are as follows: The number of neurons in the input layer is set to 12, which corresponds completely to the dimension of the preprocessed model input sequence. It is responsible for receiving the standardized spatiotemporal aggregation feature sequence data of gangs and synchronously transmitting the data to the hidden layer. The hidden layer contains two layers of LTM units. The number of LTM units in each layer is set to 64. The tanh activation function is used to capture the time dependency relationship of the spatiotemporal aggregation features of gangs and to explore the deep temporal rules of gang activities. The hidden layer has a dropout probability of 0.2, randomly discarding some neuron outputs to prevent overfitting and improve generalization ability. The output layer has 8 neurons, corresponding to preset hotspot region categories and activity intensity levels, and uses the softmax activation function to output the predicted probability and corresponding activity intensity level of each hotspot region. During model training, the Adam optimizer is used with a learning rate of 0.01, 500 iterations, and a batch size of 32. The cross-entropy loss function is used. All training parameters are determined based on historical abnormal gang behavior prediction samples and are not dynamically adjusted. The tanh activation function, softmax activation function, and cross-entropy loss function are existing technologies in this field and are not inventive solutions of this application, and will not be described in detail here.

[0225] (3) After loading, set the model parameters. The preset prediction time period is set to 72 hours. This time period is determined by combining the common activity cycle of the suspected gang and the time characteristics of the communication gang's crimes. It can fully cover the possible activity periods of the gang. The activity intensity classification standard is fixed at 4 levels, namely low intensity, medium intensity, high intensity and high intensity. Each level corresponds to a fixed range of activity frequency, which is used to quantify the activity level of the gang in the predicted hot spot area. The hot spot area screening threshold is set to 0.6, which is used to screen areas whose predicted probability of the model reaches the threshold or above as predicted hot spots, so as to ensure that the predicted hot spots have high credibility.

[0226] (4) The preprocessed standardized model input sequence is input into the loaded long short-term memory network model in chronological order. After receiving the standardized model input sequence, the input layer of the long short-term memory network model synchronously passes it to the first hidden layer. The short-term time dependency of the sequence is captured by 64 long short-term memory units. After processing by the tanh activation function, it is passed to the second hidden layer to further capture the long-term time dependency of the sequence and explore the deep temporal pattern of the spatiotemporal aggregation characteristics of the gang. At the same time, some neuron outputs are randomly discarded through the dropout mechanism. After the hidden layer completes the feature processing, the feature data is obtained and passed to the output layer. The prediction results are output by 8 neurons to obtain the prediction probability and corresponding activity intensity level of each possible region within 72 hours, forming a preliminary prediction result set.

[0227] (5) The preliminary prediction results set is screened and processed to extract areas with a prediction probability greater than or equal to 0.6. These areas are identified as hotspots of suspected gangs in the preset time period. At the same time, the activity intensity level corresponding to each hotspot is extracted to clarify the activity intensity attribution of each hotspot. After the screening is completed, all hotspots and their corresponding activity intensity levels are sorted and organized according to the prediction time order to clarify the prediction time period, prediction probability and activity intensity level corresponding to each hotspot, forming a standardized prediction result set to ensure the continuity and accuracy of the prediction information.

[0228] (6) The completed prediction results are packaged to generate gang behavior prediction information. The gang behavior prediction information includes all hotspot area identifiers of suspected gang activities within 72 hours, the predicted activity time period, activity intensity level and prediction probability corresponding to each hotspot area, and comprehensively reflects the activity trend of the gang. After the packaging is completed, the validity of the generated gang behavior prediction information is verified. Each hotspot area is checked to see if it has a corresponding activity intensity level and prediction probability, and whether each prediction time period is within the preset 72-hour range. The data is confirmed to be without missing, misaligned and redundant. After the verification is passed, the gang behavior prediction information is stored in the system's designated storage area.

[0229] B4: Attach the gang behavior prediction information to the first risk alarm signal.

[0230] After performing streaming data entry and visualization of the standardized call detail record (CDR) sequence at a preset real-time streaming frame rate based on the first risk alarm signal or the optimized rule weight, and collecting the processing status identifier data of each CDR entity during the visualization process, the process further includes:

[0231] C1: Obtain the set of review call detail records (CDRs) entities that contain manual review identifiers in the handling status identifier data, and extract the first CDR entity whose review result label is marked as abnormal and the second CDR entity whose review result label is marked as normal from the set of review CDR entities;

[0232] C2: Use the four-dimensional feature vector corresponding to the first verified call detail record entity as a positive sample and the four-dimensional feature vector corresponding to the second verified call detail record entity as a negative sample to generate an incremental training dataset.

[0233] C3: According to the preset incremental learning cycle, the incremental training dataset is input into the lightweight anomaly detection model. The decision tree base learner of the lightweight anomaly detection model is incrementally updated using the online gradient boosting algorithm to generate the updated lightweight anomaly detection model. The updated lightweight anomaly detection model replaces the original lightweight anomaly detection model.

[0234] In this embodiment, the parameters of the online gradient boosting algorithm are fixed, and the specific values ​​are as follows: the learning rate is set to 0.05, the maximum number of iterations is set to 50, the batch size of samples per iteration is set to 100, the regularization coefficient is set to 0.01, L2 regularization constraint is adopted, parameter verification is performed every 10 iterations, and the convergence threshold is set to 0.001. Specifically, the calculation process of the online gradient boosting algorithm is the prior art in this field and is not an inventive solution of this application, so it will not be described in detail here.

[0235] The method further includes:

[0236] D1: Calculate the percentage concentration of the first risk abnormal call records corresponding to the first risk alarm signal in the current batch of call record data stream, and determine whether the percentage concentration is greater than the preset second threshold.

[0237] Furthermore, the specific steps of D1 include:

[0238] (1) Obtain the total number of call records in the current batch of call record data streams, and obtain the number of first risk abnormal call records carrying the first risk alarm signal in the current batch of call record data streams, and use the ratio of the number of first risk abnormal call records to the total number of call records as the percentage concentration.

[0239] (2) Obtain a preset sliding baseline window, wherein the sliding baseline window contains a historical percentage concentration sequence of the most recent batches of call detail record data streams;

[0240] (3) Calculate the mean and standard deviation of the historical proportion concentration sequence, and generate the second threshold based on the sum of the mean and the standard deviation;

[0241] (4) When the concentration ratio is greater than or equal to the second threshold, a flow peak shaping trigger signal is generated, and the flow peak shaping trigger signal is used as the start condition for executing the flow peak shaping control strategy.

[0242] D2: If the percentage concentration is greater than or equal to the second threshold, a traffic peak shaping control strategy is triggered to dynamically downgrade the detection intensity of non-core business dimensions, while increasing the inference priority of the lightweight anomaly detection model; if the percentage concentration is less than the second threshold, the current detection intensity and rule weight are maintained, and the handling status identification data is synchronously fed back to the dynamic parsing template library.

[0243] Furthermore, the triggered traffic shaping control strategy dynamically downgrades the detection intensity of non-core business dimensions while increasing the inference priority of the lightweight anomaly detection model, including:

[0244] (1) Based on the total data volume of the current batch call detail record data stream and the preset standard processing throughput, calculate the traffic backlog ratio and use the traffic backlog ratio as the execution coefficient to trigger the traffic peak shaping control strategy;

[0245] (2) Obtain a preset core business feature whitelist; the core business feature whitelist includes a list of terminal identity information of key users and a list of location area codes of key base stations;

[0246] (3) Based on the core business feature whitelist, perform feature matching on the standardized call detail record (CDR) sequence, divide the successfully matched standardized CDR sequence into core business CDR streams, and divide the unmatched standardized CDR sequence into non-core business CDR streams;

[0247] (4) For the non-core business call detail record stream, the dynamic degradation is performed, specifically: the extraction frequency of the four-dimensional feature vector in the non-core business call detail record stream is reduced from real-time extraction to batch extraction at a preset first frequency reduction period of 5 seconds, and the calculation accuracy of the abnormal risk score of the non-core business call detail record stream by the lightweight anomaly detection model is reduced from floating-point accuracy to integer accuracy.

[0248] (5) For the core business call detail record (CDR) stream, the inference priority of the lightweight anomaly detection model is increased, specifically: the core business CDR stream is allocated to a preset independent computing thread pool, and the scheduling priority of the independent computing thread pool is set to the highest priority. At the same time, an incrementing sequence priority label is assigned to each CDR entity in the core business CDR stream, so that the lightweight anomaly detection model processes the core business CDR stream carrying the sequence priority label first according to the sequence priority label.

[0249] (6) After completing the inference priority of the dynamic degradation and the improvement of the lightweight anomaly detection model, the first risk anomaly call records of the core business call records flow and the first risk anomaly call records of the non-core business call records flow are merged to generate a merged risk call records set with degradation markers, and the merged risk call records set is used as input to perform the step of identifying gang abnormal behavior based on the association graph.

[0250] Furthermore, the disposal status identification data is synchronously fed back to the dynamic parsing template library, including:

[0251] (1) Obtain the call detail record (CDR) entity that carries the manual review identifier and whose review result is a parsing error from the processing status identifier data, and extract the original CDR data stream fragment of the parsing error CDR entity and the corresponding network element device identifier field;

[0252] (2) Based on the network element device identifier field, query the current parsing template associated with the network element device identifier field from the dynamic parsing template library;

[0253] (3) Parse the original call detail record data stream segment according to the current parsing template, and record the offset position of the field that failed to be parsed;

[0254] (4) Generate a template correction instruction based on the offset position of the field that failed to be parsed. The template correction instruction includes the field offset to be modified, the field length to be modified, and the corrected field type.

[0255] (5) The template correction instruction is sent to the dynamic parsing template library. The dynamic parsing template library updates the corresponding parsing template according to the template correction instruction, generates the updated parsing template, and replaces the current parsing template with the updated parsing template for parsing the raw call detail record data stream collected in real time.

[0256] Example 2:

[0257] Please see Figure 3 Another embodiment of the present invention provides a real-time communication call detail record (CDR) parsing and anomaly detection system, comprising:

[0258] The standardized parsing module 10 is used to collect raw call detail record (CDR) data streams from different network element devices in real time, obtain dynamic parsing template library configuration data, and perform timing correction, reassembly, and session merging on out-of-order and fragmented CDRs by combining sliding time windows and sequence number verification, generate standardized CDR sequences, and support dynamic correction of parsing templates based on parsing error results verified by manual review.

[0259] The anomaly scoring module 20 is used to extract four-dimensional feature vectors of user, location, behavior and tariff from the standardized call detail record (CDR) sequence in real time, input them into the pre-trained lightweight anomaly detection model, calculate the weights of the leaf nodes through multiple decision tree-based learners and sum them up by weight, and obtain the anomaly risk score of each CDR entity through normalization mapping.

[0260] The anomaly identification module 30 constructs a user-base station association graph based on terminal identity information and base station identification, uses anomaly risk score as node attribute, extracts multidimensional similarity feature vectors and divides gang clusters through hierarchical clustering, and generates a first risk alarm signal containing anomaly evidence chain for suspected gangs that exceed the gang size threshold.

[0261] The visualization module 40 uses a distributed message queue to divide the standardized call detail record (CDR) sequence into streaming data batches according to a preset real-time streaming frame rate, and performs streaming data import and front-end visualization rendering in parallel, while monitoring the write latency and rendering frame rate in real time.

[0262] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments under the guidance of the present invention without departing from the spirit and scope of the present invention. All of these variations are within the protection scope of the present invention.

Claims

1. A method for real-time parsing and anomaly detection of communication call detail records, characterized in that, include: The system collects raw call detail record (CDR) data streams from different network element devices in real time, and simultaneously obtains preset dynamic parsing template library configuration data. Based on the corresponding rules of the dynamic parsing template library, combined with sliding time window and sequence number verification, the system performs timing correction and session merging on out-of-order and fragmented CDRs in the raw CDR data stream to generate a standardized CDR sequence. The four-dimensional feature vectors of user, location, behavior and tariff are extracted in real time from the standardized call detail record (CDR) sequence and input into a pre-trained lightweight anomaly detection model to calculate the anomaly risk score of each CDR entity and construct the association graph between users and base stations. If the abnormal risk score exceeds a preset first threshold, the abnormal behavior of the gang is identified based on the association graph, and a first risk alarm signal containing an abnormal evidence chain is generated. If the first threshold is not exceeded, the rule weights of the lightweight anomaly detection model will be automatically fine-tuned based on the obtained historical review labels. Based on the first risk alarm signal or the optimized rule weight, the standardized call detail record (CDR) sequence is streamed into the database and visualized at a preset real-time streaming frame rate, and the processing status identification data of each CDR entity is collected during the visualization process.

2. The real-time parsing and anomaly detection method for communication call detail records as described in claim 1, characterized in that, The process of generating the standardized call detail record (CDR) sequence includes: The original call detail record (CDR) data stream is captured in real time from the data output port of the network element device using a zero-copy memory mapping method, and the network element device identifier field, timestamp field, and sequence number field are extracted from the original CDR data stream. Based on the network element device identifier field in the original call detail record (CDR) data stream, the corresponding target parsing template is queried from the dynamic parsing template library configuration data; the target parsing template includes a set of field offsets, a set of field lengths, and a field type mapping table. Based on the target parsing template, the original call detail record (CDR) data stream is parsed to obtain an initial set of CDR records; For the initial call detail record set, a sliding time window is constructed with terminal identification information and base station identification as the key. Within the sliding time window, the initial call detail record set is sorted according to the sequence number field, out-of-order call detail record records with missing sequence numbers are identified, and the out-of-order call detail record records are corrected by time interpolation according to the timestamp field in the call detail record records with adjacent sequence numbers, so as to generate corrected call detail record records. For fragmented call detail records (CDRs), extract the fragment identifier field and the total fragment number field from the fragmented CDRs. Based on the fragment identifier field, the fragmented CDRs belonging to the same original message are grouped and reassembled according to the fragment sequence number field to generate reassembled CDR records. The corrected and reassembled call detail records are merged according to the terminal identification information and session identification field to generate a standardized call detail record sequence.

3. The real-time parsing and anomaly detection method for communication call detail records as described in claim 2, characterized in that, The process of constructing the user-base station association map includes: Extract user-dimensional features, location-dimensional features, behavior-dimensional features, and pricing-dimensional features from standardized call detail record (CDR) sequences; The user dimension features, location dimension features, behavior dimension features, and pricing dimension features are concatenated to generate a four-dimensional feature vector, which is then input into a lightweight anomaly detection model. The lightweight anomaly detection model adopts an ensemble learning structure based on gradient boosting decision trees, which contains multiple decision tree base learners. Each decision tree base learner splits nodes according to the feature values ​​in the four-dimensional feature vector, calculates the weight of the leaf node reached by each call single entity on each decision tree base learner, and sums the weights of the leaf nodes of all decision tree base learners in a weighted manner to generate an anomaly risk score. Based on the terminal identification information and base station identification in the standardized call detail record (CDR) sequence, an initial association graph is constructed with the terminal identification information as nodes and the base station identification as associated edges. The abnormal risk score is added as a node attribute value to the initial association graph to generate an association graph between users and base stations.

4. The real-time parsing and anomaly detection method for communication call detail records as described in claim 3, characterized in that, If the abnormal risk score exceeds a preset first threshold, then based on the association graph, abnormal group behavior is identified, and a first risk alarm signal containing an abnormal evidence chain is generated, including: Call detail records (CDRs) entities in the standardized call detail record (CDR) sequence whose abnormal risk scores exceed a preset first threshold are marked as risky CDR entities, and the terminal identity information of the risky CDR entities is extracted as risk node identifiers. In the user-base station association graph, the corresponding risk node is located according to the risk node identifier. The breadth-first search algorithm is used to start from the risk node and traverse the neighbor nodes of the risk node within the preset hop count range to generate a risk subgraph. Obtain the terminal identification information of all neighboring nodes in the risk subgraph, query historical call detail records based on the terminal identification information, and extract multidimensional similarity feature vectors.

5. The real-time parsing and anomaly detection method for communication call detail records as described in claim 4, characterized in that, If the abnormal risk score exceeds a preset first threshold, then based on the correlation graph, abnormal group behavior is identified, and a first risk alarm signal containing an abnormal evidence chain is generated, which also includes: The multidimensional similarity feature vector is input into the preset hierarchical clustering model. The hierarchical clustering model calculates the Euclidean distance between risk nodes and neighboring nodes based on the multidimensional similarity feature vector, and divides risk nodes and neighboring nodes into different group clusters according to the Euclidean distance. Clusters containing more than a preset group size threshold are marked as suspected groups. Terminal identity information, common base station identifier set, and overlapping call periods of all nodes in the suspected groups are extracted to generate an abnormal evidence chain. Based on the abnormal evidence chain, a first risk alarm signal is generated, which includes a list of terminal identification information of suspected gangs, active base station areas, and description fields of abnormal behavior characteristics.

6. The real-time parsing and anomaly detection method for communication call detail records as described in claim 5, characterized in that, If the first threshold is not exceeded, the rule weights of the lightweight anomaly detection model are automatically fine-tuned based on the obtained historical review labels, including: When the abnormal risk score does not exceed the first threshold, the handling status identification data is obtained; the handling status identification data includes a normal call identification, an abnormal warning identification, and a manual review identification. From the processing status identifier data, select the call detail record (CDR) entities carrying the manual review identifier, extract the terminal identity identifier information, review result label and review timestamp of the CDR entities carrying the manual review identifier, and generate manual review label data; Call detail records (CDRs) entities whose review results are labeled as abnormal and whose review timestamps are within the time window are marked as abnormal review samples, and CDR entities whose review results are labeled as normal and whose review timestamps are within the time window are marked as normal review samples. Extract the first historical four-dimensional feature vector corresponding to the reviewed abnormal sample, extract the second historical four-dimensional feature vector corresponding to the reviewed normal sample, input the first historical four-dimensional feature vector and the second historical four-dimensional feature vector into the lightweight anomaly detection model, calculate the hit distribution of the first leaf node of the reviewed abnormal sample on each decision tree base learner in the lightweight anomaly detection model, and calculate the hit distribution of the second leaf node of the reviewed normal sample on each decision tree base learner in the lightweight anomaly detection model. Based on the hit distribution of the first leaf node and the hit distribution of the second leaf node, the gradient descent algorithm is used to update the leaf node weight matrix of each decision tree base learner in the lightweight anomaly detection model, thereby completing the automatic fine-tuning of the rule weights.

7. The real-time parsing and anomaly detection method for communication call detail records as described in claim 1, characterized in that, Following the construction of the user-base station association graph, the following is also included: A community detection algorithm is used to calculate the modularity of the association graph and identify the tightly connected subgraphs in the association graph. Extract the terminal identity information set and base station identity set contained in the tightly connected subgraph, generate potential user groups based on the terminal identity information set, generate hotspot area sets based on the base station identity set, and associate and store the potential user groups and the hotspot area sets to generate a group area association mapping table; When the abnormal risk score in each batch of call detail record (CDR) data stream exceeds a preset first threshold, the group area association mapping table is queried based on the terminal identity information corresponding to the abnormal risk score. If the query is successful, the set of potential user groups and hotspot areas that are matched is added as context information to the first risk alarm signal.

8. The real-time parsing and anomaly detection method for communication call detail records as described in claim 5, characterized in that, After generating the first risk alarm signal, the process also includes: Obtain the historical activity base station sequence of all terminal identification information in the suspected gang; Time series pattern analysis was performed on the historical activity base station sequences to identify the base stations and time windows in which the suspected gangs periodically appeared, and to generate the gang's spatiotemporal clustering characteristics. The spatiotemporal clustering features of the gang are input into a pre-trained long short-term memory network model in chronological order. The long short-term memory network model is used to predict the hot spots and activity intensity of the suspected gang within a preset time period, thereby generating gang behavior prediction information. The gang behavior prediction information is attached to the first risk alarm signal.

9. The real-time parsing and anomaly detection method for communication call detail records as described in claim 1, characterized in that, After performing streaming data entry and visualization of the standardized call detail record (CDR) sequence at a preset real-time streaming frame rate based on the first risk alarm signal or the optimized rule weight, and collecting the processing status identifier data of each CDR entity during the visualization process, the process further includes: Obtain the set of review call detail records (CDRs) entities that contain manual review identifiers in the handling status identifier data, and extract the first CDR entity whose review result label is marked as abnormal and the second CDR entity whose review result label is marked as normal from the set of review CDR entities; The four-dimensional feature vector corresponding to the first verified call detail record entity is used as a positive sample, and the four-dimensional feature vector corresponding to the second verified call detail record entity is used as a negative sample to generate an incremental training dataset. According to the preset incremental learning cycle, the incremental training dataset is input into the lightweight anomaly detection model. The decision tree base learner of the lightweight anomaly detection model is incrementally updated using an online gradient boosting algorithm to generate an updated lightweight anomaly detection model. The updated lightweight anomaly detection model then replaces the original lightweight anomaly detection model.

10. The method for real-time parsing and anomaly detection of communication call detail records as described in claim 1, characterized in that, The method further includes: Calculate the percentage concentration of the first risk abnormal call records corresponding to the first risk alarm signal in the current batch of call record data stream, and determine whether the percentage concentration is greater than a preset second threshold. If the percentage concentration is greater than or equal to the second threshold, a traffic shaping control strategy is triggered to dynamically downgrade the detection intensity of non-core business dimensions, while increasing the inference priority of the lightweight anomaly detection model; if the percentage concentration is less than the second threshold, the current detection intensity and rule weights are maintained, and the handling status identification data is synchronously fed back to the dynamic parsing template library.

11. A real-time communication call detail record (CDR) parsing and anomaly detection system, used to implement the real-time CDR parsing and anomaly detection method according to any one of claims 1-10, characterized in that, include: The standardized parsing module is used to collect raw call detail record (CDR) data streams from different network element devices in real time, obtain configuration data of the dynamic parsing template library, and generate standardized CDR sequences by combining sliding time windows and sequence number verification. The anomaly scoring module is used to extract four-dimensional feature vectors from standardized call detail record (CDR) sequences in real time, input them into a pre-trained lightweight anomaly detection model, and obtain an anomaly risk score for each CDR entity. The anomaly detection module constructs a user-base station association graph based on terminal identity information and base station identifiers, extracts multi-dimensional similarity feature vectors, divides gang clusters through hierarchical clustering, and generates a first risk alarm signal containing anomaly evidence chains. The visualization module uses a distributed message queue to divide the standardized call detail record (CDR) sequence into streaming data batches according to a preset real-time streaming frame rate. It executes streaming data import and front-end visualization rendering in parallel, and monitors write latency and rendering frame rate in real time.

Images