Method and system for dynamically identifying abnormal SMS sending based on big data technology

Abstract

The present invention discloses a method and system for dynamically identifying abnormal SMS sending based on big data technology, the method comprising: collecting historical SMS data, and performing aggregate statistics to obtain offline statistical results; performing statistics on real-time SMS data, and merging the real-time statistical results with the offline statistical results; generating an intelligent judgment threshold according to the offline statistical results; judging whether the real-time statistical results meet the requirements according to the judgment threshold, and issuing an early warning for the real-time statistical results that do not meet the requirements. By collecting historical SMS data, feature processing can be performed on the date frequency, the corresponding sending data can be obtained, and fitting training can be performed according to the features, so as to construct a trend prediction model that can predict the number of sendings of different date frequencies, so as to output the predicted number of SMS sendings, and at the same time, the judgment threshold can be calculated in conjunction with the set artificial rule threshold, so as to dynamically identify abnormal SMS sending behavior and accurately and timely issue an early warning.

Description

Technical Field

[0001] The present invention relates to the technical field of SMS data anomaly identification, and in particular to a method and system for dynamically identifying SMS sending anomalies based on big data technology. Background Art

[0002] At present, as SMS is widely used as an important communication method for information transmission between individuals and enterprises, if the communication operator cannot timely learn about the customer's sending status, when SMS sending anomalies occur, it may lead to customer loss, and customers may receive a large number of spam SMS in a short period of time. At present, there are some SMS anomaly detection methods, but they are limited by the insufficiency of the rule base and the limitations of traditional algorithms, and cannot meet the real-time monitoring and the ever-changing SMS sending behavior. Summary of the invention

[0003] In view of this, the present invention proposes a method and system for dynamically identifying abnormal SMS sending based on big data technology, which can dynamically identify abnormal SMS sending behavior and accurately and in real time issue early warnings. The present invention provides the following technical solutions:

[0004] A method for dynamically identifying SMS sending anomalies based on big data technology, the method comprising:

[0005] Collect historical SMS data, aggregate and count the historical SMS data, and obtain offline statistical results;

[0006] Count real-time SMS data and merge the real-time statistics with offline statistics;

[0007] Generate intelligent judgment thresholds based on offline statistical results;

[0008] Whether the real-time statistical results meet the requirements is determined based on the judgment threshold, and an early warning is issued for real-time statistical results that do not meet the requirements.

[0009] Furthermore, the method of collecting historical SMS data, performing aggregate statistics on the historical SMS data, and obtaining offline statistical results includes:

[0010] Collect and import historical SMS data into the big data warehouse;

[0011] Use the offline processing framework Spark+Hive to design Spark jobs, aggregate and count historical SMS data, and obtain offline statistical results;

[0012] Store offline statistical results in the database.

[0013] Furthermore, the method of counting real-time SMS data and merging the real-time statistical results with the offline statistical results includes:

[0014] Use the real-time processing framework Flink+Kafka to count real-time SMS data;

[0015] Write real-time SMS data into Kafka and use Flink's data source to connect to Kafka.

[0016] Use Flink to process real-time SMS data;

[0017] Use Flink to group and aggregate the processed real-time SMS data to obtain real-time statistical results.

[0018] The real-time statistical results are stored in the database and merged with the offline statistical results data.

[0019] Furthermore, the method for generating an intelligent judgment threshold according to offline statistical results includes:

[0020] Calculate the statistical characteristics of different date frequencies based on offline statistical results;

[0021] Perform time feature extraction on different date frequencies to capture the sending patterns of different date frequencies;

[0022] Based on the time series model, the statistical features and time features of different date frequencies are input for fitting training to obtain the sending trend prediction model of different date frequencies. The model outputs the predicted value of the number of SMS messages sent at the next date frequency.

[0023] Set the artificial rule threshold and configure the threshold coefficient matrix based on the artificial rule threshold;

[0024] The judgment threshold is calculated through the threshold coefficient matrix and the predicted value.

[0025] Furthermore, the method of determining whether the real-time statistical results meet the requirements according to the judgment threshold and issuing an early warning for the real-time statistical results that do not meet the requirements includes:

[0026] Obtain the real-time SMS data from Flink statistics and compare it with the judgment threshold;

[0027] If the judgment threshold is reached, an early warning message is generated and pushed to the early warning message topic of Kafka;

[0028] Create an early warning project and consume Kafka's early warning message topic to push early warning messages.

[0029] Furthermore, the method of calculating the judgment threshold through the threshold coefficient matrix and the predicted value includes:

[0030] Take the product of the predicted value and the threshold coefficient of a certain date frequency in the threshold coefficient matrix to calculate the judgment threshold.

[0031] The present invention further discloses a system for dynamically identifying SMS sending anomalies based on big data technology, comprising:

[0032] The offline data processing module is used to collect historical SMS data, aggregate and count the historical SMS data, and obtain offline statistical results;

[0033] Real-time data processing module, used to count real-time SMS data and merge the real-time statistical results with the offline statistical results;

[0034] A judgment threshold calculation module is used to generate an intelligent judgment threshold according to offline statistical results;

[0035] The early warning judgment module is used to determine whether the real-time statistical results meet the requirements based on the judgment threshold, and to issue an early warning for real-time statistical results that do not meet the requirements.

[0036] The present invention further discloses a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0037] The present invention further discloses an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above method when executing the program.

[0038] The present invention further discloses a computer program product, comprising a computer program, wherein the computer program implements the above method when executed by a processor.

[0039] According to the technical solution of the present invention, by collecting historical SMS data, statistical features and time features of different date frequencies can be processed to obtain corresponding sending pattern data, and fitting training can be performed based on the statistical features and time features to build a trend prediction model that can predict the number of text messages sent at different date frequencies, so that the predicted number of text messages sent can be output, and at the same time, in conjunction with the set artificial rule threshold, the judgment threshold can be calculated, so that abnormal SMS sending behavior can be dynamically identified, and early warnings can be issued accurately and in real time, making important contributions to protecting the rights and interests of users and the interests of communication operators. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] For the purpose of illustration and not limitation, the present invention is now described in conjunction with the embodiments of the present invention and the accompanying drawings, in which:

[0041] Figure 1 It is a flowchart of a method for dynamically identifying SMS sending anomalies based on big data technology in an embodiment of the present invention;

[0042] Figure 2It is a schematic diagram of the system structure for dynamically identifying SMS sending anomalies based on big data technology in an implementation manner of the present invention;

[0043] Figure 3 is a schematic diagram of the structure of an electronic device in an embodiment of the present invention;

[0044] Figure 4 is a schematic diagram of the offline processing flow of historical SMS data in an embodiment of the present invention;

[0045] Figure 5 It is a schematic diagram of a threshold determination process for real-time SMS data in an embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the implementation mode of the present application will be clearly and completely described below in conjunction with the drawings in the implementation mode of the present application. Obviously, the described implementation mode is only a part of the implementation mode of the present application, not all the implementation modes. Based on the implementation mode in the present application, all other implementation modes obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present application.

[0047] It should be noted that the terms "first", "second", etc. in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0048] In the present application, the terms "upper", "lower", "left", "right", "front", "back", "top", "bottom", "inner", "outer", "middle", "vertical", "horizontal", "lateral", "longitudinal" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings. These terms are mainly used to better describe the present application and its implementation methods, and are not used to limit the indicated devices, elements or components to have a specific orientation, or to be constructed and operated in a specific orientation.

[0049] In addition, some of the above terms may be used to express other meanings in addition to indicating orientation or positional relationship. For example, the term "on" may also be used to express a certain dependency or connection relationship in some cases. For those of ordinary skill in the art, the specific meanings of these terms in this application can be understood according to the specific circumstances. In addition, the term "plurality" should mean two or more than two.

[0050] It should be noted that, in the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other. The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0051] For real-time data streams such as SMS data, if you want to achieve real-time monitoring, a rule-based processing method is usually adopted. However, the update of the rule base is always later than the sending method of abnormal short lines, so it cannot meet the real-time detection and the ever-changing SMS sending behavior.

[0052] This implementation method discloses a method and system for dynamically identifying SMS sending anomalies based on big data technology.

[0053] refer to Figure 1 , a method for dynamically identifying SMS sending anomalies based on big data technology, including:

[0054] S100: Collect historical SMS data, and aggregate and count the historical SMS data to obtain offline statistical results.

[0055] refer to Figure 1 and Figure 4 For the processing of historical SMS data, this implementation adopts the offline processing framework Spark+Hive to collect and count, and identifies the number of SMS sent by each mobile phone number based on the mobile phone number dimension. Specifically:

[0056] S110: Collect and import historical SMS data into a big data warehouse.

[0057] S120: Use the offline processing framework Spark+Hive to design a Spark job, aggregate and count the historical SMS data, and obtain offline statistical results; specifically, the aggregate statistics are to count the sending frequency and the number of SMS messages sent by the dimension of the mobile phone number where SMS is sent. Hive is used as a data warehouse storage, a data table is created in Hive, and the historical SMS data is loaded into the data table created in Hive. The designed Spark job batch processes the historical SMS data in the data table, and counts the sending frequency and the number of SMS messages sent by the dimension of the mobile phone number where SMS is sent.

[0058] S130: Store the offline statistical results in a database. Specifically: create a storage table in the database, and the data table stores the offline statistical results.

[0059] refer to Figure 1 , Figure 4 and Figure 5 , S200: count the real-time SMS data, and merge the real-time statistical results with the offline statistical results.

[0060] For the real-time SMS data, this implementation adopts the real-time processing framework Flink+Kafka to count, and uses the same mobile phone number dimension as the historical data to identify the amount of SMS data sent by each mobile phone number. Specifically:

[0061] S210: Use the real-time processing framework Flink+Kafka to count real-time SMS data.

[0062] S220: Write the real-time SMS data into Kafka and use Flink's data source to connect to Kafka. At the same time, in order to ensure that the real-time SMS data can be consumed accurately, relevant consumer information needs to be configured.

[0063] S230: Process the real-time SMS data through Flink, that is, parse, clean and convert the real-time SMS data for subsequent statistical analysis.

[0064] S240: The processed real-time SMS data is grouped and aggregated through Flink to obtain real-time statistical results. The window operation of the Flink framework is applied, that is, the SMS data is grouped and aggregated according to the time window. For example, one minute is used as a time window to cluster and count the number of sent SMS messages. For example, if the frequency of SMS messages sent is 100 / min, the frequency of SMS messages sent can be used as the judgment data for whether the real-time SMS data is abnormal.

[0065] S250: The real-time statistical results are stored in a database and merged with the offline statistical results.

[0066] In step S130, a storage table is created, and the real-time statistical results and the offline statistical results are stored in the storage table together. After the merging is completed, an index is created to facilitate fast retrieval and use of the model.

[0067] In this embodiment, a method of real-time SMS data processing using Spark Streaming instead of Flink framework is also disclosed, which can also realize real-time processing of SMS data.

[0068] After obtaining the offline statistical results of the historical SMS data and the real-time statistical data of the real-time SMS data, it is necessary to determine whether the real-time statistical data is abnormal. The identification method of this embodiment further includes:

[0069] S300: Generate intelligent judgment thresholds based on offline statistical results. Offline statistical results can reflect the patterns and rules of SMS sending in the past period of time. However, for real-time rule changes or continuous adjustments of special dates, it is inaccurate to rely solely on offline statistical results to judge abnormal real-time SMS data. Therefore, this implementation incorporates model intelligent prediction and artificial rules in the generation of judgment thresholds. Specifically:

[0070] S310: Calculate the statistical characteristics of different date frequencies based on the offline statistical results. Specifically:

[0071] Preprocess the offline statistical results reflecting the historical SMS data, replace the abnormal noise data and perform differential transformation to obtain a stable offline statistical result sequence, aggregate statistics according to the date frequency of monthly, weekly, daily and hourly sending, and calculate the statistical features: upper limit, lower limit, mean, median, etc., to obtain the statistical features of different frequencies:

[0072] ;

[0073] ;

[0074] ;

[0075] in, It indicates the lower quartile of the historical SMS data when they are arranged in ascending order, which means that the data below this point accounts for 25% of the total data. It represents the upper quartile of historical SMS data when they are arranged in ascending order, which means that the data below this point accounts for 75% of the total data. represents the interquartile range, which is calculated as: - , Indicates the lower limit statistical value of SMS sending data, that is, the lower limit standard for identifying SMS sending anomalies. Indicates the upper limit statistical value of SMS sending data, that is, the upper limit standard used to identify SMS sending anomalies. Indicates the value to be replaced, which can be the median value, the upper or lower statistical value, The value after replacement.

[0076] S320: Extract time features for different date frequencies to capture the sending patterns of different date frequencies. For example, for a monthly sending frequency, its time features may be the weeks of each month, and similarly, the days of each week, the hours of each day, whether it is a working day, whether it is a holiday, etc. The sending patterns of different date frequencies can be obtained by extracting time features.

[0077] S330: Based on the time series model, the statistical features and time features of different date frequencies are input for fitting training to obtain a sending trend prediction model of different date frequencies, which outputs the predicted value of the number of SMS messages sent at the next date frequency. The time series model can also be replaced by a similar time series model, which is not limited in this implementation. Specifically:

[0078] The statistical features and time features of each date frequency are input into the time series model TimesNet in a sliding window manner. After multiple rounds of iterative training and verification evaluation, the monthly, weekly, daily, and hourly trend prediction models are obtained. A time sliding window N is defined. Then, the input is defined as M=[m1,m2,…,m n ], where m n Represents the feature vector of the nth input, and the output is K=[m n+1 ], the current window model input and output can be expressed as:

[0079] K = TimesNet(M);

[0080] Exemplarily, the sending trend prediction model constructed according to different date frequencies may include:

[0081] Monthly frequency model: The input data is the statistical characteristics and time characteristics of historical SMS data in the past N months, and the output is the predicted value of SMS sending data in the next month;

[0082] Weekly frequency model: The input data is the statistical characteristics and time characteristics of the historical SMS data of the past N weeks, and the output is the predicted value of SMS sending data in the next week;

[0083] Daily frequency model: The input data is the statistical characteristics and time characteristics of historical SMS data in the past N days, and the output is the predicted value of SMS sending data for the next day;

[0084] Time-frequency model: The input data is the statistical characteristics and time characteristics of historical SMS data in the past N hours, and the output is the predicted value of SMS sending data in the next hour.

[0085] S340: Setting an artificial rule threshold, and configuring a threshold coefficient matrix based on the artificial rule threshold.

[0086] Manual rules can be formulated based on specific business knowledge and experience, and abnormal behaviors can be detected by defining rules. For example, on special holidays, different types of companies will send a large number of policy-related or business-related text messages. In this case, manual rules are required to participate in threshold correction. In addition, different threshold coefficients can be given for detection of different date frequencies. In step S330, the predicted value output by the model is obtained based on the prediction of historical sending data. When making an abnormal alarm, an alarm threshold needs to be manually formulated to determine the final result. When setting the threshold of the manual rule, in addition to the threshold for the number of text messages sent, it also includes rules such as whether to alarm when the number of text messages sent is 0, the number of consecutive alarms, etc. These manual rules are uniformly configured as a threshold coefficient matrix T:

[0087] ;

[0088] Among them, t n Indicates the nth rule threshold for each date frequency.

[0089] S350: Calculate the judgment threshold using the threshold coefficient matrix and the predicted value.

[0090] According to step S330 and step S340, the calculation method of the judgment threshold is as follows:

[0091] .

[0092] As can be seen from the above, by building a sending trend prediction model and combining it with manual rules, we can make full use of the automation capabilities of the sending trend prediction model and the expertise of manual rules, making the detection of abnormal behavior in SMS sending more comprehensive and accurate, and can also be flexibly adjusted and optimized according to special dates and other needs.

[0093] After obtaining the judgment threshold, it is stored in Redis. When making subsequent warning judgments, the threshold can be directly compared to determine whether to issue a warning.

[0094] S400: Determine whether the real-time statistical results meet the requirements according to the judgment threshold, and issue an early warning for the real-time statistical results that do not meet the requirements.

[0095] S410: Obtain the real-time SMS data collected by Flink and compare it with the judgment threshold.

[0096] Since SMS sending data is generated in real time, after completing the acquisition and statistical processing of real-time SMS data according to the date frequency, the real-time statistical results are compared with the judgment threshold stored in Redis, and the difference between the real-time SMS data and the judgment threshold can be quickly and accurately identified.

[0097] S420: If the judgment threshold is reached, a warning message is generated and pushed to the warning message topic of Kafka.

[0098] S430: Create an early warning project and consume the early warning message topic of Kafka to push the early warning message. For example, create an early warning project through Spring Boot, and consume the early warning message topic in Kafka through the project. Further, write a function to process the early warning message and push the early warning message, such as sending SMS, email, APP message push, etc.

[0099] Since historical SMS data is stored after the threshold judgment of a certain date frequency is completed, the big data warehouse used for storage will gradually be filled over time, so it is necessary to regularly back up and clean up historical data to avoid performance degradation caused by excessive data accumulation. At the same time, it is also necessary to use the CDH management platform to monitor the server's CPU, memory and other resources to avoid being occupied by other processes during model training and threshold judgment.

[0100] Compared with the existing SMS warning method, the present invention has the following advantages and effects:

[0101] Massive data: Big data technology can be used to process massive amounts of messages. Depending on the scale of the big data service cluster, early warning of TB or PB-level data volumes can be achieved.

[0102] Accuracy: Statistics are based on massive amounts of historical SMS data, and the data will be pre-processed such as cleaning, which can make the statistical results of sending behavior more accurate and provide more accurate results for the model.

[0103] Dynamic threshold: The warning threshold can be dynamically adjusted based on real-time data, which can adapt to the ever-changing abnormal behavior of SMS sending and reduce the false alarm rate.

[0104] Real-time: It can collect and analyze data in real time, and based on the judgment threshold, it can issue early warning notifications in seconds, thus protecting the rights and interests of users and the interests of communication operators in a timely manner.

[0105] Flexible and scalable: The early warning project developed based on Java can be easily integrated with other systems and its functions can be expanded according to actual needs.

[0106] refer to Figure 2 This embodiment further provides a system for dynamically identifying SMS sending anomalies based on big data technology, including:

[0107] The offline data processing module 21 is used to collect historical SMS data, aggregate and count the historical SMS data, and obtain offline statistical results; specifically includes: collecting historical SMS data and importing it into a big data warehouse; using the offline processing framework Spark+Hive to design Spark jobs, aggregate and count the historical SMS data, and obtain offline statistical results; storing the offline statistical results in a database.

[0108] The real-time data processing module 22 is used to count the real-time SMS data and merge the real-time statistical results with the offline statistical results; specifically: the real-time processing framework Flink+Kafka is used to count the real-time SMS data; the real-time SMS data is written into Kafka, and the Flink data source is used to connect Kafka; the real-time SMS data is processed through Flink; the processed real-time SMS data is grouped and aggregated through Flink to obtain real-time statistical results; the real-time statistical results are stored in the database and merged with the offline statistical results.

[0109] The judgment threshold calculation module 23 is used to generate an intelligent judgment threshold based on offline statistical results; specifically includes: calculating the statistical features of different date frequencies based on offline statistical results; extracting time features of different date frequencies to capture the sending patterns of different date frequencies; based on the time series model, inputting the statistical features and time features of different date frequencies for fitting training to obtain a sending trend prediction model for different date frequencies, which outputs a predicted value of the number of SMS messages sent on the next date frequency; setting an artificial rule threshold, and configuring a threshold coefficient matrix based on the artificial rule threshold; and calculating the judgment threshold through the threshold coefficient matrix and the predicted value.

[0110] The early warning judgment module 24 is used to determine whether the real-time statistical results meet the requirements according to the judgment threshold, and to issue an early warning for the real-time statistical results that do not meet the requirements; specifically: obtain the real-time SMS data counted by Flink and compare it with the judgment threshold; if the judgment threshold is reached, generate an early warning message and push it to the early warning message topic of Kafka; create an early warning project, and consume the early warning message topic of Kafka to realize the push of early warning messages.

[0111] Figure 3 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, such as Figure 3 As shown, the electronic device 50 includes: a processor 501 (processor), a memory 502 (memory) and a bus 503;

[0112] Among them, the processor 501 and the memory 502 communicate with each other through the bus 503; the processor 501 is used to call the program instructions in the memory 502 to execute the method for dynamically identifying SMS sending anomalies based on big data technology provided by the above-mentioned method implementation methods.

[0113] This embodiment provides a non-transitory computer-readable storage medium, which stores computer instructions. The computer instructions enable a computer to execute the method for dynamically identifying SMS sending anomalies based on big data technology provided by the above-mentioned method implementation methods.

[0114] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various storage media that can store program codes.

[0115] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0116] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0117] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for dynamically identifying SMS sending anomalies based on big data technology, characterized in that: The method comprises: Collect historical SMS data, aggregate and count the historical SMS data, and obtain offline statistical results; Count real-time SMS data and merge the real-time statistics with offline statistics; Generate intelligent judgment thresholds based on offline statistical results; including: calculating statistical features of different date frequencies based on offline statistical results; extracting time features of different date frequencies to capture sending patterns of different date frequencies; based on the time series model, inputting statistical features and time features of different date frequencies for fitting training to obtain a sending trend prediction model of different date frequencies, which outputs a predicted value of the number of SMS messages sent at the next date frequency; setting an artificial rule threshold, and configuring a threshold coefficient matrix based on the artificial rule threshold; calculating a judgment threshold through the threshold coefficient matrix and the predicted value; Whether the real-time statistical results meet the requirements is determined based on the judgment threshold, and an early warning is issued for real-time statistical results that do not meet the requirements.

2. The method for dynamically identifying SMS sending anomalies based on big data technology according to claim 1 is characterized in that: The method of collecting historical SMS data, performing aggregate statistics on the historical SMS data, and obtaining offline statistical results includes: Collect and import historical SMS data into the big data warehouse; Use the offline processing framework Spark+Hive to design Spark jobs, aggregate and count historical SMS data, and obtain offline statistical results; Store offline statistical results in the database.

3. The method for dynamically identifying SMS sending anomalies based on big data technology according to claim 1 is characterized in that: The method for counting real-time SMS data and merging the real-time statistical results with the offline statistical results includes: Use the real-time processing framework Flink+Kafka to count real-time SMS data; Write real-time SMS data into Kafka and use Flink's data source to connect to Kafka. Use Flink to process real-time SMS data; Use Flink to group and aggregate the processed real-time SMS data to obtain real-time statistical results. The real-time statistical results are stored in the database and merged with the offline statistical results data.

4. The method for dynamically identifying SMS sending anomalies based on big data technology according to claim 3 is characterized in that: The method of determining whether the real-time statistical results meet the requirements according to the judgment threshold and issuing an early warning for the real-time statistical results that do not meet the requirements includes: Obtain the real-time SMS data from Flink statistics and compare it with the judgment threshold; If the judgment threshold is reached, an early warning message is generated and pushed to the early warning message topic of Kafka; Create an early warning project and consume Kafka's early warning message topic to push early warning messages.

5. The method for dynamically identifying SMS sending anomalies based on big data technology according to claim 1 is characterized in that: The method for calculating the judgment threshold value through the threshold coefficient matrix and the predicted value includes: Take the product of the predicted value and the threshold coefficient of a certain date frequency in the threshold coefficient matrix to calculate the judgment threshold.

6. A system for dynamically identifying SMS sending anomalies based on big data technology, characterized in that: include: The offline data processing module is used to collect historical SMS data, aggregate and count the historical SMS data, and obtain offline statistical results; Real-time data processing module, used to count real-time SMS data and merge the real-time statistical results with the offline statistical results; A judgment threshold calculation module is used to generate an intelligent judgment threshold according to offline statistical results; It is also used to calculate the statistical characteristics of different date frequencies based on offline statistical results; Extract time features for different date frequencies to capture the sending patterns of different date frequencies; based on the time series model, input the statistical features and time features of different date frequencies for fitting training to obtain a sending trend prediction model for different date frequencies, which outputs the predicted value of the number of SMS messages sent on the next date frequency; Set the artificial rule threshold and configure the threshold coefficient matrix based on the artificial rule threshold; The judgment threshold is calculated through the threshold coefficient matrix and the predicted value; The early warning judgment module is used to determine whether the real-time statistical results meet the requirements based on the judgment threshold, and to issue an early warning for real-time statistical results that do not meet the requirements.

7. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method described in any one of claims 1 to 5 is implemented.

9. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 5.

Images