Short message filtering processing method and device based on Hash matching algorithm

Abstract

The invention belongs to the technical field of short message processing, and provides a short message filtering processing method and device based on a Hash matching algorithm. The method comprises the following steps: carrying out multiple kinds of preprocessing on short message contents of a to-be-processed short message, and carrying out pre-auditing on the to-be-processed short message after multiple kinds of preprocessing; adopting a bidirectional maximum matching word segmentation method to perform bidirectional matching word segmentation on the short message content of the to-be-processed short message passing the pre-auditing to obtain a word segmentation result of the to-be-processed short message, and determining a character distribution condition in a preset hash table so as to determine the complaint rate of the to-be-processed short message; and according to a comparison result of the determined complaint rate of the to-be-processed short message and a specified threshold value, keyword matching is carried out based on a keyword matching rule, and multi-level evaluation calculation is determined to further carry out multi-level control. According to the invention, more effective multi-level control is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of short message data processing, and in particular to a short message filtering processing method and device based on a hash matching algorithm. Background Art

[0002] With the rapid development of society and economy, everything is interconnected, and all walks of life are inseparable from the Internet. Many "black industries" use the convenient means of SMS to wantonly spread "bad SMS content", causing very serious harm to the people and the country. Nowadays, the makers of "bad SMS" have begun to process SMS in various ways, making it difficult for operators to distinguish some SMS and difficult to monitor and intercept them in time.

[0003] Mobile SMS is popular among users for its mobility, low price, and convenient sending and receiving. It has become an important way for people to transmit information and communicate. However, due to the development of Internet mobile communication technology, SMS messages involving fraud, deception, and behaviors that are not in line with social customs (such as gambling) have also emerged. These are collectively referred to as "spam SMS". The phenomenon of spam SMS covers every corner of the world and has become a global problem, which has also had a serious negative impact on operators. In order to maintain the normal order of SMS services, ensure the smooth flow of normal business and information security, operators are forced to take effective measures to solve the problem of spam SMS filtering.

[0004] The existing technology basically adopts two main methods for filtering spam text messages. One is to process according to the overall content of the received text messages. Usually, methods such as keyword matching, matching the credit database, and extracting keyword features are used. It is impossible to identify the content of the text messages, and there are often omissions in the identification. The second is to judge the user behavior and intercept according to the user's sending frequency and the rules of the sending number. This method has limitations and can be easily circumvented by the makers of "bad news". In addition, due to the strong subjectivity of keyword matching and the complexity of text message content, it is easy to cause false interception, which affects normal users in text message filtering management. However, with the various text message contents of fraudsters, it is impossible to supplement keywords in time. If you want to add or supplement keywords at any time, it will greatly increase the operating cost of the operator's server and affect the level of short message processing by the system.

[0005] Therefore, it is necessary to provide an efficient and practical SMS filtering processing method to solve the above problems. Summary of the invention

[0006] The present invention aims to provide a method and device for SMS filtering based on a hash matching algorithm to solve the technical problems in the prior art that the SMS content cannot be effectively identified and that the interception is incorrect due to interception based only on the user's sending frequency and the regularity of the sending number. The technical problem to be solved by the present invention is achieved through the following technical solution.

[0007] The first aspect of the present invention provides a method for filtering and processing short messages based on a hash matching algorithm, comprising: Collect the SMS messages to be processed, perform multiple preprocessing on the SMS contents of the SMS messages to be processed, and perform pre-examination and review on the SMS messages to be processed after the multiple preprocessing; adopt a two-way maximum matching word segmentation method to perform two-way matching word segmentation on the SMS contents of the SMS messages to be processed that have passed the pre-examination and review, and obtain the word segmentation results of the SMS messages to be processed; According to the obtained word segmentation results of the SMS to be processed, the character distribution in the preset hash table is determined to determine the complaint rate of the SMS to be processed; according to the comparison result of the determined complaint rate of the SMS to be processed with the specified threshold, and based on the keyword matching rules, keyword matching is performed to determine multi-level evaluation calculations for further multi-level control.

[0008] The second aspect of the present invention proposes a text message filtering and processing device based on a hash matching algorithm, which executes the text message filtering and processing method based on the hash matching algorithm described in the first aspect of the present invention, and the text message filtering and processing device includes: a collection module, which collects text messages to be processed, performs multiple preprocessing on the text message content of the text messages to be processed, and performs pre-examination and review on the text messages to be processed after the multiple preprocessing; a word segmentation module, which adopts a two-way maximum matching word segmentation method, performs two-way matching word segmentation on the text message content of the text messages to be processed that have passed the pre-examination and review, and obtains the word segmentation result of the text messages to be processed; a first determination module, which determines the character distribution in a preset hash table according to the obtained word segmentation result of the text messages to be processed, so as to determine the complaint rate of the text messages to be processed; a second determination module, which determines a multi-level evaluation calculation according to the comparison result of the determined complaint rate of the text messages to be processed with a specified threshold, and performs keyword matching based on a keyword matching rule to further perform multi-level control.

[0009] The third aspect of the present invention provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the SMS filtering processing method based on the hash matching algorithm described in the first aspect of the present invention.

[0010] The fourth aspect of the present invention provides a computer-readable medium having a computer program stored thereon, and when the computer program is executed by a processor, the multi-level cloud phone application security management and control method described in the first aspect of the present invention is implemented.

[0011] The embodiments of the present invention include the following advantages: Compared with the prior art, the present invention pre-screens and reviews the SMS messages to be processed after multiple pre-processing, and performs multiple pre-processing on the SMS messages to be processed, so as to effectively and meticulously eliminate redundant information to obtain more refined SMS content, and facilitate the use of a two-way maximum matching algorithm for word segmentation operations; a two-way maximum matching word segmentation method is used to perform two-way matching word segmentation on the SMS content of the SMS messages to be processed that have passed the pre-screening and review, to obtain the word segmentation results of the SMS messages to be processed, to determine the character distribution in a preset hash table to determine the complaint rate of the SMS messages to be processed, and to perform keyword matching based on the comparison result of the determined complaint rate of the SMS messages to be processed and a specified threshold value, and to determine multi-level evaluation calculations to further perform multi-level control, so as to control the risky numbers through the SMS content.

[0012] In addition, by calculating the corresponding risk assessment values ​​for the specific conditions of different regions and adding a hierarchical assessment switch, the control level can be flexibly switched, thereby achieving more effective multi-level control. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a flowchart of an example of a method for filtering and processing short messages based on a hash matching algorithm of the present invention; Figure 2 It is a schematic diagram of an application example of bidirectional matching in the SMS filtering processing method based on the hash matching algorithm of the present invention; Figure 3 It is a schematic diagram of an example of multi-level control in the SMS filtering processing method based on the hash matching algorithm of the present invention; Figure 4 It is a structural block diagram of a method and device for short message filtering based on a hash matching algorithm of the present invention; Figure 5 is a schematic structural diagram of an electronic device embodiment according to the present invention; Figure 6 It is a schematic diagram of the structure of a computer-readable medium embodiment according to the present invention. DETAILED DESCRIPTION

[0014] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0015] In view of the above problems, the present invention proposes a method and device for SMS filtering based on a hash matching algorithm. The method pre-examines and reviews the SMS to be processed after multiple pre-processing, and performs multiple pre-processing on the SMS to be processed, which can effectively and meticulously eliminate redundant information to obtain more refined SMS content, and facilitate the use of a two-way maximum matching algorithm for word segmentation; the two-way maximum matching word segmentation method is used to perform two-way matching word segmentation on the SMS content of the SMS to be processed that has passed the pre-examination and review, and obtain the word segmentation result of the SMS to be processed, and the hash matching algorithm is used to configure the hash key value for the word segmentation string to determine the character distribution in the preset hash table, so as to determine the complaint rate of the SMS to be processed, and the comparison result of the determined complaint rate of the SMS to be processed with the specified threshold value is compared, and keyword matching is performed based on the keyword matching rule, and multi-level evaluation calculations are determined to further perform multi-level control, and the risk number is controlled through the SMS content.

[0016] In addition, by calculating the corresponding risk assessment values ​​for the specific conditions of different regions and adding a hierarchical assessment switch, the control level can be flexibly switched, thereby achieving more effective multi-level control.

[0017] Example 1 Refer to the following Figure 1 , Figure 2 and Figure 3 , the contents of the present invention will be described in detail.

[0018] Figure 1 It is a flowchart of an example of the method for filtering and processing short messages based on a hash matching algorithm of the present invention.

[0019] like Figure 1 As shown, in step S101, the SMS messages to be processed are collected, the SMS contents of the SMS messages to be processed are pre-processed in a variety of ways, and the SMS messages to be processed after the various pre-processing are pre-examined and reviewed.

[0020] In a specific application example, the short messages to be processed are collected from a short message center of an operator (eg, operator a).

[0021] The following pre-processing is performed on the SMS to be processed: removing useless characters, interference characters, etc., and converting similar characters or similar words.

[0022] By removing useless characters, interfering characters, similar characters, etc., the content of the SMS to be processed is made more refined, and finally similar characters are converted to improve the accuracy of word segmentation.

[0023] Specifically, the reserved electronic address types in the text message content are extracted: account numbers (such as WeChat numbers, QQ numbers), short links, bank account information, payment accounts, code numbers, domain names, URLs, IP addresses and other electronic address types. These electronic address types are combined with keyword information to determine illegal information.

[0024] Furthermore, based on the extracted electronic address type and keyword information, a pre-screening template is used to pre-screen and review a variety of pre-processed text messages (specifically matching risk information, which includes illegal URLs, illegal keywords, and fraudulent content) to determine the text messages that can be released directly.

[0025] Specifically, a pre-screening template is preset, and the preset template includes a matching part corresponding to the electronic address type and keyword information. The pre-screening template includes daily conversation templates, notification templates, announcement templates, news templates, etc. New pre-screening templates can be added at any time according to business needs. Text messages containing such content are normal text messages and are directly released. For example, text messages containing content such as "The weather is very good today" and "Dad, remember to come home for dinner tonight" can be directly released.

[0026] It should be noted that the above is only described as an optional example and should not be understood as a limitation to the present invention.

[0027] Next, in step S102, a two-way maximum matching word segmentation method is used to perform two-way matching word segmentation on the SMS content of the SMS to be processed that has not passed the preliminary review, and obtain the word segmentation result of the SMS to be processed.

[0028] Specifically, a bidirectional maximum matching word segmentation method is adopted. According to the existing semantic recognition technology (which may include numbers, English, common Chinese characters, etc.), firstly, a forward match is used from left to right to match continuously. If the corresponding text is matched in the forward direction, the matched text is merged into an independent string. Then, a reverse match is performed by a reverse match from right to left. If the corresponding text is matched in the reverse direction, the matched text is merged into an independent string. Moreover, when the forward matching result (specifically, the string obtained by the match) and the reverse matching result (specifically, the string obtained by the match) are consistent, these strings are retained, and the word segmentation result of the short message to be processed is obtained. The above-mentioned bidirectional maximum matching word segmentation method has a large granularity, and each word separated contains a large number of characters, which can effectively increase the accuracy of word segmentation and ensure the rationality of subsequent threshold judgment.

[0029] For forward and reverse matching, such as Figure 2As shown, a text message to be processed containing "Mobile banking scan and pay to enjoy instant discount, limited quantity, first come first served" is matched forwardly and matched reversely. The result of the forward match is "Mobile|Bank|Scan|Pay|Enjoy|Instant Discount|Limited Quantity|,|First Come First Served". The result of the reverse match is ""Mobile|Bank|Scan|Pay|Enjoy|Instant Discount|Limited Quantity|,|First Come First Served". Moreover, the forward match result (specifically the string obtained by matching) and the reverse match result (specifically the string obtained by matching) are consistent. That is, the word segmentation result of the text message to be processed is obtained.

[0030] It should be noted that the above is only described as an optional example and should not be understood as a limitation to the present invention.

[0031] Next, in step S103, the character distribution in the preset hash table is determined according to the obtained word segmentation result of the short message to be processed, so as to determine the complaint rate of the short message to be processed.

[0032] A preset hash table is established based on the illegal keywords and the electronic address type. The preset hash table includes the electronic address type, multiple illegal keywords (i.e., multiple illegal keywords), such as fraud-related keywords, gang-related keywords, and keywords representing bad social behavior. The character string of each illegal keyword is used as a key (for example, represented by key), and the character string of each illegal keyword is configured with a value (for example, represented by value) to form a key-value combination to establish the preset hash table.

[0033] For example, a preset hash table is established in the data management system. The specific illegal keywords and email address types are used as keys, and the illegal keywords, email address types, and the corresponding number of occurrences are used as values. Then, all word segmentation combinations (key, value) are used to establish the preset hash table.

[0034] Specifically, the word segmentation result and the electronic address type of the short message to be processed obtained in the above step S102 are used as keys to match the preset hash table, and the number of occurrence points of each word in the short message to be processed in the preset hash table is obtained to calculate the complaint rate.

[0035] It should be noted that the pre-configured hash table contains the key value of the segmentation string of the SMS content, that is, the complaint rate of the SMS can be obtained through the distribution of the segmentation string in the hash table.

[0036] Specifically, assume that the content of the SMS to be processed is a segmented string of N: , The number of times each word segmentation string appears in the hash table is: ; Assume the total number of words in the preset hash table is: .

[0037] Use the following expression to calculate the complaint rate of pending SMS messages:

[0038] in, Indicates the complaint rate of the SMS to be processed, represented by the number of occurrences of the same keyword string as in the preset hash table; Respectively represent the first segmentation string, the second segmentation string, the third segmentation string, ..., the Nth segmentation string in the content of the SMS to be processed; t1, t2, t3, ..., t N Indicates the number of occurrences corresponding to the first segmentation string, the second segmentation string, the third segmentation string, ..., the Nth segmentation string; It represents the total number of occurrences of the word segments representing the keywords in the preset hash table, where S represents the total number of keywords in the hash table.

[0039] In a specific implementation, the SMS content to be processed is segmented, and the keywords contained in the SMS to be processed are obtained as feature information of the SMS to be processed, and a hash key value is assigned to each keyword segment. By querying a preset hash table, the proportion of all keyword key values ​​of the SMS to be processed to the total number of segments representing the keywords in the preset hash table is calculated to obtain the complaint rate of the SMS to be processed.

[0040] It should be noted that the above is only described as an optional example and should not be understood as a limitation to the present invention.

[0041] Next, in step S104, according to the comparison result between the determined complaint rate of the to-be-processed short message and the specified threshold, keyword matching is performed based on keyword matching rules, and multi-level evaluation calculation is determined to further perform multi-level control.

[0042] Further, the calculated complaint rate is compared with a specified threshold to determine whether to proceed to the subsequent steps.

[0043] It should be noted that, in the present invention, the specified threshold is configured after identification training with a large number of spam text message samples and non-spam text message samples.

[0044] For the determination process of the above-mentioned specified threshold, first, a large amount of historical SMS data, including spam SMS and non-spam SMS, is collected, and the data samples are ensured to be representative. The data set is divided into a training set and a test set. Among them, the collected SMS data is preprocessed and segmented, and the electronic address type is extracted. The segmentation and electronic address type are used as risk features, and the collected SMS data is annotated with complaint situations to divide risk SMS and non-risk SMS (i.e., positive samples and negative samples) to establish a training set. According to the experience of spam SMS filtering business, an intermediate value is selected as the initial threshold (i.e., the initial value of the specified threshold). The threshold is used to judge the positive and negative nature of the sample, specifically the positive sample (i.e., positive class) and the negative sample (i.e., negative class). The collected SMS data is trained and verified using the machine learning algorithm logistic regression to optimize the initial threshold and finally obtain the optimal threshold.

[0045] The specific process of determining the optimal threshold is as follows: according to the initial threshold, the complaint rate model is trained using the logistic regression algorithm and training samples, and then the complaint rate model is evaluated using the test set. The true positive rate (TPR, also known as the true positive rate TPR) and the false positive rate (FPR, also known as the false positive rate FPR) are calculated based on the prediction results determined by the complaint rate model, and the difference between the true positive rate TPR and the false positive rate FPR is calculated. Then, the threshold is continuously adjusted, the complaint rate model is trained, and a new true positive rate TPR and false positive rate FPR are obtained. When the difference between the true positive rate TPR and the false positive rate FPR is the largest, the threshold is the optimal threshold.

[0046] The optimal threshold is the threshold that maximizes TPR-FPR, which is expressed by the following expression:

[0047] Among them, Optimal Threshold represents the optimal threshold that maximizes the difference between the true positive rate (TPR, also known as the true positive rate TPR) and the false positive rate (FPR, also known as the false positive rate FPR) of the model prediction results determined by the complaint rate model; θ represents different thresholds, TPR is the true positive rate of the model prediction results, which represents the proportion of samples that are correctly classified as positive among all samples that are actually positive, the true positive rate TPR is the false positive rate of the model prediction results, and the false positive rate FPR represents the proportion of samples that are incorrectly classified as positive among all samples that are actually negative.

[0048] When the calculated complaint rate is less than the specified threshold, the SMS to be processed enters the keyword matching module, which means it needs to confirm again whether to enter risk management. When the calculated complaint rate is less than the specified threshold and when the keyword matching module determines that the risk keyword is hit, it enters risk management, that is, enters the risk area management module for multi-level risk management.

[0049] In a specific implementation, when the calculated complaint rate is less than a specified threshold and when the keyword matching module determines that the risk keyword is not hit, the SMS to be processed is released, that is, it does not enter risk control.

[0050] When the calculated complaint rate is greater than or equal to the specified threshold, the risk control will be directly entered, that is, the risk area control module will be entered to carry out multi-level risk control.

[0051] In another specific implementation, based on the comparison result of the calculated complaint rate of the to-be-processed SMS with a specified threshold value (specifically, the calculated complaint rate is less than the specified threshold value), keyword matching is performed based on keyword matching rules (specifically, matching is performed according to the keyword matching rules using a keyword matching module, and when the SMS content hits the keyword rules, the risk area management and control module is entered. When the SMS content does not hit the keyword rules, the current SMS corresponding to the SMS content is released, and a multi-level evaluation calculation is determined to further perform multi-level control.

[0052] Specifically, by sorting out the complaint data and interception data, we calculate the proportion of risk numbers in each province, city, district (county), and use them as the risk assessment values ​​of each province, city, district (county), specifically the first-level risk assessment value, the second-level risk assessment value, and the third-level risk assessment value.

[0053] The process of determining the risk assessment value specifically includes determining the following parameters: The amount of illegal data = the amount of complaint data received + the amount of intercepted data; The first-level risk assessment value = the amount of illegal data in the province / the amount of illegal data in the country; The second-level risk assessment value = the amount of illegal data in the city / the amount of illegal data in the province; The third-level risk assessment value = the amount of illegal data in the district / the amount of illegal data in the city.

[0054] The above-mentioned provincial illegal data volume, national illegal data volume, provincial illegal data volume, municipal illegal data volume, and district illegal data volume are obtained by counting the data within a specified historical time period (for example, within 6 months, within 12 months, and within 24 months).

[0055] For example, taking LN province as an example, through the province code, city code, and district (county) number, it can be known that the amount of complaint data and interception data is M in total, of which city a covers S items, so the risk assessment value of city a is P=S / M.

[0056] Risk assessments are conducted in different regions based on the calculated first-level risk assessment values, second-level risk assessment values, and third-level risk assessment values. They can be divided at the provincial level. After the risk assessment, the provincial level can continue to be calculated to conduct a city-level assessment, and further to conduct a district-level assessment. Through calculations, risk assessments can be conducted at three levels. Risk assessments are switched at each level through switches at different levels, thus realizing a multi-level risk assessment process with strong adjustability.

[0057] For each level of risk, we will judge and evaluate the switching layer by layer. First, we will analyze whether the province to which the current SMS number belongs is a high-risk province (the judgment process corresponding to the province will start at this time). If it is a high-risk province, we will enter the interception template. If it is not a high-risk province, we will continue to analyze whether the city to which it belongs is a high-risk city (the judgment process corresponding to the city will be switched at this time). If it is a high-risk city, we will enter the interception template. If it is not a high-risk city, we will continue to analyze whether the district to which it belongs is a high-risk area (the judgment process corresponding to the district will be switched at this time). If the district to which it belongs is a high-risk area, we will enter the interception template. If the district to which it belongs is not a high-risk area, it will be released directly. For details, please refer to Figure 3 .

[0058] By adopting multi-level risk assessment and coordinating different levels of switches for risk assessment switching, multi-level regional risk management is effectively achieved and the efficiency of the SMS filtering system is improved.

[0059] It should be noted that the above is only described as an optional example and should not be understood as a limitation to the present invention.

[0060] In addition, the drawings are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the drawings do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0061] Compared with the prior art, the present invention pre-screens and reviews the SMS messages to be processed after multiple pre-processing, and performs multiple pre-processing on the SMS messages to be processed, so as to effectively and meticulously eliminate redundant information to obtain more refined SMS content, and facilitate the use of a two-way maximum matching algorithm for word segmentation operations; a two-way maximum matching word segmentation method is used to perform two-way matching word segmentation on the SMS content of the SMS messages to be processed that have passed the pre-screening and review, to obtain the word segmentation results of the SMS messages to be processed, to determine the character distribution in a preset hash table to determine the complaint rate of the SMS messages to be processed, and to perform keyword matching based on the comparison result of the determined complaint rate of the SMS messages to be processed and a specified threshold value, and to determine multi-level evaluation calculations to further perform multi-level control, so as to control the risky numbers through the SMS content.

[0062] In addition, by calculating the corresponding risk assessment values ​​for the specific conditions of different regions and adding a hierarchical assessment switch, the control level can be flexibly switched, thereby achieving more effective multi-level control.

[0063] Example 2 The following are embodiments of the device of the present invention, which can be used to implement the method embodiments of the present invention. For details not disclosed in the device embodiments of the present invention, please refer to the method embodiments of the present invention.

[0064] Figure 4 It is a structural diagram of an example of a short message filtering and processing device based on a hash matching algorithm of the present invention.

[0065] Reference Figure 4 The second aspect of the present disclosure provides a text message filtering processing device based on a hash matching algorithm, which adopts the text message filtering processing method based on a hash matching algorithm described in the first aspect of the present invention.

[0066] Specifically, the SMS filtering processing device 400 includes a collection module 410 , a word segmentation module 420 , a first determination module 430 and a second determination module 440 .

[0067] In a specific implementation, the collection module 410 is used to collect the SMS messages to be processed, perform various preprocessing on the SMS contents of the SMS messages to be processed, and perform preliminary review and examination on the SMS messages to be processed after the various preprocessing.

[0068] The word segmentation module 420 adopts a two-way maximum matching word segmentation method to perform two-way matching word segmentation on the SMS content of the SMS to be processed that has passed the preliminary review and obtain the word segmentation result of the SMS to be processed. The first determination module 430 determines the character distribution in the preset hash table based on the obtained word segmentation result of the SMS to be processed to determine the complaint rate of the SMS to be processed. The second determination module 440 determines the multi-level evaluation calculation based on the comparison result of the determined complaint rate of the SMS to be processed with the specified threshold, and performs keyword matching based on the keyword matching rule to further perform multi-level control.

[0069] According to an optional implementation manner, determining the character distribution in a preset hash table based on the obtained word segmentation result of the short message to be processed to determine the complaint rate of the short message to be processed includes: Use the following expression to calculate the complaint rate of pending SMS messages:

[0070] Wherein, P represents the complaint rate of the SMS to be processed, which is represented by the number of occurrences of the same keyword string as in the preset hash table; Respectively represent the first segmentation string, the second segmentation string, the third segmentation string, ..., the Nth segmentation string in the content of the SMS to be processed; t1, t2, t3, ..., t N Indicates the number of occurrences corresponding to the first segmentation string, the second segmentation string, the third segmentation string, ..., the Nth segmentation string; It represents the total number of occurrences of the word segments representing the keywords in the preset hash table, where S represents the total number of keywords in the hash table.

[0071] According to an optional implementation manner, the method further includes: selecting an initial threshold; training a complaint rate model using a logistic regression algorithm and a training set according to the initial threshold; in the process of training the complaint rate model, optimizing the initial threshold, and obtaining an optimal threshold using the following expression:

[0072] Among them, Optimal Threshold represents the optimal threshold that maximizes the difference between the true positive rate TPR and the false positive rate FPR of the model prediction result determined by the complaint rate model; θ represents different thresholds, TPR represents the true positive rate of the model prediction result, also known as the true positive rate TPR, which represents the proportion of samples that are correctly classified as positive in all samples that are actually positive, and FPR is the false positive rate of the model prediction result, also known as the false positive rate FPR, which represents the proportion of samples that are incorrectly classified as positive in all samples that are actually negative; the obtained optimal threshold is used as the designated threshold for comparison with the complaint rate of the SMS to be processed.

[0073] According to an optional implementation, when the calculated complaint rate is less than a specified threshold and when the keyword matching module determines that a risk keyword is hit, risk management is entered, that is, the risk area management module is entered to perform multi-level risk management.

[0074] When the calculated complaint rate is greater than or equal to the specified threshold, the risk control will be directly entered, that is, the risk area control module will be entered to carry out multi-level risk control.

[0075] According to an optional implementation manner, a preset hash table is established based on illegal keywords and electronic address types, and the preset hash table includes electronic address types and multiple illegal keywords, and the illegal keywords include fraud-related keywords, deception-related keywords, and keywords representing bad social behavior, wherein the string of each illegal keyword is used as a key, and the string of each illegal keyword is configured with a value to form a key-value combination to establish the preset hash table.

[0076] According to an optional implementation method, reserved electronic address types are extracted from the text message content: chat accounts, short links, bank account information, payment accounts, code numbers, domain names, URLs, IP and other electronic address types; further, based on the extracted electronic address types and keyword information, a pre-screening template is used to perform pre-screening on a variety of pre-processed text messages to determine the text messages that can be released directly.

[0077] According to an optional implementation manner, pre-examination templates are preset, and the pre-examination templates include daily conversation templates, notification templates, announcement templates, and news templates.

[0078] Compared with the prior art, the present invention pre-screens and reviews the SMS messages to be processed after multiple pre-processing, and performs multiple pre-processing on the SMS messages to be processed, so as to effectively and meticulously eliminate redundant information to obtain more refined SMS content, and facilitate the use of a two-way maximum matching algorithm for word segmentation operations; a two-way maximum matching word segmentation method is used to perform two-way matching word segmentation on the SMS content of the SMS messages to be processed that have passed the pre-screening and review, to obtain the word segmentation results of the SMS messages to be processed, to determine the character distribution in a preset hash table to determine the complaint rate of the SMS messages to be processed, and to perform keyword matching based on the comparison result of the determined complaint rate of the SMS messages to be processed and a specified threshold value, and to determine multi-level evaluation calculations to further perform multi-level control, so as to control the risky numbers through the SMS content.

[0079] In addition, by calculating the corresponding risk assessment values ​​for the specific conditions of different regions and adding a hierarchical assessment switch, the control level can be flexibly switched, thereby achieving more effective multi-level control.

[0080] It should be noted that the SMS filtering processing method based on hash matching algorithm executed by the SMS filtering processing device based on hash matching algorithm in Example 2 is the same as the SMS filtering processing method based on hash matching algorithm in Example 1, so the description of the same parts is omitted.

[0081] Figure 5 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

[0082] like Figure 5 As shown, the electronic device is presented in the form of a general computing device. The processor may be one or more and work in coordination. The present invention does not exclude distributed processing, that is, the processor may be dispersed in different physical devices. The electronic device of the present invention is not limited to a single entity, but may also be the sum of multiple physical devices.

[0083] The memory stores a computer executable program, which is usually a machine-readable code. The computer-readable program can be executed by the processor to enable the electronic device to perform the method of the present invention, or at least part of the steps in the method.

[0084] The memory includes a volatile memory, such as a random access memory unit (RAM) and / or a cache memory unit, and may also be a non-volatile memory, such as a read-only memory unit (ROM).

[0085] Optionally, in this embodiment, the electronic device further includes an I / O interface, which is used for the electronic device to exchange data with an external device. The I / O interface can represent one or more of several types of bus structures, including a storage unit bus or a storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of a variety of bus structures.

[0086] It should be understood that Figure 5 The electronic device shown is only an example of the present invention, and the electronic device of the present invention may also include elements or components not shown in the above examples. For example, some electronic devices also include display units such as display screens, and some electronic devices also include human-computer interaction elements such as buttons, keyboards, etc. As long as the electronic device can execute the computer-readable program in the memory to implement the method of the present invention or at least part of the steps of the method, it can be considered as an electronic device covered by the present invention.

[0087] Through the above description of the implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by combining software with necessary hardware. Figure 6 As shown, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes a number of commands to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to the embodiment of the present invention.

[0088] The software product may be any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0089] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with a command execution system, device, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0090] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0091] The computer-readable medium carries one or more programs (eg, computer-executable programs). When the one or more programs are executed by a device, the computer-readable medium implements the method of the present disclosure.

[0092] Those skilled in the art will appreciate that the above modules can be distributed in the device according to the description of the embodiment, or can be changed accordingly and only used in one or more devices different from the embodiment. The modules of the above embodiments can be combined into one module, or further divided into multiple sub-modules.

[0093] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the exemplary embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several commands to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiment of the present invention.

[0094] The exemplary embodiments of the present invention are specifically shown and described above. It should be understood that the present invention is not limited to the detailed structure, configuration or implementation method described herein; on the contrary, the present invention is intended to cover various modifications and equivalent configurations included in the spirit and scope of the appended claims.

Claims

1. A method for filtering and processing short messages based on a hash matching algorithm, characterized in that: include: Collecting the SMS messages to be processed, performing various pre-processing on the SMS contents of the SMS messages to be processed, and conducting preliminary review and examination on the SMS messages to be processed after various pre-processing; A two-way maximum matching word segmentation method is used to perform two-way matching word segmentation on the SMS content of the SMS to be processed that has passed the preliminary review, and obtain the word segmentation result of the SMS to be processed; Determine the character distribution in a preset hash table according to the obtained word segmentation result of the short message to be processed, so as to determine the complaint rate of the short message to be processed; According to the comparison result between the determined complaint rate of the to-be-processed short message and the specified threshold value, keyword matching is performed based on keyword matching rules, and multi-level evaluation calculation is determined to further perform multi-level control.

2. The SMS filtering processing method based on the hash matching algorithm according to claim 1 is characterized in that: The step of determining the character distribution in a preset hash table according to the obtained word segmentation result of the short message to be processed, so as to determine the complaint rate of the short message to be processed, includes: Use the following expression to calculate the complaint rate of pending SMS messages: ； Wherein, P represents the complaint rate of the SMS to be processed, which is represented by the number of occurrences of the same keyword string as in the preset hash table; Respectively represent the first segmentation string, the second segmentation string, the third segmentation string, ..., the Nth segmentation string in the content of the SMS to be processed; t1, t2, t3, ..., t N Indicates the number of occurrences corresponding to the first segmentation string, the second segmentation string, the third segmentation string, ..., the Nth segmentation string; It represents the total number of occurrences of the word segments representing the keywords in the preset hash table, where S represents the total number of keywords in the hash table.

3. The SMS filtering processing method based on hash matching algorithm according to claim 1 is characterized in that: Further including: Select the initial threshold; Based on the initial threshold, the complaint rate model is trained using the logistic regression algorithm and the training set; In the process of training the complaint rate model, the initial threshold is optimized and the optimal threshold is obtained using the following expression: ； Among them, Optimal Threshold represents the optimal threshold that maximizes the difference between the true positive rate TPR and the false positive rate FPR of the model prediction results determined by the complaint rate model; θ represents different thresholds, TPR is the true positive rate of the model prediction results, also known as the true positive rate TPR, which represents the proportion of samples that are correctly classified as positive among all samples that are actually positive, and FPR is the false positive rate of the model prediction results, also known as the false positive rate FPR, which represents the proportion of samples that are incorrectly classified as positive among all samples that are actually negative. The obtained optimal threshold is used as a designated threshold for comparison with the complaint rate of the short message to be processed.

4. The method for filtering short messages based on a hash matching algorithm according to claim 3 is characterized in that: Further including: When the calculated complaint rate is less than the specified threshold, and when the keyword matching module determines that a risk keyword is hit, the risk control module is entered, that is, the risk area control module is entered to perform multi-level risk control; When the calculated complaint rate is greater than or equal to the specified threshold, the risk control will be directly entered, that is, the risk area control module will be entered to carry out multi-level risk control.

5. The method for filtering short messages based on a hash matching algorithm according to claim 1, characterized in that: include: A preset hash table is established based on the illegal keywords and the electronic address type. The preset hash table includes the electronic address type and multiple illegal keywords. The illegal keywords include fraud-related keywords, deception-related keywords, and keywords representing bad social behavior. The character string of each illegal keyword is used as a key, and the character string of each illegal keyword is configured with a value to form a key-value combination to establish the preset hash table.

6. The method for filtering short messages based on a hash matching algorithm according to claim 1, characterized in that: Further including: Extract the reserved electronic address types in the text message content: chat account, short link, bank account information, payment account, code number, domain name, website, IP and other electronic address types; further, based on the extracted electronic address type and keyword information, use the pre-screening template to pre-screen and review the text messages after multiple pre-processing to determine the text messages that can be released directly.

7. The method for filtering short messages based on a hash matching algorithm according to claim 1, characterized in that: Further including: Preset pre-screening templates, including daily conversation templates, notification templates, announcement templates, and news templates.

8. A short message filtering and processing device based on a hash matching algorithm, characterized in that: It implements the SMS filtering processing method based on the hash matching algorithm as described in any one of claims 1 to 7, and the SMS filtering processing device comprises: A collection module collects the SMS messages to be processed, performs various preprocessing on the SMS contents to be processed, and performs preliminary review and audit on the SMS messages to be processed after various preprocessing; The word segmentation module uses a two-way maximum matching word segmentation method to perform two-way matching word segmentation on the SMS content to be processed that has passed the preliminary review and obtain the word segmentation results of the SMS to be processed; A first determination module determines the character distribution in a preset hash table according to the obtained word segmentation result of the short message to be processed, so as to determine the complaint rate of the short message to be processed; The second determination module determines a multi-level evaluation calculation to further perform multi-level control according to the comparison result between the determined complaint rate of the to-be-processed short message and a specified threshold value and performs keyword matching based on keyword matching rules.

9. The SMS filtering processing device based on hash matching algorithm according to claim 8 is characterized in that: include: Use the following expression to calculate the complaint rate of pending SMS messages: ； Wherein, P represents the complaint rate of the SMS to be processed, which is represented by the number of occurrences of the same keyword string as in the preset hash table; Respectively represent the first segmentation string, the second segmentation string, the third segmentation string, ..., the Nth segmentation string in the content of the SMS to be processed; t1, t2, t3, ..., t N Indicates the number of occurrences corresponding to the first segmentation string, the second segmentation string, the third segmentation string, ..., the Nth segmentation string; It represents the total number of occurrences of the word segments representing the keywords in the preset hash table, where S represents the total number of keywords in the hash table.

10. The SMS filtering processing device based on hash matching algorithm according to claim 8, characterized in that: Further including: Select the initial threshold; Based on the initial threshold, the complaint rate model is trained using the logistic regression algorithm and the training set; In the process of training the complaint rate model, the initial threshold is optimized and the optimal threshold is obtained using the following expression: ； Among them, Optimal Threshold represents the optimal threshold that maximizes the difference between the true positive rate TPR and the false positive rate FPR of the model prediction results determined by the complaint rate model; θ represents different thresholds, TPR is the true positive rate of the model prediction results, also known as the true positive rate TPR, which represents the proportion of samples that are correctly classified as positive among all samples that are actually positive, and FPR is the false positive rate of the model prediction results, also known as the false positive rate FPR, which represents the proportion of samples that are incorrectly classified as positive among all samples that are actually negative. The obtained optimal threshold is used as a designated threshold for comparison with the complaint rate of the short message to be processed.

Images