Sensitive word filtering method and system based on real-time dynamic updating of distributed system

Abstract

The application discloses a sensitive word filtering method and system based on real-time dynamic updating of a distributed system, and the method comprises the following steps: configuring a sensitive word library in a configuration center, providing an interface for updating the sensitive word library, and establishing a long connection between the distributed system and the configuration center; listening to whether the sensitive word library is updated, when it is found that the sensitive word library is updated, establishing an index by using a DFA algorithm based on the updated sensitive word library, and publishing the index to the distributed system; when the distributed system is started, judging by using the storage, performing sensitive word filtering on all the string fields stored in the storage based on the index established by using the DFA algorithm, identifying the sensitive words, and replacing the sensitive words with specified strings. The application can be used for filtering sensitive words in a text published by a user, replacing the sensitive words with specified strings, and realizing real-time dynamic updating of sensitive words that need to be filtered.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information filtering, in particular to a sensitive word filtering method and system based on real-time dynamic updating of a distributed system. BACKGROUND

[0002] When developing a project, there are many places where user input text is entered. This can easily be exploited by criminals to post information such as reactionary and violent information. Therefore, filtering such information is particularly important.

[0003] At the same time, considering the distributed micro-service architecture, the part that needs to be filtered may appear in each module, and the forbidden word table also needs to be updated in real time, so it is particularly important to build a tool class that can be applied in a distributed architecture.

[0004] In addition, because the data volume of the sensitive word library is huge, considering the performance problem, the traditional string traversal replacement cannot meet the demand, and the DFA algorithm (Deterministic Finite Automaton) needs to be used to establish an index to optimize the filtering efficiency. SUMMARY

[0005] To solve the above problems in the prior art, the present application discloses a sensitive word filtering method and system based on real-time dynamic updating of a distributed system, which is used to filter sensitive words in user-posted text, replace the sensitive words with specified strings, and realize real-time dynamic updating of the sensitive words that need to be filtered.

[0006] To achieve the above purpose, the present application provides the following technical scheme:

[0007] A sensitive word filtering method based on real-time dynamic updating of a distributed system, comprising the steps of:

[0008] configuring a sensitive word library in a configuration center and providing an interface for updating the sensitive word library, and establishing a long connection between the distributed system and the configuration center;

[0009] listening to whether the sensitive word library has been updated, and when an update is detected, establishing an index based on the updated sensitive word library using a DFA algorithm and publishing it to the distributed system;

[0010] When the distributed system is started, using the storage judgment, performing sensitive word filtering on all stored string fields based on the index established by the DFA algorithm, identifying sensitive words and replacing them with specified strings.

[0011] Further, the distributed system establishes the long connection in the gateway.

[0012] Further, the step of configuring the sensitive word library further comprises: configuring an invalid word library, and the invalid characters in the invalid word library are not included in the index established based on the DFA algorithm.

[0013] Further, when the sensitive word library is configured, a weight is assigned to each sensitive word.

[0014] Further, after the sensitive word filtering, the frequency of the sensitive word is counted, and the user is scored.

[0015] Further, the index established based on the DFA algorithm is published to the database of the distributed system, and when the micro-service cluster of the distributed system is started, the index is pulled from the database to perform the sensitive word filtering.

[0016] A sensitive word filtering system based on real-time dynamic update of a distributed system, the distributed system and a configuration center establish a long connection, and the sensitive word filtering system comprises:

[0017] A configuration and update module connected with the configuration center, configured to configure a sensitive word library and provide an interface for updating the sensitive word library, and store the sensitive word library to the configuration center;

[0018] A listening module configured to listen to whether the sensitive word library is updated;

[0019] A DFA module configured to, when the listening module listens to the update, establish an index based on the updated sensitive word library by using the DFA algorithm;

[0020] A communication module connected with the distributed system, configured to publish the index established based on the DFA algorithm to the distributed system;

[0021] A filtering module configured to perform sensitive word filtering on all string fields stored in the distributed system based on the index established by the DFA algorithm by using the storage judgment, identify the sensitive word, and replace the sensitive word with a specified string.

[0022] Further, the filtering module uses a Mybatis interceptor.

[0023] Due to the above technical solutions, the application has the following beneficial effects:

[0024] The sensitive word library is stored in a configuration center, and a distributed micro-service cluster is connected with the configuration center through long connection to obtain the update of the sensitive word library and the invalid word library, and the update of the sensitive word library does not affect the normal operation of the distributed micro-service cluster; meanwhile, considering that the long connection consumes too much performance, only the long connection is established in the gateway, and when the update is listened to, the index is established based on the DFA algorithm, and then the index is published to the database, so that when other micro-services are started, the index does not need to be established for a long time, and the sensitive word filtering can be directly pulled from the database. In addition, the DFA algorithm is used to establish the index, which can optimize the filtering efficiency and greatly reduce the query complexity. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0026] Figure 1 The data interaction diagram of the sensitive word filtering system based on the real-time dynamic update of the distributed system of the present application.

[0027] Figure 2 The module diagram of the sensitive word filtering system based on the real-time dynamic update of the distributed system of the present application.

[0028] Figure 3 The architecture diagram of the distributed system. DETAILED DESCRIPTION

[0029] The specific embodiments of the present application will be further described in combination with the drawings. It should be noted that the description of these embodiments is used to help understand the present application, but does not constitute a limitation on the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as there is no conflict.

[0030] The present application provides a sensitive word filtering method based on real-time dynamic update of a distributed system, which is used to filter sensitive words in a text published by a user, replace the sensitive words with a specified string, and realize real-time dynamic update of the sensitive words to be filtered. The sensitive word filtering method mainly includes the following steps:

[0031] Step one, configure the sensitive word library in the configuration center, and provide an interface for updating the sensitive word library, and the distributed system is connected with the configuration center through long connection;

[0032] Step two, listen to whether the sensitive word library is updated, when listening to the update, based on the updated sensitive word library, using DFA algorithm to establish index, and publish to the distributed system;

[0033] Step three, when the distributed system starts, using the judgment of the warehouse, the index based on DFA algorithm is established for all the string fields in the warehouse, and the sensitive word filtering is carried out, the sensitive word is recognized and replaced into the specified string.

[0034] As shown in Figure 3 The distributed system is a software system established on the network, which usually includes gateway, message bus, registration center, configuration center, service management, hardware access, alarm configuration, data visualization, mobile device access module, etc.

[0035] The sensitive word library is stored in the configuration center, and the distributed micro-service cluster is connected with the configuration center through the establishment of long connection to obtain the update of the sensitive word library, and the update of the sensitive word library will not affect the normal operation of the distributed micro-service cluster; at the same time, considering that the establishment of long connection consumes too much performance, so the distributed micro-service cluster only establishes long connection with the configuration center in the gateway, and when listening to the update, the index is established based on DFA algorithm, and then the index is published to the Redis cache engine, so that other micro-services do not need to spend a lot of time to establish index when starting, and can directly pull the index from Redis for sensitive word filtering. In addition, the DFA algorithm is used to establish index, which can optimize the filtering efficiency and greatly reduce the query complexity. Among them, the sensitive word filtering based on DFA algorithm to establish index is prior art, which can refer to the technical content disclosed in the invention "sensitive word filtering method and system based on DFA algorithm" with Chinese patent application number 202110338568.4, which is not repeated here.

[0036] Further, in the embodiment of the present application, when the sensitive word library is configured, a weight can be assigned to each sensitive word. After sensitive word filtering, the frequency of the sensitive word appearing in the log can be counted. In this way, each user can be scored, and if the score of the user is lower than a specified score, the user's right to speak and call other services will be affected (counting the frequency of the sensitive word appearing is to prevent misjudgment due to the frequent appearance of a sensitive word, so that the user is unaware of the situation caused by a single sensitive word to score too low).

[0037] Further, in the embodiment of the present application, the step of configuring the sensitive word library further comprises: configuring an invalid word library, and the index established based on the DFA algorithm does not contain invalid characters in the invalid word library. Specifically, the invalid word library is a list composed of self-defined invalid characters, and after the invalid word library is configured, the index established based on the DFA algorithm will not count these invalid characters. For example, when a text is filtered by a sensitive word, the invalid characters in the text will not be queried in the index, and only the valid characters are matched. Only when the valid character data is set as a sensitive word, the valid character data will be uniformly replaced by a specified string. If the sensitive word cannot be queried, the original text will not be affected because no processing is performed on the original text. Therefore, by using the method of the present application, when meaningless characters are used by a user to separate sensitive words, the sensitive words can still be filtered. At the same time, if meaningless characters are used normally but no sensitive word appears, the original format input by the user will not be filtered.

[0038] Referring to Figure 2 As shown in the figure, the present application further provides a sensitive word filtering system based on real-time dynamic updating of a distributed system, and the distributed system is connected with a configuration center to establish a long connection. The sensitive word filtering system specifically comprises a configuration and updating module 1, a listening module 2, a DFA module 3, a communication module 4, and a filtering module 5.

[0039] The configuration and updating module 1 is connected with the configuration center, is used for configuring a sensitive word library and providing an interface for updating the sensitive word library, and stores the sensitive word library to the configuration center.

[0040] The listening module 2 is used for listening whether the sensitive word library is updated.

[0041] The DFA module 3 is used for, when the listening module listens that there is an update, establishing an index based on the updated sensitive word library by using the DFA algorithm.

[0042] The communication module 4 is connected with a distributed micro-service cluster, and is used for publishing the index established based on the DFA algorithm to the distributed system.

[0043] The filtering module 5 is used for, when the micro-service is used, adopting a storage judgment, performing sensitive word filtering on all string fields stored in the distributed system based on the index established based on the DFA algorithm, identifying sensitive words, and replacing the sensitive words with specified strings. The filtering module can adopt a Mybatis interceptor.

[0044] Specifically, the sensitive word filtering system of the present application can be integrated into the Starter of Springboot in order to establish the tool class, so that the sensitive word filtering function of the project can be automatically realized when the project is imported into the engineering. Among them, Springboot is an open source framework of java platform, which is used to automatically load configuration class without additional configuration by users. The core idea is to let the business personnel not to overcare the configuration, and only need to complete the business code. Starter is only a name, that is, the name of the dependent package after using springboot ***-starter. If it is customized, it is named starter-***.

[0045] By configuring the factories file, when the Springboot project is started, the file will be scanned, and the configuration class recorded in the file will be imported into the spring container, so that the function written by the system can be run.

[0046] In addition, the sensitive word filtering system of the present application can also be integrated into the framework other than Springboot. The purpose of using Springboot is to automatically configure. The system can also provide manual configuration, which can be called in the required place through the provided Factory method. Users can use the exposed interface to integrate into their own project.

[0047] When the system performs sensitive word filtering, it defaults to the judgment of the warehouse, that is, the interceptor of Mybatis is used to filter the sensitive words in all string fields entering the database by using the sensitive word filtering method based on DFA algorithm. The sensitive words are recognized and replaced with specified strings, and the information such as which account uses which sensitive word and the frequency is counted in the log.

[0048] The system can also provide the interceptor returned to the server by the front end, that is, the interceptor when the server parses parameters. That is, users can specify the effective time of the function through the configuration file, such as when the front end passes to the server, or when the server saves data in the database. At the same time, it also supports users to call through tool class, that is, custom use time, which is more flexible. In short, the system provides the interceptor of relational databases such as mysql by default, as well as the interceptor for parsing front-end parameters. At the same time, the interface is exposed to let users customize the processing.

[0049] The system also provides support for log printing function, that is, a history record is provided, for example, when a paragraph is filtered, if necessary, a series of information such as filtering time, filtering vocabulary, content before filtering, etc. can be saved for subsequent processing. That is, a voucher is left.

[0050] The log system is defined by the user in their own project, i.e. the user specifies the output location of the log file and the storage method. That is, the user can configure the log after configuration to the project, and when the sensitive word filtering tool works, the information will be printed to the log configuration specified by the user.

[0051] The system can also provide scoring for users. Alert processing for certain reactionary elements and certain illegal elements or malicious code elements. Implementing speech bans and other processing for low-quality users, or restricting access permissions (such as being unable to use WeChat, DingDing, etc.).

[0052] Cooperation Figure 1 As shown, the system defaults to injecting listeners for common open source configuration centers such as Nacos and SpringCloudConfig. The user can update the sensitive word library and invalid word library by configuring the used registration center and the file to be listened to when the corresponding file is changed. At the same time, the interface for updating the sensitive word library is exposed, i.e. the user can customize the trigger conditions for updating the sensitive word library and integrate it into their project.

[0053] The above-mentioned registration center will listen to changes in the configuration file, and when it detects a change in the configuration file, it will send information to the mounted server. When the server receives the signal, it will read the content of the configuration file and establish an index. Then publish, for use by other modules.

[0054] Because the sensitive word library needs to be called after the project starts, and because the project is a distributed project, it cannot be stopped because the sensitive word library needs to be updated, which would undoubtedly be a huge cost. And the sensitive word library is not fixed at the beginning, and needs to be updated in real time as the business develops. Therefore, the forbidden word library is stored in the configuration file of the configuration center (Nacos), and the state of the sensitive word library is updated by establishing a long connection with Nacos. At the same time, considering that establishing a long connection with Nacos consumes too much performance, only a long connection is established in the gateway. And when the configuration file changes, an index is established. Then publish the index to Redis in the data storage layer, so that other microservices in the business layer do not need to spend a lot of time establishing an index when they start, and can directly pull from Redis. At the same time, other microservices use the CAS optimistic locking mechanism, which can listen to changes in the version of the sensitive word library on Redis by comparing version numbers to update the local index library.

[0055] Further, because the sensitive word library can be as large as 100,000, the time overhead generated by traditional string comparison and replacement is huge, and the corresponding time overhead cannot be borne. Therefore, the DFA algorithm is used to establish the relevant index. By establishing the index tree and using the Bloom mapping (hashing) idea, the time complexity of the algorithm is reduced, and the query efficiency is improved.

[0056] The following illustrates the steps of establishing the index based on the DFA algorithm:

[0057] Step 1: Read each sensitive word in the configuration file.

[0058] Split each character of the sensitive word. For example, "abc" -> "a" -> "b" -> "c" -> weight, and the next one is "ade" -> "a" -> "d" -> "e" -> weight, that is, each character is stored as a separate node. After the index of the above two records is established, a is the parent node, and its child nodes are b and d. Similarly, the process is repeated.

[0059] When judging whether "abc" is in the sensitive word library, first check whether the a node exists. If the a node exists, further judge whether the child node of the a node contains b. If not, it means that abc is not in the sensitive word table. If it exists, further judge whether the child node of b contains c. When c is judged and the content of c is the weight, it means that the data abc in the sensitive word library is obtained, and the configured weight is obtained.

[0060] That is, when the sensitive word library is large, such as several thousand nodes under a, how to find whether it contains b. If compared one by one, it needs to compare an average of more than a thousand times. In the system, the hash function is established, that is, whether the data exists is calculated. The system uses the hash function provided by Java to map b to a number. For example, b is mapped to 1045, so to judge whether b exists, only need to go to the position of 1045 to see, if it is empty, it means that it does not exist.

[0061] The embodiments of the application are described in detail in combination with the drawings, but the application is not limited to the described embodiments. For those skilled in the art, various changes, modifications, replacements and variations of the embodiments can be made without departing from the principles and spirits of the application, and still fall within the protection scope of the application.

Claims

1. A sensitive word filtering method based on real-time dynamic updates of a distributed system, characterized in that, Including the following steps: Configure a sensitive word database in the configuration center and provide an interface for updating the sensitive word database. The distributed system establishes a long connection with the configuration center in the gateway. The system monitors for updates to the sensitive word database. When an update is detected, it uses the DFA algorithm to centrally build an index based on the updated sensitive word database and publishes it to the database of the distributed system. When the distributed system starts, an insertion check is performed. The index is retrieved from the database, and all inserted string fields are filtered for sensitive words based on the index. Sensitive words are identified and replaced with specified strings.

2. The sensitive word filtering method based on real-time dynamic updates of a distributed system according to claim 1, characterized in that, The steps for configuring the sensitive word library also include: configuring an invalid word library, ensuring that the index built based on the DFA algorithm does not contain invalid characters from the invalid word library.

3. The sensitive word filtering method based on real-time dynamic updates of a distributed system according to claim 1, characterized in that: When configuring the sensitive word library, assign a weight to each sensitive word.

4. The sensitive word filtering method based on real-time dynamic updates of a distributed system according to claim 1, characterized in that: After filtering for sensitive words, the frequency of the sensitive words is counted, and users are scored accordingly.

5. A sensitive word filtering system based on a distributed system with real-time dynamic updates, characterized in that, The distributed system establishes a long-lived connection with the configuration center in the gateway. The sensitive word filtering system includes: The configuration and update module connects to the configuration center, is used to configure the sensitive word library and provide an interface for updating the sensitive word library, and stores the sensitive word library in the configuration center; The monitoring module is used to monitor whether the sensitive word database has been updated. The DFA module is used to centrally build an index based on the updated sensitive word database when the monitoring module detects an update. A communication module, connected to the distributed system, is used to publish the index built based on the DFA algorithm to the database of the distributed system; The filtering module is used to perform a database entry judgment, retrieve the index from the database, filter sensitive words for all string fields entered into the distributed system based on the index, identify sensitive words and replace them with specified strings.

6. The sensitive word filtering system based on real-time dynamic updates of a distributed system according to claim 5, characterized in that, The filtering module uses Mybatis interceptors.

Images