Method and device for screening compliance keywords based on information abstract algorithm comparison

Abstract

The invention discloses a compliance keyword screening method, device and system based on information abstract algorithm comparison and a medium, and the method comprises the steps: carrying out the word segmentation of a to-be-screened text, and segmenting a continuous text sequence into a plurality of text segments with independent semantics; generating an information abstract value corresponding to each text segment; on the basis of the information abstract value, an abstract vector of the to-be-screened text is constructed, and each dimension of the abstract vector corresponds to the information abstract value of one text fragment; and quickly comparing the abstract vector of the to-be-screened text with values in the keyword abstract index table, and outputting a screening result. According to the method, the efficiency of screening compliance keywords in massive texts is greatly improved, the system resource occupation is reduced, the processing time delay is reduced, and the smooth operation of the system is ensured. The recognition capability of the semantic variant of the keyword is enhanced, the screening accuracy is improved, and the omission ratio of non-compliance information is reduced to the maximum extent.

Description

Technical Field

[0001] The present invention belongs to the field of information technology, and in particular relates to a method, device, system and medium for screening compliant keywords based on information summary algorithm comparison. Background Art

[0002] In today's era of digital information explosion, text information of all kinds is growing at a massive rate. Whether it's social media platforms, e-commerce platforms, or internal corporate document systems, vast amounts of text data are generated daily. To ensure that this information complies with laws, regulations, industry standards, and internal corporate rules and regulations, keyword screening has become a crucial task.

[0003] Traditional keyword screening methods often rely on simple string matching techniques, directly comparing character sequences in the text to be tested against a pre-set library of compliant keywords. This approach is somewhat feasible for small texts and simple keyword sets, and can quickly identify text containing specific keywords. For example, some early forum management systems used simple string matching on the titles and text of new user posts to determine whether banned words were present.

[0004] However, as information becomes increasingly complex and diverse, traditional methods have exposed the following problems: 1. Inefficiency: When dealing with massive amounts of text data, character-by-character matching is computationally intensive, consuming significant time and system resources. For example, a large social media platform receives hundreds of millions of daily user updates. Using traditional string matching to screen for compliant keywords would require the server to run at high speed for extended periods, resulting in high latency and potentially system freezes or even crashes due to resource depletion.

[0005] 2. Limited Accuracy: Due to the flexibility of natural language, keywords in text can be expressed in a variety of forms. For example, the keyword "smoking" can appear in actual text as variations such as "smoking," "inhaling a cigarette," and "holding a cigarette in mouth." Simple string matching is difficult to capture all of these semantically similar expressions, which can easily lead to missed detections and the release of non-compliant information into the public eye or internal corporate processes. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to address the shortcomings of the above-mentioned existing technologies and provide a method, device, system, and medium for screening compliant keywords based on information summary algorithm comparison. This method significantly improves the efficiency of screening compliant keywords in massive texts, reduces system resource usage, reduces processing time delays, and ensures smooth system operation. It also enhances the ability to recognize semantic variants of keywords, improves screening accuracy, and minimizes the rate of missed detection of non-compliant information.

[0007] The first aspect of the present invention discloses a method for screening compliant keywords based on information summary algorithm comparison, comprising the following steps: Perform word segmentation on the text to be screened, dividing the continuous text sequence into multiple text segments with independent semantics; Generate a message digest value corresponding to each text fragment; Based on the information summary value, a summary vector of the text to be screened is constructed, where each dimension of the summary vector corresponds to the information summary value of a text fragment; Quickly compare the summary vector of the text to be screened with the value in the keyword summary index table and output the screening results.

[0008] The above method further includes: obtaining updated information of compliant keywords in real time, and updating the keyword summary index table when obtaining compliant keywords that need to be updated; After the keyword summary index table is updated, the screened text is re-matched. During the re-match, only the text range affected by the keyword update is compared and the screening results are output.

[0009] In the above method, the keyword summary index table is obtained by the following method: Obtain compliant keywords and build a keyword library; Generate information summary value for each keyword; Create a keyword summary index table that records the summary value of each keyword and its related attributes.

[0010] The above method first performs a cleaning operation on the text to be screened before performing word segmentation on the text to be screened; The cleaning operation includes removing interference elements from the text and converting the text into a standard format; The interference elements include special symbols, punctuation marks and extra spaces in the text.

[0011] The second aspect of the present invention discloses a compliance keyword screening device based on information summary algorithm comparison, comprising a pre-processing unit, a summary generation unit, a keyword library management unit and a comparison and screening unit; The pre-processing unit is used to perform word segmentation processing on the text to be screened, dividing the continuous text sequence into multiple text segments with independent semantics; The summary generation unit is used to generate an information summary value corresponding to each text segment and construct a summary vector of the text to be screened; The keyword library management unit is used to generate and update the keyword summary index table; The comparison and screening unit is used to quickly compare the summary vector of the text to be screened with the value in the keyword summary index table and output the screening result.

[0012] The above-mentioned device also includes an update monitoring module, which is used to obtain the update information of compliant keywords in real time, and when it is obtained that the compliant keywords need to be updated, drive the keyword library management unit to update the keyword summary index table, and drive the comparison and screening unit to re-match the screened text. During the re-match, only the text range affected by the keyword update is compared and executed, and the screening results are output.

[0013] In the above-mentioned device, the keyword library management unit includes the following steps when generating the keyword summary index table: Obtain compliant keywords and build a keyword library; Generate information summary value for each keyword; Create a keyword summary index table that records the summary value of each keyword and its related attributes.

[0014] In the above-mentioned device, the pre-processing unit performs a cleaning operation on the text to be screened before performing word segmentation processing on the text to be screened; The cleaning operation includes removing interference elements from the text and converting the text into a standard format; The interference elements include special symbols, punctuation marks and extra spaces in the text.

[0015] The third aspect of the present invention discloses a compliance keyword screening system based on information summary algorithm comparison, comprising: Data collection layer: used to collect text to be screened and temporarily store data using buffer pool technology; Processing core layer: composed of a cluster of multiple devices described in the second aspect, used for parallel processing of the collected text to be screened; Result output layer: used to classify and summarize the screening results and generate detailed reports; Control Center: Responsible for configuring system parameters and monitoring system operation status.

[0016] The third aspect of the present invention discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the compliance keyword screening method based on information summary algorithm comparison described in the first aspect is implemented. Compared with the prior art, the present invention has the following advantages: 1. Efficiency: The information summary algorithm is used to quickly extract and compare key text features, avoiding the tedious character-by-character comparison of traditional string matching. When processing massive amounts of text, system resource utilization is significantly improved, and the processing speed is increased by several times or even dozens of times compared to traditional methods, which can meet the needs of large-scale real-time screening.

[0017] 2. High Accuracy: Leveraging the uniqueness and sensitivity of information summaries, combined with auxiliary methods such as semantic analysis, it can not only accurately identify the original form of keywords, but also effectively capture their semantic variations, greatly reducing the risk of missed detection, ensuring that non-compliant information has nowhere to hide, and improving the quality of content review.

[0018] 3. Strong dynamic adaptability: The unique dynamic update mechanism of the keyword library enables the system to quickly respond to changes in laws and policies, efficiently processing only the changed parts, avoiding repeated screening of the entire system, saving time and resources, ensuring that the system always operates according to the latest standards, and effectively reducing corporate compliance risks.

[0019] 4. Good flexibility and scalability: The system adopts a modular design, and each device functions independently and collaboratively, which facilitates customized configuration according to different application scenarios; the distributed architecture is easy to expand horizontally. As business grows, simply adding processing nodes can continuously improve screening capabilities and adapt to the complex and changing information processing needs of the future.

[0020] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a data processing flow chart of the present invention.

[0022] Figure 2 This is a diagram of the internal modules of the device of the present invention.

[0023] Figure 3 This is a system architecture diagram of the present invention. DETAILED DESCRIPTION

[0024] Example 1 like Figure 1 As shown, the compliance keyword screening method based on information summary algorithm comparison includes the following steps: Step 1: Receive the text to be screened and perform a cleaning operation on the text to be screened; the cleaning operation includes removing interference elements in the text and converting the text into a standard format; the interference elements include special symbols, punctuation marks, and extra spaces in the text; Step 2: Use natural language processing technology to perform word segmentation on the text to be screened, dividing the continuous text sequence into multiple text segments with independent semantics, laying the foundation for subsequent feature extraction; Step 3. Use a mature message digest algorithm such as MD5 or SHA-1 to calculate the message digest value for each preprocessed text segment (word, phrase, or short sentence). This generates a message digest value for each text segment. Mature message digest algorithms such as MD5 and SHA-1 can convert input data of varying lengths into fixed-length digest values and have excellent uniqueness. Even slight differences in the input data will result in significantly different digest values. Step 4: Based on the information summary value, a summary vector of the text to be screened is constructed. Each dimension of the summary vector corresponds to the information summary value of a text fragment. This compactly represents the key features of the text and enables dimensionality reduction storage and rapid comparison of text information. Step 5: Quickly compare the summary vector of the text to be screened with the value in the keyword summary index table and output the screening result; During the comparison, we set an appropriate similarity threshold to determine whether the text contains legal keywords. For example, if the similarity between the summary value of a certain fragment of the text to be screened and the summary value of the keyword exceeds the threshold, it is considered a match and the text is marked for further review; if it is below the threshold, it is determined that the keyword is not hit and the next text is processed; When suspected hit keywords are found, the relevant attributes of the keywords are combined. For example, keywords with high risk levels will trigger immediate warning notifications for manual review, while keywords with low risk levels can first undergo a secondary intelligent verification. By introducing contextual semantic analysis and other means, it can be confirmed whether there is a real violation, thereby improving the accuracy and flexibility of screening.

[0025] This embodiment further includes: obtaining updated information of compliant keywords in real time, and updating the keyword summary index table when obtaining compliant keywords that need to be updated; After the keyword summary index table is updated, the screened text is re-matched. During the re-match, only the text range affected by the keyword update is compared and the screening results are output.

[0026] It should be noted that keyword database update monitoring is set up to track changes in laws, regulations, industry standards, and internal corporate rules in real time. Once new keywords are added or old keywords are modified, only the summary calculation and index update of the newly added or modified parts are performed separately, without the need to reprocess the entire keyword database; For the texts that have been screened, we use incremental update technology, combined with the text's timestamp, version number and other identifiers, to quickly locate the text range that may be affected by the keyword update, and only re-screen the summary comparison of the text within this range, greatly saving update costs and time.

[0027] In this embodiment, the keyword summary index table is obtained by the following method: obtaining compliant keywords and constructing a keyword library; generating an information summary value for each keyword; and establishing a keyword summary index table, in which the summary value of each keyword and its related attributes (such as the regulatory category, risk level, etc.) are recorded.

[0028] It should be noted that after obtaining the compliant keywords, these keywords are also preprocessed, including format unification, word segmentation and other operations, and then their information summary values are calculated.

[0029] Example 2 like Figure 2 As shown, the compliance keyword screening device based on information summary algorithm comparison includes a preprocessing unit, a summary generation unit, a keyword library management unit and a comparison and screening unit; The pre-processing unit has a built-in text cleaning module and word segmentation module, which are used to perform word segmentation processing on the text to be screened, and use a natural language processing algorithm to divide the continuous text sequence into multiple text segments with independent semantics; before the text to be screened is segmented, the text to be screened is first cleaned; the cleaning operation includes removing interference elements in the text and converting the text into a standard format; the interference elements include special symbols, punctuation marks and extra spaces in the text; The summary generation unit is used to generate a message digest value corresponding to each text segment and construct a summary vector of the text to be screened. Specifically, the summary generation unit has multiple built-in message digest algorithm modules, such as an MD5 calculator and a SHA-1 processor, which receive preprocessed text data, calculate the summary value of the text segment according to the set rules, and construct the text summary vector; The keyword library management unit is used to generate and update the keyword summary index table, and to connect with the update monitoring module to promptly respond to dynamic changes in the keyword library. When generating the keyword summary index table, the keyword library management unit includes the following steps: obtaining compliant keywords and building a keyword library; generating an information summary value for each keyword; and establishing a keyword summary index table, in which the summary value and related attributes of each keyword are recorded. The comparison and screening unit is used to quickly compare the summary vector of the text to be screened with the value in the keyword summary index table and output the screening result.

[0030] It should be noted that the comparison and screening unit has a built-in similarity comparison module and a risk assessment module. When the comparison and screening unit performs the comparison, it obtains the text summary vector and the keyword summary index table, and uses the built-in similarity comparison module to determine whether the text hits the compliant keywords based on the threshold. In combination with the built-in risk assessment module, the text that hits the keywords is graded and triggered to trigger the corresponding warning or secondary verification process; This embodiment also includes an update monitoring module for obtaining update information of compliant keywords in real time, and when it is obtained that compliant keywords need to be updated, driving the keyword library management unit to update the keyword summary index table, and driving the comparison and screening unit to re-match the screened text. During the re-match, only the text range affected by the keyword update is compared and the screening results are output.

[0031] It should be noted that the update monitoring module queries information sources such as laws, policies, and industry standards in real time online, identifies changes in the keyword library through the built-in difference analysis module, and drives the keyword library management unit and the comparison and screening unit to perform local update operations to ensure that the system always adapts to the latest compliance requirements.

[0032] Example 3 like Figure 3 As shown in the figure, the compliance keyword screening system based on information summary algorithm comparison includes: Data collection layer: This layer collects text to be screened and temporarily stores it using buffer pool technology. For example, in the case of content review on social media platforms, the platform pushes newly posted text to the data collection layer in real time via an API. The data collection layer retrieves a batch of text data to be screened from the buffer pool at a set frequency (e.g., once per second), tags it with metadata such as timestamps and data source identifiers, and then distributes it to the core processing layer. Processing core layer: composed of a cluster of multiple compliant keyword screening devices described in Example 2, which are used to process the collected text to be screened in parallel; each compliant keyword screening device receives text data, and the preprocessing unit first removes interference content such as emoticons and topic tags in the text, and segments the text into word sequences; the summary generation unit calculates the SHA-256 summary value of each word and adjacent phrases to construct a text summary vector; the comparison screening unit compares the summary vector with the keyword summary index table. If it is found that the summary value of a certain text fragment is similar to the summary value of the "pornography-related" keyword by 0.85, which is higher than the threshold, then combined with the high-risk level attribute of the keyword, an early warning signal is immediately sent to the control center, and the text details are pushed to the manual review queue for further verification. If the keyword is not hit, the text processing process ends normally and waits for the next batch of text input; Result output layer: This layer is used to categorize and summarize screening results and generate detailed reports. Screened text results are aggregated to the result output layer, where they are statistically summarized by hit keyword type, risk level, and other dimensions. Visual reports are generated, displaying key information such as the number of newly added illegal texts and the ranking of the frequency of violations by keyword. Administrators can log in to the control center through a browser to view reports and, if needed, export data to Excel format for regular compliance analysis and reporting. Control Center: Responsible for configuring system parameters and monitoring system operation status.

[0033] It should be noted that when implementing the compliance keyword screening system, the hardware used was a high-performance server cluster equipped with sufficient CPU cores, memory capacity, and high-speed storage devices to meet the storage and rapid computation requirements of massive amounts of text data. Servers were connected via high-speed network switches to ensure low latency and high bandwidth for data transmission. For example, a 10 Gigabit Ethernet switch was used to build the internal network. In terms of software, the operating system should be a stable Linux distribution with good support for distributed computing, such as CentOS or Ubuntu Server; a distributed computing framework, such as Apache Flink, should be built on it to coordinate the task scheduling and resource allocation of multiple device instances in the core layer; the database should use a relational database with efficient indexing capabilities (such as MySQL) to store keyword libraries and related configuration information, combined with a distributed file system (such as Ceph) to store massive text data to be screened.

[0034] When the system is initialized, the initially collected and organized set of compliant keywords is entered into the system through the keyword library management unit, the keyword preprocessing process is started, a keyword summary index table is generated and stored in the database, and at the same time, the initial risk level of the keyword, the regulatory field to which it belongs, and other attribute information are configured in the control center.

[0035] Set text preprocessing rules in the control center, such as cleaning symbol ranges and word segmentation algorithm parameters; determine the information summary algorithm type (such as SHA-256 by default); adjust the similarity threshold for comparison screening. Based on test data and business experience, the initial threshold can be set to 0.8, and then dynamically optimized based on the actual screening results; configure the result output format and report template to ensure that the output information meets user needs.

[0036] Example 4 A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the compliance keyword screening method based on information summary algorithm comparison described in Example 1.

[0037] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any simple modification, change and equivalent structural change made to the above embodiment based on the technical essence of the present invention shall still fall within the scope of protection of the technical solution of the present invention.

Claims

1. A compliance keyword screening method based on information summary algorithm comparison is characterized by: The following steps are involved: Perform word segmentation on the text to be screened, dividing the continuous text sequence into multiple text segments with independent semantics; Generate a message digest value corresponding to each text fragment; Based on the information summary value, a summary vector of the text to be screened is constructed, where each dimension of the summary vector corresponds to the information summary value of a text fragment; Quickly compare the summary vector of the text to be screened with the value in the keyword summary index table and output the screening results.

2. The method for screening compliant keywords based on information summary algorithm comparison according to claim 1 is characterized in that: Also includes: Obtain updated information on compliant keywords in real time, and update the keyword summary index table when compliant keywords that need to be updated are obtained; After the keyword summary index table is updated, the screened text is re-matched. During the re-match, only the text range affected by the keyword update is compared and the screening results are output.

3. The method for screening compliant keywords based on information summary algorithm comparison according to claim 1 is characterized in that: The keyword summary index table is obtained by the following method: Obtain compliant keywords and build a keyword library; Generate information summary value for each keyword; Create a keyword summary index table that records the summary value of each keyword and its related attributes.

4. The method for screening compliant keywords based on information summary algorithm comparison according to claim 1 is characterized in that: Before performing word segmentation on the text to be screened, the text to be screened must be cleaned first; The cleaning operation includes removing interference elements from the text and converting the text into a standard format; The interference elements include special symbols, punctuation marks and extra spaces in the text.

5. A compliance keyword screening device based on information summary algorithm comparison, characterized in that: It includes pre-processing unit, summary generation unit, keyword library management unit and comparison and screening unit; The pre-processing unit is used to perform word segmentation processing on the text to be screened, dividing the continuous text sequence into multiple text segments with independent semantics; The summary generation unit is used to generate an information summary value corresponding to each text segment and construct a summary vector of the text to be screened; The keyword library management unit is used to generate and update the keyword summary index table; The comparison and screening unit is used to quickly compare the summary vector of the text to be screened with the value in the keyword summary index table and output the screening result.

6. The compliance keyword screening device based on information summary algorithm comparison according to claim 5 is characterized in that: It also includes an update monitoring module, which is used to obtain the update information of compliant keywords in real time, and when it is obtained that the compliant keywords need to be updated, it drives the keyword library management unit to update the keyword summary index table, and drives the comparison and screening unit to re-match the screened text. During the re-match, only the text range affected by the keyword update is compared and the screening results are output.

7. The compliance keyword screening device based on information summary algorithm comparison according to claim 5 is characterized in that: When the keyword library management unit generates the keyword summary index table, the following steps are included: Obtain compliant keywords and build a keyword library; Generate information summary value for each keyword; Create a keyword summary index table that records the summary value of each keyword and its related attributes.

8. The compliance keyword screening device based on information summary algorithm comparison according to claim 5 is characterized in that: The pre-processing unit performs a cleaning operation on the text to be screened before performing word segmentation processing on the text to be screened; The cleaning operation includes removing interference elements from the text and converting the text into a standard format; The interference elements include special symbols, punctuation marks and extra spaces in the text.

9. The compliance keyword screening system based on information summary algorithm comparison is characterized by: include: Data collection layer: used to collect text to be screened and temporarily store data using buffer pool technology; Processing core layer: composed of a cluster of multiple devices according to claim 5, 6, 7 or 8, for parallel processing of the collected text to be screened; Result output layer: used to classify and summarize the screening results and generate detailed reports; Control Center: Responsible for configuring system parameters and monitoring system operation status.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the compliance keyword screening method based on information summary algorithm comparison according to any one of claims 1 to 4 is implemented.

Images