A method for near real-time detection of sensitive words in streaming data

Abstract

The application discloses a kind of stream data sensitive word near real-time detection method, it is characterized in that by introducing segmentation, multiple detection means, solve the problem that one-time detection method needs to wait for stream response to end and then detect, client response performance reduces.By introducing suspected sensitive word mechanism, solve the problem that in the interactive mode of stream response, due to the incomplete content to be detected when detecting each time, leading to the problem that whether the content to be detected hits sensitive word cannot be accurately judged.The creative mechanism of combining suspected sensitive word with reasonable word solves the problem that polysemous words are sensitive words in some contexts, but are not sensitive words in other contexts, leading to polysemous words being misjudged as sensitive words in contexts that are not sensitive words.In the application scenario of large language model, using stream response interactive mode can detect sensitive words in stream data and avoid misjudging polysemous words as sensitive words in reasonable use contexts.

Description

Technical Field

[0001] This invention relates to the field of sensitive word detection in computer processing technology, and in particular to a method for near real-time detection of sensitive words in streaming data. Background Technology

[0002] In large language model applications, due to the characteristics of large language models, such as the long time required for word-by-word output and complete output, large language model applications usually choose to use a streaming response interaction mode to reduce user waiting time and improve the user experience.

[0003] Since both user prompts and the output of large language models may contain inappropriate texts such as those involving violence or pornography, it is necessary to add the ability to detect sensitive words to the application to prevent inappropriate information from appearing in the application.

[0004] Because some words have different meanings in different contexts, and are sensitive words in some contexts but not in others, it is necessary to correctly identify these words with multiple meanings based on the context during the sensitive word detection process to avoid affecting normal user use.

[0005] To address the aforementioned issues, existing technologies such as "Sensitive Word Recognition Method, Apparatus, Storage Medium, and Electronic Device" (Application No.: CN202311042192.8) provide a method for sensitive word detection based on a preset hash map. Existing technologies such as "Sensitive Word Detection Method, Apparatus, Device, and Storage Medium" (Application No.: CN202310929698.4) provide a method for client-side sensitive word detection based on terminal encryption and hash comparison. While both of these existing technologies can solve the sensitive word detection problem in some scenarios, they still cannot solve the problem of sensitive word detection in streaming data under streaming response conditions, nor can they solve the problem of polysemous words being misclassified as sensitive words in reasonable contexts.

[0006] Therefore, a method is needed that can detect sensitive words in near real-time and prevent polysemous words from being misjudged as sensitive words in reasonable contexts. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a method for near real-time detection of sensitive words in streaming data.

[0008] To address the aforementioned technical problems, this invention provides a method for near real-time detection of sensitive words in streaming data, characterized by the following steps:

[0009] Step 1: Upload the sensitive word file and the reasonable word file to the file server;

[0010] Step 2: Load the text of the sensitive word file and the reasonable word file into the application memory, and organize them into a sensitive word prefix tree and a reasonable word prefix tree;

[0011] Step 3: Cache the sensitive word file and the reasonable word file to the application server disk;

[0012] Step 4: Receive the streaming data containing the text to be detected from the output of the large language model;

[0013] Step 5: Concatenate the text output by the received large language model with the text in the suspected sensitive word cache to form the text to be detected. Then, perform matching on the sensitive word prefix tree and the reasonable word prefix tree. The suspected sensitive word cache includes the text to be detected that the sensitive word detection program has determined to contain suspected sensitive words. The specific matching rules are as follows:

[0014] If the text to be detected does not match any sensitive words in the sensitive word prefix tree, it is determined that the text to be detected does not match any sensitive words;

[0015] If the text to be detected has a prefix-matching text in the sensitive word prefix tree, and the prefix-matching text is located at the end of the text to be detected, then the text to be detected is suspected of hitting a sensitive word.

[0016] If the text to be detected has a completely matching text in the sensitive word prefix tree, then it is determined whether the completely matching text has context in the text to be detected. If there is context, the completely matching text is combined with the context and prefix matching is performed in the reasonable word prefix tree. If no match is found, then the text to be detected is determined to contain a sensitive word. If there is a prefix matching text, then it is determined whether the prefix matching text is located at the end of the sentence in the text to be detected. If so, then the text to be detected is suspected of containing a sensitive word; otherwise, the text to be detected is determined to contain a sensitive word.

[0017] If the text to be detected has a completely matching text in the sensitive word prefix tree, and the completely matching text also has a completely matching text in the reasonable word prefix tree, then the text to be detected is determined not to have hit a sensitive word. For texts to be detected that are suspected of hitting sensitive words, the text to be detected is added to the sensitive word cache, and will be merged and processed when the text output by the large language model is received next time. For texts to be detected that hit sensitive words, the text that hits sensitive words is replaced with a masking symbol. For texts to be detected that do not hit sensitive words and those that have been replaced with masking symbols, they are pushed to the client via SSE technology.

[0018] Step 6: Repeat steps 4 to 5 until the output end instruction of the large language model is received. For text that is suspected to exist in the sensitive word cache, it is determined to be non-sensitive text and this part of the text is pushed to the client through SSE technology.

[0019] In step 1, the sensitive word file is a collection of sensitive words used in the sensitive word detection process; the reasonable word file is a collection of reasonable words used in the sensitive word detection process; the reasonable words are non-sensitive words containing inappropriate information.

[0020] In step 2, the sensitive word prefix tree is a tree data structure formed by arranging all sensitive words in the sensitive word file according to the size of the first character and following the red-black tree algorithm; the reasonable word prefix tree is a tree data structure formed by arranging all reasonable words in the reasonable word file according to the size of the first character and following the red-black tree algorithm.

[0021] In step 4, receiving the text output by the large language model is a continuous process. Each time the text output by the large language model is received, sensitive word detection is performed.

[0022] In step 6, the suspected sensitive word cache is a string cache used to temporarily cache the text to be detected that was determined to contain suspected sensitive words in step 5.

[0023] In step 5, the SSE technology includes a streaming response technology that supports a client initiating a single request and the server pushing multiple data streams.

[0024] The beneficial effects achieved by this invention are:

[0025] This invention solves the problem that the one-time detection method requires waiting for the streaming response to finish before detection, which destroys the responsiveness of the streaming response and reduces the client response performance. By introducing segmented and multiple detection methods, this invention addresses the issue that the one-time detection method requires waiting for the streaming response to finish before detection, which leads to the destruction of the responsiveness of the streaming response and the reduction of the client response performance.

[0026] This invention solves the problem of inaccurate judgment of whether the content to be detected contains sensitive words in the interactive mode of streaming response because the content to be detected is incomplete each time.

[0027] This invention solves the problem that polysemous words are sensitive words in some contexts but not in others, leading to the misjudgment of polysemous words as sensitive words in contexts where they are not. Attached Figure Description

[0028] Figure 1 This is a simplified flowchart of the main process for near real-time detection of sensitive words in streaming data according to an exemplary embodiment of the present invention;

[0029] Figure 2 This is a simplified flowchart illustrating a process for determining whether a sensitive word is hit but a reasonable word is not hit, as shown in an exemplary embodiment of the present invention.

[0030] Figure 3A simplified flowchart illustrating the process of determining a suspected reasonable word and ultimately a reasonable word in an exemplary embodiment of the present invention;

[0031] Figure 4 A simplified flowchart illustrating the judgment process for a suspected reasonable word that ultimately fails to match a reasonable word in an exemplary embodiment of the present invention;

[0032] Figure 5 This is a simplified flowchart illustrating a process for determining suspected sensitive words and ultimately identifying them as sensitive words in an exemplary embodiment of the present invention.

[0033] Figure 6 This is a simplified flowchart illustrating a process for determining suspected sensitive words that ultimately do not match, as shown in an exemplary embodiment of the present invention. Detailed Implementation

[0034] In this case, the large language model application hopes to have a means to detect sensitive words contained in the streaming output data text and to avoid misjudging polysemous words in scenarios where they should not be identified as sensitive words.

[0035] The large language model application is an application that provides the output capability of a large language model. Because large language models, represented by GPT, have the characteristics of word-by-word output and long completion time, large language model applications usually choose to use a streaming response interaction mode to reduce user waiting time and improve the user experience.

[0036] A prefix tree is a special form of an N-ary tree. Typically, a prefix tree is used to store strings. Each node in a prefix tree represents a string (prefix). Each node has multiple child nodes, and the paths to different child nodes contain different characters. The string represented by a child node consists of the original string of the node itself, as well as all the characters on the path leading to that child node.

[0037] Traditional techniques for sensitive word detection in streaming data, such as large language model outputs, have limitations. Traditional techniques can only detect complete text, while large language models output text character by character, only showing a portion of the complete text at a time. Using traditional techniques, detection must wait for the large language model to finish outputting before proceeding. This delays the time it takes for the large language model to finish outputting, significantly reducing application responsiveness.

[0038] This invention is based on a pre-built sensitive word prefix tree, a reasonable word prefix tree, and a suspected sensitive word mechanism. By successively detecting the output of the large language model and responding in advance to the detected reasonable text, it reduces the time that the output of the large language model stays in the sensitive word detection process and improves the response performance of the application.

[0039] The present invention will be further described below with reference to the accompanying drawings and exemplary embodiments:

[0040] like Figure 1 The simplified flowchart shown in this embodiment of the invention illustrates the main steps of near real-time detection of sensitive words in streaming data.

[0041] Step 101: Upload the sensitive word file to the file server. The current sensitive word file contains the sensitive word "apple".

[0042] Upload the list of valid keywords to the file server. The current list of valid keywords contains the valid keyword "fruit tree".

[0043] Step 102: Load the text of the sensitive word file and the reasonable word file into the application memory, and organize them into a sensitive word prefix tree and a reasonable word prefix tree.

[0044] Step 103: Cache the sensitive word file and the reasonable word file to the application server disk.

[0045] Step 104: Receive the streaming data containing the text to be detected from the output of the large language model.

[0046] Step 105: Concatenate the text output by the received large language model with the text in the suspected sensitive word cache to form the text to be detected, and then match it in the sensitive word prefix tree and the reasonable word prefix tree in turn.

[0047] Step 106: Repeat steps 105 to 106 until the output end instruction of the large language model is received. For text that is suspected to exist in the sensitive word cache, it is determined to be non-sensitive text, and this part of the text is pushed to the client through SSE technology.

[0048] SSE (Server-sent Events) is a Web technology used to implement real-time data push from a server to a client. It is a lightweight real-time communication solution different from WebSocket, and uses the HTTP protocol. When the server notifies the client that it will subsequently send stream information, the client will keep the connection open to listen to the event stream and can receive data pushed by the server in real time. In this example, outputs from a large language model are received four times in total, and the texts output by the large language model are "I", "want to eat app", "le pie" and an end instruction in sequence. After the sequentially output texts of the large language model are matched in a sensitive word prefix tree and a reasonable word prefix tree respectively, the sensitive word "apple" is hit. Therefore, three pushes are performed to the client through the SSE technology, and the pushed texts are "I", "want to eat ** pie" and the end instruction respectively.

[0049] The corresponding technical effect is that when the text output by the large language model contains a sensitive word, even if the sensitive word is not included in the same output, the text hitting the sensitive word can still be identified after sensitive word detection and replaced with "**", preventing bad information from appearing in the application.

[0050] During the detection process, the output "I" of the large language model does not contain any sensitive word, and does not generate prefix matching with any sensitive word in the sensitive word prefix tree. Therefore, "I" is determined not to hit any sensitive word, and "I" is pushed to the client as the first push text.

[0051] The output "want to eat app" of the large language model does not contain any sensitive word, but the word "app" matches the prefix of the sensitive word "apple", and there is no other text after the word "app", which is at the end of the pending detection text. Therefore, "want to eat app" is determined as a suspected sensitive word.

[0052] The output "le pie" of the large language model and "want to eat app" which is determined as a suspected sensitive word form the pending detection text "want to eat apple pie", and the word "apple" in "want to eat apple pie" hits the sensitive word. However, there are other texts before and after the word "apple", which is a situation where the text hitting the sensitive word has context in the pending detection text. A series of new words formed by "apple" and its context, namely "eat apple", "want to eat apple", "apple pie", "eat apple pie" and "want to eat apple pie", are matched in the reasonable word prefix tree. Since none of the series of new words formed by "apple" and its context matches any reasonable word in the reasonable word prefix tree, the word "apple" in the pending detection text "want to eat apple pie" is determined as a sensitive word, and the word "apple" is replaced with "**". "want to eat ** pie" is pushed to the client as the second push text.

[0053] As Figure 2 shows in the simplified flow chart of hitting a sensitive word but not hitting any reasonable word in an embodiment of the present invention, the specific steps include:

[0054] Step 201: Receive the streaming data containing the text to be detected from the output of the large language model. The text received this time is "I want".

[0055] Step 202: Call the sensitive word detection program to judge the text to be detected. Since the text to be detected "I want" does not contain the sensitive word "apple", it is determined that the text to be detected does not contain the sensitive word.

[0056] Step 203: Push the detected text "I want" to the client via SSE technology.

[0057] Step 204: Receive the streaming data containing the text to be detected from the output of the large language model. The text received this time is "eat apple".

[0058] Step 205: Call the sensitive word detection program to judge the text to be detected. Since the text to be detected "eat apple" contains the sensitive word "apple" and "apple" is not included in any reasonable words, it is determined that the text to be detected contains a sensitive word. Replace the sensitive word "apple" with "**", and the text after replacement is "eat **".

[0059] Step 206: Push the replaced text "eat**" to the client via SSE technology.

[0060] Step 207: Receive the streaming data containing the text to be detected from the large language model output. The received text is empty this time, and an end command has been received. Because the suspected sensitive word cache is empty, the large language model output is empty this time, therefore the text to be detected is empty. Since an end command has been received, the large language model output ends, and the end command is pushed to the client via SSE technology.

[0061] Step 208: The client receives three push notifications in succession: "I want to", "Eat **", and "End Command". Finally, "I want to eat **" is displayed on the page.

[0062] The corresponding technical effect is that if the text output by the large model contains sensitive words, and these sensitive words are not included in any reasonable words, then the part containing the sensitive words will be replaced with "**", thus preventing harmful information from appearing in the application.

[0063] In some embodiments, this is manifested as follows: first, the sensitive word prefix tree is matched; if the sensitive word is completely matched, then the reasonable word prefix tree is matched; if the text before and after the text section that matches the sensitive word can match reasonable words, then the sensitive word in this case is exempted.

[0064] like Figure 3 The flowchart shown in this embodiment of the invention illustrates a process for identifying a suspected reasonable word and ultimately finding a reasonable word. The specific steps include:

[0065] Step 301: receiving streaming data containing text to be detected output by a large language model, wherein the text received this time is "I want".

[0066] Step 302: invoking a sensitive word detection program to determine the text to be detected, since the text to be detected "I want" does not contain the sensitive word "apple", determining that the text to be detected does not contain sensitive words.

[0067] Step 303: pushing the detected text "I want" to a client through SSE technology.

[0068] Step 304: receiving streaming data containing text to be detected output by the large language model, wherein the text received this time is "eat apple".

[0069] Step 305: invoking the sensitive word detection program to determine the text to be detected, since the text to be detected "eat apple" contains the sensitive word "apple", but "apple" is included in the reasonable word "apple pie", determining the text to be detected as a suspected sensitive word, and adding the text to be detected "eat apple" to a suspected sensitive word cache.

[0070] Step 306: receiving streaming data containing text to be detected output by the large language model, wherein the text received this time is "pie".

[0071] Step 307: invoking the sensitive word detection program to determine the text to be detected, since there is a suspected sensitive word "eat apple" in the suspected sensitive word cache, the text to be detected this time is the text obtained by splicing the text output by the large language model and the suspected sensitive word. After splicing, the text to be detected this time is "eat apple pie". Since the text to be detected "eat apple pie" contains the sensitive word "apple", but "apple" and the subsequent text "pie" together form "apple pie" which is a reasonable word, it is determined that the text to be detected does not hit a sensitive word.

[0072] Step 308: pushing the detected text "eat apple pie" to the client through SSE technology.

[0073] Step 309: receiving streaming data containing text to be detected output by the large language model, wherein the text received this time is empty, and an end instruction is received. Invoking the sensitive word detection program to determine the text to be detected, since the suspected sensitive word cache is empty and the reasoning text is empty this time, the text to be detected is empty. Since the end instruction is received, the output of the large language model is finished, and the end instruction is pushed to the client through SSE technology.

[0074] Step 310: the client successively receives three pushes of "I want", "eat apple pie" and the end instruction, and finally displays "I want eat apple pie" on a page.

[0075] The corresponding technical effect is: when the text generated by large model inference contains a sensitive word, but the sensitive word can also hit a legitimate word after being combined with the context, it is determined that the text generated by inference does not contain a sensitive word, which avoids that a polysemous word is misjudged as a sensitive word in a context where it is not a sensitive word.

[0076] As Figure 4 shows in the flowchart of an embodiment of the present invention, which is a simplified flowchart of a process that is suspected to be a legitimate word but ultimately does not hit a legitimate word, the specific steps are:

[0077] Step 401: receiving streaming data containing a text to be detected output by a large language model, the text received this time is "I want to".

[0078] Step 402: invoking a sensitive word detection program to determine the text to be detected, since the text to be detected "I want to" does not contain the sensitive word "apple", it is determined that the text to be detected does not contain a sensitive word.

[0079] Step 403: pushing the detected text "I want to" to the client via SSE technology.

[0080] Step 404: receiving streaming data containing the text to be detected output by the large language model, the text received this time is "eat apple".

[0081] Step 405: invoking the sensitive word detection program to determine the text to be detected, since the text to be detected "eat apple" contains the sensitive word "apple", but "apple" is included in the legitimate word "apple tree", it is determined that the text to be detected is a suspected sensitive word, and the text to be detected "eat apple" is added to a suspected sensitive word cache.

[0082] Step 406: receiving streaming data containing the text to be detected output by the large language model, the text received this time is "pie".

[0083] Step 407: invoking the sensitive word detection program to determine the text to be detected, since there is a suspected sensitive word "eat apple" in the suspected sensitive word cache, the current text to be detected is the text obtained by splicing the text output by the large language model and the suspected sensitive word. After splicing, the current text to be detected is "eat apple pie". Since the text to be detected "eat apple pie" contains the sensitive word "apple", and although the sensitive word "apple" is included in the legitimate word "apple tree", it can only form "apple pie" with the subsequent text "pie", and cannot be exempted through a legitimate word, therefore, it is determined that the text to be detected hits a sensitive word, the sensitive word "apple" is replaced with "**", and the text after replacement is "eat ** pie".

[0084] Step 408: pushing the replaced text "eat ** pie" to the client via SSE technology.

[0085] Step 409: receiving stream data containing text to be detected output by the large language model, wherein the text received this time is empty, and an end instruction is received. A sensitive word detection program is invoked to determine the text to be detected. Since the suspected sensitive word cache is empty and the推理 text this time is empty, the text to be detected is empty. Since the end instruction is received, the output of the large language model is ended, and the end instruction is pushed to the client through SSE technology.

[0086] Step 410: the client sequentially receives three pushes of "I want", "eat**pie" and the end instruction, and finally displays "I want eat**pie" on the page.

[0087] The corresponding technical effect is: when the text generated by large model reasoning contains a sensitive word, and although the sensitive word is suspected to hit a reasonable word during detection, it is determined that it does not hit the reasonable word after being combined with the context, it is determined that the text generated by reasoning contains the sensitive word, and the part hitting the sensitive word is replaced with "**", so as to prevent bad information from appearing in the application.

[0088] As Figure 5 shown in the simplified flow chart of a suspected sensitive word that finally hits the sensitive word in the embodiment of the present invention, the specific steps include:

[0089] Step 501: receiving stream data containing text to be detected output by the large language model, wherein the text received this time is "I want".

[0090] Step 502: invoking a sensitive word detection program to determine the text to be detected. Since the text to be detected "I want" does not contain the sensitive word "apple", it is determined that the text to be detected does not contain sensitive words.

[0091] Step 503: pushing the detected text "I want" to the client through SSE technology.

[0092] Step 504: receiving stream data containing text to be detected output by the large language model, wherein the text received this time is "eat app".

[0093] Step 505: invoking a sensitive word detection program to determine the text to be detected. Since the text to be detected "eat app" contains "app", which is the prefix of the sensitive word "apple", the text to be detected is determined as a suspected sensitive word, and the text to be detected "eat app" is added to the suspected sensitive word cache.

[0094] Step 506: receiving stream data containing text to be detected output by the large language model, wherein the text received this time is "le pie".

[0095] Step 507: Invoke a sensitive word detection program to determine the text to be detected. Since there is a suspected sensitive word "chi ping" in the suspected sensitive word cache, the text to be detected this time is the text output by the large language model spliced with the suspected sensitive word. After splicing, the text to be detected this time is "chi ping guo pai". Since the text to be detected "chi ping guo pai" contains the sensitive word "ping guo", and although this sensitive word "ping guo" is included in the reasonable word "ping guo shu", it can only form "ping guo pai" with the following text "pai" and cannot be exempted by a reasonable word. Therefore, it is determined that the text to be detected hits the sensitive word, and the sensitive word "ping guo" is replaced with "**", and the replaced text is "chi ** pai".

[0096] Step 508: Push the replaced text "chi ** pai" to the client through SSE technology.

[0097] Step 509: Receive the streaming data output by the large language model that includes the text to be detected. The text received this time is empty, and an end instruction is received. Invoke the sensitive word detection program to determine the text to be detected. Since the suspected sensitive word cache is empty and the current inference text is empty, the text to be detected is empty. Since the end instruction has been received, the output of the large language model is completed, and the end instruction is pushed to the client through SSE technology.

[0098] Step 510: The client successively receives three pushes: "I want", "chi ** pai", and the end instruction, and finally displays "I want chi ** pai" on the page.

[0099] The corresponding technical effect is that the text generated by large model inference contains a sensitive word, and since the sensitive word is split in the output of the large language model and is respectively in multiple outputs of the large language model, when the sensitive word is not completely output, the sensitive word detection program always determines that the text to be detected is a suspected sensitive word. After the sensitive word is completely output, it is confirmed that the text to be detected hits the sensitive word and does not hit a reasonable word, then it is determined that the text generated by large language model inference contains a sensitive word, and the part hitting the sensitive word is replaced with "**, so as to prevent bad information from appearing in the application.

[0100] As Figure 6 shown in the simplified flow chart of an embodiment of the present invention where a word is a suspected sensitive word but ultimately does not hit a sensitive word, the specific steps are:

[0101] Step 601: Receive the streaming data output by the large language model that includes the text to be detected, and the text received this time is "I want".

[0102] Step 602: Invoke the sensitive word detection program to determine the text to be detected. Since the text to be detected "I want" does not contain the sensitive word "xiang cai", it is determined that the text to be detected does not contain a sensitive word.

[0103] Step 603: Push the detected text "I want" to the client through SSE technology.

[0104] Step 604: Receive the streaming data containing the text to be detected output by the large language model, and the text received this time is "chi xiang".

[0105] Step 605: Invoke the sensitive word detection program to determine the text to be detected. Since the text to be detected "chi xiang" contains the prefix "xiang" of the sensitive word "xiang cai", the text to be detected is determined as a suspected sensitive word, and the text to be detected "chi xiang" is added to the suspected sensitive word cache.

[0106] Step 606: Receive the streaming data containing the text to be detected output by the large language model, and the text received this time is "jiao pai".

[0107] Step 607: Invoke the sensitive word detection program to determine the text to be detected. Since there is a suspected sensitive word "chi xiang" in the suspected sensitive word cache, the text to be detected this time is the spliced text of the text output by the large language model and the suspected sensitive word. After splicing, the text to be detected this time is "chi xiang jiao pai". Since the text to be detected "chi xiang jiao pai" does not contain the sensitive word "xiang cai", it is determined that the text to be detected does not hit any sensitive word.

[0108] Step 608: Push the detected text "chi xiang jiao pai" to the client through SSE technology.

[0109] Step 609: Receive the streaming data containing the text to be detected output by the large language model, the text received this time is empty, and an end instruction is received. Invoke the sensitive word detection program to determine the text to be detected. Since the suspected sensitive word cache is empty and the current inference text is empty, the text to be detected is empty. Since the end instruction is received, the output of the large language model is finished, and the end instruction is pushed to the client through SSE technology.

[0110] Step 610: The client receives three pushes of "I want", "chi xiang jiao pai" and the end instruction successively, and finally displays "I want chi xiang jiao pai" on the page.

[0111] The corresponding technical effect is that the text generated by large model inference contains a part of a sensitive word. Since the inference text is split into multiple outputs of the large language model, when less inference text is output, the sensitive word detection program determines the text to be detected as a suspected sensitive word; after the output increases gradually, it is confirmed that the text of the suspected sensitive word does not hit any sensitive word, and finally it is determined that the text generated by the large language model inference does not contain sensitive words.

[0112] The term "suspected sensitive words" refers to the situation in a streaming response scenario where, during each sensitive word detection, the text to be detected is not a complete text. If the end of the text to be detected has a prefix match with a sensitive word or a complete match with a sensitive word, and the text also has a prefix match with a reasonable word, then subsequent text needs to be completed before it can be determined whether a sensitive word has been hit. Such text to be detected is defined as containing suspected sensitive words.

[0113] This invention primarily provides a method for near real-time detection of sensitive words in streaming data, and its beneficial effects are:

[0114] By introducing segmented and multiple detection methods, the problem of one-time detection methods waiting for the streaming response to finish before detection is solved, which destroys the responsiveness of the streaming response and reduces the client response performance.

[0115] By introducing a suspected sensitive word mechanism, the problem of not being able to accurately determine whether the content to be detected contains sensitive words in the interactive mode of streaming response is solved because the content to be detected is incomplete each time.

[0116] By combining suspected sensitive words with reasonable words, a creative mechanism is established to solve the problem that polysemous words are sensitive words in some contexts but not in others, leading to the misjudgment of polysemous words as sensitive words in contexts where they are not sensitive words.

[0117] This invention may have many other embodiments. The above embodiments do not limit this invention in any way. Without departing from the spirit and essence of this invention, those skilled in the art can make various corresponding changes and modifications according to this invention. All other improvements and applications made to the above embodiments by equivalent transformation should fall within the protection scope of the appended claims.

Claims

1. A method for near real-time detection of sensitive words in streaming data, characterized in that, Includes the following steps: Step 1: Upload the sensitive word file and the reasonable word file to the file server; Step 2: Load the text of the sensitive word file and the reasonable word file into the application memory, and organize them into a sensitive word prefix tree and a reasonable word prefix tree; Step 3: Cache the sensitive word file and the reasonable word file to the application server disk; Step 4: Receive the streaming data containing the text to be detected from the output of the large language model; Step 5: Concatenate the text output by the received large language model with the text in the suspected sensitive word cache to form the text to be detected. Then, perform matching on the sensitive word prefix tree and the reasonable word prefix tree. The suspected sensitive word cache includes the text to be detected that the sensitive word detection program has determined to contain suspected sensitive words. The specific matching rules are as follows: If the text to be detected does not match any sensitive words in the sensitive word prefix tree, it is determined that the text to be detected does not match any sensitive words; If the text to be detected has a prefix-matching text in the sensitive word prefix tree, and the prefix-matching text is located at the end of the text to be detected, then the text to be detected is suspected of hitting a sensitive word. If the text to be detected has a completely matching text in the sensitive word prefix tree, then it is determined whether the completely matching text has context in the text to be detected. If there is context, the completely matching text is combined with the context and prefix matching is performed in the reasonable word prefix tree. If no match is found, then the text to be detected is determined to contain a sensitive word. If there is a prefix matching text, then it is determined whether the prefix matching text is located at the end of the sentence in the text to be detected. If so, then the text to be detected is suspected of containing a sensitive word; otherwise, the text to be detected is determined to contain a sensitive word. If the text to be detected has a completely matching text in the sensitive word prefix tree, and the completely matching text also has a completely matching text in the reasonable word prefix tree, then the text to be detected is determined not to have hit a sensitive word; for the text to be detected that is suspected to have hit a sensitive word, the text to be detected is added to the sensitive word cache, and will be merged and processed when the text output by the large language model is received next time; for the text to be detected that has hit a sensitive word, the text that has hit a sensitive word is replaced with a masking symbol; for the text to be detected that has not hit a sensitive word and the text that has been replaced with a masking symbol, it is pushed to the client through SSE technology; Step 6: Repeat steps 4 to 5 until the output end instruction of the large language model is received. For text that is suspected to exist in the sensitive word cache, it is determined to be non-sensitive text and this part of the text is pushed to the client through SSE technology.

2. The method for near real-time detection of sensitive words in streaming data as described in claim 1, characterized in that, In step 1, the sensitive word file is a collection of sensitive words used in the sensitive word detection process; the reasonable word file is a collection of reasonable words used in the sensitive word detection process; and the reasonable words are non-sensitive words containing inappropriate information.

3. The method for near real-time detection of sensitive words in streaming data as described in claim 2, characterized in that, In step 2, the sensitive word prefix tree is a tree data structure formed by arranging all sensitive words in the sensitive word file according to the size of the first character and following the red-black tree algorithm; the reasonable word prefix tree is a tree data structure formed by arranging all reasonable words in the reasonable word file according to the size of the first character and following the red-black tree algorithm.

4. The method for near real-time detection of sensitive words in streaming data as described in claim 3, characterized in that, In step 4, receiving the text output by the large language model is a continuous process. Each time the text output by the large language model is received, sensitive word detection is performed.

5. The method for near real-time detection of sensitive words in streaming data as described in claim 4, characterized in that, In step 6, the suspected sensitive word cache is a string cache used to temporarily cache the text to be detected that was determined to contain suspected sensitive words in step 5.

6. The method for near real-time detection of sensitive words in streaming data as described in claim 5, characterized in that, In step 5, the SSE technology includes a streaming response technology that supports a client initiating a single request and the server pushing multiple data streams.

Images