Content detection method, system, equipment and medium
By using sampling strategy and sliding window slicing technology to perform content detection during the streaming output of the digital assistant, the problem of long-term content detection in the existing technology is solved, real-time and accuracy are improved, and inappropriate Presentation of content.
Patent Information
- Application Number
- CN202410978396.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-19
- Publication Date
- 2024-11-19
AI Technical Summary
In the prior art, the process of performing content detection after generating reply information is long, and it is difficult to ensure the real-timeness of content detection, resulting in inappropriate content being presented during the user's waiting period.
During the process of streaming reply information in the language model, the content slice is determined according to the set sampling strategy, and the content is detected while streaming output, and the content is sliced using the sliding window and the step strategy to achieve the edge output edge Detection, improve the real-timeness of content detection.
It realizes real-time detection of inappropriate content when the language model is streamed to output content, avoids users waiting and reduces the presentation of inappropriate content, and improves the real-time and accuracy of content detection.
Smart Images

Figure CN118982019A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a content detection method, system, electronic device, and computer-readable storage medium. Background Art
[0002] With the rapid development of computer technology, digital assistants have emerged. Users can interact with digital assistants through human-computer dialogue. Specifically, users can input question information through digital assistants, and digital assistants can generate reply information for the question information by analyzing the question information, thereby realizing human-computer dialogue.
[0003] Normally, considering that the reply message is long, the digital assistant can use streaming output to output the content, and transmit the output content to the client in streaming output mode to avoid users waiting for a long time.
[0004] In the related art, in order to avoid inappropriate content (e.g., content that is not desired to be presented to the user) in the reply information, after the reply information is generated, content detection can be performed on the reply information. However, in the above content detection process, it takes a long time from the generation of the reply information to the completion of the content detection, and it is difficult to ensure the real-time performance of the content detection. Summary of the invention
[0005] The present application provides a content detection method. The method can perform content detection in the process of a language model streaming out reply information, thereby improving the real-time performance of content detection. The present application also provides a system, an electronic device, a computer-readable storage medium, and a computer program product corresponding to the above method.
[0006] In a first aspect, the present application provides a content detection method, the method comprising:
[0007] Continuously receive the output content of the language model streaming output;
[0008] According to the set sampling strategy, content slices are determined from the output content that has been output; content slices
[0009] Performing content detection on the content slice to obtain a detection result of the content slice;
[0010] The output content that has undergone content detection is transmitted to a client, so that the output content is displayed on the client in a streaming output manner.
[0011] In some possible implementations, determining content slices from the output content that has been output according to a set sampling strategy includes:
[0012] According to a set sliding window and / or a set step, a content slice is determined from the output content that has been output; wherein the sliding window is used to determine the number of reply characters in the content slice, and the step is used to determine the difference between content slices of two adjacent content detections.
[0013] In some possible implementations, determining the content slice from the output content that has been output according to the set sliding window and / or the set step includes:
[0014] According to a set step, a set sliding window is moved backward in a character sequence; wherein the character sequence is composed of a plurality of reply characters in the output content that has been output;
[0015] In response to the number of the reply characters in the sliding window reaching the length of the sliding window, the reply characters in the sliding window are determined as content slices.
[0016] In some possible implementations, the method further includes:
[0017] In response to the language model starting to stream output the output content, determining that the position of the sliding window in the character sequence is the front end of the character sequence;
[0018] In response to the number of the reply characters in the sliding window reaching a first preset length, the reply characters in the sliding window are determined as content slices for a first content detection, wherein the first preset length is less than the length of the sliding window.
[0019] In some possible implementations, after completing the first content detection, the method further includes:
[0020] Maintaining the position of the sliding window in the character sequence;
[0021] In response to the number of the reply characters in the sliding window reaching the length of the sliding window, the reply characters in the sliding window are determined as content slices for a second content detection.
[0022] In some possible implementations, performing content detection on the content slice to obtain a detection result of the content slice includes:
[0023] The content slice is input into a content detection model, and a detection result output by the content detection model is received; wherein the content detection model has natural language analysis capability.
[0024] In some possible implementations, the method further includes:
[0025] Based on the detection result of the content slice, determine the processing strategy for the content slice; wherein the processing strategy includes at least one of the following: a strategy for indicating whether the language model continues to stream the output content, a strategy for indicating whether to present the output content, and a strategy for indicating to present the content.
[0026] In some possible implementations, the detection result indicates that the content slice passes the content detection, and determining a processing strategy for the content slice according to the detection result of the content slice includes:
[0027] instructing the language model to continue streaming the output content; and,
[0028] Instruct the client to present a target reply character in a conversation page; wherein the conversation page is used to interact with a digital assistant, and the target reply character is a reply character in the content slice that is not presented in the conversation page.
[0029] In some possible implementations, the detection result indicates that the content slice fails content detection, and determining a processing strategy for the content slice according to the detection result of the content slice includes:
[0030] instructing the language model to stop streaming the output content; and,
[0031] The client is instructed to replace the reply characters presented in the conversation page with default characters.
[0032] In a second aspect, the present application provides a content detection system, the system comprising:
[0033] A receiving module, used to continuously receive the output content of the language model streaming output;
[0034] A determination module, configured to determine content slices from the output content that has been output according to a set sampling strategy;
[0035] A detection module, used to perform content detection on the content slice and obtain a detection result of the content slice;
[0036] The transmission module is used to transmit the output content that has undergone content detection to the client, so that the output content is displayed on the client in a streaming output manner.
[0037] In some possible implementations, the determining module is specifically configured to:
[0038] According to a set sliding window and / or a set step, a content slice is determined from the output content that has been output; wherein the sliding window is used to determine the number of reply characters in the content slice, and the step is used to determine the difference between content slices of two adjacent content detections.
[0039] In some possible implementations, the determining module is specifically configured to:
[0040] According to a set step, a set sliding window is moved backward in a character sequence; wherein the character sequence is composed of a plurality of reply characters in the output content that has been output;
[0041] In response to the number of the reply characters in the sliding window reaching the length of the sliding window, the reply characters in the sliding window are determined as content slices.
[0042] In some possible implementations, the determining module is further configured to:
[0043] In response to the language model starting to stream output the output content, determining that the position of the sliding window in the character sequence is the front end of the character sequence;
[0044] In response to the number of the reply characters in the sliding window reaching a first preset length, the reply characters in the sliding window are determined as content slices for a first content detection, wherein the first preset length is less than the length of the sliding window.
[0045] In some possible implementations, after completing the first content detection, the determination module is further configured to:
[0046] Maintaining the position of the sliding window in the character sequence;
[0047] In response to the number of the reply characters in the sliding window reaching the length of the sliding window, the reply characters in the sliding window are determined as content slices for a second content detection.
[0048] In some possible implementations, the detection module is specifically used to:
[0049] The content slice is input into a content detection model, and a detection result output by the content detection model is received; wherein the content detection model has natural language analysis capability.
[0050] In some possible implementations, the determining module is further configured to:
[0051] Based on the detection result of the content slice, determine the processing strategy for the content slice; wherein the processing strategy includes at least one of the following: a strategy for indicating whether the language model continues to stream the output content, a strategy for indicating whether to present the output content, and a strategy for indicating to present the content.
[0052] In some possible implementations, the detection result indicates that the content slice passes the content detection, and the determination module is specifically configured to:
[0053] instructing the language model to continue streaming the output content; and,
[0054] Instruct the client to present a target reply character in a conversation page; wherein the conversation page is used to interact with a digital assistant, and the target reply character is a reply character in the content slice that is not presented in the conversation page.
[0055] In some possible implementations, the detection result indicates that the content slice fails the content detection, and the determining module is specifically configured to:
[0056] instructing the language model to stop streaming the output content; and,
[0057] The client is instructed to replace the reply characters presented in the conversation page with default characters.
[0058] In a third aspect, the present application provides an electronic device, the electronic device comprising a processor and a memory. The processor and the memory communicate with each other. The processor is used to execute instructions stored in the memory so that the electronic device performs the content detection method in the first aspect or any implementation of the first aspect.
[0059] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, wherein the instructions instruct an electronic device to execute the content detection method described in the first aspect or any one of the implementations of the first aspect.
[0060] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on an electronic device, enables the electronic device to execute the content detection method described in the first aspect or any one of the implementations of the first aspect.
[0061] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations.
[0062] It can be seen from the above technical solutions that this application has the following advantages:
[0063] The present application provides a content detection method, which continuously receives output content streamed by a language model, determines content slices from the output content that has been output according to a set sampling strategy, then performs content detection on the content slices, obtains detection results of the content slices, and transmits the output content that has undergone content detection to a client, so that the output content is displayed on the client in a streaming output manner.
[0064] In this method, while the language model is outputting content in a streaming manner, the output content is continuously received, and the output content that has been output is sliced through the set sampling strategy. In this way, there is no need to wait for all the reply information to be generated before performing detection, thus achieving "detection while outputting" and improving the real-time performance of content detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0065] In order to more clearly illustrate the technical method of the embodiments of the present application, the drawings required for use in the embodiments are briefly introduced below.
[0066] Figure 1 A flowchart of a content detection method provided in an embodiment of the present application;
[0067] Figure 2 A flowchart of another content detection method provided in an embodiment of the present application;
[0068] Figure 3 A schematic diagram of the structure of a content detection system provided in an embodiment of the present application;
[0069] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0070] The terms "first" and "second" in the embodiments of the present application are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.
[0071] First, some technical terms involved in the embodiments of the present application are introduced.
[0072] With the rapid development of computer technology, digital assistants have emerged. Digital assistants usually have natural language processing capabilities. In some examples, users can interact with digital assistants through human-computer dialogue. For example, users can enter questions related to general knowledge in the digital assistant. By analyzing the content entered by the user, the digital assistant can generate reply information for questions related to general knowledge, thereby completing a round of dialogue. In other examples, the digital assistant can process natural language content according to the needs of the user. For example, the user can enter a text in the digital assistant and instruct the digital assistant to perform relevant processing (such as summarizing the text). The digital assistant can summarize the text entered by the user by identifying the user's needs and generate a reply information including the text summary content.
[0073] Considering that the reply information can be long, the digital assistant can output the content in a streaming response manner, and transmit the output content to the client in a streaming output manner. That is to say, the digital assistant can output the content in units of reply characters, output one reply character at a time, and after all reply characters are output, the reply characters form a reply message. Moreover, during the streaming output process, the reply characters are continuously transmitted to the client. For example, when the reply characters reach a certain number, the number of reply characters is transmitted to the client. For another example, at a certain transmission interval, the reply characters output within the transmission interval are transmitted to the client. In this way, the client can present the output content that the digital assistant has output in real time in a streaming output manner, without having to wait for the digital assistant to generate a complete reply message, and then present the complete reply message, thereby avoiding the user waiting time for too long and improving the user's interactive experience with the digital assistant.
[0074] Digital assistants usually generate reply information based on the natural language processing capabilities of language models. Since the corpus (i.e., training data) for training language models is large and complex, it is difficult to ensure that the reply information generated by digital assistants fully meets user needs.
[0075] In the related art, in order to avoid inappropriate content in the reply information, such as content that is not desired to be presented to the user, it is usually necessary to perform content detection on the reply information. Specifically, after the digital assistant generates the reply information, the reply information can be divided into multiple segments, and the content detection is performed using preset detection rules or keyword matching rules. However, the above content detection method is performed after the reply information is completed. The reply information with inappropriate content may have been presented to the user, and it is difficult to ensure the real-time performance of content detection.
[0076] In view of this, the present application provides a content detection method. The method continuously receives the output content of the language model in a streaming manner, determines the content slice from the output content that has been output according to the set sampling strategy, then performs content detection on the content slice, obtains the detection result of the content slice, and transmits the output content after content detection to the client, so that the output content is displayed in a streaming output manner on the client.
[0077] In this method, while the language model is outputting content in a streaming manner, the output content is continuously received, and the output content that has been output is sliced through the set sampling strategy. In this way, there is no need to wait for all the reply information to be generated before performing detection, thus achieving "detection while outputting" and improving the real-time performance of content detection.
[0078] To facilitate understanding of the technical solution provided by the embodiments of the present application, it will be described below with reference to the accompanying drawings.
[0079] See also Figure 1 The flowchart of a content detection method provided in an embodiment of the present application is shown, and the method can be executed by a server for performing content detection. The method specifically includes:
[0080] S101: Continuously receive the output content of the language model streaming output.
[0081] In an embodiment of the present application, a user can have a human-computer dialogue with a digital assistant. In some embodiments, the digital assistant can be an independent software system. For example, the digital assistant can have an application (APP) with a human-computer dialogue function. In other embodiments, the digital assistant can be a functional module of other systems. For example, the digital assistant can be deployed in a business platform and provide a human-computer dialogue function as a functional module of the business platform. In other embodiments, the digital assistant can also support online use.
[0082] The digital assistant usually provides a dialogue page, and the user can interact with the digital assistant in the dialogue page. Specifically, the dialogue page can provide an input box and an information sending control, and the user can enter the request information in the input box, for example, the user can enter the question information, or the processing requirement information for the text content in the input box. After the user completes the input of the request information, the information sending control can be triggered to send the request information to the digital assistant, and the request information is presented in the dialogue window.
[0083] Depending on the application scenario, the dialog window can have different forms of expression. In some embodiments, the digital assistant is an independent software system, in which case the dialog window can be a page loaded in the client of the digital assistant. In other embodiments, the digital assistant can be a functional module of other systems, in which case the dialog window can be a page loaded in the client of other systems. In other embodiments, the digital assistant supports online use, and the dialog window can be a page displayed in a browser.
[0084] In some possible implementations, the digital assistant may generate a reply based on the natural language processing capabilities of a language model. The language model has natural language processing (NLP) capabilities and can be used to process different types of natural language tasks. For example, the language model may be a deep learning model trained using text data.
[0085] That is to say, after the user sends a request message, the digital assistant can call the language model to process the request message. The language model analyzes the request message, identifies the user's human-computer dialogue intention, and then generates a reply message for the request message. In this way, a round of human-computer dialogue is completed.
[0086] The specific process of generating reply information by a language model is described below. The language model is usually a generative model, and the generation process of reply information may include multiple output rounds, wherein each output round outputs a reply character (token) of the reply information, and the new reply character is generated based on the request information and the reply characters generated in the previous output round.
[0087] That is, in the first output round, the language model outputs the first reply character based on the request information, and in the Nth output round, the language model generates the Nth reply character based on the request information and the N-1 reply characters that have been output in the first N-1 output rounds. Where N is an integer greater than 1. In this way, after the content output of all output rounds is completed, the reply characters output in each output round constitute the reply information output by the language model.
[0088] In an embodiment of the present application, the language model outputs content in a streaming output manner, and the server for content detection can continuously receive the output content of the language model in a streaming output. In other words, the digital assistant outputs a reply character in each output round by calling the language model, and sends the output content including the reply character to the server for content detection. In this way, the server for content detection continuously receives the output content output by the language model in units of reply characters, so as to perform content detection on the output content before presenting the reply information to the user.
[0089] S102: Determine content slices from the output content that has been output according to the set sampling strategy.
[0090] Specifically, the content slice includes at least one reply character. In the embodiment of the present application, the content slice can be understood as a detection object in content detection, that is, the content detection can detect at least one reply character in the content slice.
[0091] The sampling strategy can be understood as a strategy for determining content slices from the output content. Since the language model uses streaming output and generates one reply character per output round, the server used for content detection needs to determine the content slice to be detected from the output content (including multiple reply characters) that has been output by the language model using a certain sampling strategy.
[0092] In some possible implementations, combined with the characteristics of the language model streaming output, the server for content detection can determine the content slice through the sliding window and the step. Specifically, according to the set sliding window and / or the set step, the content slice is determined from the output content that has been output.
[0093] Among them, the sliding window can be used to process subsequences in the data sequence, maintain a fixed-length window, slide the window on the data sequence, and the data in the window forms a subsequence. The stride can be understood as the step length of the sliding window sliding on the data sequence.
[0094] In an embodiment of the present application, a data sequence may refer to a character sequence composed of multiple reply characters in the output content output by the language model. In other words, according to the order of the output rounds, the reply characters output by the language model in each output round are composed into a character sequence.
[0095] Furthermore, the sliding window can be used to determine the number of reply characters in the content slice, and the step can be used to determine the difference between the content slices of two adjacent content detections. In other words, by sliding the sliding window on the character sequence, the content slices of multiple content detections are determined in the process of the language model generating reply information, thus realizing "detection while outputting".
[0096] In a specific implementation, the server for content detection can move a set sliding window backward in the character sequence according to a set step, and in response to the number of reply characters in the sliding window reaching the length of the sliding window, determine the reply characters in the sliding window as content slices.
[0097] That is to say, in the embodiment of the present application, the number of reply characters of a content detection satisfies the length of the sliding window. For example, when the length of the sliding window is 500 characters, the number of reply characters in the content slice is 500. During subsequent content detection, content detection is performed on the 500 reply characters in the content slice. The difference between the content slices of two adjacent content detections is the number of reply characters of the step length. For example, when the length of the step is 20 characters, if the content slice of the Nth content detection includes the 301st to 800th characters in the character sequence, the content slice of the N+1th content detection includes the 321st to 820th characters in the character sequence.
[0098] It can be understood that since the language model continuously generates reply characters in a streaming output manner, after the sliding window is moved backward at a set step, the sliding window may exceed the range of the character sequence, that is, the number of reply characters in the sliding window may be less than the length of the sliding window. For example, the length of the sliding window is 500 characters, the length of the step is 20 characters, and after moving 20 characters backward in the character sequence, the number of reply characters included in the sliding window and output is only 480, which is less than the length of the sliding window. As the server used for content testing continues to receive the output content of the language model streaming output, the length of the character sequence increases accordingly. When the number of reply characters in the sliding window reaches 500, the 500 reply characters in the sliding window are used as content slices.
[0099] Thus, in view of the characteristics of the language model streaming output, the embodiment of the present application slices the generated reply characters by moving the sliding window in steps during the process of the language model generating reply information, and determines the detection objects for multiple content detections. In addition, since the output content of the language model output usually represents natural language content, in the embodiment of the present application, by moving the sliding window in steps, the content slices of two adjacent content detections include some of the same reply characters, thus avoiding the low accuracy of subsequent content detection due to problems such as inaccurate character segmentation and incoherent semantics.
[0100] Further, in response to the language model starting to stream the output content, the position of the sliding window in the character sequence is determined to be the front end of the character sequence, and in response to the number of reply characters in the sliding window reaching a first preset length, the reply characters in the sliding window are determined as content slices for the first content detection.
[0101] The first preset length is smaller than the length of the sliding window. For example, the first preset length may be the length of a step.
[0102] That is, when the language model starts to generate reply information, the character sequence does not include the reply characters that have been output, and the initial position of the sliding window is determined as the front end of the character sequence, and the first content slice is determined after the language model generates reply characters of the first preset length. In this way, the first content detection is performed after a short waiting time.
[0103] Taking the first preset length as the length of the step as an example, when the length of the step is 20 characters, the position of the sliding window is the front end of the character sequence. After waiting for the language model to output to the 20th reply character, the 1st reply character to the 20th reply character is determined as the content slice for the first content detection.
[0104] Further, after completing the first content detection, the position of the sliding window in the character sequence is maintained, and in response to the number of reply characters in the sliding window reaching the length of the sliding window, the reply characters in the sliding window are determined as content slices for the second content detection.
[0105] That is to say, the first content slice includes fewer reply characters of a first preset length. After the first content detection is completed, the sliding window is not moved backward, and the language model continues to be waited for to output content until the number of reply characters output reaches the length of the sliding window, and the reply characters of the sliding window length are used as the second content slice.
[0106] For example, the first preset length is the length of the step. When the length of the step is 20 characters and the length of the sliding window is 500 characters, the content slice for the first content detection includes the 1st reply character to the 20th reply character. After waiting for the language model to stream output to the 500th reply character, the 1st reply character to the 500th reply character is determined as the content slice for the second content detection.
[0107] Thus, in the early stage of the reply information generation process, the reply characters are sliced by not moving the sliding window, first waiting for the language model to output the reply characters of the step length, and then waiting for the language model to output the reply characters of the sliding window length. In the subsequent content detection, the reply characters are sliced by sliding the sliding window backwards and waiting for the number of reply characters in the sliding window to reach the sliding window length.
[0108] Based on the above content, it can be known that in the embodiment of the present application, the content slicing determination process is related to the length of the sliding window and the step length, so the user can configure the length of the sliding window and the step length by himself. In other words, the user can configure the corresponding sliding window length and step length based on the actual business scenario to meet the content detection requirements in different business scenarios and improve flexibility.
[0109] S103: Perform content detection on the content slice to obtain the detection result of the content slice.
[0110] In the embodiment of the present application, the content detection of the content slice can be used to detect whether there is inappropriate content in the content slice that is not desired to be presented to the user. In specific implementation, the server for content detection can input the content slice into the content detection model and receive the detection result output by the content detection model.
[0111] The content detection model has a natural language analysis capability, for example, the content detection model can be a natural language analysis model. By calling the content detection model and utilizing the natural language analysis capability of the content detection model, it is determined whether the content slice passes the content detection.
[0112] Compared with the keyword matching method used in related technologies, the content detection model can accurately detect richer features such as the semantics of reply characters in content slices, thereby improving the accuracy of content detection and reducing false positives and negatives.
[0113] It should be noted that the embodiments of the present application do not limit the specific method of content detection. In some possible implementation methods, content detection for content slices can also be achieved by using set content detection rules and other methods.
[0114] In an embodiment of the present application, by determining content slices, performing content detection on content slices, determining content slices again, and performing content detection on content slices again, multiple content detections are performed during the process of the language model generating reply information to meet the real-time content detection requirements in streaming answers.
[0115] S104: Transmit the output content after content detection to the client, so that the output content is displayed on the client in a streaming output manner.
[0116] In the embodiment of the present application, the output content of the language model streaming output is also transmitted to the client of the digital assistant in the form of streaming output. In this way, when the language model generates reply information, the client of the digital assistant can present the output content that has been output to the user in the form of streaming output, realizing "output and presentation at the same time", reducing the user's waiting time.
[0117] At the same time, in the embodiment of the present application, the output content needs to be detected before being transmitted to the client of the digital assistant. In other words, the output content that has not been detected will not be transmitted to the client and will not be presented to the user. In this way, the output content including inappropriate content is avoided from being presented to the user.
[0118] Furthermore, in some embodiments, the server for performing content detection may also determine a processing strategy for the content slice based on the detection result of the content slice.
[0119] The processing strategy may include at least one of the following: a strategy for indicating whether the language model continues to stream output the aforementioned output content, a strategy for indicating whether to present the output content, and a strategy for indicating to present the content.
[0120] That is to say, after the server used for content detection determines the content detection result, it can judge whether the language model should continue to reply to the user's request information, whether the current output content should be presented to the user, and whether the presented content should be processed based on the content detection result.
[0121] like Figure 2 As shown, when the detection result indicates that the content slice passes the content detection, the server for content detection can instruct the language model to continue streaming the aforementioned output content, and instruct the client to present the target reply character in the conversation page. The target reply character is the reply character in the content slice that is not presented in the conversation page.
[0122] It can be understood that since the language model outputs content in a streaming output manner, and the output content is transmitted to the digital assistant client in a streaming output manner, in an embodiment of the present application, the content slices of two adjacent content detections may include the same reply characters. For example, the content slice of the Mth content detection is the 21st reply character to the 520th reply character, and the content slice of the M+1th content detection is the 41st reply character to the 540th reply character, wherein the 41st reply character to the 520th reply character are the same reply characters. At this time, if the Mth detection result indicates that the content slice passes the content detection, the 1st reply character to the 520th reply character may be presented in the conversation page. If the M+1th detection result indicates that the content slice passes the content detection, at this time, the target reply characters are the 521st reply characters to the 540th reply characters.
[0123] In this way, when the content slice does not include inappropriate content, the language model is instructed to output subsequent reply characters normally, and the client is instructed to present the output content that has been output to the user normally.
[0124] When the detection result indicates that the content slice fails the content detection, the language model is instructed to stop streaming the output content, and the client is instructed to replace the reply characters presented in the conversation page with default characters.
[0125] That is, when the content slice includes inappropriate content, the language model is instructed to stop generating subsequent reply characters and end the current human-computer dialogue. In addition, in order to avoid presenting inappropriate content to the user, the presented content in the dialogue page is replaced in time. For example, the default character can be "I can't answer this question for the time being, try another question."
[0126] It should be noted that in the embodiments of the present application, other methods can be used to avoid presenting inappropriate content to users. For example, keywords in inappropriate content can be replaced with default words, and inappropriate content can be presented with an "*" symbol, which is not limited in the embodiments of the present application.
[0127] In some possible implementations, the number of reply characters in the sliding window may not reach the length of the sliding window. For example, the length of the sliding window is 500 characters, the length of the reply characters in the current sliding window is 490 characters, but the reply information has been generated. In view of the above situation, the embodiments of the present application provide two different processing modes: In some embodiments, the server for performing content detection may default to the reply characters in the sliding window passing the content detection, thereby achieving a more relaxed content detection. In other embodiments, the server for performing content detection may not perform content detection on the reply characters in the sliding window, so that the reply characters in the sliding window will not be presented to the user, thereby avoiding the presentation of inappropriate content to the user and achieving a more stringent content detection.
[0128] Based on the above content description, an embodiment of the present application provides a content detection method. The method continuously receives the output content of the language model streaming output, determines the content slice from the output content that has been output according to the set sampling strategy, then performs content detection on the content slice, obtains the detection result of the content slice, and transmits the output content after content detection to the client, so that the output content is displayed in a streaming output manner on the client.
[0129] In this method, while the language model is outputting content in a streaming manner, the output content is continuously received, and the output content that has been output is sliced through the set sampling strategy. In this way, there is no need to wait for all the reply information to be generated before performing detection, thus achieving "detection while outputting" and improving the real-time performance of content detection.
[0130] Combination of the above Figure 1 and Figure 2 The content detection method provided in the embodiment of the present application is introduced in detail. The system and device provided in the embodiment of the present application will be introduced in conjunction with the accompanying drawings.
[0131] See also Figure 3 The structural diagram of the content detection system shown in FIG. 30 includes:
[0132] A receiving module 301 is used to continuously receive output content streamed from a language model, wherein the output content includes a plurality of reply characters;
[0133] A determination module 302 is used to determine content slices from the output content that has been output according to a set sampling strategy;
[0134] A detection module 303 is used to perform content detection on the content slice to obtain a detection result of the content slice;
[0135] The transmission module 304 is used to transmit the output content that has undergone content detection to the client, so that the output content is displayed on the client in a streaming output manner.
[0136] In some possible implementations, the determining module 302 is specifically configured to:
[0137] According to a set sliding window and / or a set step, a content slice is determined from the output content that has been output; wherein the sliding window is used to determine the number of reply characters in the content slice, and the step is used to determine the difference between content slices of two adjacent content detections.
[0138] In some possible implementations, the determining module 302 is specifically configured to:
[0139] According to a set step, a set sliding window is moved backward in a character sequence; wherein the character sequence is composed of a plurality of reply characters in the output content that has been output;
[0140] In response to the number of the reply characters in the sliding window reaching the length of the sliding window, the reply characters in the sliding window are determined as content slices.
[0141] In some possible implementations, the determining module 302 is further configured to:
[0142] In response to the language model starting to stream output the output content, determining that the position of the sliding window in the character sequence is the front end of the character sequence;
[0143] In response to the number of the reply characters in the sliding window reaching a first preset length, the reply characters in the sliding window are determined as content slices for a first content detection, wherein the first preset length is less than the length of the sliding window.
[0144] In some possible implementations, after completing the first content detection, the determination module 302 is further configured to:
[0145] Maintaining the position of the sliding window in the character sequence;
[0146] In response to the number of the reply characters in the sliding window reaching the length of the sliding window, the reply characters in the sliding window are determined as content slices for a second content detection.
[0147] In some possible implementations, the detection module 303 is specifically configured to:
[0148] The content slice is input into a content detection model, and a detection result output by the content detection model is received; wherein the content detection model has natural language analysis capability.
[0149] In some possible implementations, the determining module 302 is further configured to:
[0150] Based on the detection result of the content slice, determine the processing strategy for the content slice; wherein the processing strategy includes at least one of the following: a strategy for indicating whether the language model continues to stream the output content, a strategy for indicating whether to present the output content, and a strategy for indicating to present the content.
[0151] In some possible implementations, the detection result indicates that the content slice passes the content detection, and the determination module 302 is specifically configured to:
[0152] instructing the language model to continue streaming the output content; and,
[0153] Instruct the client to present a target reply character in a conversation page; wherein the conversation page is used to interact with a digital assistant, and the target reply character is a reply character in the content slice that is not presented in the conversation page.
[0154] In some possible implementations, the detection result indicates that the content slice fails the content detection, and the determination module 302 is specifically configured to:
[0155] instructing the language model to stop streaming the output content; and,
[0156] The client is instructed to replace the reply characters presented in the conversation page with default characters.
[0157] The content detection system 30 according to the embodiment of the present application may correspond to executing the method described in the embodiment of the present application, and the above and other operations and / or functions of each module / unit of the content detection system 30 are respectively to implement Figure 1 or Figure 2 For the sake of brevity, the corresponding processes of each method in the illustrated embodiment are not described in detail here.
[0158] The present application also provides an electronic device. The electronic device is specifically used to implement Figure 3 Functionality of content detection system 30 in the illustrated embodiment.
[0159] Figure 4 A schematic diagram of the structure of an electronic device 400 is provided. Figure 4 As shown, the electronic device 400 includes a bus 401, a processor 402, a communication interface 403 and a memory 404. The processor 402, the memory 404 and the communication interface 403 communicate with each other via the bus 401.
[0160] The bus 401 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0161] The processor 402 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).
[0162] The communication interface 403 is used for communicating with the outside. For example, the communication interface 403 can be used for communicating with a terminal.
[0163] The memory 404 may include a volatile memory, such as a random access memory (RAM). The memory 404 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).
[0164] The memory 404 stores executable codes, and the processor 402 executes the executable codes to perform the aforementioned content detection method.
[0165] Specifically, in implementing Figure 3 In the case of the embodiment shown, and Figure 3 When each module or unit of the content detection system 30 described in the embodiment is implemented by software, the execution Figure 3 The software or program code required for the functions of each module / unit in the system may be partially or completely stored in the memory 404. The processor 402 executes the program code corresponding to each unit stored in the memory 404 to perform the aforementioned content detection method.
[0166] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the above-mentioned file processing method applied to the content detection system 40.
[0167] The embodiment of the present application further provides a computer program product, which includes one or more computer instructions. When the computer instructions are loaded and executed on a computing device, the process or function described in the embodiment of the present application is generated in whole or in part.
[0168] The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer or data center to another website, computer or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.
[0169] When the computer program product is executed by a computer, the computer executes any of the aforementioned file processing methods. The computer program product may be a software installation package, and when any of the aforementioned file processing methods is needed, the computer program product may be downloaded and executed on a computer.
[0170] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to each embodiment of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0171] The units involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the unit / module does not, in some cases, constitute a limitation on the unit itself.
[0172] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.
[0173] In the context of the present application embodiment, machine-readable medium can be a tangible medium that can contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable medium can include but is not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the above. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.
[0174] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the system or device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.
[0175] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0176] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0177] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
[0178] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A content detection method, It is characterized in that The method comprises: Continuously receive the output content of the language model streaming output; According to the set sampling strategy, content slices are determined from the output content that has been output; content slices Performing content detection on the content slice to obtain a detection result of the content slice; The output content that has undergone content detection is transmitted to a client, so that the output content is displayed on the client in a streaming output manner.
2. The method according to claim 1, It is characterized in that The step of determining content slices from the output content that has been output according to the set sampling strategy includes: According to a set sliding window and / or a set step, a content slice is determined from the output content that has been output; wherein the sliding window is used to determine the number of reply characters in the content slice, and the step is used to determine the difference between content slices of two adjacent content detections.
3. The method according to claim 2, It is characterized in that The determining of content slices from the output content that has been output according to the set sliding window and / or the set step includes: According to a set step, a set sliding window is moved backward in a character sequence; wherein the character sequence is composed of a plurality of reply characters in the output content that has been output; In response to the number of the reply characters in the sliding window reaching the length of the sliding window, the reply characters in the sliding window are determined as content slices.
4. The method according to claim 3, It is characterized in that The method further comprises: In response to the language model starting to stream output the output content, determining that the position of the sliding window in the character sequence is the front end of the character sequence; In response to the number of the reply characters in the sliding window reaching a first preset length, the reply characters in the sliding window are determined as content slices for a first content detection, wherein the first preset length is less than the length of the sliding window.
5. The method according to claim 4, It is characterized in that After completing the first content detection, the method further includes: Maintaining the position of the sliding window in the character sequence; In response to the number of the reply characters in the sliding window reaching the length of the sliding window, the reply characters in the sliding window are determined as content slices for a second content detection.
6. The method according to claim 1, It is characterized in that The performing content detection on the content slice to obtain the detection result of the content slice includes: The content slice is input into a content detection model, and a detection result output by the content detection model is received; wherein the content detection model has natural language analysis capability.
7. The method according to claim 1, It is characterized in that The method further comprises: Based on the detection result of the content slice, determine the processing strategy for the content slice; wherein the processing strategy includes at least one of the following: a strategy for indicating whether the language model continues to stream the output content, a strategy for indicating whether to present the output content, and a strategy for indicating to present the content.
8. The method according to claim 7, It is characterized in that The detection result indicates that the content slice passes the content detection, and the determining, according to the detection result of the content slice, a processing strategy for the content slice includes: instructing the language model to continue streaming the output content; and, Instruct the client to present a target reply character in a conversation page; wherein the conversation page is used to interact with a digital assistant, and the target reply character is a reply character in the content slice that is not presented in the conversation page.
9. The method according to claim 7, It is characterized in that The detection result indicates that the content slice fails the content detection, and the determining, according to the detection result of the content slice, a processing strategy for the content slice includes: instructing the language model to stop streaming the output content; and, The client is instructed to replace the reply characters presented in the conversation page with default characters.
10. A content detection system, It is characterized in that The system comprises: A receiving module, used to continuously receive the output content of the language model streaming output; A determination module is used to determine content slices from the output content that has been output according to a set sampling strategy; content slices A detection module, used to perform content detection on the content slice and obtain a detection result of the content slice; The transmission module is used to transmit the output content that has undergone content detection to the client, so that the output content is displayed on the client in a streaming output manner.
11. An electronic device, It is characterized in that The electronic device comprises a processor and a memory; The processor is configured to execute instructions stored in the memory, so that the electronic device performs the method according to any one of claims 1 to 9.
12. A computer-readable storage medium, It is characterized in that The method comprises instructions, wherein the instructions instruct an electronic device to execute the method as claimed in any one of claims 1 to 9.
Citation Information
Cited By
Content testing method, system, device, and medium
WO2026016553A1
