A method, system and electronic device for security detection of streaming generated content

Abstract

This specification discloses a security detection method, system, and electronic device for streaming content. The method includes: during the continuous acquisition of streaming content, segmenting the acquired content into corresponding content units according to preset content segmentation rules; upon detecting a newly generated content unit, determining its corresponding risk probability value, and updating a sliding window based on the determined risk probability value; calculating a cumulative risk score using a preset aggregation function based on the risk probability values ​​of each content unit currently contained in the sliding window; and executing a corresponding security handling strategy on the streaming content based on the cumulative risk score. This specification embodiment aggregates and calculates the risk probability values ​​of multiple content units using a sliding window, reducing false alarms caused by fluctuations in single-point judgments, while also identifying cumulative risks scattered across multiple content units, meeting the real-time requirements of streaming scenarios.

Description

Technical Field

[0001] This specification relates to the field of content security detection technology, specifically to a security detection method, system, and electronic device for streaming content. Background Technology

[0002] With the rapid development of content generation technology, various content generation services are widely used in scenarios such as dialogue interaction, content creation, and information retrieval. In these scenarios, generated content is often produced and presented to users gradually in a streaming manner, rather than being output all at once after all content has been generated. To ensure the security of the output content, real-time security detection is required during the content generation process so that intervention measures can be taken in time before risky content is presented to users.

[0003] Currently, security detection for streamed content primarily employs a successive detection scheme based on independent content units. This scheme independently classifies each newly generated content unit for security, outputs the risk probability value of that unit, and compares it with a preset threshold to determine whether the unit is secure. However, this scheme has the following two shortcomings.

[0004] On the one hand, since the judgment of a security classification model on a single content unit inevitably has random fluctuations, a single accidental probability fluctuation may lead to a misjudgment, causing actually safe content to be blocked incorrectly, or content that is actually risky to be allowed to pass incorrectly.

[0005] On the other hand, some risks are not generated independently by a single content unit, but rather by the combined effect of multiple content units. In such scenarios, each content unit individually may pose a low level of risk, but the combination of multiple content units constitutes a clear risk. Because the successive detection scheme assesses each content unit in isolation and cannot perceive the risk correlation between multiple content units, it is prone to underestimating risks arising from such combinations.

[0006] In summary, how to reduce misjudgments caused by fluctuations in single-point judgments and effectively identify risks generated by multiple content units in the security detection of streaming content remains a technical problem that needs further research in this field. Summary of the Invention

[0007] The purpose of the embodiments in this specification is to provide a method, system and electronic device for security detection of streaming content, so as to solve the problem that the security detection scheme for streaming content in the related art is prone to misjudgment due to single-point fluctuations because it makes independent judgment on individual content units.

[0008] According to a first aspect of this specification, a method for security detection of streamed content is provided, comprising: During the continuous acquisition of streaming content, the acquired content is segmented according to preset content segmentation rules to generate corresponding content units; Upon detecting a newly generated content unit, its corresponding risk probability value is determined, and the sliding window is updated according to the determined risk probability value. The sliding window is used to retain a preset number of recently generated content units and their corresponding risk probability values. Based on the risk probability value of each content unit currently contained in the sliding window, a cumulative risk score is calculated using a preset aggregation function; Based on the accumulated risk score, a corresponding security handling strategy is implemented for the streaming content.

[0009] According to a second aspect of this specification, a security detection system for streaming content is provided, comprising: The segmentation unit is configured to segment the acquired content according to a preset content segmentation rule to generate corresponding content units during the continuous acquisition of streaming content. The probability determination and window update unit is configured to determine the risk probability value corresponding to a newly generated content unit when the newly generated content unit is detected, and update the sliding window according to the determined risk probability value. The sliding window is used to retain a preset number of recently generated content units and their corresponding risk probability values. The cumulative calculation unit is configured to calculate a cumulative risk score based on the risk probability value of each content unit currently contained in the sliding window using a preset aggregation function; The processing execution unit is configured to execute a corresponding security processing strategy on the streaming content based on the accumulated risk score.

[0010] According to a third aspect of this specification, an electronic device is provided, including at least one processor and a memory communicatively connected to said at least one processor. The memory stores instructions executable by said at least one processor to enable the at least one processor to perform the method described in the first aspect of this specification.

[0011] According to a fourth aspect of this specification, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the steps of the method as described in the first aspect of this specification.

[0012] According to a fifth aspect of this specification, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the steps of the method as described in the first aspect of this specification.

[0013] The technical solutions provided in the embodiments of this specification have the following beneficial effects: By calculating a cumulative risk score using a preset aggregation function based on the risk probability values ​​of each content unit currently contained in the sliding window, the aggregation operation integrates the risk probability values ​​of multiple content units within the window. This means that fluctuations in the judgment of a single content unit are balanced by the values ​​of other content units during the aggregation process, thus reducing misjudgments caused by fluctuations in single-point judgments. Furthermore, since the cumulative risk score comprehensively reflects the overall risk level of multiple content units within the window, when multiple content units jointly generate risk, even if the individual risk probability value of each content unit is low, their combined effect can still be reflected in the cumulative risk score. This allows for accurate detection of risks jointly generated by multiple content units, reducing missed judgments.

[0014] By updating the sliding window when a new content unit is detected, the sliding window always retains a preset number of recently generated content units and their corresponding risk probability values. Since the capacity of the sliding window is a preset number, no matter how much the total amount of streaming content grows, the amount of data to be processed for each calculation remains constant, thereby keeping storage and computational overhead within a fixed range and meeting the real-time requirements of streaming scenarios. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0016] Figure 1 The main flowchart of the security detection method for streaming content provided in the embodiments of this specification is shown below. Figure 2 A schematic diagram of the security detection framework for streaming content provided in the embodiments of this specification; Figure 3 This is a schematic diagram of multi-level threshold decision-making and state transition provided in the embodiments of this specification; Figure 4 This is a schematic diagram illustrating parallel processing of multiple risk categories provided in the embodiments of this specification; Figure 5 A schematic diagram of the security detection system for streaming content provided in the embodiments of this specification; Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this specification. Detailed Implementation

[0017] Before detailing the embodiments described in this specification, it should be understood that although specific embodiments are described herein, this does not limit the scope of protection of the embodiments described herein. The embodiments described herein can be applied to various security detection scenarios for streamed content, including but not limited to text-based dialogue interaction, speech synthesis streaming output, and video stream generation.

[0018] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties. The collection, use and processing of related data shall comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals shall be provided for users to choose to authorize or refuse.

[0019] To facilitate a thorough understanding of the technical solutions presented in this specification by those skilled in the art, several key concepts involved in the embodiments of this specification are first explained in a unified manner. These key concepts form the basis of the technical solutions presented in this specification and are applied throughout the subsequent descriptions of the various embodiments.

[0020] Streaming content, in this specification, can be understood as a sequence of data generated progressively in a streaming manner by a content generation model or generation service. Unlike the mode where the complete content is output all at once after all content has been generated, streaming content can be generated and transmitted character by character or segment by segment over time, and the receiving end can begin receiving and processing the content before it is fully generated. Streaming content can include data in various modalities such as text, audio, and video.

[0021] In this specification, a content unit can be understood as the basic processing unit obtained after segmenting the streamed content. By segmenting a continuous stream of data according to preset content segmentation rules, a series of semantically or temporally independent content units can be obtained. In a text scenario, a content unit can be a sentence or a paragraph; in an audio scenario, a content unit can be a speech segment divided by a silence interval; in a video scenario, a content unit can be a video segment divided by a scene transition point.

[0022] In this specification, the risk probability value can be understood as a numerical value used to characterize the confidence level that a single content unit possesses content risk. This value is typically output by a security classification model and ranges from 0.0 to 1.0. A higher value indicates a higher confidence level that the content unit contains risky content. Content risk can include various types such as harmful content, biased content, and illegal or non-compliant content.

[0023] In this specification, a sliding window can be understood as a caching structure used to maintain a fixed-capacity data set in a continuous stream of content. As new content units are continuously generated, new units can be added to the caching structure, while the oldest units can be removed, thus ensuring that the caching structure always retains a preset number of recently generated content units and their corresponding risk probability values.

[0024] In this specification, the cumulative risk score can be understood as a comprehensive risk quantification value obtained by aggregating the risk probability values ​​of multiple content units within a sliding window. This score reflects the overall risk level of multiple content units within the window, rather than the instantaneous risk status of any single unit.

[0025] In this specification, a security handling strategy can be understood as the response measures taken in response to detected risks. Security handling strategies can be fixed measures or tiered according to the degree of risk. For example, allowing content to continue generating, triggering risk mitigation actions, or executing risk interception responses.

[0026] Based on an understanding of the above key concepts, the technical solutions of the embodiments of this specification will be described in detail below with reference to the accompanying drawings.

[0027] Figure 1 This is the main flowchart of the security detection method for streaming content provided in the embodiments of this specification. Figure 1 As shown, the method may include the following steps.

[0028] Step S101: During the continuous acquisition of streaming content, the acquired content is segmented according to preset content segmentation rules to generate corresponding content units.

[0029] Specifically, during the continuous output of streaming content, it can exist in the form of a character stream or a frame stream. The obtained content needs to be segmented according to preset content segmentation rules in order to subsequently conduct risk assessments on the segmented content units.

[0030] There are various ways to implement content segmentation rules. In one implementation, segmentation can be achieved by detecting semantic boundary markers in the streamed content. When such a semantic boundary marker is detected, the currently accumulated content can be output as a single content unit. The specific form of the semantic boundary marker can be flexibly chosen according to the modality of the content. In text generation scenarios, semantic boundary markers may include at least one of sentence-ending punctuation marks (such as periods, question marks, and exclamation marks), paragraph endings, or line breaks. In audio generation scenarios, semantic boundary markers may be silence intervals. In video generation scenarios, semantic boundary markers may be scene transition points. This specification does not limit the specific implementation method of the content segmentation rules.

[0031] To better understand the above segmentation process, let's take a text scenario as an example for tracing and explanation. Assume the content generator outputs a dialogue response sequentially in a streaming manner. The complete content is: "This topic involves multiple aspects. From a historical perspective, there are some viewpoints worth exploring. It should be noted that people's understanding of this issue may differ across different cultural backgrounds. Some viewpoints may be controversial, but are worth discussing from an academic perspective." When the first period is detected, "This topic involves multiple aspects." will be treated as a content unit. Output. When the second period is detected, "From a historical perspective, there are some points worth exploring." will be output as the content unit. Output. When the third period is detected, the sentence "It should be noted that people's understanding of this issue may differ across different cultural backgrounds" will be output as the content unit. Output. When the fourth period is detected, the sentence "Some viewpoints may be controversial, but are worth discussing from an academic perspective." will be output as a content unit. Output. If the content generation end continues to output new sentences, then generate them sequentially. , These are content units. These content units will serve as the basis for subsequent risk assessments.

[0032] by Figure 2 The above segmentation process will be further explained using an example. Figure 2 This is a schematic diagram of a security detection framework for streaming content provided in the embodiments of this specification. Figure 2 As shown in the "Content Generation and Segmentation Stage," the main nodes involved in this stage include a content generation end and a content segmentation module. The content generation end can be a hardware or software module used to perform streaming content generation. During operation, the content generation end outputs the generated content step-by-step in a streaming manner. In one embodiment, the content generation end can be a large language model, which generates response text word-by-word based on user input prompts and transmits the generated text content to the downstream detection module in real time in a streaming manner. In another embodiment, the content generation end can also be a speech synthesis engine or a video generation model. This specification does not limit the specific type of content generation end. The content segmentation module can receive the content stream output by the content generation end and segment it according to preset content segmentation rules, dividing the continuous data stream into various content units. Data transmission between the content segmentation module and the content generation end can be real-time; that is, the content segmentation module can detect and process each character or frame output by the content generation end.

[0033] After segmenting the content units, it is necessary to determine their risk probability values ​​and maintain the sliding window. The following describes step S102.

[0034] Step S102: When a newly generated content unit is detected, its corresponding risk probability value is determined, and the sliding window is updated according to the determined risk probability value. The sliding window is used to retain a preset number of recently generated content units and their corresponding risk probability values.

[0035] Specifically, after detecting a newly generated content unit, the security classification model can extract and evaluate the risk features of that unit, and output the risk probability value corresponding to that content unit. The specific architecture of the security classification model does not constitute a limitation on the scheme described in this specification, as long as it can output continuous risk probability values ​​for the input content units. For example, the security classification model can be a binary classifier fine-tuned based on a pre-trained language model, a classifier based on a lightweight Transformer architecture, or a sequence classifier based on a recurrent neural network. This specification does not limit the specific architecture of the security classification model.

[0036] Continuing with the previous example, let's assume that the security classification model applies to content units. to The output risk probability values ​​are as follows: , , , , , It can be observed that as the dialogue progresses, the risk probability value of the content unit shows a gradual upward trend.

[0037] After determining the risk probability value corresponding to the newly generated content unit, the system updates the sliding window based on the determined risk probability value. The sliding window has a preset capacity. .by For example, at time At that time, the sliding window retains the 5 most recently generated content units and their corresponding risk probability values, i.e. The corresponding risk probability values ​​are respectively , , , , .

[0038] At time Newly generated content unit detected And its risk probability value Once determined, the tuple It can be added to a sliding window. Meanwhile, to maintain a fixed window size of 5, the oldest tuple... It can be moved out of the window, creating a sliding window. At this time, the data within the sliding window is updated. The corresponding risk probability value sequence is updated to Since the sliding window has a preset capacity, the amount of data stored in the window remains constant regardless of how much the total amount of content generated by streaming increases, thus keeping storage overhead within a fixed range.

[0039] When a window reaches its capacity and content units need to be removed, in addition to removing the earliest content units in chronological order, a selective removal strategy based on retention priority scores can also be adopted. In one implementation, the retention priority score of each historical content unit in the sliding window can be calculated, and the historical content unit with the lowest score and its corresponding risk probability value can be removed first.

[0040] The determination of priority scores can take into account a variety of factors. Each factor will be explained below.

[0041] The first factor is the risk probability value of the historical content unit itself. This factor is based on the premise that the higher the risk probability value of a content unit, the stronger the risk signal it contains, and the greater its contribution to the subsequent accumulation of risk; therefore, it should be given a higher priority for retention. For example, continuing the previous example, if within the window... The risk probability value is , The risk probability value is ,but The contribution from this factor to the retention priority score is higher than .

[0042] The second factor is the semantic relevance between historical content units and newly generated content units. This factor is based on the premise that historical content units with a closer semantic connection to the current content unit are more likely to jointly constitute a risk context with the current content unit, and therefore should be prioritized for retention. For example, if the newly generated content unit... The discussion focuses on topics related to "historical perspectives," while the window contains... ("From a historical perspective, there are some viewpoints worth exploring.") If they are highly related semantically, then The contribution from this factor is relatively high in the retention priority score. Semantic relevance can be measured by calculating the cosine similarity between text vectors, or it can be determined by other semantic matching methods. This specification does not limit the specific calculation method of semantic relevance.

[0043] The third factor is the historical contribution record of content units in historical risk assessment. This factor is based on the following: if a historical content unit has significantly influenced the final cumulative risk score in multiple past cumulative risk assessments (e.g., it has been given a high aggregation weight), it indicates that the unit has a high risk-building capability and should be given a higher retention priority. For example, if... ("It should be noted that people's understanding of this issue may differ in different cultural contexts.") In previous window aggregation calculations, it was given a high weight by the time decay mechanism or the attention mechanism, so its contribution record was better and the retention priority score was correspondingly improved.

[0044] It should be noted that the three factors mentioned above are merely illustrative examples, and this specification does not limit the factors for determining the priority score. In practical applications, other factors can be introduced according to business needs, or different weights can be assigned to the above factors for comprehensive calculation.

[0045] The system can calculate the retention priority score for each historical content unit by considering one or more of the above factors, and then prioritize removing the historical content unit with the lowest retention priority score. This selective removal strategy allows the sliding window to retain more content units that are valuable for current risk assessment within its limited capacity, thus improving the quality of information within the window.

[0046] by Figure 2 Let's take an example to further illustrate the above process. For example... Figure 2 As shown in the "Risk Assessment Phase" and "Sliding Window and Cumulative Calculation Phase," after the content segmentation module outputs a new content unit, this unit is passed to the security classification model for risk assessment. The output risk probability value is then fed into the sliding window. The sliding window maintains the most recent... Each content unit and its corresponding risk probability value are displayed. When a new unit is added, the oldest unit is removed, resulting in a continuously sliding window.

[0047] After the sliding window is updated, the risk probability values ​​within the window need to be aggregated and calculated. The following explains step S103.

[0048] Step S103: Calculate the cumulative risk score based on the risk probability value of each content unit currently contained in the sliding window using a preset aggregation function.

[0049] Specifically, the risk probability value of a single content unit only reflects the risk signal at that instant. To reduce the impact of fluctuations in single-point judgments, it is necessary to comprehensively calculate multiple probability values ​​within the window. Furthermore, some risks are not generated independently by a single content unit, but rather by the combined effect of multiple content units—each content unit individually has a low risk level, but the combination of multiple content units constitutes a clear risk. By aggregating the risk probability values ​​of multiple content units within the window, the combined risk effect can be reflected, thereby accurately detecting joint risks. At time... The sliding window retains the most recently generated [data / data]. The data structure of a content unit and its corresponding risk probability value can be represented as follows: in, Indicates the first Each content unit This indicates the corresponding risk probability value. The range of values ​​is to The above text is from a time when... Window capacity For example, the current sliding window retains content units. The corresponding risk probability value sequence is There are various ways to implement aggregate functions, which aim to quantify the overall risk level within a window.

[0050] In one implementation, the aggregation function can employ a weighted aggregation method. A weight coefficient is assigned to each content unit within the window, and the cumulative risk score is obtained by multiplying each risk probability value by its corresponding weight and then summing the results. The calculation process can be represented as follows: in, For the sliding window capacity, For the first in the window Risk probability value for each content unit The corresponding weights are defined, and each weight satisfies the normalization constraint. Through this aggregation operation, the fluctuations in the judgment of a single content unit are balanced by the values ​​of other content units during the summation process, thereby reducing misjudgments caused by fluctuations in single-point judgments.

[0051] To more intuitively understand the effect of aggregation operations, two comparative examples are provided below.

[0052] Example 1 (High single-point probability value but low overall score): Suppose the security classification model is applied to a certain content unit. It output a high risk probability value. (Possibly due to random fluctuations in the classifier), while the risk probability values ​​for the other four content units within the window are respectively , , , If uniform weights are used (each weight is 1), Then the cumulative risk score This result is far lower than the single-point probability value. It is also lower than the preset observation threshold (e.g.) Therefore, the system determines it is safe and allows content generation to continue. This example illustrates that aggregation operations can effectively smooth out random fluctuations at a single point and avoid false positives caused by classifier noise.

[0053] Example 2 (Multiple individual probability values ​​are low, but the overall score is high): Assume the risk probability values ​​of the five content units within the window are as follows: , , , , Each single-point probability value is lower than the single-point risk threshold (e.g.) If each point is judged independently, no alarm will be triggered. However, if uniform weighting is used for aggregation, the cumulative risk score will be calculated. The result reached the preset observation threshold. The system triggers a risk mitigation operation. This example illustrates that aggregation operations can capture the gradual accumulation trend of risk signals from multiple content units, identifying progressive risks that cannot be detected by individual point-by-point assessments.

[0054] There are several ways to determine the weights. In one implementation, the weights can be negatively correlated with the time difference between the generation time of the corresponding content unit and the current time. That is, the closer the content unit was generated to the current time, the higher the weight is assigned, so that the cumulative risk score can more sensitively reflect the risk status of the latest content. This time-decay weighted sum can be determined using the exponential decay formula: in, For time decay parameters, This represents the time offset of the content unit from the current time. The range of values ​​is to . The larger the value, the more prominent the weight of recent content units, and the more sensitive the aggregation results are to changes in the latest content; The smaller the value, the closer the weight of each content unit is to a uniform distribution, and the smoother the aggregation result. (With the current window...) For example, if set ,but The most recently generated unit has the highest weight. The oldest unit has the smallest weight. Continuing the previous example, after normalization, from... arrive The weights are approximately as follows: , , , , According to formula (2), the cumulative risk score is approximately .because , The risk probability value is relatively high and is closer to the current time, so its weight is relatively large, and the aggregation result can more sensitively reflect the trend of risk escalation.

[0055] In another implementation, a lightweight attention network can be introduced to process the representations of each content unit and dynamically learn the contribution weight of each content unit to the current overall risk. Specifically, this can be achieved using a small neural network. The representation of the content unit is processed, and the attention weights are obtained by normalization using the Softmax function: in, For the first in the window Lightweight semantic representation of each content unit This means that the semantic representation and the risk probability value are concatenated and then input into the network. , The range of values ​​is to Based on attention weights, the cumulative risk score can be calculated as follows: Since the number of parameters in an attention network is much smaller than that in a safety classification model, the additional computational overhead it introduces can be kept at a low level.

[0056] The aforementioned weighted aggregation performs well in smoothing noise, but when a content unit itself carries an extremely high risk signal, if conventional weighted aggregation is still used, this extreme value may be diluted by the probabilities of other low-risk units within the window. To balance trend accumulation and peak capture, the weighted aggregation function can be further designed as a combination function. Specifically, the weighted aggregation result can be combined with the maximum risk probability value in the sliding window to obtain the cumulative risk score: in, This refers to the combined weighting coefficients. By introducing a maximum value term, even if other units have low risks, the cumulative risk score will be effectively boosted if an extremely high risk probability value appears within the window. Continuing the previous example, if the maximum risk probability value within the window is... The weighted aggregation result is approximately ,Pick The cumulative risk score is approximately The results are closer to the actual risk level within the window than those obtained by simple weighted aggregation.

[0057] In another implementation, the aggregation function can also be a maximum value function. That is, the maximum risk probability value among all the content units currently contained in the sliding window is directly selected as the cumulative risk score: Taking the current window as an example, in the probability value sequence In the middle, the maximum value is Therefore, the cumulative risk score is This approach has extremely low computational overhead and focuses on short-term risk peaks, making it suitable for deployment scenarios that are highly sensitive to extreme single-point risks and have limited computing resources.

[0058] The choice of aggregation function can be configured based on the risk characteristics of the business scenario, and different aggregation functions can be used for different risk categories. This specification does not restrict the specific form of the aggregation function.

[0059] by Figure 2 The above aggregation calculation process will be further explained using an example. Figure 2 As shown in the “Sliding Window and Cumulative Calculation” section, the aggregation calculation module reads the risk probability value sequence of each content unit currently contained in the sliding window, and performs comprehensive calculation on these values ​​through a preset aggregation function to output the cumulative risk score. Figure 2 The diagram simultaneously illustrates the data flow and output of three parallel computation paths: weighted aggregation, attention aggregation, and maximum value extraction.

[0060] After obtaining the cumulative risk score, it is necessary to implement the corresponding security measures. The following describes step S104.

[0061] Step S104: Based on the accumulated risk score, execute the corresponding security handling strategy on the streaming content.

[0062] Specifically, the cumulative risk score can be compared with a preset threshold, and corresponding decision instructions can be generated based on the comparison results. In its most basic implementation, the decision-making process can be represented as: in, A preset threshold is set. When the accumulated risk score does not exceed the threshold, it is considered safe (SAFE), and content output is allowed to continue; when the accumulated risk score exceeds the threshold, an alarm (ALARM) or intervention action (INTERVENE) is triggered.

[0063] Since the cumulative risk score is a continuous scalar value, there are multiple ways to implement the safety response strategy. The safety response strategy can be a fixed measure or it can be graded according to differences in risk level. In one implementation, a multi-level threshold-based response mechanism can be used. Specifically, two thresholds can be configured: an observation threshold and a risk level threshold. and interception threshold The observation threshold is lower than the interception threshold. Different intervention intensities are implemented based on the range of the cumulative risk score.

[0064] Figure 3 This is a schematic diagram illustrating multi-level threshold decision-making and state transitions. For example... Figure 3 As shown, the decision space is divided into three state intervals. State interval 301 corresponds to a cumulative risk score less than the observation threshold. A safe state; state interval 302 corresponds to a cumulative risk score greater than or equal to the observation threshold. And less than the interception threshold The observed state; state interval 303 corresponds to a cumulative risk score greater than or equal to the interception threshold. The interception state.

[0065] The cumulative risk score calculated in step S103 For example, assuming a preset observation threshold Interception threshold .because The current state transitions from the safe state to the observation state. In the observation state, risk mitigation actions are triggered. Specific actions for risk mitigation may include triggering manual review in the background, increasing the frequency of security checks for the session, or providing partial rewriting prompts for streamed content.

[0066] As streaming content generation continues, if the risk probability value of subsequent content units entering the sliding window remains high, causing the cumulative risk score to continue rising... At this point, the state will transition from the observation state to the interception state. In the interception state, a risk interception response is executed. The intervention intensity of the risk interception response is higher than that of the risk mitigation operation. Its specific actions may include interrupting the output stream of the content generation end, blocking content that has been generated but not yet presented to the user, or directly replacing it with a preset security response.

[0067] Conversely, if the risk probability value of subsequent content units entering the sliding window is low, the cumulative risk score will decrease as older high-risk units are removed. When the cumulative risk score drops to the observation threshold... In the following situation, the system can fall back from the observation state to the safe state and resume normal streaming generation and output.

[0068] The specific values ​​for the observation threshold and the blocking threshold can be configured according to the risk preferences of the business scenario. This manual does not impose any restrictions on the specific values ​​of the thresholds.

[0069] by Figure 2 Let's take an example to further illustrate the threshold decision-making process described above. Figure 2 As shown in the “Threshold Decision and Security Handling Process”, the cumulative risk score output by the aggregation calculation module is transmitted to the threshold decision module. The threshold decision module compares it with the preset threshold and generates a decision instruction. The response execution module executes the corresponding security handling strategy (allow or block / mitigate) on the streaming content according to the instruction.

[0070] The aforementioned multi-level decision-making mechanism based on cumulative risk scores performs well in smoothing noise and can effectively identify progressive risks. However, when a content unit already carries an extremely high-risk signal, if we wait for that unit to enter the sliding window and undergo aggregation calculation, this high-risk signal may be diluted by the probabilities of other low-risk units within the window. Therefore, a pre-judgment step can be added after determining the risk probability value corresponding to the newly generated content unit.

[0071] In this preliminary judgment stage, the determined risk probability value can be compared with the preset single-point risk threshold. Compare the risk probability values. When the determined risk probability value is greater than the single-point risk threshold... In such cases, a risk interception response is triggered directly, skipping the steps of updating the sliding window and subsequent aggregation calculations. For example, if the single-point risk threshold... When the risk probability value of a newly generated content unit reaches In this case, the system does not need to wait for the cumulative effect and immediately executes the interception action. This single-point circuit breaker mechanism ensures that even if the cumulative score has not exceeded the limit, the system can immediately block sudden and clearly risky content, complementing the cumulative risk detection mechanism.

[0072] In practical applications, the security risks faced by streaming content are often not single-dimensional, and may include multiple potential risks such as violence, pornography, and bias. The aforementioned embodiments mainly describe the cumulative assessment of a single risk category. When it is necessary to monitor multiple risk dimensions simultaneously, the system architecture can be further expanded.

[0073] Figure 4This is a diagram illustrating parallel processing of multiple risk categories. (Example) Figure 4 As shown, in one extended embodiment, the risk probability value output by the security classification model can include multiple values ​​corresponding to multiple risk categories. For example, for content units... Its risk probability value can be represented as a vector. ,in Indicates the probability of violence. This indicates the probability of bias risk.

[0074] The system sets up corresponding sliding windows for each of the multiple risk categories. Figure 4 The diagram illustrates three parallel category processing paths, each comprising three stages: a sliding window corresponding to the category, cumulative risk score calculation, and security handling strategy determination. The sliding window for any risk category is specifically used to cache the risk probability value sequence for that category. More importantly, each risk category corresponds to independent observation and interception thresholds. This means the system can customize threshold settings based on the characteristics of each risk category. For example, for highly sensitive categories, lower observation and interception thresholds can be set to make the system more sensitive to risk signals for that category; for less sensitive categories, higher thresholds can be set to reduce false interference with normal content.

[0075] When calculating the cumulative risk score, the system independently calculates the cumulative risk score for each risk category based on the corresponding sliding window. For example, for the probability sequence within the current window, the cumulative risk score for the violence category is calculated independently. Cumulative risk score for bias category Subsequently, based on the cumulative risk score corresponding to each risk category, a corresponding safety response strategy is determined independently.

[0076] When multiple risk categories trigger security response strategies simultaneously, the final execution actions need to be coordinated. The conflict arbitration module 411 is responsible for executing this coordination step. Specifically, if the triggered security response strategies differ in intervention intensity—for example, the violence category triggers a risk interception response while the bias category only triggers a risk mitigation operation—the conflict arbitration module 411 selects the strategy with the highest intervention intensity as the final execution action to ensure timely blocking of high-risk dimensions. If the triggered security response strategies do not differ in intervention intensity—for example, both trigger risk mitigation operations, but the specific actions matched are different (one triggers manual review, the other triggers partial content rewriting)—the conflict arbitration module 411 merges and executes the differentiated response actions matched to each risk category separately. This multi-dimensional parallel processing and coordination mechanism enables the security detection system to respond precisely to complex multi-risk scenarios.

[0077] The above description uses a text generation scenario as the main example. In this scenario, the large language model generates reply text sentence by sentence in a streaming manner based on the prompts input by the user. The embodiment in this specification continuously tracks changes in risk signals through a sliding window, triggering a mitigation operation when the accumulated risk score exceeds the observation threshold, thus buying time for backend review; and immediately interrupting output when the accumulated risk score exceeds the interception threshold, thereby ensuring the smoothness of the dialogue while maintaining a safety baseline.

[0078] In another application scenario, the speech synthesis engine outputs audio data segment by segment in a streaming manner. In this case, content segmentation rules can be based on the silence intervals between sentences, with each sentence segment treated as a content unit. The security classification model assesses the risk of each sentence segment and outputs the corresponding risk probability value. The subsequent sliding window accumulation and aggregation calculation process is consistent with the text scenario. In the video generation scenario, content segmentation rules can be based on scene transition points, with each scene segment treated as a content unit, and the subsequent processing flow is also applicable. These examples illustrate that the detection mechanism provided in the embodiments of this specification is not limited to the text modality, but is also applicable to audio, video, and other content generated in a streaming manner.

[0079] Based on the above-described security detection method for streaming content, this specification also provides a security detection system for streaming content. Figure 5 This is a schematic diagram of a security detection system for streaming content. Figure 5 As shown, this system can be deployed in the aforementioned electronic device as the execution entity for each method step, and includes: The segmentation unit 501 is configured to segment the acquired content according to a preset content segmentation rule to generate corresponding content units during the continuous acquisition of streaming content.

[0080] The probability determination and window update unit 502 is configured to determine the corresponding risk probability value when a newly generated content unit is detected, and update the sliding window according to the determined risk probability value. The sliding window is used to retain a preset number of recently generated content units and their corresponding risk probability values.

[0081] The cumulative calculation unit 503 is configured to calculate a cumulative risk score based on the risk probability value of each content unit currently contained in the sliding window using a preset aggregation function.

[0082] The processing execution unit 504 is configured to execute a corresponding security processing strategy on the streaming content based on the accumulated risk score.

[0083] Optionally, the cumulative calculation unit 503 is specifically used to: assign weights to the risk probability values ​​of each content unit and then perform weighted aggregation.

[0084] Optionally, the cumulative calculation unit 503 is specifically used to: make the weight of each content unit negatively correlated with the time difference between the generation time of the corresponding content unit and the current time; or, to dynamically learn and determine the weight after processing the representation of each content unit through an attention network.

[0085] Optionally, the cumulative calculation unit 503 is specifically used to: weight the aggregated result with the maximum risk probability value in the sliding window to obtain the cumulative risk score.

[0086] Optionally, the cumulative calculation unit 503 is specifically used to: select the maximum value of the risk probability values ​​of each content unit currently contained in the sliding window as the cumulative risk score.

[0087] Optionally, the system also includes: a circuit breaker response unit 505, used to compare the determined risk probability value with a single-point risk threshold, and directly trigger a risk interception response if the probability value is greater than the single-point risk threshold.

[0088] Optionally, the handling execution unit 504 is specifically used to: allow content to continue to be generated when the accumulated risk score is less than a preset observation threshold; trigger a risk mitigation operation when the score is greater than or equal to the observation threshold and less than a preset interception threshold; and execute a risk interception response when the score is greater than or equal to the interception threshold, wherein the observation threshold is less than the interception threshold and the intervention intensity of the risk interception response is higher than that of the risk mitigation operation.

[0089] Optionally, the probability determination and window update unit 502 is specifically used to: calculate the retention priority score of each historical content unit in the sliding window, and prioritize the removal of the historical content unit with the lowest retention priority score and its corresponding risk probability value, wherein the retention priority score is determined based on at least one factor among the historical content unit's own risk probability value, semantic relevance to the newly generated content unit, and contribution record in historical risk determination.

[0090] Optionally, the system also includes: a multi-category processing module 506, used to independently calculate the cumulative risk score for each of the multiple risk categories based on the corresponding values ​​contained in the risk probability value, and independently determine the corresponding safety handling strategy based on the cumulative risk score corresponding to each risk category; when multiple risk categories trigger safety handling strategies at the same time, if there are differences in the pre-intensity, the strategy with the highest intervention intensity is selected as the final execution action; if there are no differences in the pre-intensity, the differentiated handling actions matched with each risk category are merged and executed.

[0091] Optionally, the segmentation unit 501 is specifically used to: detect semantic boundary markers in the streaming content, and output the currently accumulated content as a content unit when semantic boundary markers are detected, wherein the semantic boundary markers include at least one of sentence-end punctuation marks, paragraph end marks, line breaks, silence intervals, and scene transition points.

[0092] The specific functional implementation, interaction logic, and technical effects of each unit in the system provided in this embodiment are consistent with those described in the foregoing method embodiments, and will not be repeated here. Those skilled in the art will understand that the above-mentioned units can be software modules, hardware modules, or modules combining software and hardware; their division is merely for the convenience of describing the technical solutions in this specification and does not constitute a limitation on the device structure.

[0093] This specification also provides an electronic device through its embodiments. Figure 6 This is a schematic diagram of the structure of an electronic device. For example... Figure 6 As shown, the electronic device may include at least one processor 601, a memory 602 communicatively connected to the at least one processor 601, and a communication interface 603. The processor 601, memory 602, and communication interface 603 are interconnected via a bus 604. The communication interface 603 is used to realize data communication between the electronic device and external devices or networks, such as receiving streaming content output from a content generation end, or returning security processing results to a client. The bus 604 may be a Peripheral Component Interconnect Standard (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 604 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0094] The memory 602 serves as a non-transitory computer-readable storage medium, storing data and computer program instructions that can be used to execute the security detection method for the aforementioned streamed content. The processor 601, by calling the instructions stored in the memory 602, can execute the various steps in the above method embodiments, thereby achieving real-time security detection and handling of the streamed content.

[0095] This specification also provides a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the various steps of the security detection method for streamed content described above. The storage medium may include read-only memory (ROM), random access memory (RAM), magnetic tape, and optical storage devices, etc.

[0096] This specification also provides a computer program product, including a computer program that, when executed by a processor, implements the various steps of the security detection method for streamed content described above. This computer program product can contain or store the program code on any form of tangible medium.

[0097] From an overall architecture perspective, the synergistic effect of the above steps forms a complete streaming security detection mechanism: Step S101 segments the streaming content into semantically complete content units, providing structured input for subsequent processing; Step S102 determines the risk probability value of each content unit and maintains a sliding window to retain short-term contextual information with fixed storage overhead; Step S103 combines multiple independent risk probability values ​​into a cumulative risk score through an aggregation function, smoothing classification noise while reducing single-point misjudgments; Step S104 transforms the cumulative risk score into a security handling decision, achieving refined risk management.

[0098] The foregoing description of specific embodiments of this specification enables those skilled in the art to implement or use the embodiments of this specification. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit and scope of the embodiments of this specification. Therefore, the embodiments of this specification are not limited to those shown, but are to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0099] What those skilled in the art will understand is: In this specification, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitation, the presence of additional identical or equivalent elements in a process, method, product, or apparatus that includes said elements is not excluded.

[0100] In this specification, “a,” “an,” and “the” do not specifically refer to the singular, but may also include the plural.

[0101] In this specification, ordinal numbers such as "first," "second," etc., do not necessarily indicate order; they are often used to distinguish between objects. For example, "first server" and "second server" usually refer to two servers. To differentiate between these two servers, they are described as "first server" and "second server." Of course, sometimes these two servers may be the same server.

[0102] In this specification, unless explicitly stated otherwise, "receiving and sending data" does not necessarily mean direct receiving and sending; it can also mean indirect receiving and sending. For example, A receiving data sent by B can be understood as A directly receiving the data sent by B, or it can be understood as A indirectly receiving the data sent by B through other entities such as C. Similarly, B sending data to A can be understood as B sending the data directly to A, or it can be understood as B indirectly sending the data to A through other entities such as C. Here, C can be one entity, or it can be two or more entities.

[0103] In this specification, unless explicitly stated otherwise, the relationships between structures can be direct or indirect. For example, when describing "A is connected to B," unless it is explicitly stated that A and B are directly connected, it should be understood that A can be directly connected to B or indirectly connected to B. Similarly, when describing "A is on top of B," unless it is explicitly stated that A is directly above B (AB is adjacent and A is above B), it should be understood that A can be directly above B or indirectly above B (AB is separated by other elements, and A is above B). And so on.

[0104] This specification uses specific terms to describe embodiments thereof. Terms such as "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described herein, as well as the features of those different embodiments or examples, without contradiction.

[0105] Although one or more embodiments of this specification provide method steps as described in the embodiments or flowcharts, it is understood that the order of steps listed in the embodiments or flowcharts is only one of many possible execution orders and does not represent the only execution order. Therefore, when the claims involve method steps, any changes or adjustments to the order of such steps, or the parallelism between steps, are also within the scope of protection of the claims.

Claims

1. A security detection method for streamed content, comprising: During the continuous acquisition of streaming content, the acquired content is segmented according to preset content segmentation rules to generate corresponding content units; Upon detecting a newly generated content unit, its corresponding risk probability value is determined, and the sliding window is updated according to the determined risk probability value. The sliding window is used to retain a preset number of recently generated content units and their corresponding risk probability values. Based on the risk probability value of each content unit currently contained in the sliding window, a cumulative risk score is calculated using a preset aggregation function; Based on the accumulated risk score, a corresponding security handling strategy is implemented for the streaming content.

2. The method according to claim 1, wherein, The preset aggregation function is a weighted aggregation function, and the calculation of the cumulative risk score using the preset aggregation function includes: The risk probability values ​​of each content unit currently contained in the sliding window are assigned weights and then aggregated in a weighted manner.

3. The method according to claim 2, wherein, The weight of each content unit is determined by any of the following methods: It is negatively correlated with the time difference between the generation time of the corresponding content unit and the current time; Alternatively, the representations of each content unit can be processed using an attention network and then dynamically learned to determine its composition.

4. The method according to claim 2, wherein, The weighted aggregation function is a combination function, and the calculation of the cumulative risk score using the preset aggregation function includes: The weighted aggregation result is combined with the maximum risk probability value in the sliding window to obtain the cumulative risk score.

5. The method according to claim 1, wherein, The preset aggregation function is a maximum value function, and the calculation of the cumulative risk score using the preset aggregation function includes: The maximum risk probability value among all content units currently contained in the sliding window is selected as the cumulative risk score.

6. The method according to claim 1, wherein, After determining the risk probability value corresponding to the newly generated content unit, the following is also included: Compare the determined risk probability value with the single-point risk threshold; If the determined risk probability value is greater than the single-point risk threshold, the risk interception response is triggered directly, skipping the update sliding window and subsequent steps.

7. The method according to claim 1, wherein, The step of implementing corresponding security measures for the streaming content based on the accumulated risk score includes: If the cumulative risk score is less than a preset observation threshold, content generation is allowed to continue; If the cumulative risk score is greater than or equal to the observation threshold and less than the preset interception threshold, a risk mitigation operation is triggered; wherein the observation threshold is less than the interception threshold. If the cumulative risk score is greater than or equal to the interception threshold, the risk interception response is executed, and the intervention intensity of the risk interception response is higher than that of the risk mitigation operation.

8. The method according to claim 1, wherein, The updated sliding window includes: Calculate the retention priority score of each historical content unit in the sliding window. The retention priority score is determined based on at least one of the following factors: the risk probability value of the historical content unit itself, the semantic relevance between the historical content unit and the newly generated content unit, and the contribution record of the historical content unit in the historical risk assessment. Prioritize removing the historical content unit with the lowest retention priority score and its corresponding risk probability value.

9. The method according to claim 1, wherein, The risk probability value includes multiple values ​​corresponding to multiple risk categories; the method further includes: For each of the multiple risk categories, the cumulative risk score is calculated independently based on the corresponding value contained in the risk probability value; Based on the cumulative risk score for each risk category, a corresponding safety response strategy is determined independently.

10. The method according to claim 9, wherein, When multiple risk categories trigger the safety response strategy simultaneously, the method further includes: If the triggered security response strategies differ in their intervention intensity, the strategy with the highest intervention intensity will be selected as the final action to be taken. If the triggered safety response strategies do not differ in intervention intensity, then the differentiated response actions matched to each risk category will be combined and executed.

11. The method according to claim 1, wherein, The preset content segmentation rule includes: detecting semantic boundary markers in the streaming content, and outputting the currently accumulated content as a content unit when the semantic boundary markers are detected; The semantic boundary markers include at least one of the following: sentence-end punctuation marks, paragraph end marks, line breaks, silence intervals, and scene transition points.

12. A security detection system for streamed content, comprising: The segmentation unit is configured to segment the acquired content according to a preset content segmentation rule to generate corresponding content units during the continuous acquisition of streaming content. The probability determination and window update unit is configured to determine the risk probability value corresponding to a newly generated content unit when the newly generated content unit is detected, and update the sliding window according to the determined risk probability value. The sliding window is used to retain a preset number of recently generated content units and their corresponding risk probability values. The cumulative calculation unit is configured to calculate a cumulative risk score based on the risk probability value of each content unit currently contained in the sliding window using a preset aggregation function; The processing execution unit is configured to execute a corresponding security processing strategy on the streaming content based on the accumulated risk score.

13. An electronic device, comprising: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1 to 11.

14. A computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the steps of the method as claimed in any one of claims 1 to 11.

15. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method as claimed in any one of claims 1 to 11.

Images