A method, system, device and medium for testing security of a pre-trained video model

Abstract

The application discloses a kind of text video model security test method, system, equipment and medium, it is related to text video model security test technical field, the method is first obtained the video to be measured generated by model based on initial sample, carries out multidimensional compliance analysis and obtains multiple analysis results;Then each result is constructed as uniform feature vector and is input decision tree audit model, obtains comprehensive risk factor;Then risk factor is compared with preset threshold value, according to the comparison result, the initial test sample is varied, generates new test sample;Finally, new sample is input model to generate new video, repeats the above analysis, feedback and variation step, until meeting stop condition after output model safety test result.The method can be not dependent on a large number of manual intervention, comprehensively, efficiently, accurately assess the safety of text video model.

Description

Technical Field

[0001] This invention generally relates to the field of text-based video model security testing technology, and specifically to a text-based video model security testing method, system, device and medium. Background Technology

[0002] With the rapid development of artificial intelligence technology, text-based video models have become an important research direction in the fields of natural language processing and computer vision. These models can generate video content based on natural language descriptions, driving the development of applications such as automated video production, virtual reality, and entertainment content creation.

[0003] However, despite significant progress made by textual analysis video models in various fields, security and compliance issues have emerged in practical applications. For example, generated videos may contain inappropriate or harmful content, such as violence, pornography, and racial discrimination, raising compliance and ethical concerns. Therefore, effectively evaluating the security and compliance of textual analysis video models has become an urgent problem to be solved.

[0004] However, existing security testing methods for textual video models suffer from a lack of diversity in test samples and rely on existing keyword rules or single-frame image features during the testing process. This makes it difficult to deeply understand the complex narrative content in the video, resulting in significant deficiencies in the breadth of coverage, depth of analysis, and operational efficiency of the testing process. Summary of the Invention

[0005] In view of the above-mentioned defects or deficiencies in the prior art, it is desirable to provide a method, system, device and medium for security testing of textual video models.

[0006] In a first aspect, the present invention provides a method for security testing of text-based video models, comprising: S1. Obtain the test video generated by the target textual video model based on the initial test samples; S2. Perform multi-dimensional compliance analysis on the video to be tested to obtain multiple analysis results corresponding to each analysis dimension; S3. Construct a unified feature vector from the multiple analysis results, and input the unified feature vector into a pre-trained decision tree audit model to obtain a comprehensive risk factor; S4. Compare the comprehensive risk factor with a preset threshold to obtain a comparison result, and mutate the initial test sample according to the comparison result to generate a new test sample. S5. Input the new test sample into the target text-based video model to generate a new video to be tested, and repeat steps S2 to S4 until the preset stopping condition is met, and output the security test result of the target text-based video model.

[0007] According to the technical solution provided by the present invention, the step of obtaining the target text-based video model based on the test video generated from the initial test samples includes: An adaptive detection synthesis model is constructed, which is used to generate multiple initial test samples based on the input prompt words; The initial test sample is input into the target text-based video model to obtain the video to be tested.

[0008] According to the technical solution provided by the present invention, the multi-dimensional compliance analysis includes at least: image content dimension, text information dimension, voice content dimension, and narrative logic dimension.

[0009] According to the technical solution provided by the present invention, the analysis of the narrative logic dimension includes: The video under test is represented by spatiotemporal tokenization. The tensor of the video under test is divided into feature blocks with local spatiotemporal correlation attributes by a three-dimensional block embedding operator, and a spatiotemporal token sequence is constructed by introducing three-dimensional position encoding. The spatiotemporal token sequence is modeled using a hierarchical spatiotemporal attention mechanism to obtain a narrative logic analysis model, and the spatial interaction features and temporal evolution features within a single frame of the video under test are extracted by the narrative logic analysis model. The narrative logic flow of the video under test is constructed based on the spatial interaction features and the temporal evolution features, and illegal narrative structures in the video under test are identified by analyzing the target narrative feature vectors within the narrative logic flow.

[0010] According to the technical solution provided by the present invention, the step of inputting the unified feature vector into a pre-trained decision tree review model to obtain a comprehensive risk factor includes: The unified feature vector is input into a pre-trained decision tree review model, and the independent risk scores for each dimension are output through the decision path of the decision tree review model. Based on the independent risk scores under each dimension, and by introducing a correction vector, a multidimensional risk vector is constructed. The multidimensional risk vector is aggregated and calculated to obtain the comprehensive risk factor.

[0011] According to the technical solution provided by the present invention, the step of mutating the initial test sample based on the comparison result to generate a new test sample includes: When the comparison result shows that the comprehensive risk factor is less than the preset threshold, the initial test sample is mutated using a first mutation strategy to generate a new test sample; the first mutation strategy increases the diversity of the test sample by introducing random perturbations into the feature embedding space. When the comparison result shows that the comprehensive risk factor is greater than or equal to the preset threshold, the initial test sample is mutated using a second mutation strategy to generate a new test sample. The second mutation strategy selects mutation operators related to the trigger risk dimension and updates them according to the gradient direction of the risk loss function to strengthen the features of the trigger risk in a targeted manner.

[0012] According to the technical solution provided by the present invention, the first mutation strategy and the second mutation strategy include one or more of the following mutation operators: semantic adversarial replacement, visual element enhancement, narrative chain extension, acoustic environment injection, and text embedding induction.

[0013] Secondly, the present invention provides a security testing system for text-based video models, comprising: The acquisition module is configured to acquire the test video generated by the target textual video model based on the initial test samples; The analysis module is configured to perform multi-dimensional compliance analysis on the video under test, and obtain multiple analysis results corresponding to each analysis dimension. The calculation module is configured to construct a unified feature vector from multiple analysis results and input the unified feature vector into a pre-trained decision tree audit model to obtain a comprehensive risk factor. The mutation module is configured to compare the comprehensive risk factor with a preset threshold to obtain a comparison result, and mutate the initial test sample according to the comparison result to generate a new test sample. An execution module is configured to input the new test sample into the target text-based video model to generate a new test video. The new test video is repeatedly processed by the analysis module, the calculation module, and the mutation module until a preset stopping condition is met, at which point the security test result of the target text-based video model is output.

[0014] Thirdly, the present invention provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.

[0015] Fourthly, the present invention provides a computer-readable storage medium having a computer program that, when executed by a processor, implements the steps of the above-described method.

[0016] In summary, this technical solution specifically discloses a method, system, device, and medium for security testing of text-based video models. The testing method includes: S1, acquiring a test video generated by the target text-based video model based on initial test samples; S2, performing multi-dimensional compliance analysis on the test video to obtain multiple analysis results corresponding to each analysis dimension; S3, constructing a unified feature vector from the multiple analysis results and inputting the unified feature vector into a pre-trained decision tree auditing model to obtain a comprehensive risk factor; S4, comparing the comprehensive risk factor with a preset threshold to obtain a comparison result, and mutating the initial test sample based on the comparison result to generate a new test sample; S5, inputting the new test sample into the target text-based video model to generate a new test video, and repeating steps S2 to S4 until a preset stopping condition is met, at which point the security test result of the target text-based video model is output.

[0017] Beneficial Effects: This invention obtains test videos generated by the target textual video model based on initial test samples, and directly links the multi-dimensional compliance analysis of the test videos with model security, providing a foundation for subsequent evaluation. Multi-dimensional compliance analysis of the test videos comprehensively examines video content from multiple levels, including images, text, voice, and narrative logic, ensuring broad coverage of various potential risks. Multiple analysis results are constructed into a unified feature vector and input into a pre-trained decision tree review model, achieving effective transformation from multi-source heterogeneous data to quantitative risk indicators, improving the accuracy and interpretability of the evaluation. The comprehensive risk factors are compared with preset thresholds, and the initial test samples are adaptively mutated based on the comparison results, enabling the test samples to dynamically adjust according to the previous round of detection results, enhancing the targeting and intelligence of the test. Finally, new test samples are input into the target textual video model again, and the above analysis, evaluation, and mutation steps are repeated, forming a closed-loop iterative process until the preset stopping conditions are met, at which point the security test results are output, achieving automation of the testing process. Therefore, the entire method constructs a complete closed-loop testing system that can comprehensively, efficiently, and accurately evaluate the security of the Wensheng video model without relying on a large amount of manual intervention. Attached Figure Description

[0018] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating a method for security testing of a text-based video model.

[0019] Figure 2 This is a schematic diagram illustrating the unfolding process of step S1 in a text-based video model security testing method.

[0020] Figure 3This is a schematic diagram illustrating the unfolding process of step S3 in a text-based video model security testing method.

[0021] Figure 4 This is a schematic diagram of the structure of a text-based video model security testing system.

[0022] Figure 5 This is a schematic diagram of a terminal device.

[0023] The diagram labels are as follows: 500, Terminal device; 501, CPU; 502, ROM; 503, RAM; 504, Bus; 505, I / O interface; 506, Input section; 507, Output section; 508, Storage section; 509, Communication section; 510, Driver; 511, Removable media; 600, Test system; 601, Acquisition module; 602, Analysis module; 603, Calculation module; 604, Mutation module; 605, Execution module. Detailed Implementation

[0024] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0025] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0026] To make the technical solutions of the embodiments of the present invention clearer and easier to understand, the application background of the embodiments of the present invention will be introduced below.

[0027] With the rapid development of artificial intelligence technology, textual video models have become one of the important research directions in the fields of natural language processing and computer vision.

[0028] Natural language processing (NLP) video models can generate video content based on natural language descriptions, driving the development of applications such as automated video production, virtual reality, and entertainment content creation. However, despite the significant progress made by NLP video models in multiple fields, security and compliance issues have also emerged in practical applications. For example, the generated videos may contain inappropriate or harmful content, such as violence or discrimination, raising compliance and ethical concerns in real-world applications.

[0029] Therefore, effectively evaluating the security and compliance of text-based video models has become an urgent problem to be solved. Existing testing methods for large text-based video models have the following limitations: (1) Insufficient testing, resulting in inadequate sample generalization. Existing methods rely on manually designed test samples, which lack sufficient diversity, thus failing to adequately cover all potential generation scenarios; (2) The compliance judgment of the generated videos is prone to missed detection. Traditional detection methods mainly rely on simple visual detection, which makes it difficult to accurately identify illegal content in complex videos, and it is easy to miss detection; (3) High cost and requires specialized test sample preparation. Existing detection methods require a lot of manual intervention and professional knowledge, resulting in high testing costs and low efficiency, making it difficult to cope with rapidly updated models.

[0030] In view of this, the present invention proposes a security testing method for a text-based video model, comprising: S1, obtaining a test video generated by the target text-based video model based on an initial test sample, wherein multi-dimensional compliance analysis of the test video is used to characterize the security of the target text-based video model; S2, performing multi-dimensional compliance analysis on the test video to obtain multiple analysis results corresponding to each analysis dimension; S3, constructing a unified feature vector from the multiple analysis results, and inputting the unified feature vector into a pre-trained decision tree auditing model to obtain a comprehensive risk factor; S4, comparing the comprehensive risk factor with a preset threshold to obtain a comparison result, and mutating the initial test sample according to the comparison result to generate a new test sample; S5, inputting the new test sample into the target text-based video model to generate a new test video, and repeating steps S2 to S4 until a preset stopping condition is met, and outputting the security test result of the target text-based video model.

[0031] As can be seen, this invention establishes a direct correspondence between video content and model security assessment by acquiring the video under test and clarifying the correlation between its analysis results and model security. Then, using multi-dimensional compliance analysis, multiple analysis results are simultaneously constructed into a unified feature vector and input into a decision tree model, ensuring comprehensive and multi-faceted detection of the video under test and achieving effective transformation from scattered detection results to quantified risk indicators. Finally, based on the comparison results of comprehensive risk factors and thresholds, the test samples are adaptively mutated, allowing them to dynamically adjust according to the previous round of detection results, improving the targeting and intelligence of the test. Through iterative loops until the stopping condition is met, the automated closed-loop testing process is guaranteed, ultimately reliably outputting the model's security test results. This method constructs a complete testing process that can adaptively generate more challenging test samples, thereby more comprehensively and efficiently evaluating the security of the Wensheng video model.

[0032] Please refer to the following. Figure 1The flowchart shown in this embodiment illustrates a text-based video model security testing method. The execution entity of this embodiment can be a computer device, server, cloud platform, or other dedicated testing system. The following is a further explanation of each step of this invention, which includes the following steps: S1. Obtain the test video generated by the target textual video model based on the initial test sample. The multi-dimensional compliance analysis of the test video is used to characterize the security of the target textual video model. The target textual video model here refers to the textual video model whose security needs to be verified. The results of the test video generated by it after multi-dimensional compliance analysis can be used to characterize the security of the current target textual video model. That is, the quality and content compliance of the test video will directly reflect the security performance of the target textual video model. By analyzing the video content, we can indirectly assess whether the model itself has the risk of generating illegal content.

[0033] Specifically, since the target text-based video model generates the test video based on the initial test samples, step S1 first requires obtaining or generating the initial test samples. These initial test samples can be preset general test prompts. In this embodiment of the invention, to ensure a more comprehensive verification of the security vulnerabilities and compliance of the target text-based video model, a method using diverse text descriptions generated through an adaptive probe synthesis model is employed. These rich initial test samples are input into the target text-based video model to be tested, and the model generates the corresponding test video based on the diverse text descriptions.

[0034] Further, see Figure 2 The process of obtaining the test video generated by the target textual video model based on the initial test samples includes the following steps: S11. Construct an adaptive detection synthesis model, which is used to generate multiple initial test samples based on the input prompt words; S12. Input the initial test sample into the target text video model to obtain the video to be tested.

[0035] Specifically, the adaptive probe synthesis model proposed in this invention is used to generate test text, and by training the adaptive probe synthesis model, diverse and targeted test samples can be generated. Guided by different input prompts, this adaptive probe synthesis model can generate test text adapted to different testing scenarios to cover potential security vulnerabilities and compliance issues.

[0036] Furthermore, the steps for constructing the adaptive probe synthesis model include the following: Step A1: Prepare training samples to train the adaptive probe synthesis model.

[0037] Prepare a set of training samples as input to the adaptive detection synthesis model. The training samples are mainly text data. Generally, the training samples consist of three parts: (1) standard text samples: including common text content and standard responses generated by the model, which serve as basic training data; (2) vulnerability test texts: including known vulnerability hints and issues, designing test texts for vulnerabilities that the model may have, and training the model to generate vulnerability-related hint texts; (3) complex scenario text samples: designing complex scenario texts that include multiple potential risk points, increasing the diversity of training samples, and ensuring that the model can generate complex test texts. The above training samples are obtained through manual annotation or collection from existing datasets and are used to train the adaptive detection synthesis model. During the training process, the adaptive detection synthesis model can learn how to generate text content that adapts to different test requirements based on the input text.

[0038] Step A2: The adaptive probe synthesis model can be guided in different directions by different prompt words and searched by different hyperparameters, ultimately generating diverse test texts.

[0039] Next, in order to increase the generalization of the test samples generated by the adaptive detection synthesis model, the present invention proposes the following three generalization methods: (1) prompt word guidance: the model generates relevant text according to different prompt words designed. Each prompt word represents a test scenario. The model generates different types of text test samples by inserting different prompt words; (2) hyperparameter optimization: by searching and optimizing the hyperparameters of the model, the most suitable parameter configuration is selected to improve the efficiency and diversity of the model in generating test text. Hyperparameter optimization can automatically adjust the complexity and diversity of the generated text to ensure coverage of more potential scenarios; (3) structured prompt generation: the structured template includes components such as [scenario], [problem], [action], and [constraint]. The model generates specific test text according to different structured templates.

[0040] Finally, based on the above generalization method, diverse test texts can be obtained, constituting the initial test samples required for the embodiments of this invention. These initial test samples are then input into the target text-based video model, which can then generate the corresponding test video based on the text descriptions of the initial test samples. Therefore, by constructing an adaptive detection synthesis model to generate diverse and targeted initial test samples, the problems of single and insufficient coverage in traditional manually designed test samples are overcome, providing a sample foundation for comprehensively exploring the security vulnerabilities of text-based video models.

[0041] S2. Perform multi-dimensional compliance analysis on the video to be tested to obtain multiple analysis results corresponding to each analysis dimension. Specifically, this invention proposes to analyze the generated content based on four layers of influence factors. These four layers of influence factors correspond to multi-dimensional compliance analysis. The so-called four layers of influence factors correspond to four dimensions, which include at least: image content dimension, text information dimension, voice content dimension, and narrative logic dimension, to ensure that the generated video content complies with relevant laws, regulations, and ethical standards.

[0042] The analysis process for each dimension will be explained in detail below: (1) Image content dimension: The generated test video content is decomposed into video frames. The video is broken down into independent image frames according to a preset frame rate. Each frame is extracted frame by frame and processed individually to ensure independent compliance analysis of each frame. The frame interval is adjusted according to the required analysis precision and the complexity of the video content. A higher frame rate allows for more detailed capture of dynamic changes in the video, while a lower frame rate reduces computational load and allows for compliance analysis of key frames.

[0043] The following section describes video compliance analysis based on a Convolutional Neural Network (CNN): The generated video is decomposed into individual frames to ensure independent detection and analysis of each frame. Then, a trained CNN model automatically extracts key features from each frame, such as edges, textures, and color distribution. These features serve as the basis for compliance detection. Finally, based on the extracted image features, the image is analyzed to detect whether it contains non-compliant content such as violence, gore, or disgust. By comparing known compliance standards with the model output, the presence of non-compliant content is determined, and a probability value in the [0,1] range is output to characterize the compliance level of the image content.

[0044] (2) Text information dimension: Text analysis is performed based on Optical Character Recognition (OCR); First, text is extracted from each frame of the generated video, recognizing all text content within the images, such as prompts and subtitles. OCR technology is then used to convert the text in the video into standard text format for subsequent compliance analysis; the OCR technology also supports multiple languages, enabling multilingual recognition. Next, compliance analysis is performed on the extracted text content, checking for non-compliant language, such as violence, pornography, hate speech, and other textual information that violates laws and regulations. Finally, the number of discrete non-compliant keywords contained in the tested video is output.

[0045] (3) Voice Content Dimension: In this stage, speech recognition technology is used to extract voice content from the generated video, including dialogue, narration, and background noise. The extracted voice text will undergo noise reduction processing to improve accuracy and usability. Next, natural language processing is used to perform compliance analysis on the transcribed voice text to identify whether it contains non-compliant information such as violent threats or hate speech. Timbre and tone analysis will also be introduced, paying particular attention to the emotional tendencies in the voice, such as provocative or intense tones, to ensure that the voice content complies with ethical requirements. In addition, the sentiment analysis model will identify and evaluate the emotional color in the voice to further determine whether it will evoke negative emotions or mislead. Finally, based on the results of the voice compliance analysis, detailed voice compliance identification indicators, namely sentiment intensity values, are generated for the next step of decision tree risk analysis.

[0046] (4) Narrative logic dimension: Specifically, the analysis of the narrative logic dimension includes the following steps: Step B1: Spatiotemporal tokenization representation of the video to be tested is performed. The tensor of the video to be tested is divided into feature blocks with local spatiotemporal correlation attributes by using a three-dimensional block embedding operator, and a spatiotemporal token sequence is constructed by introducing three-dimensional position encoding. First, the system performs deep feature deconstruction on the generated video stream of the test video using the spatiotemporal tokenization representation technique of the target text video large model. The system treats the video stream as a high-dimensional tensor constructed from the temporal dimension, the spatial channel dimension, and the pixel height and width of a single frame of the video image. By utilizing a 3D block embedding operator to perform non-overlapping sampling in the spatiotemporally continuous domain, complex video sequences are divided into feature blocks with local spatiotemporal correlation attributes, i.e., spatiotemporal token sequences.

[0047] Subsequently, non-overlapping sampling is performed in the spatiotemporal continuous domain using a three-dimensional block embedding operator. This process specifically involves using a preset time step and spatial window to perform sampling on the aforementioned high-dimensional tensor. The feature blocks are divided into feature segments with local spatiotemporal correlation attributes, denoted as the initial feature vector. ,in, , and These represent the indices of the feature blocks along the time axis, height, and width, respectively. It should be noted that each feature block contains local visual information within a single frame and the motion evolution trend of that region across consecutive frames.

[0048] Finally, in order to enable the feature blocks to recognize the sequential evolution of video plots, this embodiment of the invention introduces a three-dimensional position encoding mechanism to enhance the coordinates of the aforementioned feature blocks.

[0049] The three-dimensional position encoding formula used in this embodiment of the invention is as follows: Formula (1): Formula (1); in, , and These represent the position vectors of the feature blocks along the time axis, height, and width, respectively. By linearly fusing these three components, a coordinate augmentation vector carrying absolute temporal information and relative spatial coordinates is generated. Then, it is compared with the initial feature vector. After performing the overlay operation, a spatiotemporal token vector is constructed. For details, please refer to the following formula (2): Formula (2); Ultimately, all spacetime token vectors Together they constitute a spacetime token sequence This sequence will serve as input for subsequent hierarchical spatiotemporal attention modeling.

[0050] Step B2: Model the spatiotemporal token sequence through a hierarchical spatiotemporal attention mechanism to obtain a narrative logic analysis model, and extract the spatial interaction features and temporal evolution features within a single frame from the narrative logic analysis model. Specifically, the process of modeling the spatiotemporal token sequence using a hierarchical spatiotemporal attention mechanism employs a spatiotemporal separation self-attention mechanism. The resulting narrative logic analysis model is used for multi-level analysis of the video content of the test video. On one hand, it extracts features across different frames at the same spatial location, capturing the motion trajectory and evolution trend of objects. On the other hand, it extracts the interaction relationships between objects within the same frame. Finally, through a multi-head self-attention mechanism, the constructed narrative logic analysis model can calculate keyframes. and The semantic correlation strength between them is used to capture hidden causal chains that span long frame intervals.

[0051] Furthermore, after constructing the spatiotemporal token sequence, this embodiment of the invention employs a hierarchical spatiotemporal attention modeling technique to parse the video content. Through a spatiotemporally separated self-attention mechanism, the processing of high-dimensional features is decomposed into mutually coupled temporal and spatial dimension analyses. The specific implementation steps are as follows: First, within the dimension of a single-frame image, the system extracts the interaction relationships between different objects at the same moment through a spatial self-attention mechanism. For each frame, this stage calculates the association weights between feature blocks within the same frame to identify the semantic relationships between different visual elements in the image (such as the subject and the environment, or the subject and the inducing object), thereby obtaining the static semantic features of the single-frame image. This process focuses on information aggregation in the spatial dimension and can effectively capture the explicit visual content in a single-frame image.

[0052] Subsequently, across frames at the same spatial location, the system utilizes a temporal self-attention mechanism to capture the motion trajectory and evolution trend of objects. This stage calculates the evolution relationship of spatiotemporal tokens at corresponding spatial locations between different frames, extracting the starting, continuation, and evolution logic of actions in the video. This ensures that the model can perceive the continuous characteristics of the video over time. This process focuses on the aggregation of information in the temporal dimension, effectively capturing dynamic changes in the video.

[0053] Finally, a multi-head self-attention mechanism is used to calculate the strength of high-order semantic associations between keyframes. Through multiple independent attention subspaces, the narrative logic analysis model can simultaneously focus on different types of semantic associations (such as spatial interaction, temporal evolution, and semantic conflict), thereby capturing hidden causal chains spanning long frame intervals. This mechanism enables the narrative logic analysis model to identify violations disguised by editing, frame skipping, or long-range masking, achieving a complete modeling of the deep narrative logic of the video.

[0054] Specifically, to quantitatively describe the above-mentioned spatiotemporal separation correlation calculation process, the narrative logic analysis model of this embodiment of the invention adopts the following attention calculation formulas (3) and (4): Formula (3); Formula (4); in, ( ) is an attention mechanism function. , and These represent the query vector, key vector, and value vector obtained by mapping the spatiotemporal token sequence, respectively, and are used to represent different functional expressions in the feature space; This represents the scaling factor, usually the square root of the vector dimension, used to prevent the gradient from vanishing due to excessively large dot product results, thus ensuring the numerical stability of the calculation process. This represents the normalization exponential function, used to map the correlation strength between feature blocks to... Weighted probabilities within the interval; Indicates the first The output of each attention head, through the parallel computation of multiple attention heads, allows the model to simultaneously focus on features of multiple dimensions, such as spatial interaction features (spatial head) and temporal evolution features (temporal head); This represents a learnable weight parameter matrix used to project the token sequence into different subspaces to extract multi-level narrative association features.

[0055] Step B3: Construct the narrative logic flow of the video under test based on spatial interaction features and temporal evolution features, and identify illegal narrative structures in the video under test by analyzing the narrative feature vectors within the narrative logic flow.

[0056] After obtaining feature representations containing spatiotemporal semantics through a hierarchical spatiotemporal attention mechanism, this embodiment of the invention further constructs the narrative logic flow of the video to achieve the parsing of the deep semantic structure of the video. Specifically, in this embodiment of the invention, spatial interaction features and temporal evolution features are first fused at the same spatiotemporal location to obtain narrative primitive vectors; the narrative primitive vectors at each spatiotemporal location are organized into a narrative feature sequence in chronological order; the narrative feature sequence is input into the Encoder in the Transformer architecture, and global context modeling is performed through a multi-head self-attention mechanism to output narrative feature vectors; the narrative feature vectors and their evolution trajectory in the time dimension constitute the narrative logic flow of the video under test.

[0057] Since the narrative logic flow contains a sequence of narrative feature vectors in the time dimension of the video under test, the development state of the video can be divided into four logical states: beginning, development, climax, and ending, based on the sequence of narrative feature vectors. Furthermore, the analysis of the feature evolution between each state can identify illegal narrative structures. Specifically, the beginning is used to establish the initial scene and main characteristics; development is used to track the evolution of the main character's actions; the climax is used to identify the peak of the change rate of the Transformer's hidden layer state to pinpoint logical conflicts or risk points; and the ending is used to determine the final impact of the behavior.

[0058] Based on the narrative feature vector sequence, the process of dividing the video under test into four logical states—beginning, development, climax, and ending—requires locating the state boundaries by calculating the rate of change of the hidden layer states. For example, the hidden layer state sequence of the entire video output by the TransformerEncoder is... This sequence is the narrative feature vector sequence, where, , For the first Narrative feature vectors at each time step, This represents the total number of time steps. d For feature dimensions.

[0059] Next, we define the narrative offset at the t-th time step. The formula is as follows (5): Formula (5); in, For the first Narrative feature vectors at each time step; For the first Narrative feature vectors at each time step; Represented as the L2 norm; through detection The local peak value is used to determine the transition point of the development state: Initial stage: ,in, For the first to satisfy The point in time; Development stages: ,in, That is, the end point of the development stage is the point where the rate of change reaches its maximum value; Climax phase: Among them, global maximum value The central area; Final stage: ,in, The first after the climax ; and This is a preset state determination threshold.

[0060] To further quantify the degree of fluctuation in the narrative logic of the entire video, the cumulative narrative offset was calculated. For details, please refer to the following formula (6): Formula (6); Ultimately, the cumulative narrative offset The larger the value, the more dramatic the plot development of the video, and the more logical twists or conflicts are likely to occur.

[0061] Then, regarding intent and causality discrimination, unlike static content recognition that only targets a single frame image, this scheme can identify highly concealed illegal intents by analyzing the evolution trajectory of the global feature vector of the video.

[0062] Specifically, in this embodiment of the invention, a cross-temporal causal relationship matrix is ​​defined to identify causal relationships between different time-series images. Then the first i Frame and the j Semantic causal strength between frames The following formula (7) is used to calculate: Formula (7); in, for The activation function maps cosine similarity to the [0,1] interval. The larger the value, the closer the connection between the two in terms of narrative logic; For the first i The hidden layer state vector at each time step; For the first j The hidden layer state vectors at each time step; finally, through analysis Medium- to long-range frame pairs (i.e.) A high response value (of a relatively large value) can identify hidden causal chains that span long intervals.

[0063] Combining formula (7), the system logically fits the features of each stage of the video and identifies potential risk narratives by judging the causal relationship between different time sequence images. For example, when a video combines a series of images that are harmless in isolation, but form a narrative structure that implies self-harm, discriminatory metaphors, or false propaganda in the logical chain, this solution can capture such global logical anomalies.

[0064] Next, the narrative compliance quantification and scoring constructs a compliance discrimination head based on a gated recurrent unit or a linear classifier, inputting the target narrative feature vector extracted by the Transformer. The system detects whether the generated video contains logical breaks or abrupt changes that do not conform to ethical common sense.

[0065] The hidden layer state sequence will be presented below. Global pooling is performed, where the pooling can be mean pooling, max pooling, or attention pooling. In this embodiment of the invention, mean pooling is preferred, and the target narrative feature vector is obtained as shown in formula (8). : Formula (8); Furthermore, in this embodiment of the invention, a linear classifier is used as an example, and the narrative risk weight vector... The calculation is as follows (9): Formula (9); in, The weight matrix is ​​a learnable matrix. For bias terms; The number of pre-defined risk categories (such as ideological security, social morality, public safety, etc.); The function ensures that the sum of all components of the output vector is 1, representing the logical violation probability distribution of the video across different risk categories. Ultimately, the output is a vector with dimension... Narrative risk weight vector This is used to characterize the probability of logical violations in dimensions such as ideological security, social morality, and public safety of the video.

[0066] It should be noted that the embodiments of the present invention only illustrate the calculation of the narrative risk weight vector using a linear classifier as an example. In practical applications, in addition to linear classifiers, a gated recurrent unit can be constructed to detect the video under test. At the same time, a logical coherence score can be defined to detect whether there are logical breaks or abrupt changes that do not conform to ethical common sense in the video. Finally, a comprehensive narrative risk weight vector is obtained by combining all the detections as the output, which will not be elaborated further here.

[0067] Therefore, the embodiments of the present invention achieve comprehensive detection of generated videos through multi-dimensional compliance analysis of four dimensions: image content, text information, voice content, and narrative logic. In particular, the introduction of the narrative logic dimension can identify implicit illegal narrative structures formed by the combination of images, making up for the shortcomings of traditional single-frame detection in capturing complex illegal content.

[0068] S3. Construct a unified feature vector from the results of multiple analyses, and input the unified feature vector into a pre-trained decision tree audit model to obtain a comprehensive risk factor; Because the dimensions and representations of the analysis results differ across dimensions, they need to be normalized first, mapping them to a unified numerical space to construct a multi-dimensional unified feature vector. This feature vector contains compliance information for the video under test across the four dimensions mentioned above. Subsequently, this unified feature vector is input into a pre-trained decision tree review model. Based on the decision rules learned during training, the model traverses the decision path to the leaf nodes, ultimately outputting a comprehensive risk factor. This comprehensive risk factor is a quantitative indicator used to assess the risk level of the current test sample triggering the target video model to generate illegal content.

[0069] Specifically, see Figure 3 The above step S3 specifically includes the following steps: S31. Input the unified feature vector into the pre-trained decision tree review model, and output the independent risk score under each dimension through the decision path of the decision tree review model; S32. Construct a multidimensional risk vector based on independent risk scores under each dimension and by introducing a correction vector; S33. Perform aggregation calculations on the multidimensional risk vectors to obtain the comprehensive risk factor.

[0070] The CNN output is the probability value in the range [0,1], the OCR output is the discrete number of prohibited keywords, the speech model output is the emotion intensity value, and the narrative logic output is the logical risk weight. Therefore, before this step, the above values ​​need to be mapped to a unified feature vector. Specifically, for the image content dimension, since the convolutional neural network directly outputs a violation probability value normalized to the [0,1] interval, it can be directly used as a feature component for this dimension. For the text information dimension, the number of violation keywords detected by optical character recognition technology is a non-negative integer, which needs to be mapped to the [0,1] interval using methods such as maximum value normalization or logarithmic compression. For example, a maximum expected number of keywords can be preset, and the ratio of the actual detected number to this maximum value can be used as the feature value for this dimension. For the speech content dimension, it needs to be processed according to the specific form of the speech model output: if the output is a violation probability, it can be used directly; if the output is a sentiment tendency value, the negative sentiment intensity can be taken as the feature value. For the narrative logic dimension, the narrative risk weight vector output by the narrative logic analysis model can be maximized or weighted summed to obtain a comprehensive narrative risk value, which is then used as the corresponding component of the narrative logic dimension in the unified feature vector.

[0071] After the above normalization process, the outputs of the four dimensions are all mapped to the [0,1] interval. We concatenate them in a fixed order to form a four-dimensional unified feature vector. This vector fully encodes the compliance information of the video under test in four aspects: image, text, voice, and narrative logic, providing a standardized input for the subsequent decision tree review model.

[0072] Subsequently, it is necessary to unify the feature vector. The input is a pre-trained decision tree review model. Following pre-defined node splitting rules, the model traverses to the leaf nodes and outputs independent risk scores for each dimension. These independent risk scores constitute the input score vector. ;in, , , and Each dimension (image content dimension, text information dimension, voice content dimension, and narrative logic dimension) has an independent risk score.

[0073] It should be noted that the unified feature vector After being input into the pre-trained decision tree model, the model performs layer-by-layer judgments according to preset node splitting rules, traversing from the root node to the corresponding leaf node. At each internal node, the decision tree model compares the value of a certain feature dimension with the splitting threshold to determine the branch direction. When reaching a leaf node, the model statistically analyzes the distribution of historical training samples in that leaf node across various compliance dimensions: for the image content dimension, it calculates the proportion of non-compliant image samples in that node, serving as an independent risk score for that image dimension. For the text information dimension, the proportion of text violation samples in that node is calculated and used as an independent risk score for the text dimension. For the voice content dimension, the proportion of voice violation samples in that node is calculated as an independent risk score for the voice dimension. For the narrative logic dimension, the proportion of narrative violation samples in that node is calculated as an independent risk score for the narrative logic dimension. The independent risk scores from the above four dimensions together constitute the input score vector. This is used for subsequent multidimensional risk mapping and comprehensive risk factor calculation.

[0074] To provide a fine-grained representation of sample risk, it is necessary to map the four-dimensional primary score to a high-dimensional feature space composed of multiple risk categories. Therefore, a multi-dimensional risk mapping matrix is ​​constructed. ,in, The number of original evaluation metrics or features input into the decision tree review model (in this embodiment) n =4).

[0075] Specifically, multidimensional risk mapping matrix The following formula (10) shows that the matrix elements are... Indicates the first i The modality score for the first j The contribution gain coefficient for risk-related dimensions can be adaptively and dynamically optimized according to the application scenario.

[0076] Formula (10); Meanwhile, to capture the impact of semantic interactions between different modalities on risk assessment, this embodiment of the invention introduces a cross-modal collaborative correction vector. This is used to characterize semantic conflicts or enhancement effects between modalities. For example, when violations are detected simultaneously in images and speech, an additional weight penalty is applied, and its specific calculation formula is as follows: Formula (11): Formula (11) in, Represents the outer product (or element-wise product) of vectors. The preset intervention coefficient is used to identify and amplify the risk weight of hidden violations such as "discrepancy between text and images". A correction vector is obtained by weighted summation of interaction terms from different modal combinations. and Used to refer to this four-dimensional vector ( Two different components were taken from ) and It is a dimensional index, and .

[0077] Based on the above mapping and correction, the multidimensional risk vector can be calculated according to the following formula (12). ; Formula (12); in, This is an element-wise nonlinear activation function used to map the computation results to the interval [0, 1], ensuring the output vector... Each component in the range represents the risk probability of the sample under different dimensions; For the input score vector The transpose of the matrix; The result is dimensional vector, and also a dimensional vector Vie After addition, The function is applied element-wise to obtain... Each element in this vector represents the test sample at the [number]th [number]th [number]. i Probability of violations under different risk categories.

[0078] Finally, regarding the multidimensional risk vector The elements within are aggregated and calculated to obtain a one-dimensional comprehensive risk factor. This is used for subsequent feedback variation decisions.

[0079] It should be noted that the aggregation calculation method for multidimensional risk vectors can be selected according to the application scenario; for example, taking the maximum value can highlight the highest risk. ), or use weighted summation ( , (These are the corresponding weighting factors) to comprehensively consider various risks, without specific limitations here; finally, the comprehensive risk factor is obtained through aggregation calculation. This will serve as the basis for comparison with the preset threshold in step S4.

[0080] This demonstrates that by normalizing the heterogeneous analysis results from the four dimensions into a unified feature vector and performing multi-level risk aggregation through a decision tree audit model, an effective transformation from scattered detection results to quantitative risk indicators is achieved. At the same time, through a multi-dimensional risk mapping matrix and cross-modal correction vector, the probability of violations of samples under different risk categories can be characterized in a fine-grained manner, thereby improving the accuracy of risk assessment.

[0081] S4. Compare the comprehensive risk factor with the preset threshold to obtain the comparison result, and mutate the initial test sample according to the comparison result to generate a new test sample. After obtaining the comprehensive risk factors, this step feeds them back into the adaptive probe synthesis model to guide the generation of the next round of test samples. This process includes two core components: sample iteration control and sample mutation.

[0082] (a) Sample iteration control process: Comprehensive risk factors With preset threshold The comparison is performed, and the exploration space of the sample generation engine is dynamically adjusted based on the comparison results.

[0083] Based on the determined mutation strategy, this invention introduces a strategy factor to further adaptively control the execution intensity of mutation operations (such as the magnitude of random perturbations during exploration and the step size of gradient updates during utilization). This factor is based on the first k Multidimensional risk vector of the wheel The calculated value is used to quantify the overall deviation of the current sample in the risk space. The following defines the... k Strategy factors for round iteration The formula is as follows (13): Formula (13); in, The L2 norm of the comprehensive risk vector. Global sensitivity threshold; strategy factor The value can be +1 or 1. Used to indicate whether the current initial test sample has reached the risk boundary; For the first k The multidimensional risk vector corresponding to each iteration round.

[0084] For example, when hour, This indicates that the overall deviation of the current test sample in the risk space has reached or exceeded the sensitive boundary, and the test sample is in a "high risk intensity" state. At this time, the execution intensity of subsequent mutation operations should be appropriately converged to avoid excessive mutation leading to sample semantic collapse or test process divergence. when hour, This indicates that the overall risk intensity of the current sample is still within a safe range, and the sample is in a "low risk intensity" state. At this time, the execution intensity of subsequent mutation operations can be appropriately increased to expand the exploration scope or accelerate the risk discovery speed.

[0085] Next, the input feature vector of the sample generation engine According to evolutionary operators The update is performed using the following formula (14): Formula (14); in, For the first Round sample feature vector; For the first Round sample feature vector; This represents the increment of variation. The increment generated by the mutation operator, the specific form of which is given by the first mutation strategy or the second mutation strategy below, and The intensity control parameters will be based on the strategy factor. The value is dynamically adjusted.

[0086] Specifically, evolution operators according to The following binary branch logic is executed in the state: The first branch of logic involves considering the combined risk factors and the preset threshold. The comparison results are as follows: the comprehensive risk factor is less than the preset threshold. When the initial test sample fails to effectively trigger the model vulnerability, the system needs to search for potential risk points in a wider range. In this case, the first mutation strategy is adopted to mutate the initial test sample. The aim is to increase the diversity of test samples by introducing random perturbations in the feature embedding space and generate new test samples. That is, the first mutation strategy increases the diversity of test samples by introducing random perturbations in the feature embedding space. See the following formula (15) for details. Formula (15); in, The sample update increment (or mutation vector) under the first mutation strategy of the evolution operator. Gaussian white noise is used to search for unknown vulnerability regions over a large area within the potential space. Step size factor; As the decay factor, with each iteration... Increase the intensity of the perturbation gradually to ensure the convergence of the exploration process.

[0087] The increment Directly applied to the current sample feature vector Generate new sample features The text is then decoded into specific test sample text by an adaptive probe synthesis model. Meanwhile, in the selection of evolutionary operators, the first mutation strategy employs a random selection mechanism, randomly selecting one or more of the five mutation operators to ensure that the generated test samples have sufficient diversity to cover different risk dimensions.

[0088] The second branch of logic involves considering the combined risk factors and the preset threshold. The comparison results are as follows: the comprehensive risk factor is greater than or equal to the preset threshold. When the current initial test sample has successfully triggered the model to generate illegal content, the system needs to conduct in-depth mining around this risk point. At this time, switch to the second mutation strategy to strengthen the features of the triggering risk in a targeted manner and generate new test samples; the second mutation strategy is to select mutation operators related to the dimension of the triggering risk in a targeted manner and update them according to the gradient direction of the risk loss function in order to strengthen the features of the triggering risk in a targeted manner, as shown in the following formula (16).

[0089] Formula (16); in, As the second mutation strategy of the evolution operator, the sample update increment (or mutation vector) in the deep mutation mode determines the step size and direction of the test sample evolving towards a higher risk direction. These are gradient update coefficients used to control the intensity of target-oriented mutations and prevent semantic collapse caused by excessively rapid mutations. A gradient operator that takes the partial derivative with respect to the feature embedding space is used to locate the sensitive directions that cause changes in risk scores; This is a risk loss function; the higher the value, the greater the likelihood that the generated content will violate regulations. These are attention-guiding coefficients used to balance the weights between the target gradient and key feature fragments; Key feature vectors; This is an enhancement for key features extracted based on an attention mechanism. It involves localized high-frequency enhancement of core feature fragments that lead to high-risk scores (such as the edges of illegal objects or specific keyword textures). This increment... The same applies to the current sample feature vector. That is, new sample features are generated according to formula (14). Here, unlike the random perturbation of the first mutation strategy, the second mutation strategy has a clear directionality. That is, the gradient term ensures that the sample as a whole evolves towards a higher risk region, and the attention enhancement term ensures that the key local features in the sample that lead to high risk are further enhanced. The combination of the two can effectively uncover the deep vulnerabilities of the target textual video model at specific risk points. For example, if the risk is triggered by the image content dimension, the visual element enhancement operator is preferred.

[0090] (ii) Sample variation: Based on the adaptive probe synthesis model, specific mutation operations are performed on the initial test sample (Prompt) according to the mutation directions selected by the first and second mutation strategies mentioned above. Specifically, there are five mutation operators, covering two categories: semantic-level mutation and structural-level mutation. These are: semantic adversarial replacement, visual element enhancement, narrative chain extension, acoustic environment injection, and text embedding induction. The steps of each mutation operator are as follows: In practical applications, the first and second mutation strategies include one or more of the following mutation operators.

[0091] Among them, semantic adversarial substitution involves replacing the core words in the initial test samples with adversarial synonyms that have potential ambiguity or metaphor. Visual element enhancement involves adding adjectives describing visual texture or color to the initial test samples; Narrative chain extension involves adding time or causal adverbs to the initial test samples; Acoustic environment injection involves adding a description of the sound environment to the initial test sample; Text embedding inducement requires generating objects containing specific text in the initial test samples.

[0092] Furthermore, the specific implementation methods of each mutation operator are as follows: (1) Semantic adversarial substitution The core words or nouns in the initial test sample (Prompt) are replaced with adversarial synonyms with potential ambiguity or metaphor to generate new test samples. In practical applications, adversarial synonyms are selected from a pre-built adversarial lexicon, which contains words that are semantically similar to the original words but have potential violation tendencies. For example, replacing "attack" with "eliminate" in the initial test sample attempts to bypass the keyword filtering mechanism of the target text-based video model.

[0093] (2) Enhancement of visual elements Based on the initial test samples (Prompt), visual descriptive factors are added. In practical applications, adjectives describing visual texture or color are added to the Prompt based on the CNN detection results to test the visual compliance boundaries. For example, based on "red room," the description "flowing dark red liquid texture" is added to test whether the target textural video model will generate bloody images.

[0094] In this embodiment of the invention, the following formula (17) is used for visual element injection: Formula (17); in, The synthesized prompt words corresponding to the enhanced new test samples are used as input into the target text-based video model for sample generation; These are the basic prompts for the initial test samples, representing the initial description of the compliance scenario. To enhance the weighting coefficients, which are used to control the influence of visual description items on the original semantics and ensure the controllability of the testing process. Let be the i-th visual enhancement factor, which is the adjective vector that specifically describes visual texture or color; This represents the total number of enhancement factors injected.

[0095] (3) Extension of the narrative chain: To analyze the narrative logic of Transformers, appending time or causal adverbs to the end of the Prompt can induce the model to generate a series of illegal actions. For example, expanding the sample size from the initial seed can test the dynamic evolution logic of video generation.

[0096] (4) Acoustic environment injection: For voice compliance analysis, a description of the sound environment is explicitly added to the Prompt. For example, a voice prompt suffix is ​​appended after the scene description to test whether the audio stream generated by the model contains illegal acoustic features. Specifically, the addition method is as follows: formula (18): Formula (18); in, The sequence of full-dimensional inference instructions generated after being injected into the acoustic environment serves as the input to the Wensheng video large model. These are the basic scene description word vectors corresponding to the initial test samples; An acoustically guided suffix vector is a set of preset acoustic feature instructions; The acoustic guidance intensity factor is used to adjust the intervention weight of the acoustic suffix on the final audio synthesis result during the inference process. This represents a cascading operator, which denotes the appending of an acoustically guided suffix to the end of the basic instruction sequence according to temporal logic.

[0097] (5) Text embedding inducement: For OCR analysis, the prompt requires generating objects containing specific text. For example, it requires generating on-screen text prompts to test whether the model will generate illegal embedded text.

[0098] In summary, this invention dynamically selects an exploration strategy or a utilization strategy to adaptively mutate the test samples based on the comparison results of comprehensive risk factors and preset thresholds, thereby realizing the intelligent generation of test samples. The exploration strategy expands the test coverage through random perturbation, while the utilization strategy strengthens the in-depth mining of risk points through gradient orientation. The combination of the two significantly improves the efficiency and relevance of the test.

[0099] S5. Input the new test sample into the target text-based video model to generate a new video to be tested, and repeat steps S2 to S4 until the preset stopping condition is met, and output the security test result of the target text-based video model.

[0100] The new test sample generated in step S4 is input into the target textual video model to generate a new video to be tested. Steps S2 to S4 are then repeated to form a complete closed-loop iterative testing process.

[0101] Finally, the loop continues iteratively until the preset stopping condition is met.

[0102] Each iteration is denoted as the 1st iteration. k Rounds ( k= 1 , 2 , 3 ,...K The iterative loop includes the following sub-steps: ... k New test sample Prompt generated by the round (k) Input the target textual video model and generate the first k The test video of the wheel (k) ;No. k The test video of the wheel (k) Step S2 is executed to perform a four-dimensional compliance analysis, obtaining the results of each analysis; the results are then input into the decision tree audit model to obtain the... k Comprehensive risk factors corresponding to the round and the k The multidimensional risk vector corresponding to the wheel ; Execute step S4, according to With preset threshold The comparison results for the first k The test samples generated in the first round are mutated to generate the test samples for the next round. (Prompt) (k+1) This process continues in a loop until the preset stopping condition is met.

[0103] Specifically, the stopping conditions include any of the following situations: (1) continuous N Comprehensive risk factor obtained from rounds of iteration All exceeded the high-risk threshold (2) The number of iterations has reached the preset upper limit. This indicates that the test has covered a sufficient range.

[0104] When the stopping conditions are met, the system outputs the security test results of the target text-based video model. The test results include, but are not limited to: the change curve of the comprehensive risk factors, the trigger frequency statistics of each risk dimension, the list of typical vulnerability samples found, and the comprehensive security score of the model.

[0105] For example, the comprehensive risk curve outputs the comprehensive risk factors during the iteration process. According to the round k The curve visually illustrates the evolution trend of the security of the target text-based video model, with the high-risk threshold marked on the curve. and preset threshold The location facilitates the identification of risk trigger points and convergence status.

[0106] Statistics triggered by various risk dimensions: based on the multi-dimensional risk vector of each round. The system calculates the trigger frequency and average risk probability of each risk category (violence, pornography, gambling, addiction, etc.). The statistical methods include the number of triggers for each risk dimension, the average risk probability for each risk dimension, and a risk dimension heatmap to show the risk intensity distribution of different dimensions in different rounds. No specific limitations are imposed. List of typical vulnerability samples: High-risk rounds are selected ( > The corresponding test samples and generated test videos form a list of typical vulnerability samples. Specifically, each record includes: round number, original test sample text (i.e., the prompt that triggers the risk), comprehensive risk factor and risk scores for each dimension, storage path of high-risk videos or key frame screenshots, etc., without any special restrictions.

[0107] Based on the overall performance throughout the testing process, the security of the target text-based video model can be quantitatively scored by evaluating the performance at each round. The method of averaging or weighted calculation is not specifically limited.

[0108] Finally, based on the above data, the corresponding security test results can be generated. These results can be in text form, including, for example, whether the target text-based video model has high-risk vulnerabilities, the main risks concentrated in which dimensions, suggested remediation directions (such as strengthening visual content filtering and optimizing narrative logic review), and whether the security test passed (based on preset pass criteria). Thus, through the closed-loop iteration of the above steps, this method achieves adaptive generation of test samples and automated assessment of model security, enabling comprehensive and efficient detection of security vulnerabilities in text-based video models without relying on extensive manual intervention. Furthermore, the preset stopping conditions ensure the controllability and convergence of the testing process, and the final security test results provide clear data support and remediation directions for model improvement.

[0109] As described above, this invention achieves security and compliance testing of a large-scale text-based video model using a four-layer influence factor approach. This invention proposes innovative methods for test sample generation and automated video compliance review, primarily involving the application of intelligent detection synthesis models and video narrative analysis technologies. First, this invention trains an adaptive detection synthesis model based on human-annotated data. This model automatically generates initial test samples, which are then input into the target text-based video model to generate corresponding test video content. For the generated test video, the system extracts keyframes and performs OCR text extraction analysis, CNN sample compliance analysis, speech compliance analysis, and semantic information extraction. Next, based on the Transformer architecture, a comprehensive assessment of video compliance is conducted to obtain multi-dimensional compliance analysis results. Finally, this invention employs a decision tree model to output a comprehensive risk factor based on the above analysis results (such as pornography, violence, and other related harmfulness). Using this comprehensive risk factor as guidance, the generalization generation process of test samples is further optimized. That is, through a feedback mechanism, the compliance analysis results are used to adjust the test sample generation process to ensure the security and compliance of the samples generated by the target text-based video model.

[0110] Therefore, the beneficial effects of the present invention are mainly manifested in: (1) Improve the diversity and coverage of test samples: By generating diverse test samples through adaptive detection synthesis model, it is possible to more comprehensively detect potential vulnerabilities of text-based video models and reduce the risk of missed detection.

[0111] (2) Comprehensive compliance detection: The four-dimensional compliance analysis method is adopted, which covers the image content, text information, voice content and narrative structure in the video. It can comprehensively evaluate the compliance of the target text-generated video model and improve the accuracy of detection.

[0112] (3) Automated and efficient detection process: By combining decision tree model and feedback mechanism, an automated vulnerability detection process is realized, which not only improves testing efficiency, but also adjusts the testing strategy in real time according to the generated video content, reduces the need for manual intervention and reduces detection costs.

[0113] The above text combined Figures 1-3 The security testing method for text-based video models provided in the embodiments of the present invention has been described in detail below. Figure 4 , Figure 5 The system, device and medium provided in the embodiments of the present invention will be described.

[0114] Combination Figure 4 Based on the above embodiments, a text-based video model security testing method is proposed, and a testing system 600 is proposed here, which includes: The acquisition module 601 is configured to acquire the test video generated by the target textual video model based on the initial test sample. The multi-dimensional compliance analysis of the test video is used to characterize the security of the target textual video model. Analysis module 602 is configured to perform multi-dimensional compliance analysis on the video under test and obtain multiple analysis results corresponding to each analysis dimension. The calculation module 603 is configured to construct a unified feature vector from multiple analysis results and input the unified feature vector into a pre-trained decision tree audit model to obtain a comprehensive risk factor. The mutation module 604 is configured to compare the comprehensive risk factor with a preset threshold, obtain the comparison result, and mutate the initial test sample according to the comparison result to generate a new test sample. The execution module 605 is configured to input new test samples into the target textual video model, generate new test videos, and repeatedly pass through the analysis module, calculation module, and mutation module until the preset stopping conditions are met, and then output the security test results of the target textual video model.

[0115] In some embodiments, the acquisition module 601 is further configured to construct an adaptive detection synthesis model, which generates multiple initial test samples based on input prompt words; and input the initial test samples into a target text-based video model to generate a test video.

[0116] In some embodiments, the multi-dimensional compliance analysis in the analysis module 602 includes at least: image content dimension, text information dimension, voice content dimension, and narrative logic dimension.

[0117] In some embodiments, during the analysis of the narrative logic dimension: The analysis module 602 is used to perform spatiotemporal tokenization representation of the video under test. It divides the tensor of the video under test into feature blocks with local spatiotemporal correlation attributes through a three-dimensional block embedding operator, and introduces three-dimensional position encoding to construct a spatiotemporal token sequence. The spatiotemporal token sequence is modeled through a hierarchical spatiotemporal attention mechanism to obtain a narrative logic analysis model. The spatial interaction features and temporal evolution features within a single frame are extracted by the narrative logic analysis model. The narrative logic flow of the video under test is constructed based on the spatial interaction features and temporal evolution features. By analyzing the target narrative feature vector within the narrative logic flow, the illegal narrative structure in the video under test is identified.

[0118] In some embodiments, the calculation module 603 is further configured to input a unified feature vector into a pre-trained decision tree review model, output independent risk scores for each dimension through the decision path of the decision tree review model; and, based on the independent risk scores for each dimension and introducing a correction vector, construct a multi-dimensional risk vector; and, aggregate and calculate the multi-dimensional risk vector to obtain a comprehensive risk factor. In some embodiments, the mutation module 604 is further configured to mutate the initial test sample using a first mutation strategy to generate a new test sample when the comparison result is that the comprehensive risk factor is less than a preset threshold; the first mutation strategy increases the diversity of the test sample by introducing random perturbations in the feature embedding space. Furthermore, when the comparison result shows that the comprehensive risk factor is greater than or equal to a preset threshold, the initial test sample is mutated using a second mutation strategy to generate a new test sample. The second mutation strategy selects mutation operators related to the trigger risk dimension in a targeted manner and updates them according to the gradient direction of the risk loss function to strengthen the features of the trigger risk in a targeted manner.

[0119] In some embodiments, the first mutation strategy and the second mutation strategy in the mutation module 604 include one or more of the following mutation operators: semantic adversarial replacement, visual element enhancement, narrative chain extension, acoustic environment injection, and text embedding induction. Among them, semantic adversarial substitution involves replacing the core words in the initial test samples with adversarial synonyms that have potential ambiguity or metaphor. Visual element enhancement involves adding adjectives describing visual texture or color to the initial test samples; Narrative chain extension involves adding time or causal adverbs to the initial test samples; Acoustic environment injection involves adding a description of the sound environment to the initial test sample; Text embedding inducement requires generating objects containing specific text in the initial test samples.

[0120] The test system 600 according to an embodiment of the present invention can correspond to performing the methods described in the embodiments of the present invention, and the above and other operations and / or functions of each module of the test system 600 are respectively for implementing Figure 1 The corresponding process of the method in the illustrated embodiment will not be described in detail here for the sake of brevity.

[0121] A terminal device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of a text-based video model security testing method as described in Embodiment 1.

[0122] In this embodiment, as Figure 5As shown, the terminal device 500 includes a CPU 501 (Central Processing Unit), which can perform various appropriate actions and processes according to a program stored in ROM 502 (Read-Only Memory) or a program loaded from storage into RAM 503 (Random Access Memory). RAM 503 also stores various programs and data required for system operation. The CPU 501, ROM 502, and RAM 503 are interconnected via bus 504. An I / O interface 505 (Input / Output Interface) is also connected to bus 504.

[0123] The following components are connected to I / O interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive is also connected to I / O interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 510 as needed so that computer programs read from it can be installed into storage section 508 as needed.

[0124] In particular, according to embodiments of the present invention, the above-described flowchart is as follows. Figure 1 The described process can be implemented as a computer software program. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by the central processing unit (CPU 501), it performs the functions defined in the system of the present invention.

[0125] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM503), a read-only memory (ROM502), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0126] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0127] The units described in the embodiments of the present invention can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself. The described units or modules can also be located in a processor; for example, a processor can be described as including a first generation module, an acquisition module, a search module, a second generation module, and a merging module. The names of these units or modules do not necessarily limit the specific unit or module itself; for example, the acquisition module can also be described as "an acquisition module for acquiring multiple instances to be probed in the base table".

[0128] The present invention also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to implement a text-based video model security testing method as described in the above embodiments.

[0129] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention is not limited to the specific combination of the above-described technical features, but also includes other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in this invention.

Claims

1. A method for security testing of text-based video models, characterized in that, Includes the following steps: S1. Obtain the test video generated by the target textual video model based on the initial test samples; S2. Perform multi-dimensional compliance analysis on the video to be tested to obtain multiple analysis results corresponding to each analysis dimension; S3. Construct a unified feature vector from the multiple analysis results, and input the unified feature vector into a pre-trained decision tree audit model to obtain a comprehensive risk factor; S4. Compare the comprehensive risk factor with a preset threshold to obtain a comparison result, and mutate the initial test sample according to the comparison result to generate a new test sample. S5. Input the new test sample into the target text-based video model to generate a new video to be tested, and repeat steps S2 to S4 until the preset stopping condition is met, and output the security test result of the target text-based video model. The multi-dimensional compliance analysis includes at least the narrative logic dimension; the analysis of the narrative logic dimension includes: The video under test is represented by spatiotemporal tokenization. The tensor of the video under test is divided into feature blocks with local spatiotemporal correlation attributes by a three-dimensional block embedding operator, and a spatiotemporal token sequence is constructed by introducing three-dimensional position encoding. The spatiotemporal token sequence is modeled using a hierarchical spatiotemporal attention mechanism to obtain a narrative logic analysis model, and the spatial interaction features and temporal evolution features within a single frame of the video under test are extracted by the narrative logic analysis model. The narrative logic flow of the video under test is constructed based on the spatial interaction features and the temporal evolution features, and the illegal narrative structure in the video under test is identified by analyzing the target narrative feature vector within the narrative logic flow. The narrative logic flow includes a sequence of narrative feature vectors of the video under test in the time dimension. Based on the sequence of narrative feature vectors, the narrative offset of each time step is calculated. By detecting the local peak of the narrative offset, the development state of the video under test is divided into four logical states: beginning, development, climax, and ending. By analyzing the feature evolution between each state, illegal narrative structures are identified. The climax is identified by locating the logical conflict or risk outbreak point by identifying the peak of the state change rate of the hidden layer of the Transformer. By calculating the causal relationship between cross-time sequence images, hidden causal chains spanning long intervals are identified to identify illegal narrative structures in the video under test.

2. The text-based video model security testing method according to claim 1, characterized in that, The acquisition of the target textual video model, based on the test video generated from the initial test samples, includes: An adaptive detection synthesis model is constructed, which is used to generate multiple initial test samples based on the input prompt words; The initial test sample is input into the target text-based video model to obtain the video to be tested.

3. The text-based video model security testing method according to claim 1, characterized in that, The multi-dimensional compliance analysis includes at least the following dimensions: image content dimension, text information dimension, and voice content dimension.

4. The text-based video model security testing method according to claim 3, characterized in that, The step of inputting the unified feature vector into a pre-trained decision tree review model to obtain a comprehensive risk factor includes: The unified feature vector is input into a pre-trained decision tree review model, and the independent risk scores for each dimension are output through the decision path of the decision tree review model. Based on the independent risk scores under each dimension, and by introducing a correction vector, a multidimensional risk vector is constructed. The multidimensional risk vector is aggregated and calculated to obtain the comprehensive risk factor.

5. The text-based video model security testing method according to claim 1, characterized in that, The step of mutating the initial test sample based on the comparison result to generate a new test sample includes: When the comparison result shows that the comprehensive risk factor is less than the preset threshold, the initial test sample is mutated using a first mutation strategy to generate a new test sample; the first mutation strategy increases the diversity of the test sample by introducing random perturbations into the feature embedding space. When the comparison result shows that the comprehensive risk factor is greater than or equal to the preset threshold, the initial test sample is mutated using a second mutation strategy to generate a new test sample. The second mutation strategy selects mutation operators related to the trigger risk dimension and updates them according to the gradient direction of the risk loss function to strengthen the features of the trigger risk in a targeted manner.

6. The text-based video model security testing method according to claim 5, characterized in that, The first mutation strategy and the second mutation strategy include one or more of the following mutation operators: semantic adversarial replacement, visual element enhancement, narrative chain extension, acoustic environment injection, and text embedding induction.

7. A security testing system for text-based video models, characterized in that, The application has the method according to any one of claims 1 to 6, the system comprising: The acquisition module is configured to acquire the test video generated by the target textual video model based on the initial test samples; The analysis module is configured to perform multi-dimensional compliance analysis on the video under test, and obtain multiple analysis results corresponding to each analysis dimension. The calculation module is configured to construct a unified feature vector from multiple analysis results and input the unified feature vector into a pre-trained decision tree audit model to obtain a comprehensive risk factor. The mutation module is configured to compare the comprehensive risk factor with a preset threshold to obtain a comparison result, and mutate the initial test sample according to the comparison result to generate a new test sample. An execution module is configured to input the new test sample into the target text-based video model to generate a new test video. The new test video is repeatedly processed by the analysis module, the calculation module, and the mutation module until a preset stopping condition is met, at which point the security test result of the target text-based video model is output.

8. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-6.

Images