A video content security understanding method, system, device and storage medium

Abstract

The application discloses a video content safety understanding method, system, device and storage medium, which are corresponding solutions, and in the solutions: a video frame sampling module based on a semantic scoring strategy is introduced, the video frame highly related to harmful content can be preferentially selected, so that the sampling efficiency and the capture ability of a model to a key event are improved; a thinking chain framework is proposed, a harmful content detection task is decomposed into a plurality of structured reasoning sub-steps, a model system is guided to analyze visual and text information of a video, so that the detection precision, the interpretability and the robustness are improved, and it is also beneficial to a video understanding large model to better confirm and use the captured visual and text information.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video content safety understanding, and in particular to a video content safety understanding method, system, device and storage medium. BACKGROUND

[0002] In today's era, tens of thousands of video contents are uploaded to various long and short video platforms every day, covering entertainment, education, life sharing and other fields, and have profoundly changed the mode of information dissemination and consumption.

[0003] However, the rapid growth of video content also brings serious challenges. Some video content contains harmful elements, which not only disrupts the order of cyberspace, but also poses a serious threat to the physical and mental health of viewers, especially teenagers. Compared with text or image, the review of video content is more complex due to its dynamic and multi-modal characteristics, and major platforms have to invest a lot of manpower and resources in manual review to identify harmful elements that are not conducive to the growth of teenagers. However, due to the limitations of manual review in efficiency and accuracy, video content containing harmful elements still flows into the platform. In this context, video content safety understanding based on multi-modal large models has become a promising research direction.

[0004] Video content safety understanding requires the model to identify fuzzy concepts such as "harmful elements" or "violation elements" in the video, which is more complex than traditional video understanding tasks (such as action classification, activity recognition). On the one hand, the model needs to accurately capture multiple categories of harmful element candidates in the video, on the other hand, the model also needs to judge from a semantic perspective whether these candidates indeed have harmful connotations in the video scene. This brings multiple technical challenges: current mainstream video understanding models usually use uniform sampling frames to simplify dynamic videos into static image sequences, which can easily lead to information loss of harmful video key frames. In addition, the fuzziness and context dependence of "harmful elements" concept make the model prone to hallucinations in the detection task, which is manifested as missing harmful content or misjudging harmless content, which requires the model to have stronger recognition, understanding and reasoning ability. However, due to its dynamic and multi-modal characteristics, video content safety understanding has been in a state of research vacancy for a long time, facing higher technical threshold and more complex data requirements. In addition, the lack of high-quality video content safety dataset further hinders the evaluation and optimization of model performance.

[0005] With the increasing application demand of video content review, how to design a content safety understanding model that can accurately identify multiple harmful videos and avoid misjudging ordinary videos has gradually become a hot research direction for researchers. This not only promotes technological progress, but also provides new ideas and methods for solving practical problems.

[0006] In view of this, the present application is proposed. SUMMARY

[0007] The purpose of the present application is to provide a video content safety understanding method, system, device and storage medium, which can improve the video content safety understanding ability of the video understanding large model.

[0008] The purpose of the present application is achieved by the following technical solutions:

[0009] A video content safety understanding method, comprising:

[0010] Collecting an image dataset containing harmless pictures and harmful pictures, and a video dataset containing harmless videos and harmful videos; wherein the harmful pictures and the harmful videos cover N predefined harmful categories;

[0011] Combining the image dataset and the video dataset, using an image and video mixed fine-tuning strategy to fine-tune the video understanding large model; wherein the video understanding large model is configured with a video frame sampling module based on a semantic score strategy, used to calculate the semantic relevance score of each video frame and harmful content, perform video frame sampling according to the semantic relevance score, and used for fine-tuning training;

[0012] Using the fine-tuned video understanding large model, using a thinking chain framework for reasoning, including: through the fine-tuned video understanding large model, combining the input video and text to sequentially perform event detection and harmfulness detection results, and generate the final harmful content detection result.

[0013] A video content safety understanding system for implementing the foregoing method, comprising:

[0014] A dataset collection unit for collecting an image dataset containing harmless pictures and harmful pictures, and a video dataset containing harmless videos and harmful videos; wherein the harmful pictures and the harmful videos cover N predefined harmful categories;

[0015] A fine-tuning training unit for combining the image dataset and the video dataset, using an image and video mixed fine-tuning strategy to fine-tune the video understanding large model; wherein the video understanding large model is configured with a video frame sampling module based on a semantic score strategy, used to calculate the semantic relevance score of each video frame and harmful content, perform video frame sampling according to the semantic relevance score, and used for fine-tuning training;

[0016] A detection unit for using the fine-tuned video understanding large model, using a thinking chain framework for reasoning, including: through the fine-tuned video understanding large model, combining the input video and text to sequentially perform event detection and harmfulness detection results, and generate the final harmful content detection result.

[0017] A processing device comprising: one or more processors; a memory for storing one or more programs;

[0018] wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the foregoing method.

[0019] A readable storage medium storing a computer program, which, when executed by a processor, implements the foregoing method.

[0020] As can be seen from the technical solutions of the present application, the video frame sampling module based on the semantic scoring strategy is introduced, which can preferentially select video frames highly related to harmful content, thereby improving the sampling efficiency and the model's ability to capture key events; the proposed thinking chain framework divides the harmful content detection task into several structured reasoning sub-steps, guides the model system to analyze the visual and text information of the video, thereby improving the detection accuracy, interpretability and robustness, and is also conducive to the video understanding large model to better confirm and use the captured visual and text information. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained according to these drawings without creative labor for those skilled in the art.

[0022] Figure 1 A flowchart of a video content security understanding method provided for the embodiments of the present application.

[0023] Figure 2 A video frame merging strategy diagram provided for the embodiments of the present application.

[0024] Figure 3 A video content security understanding thinking chain framework diagram provided for the embodiments of the present application.

[0025] Figure 4 A schematic diagram of a video content security understanding system provided for the embodiments of the present application.

[0026] Figure 5 A schematic diagram of a processing device provided for the embodiments of the present application. DETAILED DESCRIPTION

[0027] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0028] Firstly, the terms possibly used in the present application are described as follows:

[0029] The terms "comprise", "contain", "include", "have" or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, the inclusion of a technical feature element (such as raw materials, components, ingredients, carriers, dosage forms, materials, sizes, parts, components, mechanisms, devices, steps, processes, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or articles, etc.) should be interpreted as not only including the explicitly listed technical feature element, but also including other technical feature elements not explicitly listed in the art.

[0030] The term "consisting of" means excluding any technical feature element not explicitly listed. If this term is used in a claim, the term will make the claim closed, so that it does not contain technical feature elements other than the explicitly listed technical feature elements, except for conventional impurities related thereto. If the term only appears in a certain clause of the claim, it is only limited to the elements explicitly listed in that clause, and the elements described in other clauses are not excluded from the overall claim.

[0031] Unless otherwise explicitly specified or limited, the terms "mount", "connect", "connect", "fix", and the like should be broadly understood, for example: it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium, or it can be connected inside two elements. For those of ordinary skill in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0032] When the concentration, temperature, pressure, size or other parameters are expressed in the form of a numerical range, the numerical range should be understood as specifically disclosing all ranges formed by any upper limit value, lower limit value, preferred value in the numerical range, regardless of whether the range is explicitly described; for example, if the numerical range "2-8" is described, the numerical range should be interpreted as including "2-7", "2-6", "5-7", "3-4 and 6-7", "3-5 and 7", "2 and 5-7", etc. Unless otherwise specified, the numerical range described herein includes all integers and fractions within the numerical range, including the end values.

[0033] Embodiment one

[0034] The embodiment of the present application provides a video content safety understanding method, as shown in the figure, which mainly comprises the following steps: Figure 1

[0035] Step 1, collect image and video data sets.

[0036] In the embodiment of the present application, image data sets containing harmless pictures and harmful pictures, and video data sets containing harmless videos and harmful videos are collected; wherein the harmful pictures and the harmful videos cover N predefined harmful categories.

[0037] Step 2, fine-tuning training of a video understanding large model.

[0038] In the embodiment of the present application, combined with the image data set and the video data set, the image and video mixed fine-tuning strategy is used to fine-tune the training of the video understanding large model.

[0039] In the embodiment of the present application, the video understanding large model is configured with a video frame sampling module based on a semantic scoring strategy, which is used to calculate the semantic relevance score of each video frame and harmful content, sample video frames according to the semantic relevance score, and fine-tune the training.

[0040] During fine-tuning training, the image data or the video frame and the question are uniformly encoded into an input format suitable for the model, the spatiotemporal features of the image data or the video frame are extracted through the pre-trained visual backbone network, and the question text is converted into a semantic vector through the text encoder. Subsequently, these features are sent to the multi-modal fusion module for alignment. Considering the process of this part, the conventional technology can be referred to, and thus no further description is made.

[0041] In the embodiment of the present application, the calculation of the semantic relevance score of each video frame and harmful content comprises: using the semantic scoring strategy of CLIP to calculate the semantic relevance score, for the expression T related to harmful content, the semantic relevance score between the given video frame is:

[0042] ;

[0043] Wherein, CLIP is a contrastive language-image pre-training model, is the semantic relevance score between the given video frame and the expression T related to harmful content.

[0044] ​In the embodiment of the present application, the video frame sampling according to the semantic correlation score and used for fine-tuning training comprises: the higher the semantic correlation score, the higher the priority of the corresponding video frame sampling; after the video frame sequence is sampled, a similar frame merging algorithm is used to compress the video frame sequence, comprising: calculating the image similarity of adjacent video frame pairs, selecting the adjacent video frame pair with the highest image similarity, denoted as , and fusing the image features of the two as the representative frame , replacing the video frame in the video frame sequence , and deleting the video frame and ; continuously repeating until the length of the video frame sequence does not exceed the preset length L, obtaining the compressed video frame sequence; and using the compressed video frame sequence for fine-tuning training.

[0045] In the embodiment of the present application, the fine-tuning training of the video understanding large model comprises: using low-rank adaptation technology for parameter fine-tuning; wherein the parameters of the video understanding large model are fixed, two weight matrices are introduced, and the two weight matrices are fine-tuned.

[0046] The two weight matrices comprise: a matrix A with a dimension of , and a matrix B with a dimension of ; the parameters (pre-training parameters) of the video understanding large model are denoted as , the dimension is , and the forward propagation process during fine-tuning training is represented as:

[0047] ;

[0048] wherein, , and are all set positive integers; represents an output vector in the forward propagation process, x is an input vector in the forward propagation process, is a weight update matrix.

[0049] Based on the above forward propagation process, the fine-tuning process of the two weight matrices is as follows: randomly initializing the matrices A and B, multiplying them to obtain an initial weight update matrix , which is added to the original parameter , and the effect is determined according to the loss function; thereafter, the gradients (i.e. adjustment amounts) of the matrices A and B are calculated through back propagation, and the process is continuously iterated until the loss function converges or has been iterated for a preset number of times. The loss function involved here can be selected by the user according to the actual situation, and the present application does not make any limitation. After the iteration is completed, the fine-tuned matrices A and B are obtained, and the fine-tuned weight update matrix is obtained by multiplying BA, which is added to the original parameter The final updated weights are obtained.

[0050] Step 3, using the fine-tuned video understanding large model to perform reasoning in the thinking chain framework.

[0051] In the embodiment of the application, the step of reasoning in the thinking chain framework comprises: performing event detection and harmfulness detection results in turn by combining the input video and the text through the fine-tuned video understanding large model, and generating the final harmful content detection result.

[0052] (1) Event detection.

[0053] Design prompt words , prompt words contain the definition of the predefined N harmful categories, and require the fine-tuned video understanding large model to focus on the visual features and semantic information of the input video.

[0054] Through the prompt words , the fine-tuned video understanding large model generates the text summary of the event in the input video , which is represented as:

[0055] ;

[0056] The text summary of the event is the event detection result.

[0057] (2) Harmfulness detection.

[0058] Based on the event detection result and the definition of the predefined N harmful categories, the semantic similarity between the text summary of the event and the harmful category is calculated to generate a preliminary harmfulness score as the harmfulness detection result.

[0059] (3) Generate the final harmful content detection result.

[0060] This part is represented as:

[0061] ;

[0062] Wherein, is the event detection result, which is the text summary of the event in the input video , is the input text, is the fine-tuned video understanding large model, and the final harmful content detection result. ​​

[0063] Further, the present application also includes: constructing a data set with video content safety understanding challenges for evaluating the fine-tuned trained video understanding large model.

[0064] The construction method includes:

[0065] (1) Collecting video data, combining a predefined N harmful category to guide an external large language model to re-label the video data, that is, mapping each video data to N harmful categories or marking as harmless category;

[0066] (2) Taking N harmful categories as keywords, combining content filtering and quality evaluation technology to screen out videos from public video resources, supplementing the re-labeled video data, so that the constructed data set with video content safety understanding challenges is balanced in categories.

[0067] The present application can be applied to various video understanding large model video content detection tasks, helping the video understanding large model better understand the events occurring in the video and whether they are harmful in the video scene, so as to better confirm whether the video has unsafe or illegal harmful content.

[0068] In order to more clearly show the technical solutions provided by the present application and the technical effects produced, the method provided by the embodiments of the present application is described in detail below with specific embodiments.

[0069] I. Constructing a data set with video content safety understanding challenges.

[0070] The purpose of the present application is to investigate and improve the ability of video understanding large model in the field of video content safety understanding, therefore, constructing a data set with video content safety understanding challenges (BVD data set) is crucial for the training and evaluation of the model.

[0071] The present application is based on the BVD data set made on the basis of the public data set, for example, it can be made on the basis of VHD11K data set, which contains 1000 videos, and the harmful and harmless data ratio is 1:1. However, VHD11K has significant defects: the number of videos is limited, only 1000, the average duration is only 32.86 seconds, and the difficulty of video content safety understanding is low; the original annotation is generated by a large language model, which contains 257 harmful categories, and the distribution is highly dispersed, some categories only appear once, which is difficult to support the full training and quantitative evaluation of the model.

[0072] In order to improve the quality of the data set, the present application redesigns the harmful content classification system, and integrates the harmful video types into N explicit categories.

[0073] In the embodiments of the present application, N is set to 9, and there are 9 clear categories in total: dangerous activities, self-harm behavior, sexual exposure, vulgar behavior, gun violence, hate consciousness, drug / alcohol / tobacco abuse, false information (misleading information generated by artificial intelligence, such as face forgery, etc.), abuse and bullying. Each category is equipped with clear definitions and typical examples to ensure the standardization and operability of labeling. This classification system not only covers common harmful content types, but also fully considers the diversity and complexity of real-world scenarios, providing a unified framework for model training and evaluation. According to this classification system, the present application re-labels the VHD11K's labeling file, calls the API interface (application program interface) of Qwen-Long (a general-purpose Qwen model with long text processing capabilities), and maps the original harmful categories to the 9 pre-defined categories or labels as harmless through targeted prompt words. To ensure quality, 10% of the videos are randomly selected for manual verification, and the verification results are highly consistent with the category definitions.

[0074] In practical applications, the specific number of harmful categories and their specific definitions can be set according to actual conditions, and the present application does not make specific limitations.

[0075] The re-labeled dataset solves the problem of inconsistent categories, but still faces the challenge of uneven category distribution. To solve the problem of uneven category distribution, the present application supplements a large number of harmful videos through internet crawling. The specific method is to use the keywords of the 9 harmful categories as the basis, combined with content filtering and quality evaluation, to select videos with large inter-frame differences and high-quality content from some public video resources. The supplemented dataset contains 2300 videos, each of which contains 200 video samples of each harmful category, and 500 harmless videos, with a total average duration of 81.77 seconds.

[0076] The construction of the BVD dataset provides reliable data support for the training and evaluation of video understanding large models in the harmful content detection task. Subsequent experiments will be based on the BVD dataset to fine-tune the model and evaluate its performance in binary classification and fine-grained classification tasks to verify its effectiveness; considering that the present application uses BVD for performance evaluation, the same method is used to construct the corresponding training set, and the BVD dataset can be provided to other researchers for model training.

[0077] II. Video content safety understanding scheme

[0078] This part includes three parts: video frame sampling module design, fine-tuning training, and thought chain understanding framework.

[0079] 1. Video frame sampling module

[0080] To enhance the ability of large video understanding models to detect and capture harmful visual elements, this invention proposes a frame sampling method based on semantic guidance and memory mechanism optimization, forming a video frame sampling module. This module combines a semantic scoring strategy (e.g., a CLIP-based semantic scoring strategy) to prioritize key frames highly relevant to harmful content, thereby improving sampling efficiency and the model's ability to capture key events.

[0081] The principle of CLIP's semantic scoring strategy is to use a pre-trained CLIP (Contrastive Language-Image Pretraining) model to obtain a semantic relevance score (i.e., CLIPScore) by calculating the semantic similarity between the image and the text content.

[0082] Specifically, given a sequence of video frames The semantic relevance score for expressions T related to harmful content (such as "harmful elements" or "inappropriate content") is calculated as follows:

[0083] ;

[0084] in, Indicates a given video frame The semantic relevance score between expressions T related to harmful content, i=1,2,…,M, where M is the length of the video frame sequence. Based on High-resolution frames are prioritized as sampling candidates. Compared with random uniform sampling, CLIPScore-based sampling can significantly improve the ability to capture semantic keyframes.

[0085] To reduce the storage and processing of redundant frames, such as Figure 2 As shown, this invention uses a simple and intuitive video frame merging strategy to compress the sampled video frame sequence. The image similarity s between adjacent frames is calculated, and its expression is shown in the following formula:

[0086] ;

[0087] in, Represents video frames The CLIP feature vectors (image feature vectors extracted by the CLIP model) This represents the number of feature vectors. For a sampled video frame sequence, the model finds the pair of neighboring frames with the highest similarity. By merging the features of these two frames using a weighted average, a representative frame is generated. , delete , , update the frame sequence. Repeat the above steps until the frame number reaches the preset input length L, and take the merged frame sequence as the input of the projection layer for subsequent alignment of visual features and text features.

[0088] After introducing the CLIPScore-based video frame sampling module, the accuracy of the video understanding large model on the binary classification and fine-grained classification tasks can be improved. For example, taking the VideoLLaMA2 model open sourced by the Alibaba DAMO Academy as an example, the accuracy on the binary classification and fine-grained classification tasks is improved by 1.91% and 1.78%, respectively. This shows that CLIPScore can effectively reduce the interference of redundant information and enhance the model's ability to perceive harmful visual content by focusing on key frames.

[0089] 2. Fine-tuning training.

[0090] After key frame sampling of the video, it is crucial to improve the model's understanding of these key frames that may contain harmful elements. The present application is based on a pre-trained video understanding large model (for example, the VideoLLaMA2 model mentioned earlier), which uses pre-trained weights as initial parameters to retain its existing visual and temporal feature extraction capabilities, and designs an image-video hybrid fine-tuning strategy.

[0091] In the embodiment of the present application, an image dataset containing harmless pictures and harmful pictures, and a video dataset containing harmless videos and harmful videos are collected; wherein the harmful pictures and harmful videos cover N predefined harmful categories; for example, N=9 can be set, and the specific 9 harmful categories can be referred to the introduction in the foregoing.

[0092] For example: in terms of image fine-tuning, 4700 image data can be carefully selected from the SMID dataset and the VHD11K dataset, including 2000 harmless pictures and 2700 harmful pictures, covering 9 predefined harmful categories (300 pictures for each category); in terms of video fine-tuning, 900 harmless videos are randomly selected from public video datasets such as ActivityNet, YouCook2 and COIN, and 100 videos for each of the 9 harmful video categories are collected through internet crawling and manual screening, totaling 900 harmful videos.

[0093] The 4700 image data and 1800 video data are comprehensively utilized to fine-tune the video understanding large model, so that the model can more comprehensively learn the feature correlation of static and dynamic harmful elements in a complex scene. At the same time, in order to reduce the calculation resource overhead during training and speed up the training, the LoRA (Low-Rank Adaptation) technology is introduced for efficient parameter fine-tuning. The background of this technology is that it is assumed that there is an extremely low-dimensional parameter in the pre-trained model, and fine-tuning it and fine-tuning in the full parameter space can have the same effect. At the same time, after pre-training, the larger the model, the smaller the intrinsic dimension. Therefore, two smaller weight matrices are defined, and only these two matrices are fine-tuned during training, which can greatly save the calculation resources of fine-tuning.

[0094] The matrix calculation formula is as follows (where the pre-training parameter has a dimension of , the matrix A has a dimension of , and the matrix B has a dimension of ):

[0095] .

[0096] In the above dimension representation, the two items before and after the multiplication sign respectively correspond to the number of rows and columns of the matrix, and the specific values can be set by the user according to the actual situation or experience, and the present application does not make any limitation.

[0097] During the training process, the parameter is frozen, and only the parameters in the matrices A and B are trained. For the process of regular forward propagation , the forward propagation process becomes:

[0098] .

[0099] After targeted LoRA fine-tuning, the accuracy of the video understanding large model on the binary classification and fine-grained classification tasks can be improved. Similarly, taking the VideoLLaMA2 model opened by the Alibaba DAMO Academy as an example, the accuracy of the fine-tuned model on the binary classification and fine-grained classification tasks is improved by 4.34% and 5.69% respectively. The experimental results show that the mixed fine-tuning strategy integrates the static features of images and the dynamic information of videos to construct a more comprehensive feature representation, thereby achieving excellent performance in binary classification and fine-grained classification tasks.

[0100] 3. Thought chain understanding framework.

[0101] After detecting possible harmful content, it is also necessary to combine video topics and summaries to further analyze whether these contents are harmful in the video scene. In order to improve the reasoning ability of the fine-tuned model, the present application proposes a video harmful content detection method based on a chain-of-thought (CoT) framework. Traditional video understanding models usually directly output classification results, lack of explainability of reasoning process, and are difficult to cope with the inter-frame dynamic semantic changes and fine-grained harmful category division of the BVD dataset. In contrast, the chain-of-thought framework proposed by the present application divides the harmful content detection task into several structured reasoning sub-steps, guides the model to systematically analyze the visual and textual information of the video, thereby improving the detection accuracy, explainability and robustness. It is also conducive to the better confirmation and use of the captured visual and textual information by the large model. As shown in Figure 3 The chain-of-thought understanding framework includes the following three core steps.

[0102] The first step of the chain-of-thought framework is to extract key events related to harmful content from the video (event detection). Given a video understanding large model , a video and an input text (such as a question or a hint), the present application designs a specific hint word to guide the model to focus on events that may be related to harmful content. The hint word is combined with the definition of the 9 harmful categories of the BVD dataset, requiring the model to pay attention to the visual features and semantic information of the video. Through hint guidance, the model generates a brief text summary of the key event, which is represented as follows:

[0103] .

[0104] For example, for a video containing a street fight, the model may generate a summary: "The video shows a group of people using weapons to violently clash." This step reduces irrelevant information interference by focusing on key events, laying the foundation for subsequent reasoning.

[0105] The second step is to judge whether the video content belongs to the harmful category based on the event detection result (harmfulness detection). For the fine-grained harmful category division of the BVD dataset, the model calculates the semantic similarity between each event and the category based on the text summary and the definition of the 9 harmful categories, and generates a preliminary harmfulness score . This method effectively distinguishes fine-grained harmful categories (such as "gun violence" and "physical violence") and reduces the risk of misjudgment. Compared with traditional end-to-end classification methods, step-by-step reasoning significantly improves the detection accuracy in complex scenarios.

[0106] The last step of the framework is to combine the video and the text input generating a final harmful content detection result which is expressed as follows:

[0107] .

[0108] Specifically, the model utilizes text summarization and text input , comprehensively analyzes the visual features, text information and reasoning process of the video, and outputs a harmful category or "harmless" judgment. To cope with the challenge of the fine-grained category division of the BVD dataset, the model can be attached with a brief reasoning explanation, for example: "the video contains a weapon conflict scene, the visual features are highly matched with the violence category, and it is determined as gun violence." This explanation not only improves the interpretability of the result, but also facilitates the user to verify the decision logic of the model. Through the above thought chain understanding framework, the present application not only improves the performance of the model in the video content safety understanding task, but also provides higher transparency and application value. Experiments show that the accuracy of the method in the binary classification and fine-grained classification tasks is improved by about 6.95% and 11.79% respectively, which shows that it has more obvious advantages in scenes requiring multi-step reasoning.

[0109] The above scheme provided by the embodiment of the present application: (1) a dataset with a video content safety understanding challenge is proposed, which can effectively examine the harmful visual element capture ability and harmful video semantic content understanding ability of existing and future video understanding large models, and provide a baseline for subsequent related research in the industry; (2) the present application simultaneously proposes a video frame sampling module and a video content safety understanding framework based on thought chain technology, which can be applied to any video understanding large model, fully excavates the potential of the video understanding large model through the capture of key harmful visual information and the guidance of multi-round question and answer, and helps it to better complete the task of video content safety understanding.

[0110] Through the description of the above implementation mode, those skilled in the art can clearly understand that the above-mentioned embodiments can be realized by software, or can be realized by means of software plus necessary general hardware platform. Based on such understanding, the technical solutions of the above-mentioned embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be CD-ROM, U disk, mobile hard disk, etc.), including a plurality of instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute the method described in each embodiment of the present application.

[0111] Embodiment two

[0112] The present application also provides a video content safety understanding system, which is mainly used to realize the method provided by the above-mentioned embodiments, as shown in Figure 4 the system mainly includes:

[0113] a data set collection unit configured to collect an image data set containing harmless pictures and harmful pictures, and a video data set containing harmless videos and harmful videos; wherein the harmful pictures and the harmful videos cover a predefined N harmful categories;

[0114] a fine-tuning training unit configured to fine-tune a video understanding large model by using an image and video mixed fine-tuning strategy in combination with the image data set and the video data set; wherein the video understanding large model is configured with a video frame sampling module based on a semantic scoring strategy, for calculating a semantic correlation score of each video frame and harmful content, performing video frame sampling according to the semantic correlation score, and for fine-tuning training;

[0115] a detection unit configured to use the fine-tuned video understanding large model to perform reasoning by using a thinking chain framework, including: sequentially performing event detection and harmfulness detection results by the fine-tuned video understanding large model in combination with an input video and text, and generating a final harmful content detection result.

[0116] Since the main technical details involved in the system have been described in detail in the previous embodiments, further description is omitted.

[0117] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional modules is taken as an example for illustration, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the system is divided into different functional modules to complete all or part of the functions described above.

[0118] Embodiment Three

[0119] The application also provides a processing device, as shown in the figure, mainly comprising: one or more processors; a memory for storing one or more programs; wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided by the preceding embodiments. Figure 5 Further, the processing device further comprises at least one input device and at least one output device; in the processing device, the processor, the memory, the input device and the output device are connected through a bus.

[0120] In the embodiments of the application, the specific types of the memory, the input device and the output device are not limited; for example:

[0121] The input device can be a touch screen, an image acquisition device, a physical button or a mouse, etc.

[0122]

[0123] ​The output device can be a display terminal.

[0124] The memory can be a Random Access Memory (RAM), and can also be a non-volatile memory, such as a disk memory.

[0125] Embodiment Four

[0126] The application further provides a readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the method provided by the foregoing embodiments is realized.

[0127] The readable storage medium in the embodiments of the application as a computer readable storage medium can be arranged in the foregoing processing device, for example, as a memory in the processing device. In addition, the readable storage medium can also be a U disk, a mobile hard disk, a Read-Only Memory (ROM), a magnetic disk or an optical disk, and various media capable of storing program codes.

[0128] The above merely describes the preferred embodiments of the application, but the protection scope of the application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the application, which should be covered in the protection scope of the application. Therefore, the protection scope of the application should be subject to the protection scope of the claims. The information disclosed in the background section of the present application is merely intended to deepen the understanding of the general background of the application, and should not be regarded as acknowledging or implying in any form that the information constitutes the prior art known by the person skilled in the art.

Claims

1. A method of video content security understanding, the method comprising: The method comprises the following steps: Collecting an image dataset containing harmless pictures and harmful pictures, and a video dataset containing harmless videos and harmful videos; wherein the harmful pictures and the harmful videos cover a predefined N harmful categories; Combining the image dataset and the video dataset, and using an image and video mixed fine-tuning strategy to fine-tune a video understanding large model; wherein the video understanding large model is configured with a video frame sampling module based on a semantic scoring strategy, used to calculate the semantic correlation score of each video frame and harmful content, sample video frames according to the semantic correlation score, and used for fine-tuning training; Using the fine-tuned video understanding large model to perform reasoning using a thinking chain framework, including: using the fine-tuned video understanding large model to sequentially perform event detection and harmfulness detection on the input video and text, and generate a final harmful content detection result; The process of combining input video and text for event detection and harmfulness detection includes: designing prompts. prompt words It contains predefined definitions of N harmful categories and requires fine-tuning of the trained large-scale video understanding model. Focus on the visual features and semantic information of the input video; use prompts... Guided fine-tuning of the trained large video understanding model Generate input video Textual summary of the event , represented as: ； wherein the text summary of the event i.e., the event detection result; Based on the event detection results, and the definition of the predefined N harmful categories, computing a textual summarization of the event With the semantic similarity to the harmful categories, generating a preliminary harmfulness score As a harmfulness detection result; The final harmful content detection result is represented as: ； wherein, is the event detection result, which is the input video is the text summary of the event in the input video , is the preliminary harmfulness score, which is the harmfulness detection result; is the input text, is the fine-tuned trained video understanding large model, is the final harmful content detection result.

2. The method of claim 1, wherein, The calculation of the semantic correlation score of each video frame and harmful content includes: The semantic relevance score is computed using a semantic scoring strategy of CLIP, for a harmful content related expression T, it is the semantic relevance score of the given video frame ​ ； wherein CLIP is a contrastive language-image pre-training model, a semantic correlation score between the expression T related to harmful content for a given video frame a semantic correlation score between the expression T related to harmful content for a given video frame 3. The method of claim 1, wherein, The video frame sampling according to the semantic correlation score and the fine-tuning training includes: The higher the semantic correlation score is, the higher the video frame sampling priority is. After the video frame sequence is sampled, a similar frame merging algorithm is used to compress the video frame sequence, including: calculating the image similarity of adjacent video frame pairs, selecting the adjacent video frame pair with the highest image similarity, denoted as , and fusing the image features of the two as the representative frame , replacing the video frame in the video frame sequence with the representative frame , and deleting the video frame and ; continuously repeating until the length of the video frame sequence does not exceed the preset length L, and obtaining the compressed video frame sequence; Using the compressed video frame sequence for fine-tuning training.

4. The method of claim 1, wherein, The fine-tuning training of the video understanding large model includes: using a low-rank adaptation technology to fine-tune the parameters; wherein the parameters of the video understanding large model are fixed, two weight matrices are introduced, and the two weight matrices are fine-tuned; The two weight matrices include: a matrix A with dimensions , a matrix B with dimensions ; the parameters of the video understanding large model are denoted as , with dimensions , and the forward propagation process during fine-tuning training is represented as: ； wherein, , and are set positive integers; denotes the output vector during forward propagation, x is the input vector during forward propagation, is the weight update matrix.

5. The method of claim 1, wherein, Further comprising: Building a dataset with video content safety understanding challenges for evaluating the fine-tuned video understanding large model; The building method comprises: Collecting video data, and guiding an external large language model to re-label the video data according to the predefined N harmful categories, that is, mapping each video data to the N harmful categories, or marking it as a harmless category; Using the N harmful categories as keywords, combining content filtering and quality evaluation technology, and screening videos from public video resources to supplement the re-labeled video data, so that the built dataset with video content safety understanding challenges is balanced in categories.

6. A video content security understanding system characterized by, For implementing the method of any one of claims 1-5, comprising: A dataset collection unit is configured to collect an image dataset containing harmless pictures and harmful pictures, and a video dataset containing harmless videos and harmful videos; wherein the harmful pictures and the harmful videos cover a predefined N harmful categories; A fine-tuning training unit is configured to combine the image dataset and the video dataset, and use an image and video mixed fine-tuning strategy to fine-tune a video understanding large model; wherein the video understanding large model is configured with a video frame sampling module based on a semantic scoring strategy, used to calculate the semantic correlation score of each video frame and harmful content, sample video frames according to the semantic correlation score, and used for fine-tuning training; A detection unit is configured to use the fine-tuned video understanding large model to perform reasoning using a thinking chain framework, including: using the fine-tuned video understanding large model to sequentially perform event detection and harmfulness detection on the input video and text, and generate a final harmful content detection result.

7. A processing device, characterized by Comprise: One or more processors; Memory for storing one or more programs; wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to carry out the method of any one of claims 1-5.

8. A readable storage medium, storing a computer program, characterized in that, The computer program, when executed by the processor, carries out the method of any one of claims 1-5.

Images