Sensitive content detection method and apparatus, electronic device, and storage medium

Abstract

This invention provides a method, apparatus, electronic device, and storage medium for detecting sensitive content, relating to the field of content security technology. The method includes: acquiring a video to be detected; inputting the video to be detected into a visual element analysis model to obtain visual element results and visual features output by the visual element analysis model; inputting the video to be detected into an auditory element analysis model to obtain auditory element results and auditory features output by the auditory element analysis model; inputting the visual features and auditory features into an event detection model to output event detection results characterizing whether sensitive content is contained; matching the visual element results and auditory element results with a sensitive content rule base to output sensitive event types; and combining the event detection results and sensitive event types to determine the sensitive content detection result. This invention enables comprehensive detection of sensitive content in videos, improving detection flexibility and accuracy.

Description

Technical Field

[0001] This invention relates to the field of content security technology, and in particular to a method, apparatus, electronic device, and storage medium for detecting sensitive content. Background Technology

[0002] In today's society, watching videos via the internet or television signals has become an indispensable activity in people's daily lives. Everyone can watch all videos on the internet or television. However, some videos containing sensitive content are not suitable for users to watch, especially not for children or teenagers. Therefore, it is very important to know how to detect whether a video contains sensitive content.

[0003] In existing technologies, a common approach is to predefine a set of security events and train a neural network using sample videos containing these events to obtain multiple detection models. These trained models are then used to detect video content. However, in this method, the detection models typically only detect one type of security event, resulting in incomplete and inaccurate detection results. Summary of the Invention

[0004] This invention provides a sensitive content detection method, apparatus, electronic device, and storage medium to address the shortcomings of incomplete detection results in existing technologies, achieve comprehensive detection of sensitive video content, and improve detection flexibility and accuracy.

[0005] This invention provides a method for detecting sensitive content, comprising:

[0006] Obtain the video to be tested;

[0007] The video to be detected is input into the visual element analysis model to obtain the visual element results and visual features output by the visual element analysis model.

[0008] The video to be detected is input into the auditory element analysis model to obtain the auditory element results and auditory features output by the auditory element analysis model;

[0009] The visual and auditory features are input into the event detection model, and the event detection result is output to characterize whether sensitive content is contained.

[0010] The visual element results and the auditory element results are matched with the sensitive content rule base to output the sensitive event type. The sensitive content detection result is determined by combining the event detection result and the sensitive event type.

[0011] According to the sensitive content detection method provided by the present invention, the step of inputting the video to be detected into a visual element analysis model to obtain the visual element results and visual features output by the visual element analysis model includes:

[0012] The video to be detected is input into the visual classification model of the visual element analysis model, and the first visual element result and the corresponding first visual feature are output. The first visual element result is used to characterize the visual elements of people and objects in the video to be detected.

[0013] The video to be detected is input into the detection model of the visual element analysis model, and the second visual element result and the corresponding second visual feature are output. The second visual element result is used to characterize the positional relationship between people and objects in the video to be detected.

[0014] The video to be detected is input into the interactive model of the visual element analysis model, and the third visual element results and corresponding third visual features are output. The third visual element results are used to characterize the interaction relationship between people and objects and the interaction relationship between people in the video to be detected.

[0015] The visual element result is determined based on the first visual element result, the second visual element result, and the third visual element result;

[0016] The visual features are determined based on the first visual feature, the second visual feature, and the third visual feature.

[0017] According to the sensitive content detection method provided by the present invention, the step of inputting the video to be detected into an auditory element analysis model to obtain the auditory element results and auditory features output by the auditory element analysis model includes:

[0018] The video to be detected is input into the auditory classification model of the auditory element analysis model, and the first auditory element result and the corresponding first auditory feature are output. The first auditory element result is used to characterize the auditory elements of people, objects and environment in the video to be detected.

[0019] The video to be detected is input into the localization model of the auditory element analysis model, and the second auditory element result and the corresponding second auditory feature are output. The second auditory element result is used to characterize the sound source location of people and objects in the video to be detected.

[0020] Based on the first auditory element result and the second auditory element result, the auditory element result is determined;

[0021] The auditory features are determined based on the first auditory feature and the second auditory feature.

[0022] According to the sensitive content detection method provided by the present invention, the step of inputting the visual features and the auditory features into an event detection model and outputting an event detection result characterizing whether sensitive content is contained includes:

[0023] The visual features are input into the visual feature processing model of the event detection model, and the visual fusion features are output.

[0024] The auditory features are input into the auditory feature processing model of the event detection model, and the auditory fusion features are output.

[0025] The visual fusion features and the auditory fusion features are input into the multimodal fusion model of the event detection model, and the event detection result is output to characterize whether sensitive content is contained.

[0026] According to the sensitive content detection method provided by the present invention, the step of matching the visual element results and the auditory element results with a sensitive content rule base, outputting a sensitive event type, and combining the event detection results and the sensitive event type to determine the sensitive content detection result includes:

[0027] The visual element results are arbitrarily combined and matched with the visual element rules in the sensitive content rule base, and the first sensitive event type corresponding to the matched visual element rule is output.

[0028] The auditory element results are arbitrarily combined and matched with the auditory element rules in the sensitive content rule base, and the second sensitive event type corresponding to the matched auditory element rule is output.

[0029] The visual element results and the auditory element results are arbitrarily combined and matched with the cross-element rules in the sensitive content rule base, and the third sensitive event type corresponding to the matched cross-element rule is output.

[0030] Based on the first sensitive event type, the second sensitive event type, and the third sensitive event type, the sensitive event type corresponding to the video to be detected is statistically obtained;

[0031] Based on the event detection results and the sensitive event types, the sensitive content detection results corresponding to the video to be detected are determined.

[0032] The sensitive content detection method provided by the present invention further includes:

[0033] A new sensitive event is acquired, and the video corresponding to the new sensitive event is input into the visual element analysis model and the auditory element analysis model respectively to obtain the new visual element results and new visual features output by the visual element analysis model, and the new auditory element results and new auditory features output by the auditory element analysis model.

[0034] Based on the new visual element results and the new auditory element results, new visual element rules, new auditory element rules, and new cross-element rules corresponding to the new sensitive event are determined.

[0035] If the video to be detected matches at least one of the new visual element rules, the new auditory element rules, and the new cross-element rules, the sensitive event type is determined as the sensitive content detection result.

[0036] The present invention also provides a sensitive content detection device, comprising:

[0037] The acquisition module is used to acquire the video to be detected;

[0038] The visual element analysis module is used to input the video to be detected into the visual element analysis model to obtain the visual element results and visual features output by the visual element analysis model.

[0039] The auditory element analysis module is used to input the video to be detected into the auditory element analysis model to obtain the auditory element results and auditory features output by the auditory element analysis model;

[0040] The event detection module is used to input the visual features and the auditory features into the event detection model and output the event detection result to characterize whether sensitive content is contained.

[0041] The multi-clue reasoning module is used to match the visual element results and the auditory element results with the sensitive content rule base, output the sensitive event type, and combine the event detection results and the sensitive event type to determine the sensitive content detection result.

[0042] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the sensitive content detection method as described above.

[0043] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the sensitive content detection method as described above.

[0044] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the sensitive content detection method as described above.

[0045] The sensitive content detection method, apparatus, electronic device, and storage medium provided by this invention input the target to be detected into a visual element analysis model and an auditory element analysis model respectively to obtain visual element results, visual features, auditory element results, and auditory features. The basic unit of sensitive content is further parsed from events into elements. Then, the visual features and auditory features are input into an event detection model to obtain event detection results that characterize whether sensitive content is contained. At the same time, the visual element results, auditory element results, and any combination thereof are matched with a sensitive content rule base to obtain sensitive event types. The event detection results and sensitive event types are combined for fusion decision-making to obtain the final sensitive content detection result. Based on elements and their combinations, sensitive content is comprehensively detected, improving detection flexibility and accuracy. At the same time, it can meet the detection requirements of various scenarios and expand the scope of application. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0047] Figure 1 This is one of the flowcharts of the sensitive content detection method provided by the present invention;

[0048] Figure 2 This is the second flowchart of the sensitive content detection method provided by the present invention;

[0049] Figure 3 This is a schematic diagram of the sensitive content detection device provided by the present invention;

[0050] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0052] In today's society, watching videos via the internet or television signals has become an indispensable activity in people's daily lives. Everyone can watch all videos on the internet or television. However, some videos containing sensitive content are not suitable for users to watch, especially not for children or teenagers. Therefore, it is very important to know how to detect whether a video contains sensitive content.

[0053] In existing technologies, a common approach is to predefine a set of security events and train a neural network using sample videos containing these events, thereby obtaining multiple detection models. These trained models are then used to detect video content. However, in this method, the detection models can typically only detect one type of security event, resulting in incomplete and inaccurate detection results. Furthermore, the basic unit of these conventional methods is the security event. If a sudden new sensitive event needs to be detected, the entire detection model must be retrained. The cost of updating and adapting to the new sensitive event is high, and it may further lead to the detection model being unable to effectively detect sudden security events during the adaptation period, resulting in security vulnerabilities.

[0054] To address the aforementioned problems, this invention provides a sensitive content detection method that further decomposes security events into visual and auditory elements. These elements are then used as the most basic units for learning, followed by detection and decision-making. Specifically, the sensitive content detection method provided in this invention first defines basic visual and auditory elements in the training dataset. Then, it uses techniques such as multi-task and multi-element interaction networks to learn and detect elements at the element dimension. After obtaining the perceptual ability of the basic elements, the elements are integrated based on rules and models to obtain the sensitive content detection result. Furthermore, the model framework provided in this invention, when encountering new sensitive events, only requires key element analysis and rule adjustment to achieve good detection capabilities for sudden events. Simultaneously, the model is periodically updated; only the event detection model needs to be updated, without updating the visual element analysis model and the auditory element analysis model, further improving the detection capability for new events.

[0055] Figure 1 This is one of the flowcharts illustrating the sensitive content detection method provided by the present invention, such as... Figure 1 As shown, the method includes:

[0056] Step 110: Obtain the video to be tested;

[0057] Step 120: Input the video to be detected into the visual element analysis model to obtain the visual element results and visual features output by the visual element analysis model;

[0058] Step 130: Input the video to be detected into the auditory element analysis model to obtain the auditory element results and auditory features output by the auditory element analysis model;

[0059] Step 140: Input the visual features and the auditory features into the event detection model, and output the event detection result to characterize whether sensitive content is contained;

[0060] Step 150: Match the visual element results and the auditory element results with the sensitive content rule base, output the sensitive event type, and combine the event detection results and the sensitive event type to determine the sensitive content detection result.

[0061] Specifically, existing solutions provide only one type of sensitive content detection result for the video to be detected, resulting in a rather one-sided assessment. In this embodiment of the invention, the acquired video to be detected is input into a visual element analysis model and an auditory element analysis model, respectively. This yields visual element results and visual features output by the visual element analysis model, as well as auditory element results and auditory features output by the auditory element analysis model. The basic unit of sensitive content is further analyzed from events to elements. Then, the visual and auditory features are input into an event detection model to detect whether sensitive content is contained. Simultaneously, the visual element results, auditory element results, and any combination thereof are matched with a sensitive content rule base to statistically determine specific sensitive event types. Based on the application requirements of different scenarios, the event detection results and sensitive event types are fused and decided to obtain the final sensitive content detection result. Sensitive content detection is performed on an element-by-element basis, making the detection results more comprehensive and accurate, and the detection process more flexible.

[0062] Optionally, the video to be detected can be video data extracted from network or television signals that needs to be detected for sensitive content.

[0063] Optionally, before performing step 120, a visual training dataset can be pre-constructed to train the initial visual element analysis model. When constructing the visual training dataset, visual elements are first defined, and video data related to each visual element is collected to build the training dataset. The visual elements are defined based on various computer vision tasks and determined after incorporating extensive expert experience. Specifically, the visual elements include:

[0064] (1) Visual elements of a person, including but not limited to: a person's identity, clothing, makeup, location, frequency of appearance, and movement trajectory;

[0065] (2) Visual elements of an object, including but not limited to: the type of the object, the position of the object, the material of the object, and the trajectory of the object's movement;

[0066] (3) The interaction between people and objects, including but not limited to: the positional relationship between people and objects, people holding objects, people looking at objects, people throwing objects, etc.;

[0067] (4) Interactions between people, including but not limited to: conversations between people, physical contact between people, and conflicts between people.

[0068] Optionally, since there are multiple categories of defined visual elements, the initial visual element analysis model includes an initial visual classification model, an initial detection model, and an output interaction model, wherein:

[0069] The visual element samples of people and their corresponding sample labels, and the visual element samples of objects and their corresponding sample labels are input into the initial visual classification model for training. Training is completed when the visual classification loss function is minimized, thus obtaining the visual classification model. The input of the visual classification model is a video image, and the output is the classification element information and features of the corresponding video image. In addition, the initial visual classification model can be a network such as ResNet or Transformer, and a multi-task network obtained by connecting multiple classifiers. The visual classification loss function can be the sum of the classification losses of N tasks, as shown in Equation (1), which is:

[0070] ;

[0071] in, Indicates the first There are N tasks, where N represents the total number of tasks. Indicates the first The set of input visual elements for each task. Indicates the first The set of visual element tags corresponding to each task This represents the classification loss calculation function, which can be the standard classification cross-entropy loss function. Indicates the first A classification model for each task.

[0072] The positional relationship samples of people and objects, along with their corresponding sample labels, are input into the initial detection model. Training of the initial detection model is completed when the multi-task detection loss function is minimized, resulting in a detection model. The input to the detection model is a video image, and the output includes the position of people, the categories of various objects of interest, and their positions. The initial detection model can be a Faster R-CNN network. The multi-task detection loss function is shown in equation (2), which is:

[0073] ;

[0074] in, Indicates the loss of human detection, This indicates the loss of the object being detected.

[0075] The interaction relationship samples between people and objects and their corresponding sample labels, and the interaction relationship samples between people and their corresponding sample labels are input into the initial interaction model for training. The training is completed when the interaction loss function is minimized, and the interaction model is obtained. The input of the interaction model is a video image carrying the image regions of people, objects, and other people, and the output is the interaction relationship between the two. The initial interaction model can be a fusion network of Siamese neural network and 3-layer perceptron (MLP), and the interaction loss can be the visual classification loss function as shown in Equation (1).

[0076] After training the visual classification model, detection model, and interaction model, a trained visual element analysis model is obtained. This model is then used to analyze the visual elements and features in the video to be detected.

[0077] Optionally, when constructing the visual training dataset, the proportion of visual element samples of each category in the visual training dataset should be kept as balanced as possible to avoid the deviation of visual element results and visual features caused by the excessive proportion of one type of visual element samples, so as to further ensure the rationality and accuracy of the elements analyzed by the visual element analysis model.

[0078] Optionally, the step of inputting the video to be detected into the visual element analysis model to obtain the visual element results and visual features output by the visual element analysis model includes:

[0079] The video to be detected is input into the visual classification model of the visual element analysis model, and the first visual element result and the corresponding first visual feature are output. The first visual element result is used to characterize the visual elements of people and objects in the video to be detected.

[0080] The video to be detected is input into the detection model of the visual element analysis model, and the second visual element result and the corresponding second visual feature are output. The second visual element result is used to characterize the positional relationship between people and objects in the video to be detected.

[0081] The video to be detected is input into the interactive model of the visual element analysis model, and the third visual element results and corresponding third visual features are output. The third visual element results are used to characterize the interaction relationship between people and objects and the interaction relationship between people in the video to be detected.

[0082] The visual element result is determined based on the first visual element result, the second visual element result, and the third visual element result;

[0083] The visual features are determined based on the first visual feature, the second visual feature, and the third visual feature.

[0084] Specifically, to analyze visual elements, the video to be detected is input into a visual classification model, a detection model, and an interaction model, respectively. This yields the first visual element result and first visual feature output by the visual classification model, the second visual element result and second visual feature output by the detection model, and the third visual element result and third visual feature output by the interaction model. The first visual element result, the second visual element result, and the third visual element result together form the visual element result, such as the identity of a person or the category of an object. The first visual feature, the second visual feature, and the third visual feature together form the visual feature, such as the 128-dimensional vector feature of the identity of a person.

[0085] Optionally, before performing step 130, an auditory training dataset can be pre-constructed to train the initial auditory element analysis model. When constructing the auditory training dataset, auditory elements are first defined, and video data related to each auditory element is collected to build the training dataset. Auditory elements can be defined based on various tasks such as speech processing and voiceprint recognition, and determined in conjunction with expert experience. Specifically, these include:

[0086] (1) Auditory elements of a person, including but not limited to: a person's identity, tone of voice, content of speech, vocal position, speech rate, and emotional state.

[0087] (2) Auditory elements of an object, including but not limited to: the position of the object and the frequency of the object's sound;

[0088] (3) Environmental elements, including but not limited to: whether the environment is noisy, the type of environment you may be in, etc.

[0089] Optionally, since there are multiple categories of defined auditory elements, the initial auditory element analysis model includes an initial auditory classification model and an initial localization model, wherein:

[0090] The auditory element samples of people and their corresponding sample labels, the auditory element samples of objects and their corresponding sample labels, and the environmental element samples and their corresponding sample labels are input into the initial auditory classification model for training. Training is completed when the auditory classification loss function is minimized, thus obtaining the auditory classification model. The input of the auditory classification model is a video carrying an audio signal, and the output is the classification element information and features of the corresponding video. In addition, the initial auditory classification model can be a multi-task network obtained by connecting multiple classifiers to a network such as the PSNN library. The auditory classification loss function can be the sum of the classification losses of N tasks, as shown in Equation (3), which is:

[0091] ;

[0092] in, Indicates the first There are N tasks, where N represents the total number of tasks. Indicates the first The set of auditory input elements for each task. Indicates the first The set of auditory element tags corresponding to each task This represents the classification loss calculation function, which can be the standard classification cross-entropy loss function. Indicates the first A classification model for each task.

[0093] The sound source location relationship samples of people and objects, along with their corresponding sample labels, are input into the initial localization model. Training of the initial localization model is completed when the sound source detection loss function is minimized, resulting in the localization model. The input of the localization model is a video image carrying audio signals, and the output is the sound source location. The initial localization model can be a Faster R-CNN. The sound source detection loss function... As shown in equation (4), equation (4) is:

[0094] ;

[0095] in, This represents the loss of localization of the sound source in the video image.

[0096] After training the auditory classification model and localization model, a trained auditory element analysis model is obtained, and the auditory element analysis model is used to analyze the auditory elements and auditory features in the video to be tested.

[0097] Optionally, when constructing the auditory training dataset, the proportion of auditory element samples of each category in the auditory training dataset should be kept as balanced as possible to avoid the deviation of auditory element results and auditory features caused by the excessive proportion of auditory element samples of one category, and to further ensure the rationality and accuracy of the elements analyzed by the auditory element analysis model.

[0098] Optionally, the step of inputting the video to be detected into the auditory element analysis model to obtain the auditory element results and auditory features output by the auditory element analysis model includes:

[0099] The video to be detected is input into the auditory classification model of the auditory element analysis model, and the first auditory element result and the corresponding first auditory feature are output. The first auditory element result is used to characterize the auditory elements of people, objects and environment in the video to be detected.

[0100] The video to be detected is input into the localization model of the auditory element analysis model, and the second auditory element result and the corresponding second auditory feature are output. The second auditory element result is used to characterize the sound source location of people and objects in the video to be detected.

[0101] Based on the first auditory element result and the second auditory element result, the auditory element result is determined;

[0102] The auditory features are determined based on the first auditory feature and the second auditory feature.

[0103] Specifically, to analyze auditory elements, the video to be detected is input into the auditory classification model and the localization model respectively, to obtain the first auditory element result and the first auditory feature output by the auditory classification model, and the second auditory element result and the second auditory feature output by the localization model. The first auditory element result and the second auditory element result constitute the auditory element result, such as the identity of the person, whether the environment is noisy, etc. The first auditory feature and the second auditory feature constitute the auditory feature, such as the 128-dimensional vector feature of the person's identity.

[0104] Optionally, after obtaining the visual element results, visual features, auditory element results, and auditory features, the above data can be smoothed in the temporal domain to reduce noise interference.

[0105] Optionally, before performing step 140, an event training dataset can be pre-constructed to train the initial event detection model. The initial event detection model includes: an initial visual feature processing model, an initial auditory feature processing model, and an initial multimodal fusion model, wherein:

[0106] The initial visual feature processing model can be a residual network (ResNet), InceptionNet (GoogLeNet), DenseNet (Densely connected convolutional networks), or ViT (Vision Transformer), etc. Visual feature samples can be input into the initial visual feature processing model for single-modal pre-training. When the autoencoder loss function is minimized, the training is completed, and the visual feature processing model is obtained. The input of the visual feature processing model is the visual features corresponding to the visual element results, and the output is a 256-dimensional visual fusion feature.

[0107] The initial auditory feature processing model can be FSMN (Feedforward Sequential Memory Networks) or Recurrent Neural Network (RNN), etc. Auditory feature samples can be input into the initial auditory feature processing model for single-modal pre-training. When the autoencoder loss function is minimized, the training is completed, and the auditory feature processing model is obtained. The input of the auditory feature processing model is the auditory features corresponding to the auditory element results, and the output is the auditory fusion feature with a dimension of 128.

[0108] The initial multimodal fusion model can be a Transformer model, capable of processing video and audio information in the same format. During training, visual feature samples are input into a pre-trained visual feature processing model, and auditory feature samples are input into a pre-trained auditory feature processing model. The visual and auditory fusion features output by the two models are then input into the initial multimodal fusion model for joint training. Training is completed when the loss function for sensitive event classification is minimized, resulting in the multimodal fusion model. The input to the multimodal fusion model is the visual and auditory fusion features, and the output is a binary classification result indicating whether sensitive content is contained, i.e., the event detection result.

[0109] After training the visual feature processing model, auditory feature processing model, and multimodal fusion model, a trained event detection model is obtained. The event detection model is then used to analyze the visual and auditory features in the video to be detected and output the event detection result indicating whether sensitive content is contained.

[0110] Optionally, after the event detection model is trained, the step of inputting the visual features and the auditory features into the event detection model and outputting an event detection result characterizing whether sensitive content is contained includes:

[0111] The visual features are input into the visual feature processing model of the event detection model, and the visual fusion features are output.

[0112] The auditory features are input into the auditory feature processing model of the event detection model, and the auditory fusion features are output.

[0113] The visual fusion features and the auditory fusion features are input into the multimodal fusion model of the event detection model, and the event detection result is output to characterize whether sensitive content is contained.

[0114] Specifically, in order to meet the detection requirements of multiple application scenarios, in this embodiment of the invention, visual features and auditory features are input into the event detection model to detect whether the video to be detected contains sensitive content and output the corresponding event detection results, which can be applied to scenarios where it is not necessary to determine the specific sensitive event type.

[0115] Optionally, a sensitive content rule base can be pre-built before performing step 150 to facilitate matching with visual and auditory element results. The sensitive content rule base includes visual element rules, auditory element rules, and cross-element rules, wherein:

[0116] Visual element rules can be logical combinations of visual sub-elements in various types of sensitive events; auditory element rules can be logical combinations of auditory sub-elements in various types of sensitive events; cross element rules can be logical combinations of visual and auditory sub-elements in various types of sensitive events, and cross element rules are used to fill in gaps in visual and auditory element rules.

[0117] Optionally, the step of matching the visual element results and the auditory element results with a sensitive content rule base, outputting a sensitive event type, and combining the event detection results and the sensitive event type to determine the sensitive content detection result includes:

[0118] The visual element results are arbitrarily combined and matched with the visual element rules in the sensitive content rule base, and the first sensitive event type corresponding to the matched visual element rule is output.

[0119] The auditory element results are arbitrarily combined and matched with the auditory element rules in the sensitive content rule base, and the second sensitive event type corresponding to the matched auditory element rule is output.

[0120] The visual element results and the auditory element results are arbitrarily combined and matched with the cross-element rules in the sensitive content rule base, and the third sensitive event type corresponding to the matched cross-element rule is output.

[0121] Based on the first sensitive event type, the second sensitive event type, and the third sensitive event type, the sensitive event type corresponding to the video to be detected is statistically obtained;

[0122] Based on the event detection results and the sensitive event types, the sensitive content detection results corresponding to the video to be detected are determined.

[0123] Specifically, to obtain specific sensitive event types, in this embodiment of the invention, the visual element results and any combination of their visual sub-elements are matched with visual element rules to output the first sensitive event type corresponding to the matched visual element rule; the auditory element results and any combination of their auditory sub-elements are matched with auditory element rules to input the second sensitive event type corresponding to the matched auditory element rule; to fill in any gaps, the visual and auditory sub-elements are arbitrarily combined and matched with cross-element rules to output the third sensitive event type corresponding to the matched cross-element rule; statistical analysis is performed on the first, second, and third sensitive event types, and after deduplication and combination, the sensitive event type corresponding to the video to be detected is obtained. Furthermore, according to the detection requirements of the application scenario, a fusion decision can be made on the sensitive event type and the event detection result to obtain the final sensitive content detection result, wherein the sensitive content detection result includes not only sensitive content but also the specific sensitive event type.

[0124] For example, when the detection rate of sensitive events is required to be high, that is, in a high detection rate scenario, the conclusion that the detection result of sensitive content is risk-free can only be obtained when both the sensitive event type and the event detection result are risk-free. All other situations are risky.

[0125] When the detection rate requirement for sensitive events is low, i.e. in a high pass rate scenario, the conclusion that the sensitive content detection result is risky can only be obtained when both the sensitive event type and the event detection result are risky. In other cases, it is considered a risk-free situation.

[0126] Optionally, the sensitive content detection method provided in this embodiment of the invention further includes:

[0127] New sensitive events are acquired, and the videos corresponding to the new sensitive events are input into the visual element analysis model and the auditory element analysis model respectively to obtain the new visual element results and new visual features output by the visual element analysis model, and the new auditory element results and new auditory features output by the auditory element analysis model.

[0128] Based on the new visual element results and the new auditory element results, new visual element rules, new auditory element rules, and new cross-element rules corresponding to the new sensitive event are determined.

[0129] If the video to be detected matches at least one of the new visual element rules, the new auditory element rules, and the new cross-element rules, the sensitive event type is determined as the sensitive content detection result.

[0130] Specifically, when a new sensitive event is added, existing technologies require retraining the entire detection model, making the cost of updating and adapting to the new sensitive event relatively high. Furthermore, during the update and adaptation period, it is impossible to effectively detect security vulnerabilities. To address these issues, in this embodiment of the invention, when a new sensitive event occurs, the video of the new sensitive event can be input into both a visual element analysis model and an auditory element analysis model. This yields visual element results from the visual element analysis model and auditory element results from the auditory element analysis model. Visual element rules for the new sensitive event are constructed based on the visual element results and any combination of their visual sub-elements. Auditory element rules are also constructed based on the auditory element results and any combination of their auditory sub-elements. Cross-element rules for the new sensitive event are constructed based on any combination of visual and auditory sub-elements to update the sensitive content rule base. Once the video to be detected matches the new rules, it is determined to be risky content, without requiring a fusion decision based on the sensitive event type and event detection results. In addition, based on the new sensitive events, the event detection model can be periodically updated and optimized to further improve detection performance.

[0131] Furthermore, the basic unit of the aforementioned conventional methods is a security event. Once a sudden new sensitive event needs to be detected, the entire detection model needs to be retrained. The cost of updating and adapting to the new sensitive event is relatively high, and it may further lead to the detection model being unable to effectively detect sudden security events during the adaptation period, resulting in security vulnerabilities.

[0132] Optionally, the embodiments of the present invention are not limited by hardware platforms or programming languages. Programming languages ​​such as C, C++, and Python can all be used to implement the sensitive content detection method described in the embodiments of the present invention and achieve the expected results. Furthermore, the implementation environment of the embodiments of the present invention can be a server using a 2.9GHz 128-core CPU and 128GB of memory. The training and detection of the models involved in the embodiments of the present invention are both processed on the eight Nvidia Tesla A100 GPUs mounted on the aforementioned server. On the GPUs, the embodiments of the present invention use PyTorch as the software platform for development and implementation. In practical applications, adjustments can be made flexibly according to specific application scenarios, and the embodiments of the present invention do not impose limitations in this regard.

[0133] The sensitive content detection method provided by this invention inputs the target to be detected into a visual element analysis model and an auditory element analysis model respectively, obtaining visual element results, visual features, auditory element results, and auditory features. The basic unit of sensitive content is further parsed from events into elements. Then, the visual features and auditory features are input into an event detection model to obtain event detection results that characterize whether sensitive content is contained. At the same time, the visual element results, auditory element results, and any combination thereof are matched with a sensitive content rule base to obtain sensitive event types. The event detection results and sensitive event types are combined for fusion decision-making to obtain the final sensitive content detection result. Based on elements and their combinations, sensitive content is comprehensively detected, improving detection flexibility and accuracy. At the same time, it can meet the detection requirements of various scenarios and expand the scope of application.

[0134] The sensitive content detection device provided by the present invention is described below. The sensitive content detection device described below and the sensitive content detection method described above can be referred to in correspondence.

[0135] Figure 3 This is a schematic diagram of the sensitive content detection device provided by the present invention, as shown below. Figure 3 As shown, the sensitive content detection device 200 includes: an acquisition module 201, a visual element analysis module 202, an auditory element analysis module 203, an event detection module 204, and a multi-clue reasoning module 205, wherein:

[0136] Module 201 is used to acquire the video to be detected;

[0137] The visual element analysis module 202 is used to input the video to be detected into the visual element analysis model to obtain the visual element results and visual features output by the visual element analysis model.

[0138] The auditory element analysis module 203 is used to input the video to be detected into the auditory element analysis model to obtain the auditory element results and auditory features output by the auditory element analysis model;

[0139] The event detection module 204 is used to input the visual features and the auditory features into the event detection model and output the event detection result that characterizes whether sensitive content is contained.

[0140] The multi-clue reasoning module 205 is used to match the visual element results and the auditory element results with the sensitive content rule base, output the sensitive event type, and combine the event detection results and the sensitive event type to determine the sensitive content detection result.

[0141] The sensitive content detection device provided by this invention inputs the target to be detected into a visual element analysis model and an auditory element analysis model respectively to obtain visual element results, visual features, auditory element results, and auditory features. The basic unit of sensitive content is further parsed from events into elements. Then, the visual features and auditory features are input into an event detection model to obtain event detection results that characterize whether sensitive content is contained. At the same time, the visual element results, auditory element results, and any combination thereof are matched with a sensitive content rule base to obtain sensitive event types. The event detection results and sensitive event types are combined for fusion decision-making to obtain the final sensitive content detection result. Based on elements and their combinations, sensitive content is comprehensively detected, improving detection flexibility and accuracy. At the same time, it can meet the detection requirements of various scenarios and expand the scope of application.

[0142] Optionally, the visual element analysis module 202 is specifically used for:

[0143] The video to be detected is input into the visual classification model of the visual element analysis model, and the first visual element result and the corresponding first visual feature are output. The first visual element result is used to characterize the visual elements of people and objects in the video to be detected.

[0144] The video to be detected is input into the detection model of the visual element analysis model, and the second visual element result and the corresponding second visual feature are output. The second visual element result is used to characterize the positional relationship between people and objects in the video to be detected.

[0145] The video to be detected is input into the interactive model of the visual element analysis model, and the third visual element results and corresponding third visual features are output. The third visual element results are used to characterize the interaction relationship between people and objects and the interaction relationship between people in the video to be detected.

[0146] The visual element result is determined based on the first visual element result, the second visual element result, and the third visual element result;

[0147] The visual features are determined based on the first visual feature, the second visual feature, and the third visual feature.

[0148] Optionally, the auditory element analysis module 203 is specifically used for:

[0149] The video to be detected is input into the auditory classification model of the auditory element analysis model, and the first auditory element result and the corresponding first auditory feature are output. The first auditory element result is used to characterize the auditory elements of people, objects and environment in the video to be detected.

[0150] The video to be detected is input into the localization model of the auditory element analysis model, and the second auditory element result and the corresponding second auditory feature are output. The second auditory element result is used to characterize the sound source location of people and objects in the video to be detected.

[0151] Based on the first auditory element result and the second auditory element result, the auditory element result is determined;

[0152] The auditory features are determined based on the first auditory feature and the second auditory feature.

[0153] Optionally, the event detection module 204 is specifically used for:

[0154] The visual features are input into the visual feature processing model of the event detection model, and the visual fusion features are output.

[0155] The auditory features are input into the auditory feature processing model of the event detection model, and the auditory fusion features are output.

[0156] The visual fusion features and the auditory fusion features are input into the multimodal fusion model of the event detection model, and the event detection result is output to characterize whether sensitive content is contained.

[0157] Optionally, the multi-threaded reasoning module 205 is specifically used for:

[0158] The visual element results are arbitrarily combined and matched with the visual element rules in the sensitive content rule base, and the first sensitive event type corresponding to the matched visual element rule is output.

[0159] The auditory element results are arbitrarily combined and matched with the auditory element rules in the sensitive content rule base, and the second sensitive event type corresponding to the matched auditory element rule is output.

[0160] The visual element results and the auditory element results are arbitrarily combined and matched with the cross-element rules in the sensitive content rule base, and the third sensitive event type corresponding to the matched cross-element rule is output.

[0161] Based on the first sensitive event type, the second sensitive event type, and the third sensitive event type, the sensitive event type corresponding to the video to be detected is statistically obtained;

[0162] Based on the event detection results and the sensitive event types, the sensitive content detection results corresponding to the video to be detected are determined.

[0163] Optionally, the sensitive content detection device further includes: an update module, specifically used for:

[0164] New sensitive events are acquired, and the videos corresponding to the new sensitive events are input into the visual element analysis model and the auditory element analysis model respectively to obtain the new visual element results and new visual features output by the visual element analysis model, and the new auditory element results and new auditory features output by the auditory element analysis model.

[0165] Based on the new visual element results and the new auditory element results, new visual element rules, new auditory element rules, and new cross-element rules corresponding to the new sensitive event are determined.

[0166] If the video to be detected matches at least one of the new visual element rules, the new auditory element rules, and the new cross-element rules, the sensitive event type is determined as the sensitive content detection result.

[0167] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device 300 may include a processor 310, a communications interface 320, a memory 330, and a communication bus 340. The processor 310, communications interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 can invoke logical instructions stored in the memory 330 to execute a sensitive content detection method, which includes:

[0168] Obtain the video to be tested;

[0169] The video to be detected is input into the visual element analysis model to obtain the visual element results and visual features output by the visual element analysis model.

[0170] The video to be detected is input into the auditory element analysis model to obtain the auditory element results and auditory features output by the auditory element analysis model;

[0171] The visual and auditory features are input into the event detection model, and the event detection result is output to characterize whether sensitive content is contained.

[0172] The visual element results and the auditory element results are matched with the sensitive content rule base to output the sensitive event type. The sensitive content detection result is determined by combining the event detection result and the sensitive event type.

[0173] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0174] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is capable of executing the sensitive content detection methods provided by the above methods, the method comprising:

[0175] Obtain the video to be tested;

[0176] The video to be detected is input into the visual element analysis model to obtain the visual element results and visual features output by the visual element analysis model.

[0177] The video to be detected is input into the auditory element analysis model to obtain the auditory element results and auditory features output by the auditory element analysis model;

[0178] The visual and auditory features are input into the event detection model, and the event detection result is output to characterize whether sensitive content is contained.

[0179] The visual element results and the auditory element results are matched with the sensitive content rule base to output the sensitive event type. The sensitive content detection result is determined by combining the event detection result and the sensitive event type.

[0180] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the sensitive content detection method provided by the above methods, the method comprising:

[0181] Obtain the video to be tested;

[0182] The video to be detected is input into the visual element analysis model to obtain the visual element results and visual features output by the visual element analysis model.

[0183] The video to be detected is input into the auditory element analysis model to obtain the auditory element results and auditory features output by the auditory element analysis model;

[0184] The visual and auditory features are input into the event detection model, and the event detection result is output to characterize whether sensitive content is contained.

[0185] The visual element results and the auditory element results are matched with the sensitive content rule base to output the sensitive event type. The sensitive content detection result is determined by combining the event detection result and the sensitive event type.

[0186] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0187] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0188] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting sensitive content, characterized in that, include: Obtain the video to be tested; The video to be detected is input into the visual element analysis model to obtain the visual element results and visual features output by the visual element analysis model. The visual element results include the first visual element results, the second visual element results, and the third visual element results; The first visual element result is used to characterize the visual elements of people and objects in the video to be detected; The second visual element result is used to characterize the positional relationship between people and objects within the video to be detected; The third visual element results are used to characterize the interaction relationships between people and objects, as well as the interaction relationships between people, within the video to be detected. The video to be detected is input into the auditory element analysis model to obtain the auditory element results and auditory features output by the auditory element analysis model; the auditory element results include a first auditory element result and a second auditory element result; the first auditory element result is used to characterize the auditory elements of people, objects, and environment in the video to be detected; the second auditory element result is used to characterize the sound source positions of people and objects in the video to be detected. The visual and auditory features are input into the event detection model, and the event detection result is output to characterize whether sensitive content is contained. The visual element results and the auditory element results are matched with the sensitive content rule base to output the sensitive event type. The sensitive content detection result is determined by combining the event detection result and the sensitive event type.

2. The sensitive content detection method according to claim 1, characterized in that, The step of inputting the video to be detected into the visual element analysis model to obtain the visual element results and visual features output by the visual element analysis model includes: The video to be detected is input into the visual classification model of the visual element analysis model, and the first visual element result and the corresponding first visual feature are output. The video to be detected is input into the detection model of the visual element analysis model, and the second visual element result and the corresponding second visual feature are output. The video to be detected is input into the interactive model of the visual element analysis model, and the third visual element result and the corresponding third visual feature are output. The visual element result is determined based on the first visual element result, the second visual element result, and the third visual element result; The visual features are determined based on the first visual feature, the second visual feature, and the third visual feature.

3. The sensitive content detection method according to claim 1, characterized in that, The step of inputting the video to be detected into the auditory element analysis model to obtain the auditory element results and auditory features output by the auditory element analysis model includes: The video to be detected is input into the auditory classification model of the auditory element analysis model, and the first auditory element result and the corresponding first auditory feature are output. The video to be detected is input into the localization model of the auditory element analysis model, and the second auditory element result and the corresponding second auditory feature are output. Based on the first auditory element result and the second auditory element result, the auditory element result is determined; The auditory features are determined based on the first auditory feature and the second auditory feature.

4. The sensitive content detection method according to claim 1, characterized in that, The step of inputting the visual features and the auditory features into the event detection model and outputting an event detection result characterizing whether sensitive content is contained includes: The visual features are input into the visual feature processing model of the event detection model, and the visual fusion features are output. The auditory features are input into the auditory feature processing model of the event detection model, and the auditory fusion features are output. The visual fusion features and the auditory fusion features are input into the multimodal fusion model of the event detection model, and the event detection result is output to characterize whether sensitive content is contained.

5. The sensitive content detection method according to claim 1, characterized in that, The step of matching the visual element results and the auditory element results with the sensitive content rule base, outputting the sensitive event type, and combining the event detection results and the sensitive event type to determine the sensitive content detection result includes: The visual element results are arbitrarily combined and matched with the visual element rules in the sensitive content rule base, and the first sensitive event type corresponding to the matched visual element rule is output. The auditory element results are arbitrarily combined and matched with the auditory element rules in the sensitive content rule base, and the second sensitive event type corresponding to the matched auditory element rule is output. The visual element results and the auditory element results are arbitrarily combined and matched with the cross-element rules in the sensitive content rule base, and the third sensitive event type corresponding to the matched cross-element rule is output. Based on the first sensitive event type, the second sensitive event type, and the third sensitive event type, the sensitive event type corresponding to the video to be detected is statistically obtained; Based on the event detection results and the sensitive event types, the sensitive content detection results corresponding to the video to be detected are determined.

6. The sensitive content detection method according to any one of claims 1 to 5, characterized in that, Also includes: A new sensitive event is acquired, and the video corresponding to the new sensitive event is input into the visual element analysis model and the auditory element analysis model respectively to obtain the new visual element results and new visual features output by the visual element analysis model, and the new auditory element results and new auditory features output by the auditory element analysis model. Based on the new visual element results and the new auditory element results, new visual element rules, new auditory element rules, and new cross-element rules corresponding to the new sensitive event are determined. If the video to be detected matches at least one of the new visual element rules, the new auditory element rules, and the new cross-element rules, the sensitive event type is determined as the sensitive content detection result.

7. A sensitive content detection device, characterized in that, include: The acquisition module is used to acquire the video to be detected; The visual element analysis module is used to input the video to be detected into the visual element analysis model to obtain the visual element results and visual features output by the visual element analysis model; the visual element results include a first visual element result, a second visual element result, and a third visual element result; The first visual element result is used to characterize the visual elements of people and objects in the video to be detected; The second visual element result is used to characterize the positional relationship between people and objects within the video to be detected; The third visual element results are used to characterize the interaction relationships between people and objects, as well as the interaction relationships between people, within the video to be detected. The auditory element analysis module is used to input the video to be detected into the auditory element analysis model to obtain the auditory element results and auditory features output by the auditory element analysis model; the auditory element results include a first auditory element result and a second auditory element result; the first auditory element result is used to characterize the auditory elements of people, objects, and environment in the video to be detected; the second auditory element result is used to characterize the sound source positions of people and objects in the video to be detected; The event detection module is used to input the visual features and the auditory features into the event detection model and output the event detection result to characterize whether sensitive content is contained. The multi-clue reasoning module is used to match the visual element results and the auditory element results with the sensitive content rule base, output the sensitive event type, and combine the event detection results and the sensitive event type to determine the sensitive content detection result.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the sensitive content detection method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the sensitive content detection method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the sensitive content detection method as described in any one of claims 1 to 6.

Images