Multi-mode illegal short video advertisement detection method and related equipment
By combining contrastive learning, membership networks, and gating networks in a deep fuzzy neural network model, the problems of high computational cost and poor interpretability in multimodal short video ad detection are solved, achieving efficient and accurate detection results.
Patent Information
- Application Number
- CN202511712312.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-20
- Publication Date
- 2026-02-27
AI Technical Summary
Existing models for detecting illegal short video ads are computationally expensive when processing multimodal data, struggle to capture intermodal relationships, and lack interpretability in neural networks, resulting in low detection efficiency and accuracy.
A deep fuzzy neural network model is adopted. Image, text and audio features are extracted through a contrastive learning network module, feature fuzzification is performed using a membership network module, control rules are generated by a gating network, and finally the detection results are output by the fuzzy neural network module.
It improves the efficiency and accuracy of short video ad detection, better captures multimodal feature differences, and enhances the interpretability and credibility of the detection.
Smart Images
Figure CN121582846A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of illegal video detection technology, and in particular to a multimodal method and related equipment for detecting illegal short video advertisements. Background Technology
[0002] With the development of cloud computing, big data, the Internet of Things, mobile internet, and artificial intelligence, new business formats, new models, and new consumption patterns in the digital society have become powerful engines driving the innovative development of the digital advertising industry. Under this new development context, digital advertising has developed new characteristics such as e-commerce integration, social interaction, dynamism, contextualization, and intelligence.
[0003] With the development of technology and the rise of video platforms, video content has become the most popular information dissemination medium. Among them, short video platforms have become the most mainstream media in recent years and an important platform for digital advertising. However, illegal and irregular advertisements inevitably exist, which poses a challenge to the online advertising supervision work of relevant departments and enterprises.
[0004] Short video detection differs from traditional advertising detection, which relies on single image or text analysis. Advertisements in short videos are often new forms of advertising, such as product placement ads and "advertorials," which are difficult for machines to detect. These short video ads often have multiple modalities, and these modalities are interconnected. Detecting only a single modality often results in the loss of crucial violation information. Traditional illegal and non-compliant advertising detection models are limited by their data pathways and complex structures. When dealing with large-scale, high-dimensional data, they often require extremely high computational costs and struggle to express the relationships between data. This results in fuzzy inference systems having lower prediction accuracy than some algorithms with network structures. While traditional neural networks can effectively process high-dimensional data, the feedforward propagation network path constructed through the topological relationships of numerous neurons means that the weight coefficients generated by the iterative processes of neurons in the hidden layers cannot directly relate to the results or represent any real-world significance directly related to the task being processed—a phenomenon commonly referred to as the "black box" nature. This leads to a lack of interpretability in neural networks, affecting the credibility and acceptability of the results.
[0005] In summary, due to the lack of standards, rules, and relevant data for identifying illegal short video advertisements, it is difficult to build traditional rule-based detection models. Neural network models also lack interpretability, making it difficult to train models to achieve good results. Furthermore, most traditional models only perform distributed detection for a single modality, such as images or text, which is significantly less efficient for detecting highly perturbative short videos. Summary of the Invention
[0006] This invention provides a multimodal method and related equipment for detecting illegal short video advertisements, with the aim of improving the detection efficiency and accuracy of short video advertisements.
[0007] To achieve the above objectives, the present invention provides a multimodal method for detecting illegal short video advertisements, comprising:
[0008] Step 1: Based on the acquired short video ad dataset, obtain image data, text data, and audio data. The short video ad dataset includes multiple compliant short video ad data and multiple non-compliant short video ad data.
[0009] Step 2: Input image data, text data, and audio data into the constructed deep fuzzy neural network model, train the deep fuzzy neural network model, and obtain the illegal short video advertisement detection model;
[0010] Step 3: Input the image data, text data, and audio data from the target short video ad data into the illegal short video ad detection model for detection, and obtain the detection results;
[0011] The deep fuzzy neural network model includes a contrastive learning network module, a membership network module, a gating network module, and a fuzzy neural network module;
[0012] The contrastive learning network module is used to perform contrastive learning on the input data in order to output contrastive feature vectors;
[0013] The membership network module is used to calculate the membership degree of the contrastive feature vectors output by the contrastive learning network module in order to output fuzzy variables;
[0014] The gating network module is only used to generate control rules during the training process of the deep fuzzy neural network model;
[0015] During training, the fuzzy neural network module is used to defuzzify fuzzy variables according to the fuzzy weights in the control rules to output detection results. During application, it is used to defuzzify the fuzzy variables output by the member network module to output detection results.
[0016] Furthermore, based on the acquired short video ad dataset, image data, text data, and audio data are obtained, including:
[0017] Preprocess the short video ad dataset and extract multiple video frames as image data;
[0018] The short video ad dataset is segmented based on each video frame to obtain multiple video segments;
[0019] Extract the audio and text data corresponding to each video segment.
[0020] To elaborate further, the contrastive learning network module includes:
[0021] An image encoder is used to encode image data to obtain multiple image vectors;
[0022] A text encoder is used to encode text data to obtain multiple text vectors;
[0023] An audio encoder is used to encode audio data to obtain multiple audio vectors;
[0024] The first vector space is used to learn by comparing each image vector with each text vector to obtain the first contrast feature vector;
[0025] The second vector space is used to learn by comparing every two image vectors to obtain the second contrast feature vector.
[0026] The third vector space is used to compare and learn every two text vectors to obtain the third contrast feature vector.
[0027] The fourth vector space is used to learn by comparing every two audio vectors to obtain the fourth contrast feature vector.
[0028] Furthermore, the membership network module is constructed based on the membership function, the expression of which is:
[0029]
[0030] in, Indicates the membership degree value. This represents the input contrast feature vector. Represents the weight matrix. This represents the bias vector.
[0031] Furthermore, the fuzzy weights are obtained by performing fuzzy logic operations on each membership value, and the expression is:
[0032]
[0033] in, Indicates the weight of the control rules. This represents the probability that a fuzzy variable is selected by each rule. This represents the membership value of the first contrastive feature vector. This represents the membership value of the second contrastive feature vector. This represents the membership value of the third contrastive eigenvector. This represents the membership value of the fourth contrastive eigenvector.
[0034] Furthermore, the expression for calculating the detection result is:
[0035]
[0036] in, Indicates the test results. Indicates the normalized weights. represents a fuzzy variable, and represents the total number of fuzzy variables.
[0037] Furthermore, training a deep fuzzy neural network model includes:
[0038] The loss value is calculated using partial detection results and mean square error output from the fuzzy deep network module:
[0039]
[0040] in, Indicates the loss value. Indicates the number of samples. This represents the true value of the t-th sample. This represents the predicted value of the t-th sample;
[0041] The backpropagation gradient is calculated based on the loss value to update the parameters of the deep fuzzy neural network model layer by layer. The calculation expression is as follows:
[0042]
[0043] in, Represents the gradient value. Indicates the first The first in the layer One parameter, Indicates the first The layer is affected by the first The number of nodes affected by a node in a layer.
[0044] The present invention also provides a multimodal illegal short video advertisement detection device, comprising:
[0045] The acquisition module is used to obtain image data, text data, and audio data based on the acquired short video ad dataset. The short video ad dataset includes multiple compliant short video ad data and multiple non-compliant short video ad data.
[0046] The training module is used to input image data, text data, and audio data into a deep fuzzy neural network model, train the deep fuzzy neural network model, and obtain an illegal short video advertisement detection model.
[0047] The detection module is used to input the image data, text data, and audio data from the target short video ad data into the illegal short video ad detection model for detection and to obtain the detection results;
[0048] The deep fuzzy neural network model includes a contrastive learning network module, a membership network module, a gating network module, and a fuzzy neural network module;
[0049] The contrastive learning network module is used to perform contrastive learning on the input data in order to output contrastive feature vectors;
[0050] The membership network module is used to calculate the membership degree of the contrastive feature vectors output by the contrastive learning network module in order to output fuzzy variables;
[0051] The gating network module is only used to generate control rules during the training process of the deep fuzzy neural network model;
[0052] During training, the fuzzy neural network module is used to defuzzify fuzzy variables according to the fuzzy weights in the control rules to output detection results. During application, it is used to defuzzify the fuzzy variables output by the member network module to output detection results.
[0053] The present invention also provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a method for detecting multimodal illegal short video advertisements.
[0054] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements a method for detecting multimodal illegal short video advertisements.
[0055] The above-described solution of the present invention has the following beneficial effects:
[0056] This invention is based on a short video ad dataset containing multiple compliant and multiple non-compliant short video ad data. Image, text, and audio data are obtained and input into a constructed deep fuzzy neural network model. The deep fuzzy neural network model is trained to obtain a non-compliant short video ad detection model. The image, text, and audio data from the target short video ad data are then input into the non-compliant short video ad detection model for detection, yielding detection results. The deep fuzzy neural network model includes a contrastive learning network module, a membership network module, a gating network module, and a fuzzy neural network module. Compared with existing technologies, this invention extracts feature data from three modalities (image, text, and audio) in short videos through contrastive learning, better capturing the differences between features of different modalities and converting them into contrastive feature vectors that can be used for classification. Then, the membership network module is used for feature fuzzification. During the training process, the gating network module forms control rules, enabling the fuzzy neural network module to output detection results based on these control rules. This improves the detection efficiency and accuracy of short video ad data from the perspectives of image, text, and audio modalities.
[0057] Other beneficial effects of the present invention will be described in detail in the following detailed description section. Attached Figure Description
[0058] Figure 1 This is a flowchart illustrating an embodiment of the present invention;
[0059] Figure 2 This is a framework diagram of the deep fuzzy neural network model in an embodiment of the present invention;
[0060] Figure 3 This is a schematic diagram of the multimodal illegal short video advertisement detection device in an embodiment of the present invention;
[0061] Figure 4 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation
[0062] To make the technical problems, solutions, and advantages of this invention clearer, a detailed description will be provided below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0063] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.
[0064] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a locking connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.
[0065] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0066] This invention addresses existing problems by providing a multimodal method and related equipment for detecting illegal short video advertisements.
[0067] like Figure 1 , Figure 2 As shown, an embodiment of the present invention provides a method for detecting multimodal illegal short video advertisements, including:
[0068] Step 1: Based on the acquired short video ad dataset, obtain image data, text data, and audio data. The short video ad dataset includes multiple compliant short video ad data and multiple non-compliant short video ad data.
[0069] Step 2: Input image data, text data, and audio data into the constructed deep fuzzy neural network model, train the deep fuzzy neural network model, and obtain the illegal short video advertisement detection model;
[0070] Step 3: Input the image data, text data, and audio data from the target short video ad data into the illegal short video ad detection model for detection, and obtain the detection results;
[0071] The deep fuzzy neural network model includes a contrastive learning network module, a membership network module, a gating network module, and a fuzzy neural network module;
[0072] The contrastive learning network module is used to perform contrastive learning on the input data in order to output contrastive feature vectors;
[0073] The membership network module is used to calculate the membership degree of the contrastive feature vectors output by the contrastive learning network module in order to output fuzzy variables;
[0074] The gating network module is only used to generate control rules during the training process of the deep fuzzy neural network model;
[0075] During training, the fuzzy neural network module is used to defuzzify fuzzy variables according to the fuzzy weights in the control rules to output detection results. During application, it is used to defuzzify the fuzzy variables output by the member network module to output the detection results of the target short video advertisement data. The detection results are divided into illegal short video advertisements and compliant short video advertisements.
[0076] Specifically, obtaining the short video ad dataset includes:
[0077] Obtaining short video advertising datasets ;
[0078] Short video advertising dataset The data is categorized and labeled to form compliant short video advertising data. The data on illegal short video ads is categorized and classified to form illegal video ad data. ;
[0079] Data on illegal video ads The test set was obtained by randomly allocating the samples in a 7:3 ratio. and part of the training set The real training set is .
[0080] This invention defines short video advertisements that involve content related to guidance issues, politically sensitive information, absolute and exaggerated language, sensitive or prohibited words, or suspected medical terminology as illegal short video advertisements.
[0081] Specifically, based on the acquired short video ad dataset, image data, text data, and audio data are obtained, including:
[0082] Preprocess the short video ad dataset and extract multiple video frames as image data;
[0083] The short video ad dataset is segmented based on each video frame to obtain multiple video segments;
[0084] Extract the audio and text data corresponding to each video segment.
[0085] It should be noted that the preprocessing method can be to extract multiple video frames from the short video ad dataset at preset time intervals, or to obtain multiple video frames through optical flow calculation. This embodiment of the invention does not specifically limit the method of collecting image data from the short video ad dataset.
[0086] In this embodiment of the invention, the contrastive learning network module is used to perform contrastive learning on images, text, and audio respectively, and outputs contrastive feature vectors that are not limited to image-text, image-image, audio-audio, and text-text. For example, image and text features are embedded into a shared vector space at the same time, so that similar images and texts are closer in this space, while dissimilar images and texts are farther apart, and finally outputs contrastive feature vectors that show that the images and texts have similar semantic expressions.
[0087] The preferred configuration is a contrastive learning network module that includes:
[0088] An image encoder is used to encode image data to obtain multiple image vectors;
[0089] A text encoder is used to encode text data to obtain multiple text vectors;
[0090] An audio encoder is used to encode audio data to obtain multiple audio vectors;
[0091] The first vector space is used to learn by comparing each image vector with each text vector to obtain the first contrast feature vector;
[0092] The second vector space is used to learn by comparing every two image vectors to obtain the second contrast feature vector.
[0093] The third vector space is used to compare and learn every two text vectors to obtain the third contrast feature vector.
[0094] The fourth vector space is used to learn by comparing every two audio vectors to obtain the fourth contrast feature vector.
[0095] It should be noted that image encoders, text encoders, and audio encoders are implemented using tools such as, but not limited to, visual transformers or Transformers. Figure 2 The Encoder in the code obtains feature vectors for each modality of data. For example, in the image-text contrastive learning process, the image vectors are processed into feature vectors. and text vector Embedded into a shared vector space of size N×N, then cosine similarity is calculated, and temperature parameter is used. The data is scaled, then normalized to a probability distribution using softmax. Finally, image-text feature pairs with a similarity greater than k for the violation features are selected as the first contrast feature vector X. The first contrast feature vector includes... Based on the same contrastive learning method, we can obtain graph-graph feature pairs with a similarity greater than k as the second contrastive feature vector Y, text-text feature pairs with a similarity greater than k as the third contrastive feature vector Z, and audio-audio feature pairs with a similarity greater than k as the fourth contrastive feature vector O.
[0096] The preferred embodiment is that the membership network module is constructed based on the membership function, which is set in the fully connected layer. The expression of the fully connected layer is as follows: sigmoid is the activation function The purpose is to map feature vectors to the 0 and 1 intervals of a fuzzy set, that is, to map vectors to fuzzy sets, specifying fuzzy variables that a given vector belongs to different fuzzy sets. , , , The degree of fuzziness, the expression for the membership function is:
[0097]
[0098] in, Indicates the membership degree value. This represents the input contrast feature vector. , Represents the weight matrix. This represents the bias vector.
[0099] In this embodiment of the invention, the membership network module is essentially composed of a linear layer and an activation layer, used to output multiple fuzzy variables based on fuzzy sets, such as... .
[0100] In this embodiment of the invention, the gated network module includes multiple gated network sub-modules, and each gated network sub-module includes three gate units (Gates).
[0101] It should be noted that the gating network submodule is the same as the gating network.
[0102] Specifically, the gating network module uses a set of gating units to control the transmission process between the member network module and the fuzzy neural network module using 0 or 1, generating control rules as follows: yes , yes , yes , yes "This stage is only adjusted during the training process."
[0103] During training, the gated network module receives fuzzy variables from its member network modules as input and outputs control rules. However, fuzzy variables are not used as inputs to the gate unit (Gate) in the gated network module; ultimately, the control rules are used instead. The selection of the control gating unit is defined in this embodiment of the invention. If there are multiple control rules, then each fuzzy variable is selected and output by the gating unit. There are 1 control rule weights, which represent the probability that each fuzzy variable will be selected by each rule. The expression for the selection function is:
[0104]
[0105]
[0106]
[0107] Final choice Membership values greater than the threshold k are input into the fuzzy neural network module for fuzzy logic operations. Values .
[0108] Specifically, the fuzzy weights are obtained by performing fuzzy logic operations on each membership value, as expressed in the following expression:
[0109]
[0110] in, Indicates the weight of the control rules. This represents the probability that a fuzzy variable is selected by each rule. This represents the membership value of the first contrastive feature vector. This represents the membership value of the second contrastive feature vector. This represents the membership value of the third contrastive eigenvector. This represents the membership value of the fourth contrastive eigenvector.
[0111] In this embodiment of the invention, the fuzzy neural network module mainly consists of a deconvolution submodule and a fully connected submodule. The deconvolution submodule includes a one-dimensional convolutional layer, a ReLU activation layer, and a max-pooling layer. The fully connected submodule includes a linear layer and a ReLU activation layer. The feedforward propagation formula of the fuzzy neural network module is:
[0112]
[0113] The activation function of the ReLU activation layer is defined as follows: The input control rule weights are processed using one-dimensional convolutional layers; after convolution, they are expanded to a higher dimension by the one-dimensional convolutional kernel, and then flattened back to the initial dimension in a fully connected submodule. Indicates the first Each node in the layer is connected to the first All nodes of the layer, parameters Represents connection weights. Represents the offset of the node.
[0114] Specifically, the weights output by the deep fuzzy neural network module The normalized weights are calculated using a normalization algorithm. The normalization formula is as follows:
[0115]
[0116] This invention defines the total output of a deep fuzzy neural network model. The weighted sum of the outputs of all control rules is converted into a numerical output, allowing the model to output the detection results. The calculation expression is:
[0117]
[0118] in, Indicates the test results. Indicates the normalized weights. Representing fuzzy variables, This represents the total number of fuzzy variables.
[0119] Specifically, training a deep fuzzy neural network model includes:
[0120] The loss value is calculated using partial detection results and mean square error output from the fuzzy deep network module:
[0121]
[0122] in, Indicates the loss value. Indicates the number of samples. This represents the true value of the t-th sample. This represents the predicted value of the t-th sample;
[0123] The backpropagation gradient is calculated based on the loss value to update the parameters of the deep fuzzy neural network model layer by layer. The calculation expression is as follows:
[0124]
[0125] in, Represents the gradient value. Indicates the first The first in the layer One parameter, Indicates the first The layer is affected by the first The number of nodes affected by a node in a layer.
[0126] In this embodiment of the invention, the training process of the deep fuzzy neural network model is as follows:
[0127] Image, text, and audio data obtained from short video ad datasets are input into a deep fuzzy neural network model.
[0128] First, the image encoder, text encoder, and audio encoder in the contrastive learning network module encode the image data, text data, and audio data respectively to obtain image vectors, text vectors, and audio vectors. Then, the image vectors are compared and learned one by one with the text vectors, and every two image vectors, every two text vectors, and every two audio vectors are compared and learned to obtain a variety of contrastive feature vectors.
[0129] Then, the membership degree of each comparative feature vector is calculated by the membership degree function in the membership degree network module, and the membership degree value is used as a fuzzy variable.
[0130] Next, the control rules are generated by controlling 0 or 1 through the transmission process between the gated network module, the membership network module, and the fuzzy neural network module;
[0131] Then, the fuzzy variables are defuzzified by the deconvolution submodule and the fully connected submodule in the fuzzy neural network module to obtain the weighted sum of the outputs under all control rules. The weighted sum is then converted into a numerical output to output the detection results of the short video advertising data.
[0132] Finally, the loss value is calculated based on the detection results and mean squared error to calculate the backpropagation gradient, and then the parameters of the deep fuzzy neural network model are updated layer by layer to obtain the illegal short video advertisement detection model.
[0133] Specifically, step 3 includes:
[0134] Input the image data, text data, and audio data from the target short video ad data into the illegal short video ad detection model;
[0135] The image encoder, text encoder, and audio encoder in the contrastive learning module encode the image data, text data, and audio data in the target short video advertisement data respectively, to obtain image vectors, text vectors, and audio vectors. The image vectors are compared and learned one by one with the text vectors, and every two image vectors, every two text vectors, and every two audio vectors are compared and learned to obtain a variety of contrastive feature vectors.
[0136] Each contrast feature vector is input into the membership network module for synchronous calculation, and the membership value corresponding to each contrast feature vector is used as a fuzzy variable.
[0137] Each fuzzy variable is directly input into the fuzzy neural network module for defuzzification, resulting in a weighted sum of outputs under all control rules. This weighted sum is then converted into a numerical output, which is used to output the detection results of the target short video ad data. This enables the detection of illegal ads, and the detection results of the target short video ad data are divided into illegal short video ads and compliant short video ads.
[0138] This invention is based on a short video ad dataset containing multiple compliant and multiple non-compliant short video ad data. Image, text, and audio data are obtained and input into a constructed deep fuzzy neural network model. The deep fuzzy neural network model is trained to obtain a non-compliant short video ad detection model. Target short video ad data is then input into the non-compliant short video ad detection model for detection, yielding detection results. The deep fuzzy neural network model includes a contrastive learning network module, a membership network module, a gating network module, and a fuzzy neural network module. Compared with existing technologies, this invention extracts feature data from three modalities (image, text, and audio) in short videos through contrastive learning, better capturing the differences between features of different modalities and converting them into contrastive feature vectors that can be used for classification. Then, the membership network module is used for feature fuzzification. During the training process, the gating network module forms control rules, enabling the fuzzy neural network module to output detection results based on these control rules. This improves the detection efficiency and accuracy of short video ad data from the perspectives of image, text, and audio modalities.
[0139] Corresponding to the multimodal illegal short video ad detection method described in the above embodiments, such as Figure 3 As shown, this embodiment of the invention also provides a multimodal illegal short video advertisement detection device 100, which includes:
[0140] The acquisition module 101 is used to obtain image data, text data and audio data based on the acquired short video advertising dataset. The short video advertising dataset includes multiple compliant short video advertising data and multiple non-compliant short video advertising data.
[0141] Training module 102 is used to input image data, text data and audio data into the constructed deep fuzzy neural network model, train the deep fuzzy neural network model, and obtain the illegal short video advertisement detection model.
[0142] The detection module 103 is used to input the image data, text data and audio data in the target short video advertisement data into the illegal short video advertisement detection model for detection and to obtain the detection results;
[0143] The deep fuzzy neural network model includes a contrastive learning network module, a membership network module, a gating network module, and a fuzzy neural network module;
[0144] The contrastive learning network module is used to perform contrastive learning on the input data in order to output contrastive feature vectors;
[0145] The membership network module is used to calculate the membership degree of the contrastive feature vectors output by the contrastive learning network module in order to output fuzzy variables;
[0146] The gating network module is only used to generate control rules during the training process of the deep fuzzy neural network model;
[0147] During training, the fuzzy neural network module is used to defuzzify fuzzy variables according to the fuzzy weights in the control rules to output detection results. During application, it is used to defuzzify the fuzzy variables output by the member network module to output detection results.
[0148] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.
[0149] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0150] This invention also provides a terminal device, such as... Figure 4 As shown, the terminal device D10 of this embodiment includes: at least one processor D100 ( Figure 4 The diagram shows only one processor, a memory D101, and a computer program D102 stored in the memory D101 and executable on the at least one processor D100. When the processor D100 executes the computer program D102, it implements the above-described multimodal illegal short video advertisement detection method.
[0151] The terminal device D10 can be a desktop computer, laptop, handheld computer, server, server cluster, or cloud server, etc. This terminal device may include, but is not limited to, a processor D100 and a memory D101. Those skilled in the art will understand that... Figure 4This is merely an example of terminal device D10 and does not constitute a limitation on terminal device D10. It may include more or fewer components than shown in the figure, or combine certain components, or different components, such as input / output devices, network access devices, etc.
[0152] The processor D100 can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.
[0153] In some embodiments, the memory D101 may be an internal storage unit of the terminal device D10, such as a hard disk or memory of the terminal device D10. In other embodiments, the memory D101 may be an external storage device of the terminal device D10, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the terminal device D10. Furthermore, the memory D101 may include both internal and external storage units of the terminal device D10. The memory D101 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory D101 can also be used to temporarily store data that has been output or will be output.
[0154] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.
[0155] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0156] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements a method for detecting multimodal illegal short video advertisements.
[0157] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a building device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks.
[0158] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A method for detecting multimodal illegal short video advertisements, characterized in that, include: Step 1: Based on the acquired short video ad dataset, obtain image data, text data, and audio data. The short video ad dataset includes multiple compliant short video ad data and multiple non-compliant short video ad data. Step 2: Input the image data, text data, and audio data into the constructed deep fuzzy neural network model, train the deep fuzzy neural network model, and obtain the illegal short video advertisement detection model; Step 3: Input the image data, text data, and audio data from the target short video ad data into the illegal short video ad detection model for detection, and obtain the detection results; The deep fuzzy neural network model includes a contrastive learning network module, a membership network module, a gating network module, and a fuzzy neural network module; The contrastive learning network module is used to perform contrastive learning on the input data to output a contrastive feature vector; The membership network module is used to calculate the membership degree of the contrastive feature vector output by the contrastive learning network module in order to output fuzzy variables; The gated network module is only used to generate control rules during the training process of the deep fuzzy neural network model; During training, the fuzzy neural network module is used to defuzzify the fuzzy variables according to the fuzzy weights in the control rules to output detection results. During application, it is used to defuzzify the fuzzy variables output by the member network module to output detection results.
2. The multimodal illegal short video advertisement detection method according to claim 1, characterized in that, Based on the acquired short video ad dataset, image data, text data, and audio data were obtained, including: The short video ad dataset is preprocessed to extract multiple video frames as image data; The short video ad dataset is segmented based on each video frame to obtain multiple video segments; Extract the audio and text data corresponding to each video segment.
3. The multimodal illegal short video advertisement detection method according to claim 2, characterized in that, The contrastive learning network module includes: The image encoder is used to encode the image data to obtain multiple image vectors; The text encoder is used to encode the text data to obtain multiple text vectors; The audio encoder is used to encode the audio data to obtain multiple audio vectors; The first vector space is used to learn by comparing each image vector with each text vector to obtain the first contrast feature vector; The second vector space is used to learn by comparing every two image vectors to obtain the second contrast feature vector. The third vector space is used to compare and learn every two text vectors to obtain the third contrast feature vector. The fourth vector space is used to learn by comparing every two audio vectors to obtain the fourth contrast feature vector.
4. The multimodal illegal short video advertisement detection method according to claim 3, characterized in that, The membership network module is constructed based on the membership function, the expression of which is: in, Indicates the membership degree value. This represents the input contrast feature vector. Represents the weight matrix. This represents the bias vector.
5. The multimodal illegal short video advertisement detection method according to claim 4, characterized in that, The fuzzy weights are obtained by performing fuzzy logic operations on each membership value, and the expression is as follows: in, Indicates the weight of the control rules. This represents the probability that a fuzzy variable is selected by each rule. This represents the membership value of the first contrastive feature vector. This represents the membership value of the second contrastive feature vector. This represents the membership value of the third contrastive eigenvector. This represents the membership value of the fourth contrastive eigenvector.
6. The multimodal illegal short video advertisement detection method according to claim 5, characterized in that, The calculation expression for the detection result is as follows: in, Indicates the test results. Indicates the normalized weights. represents a fuzzy variable, and represents the total number of fuzzy variables.
7. The multimodal illegal short video advertisement detection method according to claim 6, characterized in that, Training the deep fuzzy neural network model includes: The loss value is calculated using the partial detection results and mean square error output by the fuzzy deep network module: in, Indicates the loss value. Indicates the number of samples. This represents the true value of the t-th sample. This represents the predicted value of the t-th sample; Based on the loss value, the backpropagation gradient is calculated to update the parameters of the deep fuzzy neural network model layer by layer. The calculation expression is as follows: in, Represents the gradient value. Indicates the first The first in the layer One parameter, Indicates the first The layer is affected by the first The number of nodes affected by a node in a layer.
8. A multimodal illegal short video advertisement detection device, characterized in that, include: The acquisition module is used to obtain image data, text data, and audio data based on the acquired short video advertisement dataset, wherein the short video advertisement dataset includes multiple compliant short video advertisement data and multiple non-compliant short video advertisement data; The training module is used to input the image data, the text data, and the audio data into a deep fuzzy neural network model, train the deep fuzzy neural network model, and obtain an illegal short video advertisement detection model. The detection module is used to input the image data, text data, and audio data from the target short video advertisement data into the illegal short video advertisement detection model for detection, and obtain the detection results; The deep fuzzy neural network model includes a contrastive learning network module, a membership network module, a gating network module, and a fuzzy neural network module; The contrastive learning network module is used to perform contrastive learning on the input data to output a contrastive feature vector; The membership network module is used to calculate the membership degree of the contrastive feature vector output by the contrastive learning network module in order to output fuzzy variables; The gated network module is only used to generate control rules during the training process of the deep fuzzy neural network model; During training, the fuzzy neural network module is used to defuzzify the fuzzy variables according to the fuzzy weights in the control rules to output detection results. During application, it is used to defuzzify the fuzzy variables output by the member network module to output detection results.
9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the multimodal illegal short video advertisement detection method as described in any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the multimodal illegal short video advertisement detection method as described in any one of claims 1 to 7.