Multi-mode video information monitoring method and device, computer equipment and medium

Abstract

The invention provides a multi-modal video information monitoring method and device, computer equipment and a medium, and belongs to the field of information recognizing.The method comprises the steps that a multi-modal sensitive information recognition model is constructed; obtaining a target video and historical sensitive information of a user corresponding to the target video; determining an image type of the sensitive information, and extracting a first image frame of the target video according to the image type; extracting the text information and the audio information of the target video, and adjusting the modal weight in the SE through the historical sensitive information; and inputting the text information, the first image frame and the audio information into a multi-mode sensitive information identification model to obtain an identification result of the sensitive information. According to the method, firstly, image frames related to sensitive information are extracted, invalid information in a target video is reduced, secondly, modal weights are adjusted through historical sensitive information records, the accuracy of sensitive information recognition is improved, then features of various modal information are fused, and recognition of multi-modal sensitive information is further improved.

Description

Technical Field

[0001] The present invention belongs to the field of information identification, and specifically relates to a multimodal video information monitoring method, device, computer equipment and medium. Background Art

[0002] With the development of various online videos such as short videos, web dramas or live broadcasts, in addition to enriching people's spiritual activities, the sensitive information that may exist also brings related risks.

[0003] Generally speaking, sensitive information involves many data forms, such as confidential information and illegal information involving political topics, pornography, violence, or sensitive privacy. These sensitive information generally involve three modes: text, image, and audio. In addition, sensitive information also has spatiotemporal dynamics and scene-dependent characteristics. In other words, sensitive information usually has the characteristics of massive multimodality, spatiotemporal dynamics, and scene-dependent.

[0004] Traditional methods tend to focus only on data of a single modality, for example, only text or just images. However, with the evolution of multimedia technology, traditional single-modal methods are unable to cope with complex content such as online videos. One obvious problem is that single-modal methods may miss key information in other modalities, resulting in inaccurate or incomplete recognition. In addition, single-modal methods also have difficulty in handling complementary and conflicting information between multiple modalities, and it is difficult to capture such complex relationships. Traditional sensitive information identification methods usually rely on manual review or rule-based screening. However, these methods are not only time-consuming and labor-intensive, but also prone to errors or missed detections.

[0005] In recent years, deep learning technology has demonstrated its great potential in many fields, from image recognition, speech processing to natural language understanding. The cross-modal sensitive information recognition method is a method that uses deep learning technology to identify sensitive information of different types of data (such as text, images, audio, video, etc.). However, the inventors found that the relevant technologies for sensitive information recognition generally identify and predict data of different modalities separately. For example, the data of the text modality is sent to the text detection interface to obtain the sensitive information recognition result; the data of the image modality is sent to the corresponding picture detection interface to obtain the sensitive information recognition result. This recognition process is often inefficient, and the correlation between multiple modal data cannot be reflected, resulting in low accuracy in the recognition of sensitive information. Summary of the invention

[0006] In order to solve the problem of low accuracy in identifying multimodal sensitive information, the present invention provides a multimodal video information monitoring method, apparatus, computer equipment and medium.

[0007] In order to achieve the above object, the present invention provides the following technical solutions: First, a multimodal video information monitoring method is provided, the method comprising: The word embedding model Word2Vec, the convolutional neural network CNN and the recurrent neural network RNN ​​are fully connected and fused, and the attention module SE is embedded to obtain the SE enhanced fusion model SE-WRC Model; the SE-WRC Model is trained with the pre-acquired training samples to obtain a multimodal sensitive information recognition model; Obtain the target video and the historical sensitive information of the user corresponding to the target video; Extracting text information and audio information of the target video; and extracting the first image frame of the target video according to the type of preset target sensitive information; Adjusting the modality weight in SE through the historical sensitive information; The text information, the first image frame and the audio information are input into the multimodal sensitive information recognition model, and the feature vectors of the text information, the first image frame and the audio information are extracted respectively through Word2Vec, CNN and RNN, and the feature vectors of the text information, image information and audio information are mapped to the same dimension through SE, and the feature vectors are weightedly spliced ​​according to the modal weights to obtain fused features; finally, the fused features are identified by the classifier to obtain the recognition result of the preset target sensitive information in the target video.

[0008] Optionally, extracting the first image frame of the target video according to the type of preset target sensitive information includes: Determining the corresponding image type according to the image scene of the preset target sensitive information; Image frames are extracted from the target video according to the image type to obtain a first image frame of the same image type as the preset target sensitive information.

[0009] Optionally, extracting text information and audio information of the target video includes: Extracting a second image frame containing text information from the target video; extracting the text information from the second image frame by optical character recognition (OCR); Use the audio processing library to separate the audio stream from the target video.

[0010] Optionally, adjusting the modality weight in the SE by using the historical sensitive information includes: Performing importance evaluation on feature vectors of different modes according to the historical sensitive information to obtain importance evaluation results; A learnable weight vector is introduced into SE, and the learnable weight vector is adjusted according to the importance evaluation result to obtain the modal weight.

[0011] Optionally, importance evaluation is performed on feature vectors of different modes according to the historical sensitive information, and the importance evaluation results obtained include: Determine the number of occurrences of sensitive information of different modalities in the historical sensitive information, and determine the first importance of features of different modalities according to the number of occurrences; Taking the historical sensitive information as input, predicting user behavior by using a random forest method, and determining the second importance of different modal features according to feature weights of the random forest method; An importance evaluation result is determined according to the first importance and the second importance.

[0012] Optionally, performing weighted concatenation on the feature vectors according to the modality weights to obtain fusion features includes: The correlation of the feature vectors corresponding to different modal information is calculated through the query vector of SE to obtain the correlation weight; Performing feature fusion on the feature vectors corresponding to the different modal information according to the correlation weights to obtain comprehensive features corresponding to the different modal information; The comprehensive features corresponding to different modal information are spliced ​​according to the modal weights to obtain fused features.

[0013] Optionally, the training samples include sample images, sample texts and sample audios with sensitive information, and sensitive information corresponding to the sample images, sample texts and sample audios; and the multimodal sensitive information recognition model is obtained by training the SE-WRCModel using the pre-acquired training samples, including: Input the sample image, sample text and sample audio into the SE-WRC Model to obtain a recognition result; The SE-WRC Model is trained with the goal of minimizing the difference between the recognition result and the sensitive information to obtain a multimodal sensitive information recognition model.

[0014] Secondly, a multi-modal video information monitoring device is also provided, the device comprising: A construction module is used to fuse the word embedding model Word2Vec, the convolutional neural network CNN and the recurrent neural network RNN ​​in a fully connected layer and embed the attention module SE to obtain the SE enhanced fusion model SE-WRC Model; the SE-WRC Model is trained with the pre-acquired training samples to obtain a multimodal sensitive information recognition model; An acquisition module is used to acquire the target video and historical sensitive information of the user corresponding to the target video; An extraction module, used to extract text information and audio information of a target video; and extract a first image frame of the target video according to a type of preset target sensitive information; An adjustment module, used to adjust the modality weight in SE according to the historical sensitive information; The recognition module is used to input the text information, the first image frame and the audio information into the multimodal sensitive information recognition model, extract the feature vectors of the text information, the first image frame and the audio information through Word2Vec, CNN and RNN respectively, map the feature vectors of the text information, image information and audio information to the same dimension through SE, and weightedly splice the feature vectors according to the modal weights to obtain fused features; finally, the fused features are recognized by the classifier to obtain the recognition result of the preset target sensitive information in the target video.

[0015] In addition, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the multimodal video information monitoring method described above is implemented.

[0016] Finally, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned multimodal video information monitoring method when executing the program.

[0017] The multi-modal video information monitoring method provided by the present invention has the following beneficial effects: Firstly, by analyzing the sensitive information, the image frames related to the target video and the sensitive information are extracted, and the preliminary screening of the target video is realized, which reduces the invalid information in the target video and is conducive to improving the efficiency and accuracy of sensitive information identification. Secondly, the modal weights of different modal information are adjusted according to the historical sensitive information records of the user corresponding to the target video. In this way, the user's habit of using sensitive information is determined according to the user's historical behavior, thereby improving the accuracy of sensitive information identification. Then, the information features of different modalities are extracted respectively through the multimodal sensitive information identification model, and the features of multiple modal information are fused. In this way, the information of different modalities is semantically associated through the feature fusion of multi-module information, which strengthens the correlation between different modal information, improves the semantic analysis ability of multimodal information, and further improves the accuracy of multimodal sensitive information identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiment of the present invention and its design scheme, the following briefly introduces the drawings required for this embodiment. The drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 The present invention is a flowchart of a multimodal video information monitoring method provided according to an exemplary embodiment.

[0020] Figure 2 A schematic diagram of a CNN model architecture provided by the present invention according to an exemplary embodiment.

[0021] Figure 3 A training effect comparison diagram provided by the present invention according to an exemplary embodiment.

[0022] Figure 4 The present invention is a block diagram of a multimodal video information monitoring device according to an exemplary embodiment. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the technical solution of the present invention and implement it, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the scope of protection of the present invention.

[0024] The technical solutions provided by various embodiments of the present invention are described in detail below in conjunction with the accompanying drawings.

[0025] First, the present invention provides a multi-modal video information monitoring method, specifically as follows Figure 1 As shown, the following steps are included: S101. Construct a multimodal sensitive information recognition model.

[0026] Specifically, the word embedding model Word2Vec, convolutional neural network CNN and recurrent neural network RNN ​​are fully connected and fused, and embedded in the attention module SE to obtain the SE enhanced fusion model SE-WRC Model.

[0027] For example, CBOW (Continuous Bag-of-Words) is used as the architecture of the word embedding model Word2Vec, and an input layer and a hidden layer are constructed respectively. The input layer is used to receive the word vectors in the context; the hidden layer is used to perform weighted averaging on the word vectors of the input layer, and then determine the product of the weight matrix and the input vector through a linear transformation to obtain the representation of the hidden layer. The weight matrix here can be regarded as a word embedding matrix, in which each row represents the embedding vector of a word; the output layer: uses the softmax function to calculate the probability of each word in the vocabulary being the target word.

[0028] Construct a CNN model, such as Figure 2As shown, it includes an input layer, a convolution layer, a pooling layer and a fully connected layer; the input layer includes 3 color channels and adjusts the input size to 32x32x3; the convolution layer includes convolution layer 1 and convolution layer 2, the convolution layer 1 includes 6 convolution kernels of 5x5 size and activation function ReLU, and the output feature map size is 28x28x6; the convolution layer 2 includes 16 convolution kernels of 5x5 size and activation function ReLU, and the output feature map size is 8x8x16; the pooling layer includes pooling layer 1 and pooling layer 2, the pooling layer 1 is maximum pooling, the pooling window size is 2x2, the step size is 2, and the output feature map size is 12x12x6; the pooling layer 2 is maximum pooling, the pooling window size is 2x2, the step size is 2, and the output feature map size is 4x4x16; finally, a fully connected layer is constructed, and the number of input neurons and the number of output neurons are set.

[0029] Build an RNN model, including an input layer and an RNN layer; the output layer is used to receive the audio feature sequence as input, and the feature vector of each time step (such as MFCC features) will be used as an input to the RNN; use GRU units to build the RNN layer, which can capture the temporal dependencies in the audio signal. And set the number of units, the number of layers, and the sigmoid activation function.

[0030] In the present invention, firstly, the input layer and hidden layer of the above-mentioned CBOW, the input layer, convolution layer, pooling layer and fully connected layer of CNN, and the input layer and RNN layer of RNN are constructed. Secondly, the hidden layer of CBOW, the fully connected layer of CNN and the RNN layer of RNN are further fused with the features of different modalities through the average pooling layer of the embedded attention module SE, and the fused features of different modalities are weightedly connected. Then, a classifier is constructed for feature recognition, and the recognition result is used as output to obtain the SE enhanced fusion model SE-WRC Model.

[0031] After obtaining the SE-WRC Model, the model needs to be trained before it can be put into use.

[0032] Specifically, the SE-WRC Model can be trained with pre-acquired training samples to obtain a multimodal sensitive information recognition model. The training samples include sample images, sample texts, and sample audios with sensitive information, as well as sensitive information corresponding to the sample images, sample texts, and sample audios. For example, the sample image can be a photo or picture information involving confidential information or illegal information, the sample text can be text information involving confidential information or illegal information, and the sample audio can be voice information involving confidential information or illegal information, and the corresponding sensitive information is the confidential information or illegal information in the above-mentioned photo, picture, text, or voice content.

[0033] The sample image, sample text and sample audio are input into the SE-WRC Model to obtain a recognition result; the SE-WRC Model is trained with the goal of minimizing the difference between the recognition result and the true result to obtain a multimodal sensitive information recognition model.

[0034] S102: Obtain the target video and historical sensitive information of the user corresponding to the target video.

[0035] In this step, a database of sensitive information can be constructed to store historical records of sensitive information and corresponding user information, so as to facilitate querying historical sensitive information at any time.

[0036] For example, the target video is a video posted by user A, and user A has posted five illegal pictures. Then, the history of user A posting image-type sensitive information can be obtained through a sensitive information database query, that is, user A has five image-type historical sensitive information.

[0037] S103, extracting text information and audio information of the target video; and extracting the first image frame of the target video according to the type of preset target sensitive information.

[0038] In one embodiment, since the identification of sensitive information of the image type is the most complex, the target video may first be preliminarily screened according to the preset target sensitive information of the image type. Specifically, the image scene of the preset target sensitive information to be screened may be first determined, and the corresponding image type may be determined according to the image scene of the preset target sensitive information; image frames may be extracted from the target video according to the image type to obtain a first image frame of the same image type as the preset target sensitive information.

[0039] For example, if it is necessary to automatically detect and process images containing person A (person A is used as an example of sensitive information, i.e., preset target sensitive information) in a surveillance video stream, then it can be determined that the image scenes that may contain the sensitive information are all scenes where people appear, i.e., the image types containing people. Therefore, the goal of this step is to identify the image types containing people in the surveillance video stream, and then extract these image frames containing people from the surveillance video stream.

[0040] In the above video stream processing process, the video needs to be decoded. A video processing library (such as OpenCV or FFmpeg) is used to decode the video file into a series of image frames, and the image type is continuously tracked. When the "containing person" image type is detected, the image frame is extracted as the first image frame, or multiple frames are extracted as needed to cover the complete time period when the face appears.

[0041] In addition, a timestamp vector may be added to the extracted first image frame to facilitate semantic association with feature vectors corresponding to other modal information.

[0042] In another embodiment, in order to further monitor the multimodal information of the target video, it is also necessary to extract text information and audio information from the target video.

[0043] For example, first, a second image frame containing text information in the target video may be extracted; and the text information of the second image frame may be extracted by optical character recognition (OCR).

[0044] For example, a suitable OCR library can be selected, such as Tesseract, Google Cloud Vision API, or Microsoft Azure Computer Vision. In addition, in order to improve the accuracy of OCR, the extracted second image frame is preprocessed, including grayscale, binarization, denoising, contrast adjustment, etc. The preprocessed second image frame is sent to the OCR engine for text recognition. Finally, the recognized text is integrated in time sequence or frame sequence to form complete text information.

[0045] For the extraction of audio information, you can use an audio processing library (such as librosa, pydub, or FFmpeg's audio function) to separate the audio stream from the video file.

[0046] In addition, you can also transcribe the audio and select a speech transcription service, such as Google Speech-to-Text API, IBM Watson Speech to Text, Amazon Transcribe, etc. In order to improve the accuracy of audio extraction, you can also pre-process the audio and perform pre-processing such as noise reduction and format conversion according to the requirements of the speech transcription service. Finally, transcribe the audio: send the audio file to the selected speech transcription service for transcription and obtain the audio information in text form.

[0047] Finally, the information is integrated by combining the text information extracted by OCR and the audio information obtained by speech transcription according to timestamps or other logic.

[0048] S104: Adjust the modality weight in SE using the historical sensitive information.

[0049] In another embodiment, the importance of feature vectors of different modalities is evaluated based on the historical sensitive information to obtain an importance evaluation result; a learnable weight vector is introduced into SE, and the learnable weight vector is adjusted according to the preliminary evaluation result to obtain the modality weight.

[0050] Specifically, the number of occurrences of sensitive information of different modalities in the historical sensitive information can be determined, and the first importance of different modal features can be determined based on the number of occurrences. For example, the respective proportions of sensitive information of different modalities in the historical sensitive information can be determined based on the number of occurrences, and the proportions can be determined as the first importance; the historical sensitive information is used as input, and the user behavior is predicted through the random forest method, and the second importance of different modal features is determined based on the feature weights of the random forest method; a preliminary evaluation result is determined based on the first importance and the second importance, such as the preliminary evaluation result can be the sum or product of the first importance and the second importance.

[0051] For example, first, the global features of different modal information are extracted through the average pooling layer of SE; secondly, a dynamic weight adjustment mechanism is designed, for example, a learnable weight vector is introduced in the SE module, which is used to control the importance of different modal features; then the weight vector is initialized based on historical sensitive information. For example, if the value of the preliminary evaluation result of the image modality is the highest according to the historical sensitive information, the initial weight of the image modality can be set higher, or the weight vector can be set directly in proportion to the value of the preliminary evaluation result.

[0052] In addition, the characteristics of the preset target sensitive information can be analyzed to identify the common characteristics corresponding to the preset target sensitive information of different modalities, such as specific colors in the image modality, specific words in the text modality, or specific audio patterns in the audio modality; in the subsequent use of the model, the model can dynamically adjust the weight vector of SE according to the modal characteristics of the preset target sensitive information. For example, if a specific color appears frequently in the preset target sensitive information and the other two modalities do not have obvious common characteristics, the modal weight of the video modality can be increased. Specifically, optimization algorithms such as gradient descent can be used to update the modal weight.

[0053] S105: Input the text information, the first image frame, and the audio information into a multimodal sensitive information recognition model to obtain a recognition result of preset target sensitive information in the target video.

[0054] Specifically, the text information, the first image frame and the audio information are input into the multimodal sensitive information recognition model, and the feature vectors of the text information, the first image frame and the audio information are extracted by Word2Vec, CNN and RNN respectively. The feature vectors of the text information, the image information and the audio information are mapped to the same dimension by SE, and then the feature vectors of the text information, the image information and the audio information can be timestamp aligned, and the feature vectors are weighted spliced ​​according to the modal weight to obtain the fusion feature; finally, the fusion feature is identified by the classifier to obtain the recognition result of the preset target sensitive information in the target video. In this way, the features of multiple modal information are fused and the different modal information is semantically associated, which can effectively improve the recognition of multimodal sensitive information.

[0055] Among them, SE first captures the global information of each feature channel through a global average pooling operation, and concatenates the feature vectors of the global information corresponding to the same modal information. For example, the query vector of SE can be used to calculate the correlation of the feature vectors corresponding to each of the different modal information to obtain the correlation weight; according to the correlation weight, the feature vectors corresponding to each of the different modal information are fused to obtain the comprehensive features corresponding to the different modal information; for example, the correlation between multiple feature vectors of the image modality is determined according to the query vector, and the feature vectors with higher correlation are fused to obtain the comprehensive features of the image modality after the fusion of multiple feature vectors. Then, the comprehensive features corresponding to the different modal information are concatenated according to the modal weight to obtain the fused features. The fused features can be fused features of multimodal information integrated according to timestamps.

[0056] In addition, the present invention has conducted multiple experimental comparisons on the training effects of the multimodal sensitive information recognition model obtained by training based on the fusion model SE-WRC Model and the recognition model obtained by traditional CNN training. Figure 3 As shown, the multimodal sensitive information recognition model provided by the present invention has obviously higher accuracy and is more superior than the traditional CNN model.

[0057] By adopting the above method, firstly, by analyzing the sensitive information, the image frames related to the target video and the sensitive information are extracted, thereby realizing the preliminary screening of the target video and reducing the invalid information in the target video. Secondly, the modal weights of different modal information are adjusted according to the historical sensitive information records of the user corresponding to the target video. In this way, the user's habit of using sensitive information is determined according to the user's historical behavior, thereby improving the accuracy of sensitive information recognition. Then, the features of multiple modal information are fused through the multimodal sensitive information recognition model, thereby semantically associating different modal information and further improving the recognition of multimodal sensitive information.

[0058] Secondly, the present invention also provides a multi-modal video information monitoring device, such as Figure 4 As shown, including: The construction module 401 is used to fuse the word embedding model Word2Vec, the convolutional neural network CNN and the recurrent neural network RNN ​​in a fully connected layer, and embed the attention module SE to obtain the SE enhanced fusion model SE-WRC Model; the SE-WRC Model is trained by using the pre-acquired training samples to obtain a multimodal sensitive information recognition model.

[0059] The acquisition module 402 is used to acquire the target video and the historical sensitive information of the user corresponding to the target video.

[0060] The extraction module 403 is used to extract the text information and audio information of the target video; and extract the first image frame of the target video according to the type of preset target sensitive information.

[0061] The adjustment module 404 is used to adjust the modality weight in the SE according to the historical sensitive information.

[0062] The recognition module 405 is used to input the text information, the first image frame and the audio information into the multimodal sensitive information recognition model, extract the feature vectors of the text information, the first image frame and the audio information through Word2Vec, CNN and RNN respectively, map the feature vectors of the text information, the image information and the audio information to the same dimension through SE, and weightedly concatenate the feature vectors according to the modal weight to obtain fused features; finally, the fused features are recognized by the classifier to obtain the recognition result of the sensitive information in the target video.

[0063] By using the above-mentioned device, firstly, by analyzing the sensitive information, image frames related to the target video and sensitive information are extracted, thereby realizing preliminary screening of the target video and reducing invalid information in the target video. Secondly, the modal weights of different modal information are adjusted according to the historical sensitive information records of the user corresponding to the target video. In this way, the user's habit of using sensitive information is determined according to the user's historical behavior, thereby improving the accuracy of sensitive information recognition. Then, the features of multiple modal information are fused through a multimodal sensitive information recognition model, thereby semantically associating different modal information and further improving the recognition of multimodal sensitive information.

[0064] The present invention also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 The steps of the multimodal video information monitoring method are provided.

[0065] The present invention also provides a computer device. At the hardware level, the computer device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory. Of course, it may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The steps of the multimodal video information monitoring method are provided.

[0066] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0067] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as a combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0068] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0069] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0070] It should be noted that the above specific implementation method can enable those skilled in the art to understand the invention more comprehensively, but does not limit the invention in any way. Therefore, although the present invention has been described in detail in this specification, those skilled in the art should understand that the invention can still be modified or replaced by equivalents; and all technical solutions and improvements that do not deviate from the spirit and scope of the invention are included in the protection scope of the patent for the invention. Any figure mark in the claims should not be regarded as limiting the claims involved.

Claims

1. A multimodal video information monitoring method, characterized in that: The method comprises: The word embedding model Word2Vec, the convolutional neural network CNN and the recurrent neural network RNN ​​are fully connected and fused, and the attention module SE is embedded to obtain the SE enhanced fusion model SE-WRC Model; the SE-WRCModel is trained with the pre-acquired training samples to obtain a multimodal sensitive information recognition model; Obtain the target video and the historical sensitive information of the user corresponding to the target video; Extracting text information and audio information of the target video; and extracting the first image frame of the target video according to the type of preset target sensitive information; Adjusting the modality weight in SE through the historical sensitive information; The text information, the first image frame and the audio information are input into the multimodal sensitive information recognition model, and the feature vectors of the text information, the first image frame and the audio information are extracted respectively through Word2Vec, CNN and RNN, and the feature vectors of the text information, image information and audio information are mapped to the same dimension through SE, and the feature vectors are weightedly spliced ​​according to the modal weights to obtain fused features; finally, the fused features are identified by the classifier to obtain the recognition result of the preset target sensitive information in the target video.

2. A multimodal video information monitoring method according to claim 1, characterized in that: The step of extracting the first image frame of the target video according to the type of preset target sensitive information includes: Determining the corresponding image type according to the image scene of the preset target sensitive information; Image frames are extracted from the target video according to the image type to obtain a first image frame of the same image type as the preset target sensitive information.

3. A multimodal video information monitoring method according to claim 1, characterized in that: The text information and audio information of the target video are extracted including: Extracting a second image frame containing text information from the target video; extracting the text information from the second image frame by optical character recognition (OCR); Use the audio processing library to separate the audio stream from the target video.

4. A multimodal video information monitoring method according to claim 1, characterized in that: Adjusting the modality weight in SE by using the historical sensitive information includes: Performing importance evaluation on feature vectors of different modes according to the historical sensitive information to obtain importance evaluation results; A learnable weight vector is introduced into SE, and the learnable weight vector is adjusted according to the importance evaluation result to obtain the modal weight.

5. A multimodal video information monitoring method according to claim 4, characterized in that: The importance of feature vectors of different modes is evaluated according to the historical sensitive information, and the importance evaluation results obtained include: Determine the number of occurrences of sensitive information of different modalities in the historical sensitive information, and determine the first importance of features of different modalities according to the number of occurrences; Taking the historical sensitive information as input, predicting user behavior by using a random forest method, and determining the second importance of different modal features according to feature weights of the random forest method; An importance evaluation result is determined according to the first importance and the second importance.

6. A multimodal video information monitoring method according to claim 1, characterized in that: The weighted concatenation of the feature vectors according to the modal weights to obtain the fusion features includes: The correlation of the feature vectors corresponding to different modal information is calculated through the query vector of SE to obtain the correlation weight; Performing feature fusion on the feature vectors corresponding to the different modal information according to the correlation weights to obtain comprehensive features corresponding to the different modal information; The comprehensive features corresponding to different modal information are spliced ​​according to the modal weights to obtain fused features.

7. A multimodal video information monitoring method according to claim 1, characterized in that: The training samples include sample images, sample texts and sample audios with sensitive information, and the sensitive information corresponding to the sample images, sample texts and sample audios; The SE-WRC Model is trained by using the pre-acquired training samples to obtain a multimodal sensitive information recognition model, including: Input the sample image, sample text and sample audio into the SE-WRC Model to obtain a recognition result; The SE-WRC Model is trained with the goal of minimizing the difference between the recognition result and the sensitive information to obtain a multimodal sensitive information recognition model.

8. A multimodal video information monitoring device, characterized in that: The device comprises: A construction module is used to fuse the word embedding model Word2Vec, the convolutional neural network CNN and the recurrent neural network RNN ​​in a fully connected layer and embed the attention module SE to obtain the SE enhanced fusion model SE-WRC Model; the SE-WRC Model is trained with the pre-acquired training samples to obtain a multimodal sensitive information recognition model; An acquisition module is used to acquire the target video and historical sensitive information of the user corresponding to the target video; An extraction module, used to extract text information and audio information of a target video; and extract a first image frame of the target video according to a type of preset target sensitive information; An adjustment module, used to adjust the modality weight in SE according to the historical sensitive information; The recognition module is used to input the text information, the first image frame and the audio information into the multimodal sensitive information recognition model, extract the feature vectors of the text information, the first image frame and the audio information through Word2Vec, CNN and RNN respectively, map the feature vectors of the text information, image information and audio information to the same dimension through SE, and weightedly splice the feature vectors according to the modal weights to obtain fused features; finally, the fused features are recognized by the classifier to obtain the recognition result of the preset target sensitive information in the target video.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the program.

Images