A sensitive information identification and early warning method, system and medium based on short video

Abstract

The present invention provides a sensitive information identification and early warning method, system and medium based on short videos, which relates to the field of information identification technology, including: converting audio data into text data; extracting image features and text features, and matching and associating them to form multiple image and text feature clusters; obtaining the overlap between each image and text feature cluster, and converting it into the overlap ratio of each feature in each image and text feature cluster, reducing the weight of image features and text features with high overlap ratio in the image and text feature cluster, and performing feature fusion; sorting the image and text feature clusters from small to large according to the overlap ratio, and determining the sequence of key feature clusters; inputting the fused features into the sensitive information identification model in sequence for identification, and stopping after the sensitive information is identified. The method amplifies the hidden feature information of sensitive data and improves the accuracy of judgment and identification by performing time-series feature association on the multimodal data extracted by the short video and reducing the weight of repeated feature data.

Description

Technical Field

[0001] The present invention relates to the technical field of sensitive information identification, and in particular to a sensitive information identification and early warning method, system and medium based on short videos. Background Art

[0002] On major short video platforms, the screening of sensitive information has long relied on manual review. Although this traditional method can ensure the accuracy of the review to a certain extent, in the face of massive uploaded video content, manual review is not only time-consuming and laborious, but also prone to unstable review efficiency and quality due to human subjective factors and fatigue. With the rapid development of artificial intelligence and machine learning technology, this field is undergoing profound changes.

[0003] Currently, the industry has begun to explore and apply mature network models to automatically review sensitive information in short videos. These models are based on deep learning algorithms and can learn the underlying patterns and features in large amounts of data, thereby achieving intelligent identification of sensitive content. This technological innovation has significantly improved the efficiency of review, reduced labor costs, and enhanced the objectivity and consistency of review. By continuously optimizing the algorithm model, the system can adapt to new types of sensitive information more quickly and maintain the effectiveness of review.

[0004] However, sensitive information in short videos is often concealed and transient. They may not appear continuously throughout the entire video, but are cleverly hidden in a certain segment or even just a few frames. This characteristic poses a great challenge to the identification of sensitive information, because traditional recognition methods often rely on stable feature information, which is difficult to effectively capture in rapidly switching video frames. On the one hand, if a frame-by-frame recognition method is adopted, although all possible sensitive information can be captured in theory, short videos usually consist of tens of thousands of frames. Such a processing method will lead to a sharp increase in the amount of data, greatly slowing down the recognition speed and reducing the overall processing efficiency. On the other hand, if frame-by-frame analysis is abandoned and interval sampling or key frame extraction is adopted instead, although it can reduce the amount of calculation and increase the recognition speed, it is likely to miss sensitive content that only appears for a very short time, resulting in a significant decrease in recognition accuracy. Summary of the invention

[0005] To solve the above problems, the present invention provides a sensitive information identification and early warning method based on short videos. The method amplifies the hidden feature information of sensitive data and improves the accuracy of judgment and identification by performing time-series feature association on the multimodal data extracted from the short videos and reducing the weight of repeated feature data.

[0006] To achieve the above object, the present invention provides the following technical solutions.

[0007] The present invention provides a sensitive information identification and early warning method based on short videos, comprising the following steps:

[0008] Extract audio data and video data of short videos; convert audio data into text data, and divide the text data into multiple time windows according to timing information; convert video data frame by frame into multiple image frames, and obtain key image frames after deduplication;

[0009] Construct two feature extraction networks to extract image features of multiple key image frames and text features of multiple time windows of text data, respectively, and match and associate image features with text features under multiple time windows according to the timestamps of key image frames to form multiple image-text feature clusters;

[0010] Perform cosine similarity judgment on multiple image and text feature clusters to obtain the overlap between each image and text feature cluster; convert the overlap between image and text feature clusters into the overlap ratio of each feature in each image and text feature cluster, reduce the weight of image features and text features with high overlap ratio in the image and text feature cluster, and perform feature fusion; sort the image and text feature clusters from small to large according to the overlap ratio to determine the key feature cluster sequence;

[0011] Construct a network model; train the network model through the sensitive video training set to obtain a sensitive information recognition model; input the fused features in the graphic feature clusters into the sensitive information recognition model in sequence according to the key feature cluster sequence for recognition, until sensitive information is recognized or all graphic feature clusters are recognized, and determine the recognition result.

[0012] Preferably, the step of converting the video data frame by frame into a plurality of image frames and obtaining key image frames after deduplication comprises the following steps:

[0013] Convert the video data frame by frame into multiple image frames, normalize the sizes of the multiple image frames, and grayscale the multiple image frames to obtain multiple pre-processed image frames;

[0014] Perform discrete cosine transform calculation on the preprocessed image frame to obtain a discrete cosine transform coefficient matrix, extract the 8x8 matrix in the upper left corner of the discrete cosine transform coefficient matrix, and calculate the grayscale average value of all pixels in the matrix; traverse each pixel in the 8x8 matrix, compare the grayscale value with the average value, and when the grayscale value is greater than or equal to the average value, it is recorded as 1; if it is less than the average value, it is recorded as 0, and the comparison results are combined into a perceptual hash value of a binary integer;

[0015] Calculate the Hamming distance between the perceived hash values ​​of any two image frames, and compare it with a preset threshold. When the Hamming distance between the perceived hash values ​​of the two image frames is less than or equal to the threshold, the two image frames are duplicate image frames.

[0016] All image frames considered as duplicates are classified, and duplicates are deleted to determine the key image frames.

[0017] Preferably, the construction of two feature extraction networks to respectively extract image features of a plurality of key image frames and text features of a plurality of time window text data comprises the following steps:

[0018] Construct an image feature extraction model based on the ResNet model, and extract image feature vectors of key image frames through the image feature extraction model;

[0019] A text feature extraction model based on the Word2Vec model is constructed, and text feature vectors of text data in multiple time windows are extracted through the text feature extraction model.

[0020] Preferably, matching and associating the timestamps of the key image frames with the text features in multiple time windows to form multiple image-text feature clusters includes the following steps:

[0021] Determine the time window size and align the timestamp of the key image frame with the timestamp of the text data;

[0022] For each time window, image features of key image frames and text features of text data matching the time range of the time window are screened out;

[0023] Construct multiple image-text feature cluster sets, each of which includes multiple image features and text features in a time window.

[0024] Preferably, the performing cosine similarity judgment on a plurality of image-text feature clusters to obtain the overlap between the image-text feature clusters comprises the following steps:

[0025] Get the cosine similarity value between the two feature vectors of two image and text feature clusters:

[0026] ;

[0027] In the formula, and is the feature vector; and for and Model;

[0028] According to the preset similarity threshold and the solution of the cosine similarity value, it is determined whether there is similarity between the current two feature vectors;

[0029] Traverse to obtain all the feature numbers that satisfy the similarity between each image feature and text feature cluster, and determine the overlap degree of the current feature in the entire set of image and text feature clusters:

[0030] ;

[0031] In the formula, is the overlap of the current feature, i =1,2,3... k , k is the characteristic number; n is the number of all features that satisfy similarity; N is the number of image-text feature clusters in the image-text feature cluster set.

[0032] Preferably, converting the overlap between the image-text feature clusters into the overlap ratio of each feature in each image-text feature cluster, downgrading the image features and text features with high overlap ratio in the image-text feature cluster, and performing feature fusion, comprises the following steps:

[0033] Determine the current feature vector The overlap degree of the entire image and text feature cluster set , converted into the overlap ratio of each feature in each image and text feature cluster ;

[0034] Preset demotion threshold , for those who exceed the demotion threshold The feature vectors of image features and text features with high overlap ratios are weighted down in equal proportion:

[0035] ;

[0036] ;

[0037] In the formula, The number of features that overlap in the current image-text feature cluster;

[0038] The feature vectors of the down-weighted image features and text features are concatenated to determine the fused features.

[0039] Preferably, the sensitive information identification model is constructed by training using an LSTM network, and the fused features are input into the trained LSTM network to obtain a judgment result of whether sensitive information is generated.

[0040] The present invention also provides a short video-based sensitive information identification and early warning device, comprising:

[0041] The data processing module is used to extract the audio data and video data of the short video, convert the audio data into text data, and divide the text data into multiple time windows according to the timing information; it is also used to convert the video data frame by frame into multiple image frames, and obtain key image frames after deduplication;

[0042] A feature extraction module is used to extract image features of multiple key image frames and text features of multiple time window text data through two feature extraction networks, and match and associate the image features with the text features under multiple time windows according to the timestamps of the key image frames to form multiple image-text feature clusters;

[0043] The overlap calculation module is used to perform cosine similarity judgment on multiple image and text feature clusters to obtain the overlap between each image and text feature cluster;

[0044] The feature weight reduction and fusion module is used to convert the overlap between image and text feature clusters into the overlap ratio of each feature in each image and text feature cluster, reduce the weight of image features and text features with high overlap ratio in the image and text feature cluster, and perform feature fusion;

[0045] The overlap degree sorting module is used to sort the image and text feature clusters from small to large according to the overlap degree and determine the key feature cluster sequence;

[0046] The recognition module is used to train the network model through the sensitive video training set to obtain the sensitive information recognition model, input the fused feature sequence in sequence according to the key feature cluster sequence, and pass the sensitive information recognition model until the sensitive information is recognized or all the graphic feature clusters are recognized to determine the recognition result.

[0047] The present invention also provides a sensitive information identification and early warning system based on short videos, the system comprising:

[0048] processor;

[0049] a memory having stored thereon a computer program executable on the processor;

[0050] Among them, when the computer program is executed by the processor, the steps of the sensitive information identification and warning method based on short videos are implemented.

[0051] The present invention also provides a computer-readable storage medium, on which a data processing program is stored. When the data processing program is executed by a processor, the steps of the short video-based sensitive information identification and early warning method are implemented.

[0052] Beneficial effects of the present invention:

[0053] The present invention proposes a sensitive information identification and early warning method, system and medium based on short videos. The method can effectively extract time series feature association information and improve the representativeness of the features to be identified by performing time series feature association on multimodal data. The repeated feature information is further reduced in weight by deduplication of key image frames, and the repeated feature information is downgraded by the overlap information between graphic feature clusters, thereby amplifying the hidden feature information of sensitive data, which helps to improve the accuracy of judgment and identification. The method sorts graphic feature clusters from small to large, improves the priority of distinguishing feature information, speeds up the selection of hidden feature information, and improves the efficiency of judgment and identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is a flow chart of a sensitive information identification and early warning method based on short videos according to an embodiment of the present invention;

[0055] Figure 2 It is a model architecture of a sensitive information identification and early warning system based on short videos according to an embodiment of the present invention;

[0056] Figure 3 is a logic diagram of the image frame weight reduction steps of an embodiment of the present invention;

[0057] Figure 4 It is a logic diagram of the feature downgrading steps of an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0059] Example

[0060] Sensitive information in short videos is often concealed and instantaneous. They may not appear continuously in the entire video, but are cleverly hidden in a certain segment or even in just a few frames. This characteristic poses a great challenge to the identification of sensitive information, because traditional recognition methods often rely on stable feature information, and this type of information is difficult to effectively capture in rapidly switching video frames. On the one hand, if a frame-by-frame recognition method is adopted, although all possible sensitive information can be captured in theory, short videos are usually composed of tens of thousands of frames. Such a processing method will cause a sharp increase in the amount of data, greatly slowing down the recognition speed and reducing the overall processing efficiency. On the other hand, if frame-by-frame analysis is abandoned and interval sampling or key frame extraction is adopted instead, although it can reduce the amount of calculation and increase the recognition speed, it is likely to miss sensitive content that only appears in a very short time, resulting in a significant decrease in recognition accuracy. To this end, the present invention proposes a sensitive information recognition and early warning method based on short videos. The process is as follows: Figure 1 As shown, the following steps are included:

[0061] S1: Extract the audio data and video data of the short video and convert the audio data into text data. The audio data to text conversion can be performed using the Whisper model currently developed by OpenAI as the basis for subsequent text feature (semantic feature) extraction.

[0062] S2: Divide the text data into multiple time windows according to the time sequence information; convert the video data frame by frame into multiple image frames, and obtain key image frames after deduplication. The image frame deduplication method of the present invention uses perceptual hash values ​​for judgment, and the judgment logic diagram is as follows: Figure 3 As shown, the specific steps include:

[0063] S2.1: Convert the video data frame by frame into multiple image frames, normalize the sizes of the multiple image frames, and grayscale the multiple image frames to obtain multiple pre-processed image frames.

[0064] S2.2: Perform discrete cosine transform calculation on the preprocessed image frame to obtain the discrete cosine transform coefficient matrix, extract the 8x8 matrix in the upper left corner of the discrete cosine transform coefficient matrix, and calculate the grayscale average value of all pixels in the matrix; traverse each pixel in the 8x8 matrix, compare the grayscale value with the average value, when the grayscale value is greater than or equal to the average value, it is recorded as 1; if it is less than the average value, it is recorded as 0, and the comparison results are combined into a 64-bit binary integer perceptual hash value (PHash).

[0065] The present invention uses discrete cosine transform (DCT) for compression and denoising. Discrete cosine transform is a transformation method commonly used for image compression, which can retain the low-frequency information of the image and remove high-frequency noise. Usually, the 8x8 matrix in the upper left corner is taken from the DCT coefficient matrix, which presents the lowest frequency information in the image.

[0066] S2.3: Calculate the Hamming distance between the perceived hash values ​​of any two image frames. The smaller the Hamming distance, the more similar the two images are. The Hamming distance between the perceived hash values ​​of the two image frames is estimated and compared with a preset threshold. When the Hamming distance between the perceived hash values ​​of the two image frames is less than or equal to the threshold, the two image frames are duplicate image frames.

[0067] S2.4: Classify all image frames that are considered to be repeated, delete the repeated ones, and determine the key image frames.

[0068] S3: Construct two feature extraction networks to extract image features of multiple key image frames and text features of text data in multiple time windows respectively, and match and associate image features with text features in multiple time windows according to the timestamps of key image frames to form multiple image-text feature clusters.

[0069] In the present invention, an image feature extraction model based on the ResNet model is constructed, and image feature vectors of key image frames are extracted through the image feature extraction model; a text feature extraction model based on the Word2Vec model is constructed, and text feature vectors of text data in multiple time windows are extracted through the text feature extraction model.

[0070] Among them, the advantage of the ResNet model as an image feature extraction model lies in its unique residual block structure and deep network design. By introducing jump connections, ResNet effectively solves the gradient vanishing and gradient exploding problems in deep network training, allowing the network to be trained deeper and learn more complex image features. This design not only improves the model's feature extraction capabilities, enabling it to capture subtle changes and key information in the image, but also enhances the model's generalization performance and avoids overfitting problems.

[0071] Specifically, the ResNet model takes an initial convolutional layer as input, which uses a 7x7 convolution kernel and an appropriate step size to reduce the spatial dimension of the input image and increase the number of channels to extract the basic features of the image. Next, the model enters the main structure consisting of multiple residual blocks stacked together. Each residual block contains two or more convolutional layers, and each convolutional layer is followed by batch normalization and ReLU activation function processing. At the same time, the input signal is directly passed to the output of the block through a jump connection and added to the signal after convolution processing to solve the gradient vanishing problem in deep network training. As the residual blocks are stacked, the size of the feature map gradually decreases, while the number of channels gradually increases, thereby extracting deeper image features. Finally, the model converts the feature map into a one-dimensional feature vector through a global average pooling layer.

[0072] As a text feature extraction model, the Word2Vec model has the significant advantage of being able to effectively map high-dimensional, discrete vocabulary space into low-dimensional, continuous vector space. This conversion process not only greatly reduces the complexity of data processing, but also makes semantically similar words close to each other in the vector space, forming a rich semantic structure. Specifically, the Word2Vec model architecture mainly includes input layer, mapping layer and output layer. In the input layer, the model receives text data that has been preprocessed (such as word segmentation, stop word removal, etc.) and converts each word into a vector in the form of one-hot encoding. Then, these vectors enter the mapping layer, which is the hidden layer of the model, where the high-dimensional, discrete one-hot encoding vector is converted into a low-dimensional, continuous embedding vector through the weight matrix learned through training. This conversion process retains the semantic relationship between words, so that semantically similar words are closer in the embedding space. Finally, in the output layer, the model outputs the corresponding probability distribution according to the training task (such as the context prediction center word task of the CBOW model or the center word prediction context task of the Skip-gram model). By optimizing the objective function (such as maximizing the prediction probability), the model can continuously adjust the weight matrix to learn more accurate and effective text feature vectors.

[0073] Furthermore, the present invention associates the multimodal data with time series features, extracts time series feature association information, and improves the time series representativeness of the features to be identified. The method includes the following steps:

[0074] S3.1: Determine the time window size and align the timestamp of the key image frame with the timestamp of the text data.

[0075] S3.2: For each time window, filter out the image features of the key image frames and the text features of the text data that match the time range of the time window.

[0076] S3.3: Construct multiple sets of image-text feature clusters, each of which includes multiple image features and text features in a time window.

[0077] S4: Perform cosine similarity judgment on multiple image-text feature clusters to obtain the overlap between each image-text feature cluster; convert the overlap between image-text feature clusters into the overlap ratio of each feature in each image-text feature cluster, downgrade the image features and text features with high overlap ratio in the image-text feature cluster, and perform feature fusion; sort the image-text feature clusters from small to large according to the overlap ratio to determine the key feature cluster sequence.

[0078] The present invention reduces the duplication of feature data, thereby amplifying the hidden feature information of sensitive data. Specifically, Figure 4 As shown, the following steps are included:

[0079] S4.1: Get the cosine similarity value between the two feature vectors of two image and text feature clusters:

[0080] .

[0081] In the formula, and is the feature vector; and for and Model.

[0082] S4.2: According to the preset similarity threshold and the solution result of the cosine similarity value, it is determined whether there is similarity between the current two feature vectors.

[0083] S4.3: Traverse to obtain all the feature numbers of each image feature and text feature that satisfy similarity between the image and text feature clusters, and determine the overlap degree of the current feature in the entire image and text feature cluster set:

[0084] .

[0085] In the formula, is the overlap of the current feature, i =1,2,3... k , k is the characteristic number; n is the number of all features that satisfy similarity; N is the number of image-text feature clusters in the image-text feature cluster set.

[0086] S4.4: Determine the current feature vector The overlap degree of the entire image and text feature cluster set , converted into the overlap ratio of each feature in each image and text feature cluster .

[0087] S4.5: Preset demotion threshold , for those who exceed the demotion threshold The feature vectors of image features and text features with high overlap ratios are weighted down in equal proportion:

[0088] .

[0089] .

[0090] In the formula, It is the number of overlapping features in the current image and text feature cluster.

[0091] S4.6: Concatenate the feature vectors of the down-weighted image features and text features to determine the final feature representation.

[0092] In the process of feature optimization, the present invention makes full use of the overlap ratio of different features to deeply explore the similarity of features in each cluster. Specifically, the distribution of each feature in a specific cluster is firstly statistically analyzed in detail, and by analyzing the overlap degree between different clusters, the similarity degree of features in different clusters can be accurately captured. This process not only helps to more clearly understand the correlation and difference between features, but also provides a solid foundation for subsequent feature processing.

[0093] After determining the similarity of features in each cluster, the present invention adopts a clever weight reduction strategy, which can highlight those unique and hidden feature information by appropriately reducing the weight of features with high overlap. This weight reduction process not only avoids information redundancy caused by feature duplication, but also makes the feature description of each cluster more refined and accurate.

[0094] The feature information that has been de-weighted was deeply analyzed and integrated. By amplifying the more hidden feature information, more valuable information points were successfully mined. These feature information can not only effectively characterize the similar information between the image and text feature clusters, but also accurately reflect the similarity within the cluster. On the basis of fully grasping the similar feature information, targeted de-weighting processing was carried out. This step not only further reduced the redundancy of information, but also provided a clearer and more accurate feature description for subsequent recognition processing.

[0095] S5: Construct a network model; train the network model with a sensitive video training set to obtain a sensitive information recognition model; the sensitive information recognition model of the present invention is trained and constructed using an LSTM network, and the final feature representation is input into the trained LSTM network to obtain a judgment result on whether sensitive information is generated. LSTM can capture dependencies in the time dimension and has advantages in processing time series information.

[0096] The LSTM network architecture is mainly composed of an input layer, an LSTM layer, a fully connected layer, and an output layer. In the input layer, the model receives the fused feature sequence as the input sequence of the LSTM network. Next, the input sequence enters the LSTM layer, which is composed of multiple LSTM units. Each unit controls the flow and storage of information through a forget gate, an input gate, and an output gate, thereby effectively capturing long-term dependencies in the sequence. The LSTM layer is followed by a fully connected layer, which further processes and transforms the output of the LSTM layer to extract higher-level features. Finally, the corresponding judgment results are output at the output layer.

[0097] Furthermore, the fused feature sequence is input in sequence according to the key feature cluster sequence, and the sensitive information recognition model is used until the sensitive information is recognized to stop or all graphic feature clusters are recognized, the recognition result is determined, and an early warning is pushed. The method proposed in this embodiment can effectively extract temporal feature association information and improve the representativeness of the features to be identified by performing temporal feature association on multimodal data. Further, the repeated feature information is reduced in weight through deduplication of key image frames, and the repeated feature information is downgraded through the overlap information between graphic feature clusters, thereby amplifying the hidden feature information of sensitive data, which helps to improve the accuracy of judgment and recognition. This method sorts the graphic feature clusters from small to large, improves the priority of distinguishing feature information, speeds up the selection of hidden feature information, and improves the efficiency of judgment and recognition.

[0098] The above is a short video-based sensitive information identification and warning method provided by one embodiment of this embodiment. Based on the same idea, this embodiment also provides a corresponding short video-based sensitive information identification and warning system. For the specific definition of the short video-based sensitive information identification and warning system, please refer to the definition of the short video-based sensitive information identification and warning method above, which will not be repeated here. Each module in the above-mentioned short video-based sensitive information identification and warning system can be fully or partially implemented by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0099] The system model architecture is as follows Figure 2As shown, it includes a data processing module, a feature extraction module, an overlap calculation module, a feature weight reduction fusion module, an overlap sorting module and a recognition module.

[0100] This embodiment also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 A sensitive information identification and early warning method based on short videos is provided.

[0101] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0102] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A sensitive information identification and early warning method based on short videos, characterized in that: The following steps are involved: Extract audio data and video data of short videos; convert audio data into text data, and divide the text data into multiple time windows according to timing information; convert video data frame by frame into multiple image frames, and obtain key image frames after deduplication; Construct two feature extraction networks to extract image features of multiple key image frames and text features of multiple time windows of text data, respectively, and match and associate image features with text features under multiple time windows according to the timestamps of key image frames to form multiple image-text feature clusters; Perform cosine similarity judgment on multiple image and text feature clusters to obtain the overlap between each image and text feature cluster; convert the overlap between image and text feature clusters into the overlap ratio of each feature in each image and text feature cluster, reduce the weight of image features and text features with high overlap ratio in the image and text feature cluster, and perform feature fusion; sort the image and text feature clusters from small to large according to the overlap ratio to determine the key feature cluster sequence; Construct a network model; train the network model with a sensitive video training set to obtain a sensitive information recognition model; input the fused features in the image and text feature clusters into the sensitive information recognition model in sequence according to the key feature cluster sequence for recognition until sensitive information is recognized or all image and text feature clusters are recognized, and determine the recognition result; The method of matching and associating image features with text features in multiple time windows according to the timestamps of key image frames to form multiple image-text feature clusters includes the following steps: Determine the time window size and align the timestamp of the key image frame with the timestamp of the text data; For each time window, image features of key image frames and text features of text data matching the time range of the time window are screened out; Construct multiple image-text feature cluster sets, each of which includes multiple image features and text features in a time window; The method of performing cosine similarity judgment on multiple image-text feature clusters to obtain the overlap between the image-text feature clusters includes the following steps: Get the cosine similarity value between the two feature vectors of two image and text feature clusters: Where a and b are eigenvectors; ||a|| and ||b|| are the moduli of a and b; According to the preset similarity threshold and the solution of the cosine similarity value, it is determined whether there is similarity between the current two feature vectors; Traverse to obtain all the feature numbers that satisfy the similarity between each image feature and text feature cluster, and determine the overlap degree of the current feature in the entire set of image and text feature clusters: In the formula, e i is the overlap degree of the current feature, i=1,2,3...k, k is the number of features; n is the number of all features that meet the similarity; N is the number of image-text feature clusters in the image-text feature cluster set; The method converts the overlap between the image and text feature clusters into the overlap ratio of each feature in each image and text feature cluster, downgrades the image features and text features with high overlap ratio in the image and text feature cluster, and performs feature fusion, including the following steps: Determine the current feature vector a i The overlap degree e of the entire image and text feature cluster set i , converted to the overlap ratio of each feature in each image and text feature cluster∈ i ; A weight reduction threshold μ is preset, and the feature vectors of image features and text features with a high overlap ratio exceeding the weight reduction threshold μ are weighted proportionally: In the formula, j is the number of features that overlap in the current image-text feature cluster; The feature vectors of the down-weighted image features and text features are concatenated to obtain fused features; The sensitive information identification model is trained and constructed using an LSTM network, and the fused features are input into the trained LSTM network to obtain a judgment result on whether sensitive information is generated.

2. The sensitive information identification and early warning method based on short videos according to claim 1 is characterized in that: The step of converting the video data frame by frame into multiple image frames and obtaining key image frames after deduplication includes the following steps: Convert the video data frame by frame into multiple image frames, normalize the sizes of the multiple image frames, and grayscale the multiple image frames to obtain multiple pre-processed image frames; Perform discrete cosine transform calculation on the preprocessed image frame to obtain a discrete cosine transform coefficient matrix, extract the 8×8 matrix in the upper left corner of the discrete cosine transform coefficient matrix, and calculate the grayscale average value of all pixels in the matrix; traverse each pixel in the 8×8 matrix, compare the grayscale value with the average value, and when the grayscale value is greater than or equal to the average value, it is recorded as 1; if it is less than the average value, it is recorded as 0, and the comparison results are combined into a perceptual hash value of a binary integer; Calculate the Hamming distance between the perceived hash values ​​of any two image frames, and compare it with a preset threshold. When the Hamming distance between the perceived hash values ​​of the two image frames is less than or equal to the threshold, the two image frames are duplicate image frames. All image frames considered as duplicates are classified, and duplicates are deleted to determine the key image frames.

3. The sensitive information identification and early warning method based on short videos according to claim 1 is characterized in that: The method of constructing two feature extraction networks to respectively extract image features of multiple key image frames and text features of multiple time window text data includes the following steps: Construct an image feature extraction model based on the ResNet model, and extract image feature vectors of key image frames through the image feature extraction model; A text feature extraction model based on the Word2Vec model is constructed, and text feature vectors of text data in multiple time windows are extracted through the text feature extraction model.

4. A sensitive information identification and warning device based on short videos, characterized in that: include: A data processing module is used to extract audio data and video data of a short video, convert the audio data into text data, and divide the text data into multiple time windows according to the timing information; it is also used to convert the video data frame by frame into multiple image frames, and obtain key image frames after deduplication; A feature extraction module is used to extract image features of multiple key image frames and text features of multiple time window text data through two feature extraction networks, and match and associate the image features with the text features under multiple time windows according to the timestamps of the key image frames to form multiple image-text feature clusters; The overlap calculation module is used to perform cosine similarity judgment on multiple image and text feature clusters to obtain the overlap between each image and text feature cluster; The feature weight reduction and fusion module is used to convert the overlap between image and text feature clusters into the overlap ratio of each feature in each image and text feature cluster, reduce the weight of image features and text features with high overlap ratio in the image and text feature cluster, and perform feature fusion; The overlap degree sorting module is used to sort the image and text feature clusters from small to large according to the overlap degree and determine the key feature cluster sequence; The recognition module is used to train the network model through the sensitive video training set to obtain a sensitive information recognition model, input the fused feature sequence in sequence according to the key feature cluster sequence, and pass it through the sensitive information recognition model until the sensitive information is recognized or all the graphic feature clusters are recognized, and determine the recognition result; The method of matching and associating image features with text features in multiple time windows according to the timestamps of key image frames to form multiple image-text feature clusters includes the following steps: Determine the time window size and align the timestamp of the key image frame with the timestamp of the text data; For each time window, image features of key image frames and text features of text data matching the time range of the time window are screened out; Construct multiple image-text feature cluster sets, each of which includes multiple image features and text features in a time window; The method of performing cosine similarity judgment on multiple image-text feature clusters to obtain the overlap between the image-text feature clusters includes the following steps: Get the cosine similarity value between the two feature vectors of two image and text feature clusters: Where a and b are eigenvectors; ||a|| and ||b|| are the moduli of a and b; According to the preset similarity threshold and the solution of the cosine similarity value, it is determined whether there is similarity between the current two feature vectors; Traverse to obtain all the feature numbers that satisfy the similarity between each image feature and text feature cluster, and determine the overlap degree of the current feature in the entire set of image and text feature clusters: In the formula, e i is the overlap degree of the current feature, i=1,2,3...k, k is the number of features; n is the number of all features that meet the similarity; N is the number of image-text feature clusters in the image-text feature cluster set; The method converts the overlap between the image and text feature clusters into the overlap ratio of each feature in each image and text feature cluster, downgrades the image features and text features with high overlap ratio in the image and text feature cluster, and performs feature fusion, including the following steps: Determine the current feature vector a i The overlap degree e of the entire image and text feature cluster set i , converted to the overlap ratio of each feature in each image and text feature cluster∈ i ; A weight reduction threshold μ is preset, and the feature vectors of image features and text features with a high overlap ratio exceeding the weight reduction threshold μ are weighted proportionally: In the formula, j is the number of features that overlap in the current image-text feature cluster; The feature vectors of the down-weighted image features and text features are concatenated to obtain fused features; The sensitive information identification model is trained and constructed using an LSTM network, and the fused features are input into the trained LSTM network to obtain a judgment result on whether sensitive information is generated.

5. A sensitive information identification and early warning system based on short videos, characterized in that: The system comprises: processor; a memory having stored thereon a computer program executable on the processor; Wherein, when the computer program is executed by the processor, the steps of the short video-based sensitive information identification and warning method as described in any one of claims 1 to 4 are implemented.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a data processing program, and when the data processing program is executed by the processor, the steps of the short video-based sensitive information identification and early warning method as described in any one of claims 1 to 4 are implemented.

Images