A knowledge base construction method based on video content reading and analysis

Abstract

The present invention discloses a method for constructing a knowledge base based on video content reading and analysis, which relates to the field of multimedia intelligent information processing technology, including receiving and preprocessing video streams, using deep learning models to extract visual, auditory and expression features, and constructing a comprehensive feature vector set; based on the feature vector set, using long short-term memory networks and three-dimensional convolutional neural networks to capture the continuous changes of video scenes, and generate scene evolution timelines and key event annotations; based on visual, auditory and expression features, using variational autoencoders to generate cross-modal emotional semantic labels, and establish emotional semantic graphs; based on the preliminary knowledge graph structure, through the semantic Web framework and deep graph embedding, knowledge fusion is performed to construct a knowledge base. The present invention realizes accurate understanding and intelligent analysis of video content by integrating a multimodal deep learning framework, efficiently constructs a knowledge base with rich emotions and complex entity relationships, and significantly enhances the utilization value and application scenarios of video data.

Description

Technical Field

[0001] The present invention relates to the technical field of multimedia intelligent information processing, and in particular to a knowledge base construction method based on video content reading and analysis. Background Art

[0002] With the rapid development of information technology, video, as an important carrier of information dissemination, has become a research hotspot for automatic understanding and analysis of its content. Traditional video analysis technology focuses on the extraction of information from a single modality, such as focusing only on visual features or audio features, while ignoring the rich multimodal information interaction in video content. In recent years, the rise of deep learning technology has greatly promoted the progress in the field of video understanding, making it possible to automatically extract visual, auditory and expression features from videos; in addition, the emergence of variational autoencoders has provided new ideas for the generation of cross-modal emotional semantic labels and promoted the construction of emotional semantic graphs. However, although the existing technology has made significant progress in video content analysis, there are still many challenges in the systematic and comprehensive construction of knowledge bases.

[0003] Current video content analysis technology mainly faces the following limitations: First, most methods fail to fully integrate visual, auditory and expression features when processing videos, resulting in an incomplete understanding of video content and an inability to capture the continuous changes in video scenes and the occurrence of key events; second, existing technologies often lack effective multimodal information fusion strategies when generating emotional semantic tags, which limits the accuracy and richness of emotional semantic graphs; third, metadata, subtitles and description information in videos are usually ignored, while this information is crucial for building a complete knowledge graph; finally, existing technologies often rely on simple entity recognition and relationship extraction when constructing knowledge graphs, and fail to make full use of scene evolution timelines, key event annotations and emotional semantic graphs to construct deeper entity relationships, resulting in a lack of depth and breadth in the construction of the knowledge base. Summary of the invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a knowledge base construction method based on video content reading and analysis to solve the shortcomings of current video content analysis technology in terms of comprehensive understanding, emotional semantic accuracy and depth of knowledge graph construction.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In the first aspect, an embodiment of the present invention provides a method for constructing a knowledge base based on video content reading and analysis, which includes receiving and preprocessing a video stream, using a deep learning model to extract visual, auditory and expression features, and constructing a comprehensive feature vector set; based on the feature vector set, using a long short-term memory network and a three-dimensional convolutional neural network to capture the continuous changes of the video scene, and generate a scene evolution timeline and key event annotations; based on the visual, auditory and expression features, using a variational autoencoder to generate cross-modal emotional semantic tags, and establish an emotional semantic graph; identifying entities in metadata, subtitles and descriptions in the video stream, and constructing relationships between entities in combination with the scene evolution timeline, key event annotations and emotional semantic graphs to form a preliminary knowledge graph structure; based on the preliminary knowledge graph structure, performing knowledge fusion through a semantic Web framework and deep graph embedding to construct a knowledge base.

[0008] As a preferred solution of the method for constructing a knowledge base based on video content reading and analysis described in the present invention, the specific steps of receiving and preprocessing the video stream, extracting visual, auditory and expression features using a deep learning model, and constructing a comprehensive feature vector set are as follows:

[0009] Receive live video streams using the HLS protocol and decode them using the libavcodec library in the FFmpeg toolkit;

[0010] Perform frame size standardization, color space conversion, and illumination correction on the decoded video stream;

[0011] Use the YOLOv4 model to perform object detection on the processed video stream;

[0012] After cropping the area corresponding to the detected object, it is passed to the last convolutional layer of the deep learning framework to obtain the feature map, and the visual feature vector X is extracted from the feature map through global average pooling. v ;

[0013] Based on the processed video stream, the Spleeter library is used to divide the audio stream into two parts: accompaniment and vocals. The audio sampling rate is uniformly processed, and the audio feature X is extracted through the Mel frequency cepstrum coefficient. A ;

[0014] Use MTCNN to detect faces in video frames, and then use optical flow to track faces;

[0015] Use the pre-trained FER+ model to recognize facial expressions and generate facial feature vector X E ;

[0016] Concatenate the visual, audio and facial feature vectors to form a comprehensive feature vector X = [X v ;X A;X E ].

[0017] As a preferred solution of the knowledge base construction method based on video content reading and analysis of the present invention, the method uses a long short-term memory network and a three-dimensional convolutional neural network based on a feature vector set to capture the continuous changes of the video scene, generate a scene evolution timeline and key event annotations, and the specific steps are as follows:

[0018] Initialize the LSTM network unit, configure the parameters of hidden layer size and input dimension, and set the initial state;

[0019] The comprehensive feature vector is input frame by frame, and the cell state and hidden state are updated through the long short-term memory network recursive input gate, forget gate, candidate cell state and output gate process to form a dynamic spatiotemporal feature sequence;

[0020] Extracting continuous frame feature vectors from the dynamic spatiotemporal feature sequence to form a 3D data block;

[0021] Input the 3D data block into the 3D convolutional neural network structure to capture the spatiotemporal characteristics of the video and generate a feature map;

[0022] Map the feature map to the classification label space, generate the category probability distribution of each frame, and form the scene evolution timeline;

[0023] Set the threshold F, identify the time when the key event occurs, and mark the key event with the timestamp.

[0024] As a preferred solution of the knowledge base construction method based on video content reading and analysis described in the present invention, wherein: based on visual, auditory and expression features, a variational autoencoder loss model is used to generate cross-modal emotional semantic labels and establish an emotional semantic map, the specific steps are:

[0025] Introducing the variational autoencoder loss model Multimodal information fusion of vision, hearing and expression is expressed as:

[0026]

[0027] Among them, φ is the encoder parameter, θ is the decoder parameter, x is the input feature vector, Ε represents the expected value, z is the hidden variable, and p θ is the parameterized probability distribution, q φ is the posterior distribution, KL is the divergence value, σ is the standard deviation, N is the normal distribution, P(z) is the probability distribution p of the latent variable z, I is the covariance matrix, p θ (x|z) is the likelihood function generated by the decoder, q φ (z|x)=N(z;μ(x),σ 2(x)) is the Gaussian distribution of the encoder;

[0028] Input the comprehensive feature vector set X into the variational autoencoder loss model and output the mean μ(X) and variance σ of the latent variable z 2 (X);

[0029] Sampling a noise vector ∈ from N(0, I) to obtain the latent variable z is expressed as:

[0030] z=μ(X)+∈*σ(X);

[0031] Among them, ∈ is a random noise vector;

[0032] Input the hidden variable z into the variational autoencoder loss model, reconstruct the comprehensive feature vector set X, and calculate the reconstruction error -㏒p θ (x|z) and KL divergence (q φ (z|X)||p(z)) and minimize the total loss by gradient descent

[0033] When the variational autoencoder loss model is trained, cross-modal sentiment semantic information is extracted from the latent variable z, mapped into the sentiment semantic space, and the sentiment semantic label is generated.

[0034] By using a multi-layer perceptron to add attention mechanisms to visual, auditory, and facial features, a weighted feature vector X is generated. w The expression is:

[0035] X w =a V X V +a A X A +a E X E ;

[0036] Among them, a v , a A , a E are the attention weights for visual, auditory and expression features respectively;

[0037] Combine the generated emotional semantic labels with the scene evolution timeline to annotate key events;

[0038] Based on key event annotation, the triggered emotional reactions are analyzed, an intuitive visualization interface is designed, and an emotional semantic map is established.

[0039] As a preferred solution of the knowledge base construction method based on video content reading and analysis of the present invention, the specific steps of identifying entities of metadata, subtitles and descriptions in the video stream are as follows:

[0040] Remove stop words, punctuation marks, and numbers from metadata, subtitles, and descriptions in video streams;

[0041] Use the BERT-NER model for entity recognition and define entity types for people, places, organizations, time, and events;

[0042] After entity recognition is completed, entity linking and disambiguation are performed through the knowledge graph;

[0043] The similarity-based entity linking algorithm is used to calculate the semantic similarity between the candidate entity and the identified entity, and the most matching entity is selected for linking;

[0044] After completing entity recognition and linking, further explore the relationship between entities, such as kinship between people, and association between organizations and events;

[0045] Graph convolutional networks are used for relationship mining to capture the complex relationships between entities.

[0046] As a preferred solution of the knowledge base construction method based on video content reading and analysis described in the present invention, wherein: the entity relationship is constructed by combining the scene evolution timeline, key event annotations and emotional semantic graph to form a preliminary knowledge graph structure, and the specific steps are:

[0047] Combine key events on the scene evolution timeline with their corresponding emotional semantic labels to form event nodes with emotional information;

[0048] Based on the complex relationships between captured entities, find the associations between event nodes and sentiment information;

[0049] Construct an initial graph based on the established connections between entities and events;

[0050] Use graph convolutional networks to perform relational reasoning, expand the relationships between entities in the graph, and build a preliminary knowledge graph structure.

[0051] As a preferred solution of the knowledge base construction method based on video content reading and analysis of the present invention, the steps of searching for associations between event nodes with emotional information based on the captured complex relationships between entities are as follows:

[0052] Use named entity recognition (NER) natural language processing to identify entities such as people, places, and organizations from texts and conversations;

[0053] Through relationship extraction, determine the relationships between entities such as belonging, located in, friend, and hostility;

[0054] Apply VADER sentiment analysis tool to analyze the sentiment tendency of each event and output the sentiment polarity score;

[0055] Mark event types based on the ACE event type framework;

[0056] Match event annotations with entity recognition results to determine the role of the entity as the initiator, participant, or affected person in the event;

[0057] Based on the results of sentiment analysis, quantify the impact of the event on the entity's emotional state;

[0058] Use the Neo4j graph database to create nodes for each entity and event, and attach sentiment labels to event nodes;

[0059] After attaching sentiment labels to event nodes, an edge is added to the graph to connect the event nodes, and the relationship strength and sentiment tendency attributes of the edge are recorded.

[0060] As a preferred solution of the method for constructing a knowledge base based on video content reading and analysis of the present invention, the specific steps of constructing a knowledge base based on a preliminary knowledge graph structure, through a semantic Web framework and deep graph embedding, are as follows:

[0061] Based on the direct relevance of entities and the indirect association in the knowledge graph, a knowledge fusion function F in the semantic Web framework is constructed. fe , the expression is:

[0062]

[0063] Among them, e i is the first entity i, e in the knowledge graph j is the second entity j in the knowledge graph, T is the normalization factor, α is the weight parameter for balancing the influence of direct similarity, β is the weight parameter for balancing the influence of indirect association similarity, Sim is the measure of semantic similarity between entities, G is the number of indirectly associated entities considered, and k is the indirectly associated entity in the knowledge graph;

[0064] Map entities and relationships in the knowledge graph to a low-dimensional vector space and construct a deep graph embedding fusion matrix M fe , the expression is:

[0065]

[0066] Among them, ReLU is a nonlinear activation function, A is the adjacency matrix of the graph, and H l－1 is the entity embedding matrix of the l-1th layer, L is the number of layers of GCN, and l is the layer index of GCN;

[0067] Through the knowledge fusion function of the semantic Web framework and the deep graph embedding fusion matrix, a comprehensive knowledge representation is formed to build a knowledge base.

[0068] In a second aspect, an embodiment of the present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, any step of the method for constructing a knowledge base based on video content reading and analysis as described in the first aspect of the present invention is implemented.

[0069] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, any step of the method for constructing a knowledge base based on video content reading and analysis as described in the first aspect of the present invention is implemented.

[0070] The beneficial effects of the present invention are as follows: the present invention enhances the ability to capture continuous changes and key events in video scenes by comprehensively analyzing the visual, auditory and expression features in the video; improves the accuracy and richness of emotional semantic labels by effectively fusing multimodal information; at the same time, a more complete and detailed knowledge graph is constructed by utilizing video metadata, subtitles and description information; more complex associations can be established between entities by deeply analyzing the scene evolution timeline, key event annotations and emotional semantic graphs, thereby constructing a structured, semantically rich and highly relevant knowledge base, which is of great significance to the development of video understanding and intelligent application fields. This series of innovations not only makes up for the shortcomings of the existing technology, but also opens up new research directions and application prospects for video content analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0072] Figure 1 This is a flowchart of the knowledge base construction method based on video content reading and analysis in Example 1.

[0073] Figure 2 This is a flowchart for forming a preliminary knowledge graph structure in Example 1. DETAILED DESCRIPTION

[0074] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0075] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0076] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0077] Example 1, reference Figure 1 and Figure 2 , which is the first embodiment of the present invention, provides a method for constructing a knowledge base based on video content reading and analysis, comprising the following steps:

[0078] S1. Receive and preprocess the video stream, use the deep learning model to extract visual, auditory and expression features, and construct a comprehensive feature vector set.

[0079] Furthermore, the live video stream is received using the HLS protocol and the video stream is decoded by using the libavcodec library in the FFmpeg toolkit;

[0080] Preferably, the HLS protocol allows streaming media to provide high-quality video transmission under unstable network conditions, significantly improving the stability of video transmission; libavcodec provides a wide range of codec support and can efficiently process various video encoding formats.

[0081] Perform frame size standardization, color space conversion, and illumination correction on the decoded video stream;

[0082] It should be noted that size normalization is to adjust the size of each frame of video to W×H pixels, where W and H are predefined width and height, such as W=640, H=480. This step uses bilinear interpolation for size scaling to maintain image quality;

[0083] Color space conversion is to convert the video frame from BGR color space to RGB color space to meet the input requirements of the subsequent deep learning model. The conversion formula is:

[0084] R = BGR[:,:,2];

[0085] G = BGR[:,:,1];

[0086] B=BGR[:,:,0];

[0087] Among them, R=BGR[:,:,2] means that the blue channel in the third dimension of the original image data (index is 2, because the index in Python starts from 0) is assigned to the red channel (R), G=BGR[:,:,1] means that the green channel in the third dimension of the original image data is assigned to the green channel (G), and B=BGR[:,:,0] means that the red channel in the third dimension of the original image data is assigned to the blue channel (B);

[0088] Furthermore, the YOLOv4 model is used to perform object detection on the processed video stream;

[0089] Preferably, YOLOv4 is a real-time object detection algorithm with high accuracy and speed. For each object detected, YOLOv4 outputs its bounding box coordinates (x, y, w, h) and the probability distribution of the category to which it belongs;

[0090] After cropping the area corresponding to the detected object, it is passed to the last convolutional layer of the deep learning framework to obtain the feature map, and the visual feature vector X is obtained from the feature map using global average pooling. v ;

[0091] It should be noted that the expression of global average pooling is:

[0092]

[0093] Among them, W f and H f are the width and height of the feature map, respectively, and F(i, j) is the response value of the feature map at position (i, j);

[0094] Based on the processed video stream, the Spleeter library is used to divide the audio stream into two parts: accompaniment and vocals. The audio sampling rate is uniformly processed (for example, SR = 16000Hz to meet the requirements of subsequent audio feature extraction), and the audio feature X is extracted through the Mel frequency cepstrum coefficient A ;

[0095] It should be noted that the extraction process of Mel frequency cepstral coefficients (MFCCs) includes pre-emphasis, windowing, fast Fourier transform (FFT), Mel filter bank, discrete cosine transform (DCT) and other steps. The final MFCCs vector is recorded as X A .

[0096] Preferably, the Spleeter library is an open source audio source separation library developed and maintained by Deezer Research. Its main function is to separate different sound sources (such as vocals, drums, bass, piano and other instruments) in music or audio files and generate independent audio tracks respectively; the Spleeter library is based on a deep learning model and can effectively separate audio tracks; Mel frequency cepstral coefficients (MFCCs) can capture the spectral characteristics of audio signals and are widely used in speech recognition and music information retrieval;

[0097] Furthermore, MTCNN is used to detect faces in video frames, and then optical flow method is used for face tracking to reduce the computational burden and improve tracking accuracy;

[0098] It should be noted that MTCNN (Multi-Task Cascaded Convolutional Networks) is a deep learning model for face detection. It is a cascaded convolutional neural network proposed by Zhang Xiang and others from the Institute of Automation, Chinese Academy of Sciences. It is specially designed to solve the problems of face detection and key point location.

[0099] Use the pre-trained FER+ model to recognize facial expressions and generate facial feature vector X E ;

[0100] Preferably, FER+ (Facial Expression Recognition Plus) is an enhanced version of the facial expression recognition dataset, which is derived from the original FER2013 dataset, but adds more details and label categories; the FER+ model can recognize seven basic expressions: anger, disgust, fear, happiness, sadness, surprise and neutral; the FER+ model is usually based on a CNN architecture, and its output is the expression category and its confidence score, and the generated facial feature vector X E Contains expression categories and corresponding confidence scores;

[0101] Concatenate the visual, audio and facial feature vectors to form a comprehensive feature vector X = [X v ;X A ;X E ], where “;” here means the vertical splicing of vectors;

[0102] It should also be noted that the deep integrated preprocessing and multimodal feature extraction of video streams provides a solid foundation for subsequent video understanding, analysis and application. Through refined preprocessing steps and advanced feature extraction models, the overall performance of video processing is improved. It is suitable for various application scenarios such as real-time monitoring, intelligent analysis, video summarization, etc., and achieves a comprehensive, accurate and efficient understanding of video content.

[0103] S2. Based on the feature vector set, long short-term memory network and three-dimensional convolutional neural network are used to capture the continuous changes of video scenes and generate scene evolution timeline and key event annotations.

[0104] Furthermore, initialize the long short-term memory network unit, configure the parameters of the hidden layer size h and the input dimension d, and set the initial state (initialize the LSTM cell state C0 and hidden state H0 to zero vectors, both with a shape of 1×h, ensuring that the network's starting state is empty);

[0105] Input comprehensive feature vector X frame by frame t , through the long short-term memory network recursive input gate, forget gate, candidate cell state and output gate process, the cell state and hidden state are updated to form a dynamic spatiotemporal feature sequence ("forming a dynamic spatiotemporal feature sequence" more specifically means that after LSTM processing, the hidden state H of each time step t t , forming a dynamic spatiotemporal feature sequence, capturing the changes of video scenes over time);

[0106] It should also be noted that the input gate i t 、Forget gate t , candidate cell states Output gate o t , cell state C t and the hidden state H t , the specific expression is as follows:

[0107] i t =σ(W xi x t +W hi h t-1 +b i );

[0108] f t =σ(W xf x t +W hf h t-1 +b f );

[0109]

[0110] o t =σ(W xo x t +W ho h t-1 +b o );

[0111]

[0112] Ht =o t *tan h(C t );

[0113] Among them, * represents element multiplication, W and b are weight matrix and bias term respectively, σ and tanh represent sigmoid and hyperbolic tangent activation functions respectively;

[0114] Extracting feature vectors of consecutive frames from the dynamic spatiotemporal feature sequence to form a 3D data block (“forming a 3D data block” more specifically means extracting feature vectors of consecutive T frames from the dynamic spatiotemporal feature sequence to form a T×d 3D data block, where T is the time window size);

[0115] Input the 3D data block into the 3D convolutional neural network structure to capture the spatiotemporal characteristics of the video and generate a feature map;

[0116] Map the feature map to the classification label space, generate the category probability distribution of each frame, and form the scene evolution timeline;

[0117] Set a threshold F to identify the moment when a key event occurs. When the category probability of a frame exceeds the threshold F, the key event is annotated in combination with the timestamp, and the time and type of the event are recorded.

[0118] It should be noted that the threshold F is a pre-set numerical standard used to determine when an event in a video can be identified as a "critical event"; when processing video data and generating a category probability distribution for each frame, each frame will have one or more probability values ​​associated with it. These probability values ​​reflect the model's confidence that the frame belongs to a certain category. For example, if we are looking for an intruder in a surveillance video, the model may calculate a probability of "someone breaking in" for each frame; the threshold F acts as a judgment criterion here. If the probability of "someone breaking in" for a frame exceeds F, then the frame will be marked as possibly containing a critical event (i.e., intrusion behavior). If F is not exceeded, it is considered that no critical event has occurred, or at least not enough to trigger an alarm or further attention; for example, if the threshold F is set to 0.8, it means that only when the model's prediction probability for a certain category reaches or exceeds 80%, the critical event corresponding to the category will be considered to have occurred. This can filter out low-confidence predictions and reduce false alarms, while ensuring that high-confidence events are captured in a timely manner; the setting of threshold F is usually based on experience, the needs of specific applications, or through statistical methods such as cross-validation. Thresholds that are too high or too low may lead to false alarms or missed alarms, so it is very important to find a suitable threshold. In practical applications, this threshold may need to be fine-tuned according to different environmental conditions and expected false alarm / missing alarm rates.

[0119] It should also be noted that through the above steps, not only the continuous changes of the video scene are captured, but also a detailed scene evolution timeline is generated, and key events are accurately labeled, which provides richer information for subsequent video understanding and analysis, while also optimizing the efficiency and accuracy of applications such as real-time monitoring and intelligent analysis. This solution integrates the advantages of LSTM and 3D CNN, effectively processes the time series characteristics of video data, and has significant improvements in scene understanding compared to traditional methods.

[0120] S3. Based on visual, auditory and facial expression features, a variational autoencoder is used to generate cross-modal emotional semantic labels and establish an emotional semantic map.

[0121] Furthermore, the variational autoencoder loss model is introduced Multimodal information fusion of vision, hearing and expression is expressed as:

[0122]

[0123] p(z)=N(0,I);

[0124] Among them, φ is the encoder parameter, θ is the decoder parameter, x is the input feature vector, Ε represents the expected value, z is the hidden variable, and p θ is the parameterized probability distribution, q φ is the posterior distribution, KL is the divergence value, σ is the standard deviation, N is the normal distribution, P(z) is the probability distribution p of the latent variable z, I is the covariance matrix, p θ (x|z) is the likelihood function generated by the decoder, q φ (z|x)=N(z;μ(x),σ 2 (x)) is the Gaussian distribution of the encoder;

[0125] It should be noted that the introduction of the variational autoencoder (VAE) loss model The main purposes are as follows: VAE can learn a latent, continuous, low-dimensional space representation that can capture the intrinsic structure and statistical characteristics of the input data. For multimodal data (such as visual, auditory, and expression features), VAE can map these different modes of data into the same latent space, thereby achieving cross-modal information fusion; by encoding multimodal features into a compact latent variable representation, VAE can remove redundant information and noise and retain the most core features. This is very beneficial for subsequent tasks (such as emotion recognition, scene understanding, etc.) because it simplifies the data and reduces computational complexity while maintaining key information; VAE is not only an excellent feature encoder, but also a powerful generative model, which means that it can generate new samples, which is particularly useful for application areas such as data enhancement, virtual reality, and augmented reality. In the context of multimodal sentiment analysis, VAE can help generate synthetic emotional expressions for training more robust classifiers or for creating more realistic virtual characters; data of different modalities often carry different information. For example, visual features may capture facial expressions and body language, while auditory features can capture tone and speech content, and expression features are directly related to emotional states. Through VAE's multimodal fusion, the missing information between different modalities can be complemented, so that the final model can understand the emotional state more comprehensively and accurately; in the latent space, normal data points should be clustered together, while outliers should be far away from the group. VAE can help detect anomalies by comparing the differences between input data and decoded reconstructed data, which is particularly important for scenarios such as security monitoring and health monitoring. By learning from multimodal data, VAE can capture a wider range of patterns, which helps the model have better generalization capabilities when facing new data, especially when the amount of data is limited. This ability is particularly critical.

[0126] Input the comprehensive feature vector set X into the variational autoencoder loss model and output the mean μ(X) and variance σ of the latent variable z 2 (X);

[0127] Sampling a noise vector ∈ from N(0, I) to obtain the latent variable z is expressed as:

[0128] z=μ(X)+∈*σ(X);

[0129] Among them, ∈ is a random noise vector;

[0130] Furthermore, the latent variable z is input into the variational autoencoder loss model to reconstruct the comprehensive feature vector set X and calculate the reconstruction error - ㏒p θ (x|z) and KL divergence (q φ (z|X)||p(z)) and minimize the total loss by gradient descent

[0131] When the variational autoencoder loss model is trained, cross-modal sentiment semantic information is extracted from the latent variable z, mapped into the sentiment semantic space, and the sentiment semantic label is generated.

[0132] By using a multi-layer perceptron to add attention mechanisms to visual, auditory, and facial features, a weighted feature vector X is generated. w The expression is:

[0133] X w =a V X V +a A X A +a E X E ;

[0134] Among them, a v , a A , a E are the attention weights for visual, auditory and expression features respectively;

[0135] Preferably, the attention mechanism is added to better integrate visual, auditory and expression features;

[0136] Combine the generated emotional semantic labels with the scene evolution timeline to annotate key events;

[0137] Based on key event annotation, the triggered emotional reactions are analyzed, an intuitive visualization interface is designed, and an emotional semantic map is established.

[0138] It should be noted that by integrating cross-modal information (vision, hearing and expression) and using variational autoencoders to generate emotional semantic labels, the understanding of the emotional level of video content is significantly enhanced. At the same time, the feature fusion process is optimized by introducing the attention mechanism, enabling the system to more accurately capture and analyze emotional semantic changes in complex scenarios. In addition, combining emotional semantic labels with the scene evolution timeline provides a deeper insight into the video content, which is conducive to the development and application of intelligent analysis and decision support systems.

[0139] S4. Identify entities in metadata, subtitles, and descriptions in video streams, build relationships between entities based on scene evolution timelines, key event annotations, and sentiment semantic graphs, and form a preliminary knowledge graph structure.

[0140] Furthermore, the metadata, subtitles and descriptions in the video stream are processed to remove non-entity components such as stop words, punctuation marks and numbers;

[0141] It should be noted that stop words refer to words that appear frequently in the text but contribute little to understanding the meaning of the text, such as "the", "is", "at", "which", etc.

[0142] Use the BERT-NER model for entity recognition and define entity types for people, places, organizations, time, and events;

[0143] Preferably, BERT-NER is a named entity recognition (NER) model based on BERT, which leverages BERT’s powerful pre-training capabilities to identify entities in text. First, we will prepare a large annotated dataset containing video metadata, subtitles, and descriptions for training the BERT-NER model. Training the model involves adjusting hyperparameters such as learning rate, batch size, and number of training epochs to optimize performance;

[0144] Collect and annotate a large amount of text data containing video metadata, subtitles, and descriptions as a training set;

[0145] Use the BERT model for entity recognition training and adjust hyperparameters to optimize the performance of the Bert-based NER model;

[0146] Evaluate model performance on the validation set, use the F1 score as the main indicator, and continuously iterate and optimize the BERT-NER model;

[0147] After entity recognition is completed, entity linking and disambiguation are performed through the knowledge graph;

[0148] Adopting a similarity-based entity linking algorithm, by calculating the semantic similarity between candidate entities and the identified entities, the best matching entity is selected for linking (for example, if "Apple" appears in the text, it may be necessary to determine whether this refers to the fruit or the company. This step helps to clarify the specific meaning of the entity in the context);

[0149] After completing entity recognition and linking, further explore the relationship between entities, such as kinship between people, and association between organizations and events;

[0150] Use graph convolutional networks for relationship mining to capture complex relationships between entities;

[0151] Preferably, a graph convolutional network (GCN) is a deep learning model specially designed for processing graph data, which can learn the relationships between nodes and infer unknown relationships;

[0152] Furthermore, the key events on the scene evolution timeline are combined with their corresponding emotional semantic labels to form event nodes with emotional information;

[0153] Based on the complex relationships between captured entities, find the associations between event nodes and sentiment information;

[0154] It should be noted that based on the complex relationships between captured entities, the association between event nodes and sentiment information is found. The specific steps are as follows: using named entity recognition (NER) natural language processing methods, entities of people, places, and organizations are identified from texts and conversations; by extracting relationships, the relationships between entities such as belonging, located, friends, and enemies are determined; using the VADER sentiment analysis tool, sentiment analysis is performed on each event, and sentiment polarity scores are output; based on the ACE event type framework, event types are marked; event annotations are matched with entity recognition results to determine the roles of entities as initiators, participants, and affected persons in the event; based on sentiment analysis results, the degree of influence of events on the emotional state of entities is quantified; using the Neo4j graph database to create nodes for each entity and event, and attach sentiment labels to event nodes; after attaching sentiment labels to event nodes, an edge is added to the graph to connect event nodes, and the relationship strength and sentiment tendency attributes of the edge are recorded. Based on the established connections between entities and events, an initial graph is constructed; using graph convolutional networks for relational reasoning, the relationships between entities in the graph are expanded, and a preliminary knowledge graph structure is constructed;

[0155] It should also be noted that through the above steps, a preliminary knowledge graph structure containing entity, event and emotional semantic information in the video content is constructed, which provides strong support for video understanding and intelligent analysis. Through the combination of deep learning and graph theory, compared with the existing technology, the depth and breadth of understanding of video content are significantly improved, especially in the analysis of the emotional level and the mining of complex relationships between entities. The solution is feasible and aims to promote technological innovation and application development in the field of video intelligent analysis.

[0156] S5. Based on the preliminary knowledge graph structure, knowledge fusion is performed through the semantic Web framework and deep graph embedding to build a knowledge base.

[0157] Furthermore, based on the direct relevance of entities and the indirect association in the knowledge graph, a knowledge fusion function F in the semantic Web framework is constructed. fe , the expression is:

[0158]

[0159] Among them, e i is the first entity i, e in the knowledge graph jis the second entity j in the knowledge graph, T is the normalization factor, α is the weight parameter for balancing the influence of direct similarity, β is the weight parameter for balancing the influence of indirect association similarity, Sim is the measure of semantic similarity between entities, G is the number of indirectly associated entities considered, and k is the indirectly associated entity in the knowledge graph;

[0160] Preferably, by fusion function F fe , which can simultaneously consider the direct similarity between entities and the indirect associations established through other entities, which enables the knowledge graph to more comprehensively reflect the relationships between entities, including potential connections between entities that are not directly connected;

[0161] Map entities and relationships in the knowledge graph to a low-dimensional vector space and construct a deep graph embedding fusion matrix M fe , the expression is:

[0162]

[0163] Among them, ReLU is a nonlinear activation function; A is the adjacency matrix of the graph, H l－1 is the entity embedding matrix of the l-1th layer, L is the number of layers of GCN, and l is the layer index of GCN;

[0164] Preferably, through deep graph embedding, high-dimensional entity features are compressed into low-dimensional vector space, which not only reduces storage requirements but also speeds up the computation process, especially when processing large-scale knowledge graphs.

[0165] Through the knowledge fusion function of the semantic Web framework and the deep graph embedding fusion matrix, a comprehensive knowledge representation is formed to build a knowledge base;

[0166] It should be noted that as video streams are continuously input and processed, the knowledge graph and knowledge base need to be continuously updated to reflect the latest entity relationships and events; semantic query languages ​​such as SPARQL are used to query the knowledge base to obtain detailed information about the video content; machine learning algorithms are combined to conduct in-depth analysis of the information in the knowledge base to discover patterns, trends, and anomalies; user-friendly interfaces are designed to graphically display the entity relationships in the knowledge base for easy understanding and exploration by users;

[0167] It should also be noted that by building a dynamically updated knowledge base, the ability of video understanding and intelligent analysis is optimized, providing strong support for applications such as real-time monitoring, event warning, and content recommendation. Compared with existing technologies, this solution significantly improves the efficiency and accuracy of deep understanding of video content, especially in processing complex scenes and emotional semantic information.

[0168] This embodiment also provides a computer device, which is suitable for the case of a knowledge base construction method based on video content reading and analysis, including: a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute computer executable instructions to implement the knowledge base construction method based on video content reading and analysis proposed in the above embodiment.

[0169] The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covering the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0170] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the method for constructing a knowledge base based on video content reading and analysis as proposed in the above embodiment is implemented; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read Only Memory, referred to as EPROM), programmable read-only memory (Programmable Red-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0171] In summary, the present invention significantly improves the depth and breadth of video content understanding through sophisticated preprocessing, multimodal feature extraction, emotion semantic label generation, entity relationship mining and knowledge fusion construction; uses deep learning models to efficiently process video streams, capture visual, auditory and expression features, and combines variational autoencoders to generate emotion semantic labels to enhance emotion level analysis; mines complex relationships through entity recognition and graph convolutional networks, constructs a preliminary knowledge graph, and then fuses knowledge through the semantic Web framework and deep graph embedding technology to form a dynamically updated knowledge base, which provides strong support for applications such as real-time monitoring, intelligent analysis, and event warning, optimizes the efficiency and accuracy of video understanding and intelligent analysis, and is particularly excellent in processing complex scenes and emotion semantic information.

[0172] Example 2, referring to Table 1 and Table 2, is the second example of the present invention. To further verify the advancement of the present invention, experimental simulation data of a knowledge base construction method based on video content reading and analysis is provided.

[0173] The experiment selected three different video samples, namely education, entertainment and news videos, to cover different types of video content. The length of each video sample was 10 minutes, and a total of 30 minutes of video data was used for the experiment.

[0174] First, the HLS protocol is used to receive the video stream, and the libavcodec library in the FFmpeg toolkit is used to decode the video stream. The decoded video stream is preprocessed by size standardization (unified to 640x480 pixels), color space conversion (RGB to grayscale), and illumination correction. Then, the YOLOv4 model is used for object detection to extract the visual feature vector X v ; Separate the audio stream through the Spleeter library and extract the audio feature vector X A ; Use MTCNN and FER+ models to recognize facial expressions and generate facial feature vector X E ,Finally, these feature vectors are concatenated to form a comprehensive feature vector X.

[0175] Long short-term memory network (LSTM) and three-dimensional convolutional neural network (3D-CNN) are used to capture the continuous changes of video scenes and generate a scene evolution timeline. The threshold F is set to 0.8 to identify the moment when key events occur, and key events are annotated in combination with timestamps.

[0176] The variational autoencoder (VAE) is used to generate cross-modal sentiment semantic labels and build sentiment semantic maps. Multimodal information fusion is performed on the comprehensive feature vector set X, and sentiment semantic information is extracted through the trained VAE model and mapped to the sentiment semantic space.

[0177] The BERT-NER model is used to identify entities in metadata, subtitles, and descriptions in video streams, and multiple entity types are defined. Relationship mining is performed through a graph convolutional network (GCN) to construct a preliminary knowledge graph structure.

[0178] Based on the preliminary knowledge graph structure, knowledge fusion is performed through the semantic Web framework and deep graph embedding to build a knowledge base.

[0179] The details are shown in Table 1 and Table 2 below:

[0180] Table 1 Experimental record table

[0181]

[0182] Table 2 Experimental record table

[0183]

[0184] As can be seen from the table above, the method of the present invention has significant improvements in all indicators. In particular, the emotional semantic recognition ability and knowledge graph construction efficiency have been improved by 14% and 10% respectively, which shows that the present invention can more accurately understand and process the emotional information in the video content, and is more efficient in constructing the knowledge graph, reflecting the innovation and practicality of the method.

[0185] For example, for educational videos, the feature extraction accuracy of the method of the present invention reaches 94%, which is much higher than the 85% of the traditional method. It can more accurately extract key visual, auditory and expression features from the video, laying a solid foundation for subsequent scene understanding and emotional semantic analysis. In addition, the scene understanding accuracy is also improved from 82% to 92%. Through the application of long short-term memory network and three-dimensional convolutional neural network, it can effectively capture the continuous changes of video scenes, so as to more accurately generate scene evolution timeline and key event annotations. The emotional semantic recognition ability is improved from 75% to 89%, indicating the superiority of variational autoencoder in multimodal information fusion, which can more comprehensively understand the emotional semantics contained in the video and provide strong support for the establishment of emotional semantic map. The entity recognition rate is improved from 84% to 91%, indicating that the method of the present invention is more accurate in identifying entities in metadata, subtitles and descriptions in videos, which helps to build a more complete and accurate preliminary knowledge map structure. Finally, the efficiency of knowledge map construction is improved from 75% to 85%, indicating the high efficiency of semantic Web framework and deep graph embedding in knowledge fusion, making the knowledge base construction process faster and smarter.

[0186] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A method for constructing a knowledge base based on video content reading and analysis, characterized in that: include, Receive and preprocess the video stream, use deep learning models to extract visual, auditory and expression features, and construct a comprehensive feature vector set; Based on the feature vector set, the long short-term memory network and three-dimensional convolutional neural network are used to capture the continuous changes of the video scene, generate the scene evolution timeline and key event annotations; Based on visual, auditory and facial features, a variational autoencoder is used to generate cross-modal emotional semantic labels and establish an emotional semantic map. Identify entities in metadata, subtitles, and descriptions in video streams, build relationships between entities by combining scene evolution timelines, key event annotations, and sentiment semantic graphs, and form a preliminary knowledge graph structure; Based on the preliminary knowledge graph structure, knowledge fusion is performed through the semantic Web framework and deep graph embedding to build a knowledge base; The receiving and preprocessing of the video stream, using the deep learning model to extract visual, auditory and expression features, and constructing a comprehensive feature vector set, specifically comprises the following steps: Receive live video streams using the HLS protocol and decode them using the libavcodec library in the FFmpeg toolkit; Perform frame size standardization, color space conversion, and illumination correction on the decoded video stream; Use the YOLOv4 model to perform object detection on the processed video stream; After cropping the area corresponding to the detected object, it is passed to the last convolutional layer of the deep learning framework to obtain the feature map, and the visual feature vector X is extracted from the feature map through global average pooling. v ; Based on the processed video stream, the Spleeter library is used to divide the audio stream into two parts: accompaniment and vocals. The audio sampling rate is uniformly processed, and the audio feature X is extracted through the Mel frequency cepstrum coefficient. A ; Use MTCNN to detect faces in video frames, and then use optical flow to track faces; Use the pre-trained FER+ model to recognize facial expressions and generate facial feature vector X E ; Concatenate the visual, audio and facial feature vectors to form a comprehensive feature vector X = [X v ;X A ;X E ]; Based on visual, auditory and expression features, the variational autoencoder loss model is used to generate cross-modal emotional semantic labels and establish an emotional semantic map. The specific steps are as follows: Introducing the variational autoencoder loss model Multimodal information fusion of vision, hearing and expression is expressed as: p(z)=N(0,I); Among them, φ is the encoder parameter, θ is the decoder parameter, x is the input feature vector, Ε represents the expected value, z is the hidden variable, and p θ is the parameterized probability distribution, q φ is the posterior distribution, KL is the divergence value, σ is the standard deviation, N is the normal distribution, P(z) is the probability distribution p of the latent variable z, I is the covariance matrix, p θ (x|z) is the likelihood function generated by the decoder, q φ (z|x)=N(z;μ(x),σ 2 (x)) is the Gaussian distribution of the encoder; Input the comprehensive feature vector set X into the variational autoencoder loss model and output the mean μ(X) and variance σ of the latent variable z 2 (X); Sampling a noise vector ∈ from N(0, I) to obtain the latent variable z is expressed as: z=μ(X)+∈*σ(X); Among them, ∈ is a random noise vector; Input the hidden variable z into the variational autoencoder loss model, reconstruct the comprehensive feature vector set X, and calculate the reconstruction error -㏒p θ (x|z) and KL divergence (q φ (z|X)||p(z)) and minimize the total loss by gradient descent When the variational autoencoder loss model is trained, cross-modal sentiment semantic information is extracted from the latent variable z, mapped into the sentiment semantic space, and the sentiment semantic label is generated. By using a multi-layer perceptron to add attention mechanisms to visual, auditory, and facial features, a weighted feature vector X is generated. w The expression is: X w =a V X V +a A X A +a E X E ; Among them, a v , a A , a E are the attention weights for visual, auditory and expression features respectively; Combine the generated emotional semantic labels with the scene evolution timeline to annotate key events; Based on key event annotation, analyze the triggered emotional reactions, design an intuitive visualization interface, and establish an emotional semantic map; The specific steps of combining the scene evolution timeline, key event annotations and emotional semantic graph to build the relationship between entities and form a preliminary knowledge graph structure are as follows: Combine key events on the scene evolution timeline with their corresponding emotional semantic labels to form event nodes with emotional information; Based on the complex relationships between captured entities, find the associations between event nodes and sentiment information; Construct an initial graph based on the established connections between entities and events; Use graph convolutional networks to perform relational reasoning, expand the relationships between entities in the graph, and build a preliminary knowledge graph structure; Based on the preliminary knowledge graph structure, knowledge fusion is performed through the semantic Web framework and deep graph embedding to build a knowledge base. The specific steps are as follows: Based on the direct relevance of entities and the indirect association in the knowledge graph, a knowledge fusion function F in the semantic Web framework is constructed. fe , the expression is: Among them, e i is the first entity i, e in the knowledge graph j is the second entity j in the knowledge graph, T is the normalization factor, α is the weight parameter for balancing the influence of direct similarity, β is the weight parameter for balancing the influence of indirect association similarity, Sim is the measure of semantic similarity between entities, G is the number of indirectly associated entities considered, and k is the indirectly associated entity in the knowledge graph; Map entities and relationships in the knowledge graph to a low-dimensional vector space and construct a deep graph embedding fusion matrix M fe , the expression is: Among them, ReLU is a nonlinear activation function, A is the adjacency matrix of the graph, and H l－1 is the entity embedding matrix of the l-1th layer, L is the number of layers of GCN, and l is the layer index of GCN; Through the knowledge fusion function of the semantic Web framework and the deep graph embedding fusion matrix, a comprehensive knowledge representation is formed to build a knowledge base.

2. The method for constructing a knowledge base based on video content reading and analysis according to claim 1, characterized in that: Based on the feature vector set, the long short-term memory network and the three-dimensional convolutional neural network are used to capture the continuous changes of the video scene, generate the scene evolution timeline and key event annotations, and the specific steps are as follows: Initialize the LSTM network unit, configure the parameters of hidden layer size and input dimension, and set the initial state; The comprehensive feature vector is input frame by frame, and the cell state and hidden state are updated through the long short-term memory network recursive input gate, forget gate, candidate cell state and output gate process to form a dynamic spatiotemporal feature sequence; Extracting continuous frame feature vectors from the dynamic spatiotemporal feature sequence to form a 3D data block; Input the 3D data block into the 3D convolutional neural network structure to capture the spatiotemporal characteristics of the video and generate a feature map; Map the feature map to the classification label space, generate the category probability distribution of each frame, and form the scene evolution timeline; Set the threshold F, identify the time when the key event occurs, and mark the key event with the timestamp.

3. The method for constructing a knowledge base based on video content reading and analysis as claimed in claim 2, characterized in that: The specific steps of identifying entities of metadata, subtitles and descriptions in a video stream are as follows: Remove stop words, punctuation marks, and numbers from metadata, subtitles, and descriptions in video streams; Use the BERT-NER model for entity recognition and define entity types for people, places, organizations, time, and events; After entity recognition is completed, entity linking and disambiguation are performed through the knowledge graph; The similarity-based entity linking algorithm is used to calculate the semantic similarity between the candidate entity and the identified entity, and the most matching entity is selected for linking; After completing entity recognition and linking, further explore the relationship between entities, such as kinship between people, and association between organizations and events; Graph convolutional networks are used for relationship mining to capture the complex relationships between entities.

4. The method for constructing a knowledge base based on video content reading and analysis as claimed in claim 3, characterized in that: Based on the captured complex relationship between entities, the association between event nodes and sentiment information is found, and the specific steps are as follows: Use named entity recognition (NER) natural language processing to identify entities such as people, places, and organizations from texts and conversations; Through relationship extraction, determine the relationships between entities such as belonging, located in, friend, and hostility; Apply VADER sentiment analysis tool to analyze the sentiment tendency of each event and output the sentiment polarity score; Mark event types based on the ACE event type framework; Match event annotations with entity recognition results to determine the role of the entity as the initiator, participant, or affected person in the event; Based on the results of sentiment analysis, quantify the impact of the event on the entity's emotional state; Use the Neo4j graph database to create nodes for each entity and event, and attach sentiment labels to event nodes; After attaching sentiment labels to event nodes, an edge is added to the graph to connect the event nodes, and the relationship strength and sentiment tendency attributes of the edge are recorded.

5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method for constructing a knowledge base based on video content reading and analysis according to any one of claims 1 to 4 are implemented.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for constructing a knowledge base based on video content reading and analysis according to any one of claims 1 to 4 are implemented.

Images