Video forgery analysis method based on knowledge graph

Abstract

The invention provides a video forgery analysis method based on a knowledge graph, and relates to the technical field of video forgery analysis, and the method comprises the steps: inputting a target video; segmenting the target video into a plurality of video clips, and determining a key frame for each video clip; performing dynamic entity extraction on the key frames, determining a space-time relationship among the entities, and establishing a space-time knowledge graph of the target video based on the entities and the relationships extracted from the key frames; loading a predefined space-time contradiction detection rule base, and performing contradiction detection on the space-time knowledge graph based on the space-time contradiction detection rule base; and marking abnormal nodes or edges of the space-time knowledge graph based on a contradiction detection result, and outputting a marking result. According to the method, the space-time knowledge graph is established based on the key frame, so that the efficiency of forgery analysis on the video in the early stage is improved; and meanwhile, the interpretability and the accuracy of a video counterfeiting detection result are improved, and the problem that counterfeiting conforming to a visual rule but violating reality logic cannot be identified by a traditional method is solved.

Description

Technical Field

[0001] This invention relates to the field of video forgery analysis technology, and in particular to a video forgery analysis method based on knowledge graphs. Background Technology

[0002] With the rapid development of artificial intelligence technologies such as generative adversarial networks and deepfakes, video forgery has evolved from simple pixel manipulation to highly realistic semantic-level forgery, making it difficult to distinguish between real and fake. Combined with the current trend of short videos, this easily leads to the spread of false information, resulting in widespread negative impacts. Current video forgery analysis techniques mainly include methods based on physical phenomena analysis, physiological methods, and deep learning-based image recognition methods. Physical phenomena analysis methods primarily determine whether a video is forged by analyzing whether there are significant differences between the physical information in the video frame and the real world. Physiological methods mainly target the currently popular face forgery, analyzing facial movements, symmetry, and facial features. Deep learning-based image recognition methods mainly analyze whether a video is forged through image anomaly detection and image artifact recognition, but their interpretability is poor. All of the above methods suffer from poor interpretability and do not address the spatiotemporal inconsistencies in video content, and their detection accuracy needs further improvement. Summary of the Invention

[0003] The purpose of this invention is to provide a knowledge graph-based video forgery analysis method, which further improves the accuracy of video forgery detection by combining knowledge graph technology with an inference engine.

[0004] To achieve the above-mentioned objectives, the technical solution provided by this invention is as follows: A knowledge graph-based video forgery analysis method, the method comprising: Input the target video; The target video is divided into several video segments, and keyframes are determined for each video segment; Dynamic entity extraction is performed on keyframes, and the spatiotemporal relationships between entities are determined. A spatiotemporal knowledge graph of the target video is built based on the entities and relationships extracted from each keyframe. Load a predefined spatiotemporal contradiction detection rule library, and perform contradiction detection on the spatiotemporal knowledge graph based on the spatiotemporal contradiction detection rule library; Based on the contradiction detection results, abnormal nodes or edges in the spatiotemporal knowledge graph are marked, and the marking results are output.

[0005] Furthermore, the target video is divided into several video segments, specifically including the following operations: The background feature vectors of each frame of the target video are extracted, and the difference in background feature vectors between adjacent frames is calculated. According to the frame division rules, each frame of the target video is divided into corresponding video segments. The frame division rules are as follows: consecutive frames with a background feature vector difference value less than or equal to a preset difference value are divided into the same video segment, and consecutive frames with a background feature vector difference value greater than the preset difference value are divided into different video segments. If the difference in background feature vectors between all adjacent frames of the target video is less than or equal to a preset difference value, the target video is divided into N video segments according to the total duration, with each video segment containing the same number of frames.

[0006] Furthermore, keyframes are determined for each video segment, specifically as follows: Identify sensitive areas in each frame of a video clip; The degree of change in the sensitive area of ​​adjacent frames is calculated to obtain the rate of change of the sensitive area; The change rate of different sensitive areas in a frame is weighted and calculated to obtain a comprehensive change score. Keyframes for video segments are determined based on comprehensive change scores.

[0007] Furthermore, keyframes are determined for each video segment, specifically including: Extract audio features from each frame of a video clip; The degree of change in audio features between adjacent frames is calculated to obtain the rate of change in audio features; The rate of change and the degree of change of audio features in different sensitive areas of the frame are weighted and calculated to obtain a comprehensive change score.

[0008] Furthermore, when identifying sensitive areas in each frame of a video clip, the frames in the video clip are adjusted to a first resolution. After determining the key frames of the video clip, the key frames are adjusted to a second resolution, which is higher than the first resolution.

[0009] Furthermore, dynamic entity extraction is performed on keyframes, and the spatiotemporal relationships between entities are determined, specifically including: The target detection algorithm is used to detect targets in keyframes, and the multi-target tracking algorithm is used to track the motion trajectory of targets in multiple keyframes and mark the timestamps to achieve visual entity extraction. The location, time, and event are extracted from the captions of keyframes using a speech recognition model, thus achieving text entity extraction. The motion trajectory and position of visual entities are identified, and the spatiotemporal similarity between visual entities and text entities is calculated based on the identification results and timestamps. Visual entities and text entities with spatiotemporal similarity higher than a preset similarity threshold are fused.

[0010] Furthermore, the spatiotemporal contradiction detection rule base includes: Spatiotemporal conflict rules are used to detect whether there are conflicts in the spatial changes of entities at different times; Object property rules are used to detect whether entities of an object type exhibit behaviors that do not conform to physical rules. Event causality rules are used to detect whether there are contradictions between the causes and results of various events.

[0011] Furthermore, contradiction detection is performed on the spatiotemporal knowledge graph based on a spatiotemporal contradiction detection rule base, specifically including: The rules in the spatiotemporal contradiction detection rule base are parameterized, and the parameterized rules are instantiated according to the scenarios involved in the spatiotemporal knowledge graph. Determine the relationships between the instantiated rules, and configure association trigger conditions for the rules with relationships; The entity nodes in the spatiotemporal knowledge graph are converted into fact objects in the Drools working memory, and the attributes of the fact objects are configured according to the relationships between the entity nodes. The instantiated rules are invoked to analyze the attributes of the fact object, and to find contradictory or non-compliant attributes under the rule constraints. Output contradiction detection results based on the found attributes.

[0012] Furthermore, a spatiotemporal knowledge graph is used to detect contradictions through a spatiotemporal awareness-based ST-GNN anomaly detection model, specifically including: The ST-GNN anomaly detection model captures the evolution of entity node states in spatiotemporal knowledge graphs through temporal convolution. By using spatial graph convolution, the neighbor nodes of entity nodes are aggregated according to spatial relationship weights; Based on the evolution of entity node states and aggregation results, abnormal nodes are detected, and abnormal edges in the spatiotemporal knowledge graph are detected.

[0013] Compared with the prior art, the beneficial effects of the present invention are: This invention provides a video forgery analysis method based on a knowledge graph. On one hand, it segments the target video into several video segments and determines the keyframes of each video segment. Based on the entity relationships extracted from the keyframes, a spatiotemporal knowledge graph is established to detect potential forgery in the video, thus improving the efficiency of the initial video forgery analysis. On the other hand, it uses a predefined spatiotemporal contradiction detection rule base to perform contradiction detection on the spatiotemporal knowledge graph, marking abnormal nodes or edges in the spatiotemporal knowledge graph, which can effectively improve the interpretability of the video forgery detection results. At the same time, video forgery analysis based on spatiotemporal knowledge graph can directly detect semantic contradictions, solving the problem that traditional methods rely on pixel / frequency domain features and cannot identify forgeries that conform to visual rules but violate real-world logic. The method can effectively improve the accuracy of video forgery detection. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only preferred embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a schematic diagram of the overall process of a knowledge graph-based video forgery analysis method provided in an embodiment of the present invention. Detailed Implementation

[0016] The principles and features of the present invention are described below with reference to the accompanying drawings. The listed embodiments are only used to explain the present invention and are not intended to limit the scope of the present invention.

[0017] Reference Figure 1 This embodiment provides a video forgery analysis method based on knowledge graphs. The method includes: S101. Input the target video.

[0018] S102. Divide the target video into several video segments and determine the keyframes for each video segment.

[0019] In this step, at least one keyframe is determined for each video segment.

[0020] S103. Perform dynamic entity extraction on key frames and determine the spatiotemporal relationships between entities. Based on the entities and relationships extracted from each key frame, establish a spatiotemporal knowledge graph of the target video.

[0021] In this embodiment, entities are abstracted as entity nodes in a spatiotemporal knowledge graph, while the spatiotemporal interactions between different entities are abstracted as edges between corresponding entity nodes. A timestamp sequence is added to each entity node; for example, for person node A, its timestamp sequence could be [time t1 at position i, time t2 at position j, ...]. The physical distance or spatial topological relationship between entity nodes can be encoded as the weight attribute of the edges.

[0022] S104. Load the predefined spatiotemporal contradiction detection rule library, and perform contradiction detection on the spatiotemporal knowledge graph based on the spatiotemporal contradiction detection rule library.

[0023] S105. Based on the contradiction detection results, mark the abnormal nodes or edges of the spatiotemporal knowledge graph and output the marking results.

[0024] As one possible implementation, the target video is divided into several video segments, specifically including the following operations: S201. Extract the background feature vectors of each frame of the target video and calculate the difference in background feature vectors between adjacent frames.

[0025] In this implementation, the background feature vector can reflect information such as the background's color, texture, and shape. Background feature vector extraction can be achieved using the SIFT algorithm, the SURF algorithm, or other algorithms; this implementation does not specifically limit the method used.

[0026] S202. According to the frame division rules, the frames of the target video are divided into corresponding video segments. The frame division rules are as follows: consecutive frames with a background feature vector difference value less than or equal to a preset difference value are divided into the same video segment, and consecutive frames with a background feature vector difference value greater than the preset difference value are divided into different video segments.

[0027] In this implementation, if the difference in background feature vectors between adjacent frames is less than or equal to a preset difference value, it indicates that the background differences between the adjacent frames are very small, and the two frames are likely to belong to the same scene, so they can be classified into the same video segment. If the difference in background feature vectors between adjacent frames is greater than the preset difference value, it indicates that the background of the two adjacent frames has changed significantly, meaning that a transition may have occurred in the video, and such adjacent frames need to be classified into different video segments.

[0028] S203. If the difference value of the background feature vector between all adjacent frames of the target video is less than or equal to the preset difference value, the target video is divided into N video segments according to the total duration, and each video segment contains the same number of frames.

[0029] In this implementation, if the difference value of the background feature vector between all adjacent frames of the target video is less than or equal to the preset difference value, it means that all the frames of the target video are likely in the same scene. In this case, the target video is divided into N video segments of the same duration to improve the analysis efficiency.

[0030] As another possible implementation, keyframes are determined for each video segment, specifically including: S301. Identify sensitive areas in each frame of a video clip.

[0031] In this embodiment, the sensitive area is the area in the video most likely to be forged, such as faces, subtitles, and fast-moving objects.

[0032] S302. Calculate the degree of change in the sensitive area of ​​adjacent frames to obtain the rate of change of the sensitive area.

[0033] For example, the degree of change in sensitive areas between adjacent frames can be calculated using SSIM structural similarity.

[0034] S303. The change rate of different sensitive areas of the frame is weighted and calculated to obtain a comprehensive change score.

[0035] In this implementation, different weights are assigned to the rate of change of different sensitive areas in adjacent frames. Then, the rate of change of different sensitive areas in adjacent frames is weighted and summed to obtain a comprehensive change score. For example, the weight of the rate of change of the face region can be set to 0.4, and the weight of the rate of change of the subtitle region and the high-speed moving object can be set to 0.3.

[0036] S304. Determine the keyframes of the video segment based on the comprehensive change score.

[0037] This implementation calculates a comprehensive change score by weighted summation of the change rates of multiple sensitive areas in adjacent frames, and then determines keyframes based on this comprehensive change score. A higher comprehensive change score indicates a greater degree of change in the frame, making it more likely to be forged; therefore, frames with high comprehensive change scores are preferentially selected as keyframes.

[0038] As a further possible implementation, when identifying sensitive areas of each frame in a video segment, the frames in the video segment are adjusted to a first resolution. After determining the key frames of the video segment, the key frames are adjusted to a second resolution, which is higher than the first resolution.

[0039] For example, when identifying sensitive areas, the frame can be adjusted to 720P resolution, and after determining the keyframes of the video segment, the keyframes can be adjusted to 4K resolution. This implementation reduces the consumption of computing resources and improves the efficiency of early-stage calculations by using low resolution when identifying sensitive areas of the frame. Then, by adjusting the determined keyframes to high resolution, it helps to extract more image details from the keyframes, ensuring that there is enough data available for extraction in subsequent analysis.

[0040] As another possible implementation, dynamic entity extraction is performed on keyframes, and the spatiotemporal relationships between entities are determined, specifically including: S401. Targets in keyframes are detected using a target detection algorithm, and the motion trajectories of targets in multiple keyframes are tracked using a multi-target tracking algorithm, with timestamps added, to achieve visual entity extraction.

[0041] For example, the targets include people and objects, and the target detection algorithm can be YOLOv7. The multi-target tracking algorithm can be DeepSORT, which tracks the same target in different keyframes, analyzes its motion trajectory, and records the corresponding timestamps to complete dynamic visual entity extraction.

[0042] S402. Extract location, time and event from the captions of keyframes using a language recognition model to achieve text entity extraction.

[0043] S403. Identify the movement trajectory and position of visual entities, and calculate the spatiotemporal similarity between visual entities and text entities based on the identification results and timestamps.

[0044] Location, time, and event in text entities usually correspond to the position or movement of people or objects in the image. This implementation identifies the position of the motion trajectory in the visual entity and combines it with the timestamp to determine the spatiotemporal similarity between the visual entity and the text entity, thereby being able to determine whether the video is likely to be forged based on the differences between the visual entity and the text entity.

[0045] S404. Merge visual entities and text entities whose spatiotemporal similarity is higher than a preset similarity threshold.

[0046] This implementation extracts visual entities and text entities from keyframes, and then calculates the spatiotemporal similarity between the visual entities and text entities to determine whether they actually correspond. If the spatiotemporal similarity is higher than a preset similarity threshold, the corresponding visual entities and text entities are fused together, thereby avoiding the subsequent generation of a spatiotemporal knowledge graph from being too bloated and improving the efficiency of graph analysis.

[0047] In this embodiment, the spatiotemporal conflict detection rule base includes, but is not limited to, spatiotemporal conflict rules, object property rules, and event causality rules.

[0048] Spatiotemporal conflict rules are used to detect whether there are conflicts in the spatial changes of an entity at different times. For example, whether a character moves an unreasonable distance in a short period of time.

[0049] Object property rules are used to detect whether entities of an object type exhibit behaviors that do not conform to physical rules. For example, whether there are obviously unreasonable object placements, or whether there are phenomena that clearly do not conform to the rules of reality, such as the inconsistency between the direction of scene light sources and the direction of character shadows, or the inconsistency between the direction of smoke drifting and the wind direction.

[0050] Event causality rules are used to detect whether there are contradictions between the causes and effects of various events.

[0051] As another possible implementation method, contradiction detection of spatiotemporal knowledge graphs is performed based on a spatiotemporal contradiction detection rule base, specifically including: S501. Parameterize the various rules in the spatiotemporal contradiction detection rule base, and instantiate the parameterized rules according to the scenarios involved in the spatiotemporal knowledge graph.

[0052] Parameterization involves converting the rules into data types that are easier for computers to understand, such as numeric or Boolean types. It is understood that different video content corresponds to different scenarios, and therefore, different detection rules are applicable to each. This implementation selects and instantiates parameterized rules based on the scenarios involved in the spatiotemporal knowledge graph, thereby invoking the appropriate detection rules according to actual needs. This avoids using too many inapplicable detection rules to analyze a single video, improving analysis efficiency and reducing the algorithm's consumption of computing resources.

[0053] S502. Determine the relationships between the instantiated rules and configure association triggering conditions for the rules that have relationships.

[0054] In this implementation, when a rule is invoked, other rules that are related to it will also be invoked.

[0055] S503. Convert entity nodes in the spatiotemporal knowledge graph into fact objects in the Drools working memory, and configure the attributes of the fact objects according to the relationships between entity nodes.

[0056] S504. Invoke the instantiated rules to analyze the attributes of the fact object and find contradictory or non-compliant attributes under the rule constraints.

[0057] S505. Output the contradiction detection results based on the found attributes.

[0058] This implementation parameterizes the rules in the spatiotemporal contradiction detection rule base and instantiates the parameterized rules according to the scenarios involved in the spatiotemporal knowledge graph. This allows for the detection of forged content within the spatiotemporal knowledge graph under rule constraints. By setting associated trigger conditions for related instantiated rules, when a rule is triggered, other associated instantiated rules are simultaneously invoked, thereby detecting other abnormal content in the spatiotemporal knowledge graph. When contradictory attributes or attributes that do not conform to the rules are found, the corresponding edges or entity nodes in the spatiotemporal knowledge graph are traced and marked, thus improving the visualization and interpretability of the contradiction detection results.

[0059] As a further possible implementation, the method also performs contradiction detection on the spatiotemporal knowledge graph using a spatiotemporally aware ST-GNN anomaly detection model, specifically including: The S601 and ST-GNN anomaly detection models capture the evolution of entity node states in spatiotemporal knowledge graphs through temporal convolution.

[0060] In this implementation, the ST-GNN model is a graph neural network with spatiotemporal awareness. By learning the temporal patterns of normal behavior, it can detect anomalies when the state of an entity node deviates from the normal pattern. For example, if it normally takes 4 seconds for a person to move from the bedroom to the living room, but analysis of the state of the entity node corresponding to the person reveals that it deviates from this pattern, then the entity node is considered to be potentially forged.

[0061] S602. Aggregate the neighbor nodes of the entity node according to the spatial relationship weights through spatial graph convolution.

[0062] In this implementation, the spatial aggregation result of normal entity nodes should satisfy:

[0063] in, As an empirical threshold, The features of the aggregated neighboring entity nodes, Features of neighboring entity nodes before aggregation.

[0064] S603. Detect abnormal nodes based on the evolution law of entity node states and aggregation results, and detect abnormal edges in the spatiotemporal knowledge graph.

[0065] Forged nodes often lead to abnormal attention weights for entity nodes, such as high weights for nodes that are far away in the graph, or aggregated features that deviate from the normal distribution. This implementation analyzes the temporal state evolution and spatial relationships of entity nodes in the spatiotemporal knowledge graph using the ST-GNN model. By fusing graph structure, temporal dynamics, and spatial relationships, it identifies and marks contradictory nodes or edges in the spatiotemporal knowledge graph that violate real-world rules.

[0066] This implementation method, based on explicit contradiction analysis of the spatiotemporal knowledge graph using a spatiotemporal contradiction detection rule base, further analyzes the spatiotemporal knowledge graph using implicit anomaly propagation based on graph neural networks. This allows for more accurate and comprehensive detection of abnormal regions in the spatiotemporal knowledge graph, which helps to further improve the accuracy of video forgery analysis.

[0067] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A video forgery analysis method based on knowledge graphs, characterized in that, The method includes: Input the target video; The target video is divided into several video segments, and keyframes are determined for each video segment; Dynamic entity extraction is performed on keyframes, and the spatiotemporal relationships between entities are determined. A spatiotemporal knowledge graph of the target video is built based on the entities and relationships extracted from each keyframe. Load a predefined spatiotemporal contradiction detection rule library, and perform contradiction detection on the spatiotemporal knowledge graph based on the spatiotemporal contradiction detection rule library; Based on the contradiction detection results, abnormal nodes or edges in the spatiotemporal knowledge graph are marked, and the marking results are output.

2. The video forgery analysis method based on knowledge graphs according to claim 1, characterized in that, The target video is divided into several video segments, specifically including the following operations: The background feature vectors of each frame of the target video are extracted, and the difference in background feature vectors between adjacent frames is calculated. According to the frame division rules, each frame of the target video is divided into corresponding video segments. The frame division rules are as follows: consecutive frames with a background feature vector difference value less than or equal to a preset difference value are divided into the same video segment, and consecutive frames with a background feature vector difference value greater than the preset difference value are divided into different video segments. If the difference in background feature vectors between all adjacent frames of the target video is less than or equal to a preset difference value, the target video is divided into N video segments according to the total duration, with each video segment containing the same number of frames.

3. The video forgery analysis method based on knowledge graphs according to claim 2, characterized in that, For each video segment, keyframes are determined as follows: Identify sensitive areas in each frame of a video clip; The degree of change in the sensitive area of ​​adjacent frames is calculated to obtain the rate of change of the sensitive area; The change rate of different sensitive areas in a frame is weighted and calculated to obtain a comprehensive change score. Keyframes for video segments are determined based on comprehensive change scores.

4. The video forgery analysis method based on knowledge graphs according to claim 3, characterized in that, Determining keyframes for each video segment also includes: Extract audio features from each frame of a video clip; The degree of change in audio features between adjacent frames is calculated to obtain the rate of change in audio features; The rate of change and the degree of change of audio features in different sensitive areas of the frame are weighted and calculated to obtain a comprehensive change score.

5. The video forgery analysis method based on knowledge graph according to claim 3, characterized in that, When identifying sensitive areas in each frame of a video clip, the frames in the video clip are adjusted to the first resolution. After determining the key frames of the video clip, the key frames are adjusted to the second resolution, which is higher than the first resolution.

6. The video forgery analysis method based on knowledge graph according to claim 1, characterized in that, Dynamic entity extraction is performed on keyframes, and the spatiotemporal relationships between entities are determined, specifically including: The target detection algorithm is used to detect targets in keyframes, and the multi-target tracking algorithm is used to track the motion trajectory of targets in multiple keyframes and mark the timestamps to achieve visual entity extraction. The location, time, and event are extracted from the captions of keyframes using a speech recognition model, thus achieving text entity extraction. The motion trajectory and position of visual entities are identified, and the spatiotemporal similarity between visual entities and text entities is calculated based on the identification results and timestamps. Visual entities and text entities with spatiotemporal similarity higher than a preset similarity threshold are fused.

7. The video forgery analysis method based on knowledge graph according to claim 1, characterized in that, The spatiotemporal contradiction detection rule base includes: Spatiotemporal conflict rules are used to detect whether there are conflicts in the spatial changes of entities at different times; Object property rules are used to detect whether entities of an object type exhibit behaviors that do not conform to physical rules. Event causality rules are used to detect whether there are contradictions between the causes and results of various events.

8. The video forgery analysis method based on knowledge graph according to claim 1, characterized in that, The spatiotemporal knowledge graph is subjected to contradiction detection based on a spatiotemporal contradiction detection rule base, specifically including: The rules in the spatiotemporal contradiction detection rule base are parameterized, and the parameterized rules are instantiated according to the scenarios involved in the spatiotemporal knowledge graph. Determine the relationships between the instantiated rules, and configure association trigger conditions for the rules with relationships; The entity nodes in the spatiotemporal knowledge graph are converted into fact objects in the Drools working memory, and the attributes of the fact objects are configured according to the relationships between the entity nodes. The instantiated rules are invoked to analyze the attributes of the fact object, and to find contradictory or non-compliant attributes under the rule constraints. Output contradiction detection results based on the found attributes.

9. The video forgery analysis method based on knowledge graph according to claim 1, characterized in that, Furthermore, a spatiotemporal knowledge graph is used to detect contradictions through a spatiotemporal awareness-based ST-GNN anomaly detection model, specifically including: The ST-GNN anomaly detection model captures the evolution of entity node states in spatiotemporal knowledge graphs through temporal convolution. By using spatial graph convolution, the neighbor nodes of entity nodes are aggregated according to spatial relationship weights; Based on the evolution of entity node states and aggregation results, abnormal nodes are detected, and abnormal edges in the spatiotemporal knowledge graph are detected.

Images