Video material misappropriation detection method, device and equipment, and storage medium

Abstract

This application discloses a method, apparatus, device, and storage medium for detecting the misappropriation of video materials, relating to the field of video review technology. The method includes: acquiring shot clips and corresponding clip information; extracting an assertion to be verified from the clip information, and selecting a target keyframe from the shot clip corresponding to the assertion to be verified; extracting multiple visual features of the target keyframe, comparing the multiple visual features with the features of reference materials in a reference material library to obtain a comparison result; obtaining candidate materials based on the comparison result, and determining the evidentiary relationship between the candidate materials and the assertion to be verified; constructing a dynamic evidence chain diagram based on the assertion to be verified, the shot clip, the target keyframe, the candidate materials, and the evidentiary relationship, and outputting the material misappropriation risk of the assertion to be verified based on the dynamic evidence chain diagram. This application can improve the accuracy of detecting the risk of video material misappropriation.

Description

Technical Field

[0001] This application relates to the field of video review technology, and in particular to a method, apparatus, equipment, and storage medium for detecting the misappropriation of video materials. Background Technology

[0002] Current short video review methods primarily rely on speech recognition, object detection, and keyword matching to determine whether a video contains inappropriate content. While these methods can identify visually similar scenes, they cannot determine whether the similar footage is factually consistent with the current video narrative. They are particularly ineffective at handling visually similar but inconsistent footage, resulting in low accuracy in detecting video footage misappropriation risks. Therefore, improving the accuracy of video footage misappropriation risk detection remains a problem that needs to be addressed.

[0003] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main objective of this application is to provide a method, apparatus, device, and storage medium for detecting the misappropriation of video materials, aiming to solve the technical problem of how to improve the accuracy of detecting the risk of video material misappropriation.

[0005] To achieve the above objectives, this application proposes a method for detecting the misappropriation of video materials, the method comprising: The video to be reviewed is obtained, and the video to be reviewed is segmented into shots and parsed with multimodal information to obtain multiple shot segments and the corresponding segment information of the shot segments; Extract the assertions to be verified from the fragment information and establish the mapping relationship between the assertions to be verified and the shot fragments; Select target keyframes from the shot segments corresponding to the assertions to be verified based on the mapping relationship; Extract multiple visual features from the target keyframe, compare the multiple visual features with the reference material features in the reference material library, and obtain the comparison results; Based on the comparison results, candidate materials are obtained from the reference material library, and the evidentiary relationship between the candidate materials and the assertion to be verified is determined. A dynamic evidence chain diagram is constructed based on the assertion to be verified, the shot clip, the target keyframe, the candidate material, and the evidence relationship, and the material misappropriation risk of the assertion to be verified is output based on the dynamic evidence chain diagram.

[0006] In one embodiment, the step of extracting the assertion to be verified from the segment information and establishing the mapping relationship between the assertion to be verified and the shot segment includes: Candidate assertions are extracted from the title, publication text, on-screen text, speech-to-text, and visual semantic description in the fragment information. The candidate assertions contain a set of event elements, which includes the event subject, action, object, location, time, attribute, and result. The verification value score of the candidate assertion is calculated based on the entity density, temporal specificity, location specificity, public communication relevance, risk propagation intensity, and assertion strength of the candidate assertion. Candidate assertions whose verification value scores are higher than a preset verification threshold are identified as assertions to be verified. Calculate the semantic correlation between each of the assertions to be verified and each of the shot segments; A mapping relationship between the assertion to be verified and the shot segment is established based on the semantic relevance.

[0007] In one embodiment, the step of selecting target keyframes from the shot segments corresponding to the assertion to be verified based on the mapping relationship includes: Candidate keyframes are determined from the shot segments corresponding to the assertions to be verified based on the mapping relationship; The reverse search value score is calculated based on whether the candidate keyframe is located at a scene change location, the amount of text information in the candidate keyframe, the salience of the location clues in the candidate keyframe, the salience of the core object in the candidate keyframe, the image clarity of the candidate keyframe, the semantic relevance between the candidate keyframe and the assertion to be verified, and the duplication between the candidate keyframe and the selected candidate keyframe. The target keyframe is selected from the candidate keyframes according to the reverse search value score.

[0008] In one embodiment, the multiple visual features include perceptual hash features, local visual features, deep semantic features, and on-screen text features. The step of comparing the multiple visual features with reference material features in a reference material library to obtain the comparison result includes: The perceptual hash feature, the local visual feature, the deep semantic feature, and the image text feature are compared with the corresponding features of each reference material in the reference material library to obtain the similarity score of each reference material at multiple granularities. The similarity scores at each granularity are weighted and fused to obtain the comprehensive visual similarity of each reference material, and the comprehensive visual similarity is used as the comparison result.

[0009] In one embodiment, the step of determining the evidentiary relationship between the candidate material and the assertion to be verified includes: Obtain the metadata of each candidate material, including the candidate material title, candidate material summary, and candidate material release time; Based on the metadata, the homology probability, event relevance probability, and irrelevance probability between each candidate material and the assertion to be verified are obtained; The re-ranking score of each candidate material is calculated based on the homology probability, the event relevance probability, and the irrelevance probability. Evidence candidate materials are selected from each of the candidate materials based on the re-ranking scores; The original source of the candidate evidence is traced to obtain the source information of the candidate evidence, which includes the publication time, the publishing entity, and the event description. The event elements of the assertion to be verified are compared with the event elements of the candidate evidence materials in terms of subject consistency, location consistency, time consistency, and result consistency to obtain the verification results. The evidentiary relationship between the candidate evidence materials and the assertion to be verified is determined based on the verification results. The evidentiary relationship includes supporting evidence, rebuttal evidence, partially supporting evidence, and evidence that cannot be determined.

[0010] In one embodiment, the step of outputting the material misappropriation risk of the assertion to be verified based on the dynamic evidence chain diagram includes: The source information of each candidate material is obtained based on the dynamic evidence chain diagram, and the source credibility score is calculated based on the source information. Calculate the supporting evidence strength and rebuttal evidence strength of each candidate material based on the dynamic evidence chain diagram; The dynamic evidence chain diagram identifies the time conflict path, location conflict path, and event conflict path between the assertion to be verified and each of the candidate materials, and calculates the conflict path risk based on the time conflict path, the location conflict path, and the event conflict path. The sufficiency score is calculated based on the dynamic evidence chain diagram. The sufficiency score is calculated based on the number of valid evidence, the diversity of evidence sources, the coverage of authoritative sources, the consistency of time, and the traceability of sources. The risk of misappropriation of materials for the assertion to be verified is calculated based on the source credibility score, the strength of supporting evidence, the strength of rebuttal evidence, the risk of conflict path, and the sufficiency of evidence score.

[0011] In one embodiment, before the step of selecting the target keyframe from the shot segment corresponding to the assertion to be verified according to the mapping relationship, the method further includes: Calculate the scores for visual material reuse, text visual event mismatch, location and time conflict, and title / subtitle manipulation between the assertion to be verified and the corresponding shot segment; Based on the visual material reuse score, the text visual event mismatch score, the location-time conflict score, and the title / subtitle inducement score, determine whether to execute the step of selecting the target keyframe from the shot segment corresponding to the assertion to be verified according to the mapping relationship.

[0012] Furthermore, to achieve the above objectives, this application also proposes a video material misappropriation detection device, which includes: The acquisition module is used to acquire the video to be reviewed, perform shot segmentation and multimodal information parsing on the video to be reviewed, and obtain multiple shot segments and the segment information corresponding to the shot segments; A module is established to extract assertions to be verified from the segment information and establish a mapping relationship between the assertions to be verified and the shot segments; The selection module is used to select target keyframes from the shot segments corresponding to the assertion to be verified according to the mapping relationship; The comparison module is used to extract multiple visual features of the target keyframe, compare the multiple visual features with the reference material features in the reference material library, and obtain the comparison results. The determination module is used to obtain candidate materials from the reference material library based on the comparison results, and determine the evidentiary relationship between the candidate materials and the assertion to be verified. The construction module is used to construct a dynamic evidence chain diagram based on the assertion to be verified, the shot clip, the target keyframe, the candidate material, and the evidence relationship, and output the material misappropriation risk of the assertion to be verified based on the dynamic evidence chain diagram.

[0013] In addition, to achieve the above objectives, this application also proposes a video material misappropriation detection device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the video material misappropriation detection method described above.

[0014] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the video material misappropriation detection method described above.

[0015] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the video material misappropriation detection method described above.

[0016] This application provides a method for detecting the misappropriation of video materials. The method involves acquiring a video to be reviewed, performing shot segmentation and multimodal information parsing on the video to obtain multiple shot segments and corresponding segment information; extracting assertions to be verified from the segment information and establishing a mapping relationship between the assertions and the shot segments; selecting target keyframes from the shot segments corresponding to the assertions based on the mapping relationship; extracting multiple visual features from the target keyframes and comparing the similarity of these visual features with reference material features in a reference material library to obtain comparison results; obtaining candidate materials from the reference material library based on the comparison results and determining the evidentiary relationship between the candidate materials and the assertions to be verified; constructing a dynamic evidence chain graph based on the assertions to be verified, the shot segments, the target keyframes, the candidate materials, and the evidentiary relationship; and outputting the material misappropriation risk of the assertions to be verified based on the dynamic evidence chain graph. This application extracts assertions from videos and establishes a mapping relationship between assertions and video clips. Subsequently, assertions, rather than video footage, can be used to determine the risk of misappropriation of video materials. This changes the focus of the review from broad video content to verifiable and specific factual statements, improving the accuracy of detecting the risk of misappropriation of video materials. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating an embodiment of the video material misappropriation detection method of this application. Figure 2 This is a flowchart illustrating Embodiment 2 of the video material misappropriation detection method of this application; Figure 3 This is a schematic diagram of the module structure of the video material misappropriation detection device according to an embodiment of this application; Figure 4 This is a schematic diagram of the hardware operating environment involved in the video material misappropriation detection method in this application embodiment.

[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0022] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0023] This application acquires a video to be reviewed, performs shot segmentation and multimodal information parsing on the video to be reviewed, obtains multiple shot segments and corresponding segment information; extracts assertions to be verified from the segment information, and establishes a mapping relationship between the assertions to be verified and the shot segments; selects target keyframes from the shot segments corresponding to the assertions to be verified according to the mapping relationship; extracts multiple visual features of the target keyframes, compares the multiple visual features with the features of reference materials in a reference material library, and obtains a comparison result; obtains candidate materials from the reference material library according to the comparison result, and determines the evidentiary relationship between the candidate materials and the assertions to be verified; constructs a dynamic evidence chain graph based on the assertions to be verified, the shot segments, the target keyframes, the candidate materials, and the evidentiary relationship, and outputs the material misappropriation risk of the assertions to be verified based on the dynamic evidence chain graph.

[0024] Current short video review methods primarily rely on speech recognition, object detection, and keyword matching to determine whether a video contains inappropriate content. While these methods can identify visually similar scenes, they cannot determine whether the similar footage is factually consistent with the current video narrative. They are particularly ineffective at handling visually similar but inconsistent footage, resulting in low accuracy in detecting video footage misappropriation risks. Therefore, improving the accuracy of video footage misappropriation risk detection remains a problem that needs to be addressed.

[0025] This application extracts assertions from videos and establishes a mapping relationship between assertions and video clips. Subsequently, assertions, rather than video footage, can be used to determine the risk of misappropriation of video materials. This changes the focus of the review from broad video content to verifiable and specific factual statements, improving the accuracy of detecting the risk of misappropriation of video materials.

[0026] All user-related data involved in this application were obtained with the user's permission or consent; that is, when this application is applied to specific products or technologies, user permission is required to obtain and process the relevant data, and the processing of the relevant data must comply with the relevant laws, regulations and regulatory standards of the relevant countries and regions.

[0027] Based on this, embodiments of this application provide a method for detecting the misappropriation of video materials, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the video material misappropriation detection method of this application.

[0028] In this embodiment, the video material misappropriation detection method includes steps S10~S60: Step S10: Obtain the video to be reviewed, perform shot segmentation and multimodal information parsing on the video to be reviewed, and obtain multiple shot segments and the corresponding segment information of the shot segments; It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or video material misappropriation detection device capable of performing the above functions. The following description uses a video material misappropriation detection device as an example to illustrate this embodiment and the subsequent embodiments.

[0029] It should be noted that the video to be reviewed in this embodiment is a short video, and the short video to be reviewed is denoted as:

[0030] Where F represents the video frame sequence, A represents the audio sequence, O represents the text sequence obtained by Optical Character Recognition (OCR), S represents the speech-to-text sequence obtained by speech recognition, and M represents the short video metadata. The metadata M includes the title, posting text, cover image, posting time, posting account, hashtags, source information, comment information, and other information available to the platform.

[0031] After obtaining the short video to be reviewed, it can be segmented into several shot segments:

[0032] in, Let represent the i-th shot segment, and n represent the number of shot segments. Shot segmentation can be performed using a content-aware boundary detection method based on the hue, saturation, value, and HSV color space. Specifically, it involves calculating the content difference scores formed by the HSV components and edge changes of adjacent frames frame by frame, and statistically analyzing the histogram difference values ​​between adjacent frames. When the histogram difference value exceeds a preset threshold, it is determined as a shot switch, thereby obtaining the start and end timestamps of each shot and dividing the video into several continuous shot segments.

[0033] For each shot / fragment It can extract fragment information:

[0034] in, This represents the set of keyframes for a scene segment. Indicates the time range of a shot segment. This indicates OCR subtitles or on-screen text. This represents a speech-to-text segment. Indicates audio features, Represents visual semantic description, This indicates the visual event elements identified in the segment.

[0035] Specifically, Several representative frames can be selected within the shot based on image quality, representativeness, and the degree of scene change. The start and end timestamps can be directly given from the shot segmentation results; The text can be detected and recognized from key frames within the lens using an OCR recognition engine. The corresponding audio range for this shot can be transcribed using an Automatic Speech Recognition (ASR) engine. It can be extracted through Mel spectrum, Mel-Frequency Cepstral Coefficients (MFCC) or pre-trained audio encoder; Natural language descriptions can be generated from keyframes using visual language models (such as Bootstrapping Language-Image Pre-training, BLIP, or multimodal large models); The footage can be structurally analyzed through object detection, scene recognition, or multimodal large models to identify possible subjects, actions, locations, and other elements in the footage, which can then be used as auxiliary input for subsequent event element verification.

[0036] All the segment information and metadata can also be integrated into a short video review package:

[0037] Where V represents a video clip, M represents short video metadata, and U represents all clip-level review units:

[0038] The short video review package is used for subsequent assertion extraction, segment mapping, identification of material misappropriation risks, selection of keyframes for reverse search, and construction of evidence chains.

[0039] Furthermore, to improve feasibility, processing logs for each segment can be recorded simultaneously when generating the short video review package, including shot segmentation time, OCR confidence level, speech recognition confidence level, visual semantic generation confidence level, and keyframe quality indicators. These processing logs can serve as data for subsequent manual review and threshold calibration.

[0040] Step S20: Extract the assertions to be verified from the segment information and establish a mapping relationship between the assertions to be verified and the shot segments; It should be noted that the assertions to be verified refer to specific statements in short videos that have factual verification value. Specifically, assertions to be verified can be extracted from titles, posting text, OCR subtitles, speech-to-text, and visual semantic descriptions to obtain a set of assertions to be verified.

[0041] in, This indicates the j-th assertion to be verified, and m represents the number of assertions.

[0042] The method can extract assertions to be verified based on an open-source large language model. Specifically, the title, published text, OCR subtitles, speech-to-text, and visual semantic description can be concatenated as input. The model is required to output in JSON format according to seven fields: "subject, action, object, location, time, attribute, and result" through a preset prompt template. Then, the set of assertions to be verified can be filtered out after field integrity verification, which can be used for subsequent verification value scoring.

[0043] Each assertion pending verification It can be represented as a set of event elements:

[0044] in, Indicates the subject of the event. Indicates an action or event. Represents an object, Indicates location, Indicates time, Represents attributes, Indicates a result or conclusion.

[0045] Specifically, the verification value score of candidate assertions can be calculated to select a set of assertions to be verified. Candidate assertions refer to the set of original factual statements extracted from the title, posting text, on-screen text, speech-to-text, and visual semantic description of short videos that have not yet undergone verification value assessment and screening. The formula for calculating the verification value score of candidate assertions is as follows:

[0046] in, Represents the density of the substance. Indicates the definiteness of time. Indicates the definiteness of location. Indicates the relevance of public communication. Indicates the intensity of risk transmission. Indicates the assertion strength. to For weight parameters, This refers to the Sigmoid function. A Chinese named entity recognition model based on a bidirectional long short-term memory network conditional random field (BERT-BiLSTM-CRF) using a bidirectional encoder to represent entities such as people, places, organizations, and times in candidate assertion text can be used to identify these entities. The number of entities is then divided by the total number of words in the text to obtain the result. A Chinese time expression recognition tool (such as the time phrase parsing module of the HanLP language processing package) can be used to determine whether the time expression in the assertion can be parsed into a specific date. If it can be parsed, a high score is assigned, and the time precision is obtained. The location specificity can be determined by whether the location entity identified using the Named Entity Recognition (NER) model can be precisely matched with a place name dictionary. The relevance of public communication can be obtained by directly using publicly available data such as likes, comments, and shares from the platform, which are then log-normalized and weighted summed. A predefined list of risk-sensitive terms (such as "sudden," "emergency," "policy," etc.) can be used to score keyword hits and determine the intensity of risk transmission. A predefined list of absolute qualifiers (such as "all", "no condition required", "necessary", etc.) can be used for rule matching and scoring to obtain the assertion strength. Weight to This can be achieved by training a logistic regression model on a manually annotated short video review sample set. Furthermore, the weight parameters can be obtained through training on the manually annotated short video review sample set, or they can be preset by the review scenario. For example, for campus promotional videos, the weights for location clarity, subject consistency, and source credibility can be increased; for short videos about trending events, the weights for time clarity, risk dissemination intensity, and verifiability of release time can be increased.

[0047] When the verification value score meets the following:

[0048] The candidate assertion can be identified as an assertion to be verified, where This is a preset verification threshold.

[0049] In one feasible approach, the step of extracting assertions to be verified from the fragment information and establishing a mapping relationship between the assertions to be verified and the shot fragments includes: extracting candidate assertions from the title, publication text, on-screen text, speech-to-text, and visual semantic description in the fragment information, wherein the candidate assertions contain a set of event elements, including event subject, action, object, location, time, attribute, and result; calculating the verification value score of the candidate assertions, wherein the verification value score is calculated based on the entity density, time clarification, location clarification, public communication relevance, risk dissemination intensity, and assertion strength of the candidate assertions; identifying candidate assertions with verification value scores higher than a preset verification threshold as assertions to be verified; calculating the semantic correlation degree between each assertion to be verified and each shot fragment; and establishing a mapping relationship between the assertions to be verified and the shot fragments based on the semantic correlation degree.

[0050] It should be noted that, in order to locate the shot segment corresponding to the assertion to be verified, the assertion to be verified can be calculated. With footage Semantic relevance between :

[0051] in, This indicates the similarity between the assertion to be verified and the OCR subtitle. This indicates the similarity between the assertion to be verified and the speech transcription. This indicates the similarity between the assertion to be verified and the visual semantic description. This indicates the degree of correlation between the time of occurrence of the assertion to be verified and the time range of the fragment. to These are the weight parameters.

[0052] The similarity can be calculated by vectorizing the text and then using cosine similarity. Specifically, the two text segments are converted into vectors using a model, and then the cosine of the angle between the vectors is calculated. The smaller the angle, the higher the similarity. , as well as However, tools (such as HanLP's time phrase parsing module) can be used to parse specific time points from assertions, and then calculate whether that time point falls within the start and end timestamp range of the shot, or calculate the distance between the two on the timeline, to obtain... The closer the distance, the higher the correlation.

[0053] When semantic relevance satisfy:

[0054] Then it is considered an assertion to be verified. With fragments There is a mapping relationship. It is a preset mapping threshold. The set of mapped fragments is denoted as:

[0055] By mapping assertion fragments as described above, abstract factual assertions can be bound to specific time ranges, specific shot segments, and specific keyframes, avoiding the problem that existing whole-video classification methods cannot locate risk locations.

[0056] Step S30: Select target keyframes from the shot segments corresponding to the assertion to be verified according to the mapping relationship; It should be noted that the mapping relationship can yield the corresponding shot segment for each assertion to be verified. However, these shot segments contain invalid segments, and not all frames have high evidentiary value. Therefore, target keyframes can be selected from the shot segments.

[0057] In one feasible approach, before the step of selecting a target keyframe from the shot segment corresponding to the assertion to be verified based on the mapping relationship, the method further includes: calculating a visual material reuse score, a text visual event mismatch score, a location-time conflict score, and a title / subtitle inducement score between the assertion to be verified and the corresponding shot segment; and determining whether to perform the step of selecting a target keyframe from the shot segment corresponding to the assertion to be verified based on the visual material reuse score, the text visual event mismatch score, the location-time conflict score, and the title / subtitle inducement score.

[0058] It should be noted that the risk of material misappropriation can be reviewed. Specifically, the multi-granularity anti-search evidence chain construction process can be determined based on the assertion fragment mapping results, the degree of mismatch between text and visual events, signs of visual material reuse, the degree of location and time conflict, and the misleading nature of the title and subtitles.

[0059] First, calculate the text visual event mismatch score, and then verify the assertions. Text event elements are encoded as vectors Encode visual event elements in the mapped segment into vectors Then the text visual event mismatch score for:

[0060] When the score for visual event mismatch in text is high, it means that the event described in the title, subtitle, or narration is difficult to find corresponding visual evidence in the picture.

[0061] Next, the visual material reuse score is calculated for each segment. Keyframes in The system extracts perceptual hashes. and depth visual vectors and with historical material library The visual material reuse score is obtained by matching the reference material r in the image. for:

[0062] in, Indicates Hamming distance, Indicates the hash length. and For weight parameters. Specifically, the perceptual hash h(k) can be calculated using the perceptual hash algorithm (pHash algorithm); the deep visual vector e(k) can be obtained by calculating the cosine similarity between the keyframe image embedding and the assertion text embedding using the Contrastive Language-Image Pre-training (CLIP) model.

[0063] Then, the location-time conflict score is calculated, specifically by extracting locations from the assertions. and time Extract clues such as landmarks, road signs, buildings, school gates, venues, and license plates from visual clips and candidate images, and calculate the location-time conflict score:

[0064] in, Indicates the location conflict score. Indicates the score for time conflict. and The weight parameters are used. Specifically, visual cues such as landmarks, buildings, school gates, and stadiums can be detected using the Grounding DINO (Open Vocabulary Object Detection) model. By specifying candidate category texts (such as "school gate", "teaching building", "gymnasium", etc.), the corresponding target and its confidence level can be located in the image. Textual cues such as road signs, license plates, and banners, as well as date watermarks in the image, are uniformly identified using the PaddleOCR (Optical Character Recognition) engine to obtain the specific text content.

[0065] It should be noted that comparing the location information extracted from the assertion with the location visual cues identified from the video footage to determine whether they point to the same location yields a location conflict score. The higher the degree of inconsistency, the greater the conflict score. Specifically, location elements are extracted from the assertions to be verified. And extract location visual cues from visual fragments. If the two can be precisely matched or belong to the same geographical area, then Take the lower value (closest to 0). If the two cannot match or clearly point to different locations, then Take the higher value. If no valid visual cues of location are detected in the image, this item will not be included in the trigger judgment.

[0066] It should be noted that the time information extracted from the assertion is compared with the time visual cues identified from the video footage to determine whether the two point to the same time. The higher the degree of inconsistency, the greater the conflict score. Time expression recognition tools (such as HanLP's time phrase parsing module) are used to analyze the assertions to be verified. Parsing time information The system retrieves the footage from clip c. i Extracting time clues from the image, including using OCR to recognize text containing time information such as timestamps and date watermarks, to obtain... Then and A comparison is performed; if the two are identical or the difference is within the preset tolerance range, then... Take the lower value; if the difference between the two values ​​significantly exceeds the preset tolerance range, then... Take the higher value. If no valid visual cues of time are detected in the image, this item will not be included in the trigger judgment.

[0067] Next, the title and subtitle inducement score is calculated. Specifically, the score is determined by identifying whether the title, subtitle, or narration contains expressions that induce time, location, or authenticity, such as "just now," "breaking news," "on-site exposure," "locally occurring," "officially confirmed," "real footage," "latest scene," or "emergency notice."

[0068] in, A set of words indicating urgency Indicates a set of location-referencing words. Indicates a set of time-pointing words. A collection of words that imply authoritative sources. This represents the set of words that imply authenticity. `f` is a function based on statistical results from five categories of words, representing a weighted sum of the number of words in each set, with the weights used for the summation being preset values.

[0069] Finally, the material misappropriation trigger conditions are constructed. When any of the following conditions are met, the material misappropriation multi-granularity reverse search evidence chain construction process is triggered:

[0070] or: and

[0071] or:

[0072] or: Furthermore, the footage lacks corresponding verifiable evidence.

[0073] in, , , and These are the visual reuse risk threshold, text visual mismatch risk threshold, location-time conflict risk threshold, and text-induced risk threshold. The system can determine whether the current image contains verifiable location or time clues corresponding to the assertion to be verified based on object detection results and OCR text recognition results. Specifically, Grounding DINO can be used to perform object detection based on preset landmarks, buildings, school gates, stadiums, road signs, or other scene target cues; simultaneously, PaddleOCR is used to extract school names, institution names, road signs, banners, timestamps, and location text from the image. If no relevant scene targets with a confidence level reaching the preset threshold are found in the object detection results, and no text clues matching the assertion's location or time are extracted in the OCR results, the current image is marked as "lacking verifiable visual clues," rather than directly determining that the image is inconsistent with the assertion.

[0074] To reduce the false trigger rate, a combined trigger condition can be further set: when a single risk score does not reach the strong trigger threshold but multiple weak risk indicators rise simultaneously, the system calculates a comprehensive trigger score.

[0075] in, , , as well as These are the weight parameters.

[0076] If the following conditions are met:

[0077] This triggers the process of constructing a reverse search evidence chain. Among other things, This is the comprehensive threshold.

[0078] In one feasible approach, the step of selecting a target keyframe from the shot segment corresponding to the assertion to be verified based on the mapping relationship includes: determining candidate keyframes from the shot segment corresponding to the assertion to be verified based on the mapping relationship; calculating a reverse search value score based on whether the candidate keyframe is located at a scene change location, the amount of text information in the candidate keyframe, the salience of the location clues in the candidate keyframe, the salience of the core object in the candidate keyframe, the image clarity of the candidate keyframe, the semantic relevance between the candidate keyframe and the assertion to be verified, and the repetition between the candidate keyframe and the selected candidate keyframe; and selecting a target keyframe from the candidate keyframes according to the reverse search value score.

[0079] It should be noted that candidate keyframes refer to several frames of a shot segment that have a mapping relationship with the assertion to be verified. Specifically, the reverse search value score can be calculated based on whether the candidate keyframe is located at a scene change location, the amount of text information in the candidate keyframe, the salience of the location clues in the candidate keyframe, the salience of the core object in the candidate keyframe, the image clarity of the candidate keyframe, the semantic relevance between the candidate keyframe and the assertion to be verified, and the repetition between the candidate keyframe and the selected candidate keyframes. The formula is as follows:

[0080] in, This indicates whether the keyframe is located at a point where there is a camera transition, scene change, or significant motion change; This indicates the amount of identifiable information in the keyframe, such as text, banners, logos, license plates, road signs, school names, and organization names. This indicates the salience of location cues such as landmarks, buildings, roads, school gates, venues, and indoor and outdoor environments in the keyframe; Indicates the salience of core objects such as people, vehicles, equipment, disaster scenes, event props, and ritual scenes; Indicates image clarity, searchability, and degree of occlusion; Indicates the semantic correlation between the keyframe and the assertion to be verified; This indicates the degree of repetition between the current frame and the selected frames; to These are the weight parameters.

[0081] Specifically, B(k) is directly determined by the shot boundary marker in the shot segmentation result at point 0; Ocr(k) is obtained by counting the number of text detection boxes in the PaddleOCR recognition result; Land(k) is obtained by the detection confidence of Grounding DINO for predefined landmark categories; Obj(k) is obtained by the detection confidence of Grounding DINO for predefined core object categories (people, vehicles, props, etc.); Clear(k) is obtained by calculating the variance using the Laplacian operator; Rel(k,qj) is obtained by calculating the cosine similarity between the keyframe image embedding and the assertion text embedding using CLIP; and Dup(k,K_selected) is obtained by calculating the pHash Hamming distance between the candidate frame and the selected frame. The perceptual hash distance, depth visual vector similarity, and OCR text repetition can be used to calculate the keyframes. This allows the system to retain frames with high evidentiary value while avoiding repeated reverse searches of multiple highly similar frames when selecting keyframes, thereby reducing retrieval overhead.

[0082] The system follows Sort from highest to lowest and select the top ones. Candidate keyframes:

[0083] in, Indicates a statement The selected set of reverse search keyframes As a reverse search value score, This indicates that the top K items are selected based on the sorting results.

[0084] This embodiment does not simply extract frames at time intervals, but prioritizes key frames containing landmarks, text, core objects, and event clues, thereby reducing invalid reverse searches and improving the efficiency of recalling suspected misappropriated materials.

[0085] Step S40: Extract multiple visual features of the target keyframe, compare the multiple visual features with the reference material features in the reference material library, and obtain the comparison results; It should be noted that various visual features include perceptual hash features, local visual features, deep semantic features, and OCR scene text features.

[0086] For each target keyframe The system simultaneously extracts perceptual hash features, local visual features, deep semantic features, and OCR scene text features to form a multi-granularity visual retrieval representation. The perceptual hash features are calculated using the pHash algorithm; local visual features are extracted using the Oriented FAST and Rotated BRIEF (ORB) algorithm to extract key points and their binary descriptors; deep semantic features are extracted using the CLIP (ViT-B / 32) image encoder; and OCR scene text features are extracted using PaddleOCR and converted into text representation. These four types of features are extracted in parallel and can then be used for similarity calculations at their respective granularities.

[0087] In one feasible approach, the step of comparing the multiple visual features with the features of reference materials in the reference material library to obtain the comparison result includes: comparing the perceptual hash features, the local visual features, the deep semantic features, and the image text features with the corresponding features of each reference material in the reference material library to obtain similarity scores for each reference material at multiple granularities; weighting and fusing the similarity scores at each granularity to obtain the comprehensive visual similarity of each reference material; and using the comprehensive visual similarity as the comparison result.

[0088] It should be noted that, firstly, perceptual hash feature retrieval is performed, and the system calculates keyframes. The perceptual hash h(k) of the reference material r is obtained and compared with the perceptual hash h(r) of the reference material r to obtain the hash score:

[0089] This feature is used to discover nearly duplicate footage, slightly compressed footage, slightly transcoded footage, and reused footage with minimal changes in image quality.

[0090] Then, local visual feature retrieval is performed. The system extracts the local visual features l(k) of keyframe k and matches them with the local visual features l(r) of reference material r to obtain the local visual score:

[0091] Among them, Match This indicates that a feature point matching operation is performed on the local visual features l(k) of keyframe k and the local visual features l(r) of reference material r.

[0092] This feature is used to discover similar footage after cropping, partial magnification, partial occlusion, subtitle overlay, filter processing, or aspect ratio changes. The local visual feature l(k) is extracted using the ORB algorithm: first, corner points in keyframes are detected using the FAST algorithm as keypoints, then the corresponding BRIEF binary descriptors are calculated; when matching with the local feature l(r) of the reference footage r, Hamming distance nearest neighbor matching is used combined with RANSAC for geometric consistency verification to eliminate false matches, and finally, the number of valid matching points (inliers) is used as the basis for evaluation. The value of .

[0093] Next, deep semantic feature retrieval is performed. The system inputs the keyframe k into the visual encoder to obtain the deep visual vector e(k), and calculates its deep semantic similarity with the reference material r:

[0094] This feature is used to improve the recognition capabilities of locations, organizations, event names, and event sources. Specifically, the visual encoder uses CLIP's ViT-B / 32 image encoder. CLIP is trained using contrastive learning on over 400 million image-text pairs, possessing strong cross-scene, cross-source open-domain semantic representation capabilities. Furthermore, ViT-B / 32 achieves a good balance between inference efficiency and feature representation capabilities, making it suitable for real-time retrieval scenarios with large-scale candidate material libraries in this embodiment.

[0095] Next, the system performs OCR scene text feature retrieval from keyframes. Extract the OCR scene text (text(k)) from the source material, including banners, logos, road signs, school names, organization names, event names, timestamps, etc., and match it with the OCR scene text (text(r)) of the reference material r.

[0096] in, This indicates that the text similarity calculation is performed between the OCR scene text (k) of keyframe k and the OCR scene text (r) of reference material r. Various text similarity calculation methods can be used, such as the cosine similarity calculation method.

[0097] This feature enhances the ability to identify locations, organizations, event names, and event sources, and addresses the challenge of relying solely on visual features to identify textual source clues. The OCR scene text (text(k)) is extracted using the PaddleOCR recognition engine, which demonstrates mature performance in recognizing Chinese scene text (including complex background text such as banners, road signs, school names, and organization names). It also supports integrated output of bounding box localization and text content recognition, facilitating subsequent content matching with reference OCR text.

[0098] Next, robustness enhancement retrieval is performed, considering that short video footage often undergoes mirroring, partial cropping, subtitle occlusion, compression transcoding, and aspect ratio transformation. The system then performs robustness enhancement retrieval on candidate keyframes. Generate several transformed views:

[0099] Where k is the original target keyframe. Indicates a mirrored, flipped view. This represents a cropped view of different regions. Showing a view with scale changes. This represents the view after the subtitle coverage area has been weakened or obscured. The system extracts visual features from the transformed views and performs retrieval to obtain the transformation enhancement similarity:

[0100] in to With preset weights, this step can improve the system's recall capability for secondary processed materials, partially reused materials, and subtitle-obscured materials, which is an important technical improvement that distinguishes it from ordinary image reverse search.

[0101] Finally, the overall visual recall score is calculated. The system calculates the overall visual recall score of keyframe k and reference material r:

[0102] in, to These are the weight parameters.

[0103] When the following conditions are met:

[0104] hour, The system adds reference material *r* to the candidate material set, representing the preset threshold for the overall visual recall score. (This is related to the assertion.) The set of candidate materials is denoted as:

[0105] Candidate material sources may include historical material databases, verified video databases, publicly available video indexes, news webpage indexes, school material databases, publicly available search results from platforms, and external image search results.

[0106] To avoid a large number of low-quality web pages or duplicate content in the search results, the system performs deduplication on candidate materials. Specifically, this includes near-duplicate deduplication based on image hashing, web page deduplication based on URL normalization, source merging based on publication time and publication subject, and event description deduplication based on text summary similarity.

[0107] Step S50: Based on the comparison results, obtain candidate materials from the reference material library and determine the evidentiary relationship between the candidate materials and the assertion to be verified; It should be noted that the reference material library is a massive collection of materials indexed and pre-built by the system for visual retrieval and comparison. Sources include historical material libraries, verified video libraries, publicly available video indexes, and news webpage indexes. Each material in the reference material library has multiple visual features pre-extracted to correspond to this application and associated with its metadata to support rapid similarity comparison and subsequent original source tracing. Candidate materials refer to reference materials that are visually similar to the target keyframe and are initially recalled from the reference material library by the system. Evidence relationships are the logical judgment labels ultimately established between each candidate material (after verification, it becomes evidence candidate material) and the assertion to be verified in this embodiment. Evidence relationships include supporting evidence, rebuttal evidence, partially supporting evidence, irrelevant evidence, and indeterminate evidence. Supporting evidence indicates that the candidate material can confirm the truthfulness of the current assertion; rebuttal evidence indicates that the candidate material can disprove the current assertion; partially supporting evidence indicates that the candidate material is consistent with the assertion only in some event elements; irrelevant evidence indicates that the candidate material has no substantial connection with the current assertion; and indeterminate evidence indicates that a judgment cannot be made due to insufficient information.

[0108] Step S60: Construct a dynamic evidence chain diagram based on the assertion to be verified, the shot clip, the target keyframe, the candidate material, and the evidence relationship, and output the material misappropriation risk of the assertion to be verified based on the dynamic evidence chain diagram.

[0109] It should be noted that the dynamic evidence chain graph is the core data structure that organizes fragmented review data into a structured, computable, and traceable evidence network. The dynamic evidence chain graph is a directed graph structure:

[0110] in, Represents the set of nodes in the chain of evidence. This represents the set of edges in the evidence chain. The risk of material misappropriation for the assertion to be verified can be calculated based on the dynamic evidence chain diagram.

[0111] In one feasible approach, the step of outputting the material misappropriation risk of the assertion to be verified based on the dynamic evidence chain diagram includes: obtaining source information of each candidate material based on the dynamic evidence chain diagram, and calculating a source credibility score based on the source information; calculating the supporting evidence strength and rebuttal evidence strength of each candidate material based on the dynamic evidence chain diagram; identifying time conflict paths, location conflict paths, and event conflict paths between the assertion to be verified and each candidate material based on the dynamic evidence chain diagram, and calculating conflict path risks based on the time conflict paths, location conflict paths, and event conflict paths; calculating an evidence sufficiency score based on the dynamic evidence chain diagram, wherein the evidence sufficiency score is calculated based on the number of valid evidence, the diversity of evidence sources, the coverage of authoritative sources, time consistency, and source traceability; and calculating the material misappropriation risk of the assertion to be verified based on the source credibility score, the supporting evidence strength, the rebuttal evidence strength, the conflict path risk, and the evidence sufficiency score.

[0112] It should be noted that the nodes in the evidence chain can include: the assertion node to be verified, the short video clip node, the current keyframe node, the candidate material node, the original source node, the release time node, the location node, the event description node, the source credibility node, the evidence relationship node, and the material misappropriation risk conclusion node.

[0113] The evidence chain edges can include: assertion-fragment mapping edges, fragment-keyframe inclusion edges, keyframe-candidate material similarity edges, candidate material-original source tracing edges, current assertion-candidate material subject consistency edges, current assertion-candidate material location consistency or conflict edges, current assertion-candidate material time consistency or conflict edges, current assertion-candidate material event consistency or conflict edges, candidate material-evidence relationship edges, and evidence relationship-review conclusion edges. Each edge has an edge type, edge weight, and evidence confidence level. The edge weight is jointly determined by visual similarity, event consistency, source credibility, publication time credibility, and manual review feedback, ensuring that the evidence chain diagram is not only used for display but also participates in risk calculation.

[0114] The system calculates the strength of supporting evidence and the strength of rebuttal evidence separately.

[0115] The strength of the supporting evidence is:

[0116] The strength of the rebuttal evidence is:

[0117] in, Indicates the weight of candidate materials. Indicates an indicator function. Candidate material. The source credibility score (between 0 and 1). It is the event consistency score (between 0 and 1) between the assertion and the candidate material r. Strength of evidence for misappropriation of materials (between 0 and 1). The indicator function I(·) is an indicator function: it takes the value 1 when the condition in parentheses is true, and 0 otherwise. This represents the candidate material r and the assertion to be verified q. j The evidentiary relationship between them, Support indicates supporting evidence, I ( =Support) means that the value is 1 when the candidate material r is determined to be supporting evidence, and 0 otherwise.

[0118] The sufficiency of evidence score is as follows:

[0119] in, Indicates the number of valid pieces of evidence. This indicates the diversity of sources of evidence. Indicates the coverage of authoritative sources. Indicates temporal consistency, while Trace indicates traceability of origin. to The weighting parameter is denoted as . Specifically, the number of candidate pieces of evidence that are statistically accessible, have a relevance to the assertion reaching a threshold, have complete source information, and whose evidentiary relationship is determined to be supportive, rebuttal, or partially supportive is calculated. The distribution entropy was normalized based on valid evidence from various sources, including government or institutional websites, authoritative media, original platform accounts, and ordinary web pages. The weighted proportion of evidence from authoritative sources among the valid evidence is obtained. The calculation is based on whether the time of evidence release is verifiable and whether the time sequence of the source and dissemination is reasonable. The calculation is based on whether the evidence has an accessible link, a clearly identified publishing entity, a verifiable timestamp, and a traceable upstream source. All the above indicators are normalized to the [0,1] interval.

[0120] To enable the evidence chain diagram to not only be used for display but also to participate in risk calculation, the system can further calculate the risk of conflict paths. For assertions... The set of conflict paths from the assertion node to the candidate material source node in the evidence chain graph is denoted as:

[0121] Each conflict path includes at least one of the following: time conflict path, location conflict path, event conflict path, or source conflict path.

[0122] Conflict path risk is defined as:

[0123] in, Indicates the first The weight of each conflicting path, This represents the conflict intensity of the conflict path. Specifically, ψp is the normalized weight for different conflict path types, which can be preset based on the current risk type or obtained through manual review of samples; for example, increasing the weight of time conflict paths and location conflict paths in the context of old material and new narrative. The intensity is calculated by comprehensively considering the evidence confidence, source credibility, assertion relevance, and the degree of conflict of the corresponding time, location, event, or source for each edge of the path, and then multiplying the geometric mean of the path edge weights by the degree of conflict. This avoids the problem of excessively long paths leading to unreasonably inflated scores.

[0124] Assertion-level material misappropriation risk is defined as:

[0125] in, This represents the uncertainty score of the model. This represents the risk fusion function. Model uncertainty score. The Shannon entropy method is used to calculate the entropy: The Shannon entropy is calculated and normalized for the probability distribution {P_same, P_event, P_irrelevant} output by the multimodal model. A higher entropy value indicates greater uncertainty in the model's judgment of candidate materials. Risk fusion function. Specifically, a linear weighted summation method is used, and the weight parameters are also obtained through manual verification of the training samples, i.e.:

[0126] in, to These are the weight parameters.

[0127] The system outputs a conclusion on the risk of misappropriation of materials based on the assertion-level material misappropriation risk, the strength of supporting evidence, the strength of rebuttal evidence, and the sufficiency of evidence.

[0128] If the following conditions are met:

[0129] in, A preset threshold representing the strength of rebuttal evidence.

[0130] and:

[0131] The output will then be "High risk, suspected misappropriation of materials". A preset threshold indicating the sufficiency of evidence.

[0132] If the following conditions are met:

[0133] in, This indicates the preset threshold for supporting evidence strength.

[0134] and:

[0135] The output will be "Low risk, reverse search evidence supports the current assertion".

[0136] like and If none of the values ​​reach the threshold, but there are some conflicts, then the output will be "Questionable, manual review recommended".

[0137] If the following conditions are met:

[0138] The output will be "Insufficient evidence, further verification required".

[0139] The evidence chain diagram not only serves as a report presentation structure, but also participates in assertion-level risk calculation, enabling the system to automatically calculate the risk of material misappropriation based on source tracing paths, time conflict paths, location conflict paths, and event conflict paths.

[0140] Compared to existing methods that only generate fact-checking reports or output true / false labels, this embodiment uses evidence nodes, evidence edges, and conflict paths as computable objects to participate in risk scoring, which can provide more robust review conclusions in complex scenarios such as insufficient evidence, partial conflicts, and uncertain sources.

[0141] It should be noted that manual review and evidence repatriation are also possible. After the system generates a material misappropriation review report, manual reviewers can review the following: whether the assertion to be verified is accurate, whether the mapping between the assertion and the fragment is correct, whether the reverse search keyframes have evidentiary value, whether the candidate material is similar material, whether the candidate material is the original source or an earlier source, whether the event elements of the candidate material are consistent with the current assertion, whether the determination of the evidentiary relationship is correct, and whether the final conclusion on the risk of material misappropriation is reasonable.

[0142] The result of manual review is recorded as follows:

[0143] in, Indicates the type of risk that is manually confirmed. This indicates a manually verified evidentiary relationship. This indicates the final review conclusion, and `correction_reason` indicates the reason for manual correction.

[0144] The system writes the results of manual review into the case library and uses them for subsequent model updates, threshold calibration, and reverse search ranking optimization.

[0145] Parameter updates can be represented as:

[0146] in, This represents a set of model parameters, rule weights, or thresholds. Indicates the update step size. Indicates system output Compared with manual review results The error function between them.

[0147] Manual review feedback is not simply a record of human opinions. Instead, it transforms the evidence relationships, reasons for misjudgment, reasons for omission, and final conclusions confirmed by humans into trainable samples or threshold calibration samples. These samples are used to optimize keyframe selection weights, multi-granularity retrieval weights, multimodal reordering model parameters, and evidence chain risk fusion parameters.

[0148] This embodiment acquires a video to be reviewed, performs shot segmentation and multimodal information parsing on the video to be reviewed, obtains multiple shot segments and corresponding segment information; extracts assertions to be verified from the segment information, and establishes a mapping relationship between the assertions to be verified and the shot segments; selects target keyframes from the shot segments corresponding to the assertions to be verified according to the mapping relationship; extracts multiple visual features of the target keyframes, compares the multiple visual features with the features of reference materials in the reference material library for similarity, and obtains comparison results; obtains candidate materials from the reference material library according to the comparison results, and determines the evidentiary relationship between the candidate materials and the assertions to be verified; constructs a dynamic evidence chain graph based on the assertions to be verified, the shot segments, the target keyframes, the candidate materials, and the evidentiary relationship, and outputs the material misappropriation risk of the assertions to be verified based on the dynamic evidence chain graph. This embodiment extracts assertions from the video and establishes a mapping relationship between the assertions and the shot segments. Subsequently, the assertions, rather than the video footage, can be used to determine the risk of misappropriation of video materials. This changes the review target from broad image content to verifiable and specific factual statements, improving the accuracy of video material misappropriation risk detection.

[0149] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 Before step S50, the video material misappropriation detection method further includes steps S501 to S507: Step S501: Obtain the metadata of each candidate material, the metadata including the candidate material title, candidate material summary and candidate material release time; It should be noted that this embodiment refines the steps for determining the evidentiary relationship between the candidate materials and the assertion to be verified. Multi-granularity visual retrieval can recall visually similar materials, but these may include a large number of similar but irrelevant results, different events in the same scene, similar images, or results from low-confidence sources. Therefore, this embodiment transforms the reverse search results into interpretable evidentiary relationships by acquiring candidate material metadata, constructing multimodal inputs, outputting probabilities through a classification model, calculating re-ranking scores, selecting candidate evidentiary materials, tracing the original source, and verifying the consistency of event elements.

[0150] First, multimodal input is constructed. For candidate material r, the system constructs multimodal input as follows:

[0151] in, Indicates the target keyframe. This indicates candidate content; Title indicates the short video title. This indicates a segment of subtitles. This indicates the transcription of a segment of speech. This indicates an assertion pending verification. Indicates assertion of event elements, This represents the metadata of the candidate materials.

[0152] To prevent multimodal models from making judgments based solely on visual similarity, the system inputs the current assertion event elements, candidate material release time, candidate material release entity, candidate material title summary, candidate material source type, and current short video release time into the model, enabling the model to learn fine-grained relationships such as "same source reuse", "same event but different scenes", "similar but different events", and "source cannot be determined".

[0153] Step S502: Based on the metadata, obtain the homology probability, event relevance probability, and irrelevance probability between each candidate material and the assertion to be verified; It should be noted that after inputting metadata into the multimodal model, three types of probabilities are output:

[0154] in, This indicates the probability that the candidate clip and the current keyframe are from the same source or are nearly duplicate clips; This indicates the probability that the candidate data is related to the current assertion in an event, but not necessarily of the same origin; This represents the probability that candidate material is irrelevant to the current assertion. The multimodal model can be specifically fine-tuned based on the open-source visual-language model Qwen2-VL: using Qwen2-VL's visual encoder and language model as a foundation, it takes the current keyframe, candidate material images, and text information such as titles, subtitles, and assertion event elements as input, and fine-tunes the classification head to output structured results such as homology probability, event relevance probability, and irrelevance probability. Under certain conditions, the model can also output the probability of difficult negative samples. Consistency estimate of events Used for subsequent reordering and determination of evidentiary relationships:

[0155] It should be noted that, in order for the multimodal model to distinguish between "visual similarity" and "event consistency," the system can construct multiple types of training sample sets:

[0156] in, This represents the set of positive samples reused from the same source. This represents the set of weak positive samples from different images of the same event. A set of difficult negative samples representing visually similar but inconsistent events. Represents the set of ordinary negative samples. This indicates that the source is missing or the sample set cannot be determined.

[0157] Positive samples that share the same source can include pairs of materials formed by cropping, compressing, adding subtitles, adding filters, transcoding, partially occluding, or changing the aspect ratio of the same original image or video.

[0158] Weak positive samples of the same event can include pairs of footage taken from different angles and time segments of the same news event, the same activity, or the same campus scene.

[0159] Ordinary negative samples can include pairs of materials that are visually distinct and unrelated to the events.

[0160] The sample that cannot be determined may include candidate materials that lack publication time, lack source page, have incomplete event description, or have conflicting sources.

[0161] Training samples are sourced from publicly available videos, news websites, authorized material libraries, the school's own historical material library, and manually annotated case libraries, excluding unauthorized collection of personal privacy data. Training labels primarily annotate relationships between material sources, consistency of events, and evidentiary relationships, rather than identifying natural persons or sensitive attributes.

[0162] It should be noted that this embodiment focuses on constructing difficult negative samples that are visually similar but have inconsistent events. The set of difficult negative samples is defined as follows:

[0163] in, This represents the overall visual similarity between keyframe k and candidate image r. Indicates the current assertion The event consistency score with candidate material r, Indicates the visual recall threshold. This represents the event consistency threshold. Specifically, the hard negative sample is constructed using a method that primarily involves automatic mining and secondarily involves manual review: First, multi-granularity visual retrieval is used to recall material pairs with high visual similarity S_vis(k,r). Then, based on the metadata such as the release time, location, and event description of the candidate material, it is compared with the current assertion. Material pairs with high visual similarity but significant differences in metadata (such as inconsistent location or time) are automatically marked as hard negative sample candidates. Subsequently, human reviewers sample and review the automatic mining results, and after removing mislabeled samples, they are included in the training set D_hard.

[0164] Difficult negative samples include at least the following types: similar scenarios but different events, such as two fire scenes in different regions; the same location but different times, such as activities at the same school gate in different years; the same type of activity but different subjects, such as welcoming activities at different schools; the same material incorrectly packaged with different titles; news footage similar to current short video footage but different event locations; AI-generated footage similar to real scenes but lacking a real source.

[0165] By introducing difficult negative samples, the model can learn the discrimination boundary between visually similar but inconsistent events, reducing misjudgments caused by similar but irrelevant results in ordinary reverse search.

[0166] This difficult negative sample construction mechanism is one of the key innovations that distinguishes this invention from ordinary multimodal review models. Ordinary models are usually only trained with "similar / dissimilar" or "true / false" labels, while this embodiment specifically trains the model to identify the candidate results most likely to be misjudged in material misappropriation scenarios by using visually similar but inconsistent samples.

[0167] It should be noted that sample balancing and data augmentation can also be performed. To avoid the model from being overly biased towards ordinary negative samples or homologous positive samples, the system performs class balancing on the training samples to maintain a preset proportion of homologous reused samples, weak positive samples of the same event, difficult negative samples, ordinary negative samples, and samples that cannot be judged in the training batch.

[0168] For positive samples that are reused from the same source, the system performs enhancement operations such as cropping, compression, subtitle overlay, partial occlusion, mirroring, resolution reduction, and color perturbation. For difficult negative samples, the system prioritizes material pairs with high visual similarity but at least one inconsistency in location, time, or subject. For samples that cannot be judged, the system retains samples with missing sources, missing publication times, or conflicting page content to improve the model's ability to identify uncertain evidence.

[0169] It should be noted that the multimodal reordering model training employs a joint optimization approach using classification loss, ranking loss, and event consistency constraint loss:

[0170] in, This represents the classification loss based on the relationship between candidate materials. This represents the loss from reordering candidate materials. Represents the loss due to consistency constraints on event elements. This represents the loss for difficult negative sample discrimination. and For weight parameters, The weighting parameters for the loss of difficult negative samples.

[0171] The classification loss is defined as:

[0172] in, Indicates the number of label categories. Indicates the true label, This represents the probability of the corresponding class output by the model.

[0173] The ranking loss is defined as:

[0174] in, This indicates candidate material that supports or is homologous to the current assertion. This indicates difficult negative samples or irrelevant candidate materials. Indicates the sorting interval.

[0175] The event consistency constraint loss is defined as:

[0176] in, This represents the event consistency score calculated by the event element verification module. This represents the consistency estimate of events predicted by the multimodal model.

[0177] The hard negative sample discrimination loss is defined as:

[0178] in, Labels indicating whether candidate data are hard negative samples. This represents the probability of a difficult negative sample in the model's output. This loss term enables the model to form clearer discrimination boundaries for candidate materials that are visually highly similar but have inconsistent events.

[0179] Step S503: Calculate the reordering score of each candidate material based on the homology probability, the event relevance probability, and the irrelevance probability; It should be noted that the system calculates candidate materials. Reordering score:

[0180] in, This indicates the consistency between candidate material metadata and assertion event elements. This indicates a noise penalty item caused by unreliable candidate material sources, low-quality content, missing pages, or incomplete event descriptions. This represents the probability of a difficult negative sample in the model output. to For weight parameters, MetaCons(r,qj) (metadata consistency) uses a Chinese sentence vector model (such as BGE or text2vec) to encode the candidate material page title / summary and assertion event element text respectively, and calculates the cosine similarity of their embedded vectors; Noise(r) (noise penalty) is calculated using a rule-based scoring method, specifically based on the weighted sum of three rule features: whether the candidate material source page is accessible, whether the publication time field is missing, and whether the length of the page content is lower than a preset threshold. No additional deep model is introduced to ensure the interpretability and reproducibility of this calculation.

[0181] Step S504: Select candidate evidence materials from each of the candidate materials according to the re-ranking score; It should be noted that the system selects the top scorers. One candidate material is considered as evidence for counter-search:

[0182] in, This indicates selecting the top scorer. One candidate material.

[0183] The system not only ranks images based on similarity, but also combines the current short video title, subtitles, audio, assertion event elements, and hard negative sample recognition results to determine whether candidate materials have a real event relationship with the current assertion, thereby reducing the interference of visually similar but unrelated results on the review conclusion.

[0184] Compared to ordinary visual similarity ranking, the re-ranking score in this embodiment explicitly introduces event relevance probability, metadata consistency, hard negative sample probability, and noise penalty terms, so that candidate materials that are similar but different events can be effectively downgraded, while candidate materials that constitute true evidence related to the current assertion can be preferentially retained.

[0185] Step S505: Track the original source of the candidate evidence material to obtain the source information of the candidate evidence material, the source information including the publication time, the publishing entity, and the event description; It should be noted that for the reordered candidate materials The system then further traces the original source:

[0186] in, This indicates a link to a webpage or resource. Indicates the issuing entity, Indicates the release time. Indicates the title of the candidate material. This represents a summary of the candidate materials. Indicates location clues, Indicates an event description. This refers to the image or video information of the candidate materials.

[0187] The system calculates the credibility of the candidate material sources:

[0188] in, Indicates the authority of the source. Indicates the historical reliability of the source. Indicates traceability of origin, Indicates the verifiability of the release date. This indicates the consistency of the same material across multiple sources. to These are weighted parameters. Specifically, Auth(r) (Source Authority) is scored based on a pre-defined domain authority ranking table (such as official media domains, government agency domain suffixes, etc.); Hist(r) (Historical Reliability) calculates the proportion of sources historically identified as sources of false information in the system's accumulated manual review case database; Trace(r) (Traceability) is scored based on whether the page provides verifiable original publication timestamps and author / account information, with higher scores awarded for complete fields; TimeValid(r) (Publication Time Verifiability) queries the page's historical crawl timestamps through the web archive service and cross-verifies them with the publication time marked on the page itself, with higher scores awarded for consistency; SourceConsistency(r) (Consistency) is calculated using a Chinese sentence vector model to obtain the mean cosine similarity of the title text of the same material across multiple reprinted sources. If candidate materials have multiple reprinted sources, the system selects the possible original source based on publication time, source credibility, page completeness, and similar material dissemination paths.

[0189] in, The earlier the source was published, the higher the score. Indicates the completeness of the page content. This indicates the likelihood that the source is the origin in the transmission path. to These are weighting parameters. Specifically, Propagation(s) (the likelihood of a source being the origin in the propagation path) is implemented through web archive services (such as the Internet Archive Wayback Machine): the system queries the first crawl timestamp of each reposted page for the same material in that archive service, and determines the source with the earliest first crawl time as the node with the highest likelihood of being the origin in the propagation path. This, together with Early(s) (the earlier the publication time, the higher the score), constitutes the time-series evidence upon which Path(r) is based, used to locate the original source among multiple reposted sources, rather than relying on the publication time reported by a single page. To avoid misjudging earlier reposts as the original source, the system also constructs a source time series based on the propagation path of the same material across multiple sources:

[0190] The system estimates the likelihood of a source by combining the webpage's publication time, page archiving time, the credibility of the publishing entity, and the frequency of similar images. If multiple sources have publication times close together and the source cannot be determined, the system marks candidate materials as "earlier source" or "source cannot be determined," rather than forcibly marking them as the original source. This design reduces the risk of misjudgment in source tracing and makes the chain of evidence more robust.

[0191] Step S506: Verify the consistency of the event elements of the assertion to be verified with the event elements of the candidate evidence material in terms of subject consistency, location consistency, time consistency, and result consistency, and obtain the verification results; It should be noted that the system will assert the current short video. With candidate materials The event descriptions are verified item by item. The current assertion event elements are:

[0192] The event elements of the candidate materials are:

[0193] The system calculates consistency in terms of subject, action, object, location, time, attribute, and result:

[0194]

[0195]

[0196]

[0197]

[0198]

[0199]

[0200] :

[0201] in, to These are the weight parameters.

[0202] To further identify candidate materials that are partially consistent and partially conflicting, the system calculates event conflict scores:

[0203] in, Indicates the event conflict score. to These are the weight parameters.

[0204] The strength of evidence for misappropriation of materials is defined as follows:

[0205]

[0206] in, This indicates the degree of conflict between the release time of the candidate material or the event time and the current assertion time. This indicates the degree of conflict between the candidate material location and the current assertion location. Specifically, `TimeConflict(qj,r)` calculates the absolute difference between the candidate material's release time (or the identified event time) and the assertion time, and normalizes it using a preset time window (e.g., 24 hours). When the time difference exceeds this window, the conflict degree approaches 1. `LocConflict(qj,r)` converts the assertion location and the candidate material location into latitude and longitude coordinates using a geocoding service, calculates their geographical distance, and normalizes it into a conflict score. The greater the distance, the higher the conflict score. In different review scenarios such as hot topics, campus activities, policy promotion, and AI-generated visuals, the importance of each event element varies. For example, the scenario of old material with a new narrative focuses more on time and location conflicts; the scenario of packaging material from different schools focuses more on subject and location conflicts; and the scenario of misleading policy promotion material focuses more on the result assertion and source evidence. Therefore, the system can dynamically adjust according to the risk type. to and to The weight.

[0207] Dynamic weights can be represented as:

[0208] in, Indicates the type of risk associated with misappropriation of materials. Indicates the first Feature representation of each event element and These are trainable parameters. In this way, the system can develop differentiated verification strategies for different types of material misappropriation risk.

[0209] Step S507: Determine the evidentiary relationship between the candidate evidence materials and the assertion to be verified based on the verification results. The evidentiary relationship includes supporting evidence, rebuttal evidence, partially supporting evidence, and evidence that cannot be determined.

[0210] It should be noted that the system categorizes the reverse search results into support, rebuttal, partial support, irrelevant, or indeterminate based on the relationship between the candidate material and the current assertion.

[0211] The first step is to determine the rebuttal evidence. The system will select candidate materials when the following conditions are met. Marked as rebuttal evidence:

[0212] in, To comprehensively consider visual recall scores, The threshold for high visual similarity is set, meaning only materials with sufficiently high visual similarity can be used as rebuttal evidence.

[0213] and:

[0214] in, The source credibility score for candidate materials. This is the threshold for source credibility.

[0215] And satisfy any of the following conditions: and

[0216] in, The release time of the candidate materials. This refers to the current posting time of the short video. The time of the event stated in the assertion to be verified.

[0217] or:

[0218] in, Locations identified from the candidate data. The location stated in the assertion to be verified.

[0219] or:

[0220] in, Event descriptions of candidate materials, This is an event description for an assertion that needs to be verified.

[0221] Alternatively, candidate materials may be flagged by authoritative sources as other events or historical events.

[0222] In special circumstances, when the evidence of misappropriation of candidate materials reaches a certain threshold, it can also be considered as rebuttal evidence:

[0223] in, Indicating the strength of evidence of misappropriation of materials, This is a preset intensity threshold.

[0224] at this time:

[0225] in, This indicates that the system will match the candidate material r with the assertion to be verified. The evidentiary relationship between them is determined to be "rebuttal evidence".

[0226] It should be noted that supporting evidence is determined as follows: the system will select candidate materials when the following conditions are met. Marked as supporting evidence:

[0227] and:

[0228] and:

[0229] in, The event consistency score, This is the event consistency threshold.

[0230] Furthermore, the candidate material sources can support the time, location, subject, and event description of the current assertion.

[0231] at this time:

[0232] in, This indicates that the system will match the candidate material r with the assertion to be verified. The evidentiary relationship between them is determined to be "supporting evidence".

[0233] It should be noted that partial support or partial rebuttal is determined as follows: when candidate footage is highly similar to the current keyframe, but only partially identical in event elements, the system marks it as partially support or partially rebuttal. For example, the subject and scene are the same, but the time is different; the location is the same, but the event outcome is different; the source of the footage is the same, but the title contains exaggeration or incorrect attribution.

[0234] at this time:

[0235] in, This indicates that the system will match the candidate material r with the assertion to be verified. The evidentiary relationship between them was determined to be "partially supporting evidence".

[0236] It should be noted that irrelevant evidence is determined when: the visual similarity of candidate materials is insufficient, or the multimodal model outputs a high probability of irrelevance.

[0237] in, The threshold for visual similarity is low.

[0238] or:

[0239] in, For irrelevant probabilities, This is the irrelevant probability threshold.

[0240] but:

[0241] in, This indicates that the system will match the candidate material r with the assertion to be verified. The evidentiary relationship between them was determined to be "irrelevant evidence".

[0242] It should be noted that the inability to determine evidence is defined as follows: when the source of candidate materials is unverifiable, the publication time is missing, the page content is incomplete, the event description is insufficient, or multiple sources contradict each other.

[0243] in, This represents the minimum threshold for source credibility.

[0244] Or the set of original source information for candidate material r If key traceability information is missing, then:

[0245] in, This indicates that the system will match the candidate material r with the assertion to be verified. The evidentiary relationship between them was determined to be "indeterminate evidence".

[0246] The technical advantage of this step is that the system doesn't simply output "similar images found," but further determines the evidentiary relationship between similar materials and the assertion of the current short video, enabling the reverse search results to directly serve the credible review conclusion. To improve the interpretability of manual review, the system outputs the reasons for the relationship determination along with the evidentiary relationship, such as "the posting time is earlier than the current video and the location is different," "the posting entity is an official account of another university," "the candidate material only supports the activity type but not the current location," and "the candidate source lacks a posting time and cannot be determined."

[0247] This embodiment acquires metadata for each candidate material, including the candidate material title, candidate material summary, and candidate material publication time; based on the metadata, it obtains the homology probability, event relevance probability, and irrelevance probability between each candidate material and the assertion to be verified; it calculates the re-ranking score for each candidate material based on the homology probability, event relevance probability, and irrelevance probability; it selects evidence candidate materials from the candidate materials based on the re-ranking score; it traces the original source of the evidence candidate materials to obtain the source information of the evidence candidate materials, including the publication time, publishing entity, and event description; it verifies the consistency of the event elements of the assertion to be verified with the event elements of the evidence candidate materials in terms of subject consistency, location consistency, time consistency, and result consistency to obtain the verification result; it determines the evidentiary relationship between the evidence candidate materials and the assertion to be verified based on the verification result, and the evidentiary relationship includes supporting evidence, rebuttal evidence, partially supporting evidence, and evidence that cannot be determined. This embodiment further determines the evidentiary relationship of the visual search results as supporting, refuting, partially supporting, or indeterminate, directly serving the review conclusion on the risk of material misappropriation, and improving the interpretability of the review results.

[0248] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the video material misappropriation detection method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0249] This application also provides a device for detecting the misappropriation of video footage; please refer to [reference needed]. Figure 3 The video material misappropriation detection device includes: The acquisition module 10 is used to acquire the video to be reviewed, perform shot segmentation and multimodal information parsing on the video to be reviewed, and obtain multiple shot segments and the segment information corresponding to the shot segments; Module 20 is used to extract assertions to be verified from the segment information and establish a mapping relationship between the assertions to be verified and the shot segments; The selection module 30 is used to select target keyframes from the shot segments corresponding to the assertion to be verified according to the mapping relationship; The comparison module 40 is used to extract multiple visual features of the target keyframe, compare the multiple visual features with the reference material features in the reference material library, and obtain the comparison result. The determination module 50 is used to obtain candidate materials from the reference material library based on the comparison results, and determine the evidentiary relationship between the candidate materials and the assertion to be verified. The construction module 60 is used to construct a dynamic evidence chain diagram based on the assertion to be verified, the shot clip, the target keyframe, the candidate material, and the evidence relationship, and output the material misappropriation risk of the assertion to be verified based on the dynamic evidence chain diagram.

[0250] This application acquires a video to be reviewed, performs shot segmentation and multimodal information parsing on the video to be reviewed, obtains multiple shot segments and corresponding segment information; extracts assertions to be verified from the segment information, and establishes a mapping relationship between the assertions to be verified and the shot segments; selects target keyframes from the shot segments corresponding to the assertions to be verified according to the mapping relationship; extracts multiple visual features of the target keyframes, compares the multiple visual features with the features of reference materials in a reference material library, and obtains a comparison result; obtains candidate materials from the reference material library according to the comparison result, and determines the evidentiary relationship between the candidate materials and the assertions to be verified; constructs a dynamic evidence chain graph based on the assertions to be verified, the shot segments, the target keyframes, the candidate materials, and the evidentiary relationship, and outputs the material misappropriation risk of the assertions to be verified based on the dynamic evidence chain graph. This application extracts assertions from videos and establishes a mapping relationship between assertions and video clips. Subsequently, assertions, rather than video footage, can be used to determine the risk of misappropriation of video materials. This changes the focus of the review from broad video content to verifiable and specific factual statements, improving the accuracy of detecting the risk of misappropriation of video materials.

[0251] In one embodiment, the establishing module 20 is further configured to extract candidate assertions from the title, published text, on-screen text, speech-to-text, and visual semantic description in the fragment information. The candidate assertions contain a set of event elements, including the event subject, action, object, location, time, attribute, and result. The module is also configured to calculate the verification value score of the candidate assertions, which is calculated based on the entity density, time specificity, location specificity, public communication relevance, risk propagation intensity, and assertion strength of the candidate assertions. Candidate assertions with verification value scores higher than a preset verification threshold are identified as assertions to be verified. The module is further configured to calculate the semantic correlation between each assertion to be verified and each shot fragment. Finally, the module is configured to establish a mapping relationship between the assertions to be verified and the shot fragments based on the semantic correlation.

[0252] In one embodiment, the selection module 30 is further configured to determine candidate keyframes from the shot segments corresponding to the assertion to be verified according to the mapping relationship; calculate a reverse search value score based on whether the candidate keyframe is located at a scene change location, the amount of text information in the candidate keyframe, the salience of the location clues in the candidate keyframe, the salience of the core object in the candidate keyframe, the image clarity of the candidate keyframe, the semantic correlation between the candidate keyframe and the assertion to be verified, and the repetition between the candidate keyframe and the selected candidate keyframe; and select a target keyframe from the candidate keyframes according to the reverse search value score.

[0253] In one embodiment, the comparison module 40 is further configured to compare the perceptual hash feature, the local visual feature, the deep semantic feature and the image text feature with the corresponding features of each reference material in the reference material library to obtain the similarity score of each reference material at multiple granularities; to weight and fuse the similarity scores at each granularity to obtain the comprehensive visual similarity of each reference material, and to use the comprehensive visual similarity as the comparison result.

[0254] In one embodiment, the determination module 50 is further configured to acquire metadata of each candidate material, the metadata including candidate material title, candidate material summary, and candidate material publication time; obtain the homology probability, event relevance probability, and irrelevance probability between each candidate material and the assertion to be verified based on the metadata; calculate the reordering score of each candidate material based on the homology probability, the event relevance probability, and the irrelevance probability; select evidence candidate materials from each candidate material based on the reordering score; trace the original source of the evidence candidate materials to obtain the source information of the evidence candidate materials, the source information including publication time, publishing entity, and event description; verify the consistency of the event elements of the assertion to be verified with the event elements of the evidence candidate materials in terms of subject consistency, location consistency, time consistency, and result consistency to obtain the verification result; determine the evidentiary relationship between the evidence candidate materials and the assertion to be verified based on the verification result, the evidentiary relationship including supporting evidence, rebuttal evidence, partially supporting evidence, and evidence that cannot be determined.

[0255] In one embodiment, the construction module 60 is further configured to obtain source information of each candidate material according to the dynamic evidence chain diagram, calculate a source credibility score based on the source information; calculate the supporting evidence strength and rebuttal evidence strength of each candidate material according to the dynamic evidence chain diagram; identify time conflict paths, location conflict paths, and event conflict paths between the assertion to be verified and each candidate material according to the dynamic evidence chain diagram, and calculate conflict path risks based on the time conflict paths, location conflict paths, and event conflict paths; calculate an evidence sufficiency score based on the dynamic evidence chain diagram, the evidence sufficiency score being calculated based on the number of valid evidence, the diversity of evidence sources, the coverage of authoritative sources, time consistency, and source traceability; and calculate the material misappropriation risk of the assertion to be verified based on the source credibility score, the supporting evidence strength, the rebuttal evidence strength, the conflict path risk, and the evidence sufficiency score.

[0256] In one embodiment, the selection module 30 is further configured to calculate the visual material reuse score, text visual event mismatch score, location-time conflict score, and title / subtitle inducement score between the assertion to be verified and the corresponding shot segment; and to determine whether to perform the step of selecting the target keyframe from the shot segment corresponding to the assertion to be verified based on the visual material reuse score, the text visual event mismatch score, the location-time conflict score, and the title / subtitle inducement score.

[0257] The video material misappropriation detection device provided in this application, employing the video material misappropriation detection method described in the above embodiments, can solve the technical problem of how to improve the accuracy of video material misappropriation risk detection. Compared with the prior art, the beneficial effects of the video material misappropriation detection device provided in this application are the same as those of the video material misappropriation detection method provided in the above embodiments, and other technical features in the video material misappropriation detection device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0258] This application provides a video material misappropriation detection device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the video material misappropriation detection method in the above embodiment 1.

[0259] The following is for reference. Figure 4The diagram illustrates a structural schematic suitable for implementing a video material misappropriation detection device according to embodiments of this application. The video material misappropriation detection device in embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), tablets, PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 4 The video footage misappropriation detection device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0260] like Figure 4 As shown, the video material misappropriation detection device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in ROM (Read Only Memory) 1002 or a program loaded from storage device 1003 into RAM (Random Access Memory) 1004. RAM 1004 also stores various programs and data required for the operation of the video material misappropriation detection device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via bus 1005. Input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touch screens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. The communication device 1009 allows the video footage misappropriation detection equipment to communicate wirelessly or wiredly with other devices to exchange data. While the figures show video footage misappropriation detection equipment with various systems, it should be understood that implementing or possessing all of the systems shown is not required. More or fewer systems may be implemented alternatively.

[0261] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0262] The video material misappropriation detection device provided in this application, employing the video material misappropriation detection method described in the above embodiments, can solve the technical problem of how to improve the accuracy of video material misappropriation risk detection. Compared with the prior art, the beneficial effects of the video material misappropriation detection device provided in this application are the same as those of the video material misappropriation detection method provided in the above embodiments, and other technical features in this video material misappropriation detection device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0263] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0264] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0265] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the video material misappropriation detection method in the above embodiments.

[0266] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0267] The aforementioned computer-readable storage medium may be included in the video material misappropriation detection device; or it may exist independently and not be assembled into the video material misappropriation detection device.

[0268] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by a video material misappropriation detection device, the video material misappropriation detection device performs the following: acquires a video to be reviewed; performs shot segmentation and multimodal information parsing on the video to be reviewed to obtain multiple shot segments and corresponding segment information; extracts assertions to be verified from the segment information and establishes a mapping relationship between the assertions to be verified and the shot segments; selects target keyframes from the shot segments corresponding to the assertions to be verified based on the mapping relationship; extracts multiple visual features of the target keyframes and performs similarity comparisons between the multiple visual features and reference material features in a reference material library to obtain comparison results; obtains candidate materials from the reference material library based on the comparison results and determines the evidentiary relationship between the candidate materials and the assertions to be verified; constructs a dynamic evidence chain diagram based on the assertions to be verified, the shot segments, the target keyframes, the candidate materials, and the evidentiary relationship, and outputs the material misappropriation risk of the assertions to be verified based on the dynamic evidence chain diagram.

[0269] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0270] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0271] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0272] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described video material misappropriation detection method, thereby solving the technical problem of how to improve the accuracy of video material misappropriation risk detection. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the video material misappropriation detection method provided in the above embodiments, and will not be repeated here.

[0273] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the video material misappropriation detection method described above.

[0274] The computer program product provided in this application can solve the technical problem of how to improve the accuracy of video material misappropriation risk detection. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the video material misappropriation detection method provided in the above embodiments, and will not be repeated here.

[0275] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for detecting the misappropriation of video materials, characterized in that, The method includes: The video to be reviewed is obtained, and the video to be reviewed is segmented into shots and parsed with multimodal information to obtain multiple shot segments and the corresponding segment information of the shot segments; Extract the assertions to be verified from the fragment information and establish the mapping relationship between the assertions to be verified and the shot fragments; Select target keyframes from the shot segments corresponding to the assertions to be verified based on the mapping relationship; Extract multiple visual features from the target keyframe, compare the multiple visual features with the reference material features in the reference material library, and obtain the comparison results; Based on the comparison results, candidate materials are obtained from the reference material library, and the evidentiary relationship between the candidate materials and the assertion to be verified is determined. A dynamic evidence chain diagram is constructed based on the assertion to be verified, the shot clip, the target keyframe, the candidate material, and the evidence relationship, and the material misappropriation risk of the assertion to be verified is output based on the dynamic evidence chain diagram.

2. The method as described in claim 1, characterized in that, The step of extracting the assertions to be verified from the segment information and establishing the mapping relationship between the assertions to be verified and the shot segments includes: Candidate assertions are extracted from the title, publication text, on-screen text, speech-to-text, and visual semantic description in the fragment information. The candidate assertions contain a set of event elements, which includes the event subject, action, object, location, time, attribute, and result. The verification value score of the candidate assertion is calculated based on the entity density, temporal specificity, location specificity, public communication relevance, risk propagation intensity, and assertion strength of the candidate assertion. Candidate assertions whose verification value scores are higher than a preset verification threshold are identified as assertions to be verified. Calculate the semantic correlation between each of the assertions to be verified and each of the shot segments; A mapping relationship between the assertion to be verified and the shot segment is established based on the semantic relevance.

3. The method as described in claim 1, characterized in that, The step of selecting target keyframes from the shot segments corresponding to the assertion to be verified based on the mapping relationship includes: Candidate keyframes are determined from the shot segments corresponding to the assertions to be verified based on the mapping relationship; The reverse search value score is calculated based on whether the candidate keyframe is located at a scene change location, the amount of text information in the candidate keyframe, the salience of the location clues in the candidate keyframe, the salience of the core object in the candidate keyframe, the image clarity of the candidate keyframe, the semantic relevance between the candidate keyframe and the assertion to be verified, and the duplication between the candidate keyframe and the selected candidate keyframe. The target keyframe is selected from the candidate keyframes according to the reverse search value score.

4. The method as described in claim 1, characterized in that, The various visual features include perceptual hash features, local visual features, deep semantic features, and on-screen text features. The step of comparing the various visual features with the features of reference materials in the reference material library to obtain the comparison results includes: The perceptual hash feature, the local visual feature, the deep semantic feature, and the image text feature are compared with the corresponding features of each reference material in the reference material library to obtain the similarity score of each reference material at multiple granularities. The similarity scores at each granularity are weighted and fused to obtain the comprehensive visual similarity of each reference material, and the comprehensive visual similarity is used as the comparison result.

5. The method as described in claim 1, characterized in that, The step of determining the evidentiary relationship between the candidate material and the assertion to be verified includes: Obtain the metadata of each candidate material, including the candidate material title, candidate material summary, and candidate material release time; Based on the metadata, the homology probability, event relevance probability, and irrelevance probability between each candidate material and the assertion to be verified are obtained; The re-ranking score of each candidate material is calculated based on the homology probability, the event relevance probability, and the irrelevance probability. Evidence candidate materials are selected from each of the candidate materials based on the re-ranking scores; The original source of the candidate evidence is traced to obtain the source information of the candidate evidence, which includes the publication time, the publishing entity, and the event description. The event elements of the assertion to be verified are compared with the event elements of the candidate evidence materials in terms of subject consistency, location consistency, time consistency, and result consistency to obtain the verification results. The evidentiary relationship between the candidate evidence materials and the assertion to be verified is determined based on the verification results. The evidentiary relationship includes supporting evidence, rebuttal evidence, partially supporting evidence, and evidence that cannot be determined.

6. The method as described in claim 1, characterized in that, The step of outputting the material misappropriation risk of the assertion to be verified based on the dynamic evidence chain diagram includes: The source information of each candidate material is obtained based on the dynamic evidence chain diagram, and the source credibility score is calculated based on the source information. Calculate the supporting evidence strength and rebuttal evidence strength of each candidate material based on the dynamic evidence chain diagram; The dynamic evidence chain diagram identifies the time conflict path, location conflict path, and event conflict path between the assertion to be verified and each of the candidate materials, and calculates the conflict path risk based on the time conflict path, the location conflict path, and the event conflict path. The sufficiency score is calculated based on the dynamic evidence chain diagram. The sufficiency score is calculated based on the number of valid evidence, the diversity of evidence sources, the coverage of authoritative sources, the consistency of time, and the traceability of sources. The risk of misappropriation of materials for the assertion to be verified is calculated based on the source credibility score, the strength of supporting evidence, the strength of rebuttal evidence, the risk of conflict path, and the sufficiency of evidence score.

7. The method as described in claim 1, characterized in that, Before the step of selecting the target keyframe from the shot segment corresponding to the assertion to be verified according to the mapping relationship, the method further includes: Calculate the scores for visual material reuse, text visual event mismatch, location and time conflict, and title / subtitle manipulation between the assertion to be verified and the corresponding shot segment; Based on the visual material reuse score, the text visual event mismatch score, the location-time conflict score, and the title / subtitle inducement score, determine whether to execute the step of selecting the target keyframe from the shot segment corresponding to the assertion to be verified according to the mapping relationship.

8. A device for detecting the misappropriation of video materials, characterized in that, The device includes: The acquisition module is used to acquire the video to be reviewed, perform shot segmentation and multimodal information parsing on the video to be reviewed, and obtain multiple shot segments and the segment information corresponding to the shot segments; A module is established to extract assertions to be verified from the segment information and establish a mapping relationship between the assertions to be verified and the shot segments; The selection module is used to select target keyframes from the shot segments corresponding to the assertion to be verified according to the mapping relationship; The comparison module is used to extract multiple visual features of the target keyframe, compare the multiple visual features with the reference material features in the reference material library, and obtain the comparison results. The determination module is used to obtain candidate materials from the reference material library based on the comparison results, and determine the evidentiary relationship between the candidate materials and the assertion to be verified. The construction module is used to construct a dynamic evidence chain diagram based on the assertion to be verified, the shot clip, the target keyframe, the candidate material, and the evidence relationship, and output the material misappropriation risk of the assertion to be verified based on the dynamic evidence chain diagram.

9. A video footage misappropriation detection device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the video material misappropriation detection method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the video material misappropriation detection method as described in any one of claims 1 to 7.

Images