Video positioning method and device based on text data and electronic equipment

Abstract

The application discloses a video positioning method and device based on text data and electronic equipment, and belongs to the technical field of computers; the method comprises the following steps: acquiring video data, dividing the video data into basic time units, and determining evidence information to determine a multi-source evidence sequence; acquiring text data, and generating a paragraph event graph and event complexity; the paragraph event graph comprises event nodes and event relationships; according to the event complexity corresponding to the event nodes, a candidate window is determined, and the target basic time unit corresponding to the candidate window is determined; according to the multi-source evidence sequence of the target basic time unit, candidate segment features are determined, and node features are generated according to the event nodes; cross-modal matching is performed according to the node features and the candidate segment features, and analysis is performed in combination with the evidence information and the paragraph event graph to determine the video positioning result corresponding to the event nodes in the text data; the scheme can more accurately position the video segment corresponding to the text.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a video positioning method, apparatus, and electronic device based on text data. Background Technology

[0002] In scenarios involving the management of massive amounts of long video footage and intelligent retrieval, users often want to directly locate the start and end times of multiple consecutive or related events in a video using a natural language description. For example, in a news program, a user might input "the host introduces the background of the event, then the reporter interviews the crowd, and finally the scene cuts to the event itself." The system needs to find three semantically related but visually different event segments in sequence, ensuring that the chronological order of the location results matches the text description. This type of task is often called Video Paragraph Grounding (VPG) or dense event video grounding, which essentially maps a paragraph query containing K sentences or events to K time segments in a video.

[0003] Existing video temporal sentence localization methods typically employ candidate segment scanning, cross-modal feature fusion, and temporal boundary regression to achieve localization. While these methods are effective in single-sentence localization scenarios, their core remains the matching of single sentences with single segments, making it difficult to fully express the structured relationships between multiple events within a paragraph, resulting in inaccurate video localization results. Summary of the Invention

[0004] To address the problems existing in the prior art, the present invention aims to provide a video localization method, apparatus, and electronic device based on text data. This method can extract multiple pieces of evidence information from video data and events from text data. By considering the temporal sequence and logical relationship of the events, it matches and constrains the events with the video, thereby enabling more accurate localization of text and video.

[0005] In a first aspect, this application provides a video localization method based on text data. The method includes: acquiring video data, dividing the video data into basic time units, and determining a multi-source evidence sequence, wherein the multi-source evidence sequence is determined based on evidence information of the basic time units, and the evidence information includes at least one of the following: visual features, speech recognition text, subtitle text, facial figures, shot boundary type, and scene labels; acquiring text data and generating a paragraph event graph and event complexity, wherein the paragraph event graph includes event nodes and event relationships; and the event nodes include at least one of the following: subject, action, object, attribute, location, time clue, and keywords. The event relationships include temporal or logical relationships between event nodes; the event complexity is determined based on at least one of the following indicators: sentence length, number of entities, action duration, number of modifiers, intensity of time words, and degree of relationship of event relationships; based on the event complexity corresponding to the event node, candidate windows are determined, and the target basic time unit corresponding to the candidate window is determined; based on the multi-source evidence sequence of the target basic time unit, candidate segment features are determined, and node features are generated based on the event nodes; cross-modal matching is performed based on the node features and candidate segment features, and combined with evidence information and paragraph event graphs for analysis, to determine the video localization result corresponding to the event nodes in the text data.

[0006] Optionally, the step of performing cross-modal matching based on node features and candidate segment features, and combining evidence information and paragraph event graphs for analysis to determine the video localization result corresponding to the event nodes in the text data, includes: performing cross-modal matching based on node features and candidate segment features to determine first information; performing evidence consistency analysis based on evidence information and event nodes in the paragraph event graph to determine second information; performing time prior analysis based on time-related information in the evidence information, paragraph order in the text data, and event relationships in the paragraph event graph to determine third information; and determining the association between event nodes and candidate segments based on the first, second, and third information to determine the video localization result.

[0007] Optionally, determining the association between event nodes and candidate segments based on the first, second, and third information to determine the video localization result includes: determining the association between event nodes and candidate segments based on the first, second, and third information; obtaining the event relationships between multiple event nodes, and determining the relationship loss of multiple association relationships according to the event relationships to determine the video localization result.

[0008] Optionally, the event relationships include: those corresponding to temporal relationships such as sequential relationships, overlapping relationships, and inclusion relationships, as well as causal relationships corresponding to logical relationships; the determination of the relationship loss of multiple associations according to the event relationships includes: analyzing the sequential relationship loss between the first segment corresponding to the first event and the second segment corresponding to the second event based on the sequential relationship between the first event and the second event; analyzing the overlapping relationship loss between the first segment corresponding to the first event and the second segment corresponding to the second event based on the overlapping relationship between the first event and the second event; analyzing the inclusion relationship loss between the first segment corresponding to the first event and the second segment corresponding to the second event based on the inclusion relationship between the first event and the second event; analyzing the time interval between the first segment corresponding to the first event and the second segment corresponding to the second event based on the causal relationship between the first event and the second event to determine the causal relationship loss; and determining the relationship loss of the association based on the sequential relationship loss, overlapping relationship loss, inclusion relationship loss, and causal relationship loss.

[0009] Optionally, determining candidate windows based on the event complexity corresponding to the event node includes: analyzing the maximum candidate window length, minimum candidate window length, minimum search step size, and event complexity of the event node to determine the basic window length and step size of the event node; determining multiple target windows according to the step size and basic window length, and filtering the target windows according to event relationships to determine candidate windows for video data.

[0010] Optionally, the method further includes: obtaining video positioning results corresponding to event nodes, the video positioning results including start-point information and end-point information; determining a local verification window based on the start-point information, end-point information, and boundary expansion radius; obtaining the basic time units of the local verification window, and determining local evidence features based on the multi-source evidence sequence of each basic time unit; analyzing the local evidence features and event nodes to determine correction information, and correcting the video positioning results based on the correction information.

[0011] Optionally, the step of analyzing local evidence features and event nodes to determine correction information includes: analyzing local evidence features and event nodes to determine the starting point correction amount, the ending point correction amount, and the boundary confidence level; the boundary confidence level is determined based on the matching score of local evidence features and event nodes, evidence consistency, boundary stability, and the consistency of event relationships in the segment event graph; when the boundary confidence level exceeds a preset threshold, the starting point correction amount and the ending point correction amount are used as correction information to correct the video positioning result, and an evidence chain is generated based on the evidence information and the segment event graph; when the boundary confidence level does not exceed the preset threshold, a boundary review operation is triggered, the boundary review operation including at least one of the following operations: triggering candidate window reordering to re-determine candidate windows, expanding the boundary expansion radius to re-determine local verification windows, and applying for manual review.

[0012] Optionally, the steps of performing cross-modal matching based on node features and candidate segment features, and combining evidence information and paragraph event graphs for analysis, are completed through a cross-modal analysis model. The method also includes a training step for the cross-modal analysis model: acquiring training data and annotation results, wherein the training data includes training text and training video, and the annotation results include the correspondence between training text and training video; inputting the training data into the cross-modal analysis model, determining the prediction results, and determining the joint loss based on the prediction results and annotation results, wherein the joint loss includes at least one of the following losses: matching loss between event nodes and video segments, boundary correction information loss, intersection-union ratio loss of prediction results and annotation results, event relationship loss of paragraph event graph, evidence consistency loss, and confidence loss of prediction results; and adjusting the cross-modal analysis model based on the joint loss.

[0013] Secondly, this application provides a video positioning device based on text data. The device includes: a video data acquisition module, used to acquire video data, divide the video data into basic time units, and determine a multi-source evidence sequence. The multi-source evidence sequence is determined based on evidence information from the basic time units, and the evidence information includes at least one of the following: visual features, speech recognition text, subtitle text, facial figures, shot boundary type, and scene labels; and a text data acquisition module, used to acquire text data and generate a paragraph event graph and event complexity. The paragraph event graph includes event nodes and event relationships. The event nodes include at least one of the following: subject, action, object, attribute, location, time clue, and keyword; the event relationships... The system includes the temporal or logical relationships between event nodes; the event complexity is determined based on at least one of the following indicators: sentence length, number of entities, action duration, number of modifiers, intensity of time words, and degree of relationship of event relationships; the candidate window acquisition module is used to determine candidate windows based on the event complexity corresponding to the event nodes, and to determine the target basic time unit corresponding to the candidate windows; the feature information generation module is used to determine candidate segment features based on the multi-source evidence sequence of the target basic time unit, and to generate node features based on the event nodes; the localization result acquisition module is used to perform cross-modal matching based on node features and candidate segment features, and to analyze the evidence information and paragraph event graph to determine the video localization result corresponding to the event nodes in the text data.

[0014] Thirdly, this application provides an electronic device, comprising: a memory and at least one processor; the memory being used to store computer execution instructions; the at least one processor being used to execute the computer execution instructions stored in the memory, causing the at least one processor to perform the method as described in the first aspect.

[0015] Fourthly, this application provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0016] This application can be applied to text and video matching scenarios, specifically text and video matching in news scenarios. This solution can acquire video data, divide the video data into basic time units, and then analyze the visual features, speech recognition text, subtitle text, facial figures, shot boundary types, and scene tags of the basic time units to form evidence information. After feature extraction, a multi-source evidence sequence is obtained. This solution can also acquire text data and generate a paragraph event graph, which includes event nodes and event relationships. The event nodes include at least one of the following: subject, action, object, attribute, location, timeline, and keywords. The event relationships include the temporal or logical relationships between event nodes. Some solutions use fixed-length windows with fixed-step sliding to capture video data. This solution analyzes the event complexity in text data to dynamically generate candidate windows, enabling more accurate capture of videos suitable for the event and improving video localization accuracy. Event complexity is determined based on at least one of the following metrics: sentence length, number of entities, action duration, number of modifiers, intensity of time words, and relational degree of event relationships. After determining the event complexity, candidate windows are dynamically determined, and the corresponding target base time units are identified. Candidate segment features are determined based on the multi-source evidence sequence of the target base time units, and node features are generated based on event nodes. Then, cross-modal matching is performed based on node features and candidate segment features, and evidence consistency analysis and constraint analysis are conducted in conjunction with evidence information and paragraph event graphs to determine the video localization result corresponding to the event nodes in the text data. This solution matches node features and candidate segment features, analyzes the evidence consistency of evidence information matching event nodes to videos, and combines constraint analysis in event relationships to determine the video segment corresponding to the event. Furthermore, this solution can not only analyze the correlation between a single event and a single video, but also combine the relationships between multiple events to analyze whether multiple correlations meet temporal constraints, so as to more accurately locate the video. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0018] Figure 1 This is a flowchart illustrating a video localization method based on text data according to an embodiment of this application;

[0019] Figure 2This is a schematic diagram illustrating the steps of a video localization method based on text data according to an embodiment of this application;

[0020] Figure 3 This is a schematic diagram of the structure of a video positioning device based on text data according to an embodiment of this application. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms, while “a plurality” refers to two or more, and other quantifiers are similarly understood. It should be further understood that the word “comprising” as used in this application’s specification means the presence of the stated feature, integer, step, operation, element, and / or component, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connection or wireless coupling. The word “and / or” as used herein describes the relationship between related objects, indicating that three relationships can exist; for example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character “ / ” generally indicates that the preceding and following related objects are in an “or” relationship.

[0023] This application can be applied to text and video matching scenarios, specifically in news scenarios. In broadcast television, news production, and long-form video material libraries, videos contain not only visual information but also rich evidence such as speech-recognized text, subtitles, personal identities, program names, camera cuts, station logos, locations, and reporter names. This solution can utilize this evidence to achieve more accurate location. This solution can first process the video data and text data separately. Specifically, the video data can be divided into basic time units, and then the visual features, speech-recognized text, subtitles, facial features, camera boundary types, and scene tags of the basic time units are analyzed to form evidence information. After feature extraction, a multi-source evidence sequence is obtained. Text data can be acquired and a paragraph event graph can be generated. The paragraph event graph includes event nodes and event relationships; the event nodes include at least one of the following: subject, action, object, attribute, location, timeline, and keywords; the event relationships include the temporal or logical relationships between event nodes. After processing the video and text data, candidate windows can be determined based on event complexity, and the corresponding target time units for each candidate window can be identified. Candidate segment features are then determined based on the multi-source evidence sequence of the target time units, and node features are generated based on event nodes. Subsequently, cross-modal matching is performed based on the node features and candidate segment features, and evidence consistency and constraint analysis are conducted in conjunction with evidence information and segment event graphs to determine the video localization result corresponding to the event nodes in the text data. Furthermore, this scheme can dynamically expand and verify the start and end points in the video localization result to determine more accurate video segments.

[0024] It should be noted that the process of data processing, feature extraction, feature matching, and localization boundary correction and verification of video data and text data in this solution can be completed by a cross-modal analysis model, or multiple different models can be used to complete the above process. For example, a large language model can be used to process text data, and a cross-modal model can be used to complete cross-modal matching and boundary correction.

[0025] like Figure 1 As shown, the method of this scheme is implemented according to the process of "construction of multi-source video evidence, generation of segment event graph, generation of adaptive candidate window for query, cross-modal matching of event graph constraints, boundary verification of multimodal large model, and joint training and output". The following is a detailed explanation with reference to the attached figures and formulas:

[0026] This solution implements the query process based on text paragraphs and videos. Let the input video be V, and the total video duration be T; the input paragraph query... ,in, Let represent the i-th sentence or event description, and K represent the number of events contained in the paragraph. The model needs to define each event... Output a predicted video time segment: ,in: Indicates the predicted start time; This indicates the prediction end time. The training phase is given the ground truth labeled time boundaries: ,in: Indicates the actual start time; Indicates the actual end time.

[0027] The process of constructing a multi-source video evidence sequence includes: first, preprocessing the video V; the system in this scheme divides the video into N basic time units according to shot boundaries, fixed time intervals, or semantic change points. ,in, For each basic time unit, visual features, Automatic Speech Recognition (ASR), Optical Character Recognition (OCR) subtitles, facial and human figures, shot type (shot boundary type), scene labels, and visual descriptions are extracted.

[0028] For the nth time unit, visual evidence is obtained respectively. Voice evidence Subtitle evidence Evidence of the person Camera evidence and scene evidence By aligning timestamps and projecting features, evidence from different modalities is unified into the same dimensional space, forming:

[0029] Multiple sources of evidence indicate: ;

[0030] in, , , , , , Let be the projection matrix for each modality, b be the bias term, and LN(·) denote layer normalization. To avoid the overall representation failing due to the absence of a certain modality, this scheme also introduces a modality availability mask. When a certain mode is missing, that mode does not participate in the fusion. The improved fusion formula is as follows: Formula 1:

[0031] Formula 1

[0032] Where m∈{vis, asr, ocr, face, shot, scene}, corresponding to visual evidence, audio evidence, caption evidence, character evidence, shot evidence, and scene evidence, respectively. Through the above processing, a multi-source video evidence sequence can be obtained. This sequence preserves information such as video content, audio and text, screen captions, character identities, and camera boundaries, providing a basis for subsequent cross-modal matching and boundary verification.

[0033] The process of generating paragraph event graphs includes: This approach does not directly split paragraphs into isolated sentences, but instead uses a multimodal large model or a language large model to perform structured parsing of paragraph P, generating paragraph event graphs. .in, Let R represent the set of event nodes, and let R represent the set of event relationships. Each event node... Includes main body ,action Object ,property ,Place Timeline and keywords ,express: Event Relationship Used to describe the temporal or logical relationship between the i-th event and the j-th event, including but not limited to before, after, overlap, include, parallel, causal, and transition, specifically representing: ;in, .

[0034] This solution allows you to input event nodes into a text encoder or a multimodal large model encoder to obtain event node features. To enable the model to adaptively generate candidate windows based on event difficulty, this scheme further calculates event complexity. Event complexity is determined by sentence length, number of entities, action duration, number of modifiers, strength of time terms, and relation degree. Node features Determined based on event nodes, and represented as node features. ;in, Represents text or large model encoding functions.

[0035] The event complexity can be expressed as: ;in, Indicates sentence length; Indicates the number of entities; This indicates the degree of relation of an event node in the event graph; σ(·) represents the prior of action continuity; σ(·) represents the normalized sigmoid function. The larger the value, the more complex the event semantics, the longer the duration, or the stronger the relational constraints.

[0036] The process of generating adaptive candidate windows includes: some methods typically use fixed short-term, medium-term, and long-term windows, while this solution adjusts based on event complexity. Candidate windows are dynamically generated based on the relationship between events. For the i-th event, the basic window length is first calculated. and step length ; indicates as: ; ;in, and These represent the minimum and maximum candidate window lengths, respectively. This represents the minimum search step size. This represents the step size scaling factor. When the event complexity is low, a shorter and more refined candidate window is generated; when the event complexity is high, a longer window is generated to cover the entire event. Furthermore, considering the constraints of event relationships on the candidate window, if... If the starting point of the j-th event candidate window is no earlier than the center or end point of the i-th event candidate window; if If the candidate window containing the event is within the candidate window containing the event, then the candidate window set for the i-th event is constructed. Specifically, it is represented as shown in Formula 2 below:

[0037] Formula 2

[0038] in, Indicates the candidate center point, Represents an adaptive scale set of events. This method indicates whether the candidate window satisfies the event graph relationship constraints. Compared with fixed multi-scale windows, this method can reduce invalid candidates based on query semantics and event relationships, thereby improving the efficiency of long video processing.

[0039] The process of cross-modal matching of event nodes and candidate fragments includes: for each candidate window The evidence from the underlying time units covered is pooled or attention-aggregated to obtain candidate fragment representations. , or candidate fragment features, are specifically represented as: .

[0040] The matching score between the i-th event node and the j-th candidate fragment It is composed of semantic similarity, evidence consistency, and temporal prior, and is specifically represented as shown in Formula 3 below:

[0041] Formula 3

[0042] in, For coefficients, and For the projection matrix, Indicates the consistency score of multi-source evidence. The temporal prior score is derived from time words, paragraph order, and event relationships. The evidence consistency score can be obtained by weighting ASR hit, OCR hit, face hit, visual description similarity, and shot boundary consistency.

[0043] To achieve joint localization of multiple events, this invention introduces matching variables. , indicating whether the i-th event selects the j-th candidate window. The matching objective is to maximize the cross-modal matching score while minimizing the event graph relationship violation cost, as shown in Formula 4 below:

[0044] Formula 4

[0045] in, To mitigate the cost of relationship violations, if two events have a before relationship, the center of the i-th event segment must be earlier than the center of the j-th event segment; if they have an overlap relationship, the two segments must have sufficient overlap; if they have an include relationship, the included segment must be located inside the including segment.

[0046] The event graph relation constraints include: assuming a prediction segment Its center point is The fragment length is For different relationships in the event graph, we define differentiable relationship losses respectively.

[0047] The order of priority. If Then we want event i to occur before event j, that is... Where γ is the time interval boundary, the smoothing loss is constructed using the Softplus activation function, and the sequential loss is shown in Equation 5:

[0048] Formula 5

[0049] when When the time interval is not positive, the loss approaches 0; when the prediction time sequence is reversed, the loss increases with the degree of reversal.

[0050] Overlapping relationship. If Therefore, it is desirable for the two event segments to have some temporal overlap. Let the intersection-union ratio of the predicted segments be... The overlap threshold is The overlap loss is shown in Formula 6:

[0051] Formula 6

[0052] Containment relationship. If If event j is contained within event i, then it is required that... and The loss function related to inclusion is defined as shown in Formula 7:

[0053] Formula 7

[0054] Cause and effect or transition relationship. If or This typically requires that event i occurs earlier than event j, and that the time interval between them does not exceed the upper limit. To avoid the model incorrectly connecting two segments that are too far apart, the causal relationship loss is defined as shown in Equation 8:

[0055] Formula 8

[0056] Therefore, the total event graph relationship loss is shown in Equation 9:

[0057] Formula 9

[0058] Through the above process, the structured relationships in the paragraph text are transformed into optimizable time constraints, enabling the model to focus not only on whether each event matches correctly, but also on whether the overall narrative order among multiple events is reasonable.

[0059] The process of boundary verification and correction for multimodal large models includes: obtaining coarse localization results through cross-modal matching. Subsequently, this scheme further constructs a local verification window near the boundary based on the boundary expansion radius. Where r is the boundary expansion radius, keyframes, ASR text, OCR subtitles, shot boundaries, and character evidence are extracted within this window, and event node descriptions are added. When local evidence is input together with the multimodal large model boundary verification module, it is represented as follows: The boundary verification module outputs the starting point correction amount. Endpoint correction amount Consistency score of evidence and boundary confidence The corrected segment is: ; Confidence is determined by cross-modal matching score, consistency of evidence, boundary stability, and consistency of event graph relationships. ;when Below the preset threshold When this happens, the system triggers a reordering of candidate windows, an expansion of the search scope, or manual review; when If the value is higher than the threshold, the location result and evidence chain will be output directly.

[0060] During the training phase, the joint training loss function is determined based on multiple losses, employing a combination of matching loss, boundary regression loss, intersection-over-union (IoU) loss, event graph relationship loss, evidence consistency loss, and confidence loss for joint optimization. Specifically:

[0061] Matching loss; for the i-th event, its true matching window is Using the cross-entropy loss in the form of contrastive learning, the matching loss is calculated as shown in Equation 10:

[0062] Formula 10

[0063] Boundary regression loss (or boundary correction information loss); let the true boundary correction be... , The predicted correction amount is , Using Smooth L1 loss, the boundary regression loss is calculated as shown in Equation 11:

[0064] Formula 11

[0065] Intersection over Union (IoU) loss; to directly improve the degree of boundary overlap, the IoU loss is defined and calculated using Formula 12:

[0066] Formula 12

[0067] Event graph relationship loss Based on the above definition of event graph relationship constraints, it is used to constrain the temporal relationships between multiple prediction segments.

[0068] Evidence consistency loss: If the i-th event has ASR, OCR, facial, or visual evidence in the real segment, the model is required to also provide the corresponding evidence probability in the predicted segment. Correspondingly, the calculation method for the loss of evidence consistency is shown in Formula 13:

[0069] Formula 13

[0070] Confidence loss (or confidence loss of prediction result); based on whether the IoU between the predicted segment and the true segment exceeds a threshold. Construct confidence labels Correspondingly, the confidence loss is calculated as shown in Formula 14:

[0071] Formula 14

[0072] Based on the above, the final calculation method of the joint loss function is shown in Formula 15:

[0073] Formula 15

[0074] in, to The loss weights can be adjusted based on the size of the training set and business metrics. This joint loss optimizes both the accuracy of single-event localization and the consistency of multi-event sequence, boundary stability, and interpretability of evidence.

[0075] Based on the above embodiments, this application also provides a video positioning method based on text data, such as... Figure 2 As shown, the method includes:

[0076] Step 102: Acquire video data and divide it into basic time units, then determine a multi-source evidence sequence. The multi-source evidence sequence is determined based on the evidence information of the basic time units. The evidence information includes at least one of the following: visual features, speech recognition text, subtitle text, facial figures, shot boundary type, and scene labels. Step 104: Acquire text data and generate a paragraph event graph and event complexity. The paragraph event graph includes event nodes and event relationships. The event nodes include at least one of the following: subject, action, object, attribute, location, time clue, and keywords. The event relationships include the temporal or logical relationships between event nodes. The event complexity is determined based on at least one of the following indicators: sentence length, number of entities, action duration, number of modifiers, time word intensity, and relation degree of the event relationship. Step 106: Based on the event complexity corresponding to the event nodes, determine candidate windows and the target basic time units corresponding to the candidate windows. Step 108: Based on the multi-source evidence sequence of the target basic time units, determine the candidate segment features and generate node features based on the event nodes. Step 110: Perform cross-modal matching based on node features and candidate segment features, and analyze the evidence information and paragraph event graph to determine the video localization result corresponding to the event nodes in the text data.

[0077] This application can be applied to text and video matching scenarios, specifically text and video matching in news scenarios. This solution can acquire video data, divide it into basic time units, analyze the evidence information of the basic time units, and extract features to obtain a multi-source evidence sequence. This solution can also acquire text data and generate a paragraph event graph, which includes event nodes and event relationships. This solution can analyze the event complexity in the text data to dynamically generate candidate windows, thereby more accurately capturing suitable video clips and improving the accuracy of video positioning. After determining the event complexity, candidate windows can be dynamically determined based on the event complexity, and the target basic time unit corresponding to the candidate window can be determined. Candidate segment features are determined based on the multi-source evidence sequence of the target basic time unit, and node features are generated based on the event nodes. Then, cross-modal matching is performed based on the node features and candidate segment features, and evidence consistency analysis and constraint analysis are combined with evidence information and the paragraph event graph to determine the video positioning result corresponding to the event nodes in the text data. This scheme can match node features and candidate segment features, analyze the consistency of evidence information of event nodes hitting videos, and combine constraint analysis in event relationships to determine the video segment corresponding to the event.

[0078] This scheme can perform cross-modal matching based on node features and candidate segment features to determine a matching score, and determine the matched evidence information based on node features to determine evidence consistency; based on time words, paragraph order, and event relationships in the evidence information, it determines time prior information, and analyzes information from multiple dimensions to determine the video localization result; specifically, as an optional embodiment, the step of performing cross-modal matching based on node features and candidate segment features, and combining evidence information and paragraph event graphs to determine the video localization result corresponding to the event nodes in the text data includes: performing cross-modal matching based on node features and candidate segment features to determine first information; performing evidence consistency analysis based on the evidence information and event nodes in the paragraph event graph to determine second information; performing time prior analysis based on time-related information in the evidence information, paragraph order in the text data, and event relationships in the paragraph event graph to determine third information; and determining the association between the event nodes and candidate segments based on the first, second, and third information to determine the video localization result.

[0079] This solution can also focus on the event relationships of multiple events and determine the correlation loss according to the event relationships and the events of the corresponding video segments, so as to constrain the video localization results based on the correlation loss. Specifically, as an optional embodiment, the step of determining the correlation between event nodes and candidate segments based on the first information, the second information and the third information to determine the video localization results includes: determining the correlation between event nodes and candidate segments based on the first information, the second information and the third information; obtaining the event relationships between multiple event nodes, and determining the relationship loss of multiple correlations according to the event relationships to determine the video localization results.

[0080] This solution defines event relationships as temporal constraints, using these constraints to determine the relationship loss. Specifically, as an optional embodiment, the event relationships include: those corresponding to temporal relationships (sequential, overlapping, and inclusion relationships) and those corresponding to logical relationships (causal relationships). Determining the relationship loss of multiple relationships based on event relationships includes: analyzing the sequential relationship loss between the first segment corresponding to the first event and the second segment corresponding to the second event based on the sequential relationship between the first and second events; analyzing the overlapping relationship loss between the first segment corresponding to the first event and the second segment corresponding to the second event based on the overlapping relationship between the first and second events; analyzing the inclusion relationship loss between the first segment corresponding to the first event and the second segment corresponding to the second event based on the inclusion relationship between the first and second events; analyzing the time interval between the first segment corresponding to the first event and the second segment corresponding to the second event based on the causal relationship between the first and second events to determine the causal relationship loss; and determining the relationship loss of the association based on the sequential relationship loss, overlapping relationship loss, inclusion relationship loss, and causal relationship loss. The process of determining the relationship loss can refer to the content of the event graph relationship constraints in the above embodiment.

[0081] This solution can generate corresponding windows for events of varying complexity and further filter these windows based on event relationships to determine a window set. Specifically, as an optional embodiment, determining candidate windows based on the event complexity of the event node includes: analyzing the maximum candidate window length, minimum candidate window length, minimum search step size, and event complexity of the event node to determine the basic window length and step size of the event node; determining multiple target windows according to the step size and basic window length; and filtering the target windows based on event relationships to determine candidate windows for the video data. Through further filtering of the target windows, this solution can reduce irrelevant data analysis. The target window filtering process can refer to the processing procedure corresponding to Formula 2 above.

[0082] This solution can also modify and verify the window. Specifically, as an optional embodiment, the method further includes: obtaining the video positioning result corresponding to the event node, the video positioning result including start-point information and end-point information; determining a local verification window based on the start-point information, end-point information, and boundary expansion radius; obtaining the basic time unit of the local verification window, and determining local evidence features based on the multi-source evidence sequence of each basic time unit; analyzing the local evidence features and the event node to determine correction information, and modifying the video positioning result based on the correction information. This solution can expand the start and end points and determine the local verification window, so as to analyze the local evidence features of the local verification window with the event node. In the process of analyzing the local evidence features and the event node, this solution can also combine the candidate segment features corresponding to the original video positioning result with the local evidence features, and then analyze them with the event node to determine the correction information; alternatively, it can perform feature consistency analysis and evidence information consistency analysis on the candidate segment features and the local evidence features to determine the correction information.

[0083] After analyzing local evidence features and event nodes, the correction amount and confidence level can be determined. If the confidence level meets the requirements, correction is performed. If the confidence level is low, candidate windows can be regenerated, local verification windows can be adjusted, or manual review can be introduced. Specifically, as an optional embodiment, the step of analyzing local evidence features and event nodes to determine correction information includes: analyzing local evidence features and event nodes to determine the starting point correction amount, the ending point correction amount, and the boundary confidence level; the boundary confidence level is determined based on the matching score of local evidence features and event nodes, evidence consistency, boundary stability, and the consistency of event relationships in the segment event graph; when the boundary confidence level exceeds a preset threshold, the starting point correction amount and the ending point correction amount are used as correction information to correct the video positioning result, and an evidence chain is generated based on the evidence information and the segment event graph; when the boundary confidence level does not exceed the preset threshold, a boundary review operation is triggered, which includes at least one of the following operations: triggering candidate window reordering to redetermine candidate windows, expanding the boundary expansion radius to redetermine local verification windows, and applying for manual review. The process of boundary correction and verification can refer to the process of boundary verification and correction for multimodal large models in the above embodiments.

[0084] In this scheme, the process of cross-modal matching based on node features and candidate segments can be completed using a trained model. Correspondingly, the model can employ multiple losses for joint loss analysis to optimize it. Specifically, as an optional embodiment, the steps of performing cross-modal matching based on node features and candidate segment features, combined with evidence information and paragraph event graph analysis, are completed through a cross-modal analysis model. The method also includes a training step for the cross-modal analysis model: acquiring training data and annotation results, where the training data includes training text and training video, and the annotation results include the correspondence between training text and training video; inputting the training data into the cross-modal analysis model, determining the prediction result, and determining the joint loss based on the prediction result and annotation result, where the joint loss includes at least one of the following losses: event node and video segment matching loss, boundary correction information loss, intersection-union ratio (IUU) loss between prediction and annotation results, event relationship loss of the paragraph event graph, evidence consistency loss, and confidence loss of the prediction result; and adjusting the cross-modal analysis model based on the joint loss. The model optimization process can refer to the relevant content of the joint training loss function in the above embodiment. In addition, the cross-modal analysis model of this application can also perform feature extraction, boundary correction and verification processes for video data and text data, so as to optimize the corresponding model parameters together.

[0085] Based on the above embodiments, this application also provides a video positioning device based on text data, such as... Figure 3As shown, the device includes: a video data acquisition module 202, used to acquire video data, divide the video data into basic time units, and determine a multi-source evidence sequence. The multi-source evidence sequence is determined based on the evidence information of the basic time units. The evidence information includes at least one of the following: visual features, speech recognition text, subtitle text, facial figures, shot boundary type, and scene labels. A text data acquisition module 204, used to acquire text data and generate a paragraph event graph and event complexity. The paragraph event graph includes event nodes and event relationships. The event nodes include at least one of the following: subject, action, object, attribute, location, time clue, and keyword. The event relationships include the temporal or logical relationships between event nodes. The event complexity is determined based on at least one of the following indicators: sentence length, number of entities, action duration, number of modifiers, intensity of time words, and degree of relationship of event relationships. A candidate window acquisition module 206, used to determine candidate windows based on the event complexity corresponding to the event nodes, and determine the target basic time unit corresponding to the candidate window. The feature information generation module 208 is used to determine candidate segment features based on the multi-source evidence sequence of the target's basic time unit, and to generate node features based on event nodes. The localization result acquisition module 210 is used to perform cross-modal matching based on node features and candidate segment features, and to analyze the evidence information and paragraph event graph to determine the video localization result corresponding to the event nodes in the text data.

[0086] The data processing flow in this embodiment is similar to that in the above method embodiment. For specific implementation details, please refer to the specific implementation details in the above method embodiment. These details will not be repeated here.

[0087] This application can be applied to text and video matching scenarios, specifically text and video matching in news scenarios. This solution can acquire video data, divide it into basic time units, analyze the evidence information of the basic time units, and extract features to obtain a multi-source evidence sequence. This solution can also acquire text data and generate a paragraph event graph, which includes event nodes and event relationships. This solution can analyze the event complexity in the text data to dynamically generate candidate windows, thereby more accurately capturing suitable video clips and improving the accuracy of video positioning. After determining the event complexity, candidate windows can be dynamically determined based on the event complexity, and the target basic time unit corresponding to the candidate window can be determined. Candidate segment features are determined based on the multi-source evidence sequence of the target basic time unit, and node features are generated based on the event nodes. Then, cross-modal matching is performed based on the node features and candidate segment features, and evidence consistency analysis and constraint analysis are combined with evidence information and the paragraph event graph to determine the video positioning result corresponding to the event nodes in the text data. This scheme can match node features and candidate segment features, analyze the consistency of evidence information of event nodes hitting videos, and combine constraint analysis in event relationships to determine the video segment corresponding to the event.

[0088] Based on the above embodiments, this application also provides an electronic device, including: a memory and at least one processor; the memory is used to store computer execution instructions; the at least one processor is used to execute the computer execution instructions stored in the memory, causing the at least one processor to perform the method as described in the above embodiments.

[0089] This invention also provides a computer-readable storage medium storing a computer program. When executed by a processor, this computer program implements the various processes of the above-described data processing method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0090] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0091] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0092] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0093] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0094] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0095] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0096] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0097] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0098] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0099] The above are merely embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A video localization method based on text data, characterized in that, The method includes: Acquire video data, divide the video data into basic time units, and determine a multi-source evidence sequence. The multi-source evidence sequence is determined based on the evidence information of the basic time units. The evidence information includes at least one of the following: visual features, speech recognition text, subtitle text, facial figures, shot boundary type, and scene label. Acquire text data and generate a paragraph event graph and event complexity. The paragraph event graph includes event nodes and event relationships. The event nodes include at least one of the following: subject, action, object, attribute, location, timeline, and keywords. The event relationships include the temporal or logical relationships between event nodes. The event complexity is determined based on at least one of the following indicators: sentence length, number of entities, action duration, number of modifiers, and degree of relation between event relationships. Based on the event complexity corresponding to the event node, candidate windows are determined, and the target base time unit corresponding to the candidate window is determined; the higher the event complexity, the longer the length of the candidate window; the determination of candidate windows based on the event complexity corresponding to the event node includes: The basic window length and step size of the event node are determined by analyzing the maximum candidate window length, minimum candidate window length, minimum search step size, and event complexity of the event node. Based on the step size and the base window length, multiple target windows are determined, and the target windows are filtered according to the event relationship to determine the candidate windows for video data; Candidate segment features are determined based on the multi-source evidence sequence of the target's basic time unit, and node features are generated based on event nodes; Cross-modal matching is performed based on node features and candidate segment features, and analysis is conducted in conjunction with evidence information and paragraph event graphs to determine the video localization results corresponding to event nodes in the text data; including: Cross-modal matching is performed based on node features and candidate segment features to determine the first piece of information; Based on the evidence information and the event nodes in the paragraph event diagram, an evidence consistency analysis is performed to determine the second piece of information; Based on time-related information in the evidence, paragraph order in the text data, and event relationships in the paragraph event diagram, a priori time analysis is performed to determine the third information; Based on the first, second, and third information, the correlation between event nodes and candidate segments is determined to determine the video localization result; including: Based on the first, second, and third information, determine the relationship between the event nodes and candidate fragments; Obtain the event relationships between multiple event nodes, and determine the relationship loss of multiple associations according to the event relationships in order to determine the video localization result.

2. The method according to claim 1, characterized in that, The event relationships include: those corresponding to temporal relationships such as sequential relationships, overlapping relationships, and inclusion relationships, as well as causal relationships corresponding to logical relationships; The relationship loss for determining multiple associations based on event relationships includes: Based on the chronological relationship between the first event and the second event, analyze the loss of the chronological relationship between the first segment corresponding to the first event and the second segment corresponding to the second event; Based on the overlap relationship between the first event and the second event, analyze the overlap loss between the first segment corresponding to the first event and the second segment corresponding to the second event; Based on the inclusion relationship between the first event and the second event, analyze the inclusion relationship loss between the first segment corresponding to the first event and the second segment corresponding to the second event; Based on the causal relationship between the first event and the second event, analyze the time interval between the first segment corresponding to the first event and the second segment corresponding to the second event to determine the loss of causal relationship; The relationship loss of the association is determined based on the loss of sequential relationship, the loss of overlapping relationship, the loss of inclusion relationship, and the loss of causal relationship.

3. The method according to claim 2, characterized in that, The method further includes: Obtain the video positioning result corresponding to the event node, wherein the video positioning result includes start point information and end point information; The local verification window is determined based on the starting point information, the ending point information, and the boundary expansion radius; Obtain the basic time unit of the local verification window, and determine the local evidence features based on the multi-source evidence sequence of each basic time unit; Based on the analysis of local evidence features and event nodes, correction information is determined, and the video positioning results are corrected according to the correction information.

4. The method according to claim 3, characterized in that, The process of analyzing local evidence features and event nodes to determine correction information includes: Based on the analysis of local evidence features and event nodes, the starting point correction, ending point correction, and boundary confidence are determined; the boundary confidence is determined based on the matching score of local evidence features and event nodes, evidence consistency, boundary stability, and the consistency of event relationships in the paragraph event graph. When the boundary confidence exceeds the preset threshold, the video localization result is corrected based on the starting point correction amount and the ending point correction amount, and an evidence chain is generated based on the evidence information and the segment event graph. When the boundary confidence does not exceed a preset threshold, a boundary review operation is triggered. The boundary review operation includes at least one of the following operations: triggering candidate window reordering to redetermine candidate windows, expanding the boundary expansion radius to redetermine local verification windows, and requesting manual review.

5. The method according to claim 4, characterized in that, The steps of performing cross-modal matching based on node features and candidate segment features, and combining evidence information and paragraph event graphs for analysis, are completed through a cross-modal analysis model. The method also includes a training step for the cross-modal analysis model: Acquire training data and annotation results, wherein the training data includes training text and training video, and the annotation results include the correspondence between training text and training video; The training data is input into the cross-modal analysis model to determine the prediction results. Based on the prediction results and the annotation results, the joint loss is determined. The joint loss includes at least one of the following losses: event node and video segment matching loss, boundary correction information loss, intersection-union ratio loss of prediction results and annotation results, event relationship loss of segment event graph, evidence consistency loss, and confidence loss of prediction results. The cross-modal analysis model is adjusted based on the joint loss.

6. A video positioning device based on text data, used to implement the video positioning method based on text data according to any one of claims 1-5, characterized in that, The device includes: The video data acquisition module is used to acquire video data, divide the video data into basic time units, and determine a multi-source evidence sequence. The multi-source evidence sequence is determined based on the evidence information of the basic time units. The evidence information includes at least one of the following: visual features, speech recognition text, subtitle text, facial figures, shot boundary type, and scene label. A text data acquisition module is used to acquire text data and generate a paragraph event graph and event complexity. The paragraph event graph includes event nodes and event relationships. The event nodes include at least one of the following: subject, action, object, attribute, location, timeline, and keywords. The event relationships include the temporal or logical relationships between event nodes. The event complexity is determined based on at least one of the following indicators: sentence length, number of entities, action duration, number of modifiers, and degree of relation between event relationships. The candidate window acquisition module is used to determine the candidate window based on the event complexity corresponding to the event node, and to determine the target basic time unit corresponding to the candidate window. The feature information generation module is used to determine the features of candidate segments based on the multi-source evidence sequence of the target's basic time unit, and to generate node features based on event nodes. The localization result acquisition module is used to perform cross-modal matching based on node features and candidate segment features, and to analyze the results by combining evidence information and paragraph event graphs to determine the video localization results corresponding to event nodes in the text data.

7. An electronic device, characterized in that, include: Memory and at least one processor; The memory is used to store computer-executed instructions; The at least one processor is configured to execute computer execution instructions stored in the memory, such that the at least one processor performs the method as described in any one of claims 1-5.

Images