Video pre-training data generation method and device, electronic equipment, and storage medium

Abstract

This application discloses a method, apparatus, electronic device, and storage medium for generating video pre-training data. The method includes acquiring video data to be processed and its metadata, and unifying the timeline and identifier; segmenting the video data into multiple video segments, constructing keyframe sequences and corresponding multimodal features for each video segment, extracting events from each video segment to obtain a candidate event set, constructing an event chain structure, upsampling inference paths, and generating matching inference questions and answers for each inference path; performing consistency verification and quality assessment on the generated inference questions and answers; and encapsulating the inference questions and answers that pass consistency verification and quality assessment, along with their corresponding video segments, event chain structures, and evidence location information, into video pre-training data. This scheme can automatically generate high-coverage, high-confidence inference question and answer training samples, reducing manual annotation costs and improving model inference capabilities.

Description

Technical Field

[0001] This application relates to the field of video data processing and automatic construction of training data, and in particular to a method, device, electronic device, and storage medium for generating video pre-training data based on event chain extraction and reasoning question answering. Background Technology

[0002] In recent years, video understanding and generation models have been widely used in tasks such as retrieval, question answering, summarizing, and agent decision-making. To improve the model's generality and inference capabilities, large-scale pre-training data is usually required. Compared to static images, videos contain temporal dynamics, cross-modal information, and long-range dependencies, making the construction of high-quality training data more difficult.

[0003] Existing video pre-training data mainly consists of frame-level descriptions, segment-level caption alignment, and action classification labels. While this type of data can improve the model's perceptual capabilities, it has limited impact on improving abilities such as causal inference, multi-hop reasoning, and temporal logical consistency. On the other hand, manually constructing reasoning question-answering data requires annotators to understand the video context and complete evidence localization, which is costly and difficult to guarantee consistency.

[0004] Some automated methods attempt to generate question-and-answer sessions using rule templates or single models, but they often lack explicit modeling of event relationships, easily producing answers inconsistent with the video content or questions with significant semantic ambiguity, leading to the accumulation of training noise. Especially in long video scenarios, when events occur across segments and have causal chains, the lack of an event chain structure will significantly reduce the controllability and verifiability of question-and-answer generation.

[0005] Therefore, there is an urgent need for a pre-training data generation scheme that can automatically extract events from videos, construct event chain structures, controllably generate reasoning questions and answers based on event chains, and simultaneously filter highly reliable samples through an interpretable verification mechanism. Summary of the Invention

[0006] The purpose of this application is to provide a video pre-training data generation method, apparatus, electronic device, and storage medium to solve the problems of existing automated question-answering generation that cannot guarantee reasoning consistency, evidence traceability, and long-range causal coverage, thereby generating high-quality reasoning question-answering data that can be used for pre-training at a lower manual cost.

[0007] According to a first aspect of the embodiments of this application, a method for generating video pre-training data is provided, comprising:

[0008] Acquire the video data to be processed and its metadata, and standardize the metadata into a unified timeline and a unified identifier;

[0009] The video data is segmented over time to obtain multiple video segments, and keyframe sequences and corresponding multimodal features are constructed for each video segment.

[0010] Based on the multimodal features, events are extracted from each video segment to obtain a set of candidate events containing event trigger words, participating entities, action attributes, and spatiotemporal positioning.

[0011] Event chain construction is performed on the candidate event set to infer the temporal and causal relationships between events and obtain the event chain structure.

[0012] The reasoning paths are sampled on the event chain structure, and a matching reasoning question and answer is generated for each reasoning path, wherein the reasoning question and answer includes at least the question, the answer and the location information of the evidence fragment;

[0013] Perform consistency verification and quality assessment on the generated reasoning questions and answers;

[0014] The reasoning questions and answers that have passed consistency verification and quality assessment, along with their corresponding video clips, event chain structures, and evidence location information, are encapsulated into video pre-training data.

[0015] According to a second aspect of the embodiments of this application, a video pre-training data generation apparatus is provided, comprising:

[0016] The data access module is used to acquire the video data to be processed and its metadata, and to standardize the metadata into a unified timeline and a unified identifier.

[0017] The segmentation module is used to segment the video data in time to obtain multiple video segments, and to construct keyframe sequences and corresponding multimodal features for each video segment;

[0018] The event extraction module is used to extract events from each video segment based on the multimodal features, and obtain a set of candidate events containing event trigger words, participating entities, action attributes and spatiotemporal positioning.

[0019] The event chain construction module is used to construct event chains for the candidate event set, infer the temporal and causal relationships between events, and obtain the event chain structure.

[0020] The path sampling module is used to sample the reasoning path on the event chain structure and generate a matching reasoning question and answer for each reasoning path, wherein the reasoning question and answer includes at least the question, the answer and the location information of the evidence fragment;

[0021] The question-and-answer generation module is used to perform consistency verification and quality assessment on the generated reasoning questions and answers;

[0022] The verification and storage module is used to encapsulate the reasoning questions and answers that have passed consistency verification and quality assessment, along with their corresponding video clips, event chain structures, and evidence location information, into video pre-training data.

[0023] According to a third aspect of the embodiments of this application, an electronic device is provided, comprising: one or more processors; a memory for storing one or more programs; wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in the first aspect.

[0024] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided, having stored thereon computer instructions that, when executed by a processor, implement the method described in the first aspect.

[0025] Compared with the prior art, this application has at least the following beneficial effects:

[0026] By extracting events and constructing event chains to explicitly represent key events in videos and their temporal and causal relationships, reasoning question-answering generation is given structured constraints, significantly reducing semantic drift and answer illusion.

[0027] By sampling multi-hop inference paths on the event chain to generate multi-type inference questions and answers, it is possible to cover long-range dependencies and cross-segment relationships, thereby improving the gain of pre-training data on inference capabilities.

[0028] Traceable consistency verification is achieved through symbolic execution and cross-validation with a video question-answering model, and a quality scoring mechanism is used to filter noisy samples, thereby improving the credibility and usability of the dataset.

[0029] The generated samples contain both evidence location information and optional explanatory text, which facilitates interpretable training and evaluation, and supports subsequent alignment learning and preference optimization. Attached Figure Description

[0030] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0031] Figure 1 A flowchart illustrating a video pre-training data generation method provided in an embodiment of this application.

[0032] Figure 2 This is a structural block diagram of a video pre-training data generation device provided in an embodiment of this application.

[0033] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0034] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0035] like Figure 1 As shown, the video pre-training data generation method provided in this embodiment may include steps S1 to S7. For ease of description, the following uses a single video sample as an example; in practical applications, video sets can be processed in batches.

[0036] S1: Obtain the video data to be processed and its metadata, and standardize the metadata into a unified timeline and a unified identifier;

[0037] Specifically, taking a segment of personnel activity from an indoor surveillance video as an example scenario, the original video file and associated metadata such as subtitles, speech-to-text transcription, recording time, frame rate, resolution, and source identifier are obtained. This metadata is then standardized, mapping timestamps from different sources to the same timeline to obtain a time index t in seconds. A globally unique identifier id is assigned to each video, segment, and event, and a system is established... The mapping relationship between the source identifier and the file identifier ensures that subsequent processing is based on a unified timeline and a unified identifier.

[0038] To reduce the impact of noise, deduplication, sentence segmentation, and error correction can be performed on the subtitles and transcribed text, and silence segment detection can be performed on the audio to remove invalid intervals.

[0039] S2: Divide the video data into time segments to obtain multiple video segments, and construct keyframe sequences and corresponding multimodal features for each video segment;

[0040] Specifically, taking a segment of personnel activity in an indoor surveillance video as an example scenario, shot boundary detection is performed on the video to obtain a shot sequence. Further, a scene consistency score is calculated based on the visual similarity and text topic consistency of adjacent shots, and multiple adjacent shots are merged into a semantically complete segment. The segment length can be constrained within a first threshold range, for example, 5 to 60 seconds, to ensure the stability of event extraction and question-and-answer generation, where the first threshold is the video segment duration range.

[0041] According to the preset keyframe selection rules, multiple keyframes are selected from each video segment and arranged in chronological order to obtain a keyframe sequence;

[0042] Specifically, in the aforementioned indoor surveillance video scenario, keyframes can be obtained through uniform sampling, or they can be jointly determined by the center frame of the shot, the peak motion frame, and the frame containing significant subtitle changes. For each keyframe, its timestamp (corresponding to time index t on a unified timeline) and its frame index in the original video are recorded.

[0043] Visual encoding is performed on the keyframe sequence to obtain a frame-level visual feature sequence, and audio encoding and text encoding are performed on the corresponding audio signal and subtitle text respectively to obtain audio feature sequence and text feature sequence;

[0044] Specifically, based on the timestamps of the keyframe sequence, visual encoding is performed on the keyframes to obtain a visual feature sequence. (Where the subscript t represents the time index t corresponding to the feature), align the time windows by extracting them from the corresponding audio signal and subtitle text, and perform audio encoding and text encoding respectively to obtain the audio feature sequence. With text feature sequences .

[0045] The frame-level visual feature sequence, audio feature sequence, and text feature sequence are aligned and fused according to timestamps to obtain the multimodal feature sequence corresponding to the video segment;

[0046] Specifically, the three types of features are aligned and fused based on timestamps to obtain a fragment-level multimodal feature sequence:

[0047]

[0048] in For the fused multimodal features at time index t, , , It can be a learnable linear transformation matrix or a fixed weight.

[0049] S3: Based on the multimodal features, perform event extraction on each video segment to obtain a candidate event set containing event trigger words, participating entities, action attributes, and spatiotemporal positioning; this step includes the following sub-steps:

[0050] S31: Based on a preset event pattern library, trigger word identification and argument filling are performed on the multimodal feature sequence to obtain candidate events, wherein the event arguments include at least one of the following: subject entity, object entity, location entity, and time entity, and the identified trigger words are used as event trigger words of the candidate events;

[0051] Specifically, taking a segment of personnel activity from an indoor surveillance video as an example scenario, the event type is predicted for each candidate event. and the start and end times of the event And determine the action attributes of candidate events based on action features. Determining the spatiotemporal location information of candidate events based on visual features In one embodiment, the action attribute It can be predicted by aggregating the temporal change features within the event time window, for example:

[0052]

[0053] in This represents action features constructed from visual characteristics. This is a function for predicting action attributes. Spatiotemporal positioning information. It should include at least the start and end times of the event and spatial area information, which can be obtained by target detection or visual reference localization.

[0054] Therefore, the candidate event can be represented as:

[0055] in As a trigger word, It is the set of arguments and the corresponding set of participating entities. For action attributes, and Together they constitute the spatiotemporal location information of the candidate events.

[0056] S32: Predict the event type and start and end times of the event for each candidate event, determine the action attributes of the candidate event based on action features, determine the spatiotemporal location information of the candidate event based on visual features, and generate event description text to describe the candidate event.

[0057] Specifically, taking a segment of personnel activity in an indoor surveillance video as an example scenario, the candidate event trigger words obtained in S31... Argument set Event Type Action attributes and spatiotemporal positioning information Based on the aforementioned trigger words, arguments, event types, action attributes, and spatiotemporal positioning information, event description text is generated to describe the candidate events. For example, by using template filling or generating a model:

[0058]

[0059] in A function to generate event description text.

[0060] S33: Based on the event type, event arguments, action attributes, spatiotemporal positioning information, and multimodal context information, the candidate events are mapped into event vectors, and the candidate events containing the event vectors are added to the candidate event set; wherein, the multimodal context information includes the aggregated representation of the multimodal feature sequence within the time interval of the candidate event occurrence and the semantic representation of the event description text, and the event vector is obtained by combining event type embedding, argument embedding, and multimodal context embedding;

[0061] Specifically, taking a segment of personnel activity from an indoor surveillance video as an example scenario, to facilitate subsequent relationship inference, each candidate event is mapped to an event vector:

[0062]

[0063] in Indicates the time interval of the candidate event. The corresponding multimodal feature sequence subsequence of For embedded functions, For pooling functions, For splicing operations; and will As part of the argument embedding, the event vector is obtained by combining event type embedding, argument embedding, and multimodal context embedding. Furthermore, the event description text can be... The semantic representation is obtained through text encoding and incorporated into the multimodal context embedding to meet the requirement that "multimodal context information contains the semantic representation of event description text".

[0064] S4: Perform event chain construction on the candidate event set, infer the temporal and causal relationships between events, and obtain the event chain structure; this step includes the following sub-steps:

[0065] S41: Establish a time sequence graph based on the start and end times of candidate events, and add time sequence edges to event pairs according to time constraints. The time sequence edges are directed edges from the earlier event to the later event.

[0066] Specifically, using footage of people's activities from indoor surveillance video as an example scenario, a chronology diagram is established based on the time intervals of candidate events. For event pairs... ,like And the interval between the two satisfies the preset time constraint (e.g.) If the time constraint threshold is not exceeded, then time sequence edges can be added. The chronological order edges are directed edges pointing from earlier events to later events, thus forming a chronological order graph. .

[0067] S42: Calculate the causal score of the event pair based on the causal discriminant model, and add a causal edge to the event pair when the causal score is not lower than the second threshold;

[0068] Specifically, taking a segment of personnel activity from an indoor surveillance video as an example scenario, a causal discriminant model is used to score event pairs and obtain causal scores. The causal discriminant model can be input... and output Interval score. Among them... The event vector obtained from S3 The time interval characteristic of event pairs, This provides contextual information for the event pair. When Add causal edges to event pairs when the threshold is not lower than the second threshold. The set of causal edges is obtained. .

[0069] S43: Construct a candidate event chain graph by combining the temporal sequence edges and the causal edges, apply acyclic constraints and consistency constraints to the temporal sequence graph and the causal edge set, perform loop resolution and obtain the event chain structure, wherein the event chain structure is a directed acyclic graph;

[0070] Specifically, taking a segment of personnel activity from an indoor surveillance video as an example scenario, the time sequence is set... With the set of causal edges Merge and construct a candidate event chain graph:

[0071]

[0072] Acyclic constraints and consistency constraints are applied to the candidate event chain graph: the acyclic constraints are used to ensure... It is a directed acyclic structure; the consistency constraint is used to ensure that the causal edges are consistent with the temporal order, that is, if there exists Then it should satisfy No later than When a causal edge conflicts with a temporal edge, the conflicting edge with the lowest confidence is removed based on its weight, or the edge with the lowest confidence is removed from the loop to resolve the loop, resulting in an event chain structure that is a directed acyclic graph. In addition, redundancy removal can be performed on the event chain, such as merging duplicate events of the same entity and the same action within adjacent time windows and retaining the one with the highest confidence.

[0073] S5: Sample the reasoning paths on the event chain structure and generate matching reasoning questions and answers for each reasoning path; this step includes the following sub-steps:

[0074] S51: In the event chain structure, sample multi-hop paths with the starting event as the condition and the length within the third threshold range, and output the event sequence and relationship sequence on the path;

[0075] Specifically, taking a segment of personnel activity from an indoor surveillance video as an example scenario, in the event chain structure... Upsample multi-hop inference paths and generate matching inference questions and answers for each path. An inference path can be represented as...

[0076]

[0077] in For event nodes, For relational types, The path length is defined as follows. The path length can be constrained within a third threshold range, such as 2 to 5, to balance inference difficulty and learnability, where the third threshold represents the path length range. Sampling strategies can include random walks, edge-weight-based bias sampling, or coverage-driven sampling, and the output should be a sequence of events along the path. With relation sequence :

[0078]

[0079] S52: Generate at least one question type based on the event sequence and the relationship sequence, wherein the question type includes causal question answering, time sequence question answering, and conditional inference question answering;

[0080] Specifically, taking a segment of personnel activity in an indoor surveillance video as an example scenario, based on the event sequence... With relation sequence The question types are determined, including causal question-and-answer, chronological question-and-answer, and conditional inference question-and-answer, and are based on the relationship sequence. The combination pattern of relation types serves as the basis for determining the question type: when the relation sequence contains causal edges, it corresponds to causal question and answer; when the relation sequence is mainly composed of time-sequential edges, it corresponds to time-sequential question and answer; and when the path contains conditional branching relations, it corresponds to conditional inference question and answer.

[0081] S53: Generate question text based on the question type, the event sequence, and the relationship sequence, and generate an answer that matches the question text based on the evidence fragment location information. At the same time, generate explanatory text, which contains key events and key relationships in the reasoning path. The question text, the answer, and the explanatory text constitute the reasoning question and answer.

[0082] Specifically, taking a segment of personnel activity from an indoor surveillance video as an example scenario, question text is generated based on the question type, the event sequence, and the relationship sequence. And determine the answer event on the reasoning path. The event description text corresponding to the answer event. Or the result of its trigger word and argument filling as the answer. .

[0083] Simultaneously, evidence fragment location information is generated based on the spatiotemporal location information of the answer event. In one embodiment, if the time interval of the answer event is... The time interval of the evidence fragment can then be defined as:

[0084]

[0085] in To preset the redundancy duration, This refers to the total video duration; and it can also provide the spatial location information of the answer event. Spatial location information as evidence fragments .

[0086] Furthermore, generate explanatory text. The explanatory text contains key events and key relationships within the reasoning path. This leads to the understanding of the reasoning path. Matched inference question-answering samples: in For the question text, For the answer, Locating information for evidence fragments, To interpret the text.

[0087] S6: Perform consistency verification and quality assessment on the generated reasoning question-answering; this step includes the following sub-steps:

[0088] S61: Use the event chain executor to perform symbolic reasoning on the reasoning question and answer, obtain symbolic answers, and compare the symbolic answers with the answers in the reasoning question and answer;

[0089] Specifically, taking a segment of personnel activity from an indoor surveillance video as an example scenario, the first channel of the dual-channel verification process for the generated reasoning questions and answers is symbolic execution: in the event chain structure The above-mentioned reasoning questions and answers The constraints obtained through parsing serve as input, and the symbolic answer is derived through the event chain executor. and the answers in the reasoning questions and answers. Perform consistency comparison, among which Answer generated for S5 .

[0090] S62: Use a video question answering model to reason about the questions in the reasoning question answering on the corresponding video segment to obtain the model answer, and compare the consistency between the model answer and the answer in the reasoning question answering;

[0091] Specifically, taking a segment of personnel activity in an indoor surveillance video as an example scenario, the second channel is for model verification: using a video question-answering model to answer questions in the inference question-answering process on the corresponding video segment. The model answer is obtained through reasoning. and the model answer The answer in the reasoning question and answer Perform a consistency comparison.

[0092] S63: Provided that the consistency comparison is passed, calculate the quality score based on the evidence location reliability, problem ambiguity and negative sample interference, and screen out low-quality samples based on the quality score.

[0093] Specifically, taking a segment of personnel activity in an indoor surveillance video as an example scenario, the quality score is calculated assuming that the consistency comparison between the two channels passes. The quality score can be determined by the location reliability of the evidence. Negative sample interference Ambiguity of the issue Combined, for example:

[0094]

[0095] in The weights are preset. Among them, the reliability of evidence location is... Used to measure the degree of matching between evidence fragment location information and answer event, it can be obtained by the proportion of the time interval of the answer event covered by the evidence fragment, spatial location reliability, or evidence retrieval score. Question ambiguity. This measure assesses whether there is a tendency for multiple solutions to the same question given the current evidence fragment. It can be obtained through methods such as candidate answer distribution entropy and the decrease in consistency of synonym rewriting. Negative sample interference degree. To measure the sample's robustness against distractors, it can be obtained by constructing negative samples that are similar to the correct answer and calculating the model's advantage over the correct answer. For example, the interval between the correct answer score and the score of the strongest negative sample can be used as the measure of the model's strength against distractors. Low-quality samples are screened out based on a quality score threshold.

[0096] The generated data samples can be stored in formats such as JSONL or TFRecord. Each sample must contain at least: video identifier, segment time range, question text, answer text, evidence location information, event chain structure, and optional explanatory text.

[0097] S7: Encapsulate the reasoning questions and answers that have passed consistency verification and quality assessment, along with their corresponding video clips, event chain structures, and evidence location information, into video pre-training data;

[0098] Specifically, taking footage of people's activities from indoor surveillance video as an example scenario, each filtered reasoning question-and-answer sample... Determine the video identifier to which it belongs. Corresponding video segment time interval and the event chain structure constructed on this video clip. Among them, evidence location information At least include the time location of evidence fragments Optionally includes spatial positioning .

[0099] In one embodiment, the above information is encapsulated as a structured sample. : ,in For identifying the reasoning path, The metadata field includes at least the sample generation time, question type, and quality score. And the consistency verification result identifier.

[0100] To facilitate pre-training, the encapsulated structured samples can be serialized and stored in JSONL, TFRecord, or Parquet format, and then binned according to video source, question type, and path length. The sampling ratio of different bins can be controlled to achieve a balanced sample distribution. Furthermore, near-duplicate questions within the same video segment can be deduplicated, retaining samples with higher quality scores, thus obtaining the final video pre-training dataset.

[0101] Corresponding to the aforementioned embodiments of the video pre-training data generation method, this application also provides embodiments of the video pre-training data generation apparatus.

[0102] like Figure 2 As shown, this embodiment also provides a video pre-training data generation device. The device includes:

[0103] Data access module 1 is used to acquire video data to be processed and its metadata, and to standardize the metadata into a unified timeline and a unified identifier;

[0104] Segmentation module 2 is used to segment the video data in time to obtain multiple video segments, and to construct keyframe sequences and corresponding multimodal features for each video segment;

[0105] Event extraction module 3 is used to extract events from each video segment based on the multimodal features, and obtain a set of candidate events containing event trigger words, participating entities, action attributes and spatiotemporal positioning;

[0106] Event chain construction module 4 is used to construct event chains for the candidate event set, infer the temporal and causal relationships between events, and obtain the event chain structure;

[0107] The path sampling module 5 is used to sample the reasoning path on the event chain structure and generate a matching reasoning question and answer for each reasoning path, wherein the reasoning question and answer includes at least the question, the answer and the location information of the evidence fragment;

[0108] Question and answer generation module 6 is used to perform consistency verification and quality assessment on the generated reasoning questions and answers;

[0109] The verification and storage module 7 is used to encapsulate the reasoning questions and answers that have passed consistency verification and quality assessment, along with their corresponding video clips, event chain structures, and evidence location information, into video pre-training data.

[0110] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0111] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0112] Accordingly, this application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the video pre-training data generation method described above. Figure 3 The diagram shown is a hardware structure diagram of any device with data processing capabilities, including a video pre-training data generation device provided in an embodiment of the present invention. Except for... Figure 3 In addition to the processor, memory, DMA controller, disk, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0113] Accordingly, this application also provides a computer-readable storage medium storing computer instructions thereon, which, when executed by a processor, implement the video pre-training data generation method described above. The computer-readable storage medium can be an internal storage unit of any data-processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data-processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data-processing device, and can also be used to temporarily store data that has been output or will be output.

[0114] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover various variations, uses, and adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0115] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for generating video pre-training data, characterized in that, include: Acquire the video data to be processed and its metadata, and standardize the metadata into a unified timeline and a unified identifier; The video data is segmented over time to obtain multiple video segments, and keyframe sequences and corresponding multimodal features are constructed for each video segment. Based on the multimodal features, events are extracted from each video segment to obtain a set of candidate events containing event trigger words, participating entities, action attributes, and spatiotemporal positioning. Event chain construction is performed on the candidate event set to infer the temporal and causal relationships between events and obtain the event chain structure. The reasoning paths are sampled on the event chain structure, and a matching reasoning question and answer is generated for each reasoning path, wherein the reasoning question and answer includes at least the question, the answer and the location information of the evidence fragment; Perform consistency verification and quality assessment on the generated reasoning questions and answers; use an event chain executor to perform symbolic reasoning on the reasoning questions and answers to obtain symbolic answers, and compare the symbolic answers with the answers in the reasoning questions and answers for consistency. The reasoning questions and answers that have passed consistency verification and quality assessment, along with their corresponding video clips, event chain structures, and evidence location information, are encapsulated into video pre-training data. Specifically, event chain construction is performed on the candidate event set to infer the temporal and causal relationships between events, resulting in an event chain structure, including: A chronological order graph is established based on the start and end times of candidate events, and chronological order edges are added to event pairs according to time constraints. The chronological order edges are directed edges from the earlier event to the later event. The causal score of the event pair is calculated based on the causal discriminant model, and a causal edge is added to the event pair when the causal score is not lower than the second threshold. The temporal sequence edges and the causal edges are constructed into a candidate event chain graph. Acyclic constraints and consistency constraints are applied to the temporal sequence graph and the causal edge set. Cycle resolution is performed to obtain the event chain structure, which is a directed acyclic graph.

2. The method according to claim 1, characterized in that, The video data is segmented over time to obtain multiple video clips, including: Based on lens boundary detection and scene consistency scoring, the video is divided into multiple video segments with lengths within a first threshold range, and at least one frame is selected as a keyframe for each video segment.

3. The method according to claim 2, characterized in that, Construct keyframe sequences and corresponding multimodal features for each video segment, including: According to the preset keyframe selection rules, multiple keyframes are selected from each video segment and arranged in chronological order to obtain a keyframe sequence; Visual encoding is performed on the keyframe sequence to obtain a frame-level visual feature sequence, and audio encoding and text encoding are performed on the corresponding audio signal and subtitle text respectively to obtain audio feature sequence and text feature sequence; The frame-level visual feature sequence, audio feature sequence, and text feature sequence are aligned and fused according to timestamps to obtain a segment-level multimodal feature sequence.

4. The method according to claim 1, characterized in that, Based on the multimodal features, events are extracted from each video segment to obtain a candidate event set containing event trigger words, participating entities, action attributes, and spatiotemporal positioning, including: Based on a preset event pattern library, trigger word identification and argument filling are performed on the multimodal feature sequence to obtain candidate events. The event arguments include at least one of the following: subject entity, object entity, location entity, and time entity. The identified trigger words are used as the event trigger words of the candidate events. For each candidate event, predict the event type and the start and end times of the event, determine the action attributes of the candidate event based on action features, determine the spatiotemporal location information of the candidate event based on visual features, and generate event description text to describe the candidate event. Candidate events are mapped to event vectors based on the event type, event arguments, action attributes, spatiotemporal positioning information, and multimodal context information, and candidate events containing the event vectors are added to the candidate event set; wherein, the multimodal context information includes the aggregated representation of the multimodal feature sequence within the time interval of the candidate event occurrence and the semantic representation of the event description text, and the event vector is obtained by combining event type embedding, argument embedding, and multimodal context embedding.

5. The method according to claim 1, characterized in that, Sampling inference paths on the event chain structure and generating matching inference questions and answers for each inference path, including: The event chain structure samples multi-hop paths with the starting event as the condition and the length within the third threshold range, and outputs the event sequence and relationship sequence on the path; At least one question type is generated based on the event sequence and the relationship sequence, including causal question answering, time sequence question answering, and conditional inference question answering. Based on the question type, the event sequence, and the relationship sequence, a question text is generated, and an answer matching the question text is generated based on the evidence fragment location information. At the same time, an explanatory text is generated, which contains key events and key relationships in the reasoning path. The question text, the answer, and the explanatory text constitute the reasoning question and answer.

6. The method according to claim 1, characterized in that, The generated reasoning questions and answers undergo consistency verification and quality assessment, and this also includes: The video question answering model is used to reason about the questions in the reasoning question answering on the corresponding video segment to obtain the model answer, and the model answer is compared with the answer in the reasoning question answering. Provided that the consistency comparison is passed, the quality score is calculated based on the evidence location reliability, problem ambiguity and negative sample interference, and low-quality samples are screened out according to the quality score.

7. A video pre-training data generation device, characterized in that, include: The data access module is used to acquire the video data to be processed and its metadata, and to standardize the metadata into a unified timeline and a unified identifier. The segmentation module is used to segment the video data into time segments to obtain multiple video segments, and to construct keyframe sequences and corresponding multimodal features for each video segment; The event extraction module is used to extract events from each video segment based on the multimodal features, and obtain a set of candidate events containing event trigger words, participating entities, action attributes and spatiotemporal positioning. The event chain construction module is used to construct event chains for the candidate event set, infer the temporal and causal relationships between events, and obtain the event chain structure. The path sampling module is used to sample the reasoning path on the event chain structure and generate a matching reasoning question and answer for each reasoning path, wherein the reasoning question and answer includes at least the question, the answer and the location information of the evidence fragment; The question-and-answer generation module is used to perform consistency verification and quality assessment on the generated reasoning questions and answers; it uses an event chain executor to perform symbolic reasoning on the reasoning questions and answers to obtain symbolic answers, and compares the symbolic answers with the answers in the reasoning questions and answers for consistency. The verification and storage module is used to encapsulate the reasoning questions and answers that have passed consistency verification and quality assessment, along with their corresponding video clips, event chain structures, and evidence location information, into video pre-training data. Specifically, event chain construction is performed on the candidate event set to infer the temporal and causal relationships between events, resulting in an event chain structure, including: A chronological order graph is established based on the start and end times of candidate events, and chronological order edges are added to event pairs according to time constraints. The chronological order edges are directed edges from the earlier event to the later event. The causal score of the event pair is calculated based on the causal discriminant model, and a causal edge is added to the event pair when the causal score is not lower than the second threshold. The temporal sequence edges and the causal edges are constructed into a candidate event chain graph. Acyclic constraints and consistency constraints are applied to the temporal sequence graph and the causal edge set. Cycle resolution is performed to obtain the event chain structure, which is a directed acyclic graph.

8. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by the processor, this instruction implements the steps of the method as described in any one of claims 1-6.

Images