Long video event prediction method, system and device and storage medium

Abstract

The invention discloses a long video event prediction method, system and device and a storage medium, which are one-to-one corresponding schemes, and in the scheme, more reliable future event prediction is guided by mining from a macroscopic event at the highest level and utilizing a potential evolution mode, so that the prediction efficiency is improved. Specifically, a large amount of space-time information in an original long video is subjected to semantic abstraction and induction layer by layer and further goes deep into a macroscopic event development mode to guide future event prediction, and a layered framework effectively captures and refines key semantics related to event understanding from a large amount of information. And the event evolution mode reveals the macroscopic trend of future development of the events, and these designs effectively solve the difficulties of spatio-temporal information explosion and intricate relation between the events in the long video, and generate more reliable event prediction.

Description

Technical Field

[0001] The present invention relates to the technical field of long video event prediction, and in particular to a long video event prediction method, system, device and storage medium. Background Art

[0002] Event prediction aims to predict the future situation of events based on historical situations, which helps to identify and avoid potential risks, and then provide strong support for event decision-making and emergency response. In the past, event prediction work often revolved around the textual description of historical events. With the enrichment of information collection and interaction methods, more and more events are presented in video forms such as live streaming. However, the multimodal, rich semantics, high noise and abstract expression of video expression make it difficult to migrate text-based analysis methods to video scenarios for application. Therefore, how to effectively respond to the needs of event understanding and situation awareness in video scenarios has become a timely need.

[0003] To meet this demand, early work often focused on some shallow details, such as target or action recognition, to understand and predict video events. However, these technologies often lack semantic abstraction capabilities, making it difficult to understand videos from a more macro perspective and summarize the logical chains in them. More seriously, this defect causes these methods to often only be able to process shorter videos, and they lack the processing power for long videos, which are more common in reality and have more complete and richer semantic information, making it difficult to form a significant logical chain of events. This undoubtedly severely limits its application value.

[0004] With the development of large language models and visual language technology, the semantic understanding ability of long videos has been greatly improved. These methods usually use an end-to-end approach to directly understand the video, or convert the video into text for prediction. Although they can effectively locate details and summarize events, these methods still cannot further rise to the level of events to grasp the macro trend of events, and are easily lost in the complex logical relationship between a large number of events in a long video, which greatly weakens their effectiveness.

[0005] In view of this, the present invention is proposed. Summary of the invention

[0006] The purpose of the present invention is to provide a long video event prediction method, system, device and storage medium, which can go deep into the macroscopic event level layer by layer to effectively understand the long video, and accurately predict future events based on the law of event development.

[0007] The objective of the present invention is achieved through the following technical solutions: A long video event prediction method, comprising: The input original video is divided into several continuous video clips, and the dialogue text and video character images in each video clip are extracted; Encode each video clip, the dialogue text in the video clip and the image of the video characters separately, and fuse them to generate visual description text; The dialogue text and the corresponding visual description text in each video clip are summarized as an event, and based on the common sense knowledge expert model, a knowledge-promoted retrieval strategy is adopted to connect different events into a coherent and orderly event chain according to logical associations; Capture the evolution pattern of events from the event chain and predict the next stage of events, then combine the event chain with the predicted next stage of events to predict future events.

[0008] A long video event prediction system, used to implement the above method, the system comprises: A multimodal data preprocessing unit, used to divide the input original video into a plurality of continuous video segments, and extract the dialogue text and the video character image in each video segment; A key visual description generation unit is used to encode each video segment, the dialogue text in the video segment and the video character image respectively, and fuse them to generate a visual description text; The knowledge-facilitated event chain construction unit is used to summarize the dialogue text and the corresponding visual description text in each video clip into an event, and based on the common sense knowledge expert model, adopt the knowledge-facilitated retrieval strategy to connect different events into a coherent and orderly event chain according to logical associations; The future event prediction unit is used to capture the event evolution pattern from the event chain and predict the next event stage, and then combine the event chain with the predicted next event stage to predict future events.

[0009] A processing device, comprising: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0010] A readable storage medium stores a computer program, which implements the above method when the computer program is executed by a processor.

[0011] It can be seen from the technical solution provided by the present invention that the large amount of spatiotemporal information in the original long video is semantically abstracted and summarized layer by layer, and then goes deep into the macro-event development model to guide the prediction of future events. The hierarchical framework effectively captures and refines the key semantics related to event understanding from a large amount of information, and the event evolution model reveals the macro trend of the future development of the event. These designs effectively solve the difficulties of spatiotemporal information explosion and complex connections between events in long videos, and generate more reliable event predictions. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0013] Figure 1 A flowchart of a long video event prediction method provided by an embodiment of the present invention; Figure 2 A schematic diagram of the overall architecture of a long video event prediction method provided by an embodiment of the present invention; Figure 3 A schematic diagram of a long video event prediction system provided by an embodiment of the present invention; Figure 4 A schematic diagram of a processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0014] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0015] First, the terms that may be used in this article are explained as follows: The terms "include", "comprises", "contains", "has" or other descriptions with similar semantics should be interpreted as non-exclusive inclusion. For example, including certain technical feature elements (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or products, etc.) should be interpreted as including not only certain technical feature elements explicitly listed, but also other technical feature elements known in the art that are not explicitly listed.

[0016] The term "consisting of..." means excluding any technical feature elements not explicitly listed. If this term is used in a claim, it will make the claim closed, so that it does not contain technical feature elements other than the technical feature elements explicitly listed, except for the conventional impurities related to them. If this term only appears in a clause of a claim, it only limits the elements explicitly listed in the clause, and the elements recorded in other clauses are not excluded from the overall claim.

[0017] The following is a detailed description of a long video event prediction method, system, device and storage medium provided by the present invention. The contents not described in detail in the embodiments of the present invention belong to the prior art known to professional and technical personnel in the field. If no specific conditions are specified in the embodiments of the present invention, the conventional conditions in the field or the conditions recommended by the manufacturer are followed. The instruments used in the embodiments of the present invention without indicating the manufacturer are all conventional products that can be purchased commercially.

[0018] Embodiment 1 The embodiment of the present invention provides a method for predicting long video events. Figure 1 As shown, it mainly includes the following steps: Step 1: Segment the input original video into several continuous video segments, and extract the dialogue text and video character images in each video segment.

[0019] Preferably, the input original video can be segmented based on scene change, and then the segmentation results can be connected in chronological order according to a set segment length threshold to obtain a plurality of continuous video segments.

[0020] Preferably, extracting the dialogue text and video character image in each video clip includes: for each video clip, combining speaker recognition technology and audio transcription technology to identify the video character and generate corresponding dialogue text; classifying the video character images based on face similarity, each category corresponds to a video character, and sorting the categories according to the frequency of appearance of the video character, retaining the top L categories, and for each retained category, randomly selecting any video character image in the category.

[0021] Step 2: Encode each video clip, the dialogue text in the video clip, and the video character image separately, and fuse them to generate a visual description text.

[0022] Preferably, in this step, each video clip is encoded by a video encoder and video clip features are extracted; each video character image is encoded by an image encoder and image features are extracted; and a visual description text is generated by combining the video clip features, image features and dialogue text in the video clip through the first large language model.

[0023] Step 3: Summarize the dialogue text and the corresponding visual description text in each video clip into an event, and based on the common sense knowledge expert model, adopt a knowledge-promoted retrieval strategy to connect different events into a coherent and orderly event chain according to logical associations.

[0024] Preferably, the common sense knowledge expert model may be a language model fine-tuned by a large-scale common sense knowledge graph.

[0025] Preferably, the common sense knowledge expert model adopts a knowledge-enhanced retrieval strategy to connect different events into a coherent and orderly event chain based on logical associations, including: (1) For the current event, the events before the current event are called historical events. Usually, the time stamp of the long video is used as the basis, and the time before the current event in the long video is the historical event. (2) Each historical event is input into the common sense knowledge expert model to obtain the impact of each historical event. Each historical event is connected with its corresponding impact and then encoded. The encoding results are stored in the historical event database.

[0026] In the embodiment of the present invention, the impact of various historical events may directly cause the occurrence of the current event. Revealing this potential impact helps to improve the accuracy of subsequent determination of the causal relationship between the current event and the historical event.

[0027] (3) Retrieve the top k historical events that are most relevant to the text similarity of the current event from the historical event database, input the current event and the retrieved top k historical events into the second largest language model, and classify the current event into the corresponding event chain through the second largest language model.

[0028] Step 4: Capture the evolution pattern of events from the event chain and predict the next stage of events. Then combine the event chain with the predicted next stage of events to predict future events.

[0029] Preferably, a third language model and a fourth language model are set, the fourth language model is fine-tuned using data constructed by the third language model, and step 4 is performed using the fine-tuned fourth language model as a future event prediction unit; wherein the process of fine-tuning the fourth language model using data constructed by the third language model includes: Collect a multimodal dataset for video understanding and narrative analysis. The dataset condenses a video into a series of clips, each of which is an event with a text description of the event. Obtain the summary of the video and each event in the dataset as the summary of the entire video narrative, and summarize the summary of the entire video narrative into the event evolution pattern of the entire video through the third language model; The summary of the entire video narration, the event evolution pattern and the set target future events are input into the fourth language model, and prompt words are set to guide the fourth language model to pay attention to the event evolution pattern. According to the difference between the output of the fourth language model and the set target future events, a loss function is constructed to fine-tune the fourth language model (i.e., fine-tune the parameters of the fourth language model).

[0030] Compared with the traditional method of predicting future events by identifying low-semantic information in videos such as actions and objects, the above-mentioned scheme provided by the embodiment of the present invention explores the way of imitating the hierarchical understanding of long video events by humans, and semantically abstracting and summarizing the large amount of spatiotemporal information in the original long video layer by layer, and then going deep into the macro-event development pattern to guide the prediction of future events. The hierarchical framework effectively captures and refines the key semantics related to event understanding from a large amount of information, and the event evolution pattern reveals the macro-trend of the future development of the event. These designs effectively solve the difficulties of the explosion of spatiotemporal information in long videos and the complex connections between events, and generate more reliable event predictions.

[0031] In order to more clearly demonstrate the technical solution and technical effects provided by the present invention, the method provided by the embodiment of the present invention is described in detail with reference to specific embodiments below.

[0032] 1. Overall overview of the plan.

[0033] The embodiment of the present invention provides a long video event prediction method, which adopts a hierarchical structure corresponding to the way humans understand videos to understand long videos, and guides more reliable future event predictions by mining and utilizing potential evolution patterns from the highest-level macroscopic events. First, the present invention uses a specially designed large visual model to convert key visual information into a low-noise, high-semantic-density text representation, thereby efficiently extracting the detailed information necessary to understand the event. Afterwards, the present invention uses a large language model to summarize the bottom-level details into events, and adopts a knowledge-promoted retrieval strategy to connect all isolated events discovered in the long video into a coherent and orderly event chain according to their internal logical associations. Finally, the present invention designs a thinking chain to imitate the way humans predict events, trains the language model to refine the event chain into macroscopic events step by step, mines the potential evolution patterns of macroscopic events, and combines the event evolution patterns with specific scenarios to make future event predictions.

[0034] 2. Detailed introduction of the plan.

[0035] like Figure 2As shown in the figure, the four processing parts of the present invention are shown: (1) The present invention uses some professional tools to split the original long video into segments of appropriate length and extract basic information such as text dialogues and character portraits for use by subsequent modules. (2) The present invention specially designs and trains a visual language model, which can generate key visual descriptions with less noise than ordinary video captioning models, thereby providing more accurate and concise detail information for higher-level modules. (3) The present invention summarizes details into events and organizes interrelated events into structured chains to enhance the coherence of event-level semantics. To achieve this, the present invention introduces a common sense knowledge expert model to provide causal guidance and adopts a retrieval strategy to filter related events. In this way, the present invention constructs a coherent event chain to facilitate analysis at the final state of affairs. (4) The present invention designs a pattern-centric thinking chain that explicitly captures potential situation evolution patterns and makes inferences based on them. Corresponding data is collected to fine-tune the large language model so that it can grasp the situation evolution pattern. Under the guidance of the situation evolution pattern, more reliable predictions can be made about future events in long videos. The various models involved in the above process will be introduced later. The flame symbol marked in the box where the model is located indicates that the corresponding model parameters must be optimized, and the snowflake indicates that the model parameters are frozen.

[0036] The above four parts are introduced below.

[0037] 1. Multimodal preprocessing.

[0038] (1.1) Video segmentation.

[0039] Considering the length of the video, the present invention first uses relevant tools to segment the long video into many short video clips based on scene changes. Considering that too short clips may lead to loss of context, the present invention further connects the continuous clips into clips in chronological order to ensure that the length of each clip is not less than the set length (for example, 4 minutes). It should be noted that this length is only the length used in the current implementation case. The specific length can be adjusted accordingly according to data characteristics and needs, and the corresponding number of clips N obtained by the final segmentation can also be determined according to actual conditions.

[0040] (1.2) Transcription of video dialogue.

[0041] The dialogue between characters in the video is crucial for understanding video events. In order to facilitate and effectively use the character dialogue to help understand the event, the present invention uses a professional audio transcription tool to transcribe the original video to obtain the character dialogue in text form. At the same time, considering that the characters in a long video are often not single, and the identity of the characters is indispensable information for accurately understanding the event, the present invention combines speaker recognition technology during transcription to clarify the identity of the speaker of each sentence of dialogue, thereby supplementing the character identity information on the text side.

[0042] (1.3) Collection of video portraits.

[0043] In order to further supplement the missing identity information of the characters on the visual side, the present invention collects the portraits of the main characters in the video based on the original long video. First, the present invention uses face recognition technology to collect face images in the video at a set frequency; then, all the collected face images are classified based on face similarity, and each category corresponds to a character; finally, the present invention only retains the characters with the top L appearance frequencies, and randomly samples one from the face image corresponding to the character as the character portrait and provides it to the subsequent steps.

[0044] In the embodiment of the present invention, only the first L video character portraits are retained, but all previously obtained dialogue texts are retained. The reason is that the first L video character portraits contain the identity information of the main character. Except for the first L video character portraits, the remaining character portraits are all secondary character portraits. Injecting them into the network will weaken the identity information of the main character. However, considering that the removal of the dialogues of the secondary character portraits is likely to cause local semantic loss and affect the understanding of the fragment events, all previously obtained dialogue texts are retained.

[0045] Exemplarily, the frequency may be set to 1 frame every 5 seconds. Of course, this frequency is only a parameter used in this embodiment, and the specific frequency may be set according to actual conditions or experience.

[0046] Exemplarily, L can be set to 10, that is, the top 10 characters in frequency are retained. Of course, the value of L provided here is only an example and does not constitute a limitation. Its specific value can also be set according to actual conditions or experience.

[0047] 2. Key visual description generation.

[0048] After obtaining a clip of appropriate length, the present invention plans to generate key visual descriptions from it to extract the details required for event understanding. However, common video subtitles usually contain a lot of noise that is irrelevant to the event, such as meaningless objects; and lack key information, such as character identity. These defects may lead to misunderstanding of the event. In order to retain key information while minimizing noise, the present invention introduces audio transcription technology to generate text audio description (AudioDescription), which is a data type that provides a brief audio description of key visual facts during pauses in character dialogues, designed to help visually impaired viewers enjoy the video. This specific scenario ensures that the audio description is more precise and concise than ordinary video subtitles.

[0049] The present invention specifically designs a visual language model to generate text-based audio descriptions. Since character portraits provide an important reference for character identity and dialogues contain important contextual information, the present invention introduces an additional image encoder to represent character portraits visually and incorporate nearby dialogues textually. Subsequently, a dedicated audio description generation dataset is introduced to train the model. Ultimately, high-quality visual descriptions can be generated to provide precise and concise details for subsequent event hierarchies.

[0050] Figure 2 In the figure, the key visual description generating dashed box includes the video encoder, image encoder, Q-Former (a new type of neural network architecture that focuses on improving information retrieval and representation learning through query mechanisms), mapping layer and the first language model, which together constitute the visual language model. The Q-Former and mapping layer behind the video encoder and image encoder are responsible for mapping the features extracted by the corresponding encoder into a unified vector space. It should be noted that Figure 2 The examples provided are only feasible model structures. In actual applications, users can make adjustments based on actual conditions or needs.

[0051] Exemplarily, both the video encoder and the image encoder can use the EVA-CLIP model, which is a model improved based on the contrastive language image pre-training (CLIP) technology. It can extract general visual representations containing high-level semantic information from visual signals, providing a perceptual basis for a wide range of visual understanding and visual-language multimodal tasks.

[0052] 3. Construction of event chains promoted by knowledge.

[0053] In the aforementioned process of the present invention, the generated text dialogues and key visual descriptions constitute the detailed semantics required to understand the event-level semantics. However, simply summarizing the details into events is not enough to construct a coherent event-level semantics. Logically, a long video usually contains multiple event sequences, and the events in different sequences appear alternately in chronological order. If the logical connection between events is not considered, it may lead to confusion of unrelated events, thereby affecting the clarity of the narrative.

[0054] Therefore, in addition to simply summarizing events from details, the present invention also connects interrelated events into chains to improve coherence. Although directly using a large language model is a simple approach, the complex logical relationships make it difficult to obtain satisfactory results when processing a large number of events at the same time. To solve this problem, the present invention introduces a language model fine-tuned by a large-scale common sense knowledge graph (ATOMIC) as a common sense knowledge expert model to provide causal logic guidance; and adopts a retrieval strategy to narrow the scope of the event window.

[0055] First, the present invention uses the second largest language model to summarize the text dialogue and key visual descriptions into the current event; at the same time, each historical event is input into the common sense knowledge expert model to obtain its possible impact as a causal relationship to highlight the similarity between related events. It is effective; then, each historical event and its corresponding causal relationship are connected and encoded, and the encoding results are saved in the historical event database. Subsequently, the top k historical events that are most relevant to the text similarity of the current event are retrieved from the historical event database, thereby significantly narrowing the scope of the event. Finally, the current event is input into the second largest language model together with the top k retrieved historical events to classify it into a specific event chain.

[0056] In the embodiment of the present invention, multiple event chains can be obtained (for example, they can be recorded as M, and the specific number is determined according to the actual situation). Initially, the event chain is empty, and as the video timestamp advances, the events that occur are iteratively added to the event chain. For the event to be judged currently, there are two possibilities based on its causal relationship with the events in the existing event chain: if it is determined that the current event has a clear causal relationship with the events of a certain event chain, the current event is added to the corresponding event chain; if the model believes that the event has no clear causal relationship with any historical event, the event is allowed to form a new event chain alone.

[0057] 4. Pattern-based thinking chain prediction.

[0058] The event chain provides a coherent event-level narrative for the semantic abstraction of the event level. At the event level, the long video shows the potential event evolution patterns (situation evolution patterns), which can provide key guidance for the prediction of future events. In order to utilize these patterns, the present invention introduces a third language model to capture potential event evolution patterns from existing events and reason based on these event evolution patterns. Therefore, the present invention designs a pattern-based thinking chain and specially trains the fourth language model to enhance the ability to capture situation evolution patterns and reason. Specifically, the thinking chain divides the prediction of future events into three steps: (4.1) Summarize the evolution pattern of events from the existing event chain; (4.2) Based on the evolution pattern of events, predict the next stage of events from historical events; (4.3) Combine the predicted next stage of events with the specific event context (i.e. the event chain obtained previously) to predict specific future events.

[0059] Because it is different from the commonly used chain of thinking in mathematics or symbolic reasoning, the evolution pattern of events is empirical in nature. In order to achieve empirical learning of the evolution of events, the present invention introduces the Condensed Movies Dataset (CMD) dataset as a guide, which is a multimodal dataset for video understanding and narrative analysis. This dataset condenses long videos into a series of clips. Each clip is accompanied by a textual event description and highlights the key points of the long video narrative. When the present invention is applied, according to the data characteristics and scene requirements, a suitable public dataset can be used, or the collected long videos can be similarly annotated to meet the needs of mining the evolution pattern of events.

[0060] Afterwards, for the long videos in the CMD dataset and each of the target future events therein, the corresponding summaries are retrieved from Internet resources as the benchmark for the entire long video narration. Next, the present invention uses a powerful large language model (the third language model) to summarize the summary as the evolution pattern of the entire long video to ensure its accuracy. Subsequently, the present invention inputs the summary, the evolution pattern and the target future event into the fourth language model. Since the fourth language model actually already knows the standard answer, it is only necessary to trace back the specific reasoning process to reach the answer. In order to ensure that the reasoning process meets the expectations of the present invention, the present invention uses specially designed prompt words to guide the fourth language model to focus on the evolution pattern. In this way, the present invention effectively extracts the knowledge of the evolution pattern and its reasoning ability from an omniscient and powerful large language model.

[0061] In the embodiment of the present invention, all types of models involved can be implemented using existing models. In practical applications, users can select the required model based on actual conditions or experience. Exemplary: the key visual description generation can be developed based on the large visual language model VideoLLaMA (Video Multimodal Large Model); the common sense knowledge expert model in the knowledge-promoted event chain construction can be obtained by fine-tuning the language model FLAN-T5 (text-to-text conversion model based on instruction fine-tuning). The first large language model can be implemented using the LLaMA2-7b model (Meta open source model, 7b means that the model parameters are 7 billion), the second large language model can be implemented by the closed source large language model GPT3.5-Turbo-0613 (the enhanced dialogue model released by OpenAI), and the fourth large language model can be implemented by LLaMA3.1-8b-instruc (the Chinese optimized version of the Meta open source model Llama3).

[0062] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by means of software plus necessary general hardware platforms. Based on such understanding, the technical solutions of the above embodiments can be embodied in the form of software products, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0063] Embodiment 2 The present invention also provides a long video event prediction system, which is mainly used to implement the method provided in the above embodiment, such as Figure 3 As shown, the system mainly includes: A multimodal data preprocessing unit, used to divide the input original video into a plurality of continuous video segments, and extract the dialogue text and the video character image in each video segment; A key visual description generation unit is used to encode each video segment, the dialogue text in the video segment and the video character image respectively, and fuse them to generate a visual description text; The knowledge-facilitated event chain construction unit is used to summarize the dialogue text and the corresponding visual description text in each video clip into an event, and based on the common sense knowledge expert model, adopt the knowledge-facilitated retrieval strategy to connect different events into a coherent and orderly event chain according to logical associations; The future event prediction unit is used to capture the event evolution pattern from the event chain and predict the next event stage, and then combine the event chain with the predicted next event stage to predict future events.

[0064] Technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0065] Embodiment 3 The present invention also provides a processing device, such as Figure 4 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the methods provided in the aforementioned embodiments.

[0066] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0067] In the embodiment of the present invention, the specific types of the memory, input device and output device are not limited; for example: The input device may be a touch screen, an image acquisition device, a physical button or a mouse, etc.; The output device may be a display terminal; The memory may be a random access memory (RAM) or a non-volatile memory such as a disk storage.

[0068] Embodiment 4 The present invention also provides a readable storage medium storing a computer program, which implements the method provided in the above embodiment when the computer program is executed by a processor.

[0069] In the embodiment of the present invention, the readable storage medium as a computer-readable storage medium can be set in the aforementioned processing device, for example, as a memory in the processing device. In addition, the readable storage medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk, etc., which can store program codes.

[0070] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or in any form that the information constitutes prior art known to those skilled in the art.

Claims

1. A long video event prediction method, characterized in that: include: The input original video is divided into several continuous video clips, and the dialogue text and video character images in each video clip are extracted; Encode each video clip, the dialogue text in the video clip and the image of the video characters separately, and fuse them to generate visual description text; The dialogue text and the corresponding visual description text in each video clip are summarized as an event, and based on the common sense knowledge expert model, a knowledge-promoted retrieval strategy is adopted to connect different events into a coherent and orderly event chain according to logical associations; Capture the evolution pattern of events from the event chain and predict the next stage of events, then combine the event chain with the predicted next stage of events to predict future events.

2. A long video event prediction method according to claim 1, characterized in that: The step of dividing the input original video into a plurality of continuous video segments comprises: Based on scene changes, the input original video is segmented, and then the segmentation results are connected in chronological order according to the set segment length threshold to obtain several continuous video segments.

3. A long video event prediction method according to claim 1, characterized in that: The extracting of the dialogue text and the video character image in each video clip comprises: For each video clip, speaker recognition technology and audio transcription technology are combined to identify the characters in the video and the corresponding dialogue text; The video character images are classified based on face similarity. Each category corresponds to a video character. The categories are sorted according to the frequency of appearance of the video characters. The top L categories are retained. For each retained category, any video character image in the category is randomly selected.

4. A long video event prediction method according to claim 1, characterized in that: The step of encoding each video clip, the dialogue text in the video clip, and the video character image, and fusing them to generate a visual description text comprises: Encode each video segment through a video encoder and extract features of the video segment; Each character image in the video is encoded separately through an image encoder to extract image features; The first language model is used to generate visual description text by combining video clip features, image features, and dialogue text in the video clip.

5. The long video event prediction method according to claim 1, characterized in that: The common sense knowledge expert model is a language model fine-tuned through the common sense knowledge graph.

6. A long video event prediction method according to claim 1 or 5, characterized in that: The common sense knowledge expert model adopts a knowledge-enhanced retrieval strategy to connect different events into a coherent and orderly event chain based on logical associations, including: For current events, the events at previous moments are called historical events; Input each historical event into the common sense knowledge expert model to obtain the impact of each historical event, connect each historical event with its corresponding impact and then encode it, and save the encoding result in the historical event database; The top k historical events that are most relevant to the text similarity of the current event are retrieved from the historical event database, the current event and the retrieved top k historical events are input into the second largest language model, and the current event is classified into the corresponding event chain through the second largest language model.

7. A long video event prediction method according to claim 1, characterized in that: Also includes: Setting up a third language model and a fourth language model, fine-tuning the fourth language model through the data constructed by the third language model, using the fine-tuned fourth language model as a future event prediction unit to capture the event evolution pattern from the event chain and predict the next event stage, and then combining the event chain with the predicted next event stage to predict future events; The process of fine-tuning the fourth language model using the data built by the third language model includes: Collect a multimodal dataset for video understanding and narrative analysis. The dataset condenses a video into a series of clips, each of which is an event with a text description of the event. Obtain the summary of the video and each event in the dataset as the summary of the entire video narrative, and summarize the summary of the entire video narrative into the event evolution pattern of the entire video through the third language model; The summary of the entire video narration, the pattern of events, and the set target future events are input into the fourth language model, and prompt words are set to guide the third language model to pay attention to the pattern of events. According to the difference between the output of the fourth language model and the set target future events, a loss function is constructed to fine-tune the fourth language model.

8. A long video event prediction system, characterized in that: For implementing the method described in any one of claims 1 to 7, the system comprises: A multimodal data preprocessing unit, used to divide the input original video into a plurality of continuous video segments, and extract the dialogue text and the video character image in each video segment; A key visual description generation unit is used to encode each video segment, the dialogue text in the video segment and the video character image respectively, and fuse them to generate a visual description text; The knowledge-facilitated event chain construction unit is used to summarize the dialogue text and the corresponding visual description text in each video clip into an event, and based on the common sense knowledge expert model, adopt the knowledge-facilitated retrieval strategy to connect different events into a coherent and orderly event chain according to logical associations; The future event prediction unit is used to capture the event evolution pattern from the event chain and predict the next event stage, and then combine the event chain with the predicted next event stage to predict future events.

9. A processing device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images