Structured video understanding method and device based on multi-modal large model, computer equipment and readable storage medium

Abstract

The invention discloses a structured video understanding method and device based on a multi-modal large model, computer equipment and a readable storage medium, and the method comprises the steps: firstly obtaining a to-be-processed video of a user and an interaction text describing a demand, then extracting keywords from the interaction text, and determining a focusing entity in combination with a first video multi-modal large model, and inputting the video into a second video multi-mode large model to obtain a focusing space-time scene graph of the focusing entity, and finally performing dialogue feedback on the interactive text according to the scene graph, thereby realizing efficient and accurate video content understanding and meeting user requirements.

Description

Technical Field

[0001] The present invention relates to the field of video processing, and in particular to a structured video understanding method, device, computer equipment and readable storage medium based on a multimodal large model. Background Art

[0002] In today's digital age, video, as an important information carrier, contains a vast amount of content. With the development of intelligent technology, accurate understanding of video content has become the key to meeting the needs of many application scenarios, such as intelligent monitoring, which requires analysis of abnormal behaviors in videos, and autonomous driving, which requires interpretation of road conditions. However, traditional video understanding methods face many challenges, such as difficulty in handling complex dynamic scenes and the tendency to produce inaccurate understanding results. Existing video understanding technologies often lack targeted analysis of user needs and are not efficient and accurate enough when processing video and text interactions. Summary of the invention

[0003] The object of the present invention is to provide a structured video understanding method, device, computer equipment and readable storage medium based on a multimodal large model.

[0004] In a first aspect, an embodiment of the present invention provides a structured video understanding method based on a multimodal large model, comprising:

[0005] Acquire a video to be processed and an interactive text input by a user, wherein the interactive text is a user demand description text for the video to be processed;

[0006] Extract keywords from the interactive text to obtain focused keywords, and determine focused entities corresponding to the focused keywords in the video to be processed in combination with a pre-trained first video multimodal large model;

[0007] Inputting the video to be processed into a pre-trained second video multimodal large model to obtain a focused spatiotemporal scene graph of a focused person and a focused object included in the focused entity;

[0008] A dialogue feedback is performed on the interactive text according to the focused spatiotemporal scene graph.

[0009] In a possible implementation, extracting keywords from the interactive text to obtain focused keywords includes:

[0010] A pre-trained large language model or NLTK library is called to extract keywords from the interactive text to obtain focused keywords.

[0011] In a possible implementation, the determining the focused entity corresponding to the focused keyword in the video to be processed by combining the pre-trained first video multimodal large model includes:

[0012] The first video multimodal large model obtained by fine-tuning the VILA model training is combined to determine the focus entity corresponding to the focus keyword in the video to be processed.

[0013] In a possible implementation, the step of inputting the video to be processed into a pre-trained second video multimodal large model to obtain a focused spatiotemporal scene graph of a focused person and a focused object included in the focused entity includes:

[0014] The second video multimodal large model pre-trained by the video to be processed is extracted by frame extraction to obtain a focused spatiotemporal scene graph of the focused person and the focused object included in the focused entity;

[0015] Based on the intersection-and-union ratio of the focused entities, each of the focused characters and each of the focused objects are determined, and a plurality of the focused spatiotemporal scene graphs including the interaction relationships between each of the focused characters and each of the focused objects are obtained. The focused spatiotemporal scene graphs include a unique identifier ID for each of the focused entities, which is used to track each focused entity on the timeline of the video.

[0016] In a possible implementation, the focused spatiotemporal scene graph is represented in a text form in JSON format, and the focused spatiotemporal scene graph includes frame information, node information, and edge information, the node information represents the focused entities and their positions, and the edge information represents the interaction relationship or spatial relationship between the focused entities.

[0017] In a possible implementation, the performing dialogue feedback on the interactive text according to the focused spatiotemporal scene graph includes:

[0018] Parsing the focused spatiotemporal scene graph to extract key video detail information and timing information;

[0019] The extracted information is input into a pre-trained dialogue generation model to generate a reply text to the interaction text.

[0020] In a possible implementation, the method further includes:

[0021] The first video multimodal large model and the second video multimodal large model are jointly trained, the first video multimodal large model and the second video multimodal large model share part of the training data, and the model parameters are continuously optimized through cross-validation.

[0022] In a second aspect, an embodiment of the present invention provides a structured video understanding device based on a multimodal large model, characterized in that it includes:

[0023] An acquisition module, used to acquire a video to be processed and an interactive text input by a user, wherein the interactive text is a user demand description text for the video to be processed;

[0024] The understanding module is used to extract keywords from the interactive text to obtain focused keywords, and determine the focused entities corresponding to the focused keywords in the video to be processed in combination with a pre-trained first video multimodal large model; input the video to be processed into a pre-trained second video multimodal large model to obtain a focused spatiotemporal scene graph of the focused characters and focused objects included in the focused entities; and provide dialogue feedback to the interactive text according to the focused spatiotemporal scene graph.

[0025] In a third aspect, an embodiment of the present invention provides a computer device, comprising a processor and a non-volatile memory storing computer instructions, wherein when the computer instructions are executed by the processor, the computer device executes the method described in the first aspect.

[0026] In a fourth aspect, an embodiment of the present invention provides a readable storage medium, wherein the readable storage medium includes a computer program, and when the computer program is executed, the computer device where the readable storage medium is located is controlled to execute the method described in the first aspect.

[0027] Compared with the prior art, the beneficial effects provided by the present invention include: adopting a structured video understanding method, device, computer equipment and readable storage medium based on a multimodal large model disclosed in the present invention, by obtaining the user's video to be processed and the interactive text describing the needs, then extracting keywords from the interactive text and combining the first video multimodal large model to determine the focused entity, and then inputting the video into the second video multimodal large model to obtain a focused spatiotemporal scene graph of the focused entity, and finally performing dialogue feedback on the interactive text based on this scene graph, so as to achieve efficient and accurate video content understanding and meet user needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can also be obtained based on these drawings without creative work.

[0029] Figure 1 A schematic diagram of the steps of a structured video understanding method based on a multimodal large model provided by an embodiment of the present invention;

[0030] Figure 2 A schematic block diagram of the structure of a structured video understanding device based on a multimodal large model provided by an embodiment of the present invention;

[0031] Figure 3 A schematic block diagram of the structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0033] The specific implementation modes of the present invention are described in detail below in conjunction with the accompanying drawings.

[0034] In order to solve the technical problems in the aforementioned background technology, Figure 1 A flow chart of a structured video understanding method based on a multimodal large model provided in an embodiment of the present disclosure is provided below. The structured video understanding method based on a multimodal large model is introduced in detail.

[0035] Step S201, obtaining a video to be processed and an interactive text input by a user, wherein the interactive text is a user demand description text for the video to be processed;

[0036] Step S201, extracting keywords from the interactive text to obtain focused keywords, and determining focused entities corresponding to the focused keywords in the video to be processed in combination with a pre-trained first video multimodal large model;

[0037] Step S201, inputting the video to be processed into a pre-trained second video multimodal large model to obtain a focused spatiotemporal scene graph of the focused person and focused object included in the focused entity;

[0038] Step S201: providing dialogue feedback to the interactive text according to the focused spatiotemporal scene graph.

[0039] In the embodiment of the present invention, for example, it is assumed that the server is a core component of an intelligent video analysis service platform. The user sends a request to the server through a client (such as a mobile phone APP or a web page). The user wants to know specific information in a video about the traffic conditions on the city streets, so he uploads a 30-second video file (the video to be processed) and enters the interactive text in the input box: "I want to know the interaction between the traffic police in blue uniforms and the car that ran a red light in the video."

[0040] After receiving the video file and interactive text sent by the user, the server will first check the format of the video file and perform preliminary integrity verification to ensure that the video can be read and processed normally. For the interactive text, the server will perform a simple encoding conversion (if necessary) to convert it into a format that can be processed internally for use in subsequent steps.

[0041] After receiving the interactive text, the server starts to extract keywords. First, it calls the pre-trained large language model (such as GPT-3.5) and the NLTK library to extract keywords.

[0042] For the large language model, the server will construct a specific prompt, such as "Please extract key nouns related to people, things, and behaviors from the following text: I want to know the interaction between the traffic police in blue uniform and the car that ran a red light in the video." Based on this prompt, the large language model may return keywords such as "traffic police", "blue uniform", "car", "running a red light", and "interaction".

[0043] At the same time, the NLTK library also starts working. The server preprocesses the interactive text, such as performing word segmentation, and then uses the part-of-speech tagging function of the NLTK library to identify nouns. The NLTK library may identify nouns such as "traffic police" and "car".

[0044] The server will merge and deduplicate the keywords extracted by the large language model or NLTK library, and finally obtain the focused keywords: "traffic police", "blue uniform", "car", "running a red light", and "interaction situation".

[0045] The server combines these focused keywords with a pre-trained first video multimodal large model (assumed to be a fine-tuned VILA model) to determine the focused entities.

[0046] The server inputs the focus keyword and related information of the video to be processed (such as the basic metadata of the video, including video duration, resolution, etc.) into the first video multimodal model. The model first analyzes the video and uses its visual feature recognition ability learned during the pre-training process to start searching for entities related to the focus keyword in each frame of the video.

[0047] For the keyword "traffic police", the model will search for people wearing blue uniforms (further precisely located by combining the keyword "blue uniforms") in the video, and determine their positions and related attributes in the video through the target detection algorithm. For the keyword "car", the model will identify the car object in the video, and based on the behavior keyword "running a red light", try to analyze whether the car's driving trajectory meets the characteristics of running a red light (for example, by comprehensively judging the traffic light status and the time when the vehicle passes the stop line).

[0048] After analysis by the model, the focused entities are identified as: the traffic policeman in blue uniform in the video (the specific location and appearance features are identified by the model) and the car that is judged to have run a red light (the location, color, model and other information of the car are also identified).

[0049] The server inputs the video to be processed into a pre-trained second video multimodal large model in a frame extraction manner (2 frames per second). This model can also be a model fine-tuned by a specific data set, such as the VILA model mentioned above or other suitable video multimodal large models.

[0050] When the video frames are input to the model, the model starts to analyze each frame. For the traffic police and car in the focus entity, the model uses its object detection ability to accurately locate the positions of the traffic police and the car in each frame. For example, in the first frame, the traffic police is located in the lower left corner of the picture (the position is represented by coordinate information, such as [10, 20, 30, 40], which indicates the coordinates of the upper left and lower right corners of a rectangular area of ​​the traffic police in the picture), and the car is located in the middle right position of the picture [100, 120, 150, 180].

[0051] As video frames continue to be input, the model begins to determine unique identifiers for the focused person (traffic police) and the focused object (car). Assume that the traffic police is assigned ID 1 and the car is assigned ID 2.

[0052] When constructing the focused spatiotemporal scene graph, the model needs to consider the interaction relationship and spatial relationship between entities. For spatial relationships, the model will analyze the relative position relationship between the traffic police and the car in each frame. For example, in a certain frame, the traffic police stood on the left side of the car (this spatial relationship was recorded). For interactive relationships, since the user mentioned "interaction situation", the model will pay special attention to whether there is a specific interactive behavior between the traffic police and the car. For example, starting from the 10th frame of the video, the traffic police made a gesture to signal the car to stop. After the model detects this action, it will record this interactive relationship in the focused spatiotemporal scene graph, and the edge information will be recorded as [1, "signal to stop", 2].

[0053] A focused spatiotemporal scene graph represented as text in JSON format might look like this:

[0054]

[0055] This focused spatiotemporal scene graph is continuously updated as the video frames are processed, completely recording the spatiotemporal information and interaction relationships of the focused entities in the video.

[0056] After receiving the generated focused spatiotemporal scene graph, the server first performs a parsing operation. The server reads the node information and edge information frame by frame according to the JSON format structure.

[0057] From the node information, the server obtains the position, category, and ID information of the traffic police and the car in each frame. For example, by reading the information in the above focused spatiotemporal scene graph, the server knows the initial position of the traffic police and the car in frame 1, and their position changes in frame 10. From the side information, the server obtains the interactive relationship between the traffic police and the car, such as the traffic police signaling the car to stop in frame 10.

[0058] The server inputs the parsed key video details (position changes of traffic police and cars, interactive relationships, etc.) and timing information (what happened in which frame) into the pre-trained dialogue generation model.

[0059] The dialogue generation model generates a response text based on this input information and the user's interactive text content. For example, the response text may be: "In the video, a traffic policeman in a blue uniform stands in the lower left corner of the screen, and the car is located in the middle right of the screen. In the 10th frame, the traffic policeman stands on the left side of the car and signals the car to stop."

[0060] This reply text accurately answers the user's question about the interaction between the traffic police and the car, and is generated based on an accurate understanding of the video content (reflected by focusing on the spatiotemporal scene graph). In this way, the server provides the user with an effective answer to the video content and meets the user's needs.

[0061] On the server side, the first video multimodal large model and the second video multimodal large model are jointly trained. The server prepares a large-scale data set, which contains a large number of videos and the corresponding interactive texts.

[0062] During the training process, the first video multimodal large model and the second video multimodal large model share some training data. For example, a part of the data is specifically used to train the first model's ability to determine the focused entity, and another part of the data is used to train the second model's ability to generate a focused spatiotemporal scene graph, but some intermediate data is shared, such as basic annotation information about entities in the video.

[0063] Continuously optimize the model parameters through cross-validation. For example, the server divides the dataset into multiple subsets, and each time uses a part of the subsets as the validation set and the rest as the training set. When using the first subset as the validation set, the second model is trained on the other subsets and then its performance is verified using the first subset, and vice versa. In this way, continuously adjust the model parameters to improve the accuracy and robustness of the model, so that the entire structured video understanding method based on the multimodal large model can better handle various video and interactive text inputs from users.

[0064] In the embodiment of the present invention, the extraction of the focus keywords from the interactive text can be implemented through the following examples.

[0065] Call the pre-trained large language model or the NLTK library to extract the focus keywords from the interactive text.

[0066] In the embodiment of the present invention, exemplarily, the server, as the core of the intelligent video understanding service, continuously listens for requests from users. When the user inputs the video to be processed and the interactive text (such as "I want to know what the kid holding a red balloon in the video is doing"), the server first receives and stores this information.

[0067] For the interactive text, the server needs to perform preprocessing for subsequent keyword extraction operations. The server will convert the interactive text from the format input by the user (which may be UTF-8 encoding, etc.) into the standard format for internal processing. At the same time, perform simple cleaning operations on the text, such as removing extra spaces and punctuation marks (but retain the punctuation marks that have an impact on semantics, such as the word "of" in "the kid holding a red balloon" is important for semantic understanding and cannot be removed).

[0068] The server calls the pre-trained large language model (taking GPT-3.5 as an example) for keyword extraction. The server constructs a request specifically for keyword extraction and includes the preprocessed interactive text in the request content. For example, construct a request like this: "Please perform keyword extraction on the following text, focusing on nouns related to people, objects, actions, etc.: I want to know what the kid holding a red balloon in the video is doing".

[0069] After receiving the request from the server, the large language model analyzes it according to its pre-trained algorithm and a large amount of corpus knowledge. It will identify the important elements in the text. For this example, the large language model may identify words such as "kid", "red balloon", and "doing what" as keywords. This is because the large language model has learned various language structures and semantic information during the pre-training process and can understand the importance of these words in describing a scene.

[0070] After receiving the keyword results returned by the large language model, the server will perform preliminary sorting. For example, the results returned by the large language model may contain some auxiliary explanations or other irrelevant format information. The server needs to remove this information and only retain the pure keyword content to obtain a preliminary keyword set, such as {"child", "red balloon", "what to do"}.

[0071] At the same time, the server uses the NLTK library for keyword extraction. First, the server further processes the preprocessed interactive text to make it conform to the input requirements of the NLTK library. This may include converting the text into a specific part-of-speech tagging format (such as splitting the text into a list of words). Then, the server initializes the NLTK library and loads related corpora and tools, such as part-of-speech taggers.

[0072] The NLTK library performs part-of-speech tagging on each word in the interactive text. For example, for the sentence "I want to know what the child holding the red balloon in the video is doing", the NLTK library may tag "child" as a noun, "red" as an adjective, "balloon" as a noun, and so on. Then, the server selects nouns as keyword candidates based on the part-of-speech tagging results. In this example, "child" and "balloon" are determined as keyword candidates.

[0073] The server further screens and sorts the keyword candidates determined by the NLTK library. Since "red" is an adjective describing "balloon", "red" will not be finally determined as a keyword when only focusing on nouns as keywords. Finally, a keyword set determined by the NLTK library is obtained, such as {"child", "balloon"}.

[0074] The server merges the keyword set {"child", "red balloon", "what to do"} obtained by the large language model with the keyword set {"child", "balloon"} obtained by the NLTK library. The merged keyword set becomes {"child", "red balloon", "what to do", "balloon"}.

[0075] Since "red balloon" and "balloon" have semantic overlap, the server performs a deduplication operation in order to obtain concise and accurate focused keywords. The server will deduplicate the merged keyword set according to certain semantic analysis rules (for example, "red balloon" is a more specific description of "balloon". If "red balloon" already exists, the broader keyword "balloon" can be removed), and finally obtain the focused keyword set {"children", "red balloon", "what to do"}.

[0076] In this way, the server calls the pre-trained large language model or NLTK library to extract keywords from the interactive text to obtain the focused keywords, which provides an important basis for the subsequent determination of the focused entity, the generation of the focused spatiotemporal scene graph, and the final dialogue feedback. This step fully utilizes the semantic understanding ability of the large language model and the natural language processing tools of the NLTK library to ensure the comprehensiveness and accuracy of keyword extraction.

[0077] Suppose the interaction text is "I want to know if there is a conversation between the man in the hat and the woman in the white dress in the video, and what the puppy next to them is doing."

[0078] Large language model analysis: After receiving this interactive text, the large language model will identify keywords such as "man in hat", "woman in white dress", "puppy", "conversation", and "what to do". This is because the large language model can understand the different entities described in the text and the possible relationships (conversation relationships) and actions (puppy's behavior) between them.

[0079] NLTK library processing: After NLTK library performs part-of-speech tagging on this interactive text, it determines that "man", "woman", and "puppy" are candidate noun keywords. After further screening, the descriptive "wearing a hat" and "wearing a white dress" are removed (because when determining the focus keywords, the more specific "man wearing a hat" and "woman wearing a white dress" have been identified by the large language model), and {"man", "woman", "puppy"} are obtained.

[0080] Merge and remove duplicates: The keywords obtained by the large language model or NLTK library are merged into {"man wearing a hat", "woman in a white dress", "puppy", "talk", "what to do", "man", "woman"}. After removing duplicates and based on semantic analysis, the final focus keywords are {"man wearing a hat", "woman in a white dress", "puppy", "talk", "what to do"}.

[0081] For example, the interactive text is “I want to see whether the young man running in the park in the video jumps over the bench.”

[0082] Large language model analysis: The large language model will identify keywords such as "young people", "park", "running", "bench", and "skip". These keywords cover important elements such as characters, scenes, and actions.

[0083] NLTK library processing: The NLTK library will determine "young people" and "bench" as noun keyword candidates.

[0084] Merge and remove duplicates: The merged keyword set is {"young people", "park", "running", "bench", "skip", "young people", "bench"}, and after removing duplicates, we get {"young people", "park", "running", "bench", "skip"}.

[0085] Through these scene examples of different types of interactive texts, it can be seen that when the server calls the large language model or NLTK library for keyword extraction, it can adapt to various complex interactive text contents and accurately extract the focused keywords, thus laying a solid foundation for the entire structured video understanding process based on the multimodal large model.

[0086] In an embodiment of the present invention, the determination of the focus entity corresponding to the focus keyword in the video to be processed by combining the pre-trained first video multimodal large model can be implemented through the following examples.

[0087] The first video multimodal large model obtained by fine-tuning the VILA model training is combined to determine the focus entity corresponding to the focus keyword in the video to be processed.

[0088] In an embodiment of the present invention, exemplarily, after completing keyword extraction, the server obtains a set of focused keywords such as {"child", "red balloon", "what to do"} (the previous example is continued here). The server will organize these keywords and convert them into an input format that can be understood by the first video multimodal large model (the fine-tuned VILA model). This may involve encoding the keywords or constructing a specific data structure. For example, organize the keywords into a dictionary form containing the keywords and their related attributes (such as part of speech, importance weight, etc., assuming a simple example here, the weights are all set to 1): [{"keyword":"child","part-of-speech":"noun","weight":1},{"keyword":"red balloon","part-of-speech":"noun","weight":1},{"keyword":"what to do","part-of-speech":"verb","weight":1}].

[0089] At the same time, the video to be processed is preprocessed for input into the VILA model. The server will convert the video format (if necessary) to ensure that the video encoding format, resolution, etc. meet the requirements of the VILA model. For example, the video is converted to a specific encoding format that the model can handle (such as H.264 encoding), and the video resolution is adjusted to the input size predefined by the model (such as 224x224 pixels). At the same time, the server will also extract some basic metadata of the video, such as video duration (such as 30 seconds), frame rate (such as 25fps), etc., and use these metadata together with the video data as part of the input.

[0090] The server inputs the organized focus keyword data structure and the preprocessed video to be processed (including metadata) into the fine-tuned VILA model. The VILA model is a multimodal model that can process video and text data. After fine-tuning for specific tasks (such as the video understanding task in this scenario) based on pre-training, it has the ability to analyze specific types of video content.

[0091] The VILA model first extracts features from the processed video. It uses the convolutional neural network (CNN) structure learned during pre-training to extract visual features from each frame of the video. For example, for the first frame of the video, the model identifies various visual elements in the image, such as the outline of the person, the shape of the object, and the color. At the same time, the model extracts text features from the input focus keywords and converts them into vector representations corresponding to the visual features of the video for matching and association analysis in subsequent steps.

[0092] Based on the extracted visual features of the video and the text features of the keywords, the VILA model begins to search for entities corresponding to the focused keywords in the video. For the keyword "child", the model will look for human objects that meet the characteristics of a child in each frame of the video. It will make judgments based on previously extracted visual features, such as the character's height, body shape (relatively small, consistent with the characteristics of a child), facial features, etc. At the same time, for the keyword "red balloon", the model will look for objects in the video frame that are red in color and have a shape that meets the characteristics of a balloon. In this process, the model will comprehensively consider multiple clues in the video, such as the color distribution of the object, texture and other features to accurately identify the entity.

[0093] After a comprehensive analysis of the video, the VILA model identifies the entities corresponding to the focus keywords. For example, in the 5th frame of the video, a person with short height and facial features consistent with the image of a child is found, and the child is holding a red balloon. The model identifies the child as the focus entity corresponding to the keywords "child" and "red balloon". At the same time, since this is a query about the child's behavior ("what to do"), the model will continue to track the child's activities in the video.

[0094] Once the focused entity is determined, the VILA model will annotate it. The annotations include the entity category (such as "child"), location information in the video (for example, through the coordinates in each frame, the position of the child in the 5th frame may be expressed as [100, 200, 150, 250], indicating the coordinates of the upper left and lower right corners of a rectangular area of ​​the child in the picture), and the relationship with other entities (here only the holding relationship between the child and the red balloon is involved). This annotation information will be used in subsequent steps, such as constructing a focused spatiotemporal scene graph.

[0095] Assume that the interaction text is "I want to know whether the man in the hat and the woman in the white dress in the video are talking to each other, and what the puppy next to them is doing", after keyword extraction, the focused keywords {"man in the hat", "woman in the white dress", "puppy", "talking", "doing"} are obtained.

[0096] Model input and analysis: The server inputs these keywords and pre-processed videos into the VILA model. The VILA model extracts features from the video and searches for people that meet the characteristics of "man wearing a hat" in the video. It may identify people based on the shape and color of the hat and the man's appearance; for "woman in a white dress", it will locate based on the color of the dress and the woman's appearance; for "puppy", it will search based on the appearance characteristics of the animal.

[0097] Entity identification and labeling: In different frames of the video, the model identifies three focused entities: a man wearing a hat, a woman wearing a white dress, and a puppy. For the man wearing a hat, the model labels his position in the video (e.g., [200, 300, 250, 350] in the 10th frame) and the features of the hat (e.g., black color, baseball cap style, etc.); for the woman, the details of her dress (e.g., white, long, etc.) and location information are annotated; for the puppy, its breed characteristics (e.g., small dog, white hair, etc.) and location are annotated. At the same time, the model pays special attention to whether there is an interactive conversation between the man and the woman, and the behavior of the puppy ("what to do").

[0098] For example, the interactive text is "I want to see if the young man running in the park in the video jumps over the bench", and the focus keywords are {"young man", "park", "running", "bench", "jump"}.

[0099] Model input and analysis: After the server inputs keywords and videos into the VILA model, the model searches for people with young appearance and running in the video (determining whether they are running by analyzing the person's movements and postures), and also searches for park scenes (possibly by identifying typical elements in the park such as trees and grass to determine the scene) and benches (based on the shape of the benches and their position in the scene).

[0100] Entity identification and labeling: The model identifies the young man and the bench as the focused entities. For the young man, its position in different frames (such as [300, 400, 350, 450] in the 15th frame) and running speed (estimated by analyzing the position change of the person in consecutive frames) are labeled; for the bench, its position, length, color and other features are labeled. The model also focuses on analyzing whether the young man jumps over the bench (corresponding to the keyword "jump").

[0101] Through the detailed examples of these different scenarios, we can see how the server combines the fine-tuned VILA model to accurately determine the focused entities in the video to be processed based on the focused keywords, providing key entity information for the subsequent structured video understanding process based on the multimodal large model.

[0102] In an embodiment of the present invention, the step of inputting the video to be processed into a pre-trained second video multimodal large model to obtain a focused spatiotemporal scene graph of a focused person and a focused object included in the focused entity may be implemented through the following example.

[0103] The second video multimodal large model pre-trained by the video to be processed is extracted by frame extraction to obtain a focused spatiotemporal scene graph of the focused person and the focused object included in the focused entity;

[0104] Based on the intersection-and-union ratio of the focused entities, each of the focused characters and each of the focused objects are determined, and a plurality of the focused spatiotemporal scene graphs including the interaction relationships between each of the focused characters and each of the focused objects are obtained. The focused spatiotemporal scene graphs include a unique identifier ID for each of the focused entities, which is used to track each focused entity on the timeline of the video.

[0105] In the embodiment of the present invention, illustratively, after receiving the video to be processed, the server performs a frame extraction operation on the video to be processed according to a prescribed frame extraction method (extracting 2 frames per second). For example, for a video with a duration of 30 seconds, a total of 60 frames of images will be extracted. During the frame extraction process, the server will accurately record the timestamp of each frame, and this timestamp will be used to represent the information of the time dimension when constructing the focused spatiotemporal scene graph later.

[0106] The server inputs the extracted video frames into the pre-trained second video multimodal model in sequence. This model is also a video multimodal model that has been fine-tuned with a specific data set and can process video frames and information related to the focused entity. For each frame of the image, the model performs a series of processing, including visual feature extraction, analysis related to the focused entity, and other operations.

[0107] Taking the video containing the child and the red balloon mentioned above as an example, for the focused entity child and the red balloon, the model first initially locates them in each frame. Using the target detection algorithm, the model identifies the approximate area of ​​the child in the picture (for example, represented by a rectangular box), and also determines the position of the red balloon. Then, the model extracts the visual features of the child and the red balloon, such as the child's appearance features (facial expressions, hairstyle, etc.), the color and shape features of the red balloon, etc.

[0108] Inter-frame comparison: As video frames are continuously input, the model begins to calculate the intersection over union (IoU) of the focused entities between adjacent frames. For the child, the focused character, assume that the child is located in the coordinate area [100, 200, 150, 250] in the first frame and in [105, 203, 153, 253] in the second frame. The model calculates the intersection area and union area of ​​these two rectangular areas (representing the position of the child in the two frames) to obtain the IoU. If the IoU exceeds a preset threshold (such as 0.5) and the visual features such as the appearance of the child in the two frames are similar, the model determines that the entities in the two frames are the same child.

[0109] Multi-entity comparison: For the red balloon, the focus object, the intersection-and-union calculation is also performed. In some frames, there may be multiple objects with similar balloon shapes. The model accurately distinguishes the red balloon as the focus object through the intersection-and-union calculation and visual feature comparison. For example, in the third frame, there is a yellow balloon with a similar color to the red balloon, but through the intersection-and-union calculation and the comprehensive judgment of color, shape and other features, the model can accurately identify the position and range of the red balloon.

[0110] Focus person node: For the focus person determined to be the same child, the model assigns a unique identifier ID to it, assuming that the child's ID is 1. In the focus spatiotemporal scene graph, the node information corresponding to each frame records the child's category ("child"), ID (1) and location information. For example, the node information in the first frame is {"category":"child","id":1,"location information":[100,200,150,250]}.

[0111] Focused object node: For the red balloon, assume its ID is 2. The node information in the first frame is {"category":"red balloon","id":2,"location information":[120,220,130,230]}. In this way, each frame of the focused spatiotemporal scene graph contains the node information of the focused person and the focused object.

[0112] Relationship analysis: When analyzing video frames, the model will focus on whether there is an interactive relationship between the focused person and the focused object. For example, if a child is always holding a red balloon, this holding relationship is an interactive relationship. In the focused spatiotemporal scene graph, this relationship is represented by side information. In the side information of each frame (assuming it starts from frame 1), [1,"holding",2] is recorded, indicating that the child with ID 1 is holding the red balloon with ID 2.

[0113] Complex relationship processing: In more complex scenarios, such as the relationship between multiple focused characters and objects. Assuming that the interaction text mentions the interaction between a man (focused character, ID 3) and a child (ID 1) in the video, the model will detect their behavior when analyzing the video frames. If in a certain frame (such as the 10th frame), it is found that the man handed a toy to the child (the new focused object, ID 4), then the interaction relationship records such as [3, "handed", 1] and [1, "took", 4] will be added to the side information of the 10th frame.

[0114] Time dimension integration: As the video frames are processed, the server integrates the node information and edge information corresponding to each frame to form a complete focused spatiotemporal scene graph. The focused spatiotemporal scene graph represented in JSON format may be as follows:

[0115]

[0116]

[0117] Examples of building focused spatiotemporal scene graphs in different scenarios:

[0118] 1. Scenario 1: Relationship between vehicles and pedestrians in traffic scenes:

[0119] Video frame extraction and input: Assume that the video to be processed is a video of a traffic intersection, and the interactive text is "I want to know the relationship between the pedestrian in yellow clothes and the car that ran a red light in the video." The server extracts the video frame and inputs it into the second video multimodal large model.

[0120] Entity determination: The focused entities are the pedestrian in yellow clothes (ID 1) and the car running a red light (ID 2). The model accurately identifies the positions of pedestrians and cars in each frame by calculating the intersection ratio of union. For example, in the first frame, the pedestrian is located at [50, 100, 80, 130] and the car is located at [200, 300, 250, 350].

[0121] Scene graph construction: When constructing the focused spatiotemporal scene graph, the node information records the category, ID, and location of pedestrians and cars. The edge information reflects the relationship between them. If a car is found approaching a pedestrian in the fifth frame and the pedestrian makes an evasive action, the edge information will record [2,"approach",1] and [1,"avoid",2].

[0122] 2. Scene 2: Character interaction in indoor scenes:

[0123] Video frame extraction and input: For a video of an indoor party, the interactive text is "I want to know if there is any interaction between the man wearing glasses and the woman holding a cake in the video." The server extracts frames and inputs them into the model.

[0124] Entity determination: The focus characters are the man wearing glasses (ID 1) and the woman holding a cake (ID 2). The model determines their positions in each frame, such as in the first frame the man is at [100, 150, 130, 180] and the woman is at [180, 230, 210, 260].

[0125] Scene graph construction: The node information of the focused spatiotemporal scene graph includes the category, ID, and location of the characters. The edge information records the interaction relationship. If a man is found smiling at a woman in the 10th frame, the edge information will increase [1, "Smiling gesture", 2].

[0126] Through the detailed examples of these different scenarios, we can see how the server uses frame extraction to input the video to be processed into the second video multimodal large model, determines the focused people and objects based on the intersection and union ratio of the focused entities, and constructs a focused spatiotemporal scene graph containing interactive relationships, thereby providing rich structured information for subsequent dialogue feedback.

[0127] In an embodiment of the present invention, the focused spatiotemporal scene graph is represented in a text form in JSON format, and the focused spatiotemporal scene graph includes frame information, node information and edge information, the node information represents the focused entities and their positions, and the edge information represents the interaction relationship or spatial relationship between the focused entities.

[0128] In an embodiment of the present invention, illustratively, when constructing the focused spatiotemporal scene graph, the server first determines to organize the data in JSON format. JSON (JavaScript Object Notation) is a lightweight data exchange format that is very suitable for representing such structured information. The entire focused spatiotemporal scene graph is wrapped in a large JSON object.

[0129] Frame number: For each frame in the video, there is a corresponding key-value pair to represent the information of the frame. For example, "Frame 1", "Frame 2", etc. are used as keys, and their corresponding values ​​are objects containing node information and edge information. This frame number is determined according to the playback order of the video, starting from the start frame of the video and increasing in sequence.

[0130] The significance of frame order: The order of frames is very important in representing the temporal information of the video. It can reflect the state changes of the focused entity during the video playback process. For example, in a surveillance video, the first frame may show a person (focused entity) standing at the door (represented by the position information in the node information), and as the frames progress, the person may start to move in subsequent frames and interact with other entities (such as doors, surrounding objects, etc.) (represented by side information).

[0131] Character entity: Take a video of an office scene as an example. Assume that the interactive text is "I want to know the activity trajectory of the employee holding the file in the office and the relationship with the surrounding objects." In this scene, there is a focused character who is the employee holding the file. In the focused spatiotemporal scene graph, for this employee (assuming its unique identifier ID is 1), the node information at a certain frame (such as the 5th frame) may be as follows:

[0132] {"category":"employee","id":1,"location information":[100,200,150,250]}. Here, "category" specifies that this entity is "employee", "id" is used to uniquely identify this entity on the timeline of the entire video, and "location information" is in the form of coordinates (here it is assumed to be coordinates relative to the video screen, such as the coordinates of the upper left corner and lower right corner of a rectangular area of ​​the employee in the screen) to indicate the employee's position in the screen.

[0133] Item Entity: In this office scene, there are some focus items, such as desks (assuming ID is 2) and filing cabinets (assuming ID is 3). For desks, the node information in the 5th frame may be {"category":"desk","id":2,"location information":[300,100,400,200]}; for filing cabinets, the node information may be {"category":"filing cabinet","id":3,"location information":[500,150,600,300]}.

[0134] Movement of people: As the video plays, the position of the focused person (employee) changes. For example, in the 10th frame, the employee walks to the desk, and its node information becomes {"category":"employee","id":1,"location information":[250,150,300,200]}. This change in location information accurately reflects the employee's movement trajectory in the video.

[0135] The state of the object remains unchanged: For object entities such as desks and filing cabinets, in most cases, their positions are relatively fixed in the video (unless there are special circumstances, such as being moved, etc.). So in different frames, except for the different frame numbers, their node information may remain unchanged in terms of the entity's category, ID, and location information (assuming that there is no position movement or other changes). For example, in the 10th frame, the node information of the desk is still {"category":"desk","id":2,"location information":[300,100,400,200]}.

[0136] Interaction between people and objects: Taking the above office scene as an example, in the 10th frame, when the employee walks to the desk, there may be an interactive action of placing a file on the desk. In the focused spatiotemporal scene graph, this interactive relationship is represented by side information. The side information is in the form of a triple. The side information in the 10th frame will contain [1,"Put the file on", 2], where 1 is the employee's ID and 2 is the desk's ID. "Put the file on" describes the interactive relationship between the employee and the desk.

[0137] Interaction between characters: Assume that there is another employee in the office (ID is 4), and at a certain moment in the video (such as frame 15), the two employees have a conversation. Then the side information of frame 15 will add a record like [1,"Talking with...", 4], indicating the conversation relationship between employee ID 1 and employee ID 4.

[0138] Relative position relationship: In addition to the interaction relationship, the side information can also represent the spatial relationship between the focused entities. For example, in the 5th frame, there is a spatial relationship between the employee (ID is 1) and the file cabinet (ID is 3), and the employee is to the left of the file cabinet. This spatial relationship can be represented in the side information as [1,"to the left of...",3]. The record of this spatial relationship helps to more comprehensively describe the layout relationship between the entities in the scene.

[0139] Complex spatial relationships: In some scenarios, there may be complex spatial relationships between multiple entities. For example, in a conference room scenario, there is a conference table (ID is 5), chairs (multiple, assuming one of them has ID 6), and a projector (ID is 7). In the first frame, the chairs are placed around the conference table, and the projector is at one end of the conference table. The edge information may contain triples such as [6,"around",5], [7,"at...one end",5], etc., which represent spatial relationships.

[0140] The following is an example of a JSON representation of a relatively complete focused spatiotemporal scene graph constructed based on the above office scene:

[0141]

[0142]

[0143]

[0144] This focused spatiotemporal scene graph completely records the states (node ​​information) of focused entities (employees, desks, filing cabinets, etc.) in different frames in the office scene and the relationships between them (edge ​​information), expressed in a structured, easy-to-understand and process JSON format, providing a rich information basis for the server to provide subsequent dialogue feedback based on the user's interaction text.

[0145] In the embodiment of the present invention, the dialogue feedback on the interactive text according to the focused spatiotemporal scene graph may be implemented through the following examples.

[0146] Parsing the focused spatiotemporal scene graph to extract key video detail information and timing information;

[0147] The extracted information is input into a pre-trained dialogue generation model to generate a reply text to the interaction text.

[0148] In the embodiment of the present invention, for example, it is assumed that we continue to use the example of the focused spatiotemporal scene graph of the office scene video mentioned above.

[0149] After receiving this focused spatiotemporal scene graph (expressed in JSON format), the server begins to parse it.

[0150] Parsing of frame information: The server first reads the information of each frame in sequence. Starting from "Frame 1", the server knows that this is the starting state of the video. In this frame, the node information shows that there is an employee (id=1, location information is [100,200,150,250]), a desk (id=2, location information is [300,100,400,200]) and a filing cabinet (id=3, location information is [500,150,600,300]). The edge information shows that the employee is to the left of the filing cabinet. Through this parsing, the server obtains the key information of the initial position of each focused entity at the beginning of the video and the spatial relationship between them.

[0151] Parsing of entity movement and relationship changes: As subsequent frames are parsed, such as "frame 10", the server finds that the employee's position changes to [250, 150, 300, 200], and the side information increases to [1,"Put the file here", 2]. This indicates that the employee moved to the vicinity of the desk in frame 10 and placed the file there. The server can clearly extract this change in entity position and the newly generated interaction relationship, which are important video detail information.

[0152] Integration of timing information: By analyzing each frame of information in sequence, the server constructs the timing information of the entire video. For example, it knows that the employee is first on the left side of the file cabinet, then moves to the desk and places the file, and finally (assuming there are more actions in subsequent frames) other interactions may occur. This timing information is critical to accurately answer the user's interaction text.

[0153] Let’s take a look at an example of a traffic scene video. The interactive text is “I want to know how the car that ran the red light interacted with the surrounding vehicles and pedestrians.”

[0154] Analyze the initial status of vehicles and pedestrians: In the focused spatiotemporal scene graph, the node information of "Frame 1" shows the position information of the car (assuming id = 1), other vehicles (such as id = 2, id = 3, etc.) and pedestrians (assuming id = 4, id = 5, etc.). The side information may show some initial spatial relationships, such as the front, back, left, right and right position relationship between the car and other vehicles, the distance relationship between the car and nearby pedestrians, etc.

[0155] Analyzing the interactive relationship in traffic events: In subsequent frames, the car may run a red light. The server analyzes that around "frame 10", the car crosses the stop line (judged by the change in the car's position information). At the same time, the side information may show that the car has made an emergency avoidance move with other vehicles (such as [1,"leading to... emergency avoidance", 2]) and is dangerously close to pedestrians (such as [1,"close", 4]). The server accurately extracts the key video details of the interactive relationship between the car and the surrounding vehicles and pedestrians when running a red light from these frame information, and constructs complete time sequence information based on the order of the frames.

[0156] Input to the dialogue generation model: The server extracts information from the office scene video focusing on the spatiotemporal scene graph, including the employee's position changes, the interaction relationship with the desk and filing cabinet, and the timing information of the entire video, and inputs it into the pre-trained dialogue generation model. This dialogue generation model is trained with a large amount of text-video data pairs and can generate reasonable response text based on the input video-related information.

[0157] Generate reply text: If the interaction text is "What did the employee holding the file do near the desk?", the dialogue generation model generates reply text based on the input information: "In the video, the employee holding the file is initially on the left side of the file cabinet, then moves to the desk, and places the file on the desk in the 10th frame." This reply accurately answers the user's question about the employee's behavior near the desk and is generated based on the information extracted after parsing the focused spatiotemporal scene graph.

[0158] Input into the dialogue generation model: For traffic scene videos, the server inputs the extracted interaction information and time sequence information between the car and surrounding vehicles and pedestrians when it runs a red light into the dialogue generation model.

[0159] Generate reply text: If the interaction text is "Did the car that ran the red light affect other vehicles and pedestrians?", the dialogue generation model may generate a reply: "Yes, when the car ran the red light and crossed the stop line around the 10th frame, it caused the vehicle No. 2 to make an emergency avoidance and approached the pedestrian No. 4, which had an impact on the surrounding vehicles and pedestrians."

[0160] Data source: The server obtains data from a large number of videos and corresponding interactive text datasets for model training. These videos cover various scenes, such as indoor scenes, outdoor scenes, traffic scenes, etc. The interactive texts are various questions raised in response to these videos, such as questions about character behavior, object relationships, etc.

[0161] Data preprocessing: For video data, the server will perform preprocessing operations such as format unification and resolution adjustment to make it suitable for the input requirements of the model. For interactive text, it will perform preprocessing such as keyword extraction and semantic analysis to convert it into a format that the model can understand.

[0162] Determine shared data: Select a portion of the prepared training data as shared data between the first video multimodal model and the second video multimodal model. For example, some basic video feature annotation data (such as annotations of common objects in the video, simple character action annotations, etc.) are helpful for both models to understand the video content, so they are set as shared data.

[0163] Use of shared data: When the first video multimodal large model is trained to determine the focused entity, the video feature annotations in the shared data can help the model better identify the key elements in the video, thereby accurately determining the focused entity corresponding to the focused keyword. Similarly, when the second video multimodal large model constructs the focused spatiotemporal scene graph, these basic annotation information in the shared data helps the model more accurately locate the focused people and objects, and analyze the relationship between them.

[0164] Divide data into subsets: The server divides the entire training data (including shared data and independent data) into multiple subsets. For example, the data is divided into three subsets A, B, and C.

[0165] Cross-validation process:

[0166] When the first video multimodal large model is trained with A and shared data as training sets and B as validation set, the second video multimodal large model is trained with B and shared data as training sets and A as validation set. In this way, the two models are trained under different training-validation combinations.

[0167] During the training process, the model adjusts its own parameters based on the performance on the validation set (e.g., accuracy, recall, and other evaluation indicators). For example, if the First Video Multimodal Large Model has a low accuracy rate for focusing entities on validation set B, the model will adjust internal neural network weights and other parameters to improve the ability to identify focused entities.

[0168] As the number of training rounds increases, the cross-validation process is repeated, using different subset combinations for training and validation each time. For example, in the next round, the first video multimodal large model can use B and shared data as the training set and C as the validation set, and the second video multimodal large model can use C and shared data as the training set and B as the validation set.

[0169] Continuous optimization: Through continuous cross-validation, the parameters of the two models are continuously optimized. This joint training method enables the first video multimodal large model and the second video multimodal large model to work better together, improving the performance of the entire structured video understanding method based on multimodal large models. For example, after multiple cross-validation trainings, the first video multimodal large model can more accurately determine the focused entity and provide more precise input for the second video multimodal large model, thereby making the focused spatiotemporal scene graph generated by the second video multimodal large model more accurate, ultimately improving the accuracy of dialogue feedback.

[0170] In the embodiments of the present invention, the following implementation modes are also provided.

[0171] The first video multimodal large model and the second video multimodal large model are jointly trained, the first video multimodal large model and the second video multimodal large model share part of the training data, and the model parameters are continuously optimized through cross-validation.

[0172] In the embodiment of the present invention, illustratively, in order to jointly train the first video multimodal large model and the second video multimodal large model, the server first needs to collect a large amount of video and corresponding interactive text data. These data come from a wide range of sources, such as obtaining videos from public video datasets (such as Kinetics-400, UCF-101, etc.), collecting interactive texts from manually annotated question-answering datasets or from various question-answering platforms through web crawlers.

[0173] Video data preprocessing: For the collected video data, the server needs to perform a series of preprocessing operations. The first is format conversion, which converts videos of different formats (such as MP4, AVI, etc.) into a format that the model can process, such as converting all videos to H.264 encoding format. Then, adjust the resolution of the video, because videos from different sources may have different resolutions. The server adjusts it to a resolution suitable for the model input, such as 224x224 pixels or other sizes determined according to the model requirements. At the same time, the server extracts the metadata of the video, such as frame rate, duration, video encoding parameters, and other information, and associates this information with the video data for use as auxiliary information in the training process.

[0174] Preprocessing of interactive text data: For interactive text data, the server first performs text cleaning operations to remove special characters, extra spaces and other elements that affect text processing. Then it performs lexical analysis, such as word segmentation of the text and breaking down each sentence into words or phrases, which helps with subsequent keyword extraction and semantic understanding. Next, the server will perform part-of-speech tagging on the interactive text to determine the part of speech of each word (such as noun, verb, adjective, etc.), which is very important for understanding the semantic structure of the text. In addition, the server will also classify and annotate the interactive text according to predefined rules, such as classifying the text according to the type of question (such as questions about entity relationships, questions about entity behaviors, etc.), so as to provide more targeted supervision information for the model during the training process.

[0175] The server determines which data in the sorted video and interactive text data are used as shared data for the first video multimodal large model and the second video multimodal large model. The selection of shared data is based on the analysis of the functional requirements of the two models. For example, the annotation data of basic visual elements in the video (such as common object categories, basic character movements, etc.) are very suitable as shared data. Because the first video multimodal large model needs to identify these basic elements when determining the focused entity, the second video multimodal large model also needs to accurately identify the focused people and objects and the relationship between them based on these basic elements when constructing the focused spatiotemporal scene graph.

[0176] Taking a dataset containing indoor scene videos as an example, the video annotations in the shared data may include annotation information for furniture (such as tables, chairs, sofas, etc.) and characters (basic actions such as standing, sitting, and walking) that appear in the video. These annotation information is stored in a structured form, such as storing object categories, location coordinates, and character actions in each video frame in JSON format. For the shared data of the interactive text part, there may be some general question templates related to basic visual elements, such as "Where is the table in the video?" "What actions is the character doing?" etc. These shared data are organized into a special dataset for use by the two models during training.

[0177] The server divides the entire training dataset (including shared data and independent data) into multiple subsets. For example, assuming there are 1,000 video-interaction text data pairs in total, the server divides them into four subsets: A, B, C, and D, each containing 250 data pairs.

[0178] First Video Multimodal Large Model Training and Verification:

[0179] Training set selection: The server selects subset A and shared data as the training set for the first video multimodal large model. In this training set, the model learns to determine the focus entity based on the keywords in the video content and the interactive text. For example, for an indoor scene video, when the interactive text mentions "find the person holding the cup", the model learns to identify the person holding the cup in the video as the focus entity by learning the video-text pairs in the training set.

[0180] Validation set selection and evaluation: The server uses subset B as the validation set for the first video multimodal large model. During the validation process, the model processes the video-interaction text pairs in the validation set based on the knowledge learned on the training set to determine the focused entities. The server then calculates evaluation indicators such as accuracy and recall by comparing the focused entities determined by the model with the manually annotated correct focused entities. For example, if the model correctly identifies 80% of the manually annotated focused entities, then the accuracy is 80%. If the focused entities identified by the model cover 90% of the entities that should be identified, then the recall is 90%.

[0181] Second video multimodal large model training and verification:

[0182] Training set selection: The server selects subset B and shared data as the training set for the second video multimodal large model. In this training set, the model learns to construct a focused spatiotemporal scene graph based on the video content and the already determined focused entities. For example, for the focused entity (a person holding a cup) in the above indoor scene video, the model learns to accurately track the position of this person in different frames of the video and identify his relationship with surrounding objects (such as tables, chairs, etc.), and constructs a focused spatiotemporal scene graph containing node information (category, location, ID of people and objects) and edge information (interaction relationship or spatial relationship between people and objects).

[0183] Validation set selection and evaluation: The server uses subset A as the validation set for the second video multimodal large model. During the validation process, the model constructs a focused spatiotemporal scene graph for the video-interaction text pairs in the validation set based on the knowledge learned in the training set. The server then compares the focused spatiotemporal scene graph generated by the model with the manually annotated correct focused spatiotemporal scene graph. The comparison includes the accuracy of node information (such as whether the category and position of the entity are correct) and the accuracy of edge information (such as whether the relationship between people and objects is correct). The performance indicators of the model are determined by calculating the matching degree of these contents, such as the structural similarity index (SSIM).

[0184] First Video Multimodal Large Model Parameter Adjustment: If the First Video Multimodal Large Model has a low accuracy on validation set B, for example, less than 70%, the server will analyze the types of errors the model makes when processing the data in the validation set. If it is found that the focus entity is incorrectly determined due to inaccurate recognition of certain object categories (such as small decorations), the server will adjust the parameters of the neural network layer related to object category recognition in the model. For example, the weights of certain convolution kernels in the convolutional neural network (CNN) may be adjusted so that the model can better identify these easily confused object categories, thereby improving the accuracy of focus entity determination.

[0185] Second video multimodal large model parameter adjustment: For the second video multimodal large model, if the structural similarity index on the validation set A is low, for example, lower than 0.6, the server will check the model for problems in constructing the focused spatiotemporal scene graph. If it is found that the judgment of the interaction relationship between people and objects is inaccurate (such as judging a "close" relationship as a "far" relationship), the server will adjust the parameters of the neural network module used to analyze the interaction relationship in the model. For example, adjust the weight parameters in the long short-term memory network (LSTM) or the attention mechanism (Attention) module to improve the model's understanding and judgment of the interaction relationship.

[0186] Dataset selection for the next round of cross-validation: In the second round of cross-validation, the server changes the selection of training and validation sets. The first video multimodal large model uses subset C and shared data as training sets, and subset D as validation sets; the second video multimodal large model uses subset D and shared data as training sets, and subset C as validation sets. In this way, the model is trained and validated on different data subsets to ensure that the model can generalize to different types of video-interactive text data.

[0187] Continuous optimization process: As the number of cross-validation rounds increases, the first video multimodal large model and the second video multimodal large model continuously adjust their parameters. For example, after multiple rounds of cross-validation, the accuracy of the first video multimodal large model in determining the focused entity may be improved to more than 90%, and the structural similarity index of the second video multimodal large model in constructing the focused spatiotemporal scene graph may be improved to more than 0.8. This continuous optimization process enables the two models to work better together after joint training, improving the performance of the entire structured video understanding method based on the multimodal large model.

[0188] Surveillance scenario: In the joint training scenario of surveillance videos, the interactive text may be about the activity trajectory of a specific person (such as a suspect wearing clothes of a specific color) in the surveillance area and its relationship with the surrounding environment (such as entrances and exits, vehicles, etc.).

[0189] Training process: During the joint training process, the shared data includes annotations of common objects in monitoring scenarios (such as cameras, fences, etc.) and basic human actions (such as walking, running, etc.). Through cross-validation, the first video multimodal large model learns to accurately identify the suspect as a focused entity, and the second video multimodal large model learns to construct a focused spatiotemporal scene graph that includes the relationship between the suspect and the surrounding environment. For example, on the validation set, the first video multimodal large model can accurately find suspects in complex monitoring scenes, and the second video multimodal large model can accurately construct the spatial relationship and interaction relationship between the suspect and the entrances and exits and vehicles at different times (such as approaching the entrances and exits, entering the vehicle, etc.).

[0190] Optimization results: After multiple rounds of joint training, the performance of the two models has been significantly improved when processing video-interaction text pairs related to surveillance scenarios. In practical applications, it can more quickly and accurately answer users' questions about the content of surveillance videos, such as "Did the suspect pass by a certain car?" "Which entrance or exit did the suspect appear near?" etc.

[0191] Educational scenarios: In the scenario of educational videos (such as online course videos), the interactive text may be about the interaction between teachers and students in the classroom, and the use of specific teaching tools (such as blackboards, projectors, etc.).

[0192] Training process: The shared data includes the annotations of common characters (teachers, students) in educational scenarios, teaching tools, and some basic interactive actions (such as asking questions, answering, writing, etc.). During the cross-validation process, the first video multimodal large model accurately determines the focused entities such as teachers and students, and the second video multimodal large model constructs a focused spatiotemporal scene graph that includes the relationship between teachers, students, and teaching tools. For example, on the validation set, the model can accurately identify the interactive relationship between the teacher and the students when using a projector to display content at a certain moment (such as asking students questions about the projected content).

[0193] Optimization results: As joint training progresses, the two models can provide more detailed and accurate answers when processing video-interaction text pairs related to educational scenarios. For example, for the interactive text "How do students react when the teacher writes on the blackboard?", the model can accurately answer based on the optimized performance, describing the students' concentration, whether they have questions, etc.

[0194] Through this joint training method, the first video multimodal large model and the second video multimodal large model share part of the training data and continuously optimize the model parameters through cross-validation, thereby improving the performance and accuracy of the entire structured video understanding method based on the multimodal large model in different scenarios.

[0195] In order to more clearly describe the solution provided by the embodiment of the present invention, a relatively complete implementation method is provided below.

[0196] Pre-definition:

[0197] Scene graph: is a structured data representation used to describe objects in an image or scene and the relationships between them. Scene graphs are usually represented in the form of a "graph", where "graph" refers to a topological structure, and a "graph" is represented by nodes and edges. The nodes in the scene graph represent objects or entities in the image, such as people, cars, trees, etc. Each node contains category information and location information (usually represented by a bounding box). The edges in the scene graph represent the relationship between two nodes. This relationship can be a spatial relationship (such as up, down, left, and right) or an interactive relationship (such as "holding").

[0198] Spatiotemporal scene graph: The above scene graph is for static images. In order to expand the scene graph to video analysis, there is a spatiotemporal scene graph. Compared with the scene graph, the spatiotemporal scene graph has an additional dimension of time. It is necessary to track the position and relationship changes of all nodes in the video. Therefore, different nodes need to record an id that uniquely represents the node while recording the category, which runs through the entire timeline (similar to the id in the video tracking task). Different objects have different ids, and the id of the same object in this video remains unchanged.

[0199] Text-based spatiotemporal scene graph: The spatiotemporal scene graph is represented in plain text. Although it is in text form, it can be represented in the common JSON format and has a structured form, for example:

[0200] {

[0201] Frame 1: {nodes: [{category: person, id: 1, location information: [11, 22, 33, 44]}, {category: tree, id: 2, location information: [15, 25, 35, 45]}]}], edges: [[1, "leaning", 2]]

[0202] }

[0203] The "edge" here is represented by a triple. The triple in the above example means that the entity with id 1 is next to the entity with id 2.

[0204] Focused spatiotemporal scene graph: Traditional scene graphs usually strive to include all entities in the image, while the focused spatiotemporal scene graph defined here means that the spatiotemporal scene graph only focuses on all entities that the user cares about and entities that have interactive relationships with these entities, while ignoring other entities in the video (which can be regarded as background).

[0205] Note: The contents not described in detail belong to the prior art known to the professionals in this field. Specific embodiment:

[0207] Note: This specific embodiment refers to the step of using each module for reasoning after all modules have been trained.

[0208] The present invention discloses a structured video understanding method based on a multimodal large model. The overall process of an example that can be specifically implemented is as follows: Figure 1 As shown. Taking a video and the user's question text as input, it goes through three modules in sequence, namely the focused information generation module, the focused spatiotemporal scene graph generation module and the dialogue generation module, and finally generates a reply text with high accuracy. The specific implementation steps are:

[0209] Step 1, obtain a text, which comes from the user's input, usually the user's instructions, indicating completing a task or seeking some help. Obtain a video, which also comes from the user's input. The user's input text is related to the input video. The user expects to obtain some information or results related to the input video through this method.

[0210] Step 2: Input the video and text obtained in step 1 into the focus information generation module. The main function of this module is to output the people or objects worth paying attention to in the video in the form of text based on the text and video input by the user. Different attributes are used to represent different people or objects, such as "a person wearing green clothes" and "a person wearing red clothes" to distinguish two different people.

[0211] Step 3, input the video obtained in step 1 and the focus information obtained in step 2 into a focus spatiotemporal scene graph generation module, the main function of which is to generate a focus spatiotemporal scene graph in text form based on the video content and the information to be focused.

[0212] Step 4: Input the user input text and video obtained in step 1 and the focused spatiotemporal scene graph obtained in step 3 into the dialogue generation module. The focused spatiotemporal scene graph provides relatively rich video detail information and timing information. The module will use this information to better understand the video content and finally generate a reply text to the user.

[0213] Detailed description of the plan:

[0214] The present invention includes three modules, and each module cooperates with each other when used. The following will introduce the relevant contents of each module in detail.

[0215] Focus information generation module. The function of this module is to generate people and objects that deserve attention and focus in the video based on the text and video input by the user, and output them in the form of text to provide conditional information for subsequent modules. First, the text input by the user will enter the noun extractor to extract nouns in the text. Noun extraction is one of the key steps of this module, because nouns usually represent important entities in the text, such as people, objects, places, etc., and these entities are often the focus of attention in the video content. There are many ways to implement the noun extractor, and users can choose the most suitable technical method according to their needs. One simple and effective implementation method is to extract nouns through a large language model. Large language models (such as the GPT series) can complete this task by setting appropriate prompts, and can more accurately extract nouns that need to be paid attention to from the text. However, a potential problem with this method is that the processing speed is slow, especially in scenarios where long texts are processed or quick responses are required, which may affect the overall efficiency. In order to improve the processing speed, another commonly used implementation method is to use the NLTK library to extract nouns. NLTK (Natural Language Toolkit) is one of the most popular natural language processing libraries in Python, providing a wealth of language processing tools. Through the NLTK library, users can distinguish and extract parts of speech from text, including the recognition and extraction of nouns. Compared with large language models, NLTK has a significant advantage in processing speed, especially when facing large-scale text, it can more efficiently extract important nouns in the text. However, the disadvantage of NLTK is that its accuracy may not be as good as that of large language models, especially when dealing with complex sentences and texts with obscure meanings, the accuracy of its noun extraction may decrease.

[0216] It should be noted that the text input by the user may not contain any meaningful nouns, or these nouns do not completely include all the people or objects that should be focused on, and there are other people or objects in the video that should be focused on. In order to make this method robust, a fine-tuned video multimodal model is used here, which can output the actual people or objects that should be focused on based on the input video and possible related nouns. Video multimodal models are a type of model that has made significant progress in the fields of computer vision and natural language processing in recent years. They can process video and text data at the same time, understand and generate relevant information that conforms to the video content. Any public video multimodal model can be selected for fine-tuning, such as VILA, Qwen2-VL, etc. These models have been pre-trained on large-scale video and text data and have strong cross-modal understanding capabilities, so they perform well in a variety of application scenarios. Fine-tuning the video multimodal model mainly involves constructing a corresponding dataset. One way to construct a dataset is to simulate users generating a variety of different questions through the video multimodal model, and then the annotators annotate the actual people who should be focused on based on the questions and video content. The focus should be on the main people and objects that are strongly related to the question or interact in the video, while ignoring background information.

[0217] Focused spatiotemporal scene graph generation module. The function of this module is to output a spatiotemporal scene graph in text form containing all the people and objects in the focused information in the video according to the focused information and the video. The video input adopts the form of frame extraction, and 2 frames are extracted per second. One implementation method is to use a video multimodal large model to generate a focused spatiotemporal scene graph in text form. The video multimodal large model here can be the same model as the video multimodal large model mentioned in the focused information generation module above, but it is fine-tuned according to different data; or the same large model can be fine-tuned using a mixed data set of the two. There are two reasons why the large model is only used to generate a spatiotemporal scene graph containing all the people and objects in the focused information. One is that the context length supported by the model for input and output is limited. If it is not restricted, it may output too long text and exceed the context length limit, especially in outdoor scenes; the other reason is that generating too many spatiotemporal scene graphs of people and objects that are irrelevant to the main content of the video and the content that users care about may interfere with the generation of relevant information about the people and objects that they really want to pay attention to. In addition, while learning to generate a focused spatiotemporal scene graph, the model also learns target detection, video tracking, character interaction\spatial relationship understanding capabilities, because these are elements explicitly included in the spatiotemporal scene graph. The construction of the dataset used in this module is mainly to construct a focused spatiotemporal scene graph of a video. The video dataset constructed in the previous stage can be used, which already contains the focus information. The target detection algorithm is used to automatically detect the position and category of the person focused on each frame through the focus information, and then the relationship between the characters is manually annotated. Since the result of target detection cannot guarantee that the same person has a unique ID in different frames, that is, the position and behavior of the same person in different frames cannot be directly tracked in video processing, so an additional tracking mechanism needs to be introduced to solve this problem. In the video target tracking task, a common implementation method is to determine whether it is the same person by comparing the overlap degree of the target detection frame in two adjacent frames. Specifically, if two detection frames have a high degree of overlap in two adjacent frames, and the targets they represent belong to the same category (for example, both are detected people), then the two detection frames can be considered to belong to the same person in the video. At this time, a unique ID can be assigned to this person, and the person can be continuously tracked in subsequent frames. The core of this tracking method lies in the calculation of the overlap degree of the detection frame and the consistency judgment of the category. The degree of overlap can usually be measured by calculating the intersection over union (IoU) of two detection boxes. IoU refers to the ratio of the intersection area of ​​two detection boxes to their union area. When the IoU exceeds a preset threshold (such as 0.5 or 0.7), the two boxes are usually considered to have a high degree of overlap. In addition, category consistency is also a key factor. If two overlapping detection boxes belong to different categories (for example, one is a person and the other is a vehicle), then they obviously cannot be considered the same target.This implementation is based on a basic and reasonable assumption that the interval between two video frames is short, so that the character cannot move with a large displacement.

[0218] Dialogue generation module. The main function of this module is to answer questions raised by users based on the input video and the focused spatiotemporal scene graph in text form. The focused spatiotemporal scene graph here is a structured information representation that can effectively sort out the noteworthy characters, objects and the relationships between them in the video, and provide this information in the form of text to the dialogue generation model. In this way, the dialogue generation module can better understand the video content and reduce the "hallucination" phenomenon that may be generated by the large model when generating answers, that is, generating answers that do not conform to the actual content of the video. One implementation method here is also to use a large multimodal video model, which can be trained simultaneously with the focused spatiotemporal scene graph generation module, because the two tasks are similar, one is to output the spatiotemporal scene graph, and the other is to input the spatiotemporal scene graph. The construction of the data set can also reuse the data sets between them, and only requires additional annotation of the answer to each question.

[0219] The three modules mentioned above together constitute a structured video understanding method based on a multimodal large model. This method is a robust and highly accurate video content understanding method. Video modalities are common in the real world and can handle complex dynamic scenes to meet intelligent needs. This method can accurately understand and analyze the key content of the video and provide high-quality answer analysis. It is suitable for a variety of intelligent application scenarios such as intelligent monitoring, autonomous driving, and intelligent assistants, and can provide users with accurate and efficient service experience. The focused spatiotemporal scene graph used is a structured information representation form with high organization and readability. This method can both generate and process this structured information. Through it, the model can clearly reflect the ability to understand the video content and accurately capture key people, objects and their relationships. Structured information helps to improve the model's reasoning ability, enabling it to efficiently handle complex scenes and draw accurate and reasonable inferences, making the model more robust and intelligent when dealing with diverse video content and applicable to a wide range of scenarios. The focusing strategy of this method can highlight the advantages of useful information, ignore useless background information, and thus increase the focus on key content. This helps large models reduce the illusion phenomenon during text generation, avoid the generation of information that does not match the actual content, and significantly reduce the amount of output text. This not only improves the accuracy and relevance of the generated text, but also reduces the consumption of computing resources, indirectly reduces the cost of use, and achieves a more economical operation mode, which is suitable for scenarios where large-scale data can be efficiently processed.

[0220] Please refer to Figure 2 , Figure 2 A structured video understanding device 110 based on a multimodal large model provided in an embodiment of the present invention includes:

[0221] The acquisition module 1101 is used to acquire the video to be processed and the interactive text input by the user, wherein the interactive text is a user demand description text for the video to be processed;

[0222] The understanding module 1102 is used to extract keywords from the interactive text to obtain focused keywords, and determine the focused entities corresponding to the focused keywords in the video to be processed in combination with the pre-trained first video multimodal large model; input the video to be processed into the pre-trained second video multimodal large model to obtain a focused spatiotemporal scene graph of the focused characters and focused objects included in the focused entities; and provide dialogue feedback to the interactive text according to the focused spatiotemporal scene graph.

[0223] It should be noted that the implementation principle of the aforementioned structured video understanding device 110 based on a multimodal large model can refer to the implementation principle of the aforementioned structured video understanding method based on a multimodal large model, which will not be repeated here. It should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into a physical entity, or physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the structured video understanding device 110 based on a multimodal large model can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The function of the structured video understanding device 110 based on a multimodal large model. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in a processor element or an instruction in software form.

[0224] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more microprocessors (digital signal processors, DSP), or one or more field programmable gate arrays (FPGA), etc. For another example, when a module above is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0225] The embodiment of the present invention provides a computer device 100, which includes a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device 100 executes the aforementioned structured video understanding device 110 based on a multimodal large model. Figure 3 As shown, Figure 3 The computer device 100 is a block diagram of a structure of the computer device 100 provided in an embodiment of the present invention. The computer device 100 includes a structured video understanding apparatus 110 based on a multimodal large model, a memory 111 , a processor 112 and a communication unit 113 .

[0226] In order to realize data transmission or interaction, the memory 111, the processor 112 and the communication unit 113 are electrically connected to each other directly or indirectly. For example, the electrical connection between these elements can be realized through one or more communication buses or signal lines. The structured video understanding device 110 based on the multimodal large model includes at least one software function module that can be stored in the memory 111 in the form of software or firmware or solidified in the operating system (OS) of the computer device 100. The processor 112 is used to execute the structured video understanding device 110 based on the multimodal large model stored in the memory 111, such as the software function modules and computer programs included in the structured video understanding device 110 based on the multimodal large model.

[0227] An embodiment of the present invention provides a readable storage medium, which includes a computer program. When the computer program is running, it controls the computer device where the readable storage medium is located to execute the aforementioned structured video understanding device 110 based on a multimodal large model.

[0228] For illustrative purposes, the foregoing description is made with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the present disclosure to the precise form disclosed. Numerous modifications and variations are possible in accordance with the above teachings. These embodiments are selected and described in order to best illustrate the principles of the present disclosure and its practical application, so that those skilled in the art can best utilize the present disclosure and utilize various embodiments with different modifications to suit the intended specific application.

Claims

1. A structured video understanding method based on a multimodal large model, characterized in that: include: Acquire a video to be processed and an interactive text input by a user, wherein the interactive text is a user demand description text for the video to be processed; Extract keywords from the interactive text to obtain focused keywords, and determine focused entities corresponding to the focused keywords in the video to be processed in combination with a pre-trained first video multimodal large model; Inputting the video to be processed into a pre-trained second video multimodal large model to obtain a focused spatiotemporal scene graph of a focused person and a focused object included in the focused entity; A dialogue feedback is performed on the interactive text according to the focused spatiotemporal scene graph.

2. The method according to claim 1, characterized in that The step of extracting keywords from the interactive text to obtain focused keywords includes: A pre-trained large language model or NLTK library is called to extract keywords from the interactive text to obtain focused keywords.

3. The method according to claim 1, characterized in that The step of determining the focused entity corresponding to the focused keyword in the video to be processed by combining the pre-trained first video multimodal large model includes: The first video multimodal large model obtained by fine-tuning the VILA model training is combined to determine the focus entity corresponding to the focus keyword in the video to be processed.

4. The method according to claim 1, characterized in that: The step of inputting the video to be processed into a pre-trained second video multimodal large model to obtain a focused spatiotemporal scene graph of a focused person and a focused object included in the focused entity comprises: The second video multimodal large model pre-trained by the video to be processed is extracted by frame extraction to obtain a focused spatiotemporal scene graph of the focused person and the focused object included in the focused entity; Based on the intersection-and-union ratio of the focused entities, each of the focused characters and each of the focused objects are determined, and a plurality of the focused spatiotemporal scene graphs including the interaction relationships between each of the focused characters and each of the focused objects are obtained. The focused spatiotemporal scene graphs include a unique identifier ID for each of the focused entities, which is used to track each focused entity on the timeline of the video.

5. The method according to claim 1, characterized in that The focused spatiotemporal scene graph is represented in a text form in JSON format, and the focused spatiotemporal scene graph includes frame information, node information and edge information, the node information represents the focused entities and their positions, and the edge information represents the interaction relationship or spatial relationship between the focused entities.

6. The method according to claim 1, characterized in that The performing dialogue feedback on the interactive text according to the focused spatiotemporal scene graph includes: Parsing the focused spatiotemporal scene graph to extract key video detail information and timing information; The extracted information is input into a pre-trained dialogue generation model to generate a reply text to the interaction text.

7. The method according to claim 1, characterized in that The method further comprises: The first video multimodal large model and the second video multimodal large model are jointly trained, the first video multimodal large model and the second video multimodal large model share part of the training data, and the model parameters are continuously optimized through cross-validation.

8. A structured video understanding device based on a multimodal large model, characterized in that: include: An acquisition module, used to acquire a video to be processed and an interactive text input by a user, wherein the interactive text is a user demand description text for the video to be processed; The understanding module is used to extract keywords from the interactive text to obtain focused keywords, and determine the focused entities corresponding to the focused keywords in the video to be processed in combination with a pre-trained first video multimodal large model; input the video to be processed into a pre-trained second video multimodal large model to obtain a focused spatiotemporal scene graph of the focused characters and focused objects included in the focused entities; and provide dialogue feedback to the interactive text according to the focused spatiotemporal scene graph.

9. A computer device, characterized in that: The computer device comprises a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device executes the method according to any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The readable storage medium includes a computer program, and when the computer program is executed, the computer device where the readable storage medium is located is controlled to execute the method according to any one of claims 1 to 7.

Images