Video frame determination method and apparatus, and electronic device

Abstract

The invention discloses a video frame determination method and apparatus, and an electronic device. The method comprises the steps that a problem vector of a target object is acquired, and the problem vector is used for matching a video frame from a video; a first similarity between the problem vector and a plurality of communities corresponding to the video is determined, each community is composed of a plurality of video frames, the second similarity of the multi-modal data in the video meets a first preset condition, and each video frame belongs to at least one community; and determining a target community of which the first similarity meets a second preset condition from the plurality of communities, and determining a target video frame matched with the problem vector from the target community. The technical problem that the key frame screening effect is poor due to the fact that multiple semantic roles of frames cannot be fully mined in key frame screening in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a method, device and electronic device for determining a video frame. Background Art

[0002] With the development of multimedia technology, the processing and understanding of video data has become increasingly important, especially in video question-answering systems, where users can ask questions via text. The system must quickly locate and return relevant segments or keyframes within the video. However, existing technologies treat keyframe filtering as a single clustering or classification problem, failing to fully exploit the multiple semantic roles of frames, resulting in poor keyframe filtering performance.

[0003] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0004] The embodiments of the present application provide a method, device and electronic device for determining a video frame, so as to at least solve the technical problem that the related art cannot fully exploit the multiple semantic roles of frames in key frame screening, resulting in poor key frame screening effect.

[0005] According to one aspect of an embodiment of the present application, a method for determining a video frame is provided, including: obtaining a question vector of a target object, wherein the question vector is used to match video frames from a video; determining a first similarity between the question vector and multiple communities corresponding to the video, wherein the community is composed of multiple video frames whose second similarity of multimodal data in the video meets a first preset condition, and each video frame belongs to at least one community; determining a target community from the multiple communities whose first similarity meets a second preset condition, and determining a target video frame that matches the question vector from the target community.

[0006] In some embodiments of the present application, a community is determined in the following manner: determining an initial label for each vertex in a graph structure corresponding to a video, wherein the vertices in the graph structure represent video frames in the video, the edges in the graph structure represent the second similarity of multimodal data between two vertices, and the initial label is the index of the vertex; determining an importance index for each vertex, wherein the importance index is used to quantify the degree of influence of the vertex on neighboring vertices within a preset distance range; for each vertex, updating the initial label based on the label and importance index of the neighboring vertex to obtain an updated initial label; and determining a community based on the updated initial label when the updated initial label satisfies a third preset condition.

[0007] In some embodiments of the present application, a graph structure is constructed in the following manner: obtaining multiple first video frames corresponding to a video, and determining the multiple first video frames as multiple vertices of the graph structure; determining a similarity matrix corresponding to the multiple first video frames, wherein the similarity matrix is ​​used to quantify the second similarity of multimodal data between the multiple first video frames; constructing a graph adjacency matrix based on the similarity matrix and a similarity threshold, wherein, when the second similarity of the multimodal data between any two corresponding vertices in the similarity matrix is ​​greater than the similarity threshold, it is determined that there is a connecting edge between any two vertices in the graph structure.

[0008] In some embodiments of the present application, determining a similarity matrix corresponding to multiple first video frames includes: determining a text similarity matrix corresponding to the multiple first video frames, wherein the text similarity matrix is ​​used to quantify the similarity of the descriptive texts representing the multiple first video frames; determining a visual similarity matrix corresponding to the multiple first video frames, wherein the visual similarity matrix is ​​used to quantify the similarity of the visual images representing the multiple first video frames; determining a temporal similarity matrix corresponding to the multiple first video frames, wherein the temporal similarity matrix is ​​used to quantify the similarity of the appearance time of the multiple first video frames in the video; and fusing the text similarity matrix, the visual similarity matrix, and the temporal similarity matrix to obtain a similarity matrix.

[0009] In some embodiments of the present application, determining a text similarity matrix corresponding to multiple first video frames includes: determining descriptive texts corresponding to the multiple first video frames, wherein the descriptive texts are used to describe the picture content in the first video frames; encoding the descriptive texts to obtain text feature vectors; and determining a text similarity matrix based on the text feature vectors.

[0010] In some embodiments of the present application, it also includes: obtaining an initial visual feature matrix corresponding to the graph structure, wherein the visual feature matrix is ​​composed of the initial visual feature vector of the second video frame corresponding to each vertex in the graph structure; using a graph attention network to update the initial visual feature vector of each second video frame to obtain a target visual feature matrix, wherein the graph attention network is used to update the visual features of the second video frame based on the visual features of the neighboring nodes of the second video frame.

[0011] In some embodiments of the present application, determining a target community whose first similarity satisfies a second preset condition from multiple communities includes: obtaining a first similarity set between the multiple communities and a question vector, wherein each first similarity in the first similarity set is used to represent the similarity between all video frames in the community and the question vector; sorting all the first similarities in the first similarity set to obtain a first similarity sequence; determining a preset number of communities with the highest first similarities from the first similarity sequence, and determining the preset number of communities as target communities.

[0012] In some embodiments of the present application, determining a target video frame that matches a question vector from a target community includes: determining a first target video frame, wherein the first target video frame is a video frame that appears simultaneously in at least two target communities; determining a second target video frame from each of the multiple target communities, wherein the second target video frame is a video frame in the community with the greatest third similarity to the question vector; and determining a target video frame based on the first target video frame and the second target video frame.

[0013] According to another aspect of the embodiments of the present application, a device for determining a video frame is also provided, including: an acquisition module for acquiring a question vector of a target object, wherein the question vector is used to match video frames from a video; a determination module for determining a first similarity between the question vector and multiple communities corresponding to the video, wherein the community is composed of multiple video frames whose second similarity of multimodal data in the video meets a first preset condition, and each video frame belongs to at least one community; a matching module for determining a target community whose first similarity meets a second preset condition from multiple communities, and determining a target video frame that matches the question vector from the target community.

[0014] According to another aspect of the embodiments of the present application, an electronic device is provided, including: a memory and a processor, the memory being used to store program instructions; the processor being connected to the memory and being used to execute the above-mentioned method for determining the video frame.

[0015] According to another aspect of the embodiments of the present application, a non-volatile storage medium is provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned method for determining the video frame by running the computer program.

[0016] According to another aspect of the embodiments of the present application, a computer program product is provided, including computer instructions, which implement the above-mentioned method for determining video frames when executed by a processor.

[0017] In an embodiment of the present application, a multimodal data fusion and community detection method is adopted to construct an overlapping community structure of video frames, and combine the question vector to evaluate the similarity between the community and the video frame, thereby achieving the purpose of accurately screening video frames that are highly relevant to the question and have diverse semantics, thereby achieving the technical effect of optimizing the efficiency and quality of key frame screening, and further solving the technical problem that the related technology cannot fully explore the multiple semantic roles of the frame in key frame screening, resulting in poor key frame screening effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 This is a hardware structure block diagram of a computer terminal according to a method for determining a video frame according to an embodiment of the present application;

[0020] Figure 2 is a flowchart of a method for determining a video frame according to an embodiment of the present application;

[0021] Figure 3 It is a structural diagram of a device for determining a video frame according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0023] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0024] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:

[0025] Graph Attention Network (GAT): A graph neural network model based on the attention mechanism that allows nodes (in this application example, video frames) to weight information from different neighboring nodes, thereby more effectively aggregating information and capturing complex inter-node correlation patterns. In this application example, GAT is applied to visual feature updates in video frames, enhancing feature representation through inter-node attention mechanisms to better reflect temporal and spatial relationships between frames.

[0026] Label Propagation Algorithm (LPA): An unsupervised machine learning algorithm for community detection that identifies community structures in a graph through the dynamic propagation of labels between nodes. In this embodiment of the present application, LPA is used to identify community affiliations between video frames, especially for the discovery of overlapping communities, so that each video frame can be regarded as having multiple semantic roles, improving the diversity and accuracy of key frame screening.

[0027] In video question-answering systems, keyframe screening is a core component. Its goal is to filter out diverse keyframes that are highly relevant to user questions from massive video frames, thereby improving the user's viewing experience and information acquisition efficiency. Early keyframe screening methods mainly relied on traditional image feature extraction algorithms such as SIFT and HOG, combined with statistical features such as color histograms. These features were manually designed to describe the visual content of video frames from different perspectives. However, these manually designed features have limitations when describing complex and changing video content, making it difficult to fully capture semantic information, resulting in poor keyframe screening results. Furthermore, such methods typically only focus on the relevance between frames and questions, ignoring the diversity between keyframes.

[0028] However, when using deep learning technology for key frame screening, the relevant technology still treats each frame as an independent individual for processing, fails to fully utilize the correlation structure between frames, and cannot mine the potential community structure of key frames. In addition, the relevant technology only relies on the features of a single modality and fails to fully utilize multimodal information, limiting the richness of feature representation.

[0029] In order to solve the above technical problems, the embodiments of the present application provide corresponding solutions, which are described in detail below.

[0030] The video frame determination method embodiment provided in the embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing a method for determining a video frame. Figure 1As shown, the computer terminal 10 may include one or more processors (illustrated as 102a, 102b, ..., 102n in the figure) (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions via a wired and / or wireless network connection. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. Those skilled in the art will understand that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0031] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry." The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuitry may be a single, independent processing module, or may be fully or partially integrated into any of the other components of the computer terminal 10. As discussed in the embodiments of the present application, the data processing circuitry functions as a processor control (e.g., the selection of a variable resistor terminal path connected to an interface).

[0032] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the method for determining the video frame in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implementing the above-mentioned method for determining the video frame. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0033] The transmission module 106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission module 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission module 106 may be a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0034] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0035] It should be noted that, in some optional embodiments, the above Figure 1 The computer terminal shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computer terminal described above.

[0036] In the above-mentioned operating environment, an embodiment of the present application provides an embodiment of a method for determining a video frame. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0037] Figure 2 is a flow chart of a method for determining a video frame according to an embodiment of the present application, such as Figure 2 As shown, the method includes the following steps:

[0038] Step S202 : obtaining a question vector of the target object, wherein the question vector is used to match video frames from the video.

[0039] In the above step S202 , the question vector is the question raised by the user, which is converted into a numerical vector form that can be understood and processed by a computer through natural language processing (NLP) technology.

[0040] It should be noted that in order to effectively convert the question raised by the target subject (user) into a question vector and use the vector to match video frames and filter key frames, the question may include but is not limited to at least one of the following key elements:

[0041] (1) Subject or entity: This refers to the subject or entity being asked, such as "protagonist," "scene," etc. In some embodiments of the present application, this question element will be encoded as part of the question vector and used to search for content related to the specified subject or entity in the video frame. For example, the question "What are the activities of the protagonist indoors in the video?" clearly points out the two entities "protagonist" and "indoors," which will help the system focus on specific people and scenes.

[0042] (2) Specific description of behavior or action: that is, a specific description of the behavior or action so that the system can identify the events in the video that match it. For example, in the question "What activities does the protagonist perform indoors in the video?", the word "activity" implies an inquiry about the protagonist's behavior, which will prompt the system to identify specific actions such as "walking" and "talking", and further refine the matching range of the video frame.

[0043] (3) Context and scene information: that is, context or specific scene information. In some embodiments of the present application, the word "indoor" in the question "Which scenes in the video show the protagonist's activities indoors" is a key scene description. It limits the environmental conditions for video frame search and helps the system filter out video clips that are not related to indoor scenes.

[0044] (4) Time clues: If the question contains time information, such as "the first half of the video", "the last few minutes", etc., this will serve as an important matching element. In some embodiments of the present application, even if the direct generation of the question vector does not contain time clues, by considering time similarity in the community detection stage, time information is indirectly used to optimize the screening of key frames to ensure that the returned video clips or frames are within the time range specified by the user.

[0045] (5) Question type and intent: Understanding the question type and intent, such as whether the question is about character dialogue, action details, scene description, etc., helps the system locate video frames more accurately. For example, the question "What details in the video show that the protagonist is thinking?" shows that the user pays attention to the details of the protagonist's thinking behavior, which will guide the system to find key information such as small movements or expressions that show the thinking state.

[0046] In some embodiments of the present application, the question input by the user is converted into a vector form through a pre-trained text encoder (such as BERT, T5).

[0047] Step S204 : determining a first similarity between the question vector and multiple communities corresponding to the video, wherein a community is composed of multiple video frames whose second similarity of multimodal data in the video meets a first preset condition, and each video frame belongs to at least one community.

[0048] In the above step S204, a community refers to a set of video frames in a video whose correlation of multimodal data (such as visual, text) reaches a certain threshold (first preset condition). Each community represents a visual semantic unit of a specific theme, situation or scene in the video.

[0049] The first similarity is a similarity measure between the question vector and the community feature representation. It is used to evaluate the degree of association between the question vector and each community identified in the video and determine which communities are selected as candidate communities (target communities). The second similarity is the multimodal data similarity between video frames used to construct communities. For example, it can be calculated based on the visual features and text description features of the video frames and is used to determine whether video frames belong to the same community. That is, in the community formation stage, when the inter-frame similarity meets certain conditions, these frames will be classified into one community.

[0050] In some embodiments of the present application, formulas such as cosine similarity can be used to measure the correlation between the question vector and the community feature representation. Specifically, for each community, the average value of the semantically enhanced feature vectors of all its video frames (such as the weighted fusion of visual features and text features updated using GAT) can be calculated as the feature representation of the community, and the cosine similarity between the feature representation and the question vector can be calculated as the first similarity.

[0051] Considering that different communities may contribute differently to the relevance of the question, a weighting factor can be introduced to adjust the calculation of the first similarity. For example, for each community, the average similarity between all its video frames and the question vector is first calculated, and then the average similarity is weighted according to the feature importance of the community or the matching degree between the user query and the community topic to obtain the first similarity.

[0052] It should be noted that in order to reduce the impact of community size on the first similarity calculation, a method of dynamically adjusting the community feature representation can be adopted. Specifically, for communities with a large scale (the number of video frames reaches a preset threshold), sub-communities can be further divided (such as based on the differences and time distribution of video frame features), and feature representations are calculated for each sub-community separately (such as calculating the average value of the semantic feature vectors of all its video frames), and then merged into the overall feature representation of the community through weighted averaging. Even if the community is large, the accuracy and representativeness of the feature representation can be ensured.

[0053] In some embodiments of the present application, a label propagation algorithm can be used to identify the community affiliation of nodes in a graph structure, specifically: determining the initial label of each vertex in the graph structure corresponding to the video, wherein the vertices in the graph structure represent video frames in the video, the edges in the graph structure represent the second similarity of multimodal data between two vertices, and the initial label is the index of the vertex; determining the importance index of each vertex, wherein the importance index is used to quantify the degree of influence of the vertex on neighboring vertices within a preset distance range; for each vertex, updating the initial label based on the label and importance index of the neighboring vertex to obtain an updated initial label; and determining the community based on the updated initial label when the updated initial label meets the third preset condition.

[0054] The graph structure is a mathematical model composed of vertices (video frames) and edges (second similarity between video frames). It represents the association between video frames in a video and is used to capture the complex network structure of video frames. The importance index is an indicator used to quantify the degree of influence of a vertex on neighboring vertices within a preset distance range, reflecting the relative importance of the vertex in the network.

[0055] In some embodiments of the present application, a graph structure is constructed in the following manner: obtaining multiple first video frames corresponding to a video, and determining the multiple first video frames as multiple vertices of the graph structure; determining a similarity matrix corresponding to the multiple first video frames, wherein the similarity matrix is ​​used to quantify the second similarity of multimodal data between the multiple first video frames; constructing a graph adjacency matrix based on the similarity matrix and a similarity threshold, wherein, when the second similarity of the multimodal data between any two corresponding vertices in the similarity matrix is ​​greater than the similarity threshold, it is determined that there is a connecting edge between any two vertices in the graph structure.

[0056] The first video frame is a series of video frames sampled or filtered from the video, forming part of the graph structure. For example, the video can be sampled uniformly or non-uniformly. Uniform sampling selects frames at fixed time intervals, while non-uniform sampling selects frames near key change points based on the rate of change of frame content or motion amplitude. The sampled video frame set (i.e., the first video frame set) is recorded as , ( is the total number of frames of the original video), reducing the amount of data for subsequent processing and improving computing efficiency.

[0057] The similarity matrix quantifies the second similarity of multimodal data between first video frames, where each element represents the similarity between video frame i and video frame j. The graph adjacency matrix is ​​constructed based on the similarity matrix and a similarity threshold. It represents the connection between vertices in a graph structure. When the second similarity between two vertices (video frames) is greater than the threshold, an edge connects them; otherwise, no connection exists.

[0058] In order to fully reflect the similarity matrix of the association between video frames, the similarity matrix corresponding to the multiple first video frames can be determined in the following manner: determine the text similarity matrix corresponding to the multiple first video frames, wherein the text similarity matrix is ​​used to quantify the similarity of the description texts representing the multiple first video frames; determine the visual similarity matrix corresponding to the multiple first video frames, wherein the visual similarity matrix is ​​used to quantify the similarity of the visual images representing the multiple first video frames; determine the time similarity matrix corresponding to the multiple first video frames, wherein the time similarity matrix is ​​used to quantify the similarity of the appearance time of the multiple first video frames in the video; and fuse the text similarity matrix, the visual similarity matrix and the time similarity matrix to obtain the similarity matrix.

[0059] Among them, the text similarity matrix corresponding to the multiple first video frames is determined: the description texts corresponding to the multiple first video frames are determined, wherein the description texts are used to describe the picture content in the first video frames; the description texts are encoded to obtain text feature vectors; and the text similarity matrix is ​​determined based on the text feature vectors.

[0060] In some embodiments of the present application, after sampling a video to obtain a plurality of first video frames, a graph structure may be constructed by the following steps:

[0061] (1) Video frame feature extraction, including but not limited to:

[0062] Visual feature extraction: Using open source visual encoders such as CLIP, we extract the deep feature vector for each frame of the sampled video, expressed as ,in is the total number of video frames after sampling.

[0063] Text feature extraction: Use large visual models, such as Qwen2-VL, to generate description text for the sampled video frames to obtain a description text set. , each description text is a word sequence. Use pre-trained text vector encoder (such as BGE-M3) to encode each description text. Encode and obtain a set of text feature vectors .

[0064] (2) Consider all frames of the sampled video (the first video frame) as the vertex set of the graph .

[0065] (3) Similarity matrix calculation:

[0066] Calculate the inter-frame text similarity matrix , for example, the cosine similarity formula can be used:

[0067]

[0068] Calculate the visual similarity matrix between frames , for example, the cosine similarity formula can be used:

[0069]

[0070] Calculate the inter-frame temporal similarity matrix , according to the time series position of the frame, if the time distance between two frames is less than the threshold ,but , otherwise .

[0071] Fusion strategy: fuse the text similarity matrix, visual similarity matrix and temporal similarity matrix to obtain the final inter-frame similarity matrix , the fusion formula is:

[0072]

[0073] in, is a balancing parameter that controls the weights of textual similarity and visual-temporal similarity.

[0074] (4) Constructing a graph adjacency matrix based on the similarity matrix , set the similarity threshold ,when When the vertex and vertices There are edge connections between them.

[0075] In some embodiments of the present application, the following steps can also be performed: obtaining an initial visual feature matrix corresponding to the graph structure, wherein the visual feature matrix is ​​composed of the initial visual feature vectors of the second video frame corresponding to each vertex in the graph structure; using a graph attention network to update the initial visual feature vectors of each second video frame to obtain a target visual feature matrix, wherein the graph attention network is used to update the visual features of the second video frame based on the visual features of the neighboring nodes of the second video frame.

[0076] The initial visual feature matrix is ​​composed of the initial visual feature vector corresponding to each vertex (video frame) in the graph structure. It is the original feature representation before the graph attention network performs feature update. The graph attention network can dynamically aggregate the feature information of neighboring nodes by learning the attention weights between each node (video frame) and its neighboring nodes (adjacent video frames) to update and optimize the feature representation of the nodes, strengthen the visual feature representation of key frames, and improve the accuracy of key frame screening.

[0077] The target visual feature matrix is ​​the visual feature matrix after being updated by the graph attention network. Compared with the initial visual feature matrix, it integrates the visual feature information of adjacent video frames and can more comprehensively reflect the visual content and semantic information of the video frames.

[0078] Specifically, in some embodiments of the present application, the following steps may be used to implement graph attention network (GAT) encoding:

[0079] (1) Vertex feature matrix of the input graph (visual features) and adjacency matrix .

[0080] (2) Use a two-layer graph attention network to update the visual features of the frame. The output feature calculation formula of each layer is:

[0081]

[0082] in, represents the node feature matrix in the l-th layer GAT, represents the node feature matrix after the GAT operation of the lth layer, Represents a vertex The neighbor set of It is Layer Vertex and vertices The attention coefficient between is calculated by the following formula:

[0083]

[0084] in, It is The trainable weight matrix of the layer, is the attention vector, which is used to measure the importance of the connection between vertices. is the activation function, Represents a splicing operation.

[0085] (3) Output the updated vertex feature matrix , as the semantically enhanced feature representation of the frame.

[0086] After the graph structure is constructed, the label propagation algorithm can be used to divide the community based on the graph structure. In the label propagation algorithm, the third preset condition is used to determine whether the updated initial label is stable, that is, whether it no longer changes. Meeting the third preset condition means that the algorithm iteration is completed, and the community structure can be defined based on the updated initial label.

[0087] Specifically, in some embodiments of the present application, community division can be performed through the following steps:

[0088] (1) Initialize the label of each vertex (initial label) to its own index, which is used to identify the vertex and form an initial label set .

[0089] (2) Calculate the importance of vertices (Importance index), as the weight of label propagation, is calculated as follows: Represents a vertex The influence on surrounding vertices, where Represents a vertex The set of neighbors.

[0090] (3) Perform label propagation iterations. For each vertex , update its label according to the label and weight of the neighbor vertex, and the update formula is:

[0091]

[0092] in, Indicates the Vertex Tags, Convert labels to one-hot encoding.

[0093] (4) Stop when the label no longer changes or the maximum number of iterations is reached (the third preset condition) and obtain the community division result of the frame. For example, after the iteration is completed, check the final label set and regard the vertex set with the same label as belonging to the same community.

[0094] It should be noted that in some embodiments of the present application, due to the adoption of an overlapping community discovery strategy, a single vertex (video frame) may retain multiple labels, which means that it may belong to multiple communities at the same time, reflecting the multiple semantic roles of the video frame. Therefore, for each vertex, it is necessary to check whether it has multiple labels. If there are multiple labels, the vertex is considered to overlap in multiple communities.

[0095] Step S206 : determining a target community whose first similarity satisfies a second preset condition from the multiple communities, and determining a target video frame matching the question vector from the target community.

[0096] In the above step S206, the target video frames are video frames selected from the target community that best match the question vector. These frames are not only highly relevant to the question, but also representative within the community, and ultimately constitute a set of key frames to answer the user's video question-and-answer needs.

[0097] In order to effectively balance the relevance and diversity of key frames and ensure that the selected frames are not only highly relevant to the question but also cover multiple semantic scenes in the video, the target community can be determined in the following way: a first similarity set between multiple communities and the question vector is obtained, wherein each first similarity in the first similarity set is used to represent the similarity between all video frames in the community and the question vector; all first similarities in the first similarity set are sorted to obtain a first similarity sequence; a preset number of communities with the highest first similarities are determined from the first similarity sequence, and the preset number of communities are determined as target communities.

[0098] In some embodiments of the present application, for each community, the average cosine similarity score between the semantically enhanced feature vectors and the question vector of all video frames in the community can be calculated, and the first similarity scores of all communities are collected into a set to form a first similarity set; a sorting algorithm (such as quick sort or heap sort) is used to arrange all the first similarity scores in the first similarity set in descending order to obtain a first similarity sequence; and the top k communities with the highest scores from the first similarity sequence are selected as target communities.

[0099] In order to improve the accuracy of key frame screening, the target video frame can be determined in the following manner: determining a first target video frame, wherein the first target video frame is a video frame that appears simultaneously in at least two target communities; determining a second target video frame from multiple target communities, wherein the second target video frame is the video frame in the community with the third largest similarity to the question vector; and determining the target video frame based on the first target video frame and the second target video frame.

[0100] Specifically, the first target video frames are those that appear simultaneously in at least two target communities. These frames typically have multiple semantic roles and belong to multiple related communities, representing key moments where complex scenes and concepts intersect in the video. The second target video frame is the video frame with the third highest similarity within each target community, selected as the representative. This means that the video frame has the highest similarity to the question vector within the community.

[0101] In some embodiments of the present application, the first target video frame set can be merged with the second target video frame sets of all target communities to form a preliminary target video frame set; in order to avoid duplication and redundancy, the set is deduplicated and the position where each video frame first appears is retained; if the video frames in the first target video frame set happen to be the second target video frames of certain target communities, they are retained first to reflect the multiple semantic roles of these frames; in addition, if the size of the preliminary target video frame set exceeds a predetermined number, the third similarity between each video frame and the question vector can be calculated again, and the video frames with the highest scores can be selected as the final target video frame set.

[0102] In some embodiments of the present application, overlapping community discovery and key frame screening can be achieved through the following steps:

[0103] (1) Based on the community division results, frames in the same community are regarded as a visual semantic unit. In particular, frames are allowed to belong to multiple communities to capture the multiple semantic roles of frames.

[0104] (2) Calculate each community and question vector Relevance score , the formula is:

[0105]

[0106] in, Represents a community, The pinnacle of the community Semantic enhancement features.

[0107] (3) Sort by relevance score and select the one with the highest score communities.

[0108] (4) Select the frame with the largest cosine similarity to the question vector from each community, while considering the overlapping belonging of frames in multiple communities to ensure that the key frame set covers multiple communities and is diverse.

[0109] Through the above steps S202 to S206, multimodal data fusion and community detection are adopted. By constructing an overlapping community structure of video frames and combining the question vector to evaluate the similarity between the community and the frame, the purpose of accurately screening video frames that are highly relevant to the question and have diverse semantics is achieved, thereby achieving the technical effect of optimizing the efficiency and quality of key frame screening, and further solving the technical problem that related technologies cannot fully explore the multiple semantic roles of frames in key frame screening, resulting in poor key frame screening effect.

[0110] To facilitate understanding of the above-mentioned process of determining the video frame, some specific embodiments are explained below.

[0111] Taking a video containing 1,000 frames as an example, suppose the question asked by the user is "Which scenes in the video show the protagonist's activities indoors?"

[0112] Step 1, video sampling: Perform non-uniform sampling on the video, select frames near key change points based on the frame content change rate, and reduce the number of video frames after sampling to 200 frames.

[0113] Step 2: Video frame feature extraction:

[0114] Use CLIP to extract visual features and obtain a set of visual feature vectors , use the Qwen2.5-VL visual model to generate description text for video frames, such as "the protagonist walks indoors", "the protagonist talks to others", etc., and then use the BGE vector encoding model to encode these texts to obtain a set of text feature vectors .

[0115] Step 3, graph construction:

[0116] Calculate the inter-frame text similarity matrix separately , visual similarity matrix and the temporal similarity matrix , set the balance parameters , fusion obtains the inter-frame similarity matrix , set the similarity threshold , construct the graph adjacency matrix .

[0117] Step 4, graph attention network encoding:

[0118] enter and , use two layers of GAT to update visual features and output the updated vertex feature matrix .

[0119] Step 5: Label propagation algorithm community detection:

[0120] Initialize the tag set, , calculate the importance of the vertex , perform label propagation iteration, set the maximum number of iterations to 100, and finally obtain the community division result of the frame.

[0121] Step 6: Overlapping community discovery and keyframe screening:

[0122] Allowing frames to belong to multiple communities, capturing the multiple semantic roles of frames, and encoding the question "Which scenes in the video show the protagonist's indoor activities?" into a question vector , calculate each community and Relevance score , select the three communities with the highest scores, and select the The frame with the largest cosine similarity is selected, and the overlapping belonging of frames in multiple communities is considered to finally obtain the key frame set.

[0123] Figure 3 is a structural diagram of a device for determining a video frame according to an embodiment of the present application, such as Figure 3 As shown, the device includes:

[0124] An acquisition module 302 is configured to acquire a question vector of a target object, wherein the question vector is used to match a video frame from a video;

[0125] a determination module 304 for determining a first similarity between the question vector and a plurality of communities corresponding to the video, wherein the community is composed of a plurality of video frames whose second similarity of the multimodal data in the video satisfies a first preset condition, and each video frame belongs to at least one community;

[0126] The matching module 306 is configured to determine a target community whose first similarity satisfies a second preset condition from the multiple communities, and determine a target video frame that matches the question vector from the target community.

[0127] It should be noted that Figure 3 The video frame determination device shown is used to perform Figure 2 The method for determining the video frame shown is therefore Figure 2 The explanations in the method for determining the video frame in also apply to Figure 3 The apparatus for determining the video frame shown will not be described in detail here.

[0128] An embodiment of the present application further provides an electronic device, which includes a memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute the steps of the method for determining the video frame in each embodiment of the present application.

[0129] An embodiment of the present application further provides a non-volatile storage medium, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the steps of the method for determining the video frame in each embodiment of the present application by running the computer program.

[0130] An embodiment of the present application further provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the method for determining a video frame in each embodiment of the present application.

[0131] The embodiments of the present application further provide a computer program, which, when executed by a processor, implements the steps of the method for determining a video frame in each embodiment of the present application.

[0132] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0133] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0134] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0135] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0136] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0137] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the existing technology, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc. Various media that can store program code.

[0138] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for determining a video frame, characterized in that: include: Obtaining a question vector of a target object, wherein the question vector is used to match a video frame from a video; determining a first similarity between the question vector and a plurality of communities corresponding to the video, wherein the communities are composed of a plurality of video frames whose second similarity of multimodal data in the video satisfies a first preset condition, and each of the video frames belongs to at least one of the communities; A target community in which the first similarity satisfies a second preset condition is determined from the multiple communities, and a target video frame matching the question vector is determined from the target community.

2. The method according to claim 1, characterized in that The communities are determined by: Determining an initial label for each vertex in a graph structure corresponding to the video, wherein a vertex in the graph structure represents a video frame in the video, an edge in the graph structure represents a second similarity of multimodal data between two vertices, and the initial label is an index of the vertex; Determining an importance index for each vertex, wherein the importance index is used to quantitatively represent the degree of influence of the vertex on neighboring vertices within a preset distance range; For each of the vertices, updating the initial label according to the label of the neighboring vertex and the importance index to obtain an updated initial label; In a case where the updated initial label meets a third preset condition, the community is determined according to the updated initial label.

3. The method according to claim 2, characterized in that The graph structure is constructed in the following way: Acquire a plurality of first video frames corresponding to the video, and determine the plurality of first video frames as a plurality of vertices of the graph structure; Determining a similarity matrix corresponding to the plurality of first video frames, wherein the similarity matrix is ​​used to quantitatively represent a second similarity of the multimodal data between the plurality of first video frames; A graph adjacency matrix is ​​constructed based on the similarity matrix and the similarity threshold, wherein when the second similarity of the multimodal data between any two corresponding vertices in the similarity matrix is ​​greater than the similarity threshold, it is determined that the any two vertices have a connecting edge in the graph structure.

4. The method according to claim 3, characterized in that Determining a similarity matrix corresponding to the plurality of first video frames includes: Determining a text similarity matrix corresponding to the plurality of first video frames, wherein the text similarity matrix is ​​used to quantify similarities of description texts representing the plurality of first video frames; Determining a visual similarity matrix corresponding to the plurality of first video frames, wherein the visual similarity matrix is ​​used to quantitatively represent similarities of visual images of the plurality of first video frames; Determine a temporal similarity matrix corresponding to the plurality of first video frames, wherein the temporal similarity matrix is ​​used to quantitatively represent similarities in appearance times of the plurality of first video frames in the video; The text similarity matrix, the visual similarity matrix and the time similarity matrix are fused to obtain the similarity matrix.

5. The method according to claim 4, characterized in that Determining a text similarity matrix corresponding to the plurality of first video frames includes: Determining description texts corresponding to the plurality of first video frames respectively, wherein the description texts are used to describe the picture content in the first video frames; Encoding the description text to obtain a text feature vector; The text similarity matrix is ​​determined based on the text feature vector.

6. The method according to claim 2, characterized in that The method further comprises: Acquire an initial visual feature matrix corresponding to the graph structure, wherein the visual feature matrix is ​​composed of an initial visual feature vector of the second video frame corresponding to each vertex in the graph structure; A graph attention network is used to update the initial visual feature vector of each second video frame to obtain a target visual feature matrix, wherein the graph attention network is used to update the visual features of the second video frame based on the visual features of the neighboring nodes of the second video frame.

7. The method according to claim 1, characterized in that Determining a target community from the multiple communities whose first similarity satisfies a second preset condition includes: Obtaining a first similarity set between the plurality of communities and the question vector, wherein each first similarity in the first similarity set is used to represent a similarity between all video frames in the community and the question vector; sorting all first similarities in the first similarity set to obtain a first similarity sequence; A preset number of communities having the highest first similarities are determined from the first similarity sequence, and the preset number of communities are determined as the target communities.

8. The method according to claim 1, characterized in that Determining a target video frame matching the question vector from the target community includes: Determining a first target video frame, wherein the first target video frame is a video frame that appears simultaneously in at least two of the target communities; Determining a second target video frame from each of the plurality of target communities, wherein the second target video frame is a video frame in the community having the greatest third similarity to the question vector; The target video frame is determined according to the first target video frame and the second target video frame.

9. A device for determining a video frame, characterized in that: include: An acquisition module, configured to acquire a question vector of a target object, wherein the question vector is used to match a video frame from a video; a determination module, configured to determine a first similarity between the question vector and a plurality of communities corresponding to the video, wherein the communities are composed of a plurality of video frames whose second similarity of multimodal data in the video satisfies a first preset condition, and each of the video frames belongs to at least one of the communities; The matching module is configured to determine a target community from the multiple communities, where the first similarity satisfies a second preset condition, and determine a target video frame that matches the question vector from the target community.

10. An electronic device, characterized in that: include: A memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute the method for determining the video frame according to any one of claims 1 to 8.

11. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the method for determining the video frame according to any one of claims 1 to 8 by running the computer program.

12. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the method for determining a video frame according to any one of claims 1 to 8 is implemented.

Images