Video question answering method, device and medium
By calculating the motion amplitude and clustering weight of video frames to determine the centroid video frames, and selecting key video frames based on similarity, the problem of accuracy in video information extraction is solved, and efficient key information extraction and summary generation are achieved.
Patent Information
- Application Number
- CN202510963789.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-14
- Publication Date
- 2025-12-23
- Estimated Expiration
- 2045-07-14
Smart Images
Figure CN120472379B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of image processing, in particular to a video question answering method, device and medium. BACKGROUND
[0002] Through information analysis and extraction on video data, downstream tasks such as video question answering, video generation and video recommendation can be realized.
[0003] For example, video question answering refers to generating an answer to a given question by understanding the relevant video content. With the development of neural network models, video question answering is used in more and more scenarios. However, since video content often has dynamic characteristics, it is necessary to understand continuous video frames to answer some questions. However, only a small part of a video may be related to a question. How to obtain the relevant part from the entire video and then analyze and answer is a key point.
[0004] Therefore, how to improve the accuracy of video information analysis and extraction is a technical problem to be solved by those skilled in the art. SUMMARY
[0005] To solve the above technical problems, the present application at least provides a video key information extraction method, a video question answering method, a device and a medium.
[0006] The first aspect of the present application provides a video key information extraction method, which comprises: obtaining a video frame sequence, calculating the degree of motion intensity of pixels in the video frame sequence to obtain a motion amplitude value corresponding to each video frame in the video frame sequence; calculating a clustering weight corresponding to each video frame based on the motion amplitude value corresponding to each video frame; wherein the motion amplitude value and the clustering weight are positively correlated; performing weighted clustering on each video frame based on the clustering weight corresponding to each video frame to obtain a plurality of clustering clusters, and taking the video frame corresponding to the cluster center point of each clustering cluster as a centroid video frame; calculating the image similarity between each video frame and each centroid video frame to obtain a first similarity, selecting a video frame whose first similarity satisfies a preset condition to obtain a key video frame; and performing image information extraction on each key video frame to obtain key information corresponding to the video frame sequence.
[0007] In an embodiment, the method further comprises: taking any video frame in the video frame sequence as a to-be-compressed video frame, obtaining a video frame adjacent to the to-be-compressed video frame, to obtain an adjacent video frame corresponding to the to-be-compressed video frame; calculating an image similarity between the to-be-compressed video frame and the adjacent video frame to obtain a second similarity; if the second similarity is greater than a preset similarity threshold, fusing the to-be-compressed video frame and the adjacent video frame to obtain a fused video frame; adding the fused video frame as a new video frame to the video frame sequence, and deleting the to-be-compressed video frame and the adjacent video frame in the video frame sequence to obtain a compressed video frame sequence; calculating an image similarity between each video frame and each centroid video frame to obtain a first similarity, and selecting a video frame whose first similarity satisfies a preset condition to obtain a key video frame, comprising: calculating an image similarity between each video frame in the compressed video frame sequence and each centroid video frame to obtain a first similarity; selecting a video frame whose first similarity satisfies a preset condition from each video frame in the compressed video frame sequence to obtain a key video frame.
[0008] In an embodiment, image information extraction is performed on each key video frame to obtain key information corresponding to the video frame sequence, comprising: obtaining a video frame adjacent to the key video frame from the video frame sequence, and combining the key video frame and the video frame adjacent to the key video frame to obtain a key video segment corresponding to the key video frame; performing image information extraction on the key video segment corresponding to each key video frame to obtain key information corresponding to the video frame sequence.
[0009] In an embodiment, image information extraction is performed on the key video segment corresponding to each key video frame to obtain key information corresponding to the video frame sequence, comprising: inputting the key video segment into a pre-trained video summary generation model to obtain a video summary text output by the video summary generation model; inputting the video summary text into a pre-trained text feature extraction model to obtain a video text vector output by the text feature extraction model; and / or, inputting the key video frame corresponding to the key video segment into a pre-trained multi-modal feature extraction model to obtain a video image vector output by the multi-modal feature extraction model; based on the video text vector and / or the video image vector corresponding to each key video frame, obtaining key information corresponding to the video frame sequence.
[0010] The second aspect of the present application provides a video question and answer method, the method comprising: obtaining video question data input by a user, and obtaining key information corresponding to a base library video frame sequence; wherein the key information corresponding to the base library video frame sequence is obtained based on any video key information extraction method; generating video answer data based on the video question data and the key information corresponding to the base library video frame sequence.
[0011] In an embodiment, the base video frame sequence includes a plurality of key video frames, the key information includes a video text vector corresponding to each key video frame and / or a video image vector corresponding to each key video frame, and the video question data includes a question text; based on the video question data and the key information corresponding to the base video frame sequence, the video answer data is generated, including: inputting the question text into a pre-trained text feature extraction model to obtain a question text vector output by the text feature extraction model; and / or inputting the question text into a pre-trained multi-modal feature extraction model to obtain a multi-modal vector output by the multi-modal feature extraction model; calculating a similarity between the video text vector of each key video frame and the question text vector to obtain a text vector similarity corresponding to each key video frame; and / or calculating a similarity between the video image vector of each key video frame and the multi-modal vector to obtain a multi-modal vector similarity corresponding to each key video frame; based on the text vector similarity and / or the multi-modal vector similarity corresponding to each key video frame, a question matching degree corresponding to each key video frame is calculated respectively; a key video frame satisfying a preset condition in terms of the question matching degree is selected to obtain a matching video frame; and based on each matching video frame, the video answer data is generated.
[0012] In an embodiment, the base video frame sequence further includes a plurality of normal video frames; based on each matching video frame, the video answer data is generated, including: selecting a normal video frame adjacent to the matching video frame from the base video frame sequence, combining the matching video frame and the normal video frame adjacent to the matching video frame to obtain a matching video clip corresponding to the matching video frame; inputting the matching video clip and the question text into a pre-trained question response model to obtain an initial answer of the matching video clip relative to the question text output by the question response model; and based on the initial answer corresponding to each matching video clip, the video answer data is generated.
[0013] In an embodiment, based on the initial answer corresponding to each matching video clip, the video answer data is generated, including: based on time information of each matching video clip, the initial answer corresponding to each matching video clip is sorted in chronological order to obtain a result sorting sequence; and inputting the result sorting sequence and the question text into a pre-trained answer summarization model to obtain the video answer data output by the answer summarization model.
[0014] The third aspect of the present application provides a video key information extraction device, the device comprising: a motion calculation module, configured to obtain a video frame sequence, calculate the intensity of motion of pixels in the video frame sequence, and obtain a motion amplitude value corresponding to each video frame in the video frame sequence; a weight calculation module, configured to calculate a clustering weight corresponding to each video frame based on the motion amplitude value corresponding to each video frame; wherein the motion amplitude value and the clustering weight are positively correlated; a weighted clustering module, configured to perform weighted clustering on each video frame based on the clustering weight corresponding to each video frame to obtain a plurality of clustering clusters, and take a video frame corresponding to a cluster center point of each clustering cluster as a centroid video frame; a key frame selection module, configured to calculate an image similarity between each video frame and each centroid video frame to obtain a first similarity, select a video frame satisfying a preset condition in the first similarity, and obtain a key video frame; and an information extraction module, configured to perform image information extraction on each key video frame to obtain key information corresponding to the video frame sequence.
[0015] The fourth aspect of the present application provides a video question and answer device, the device comprising: a data acquisition module, configured to acquire video question data input by a user, and acquire key information corresponding to a base video frame sequence; wherein the key information corresponding to the base video frame sequence is obtained based on any one of the video key information extraction methods; and an answer generation module, configured to generate video answer data based on the video question data and the key information corresponding to the base video frame sequence.
[0016] The fifth aspect of the present application provides an electronic device, comprising a memory and a processor, the processor being configured to execute program instructions stored in the memory to implement the video key information extraction method or the video question and answer method.
[0017] The sixth aspect of the present application provides a computer readable storage medium, having program instructions stored thereon, the program instructions being executed by a processor to implement the video key information extraction method or the video question and answer method.
[0018] The above scheme, by obtaining a video frame sequence, calculating the intensity of motion of pixels in the video frame sequence, obtaining a motion amplitude value corresponding to each video frame in the video frame sequence, calculating a clustering weight corresponding to each video frame based on the motion amplitude value corresponding to each video frame, performing weighted clustering on each video frame based on the clustering weight corresponding to each video frame to obtain a plurality of clustering clusters, taking a video frame corresponding to a cluster center point of each clustering cluster as a centroid video frame, can better process video frames with different importance, thereby obtaining more accurate and reliable centroid video frames, then, according to the image similarity between each video frame in the video frame sequence and each centroid video frame, key video frames are selected, image information extraction is performed on each key video frame, and more accurate key information can be extracted, ensuring that the key information is not lost.
[0019] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and are not restrictive of the application. BRIEF DESCRIPTION OF DRAWINGS
[0020] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and serve to explain the technical solutions of the application together with the specification.
[0021] Figure 1 is a schematic diagram of a scheme implementation environment shown by an exemplary embodiment of the application;
[0022] Figure 2 is a flowchart of a video key information extraction method shown by an exemplary embodiment of the application;
[0023] Figure 3 is a flowchart of a video key information extraction method shown by another exemplary embodiment of the application;
[0024] Figure 4 is a flowchart of a video question and answer method shown by an exemplary embodiment of the application;
[0025] Figure 5 is a flowchart of a video question and answer method shown by another exemplary embodiment of the application;
[0026] Figure 6 is a block diagram of a video key information extraction apparatus shown by an exemplary embodiment of the application;
[0027] Figure 7 is a block diagram of a video question and answer apparatus shown by an exemplary embodiment of the application;
[0028] Figure 8 is a structural schematic diagram of an electronic device shown by an exemplary embodiment of the application;
[0029] Figure 9 is a structural schematic diagram of a computer readable storage medium shown by an exemplary embodiment of the application. DETAILED DESCRIPTION
[0030] The scheme of the embodiments of the application will be described in detail below with reference to the accompanying drawings.
[0031] In the following description, for the purpose of explanation and not limitation, specific details are set forth, such as particular system configurations, interfaces, techniques, in order to provide a thorough understanding of the application.
[0032] The term "and / or", used in the present text, is only used to describe an associated relationship between associated objects, indicating that there can be three relationships, for example, A and / or B, which can represent three cases: A exists alone, A and B exist together, and B exists alone. In addition, the character " / " in the present text generally represents an "or" relationship between the associated objects before and after it. In addition, "multiple" in the present text means two or more than two. In addition, the term "at least one" in the present text means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B and C, which can mean including any one or more elements selected from the set consisting of A, B and C.
[0033] The video key information extraction method and the video question and answer method provided by the embodiments of the present application will be described below.
[0034] Please refer to Figure 1 , Figure 1 is a schematic diagram of a scheme implementation environment shown by an exemplary embodiment of the present application. The scheme implementation environment can include a terminal 110 and a server 120, which are connected in communication with each other.
[0035] The number of terminals 110 can be one or more. The terminal 110 can be a camera, a smartphone, a tablet computer, a notebook computer, a desktop computer, a smart watch, etc., but is not limited thereto.
[0036] The server 120 can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms, etc. basic cloud computing services.
[0037] In one example, the server 120 can perform key information extraction processing on the video frame sequence obtained from the terminal 110 to obtain the key information corresponding to the video frame sequence, or the server 120 can perform question answering processing on the video question data obtained from the terminal 110 to obtain video answer data. Of course, the server 120 can store the key information corresponding to the video frame sequence or the video answer data locally, back to the terminal 110 or transmit to other terminals.
[0038] In one example, a client of a target application installed and running in the terminal 110, such as the target application can be an application providing a key information extraction function or a question answering function, performs key information extraction processing on the video frame sequence to obtain key information corresponding to the video frame sequence, or performs question answering processing on the video question data to obtain video answer data. The server 120 can be a background server of the target application, configured to provide background service for the client of the target application.
[0039] The video key information extraction method or the video question answering method provided by the embodiments of the present application can be performed by the terminal 110, such as the client of the target application installed and running in the terminal 110, or the server 120, or by the terminal 110 and the server 120 in cooperation.
[0040] Please refer to Figure 2 , Figure 2 is a flowchart of a video key information extraction method according to an example embodiment of the present application. The video key information extraction method can be applied to the implementation environment shown in Figure 1 and performed by the server in the implementation environment. It should be understood that the method can also be applied to other example implementation environments and performed by devices in other implementation environments, and the embodiments of the present application do not limit the implementation environment to which the method is applied.
[0041] As shown in Figure 2 , the video key information extraction method includes at least steps S210 to S250, which are described in detail as follows.
[0042] Step S210: Obtain a video frame sequence, and calculate the degree of motion of pixels in the video frame sequence to obtain a motion amplitude value corresponding to each video frame in the video frame sequence.
[0043] The video frame sequence includes a plurality of video frames sorted in time sequence.
[0044] The video frame sequence can be generated by image acquisition of a preset scene by a terminal, and the server obtains the video frame sequence to be analyzed from the terminal; or the video frame sequence can be stored in a preset database, and the server obtains the video frame sequence to be analyzed from the preset database, and the present application does not limit the manner of obtaining the video frame sequence.
[0045] The degree of motion of pixels in the video frame sequence is calculated to obtain a motion amplitude value corresponding to each video frame in the video frame sequence.
[0046] Exemplarily, the intensity of motion of pixels in the video frame sequence can be calculated by an optical flow method, and / or an inter-frame difference method, and / or constructing a displacement-time image of the object, etc., to obtain the motion amplitude value corresponding to each video frame in the video frame sequence.
[0047] For example, the optical flow represents the displacement of pixels in the same scene collected at different times. By calculating the optical flow, the motion trajectory and speed of the object in the image can be obtained. Specifically, the motion vector of the pixel in the video frame sequence is calculated, the optical flow amplitude value representing the motion speed is extracted from the motion vector, the intensity of motion of pixels in adjacent two video frames is represented by the optical flow amplitude value, and the optical flow amplitude value is taken as the motion amplitude value corresponding to the video frame. The larger the motion amplitude value is, the more intense the motion event or behavior between the two video frames is, that is, the more likely it contains key information. Conversely, the smaller the motion amplitude value is, the more stable and static the content between the two video frames is, and the less likely it contains key information.
[0048] For another example, the inter-frame difference method calculates the pixel value difference between adjacent video frames. The larger the pixel value difference between adjacent video frames is, the larger the motion amplitude value corresponding to the video frame is; conversely, the smaller the pixel value difference between adjacent video frames is, the smaller the motion amplitude value corresponding to the video frame is. The intensity of motion of pixels is represented by the pixel value difference between adjacent video frames, and the motion amplitude value corresponding to each video frame in the video frame sequence is obtained.
[0049] For another example, in the displacement-time image of the object, the size of the slope represents the speed of the motion of the object in the video frame. The larger the absolute value of the slope is, the faster the object moves, and the larger the motion amplitude value corresponding to the video frame is; conversely, the slower the object moves, and the smaller the motion amplitude value corresponding to the video frame is. By analyzing the displacement-time image, the intensity of motion of pixels is determined, and the motion amplitude value corresponding to each video frame in the video frame sequence is obtained.
[0050] Step S220: based on the motion amplitude value corresponding to each video frame, the clustering weight corresponding to each video frame is calculated respectively.
[0051] The motion amplitude value and the clustering weight are positively correlated, that is, the larger the motion amplitude value of the video frame is, the larger the clustering weight of the video frame is, and the smaller the motion amplitude value of the video frame is, the smaller the clustering weight of the video frame is.
[0052] Exemplarily, a weight calculation formula can be pre-built to describe the mapping relationship between the motion amplitude value and the clustering weight, and the clustering weight corresponding to the video frame is calculated by inputting the motion amplitude value into the weight calculation formula. The weight calculation formula can be flexibly set according to the specific application scenario, and the motion amplitude value and the clustering weight in the weight calculation formula are in a positive correlation relationship, which is not limited by the implementation manner of the weight calculation formula.
[0053] Exemplarily, the motion amplitude value can also be compared with a pre-set threshold, different threshold values correspond to different clustering weights, and the clustering weight corresponding to the video frame is determined according to the size comparison relationship between the motion amplitude value and the threshold.
[0054] Of course, other weight distribution algorithms can also be used to calculate the clustering weight of the video frame corresponding to the motion amplitude value, such as comparing the sizes of the motion amplitude values of each video frame to set the clustering weight corresponding to each video frame, and the specific calculation manner of the clustering weight is not limited by the present application.
[0055] Step S230: based on the clustering weight corresponding to each video frame, each video frame is weighted and clustered to obtain a plurality of clustering clusters, and the video frame corresponding to the cluster center point of each clustering cluster is taken as the centroid video frame.
[0056] The weighted clustering algorithm considers the importance of each video frame when calculating the cluster center point of the clustering cluster, and the importance of the video frame is represented according to the clustering weight of each video frame, so that the video frame with a larger weight has a greater influence when calculating the cluster center point of the clustering cluster.
[0057] The clustering algorithm can be a K-means clustering algorithm (K-means), a hierarchical clustering algorithm, a spectral clustering algorithm, etc., and the type of the clustering algorithm used is not limited by the present application.
[0058] Taking K-means as an example: K video frames are selected as initial cluster center points. According to the distance of each video frame to each cluster center point, the video frame is assigned to the nearest cluster. According to the clustering weight and coordinates of the video frames in each cluster, a new cluster center point is calculated, wherein the video frame with a larger clustering weight has a greater contribution to the calculation of the cluster center point. According to the distance of each video frame to the newly calculated cluster center point, the clustering weight of the video frame is updated. The above steps are repeated until the stop condition is met, for example, the maximum number of iterations is reached or the cluster center point converges, and the K cluster center points and the cluster label of each video frame obtained finally are the output results of the weighted K-means algorithm.
[0059] The video frame corresponding to the cluster center point of each clustering cluster is taken as the centroid video frame.
[0060] For example, weighted K-means clustering is performed, and finally K centroid video frames are obtained. The greater the value of K is, the smaller the difference between the centroid video frames is, and the smaller the information loss is. However, the subsequent redundant information and the calculation overhead caused thereby are also increased. Therefore, the value of K can be set according to the calculation resources available in the current application scenario. The greater the calculation resources available are, the greater the value of K is. Of course, the value of K can also be set according to experience, which is not limited in the present application.
[0061] By assigning a clustering weight to each video frame according to the motion amplitude value corresponding to the video frame, and performing weighted clustering on the video frame sequence based on the clustering weight, the video frames with different importance can be better processed, so that a more accurate and reliable clustering result is obtained.
[0062] Step S240: Calculate the image similarity between each video frame and each centroid video frame to obtain a first similarity, and select a video frame whose first similarity satisfies a preset condition to obtain a key video frame.
[0063] After obtaining the centroid video frames, the image similarity between each video frame in the video frame sequence and each centroid video frame is calculated to obtain a first similarity.
[0064] For example, image feature vectors can be extracted from the video frames and the centroid video frames, and the vector distance between the extracted image feature vectors is calculated to obtain the image similarity between the video frames and the centroid video frames. Alternatively, the pixel value difference between the video frames and the centroid video frames can be calculated to obtain the image similarity between the video frames and the centroid video frames according to the size of the pixel value difference. The specific calculation method of the image similarity can be flexibly selected according to the actual application, which is not limited in the present application.
[0065] The video frame whose first similarity satisfies the preset condition is selected to obtain a key video frame.
[0066] For example, it can be determined whether the first similarity between the video frame and any centroid video frame is greater than a preset similarity threshold. If the first similarity between the video frame and any centroid video frame is greater than the preset similarity threshold, the video frame is reserved as a key video frame.
[0067] For example, the first similarity between the video frame and each centroid video frame can be obtained, and the number of all first similarities greater than a preset similarity threshold corresponding to the video frame is counted. If the number is greater than a preset number threshold, the video frame is reserved as a key video frame.
[0068] For example, the first similarity between the video frame and each centroid video frame can be obtained, the average of all the first similarities corresponding to the video frame is calculated, and it is determined whether the average is greater than a preset preset similarity threshold. If the average is greater than the preset preset similarity threshold, the video frame is reserved as a key video frame.
[0069] For example, the first similarity between the video frame and each centroid video frame can be obtained, the maximum value in all the first similarities corresponding to the video frame is counted, and it is determined whether the maximum value is greater than a preset preset similarity threshold. If the maximum value is greater than the preset preset similarity threshold, the video frame is reserved as a key video frame.
[0070] The preset similarity threshold can be pre-set according to experience, or can be flexibly calculated according to actual application conditions. For example, the preset similarity threshold can be set according to the available computing resources in the current application scenario. The more computing resources that can be used, the smaller the preset similarity threshold, and the more key video frames that are finally retained. The application does not limit the setting method of the preset similarity threshold.
[0071] Step S250: image information extraction is performed on each key video frame to obtain key information corresponding to the video frame sequence.
[0072] The image information of the key video frame is extracted, and the extraction result corresponding to each key video frame is taken as the key information corresponding to the video frame sequence.
[0073] The image information includes but is not limited to scene information, target object information, behavior event information, time information, weather information, etc. The specific image information can be flexibly selected according to the actual application scenario, and the application does not limit the type of image information.
[0074] The extracted key information corresponding to the video frame sequence can be used to implement downstream tasks such as video question answering, video generation, and video recommendation.
[0075] The application can better process video frames with different importance by assigning a clustering weight to each video frame according to the motion amplitude value of the video frame, weighting and clustering the video frame sequence based on the clustering weight, thereby obtaining more accurate and reliable centroid video frames. Then, according to the image similarity between each video frame in the video frame sequence and each centroid video frame, the key video frame is selected, and the image information of each key video frame is extracted, so that more accurate key information can be extracted, and the loss of key information is prevented.
[0076] Next, some embodiments of the application will be described in detail.
[0077] In some embodiments, the image similarity between each video frame and each centroid video frame is calculated in step S240 to obtain a first similarity, and video frames satisfying a preset condition in the first similarity are selected to obtain the key video frames, including steps S241 to S243.
[0078] Step S241: The video frame sequence is compressed to obtain a compressed video frame sequence.
[0079] Step S242: The image similarity between each video frame in the compressed video frame sequence and each centroid video frame is calculated to obtain a first similarity.
[0080] Step S243: From each video frame in the compressed video frame sequence, a video frame satisfying a preset condition in the first similarity is selected to obtain the key video frames.
[0081] By compressing the video frame sequence, the redundant information of the video frame sequence is removed.
[0082] Exemplarily, the step of compressing the video frame sequence includes steps S2411 to S2415.
[0083] Step S2411: Any video frame in the video frame sequence is taken as a to-be-compressed video frame, and adjacent video frames of the to-be-compressed video frame are obtained to obtain adjacent video frames corresponding to the to-be-compressed video frame.
[0084] The first N video frames and / or the last N video frames of the to-be-compressed video frame in the video frame sequence can be taken as the adjacent video frames of the to-be-compressed video frame. N is a positive integer greater than or equal to 1, and the value of N can be pre-set according to experience, or the value of N can be set according to the available computing resources in the current application scenario, for example, the more the available computing resources, the greater the value of N, which is not limited in the present application.
[0085] Step S2412: The image similarity between the to-be-compressed video frame and the adjacent video frames is calculated to obtain a second similarity.
[0086] The specific calculation method of the image similarity can refer to step S240 described above, which is not repeated here.
[0087] Step S2413: It is judged whether the second similarity is greater than a preset similarity threshold, if yes, steps S2414 and S2415 are executed, and if not, step S2411 is continuously executed until all video frames in the video frame sequence are traversed.
[0088] Step S2414: The to-be-compressed video frame and the adjacent video frames are fused to obtain a fused video frame.
[0089] The image fusion method includes pixel-level fusion and feature-level fusion. The pixel-level fusion obtains a fused image by weighted average or other operations on the pixels of the images, and the pixel-level fusion includes weighted average method, wavelet transform method, etc.; the feature-level fusion obtains a fused image by extracting features from the images and then fusing the features, and the feature-level fusion includes Principal Components Analysis (PCA), Scale-invariant feature transform (SIFT), etc.
[0090] The pixel-level fusion and / or the feature-level fusion are performed on the to-be-compressed video frame and the adjacent video frames to obtain a fused video frame.
[0091] In step S2415, the fused video frame is added to the video frame sequence as a new video frame, and the to-be-compressed video frame and the adjacent video frames in the video frame sequence are deleted to obtain a compressed video frame sequence.
[0092] In some embodiments, if the second similarity between the to-be-compressed video frame and any adjacent video frame is greater than the preset similarity threshold, one of the to-be-compressed video frame and the adjacent video frame can also be deleted.
[0093] By compressing the video frame sequence in the above embodiments, the subsequent calculation cost and calculation time consumption are greatly reduced on the premise of ensuring that the video information is not lost.
[0094] In some embodiments, the image information extraction in step S250 on each key video frame to obtain the key information corresponding to the video frame sequence includes steps S251 to S252.
[0095] In step S251, the video frames adjacent to the key video frame are obtained from the video frame sequence, and the key video frame and the video frames adjacent to the key video frame are combined to obtain a key video clip corresponding to the key video frame.
[0096] The acquisition method of the adjacent video frames can be referred to step S2411, which is not described herein.
[0097] The key video frame and the video frames adjacent to the key video frame are combined in time sequence to obtain a key video clip corresponding to the key video frame.
[0098] In step S252, the image information extraction is performed on the key video clip corresponding to each key video frame to obtain the key information corresponding to the video frame sequence.
[0099] The key video clip retains more image information, and the image information extraction is performed on each key video clip to obtain the key information corresponding to the video frame sequence, thereby avoiding omission of the key information.
[0100] Exemplarily, the key video clip is input into a pre-trained video summary generation model to obtain a video summary text output by the video summary generation model; the video summary text is input into a pre-trained text feature extraction model to obtain a video text vector output by the text feature extraction model; and / or the key video frame corresponding to the key video clip is input into a pre-trained multi-modal feature extraction model to obtain a video image vector output by the multi-modal feature extraction model; based on the video text vector and / or the video image vector corresponding to each key video frame, key information corresponding to the video frame sequence is obtained.
[0101] The video summary generation model can be constructed based on a visual language model (VLM), a convolutional neural network (CNN), a recurrent neural network (RNN), etc., and the implementation of the video summary generation model is not limited in the present application.
[0102] Taking the VLM model as an example, the VLM is used to generate a summary description for each key video clip. Specifically, the key video clip is input into the VLM as an input prompt to generate a summary description, and the template of the input prompt is as follows:
[0103] {{text: The multiple images given below are continuous video frames contained in the key video clip. Analyze the content of the key video clip to output a video summary text. Pay attention to the following aspects of the key video clip: time information, scene / place / environment information, target objects appearing in the video, behavior events occurring in the video, and output the results according to the following template:
[0104] {Time:
[0105] Scene / place / environment:
[0106] Target object information:
[0107] Behavior event:
[0108] }}
[0109] {image_url: Video frame 1 of the key video clip} ...
[0110] {image_url: Video frame M of the key video clip}
[0111] Wherein, M is the total number of video frames in the key video segment, and the information that needs to be focused on in the key video segment can be flexibly set according to actual conditions when generating the summary.
[0112] Then, the video summary text is input into the pre-trained text feature extraction model to obtain a video text vector output by the text feature extraction model; and / or, the key video frames corresponding to the key video segment are input into the pre-trained multi-modal feature extraction model to obtain a video image vector output by the multi-modal feature extraction model, and the video text vector and / or the video image vector corresponding to each key video frame are integrated to obtain key information corresponding to the video frame sequence.
[0113] Wherein, the text feature extraction model is used for encoding the input text; the multi-modal feature extraction model is used for encoding the input image and / or text, and can realize comprehensive processing of multiple modal data. The text feature extraction model and the multi-modal feature extraction model can be constructed based on a bidirectional transformer model (Bidirectional Encoder Representations from Transformers, BERT), CNN, RNN, etc., and the implementation of the text feature extraction model and the multi-modal feature extraction model is not limited in the present application.
[0114] By integrating the features of multiple dimensions of text and image, the representation ability of the extracted key information is improved, which facilitates the execution of downstream tasks.
[0115] In some embodiments, please refer to Figure 3 , Figure 3 is a flow chart of a video key information extraction method according to another exemplary embodiment of the present application, as shown in Figure 3 , comprising steps S301 to S307.
[0116] Step S301: Obtain a video frame sequence.
[0117] Step S302: Calculate the optical flow amplitude value corresponding to each video frame in the video frame sequence, and set the clustering weight corresponding to each video frame according to the optical flow amplitude value.
[0118] Step S303: According to the clustering weight corresponding to each video frame, each video frame is weighted and clustered to obtain a centroid video frame.
[0119] Step S304: Compress the video frame sequence to obtain a compressed video frame sequence.
[0120] The step S304 has no execution sequence relationship with the steps S302 and S303, and can be executed in parallel with the steps S302 and S303, or the steps S302 and S303 can be executed first, and then the step S304 is executed, or the step S304 can be executed first, and then the steps S302 and S303 are executed.
[0121] Step S305: From each video frame in the compressed video frame sequence, a video frame that meets a preset condition in image similarity with the centroid video frame is selected, to obtain a key video frame.
[0122] Step S306: A key video segment corresponding to the key video frame is obtained, and a video summary text corresponding to the key video segment is generated.
[0123] Step S307: A video text vector is obtained by encoding the video summary text, and a video image vector is obtained by encoding the key video frame corresponding to the key video segment, to obtain key information corresponding to the key video frame.
[0124] The specific implementation of each step is described in the foregoing embodiments, and will not be repeated here.
[0125] Please refer to Figure 4 , Figure 4 is a flowchart of a video question-answering method according to an example embodiment of the present application. The video question-answering method can be applied to Figure 1 the implementation environment shown in the figure, and is specifically executed by a server in the implementation environment. It should be understood that the method can also be applied to other example implementation environments, and is specifically executed by devices in other implementation environments, and the present embodiment does not limit the implementation environment to which the method is applied.
[0126] As shown in Figure 4 , the video question-answering method at least includes steps S410 to S420, which are described in detail as follows.
[0127] Step S410: Video question data input by a user is obtained, and key information corresponding to a base library video frame sequence is obtained.
[0128] The key information corresponding to the base library video frame sequence is obtained based on any one of the video key information extraction methods described above.
[0129] The video question data includes but is not limited to text, audio, image, etc., and the present application does not limit the data type of the video question data.
[0130] The video question data can be used to indicate video query, video content analysis, etc. For example, the video question data can be "query video of rainy weather", to indicate that the video segment related to the weather information of raining is queried; the video question data can also be "query video of rainy weather".
[0131] Step S420: generating video answer data based on the video question data and the key information corresponding to the base video frame sequence.
[0132] According to the key information corresponding to the base video frame sequence, the video question data and each video frame in the base video frame sequence are matched to determine the video frame in the base video frame sequence that matches the video question data, and the video answer data is generated according to the matched video frame.
[0133] For example, if the video query question, the video query result can be directly generated according to the matched video frame, and the video query result is taken as the video answer data. If the video content analysis service, after obtaining the matched video frame, the image content of the matched video frame is analyzed according to the video question data, such as analyzing the time, scene, location, environment, target object information, behavior event and other information of the matched video frame, to obtain the analysis result related to the video question data, and the analysis result is taken as the video answer data. It should be noted that the specific mode of the video answer data can be flexibly selected according to the specific business type, the questioning intention of the video question and answer data, and the like, which is not limited in the present application.
[0134] Exemplarily, the base video frame sequence contains a plurality of key video frames, the key information includes a video text vector corresponding to each key video frame, and / or a video image vector corresponding to each key video frame, and the video question data contains a question text. Based on the video question data and the key information corresponding to the base video frame sequence, the video answer data is generated, including: inputting the question text into the text feature extraction model to obtain the question text vector output by the text feature extraction model; and / or inputting the question text into the multi-modal feature extraction model to obtain the multi-modal vector output by the multi-modal feature extraction model; calculating the similarity between the video text vector of each key video frame and the question text vector to obtain the text vector similarity corresponding to each key video frame; and / or calculating the similarity between the video image vector of each key video frame and the multi-modal vector to obtain the multi-modal vector similarity corresponding to each key video frame; based on the text vector similarity and / or the multi-modal vector similarity corresponding to each key video frame, the question matching degree corresponding to each key video frame is calculated respectively; selecting the key video frame whose question matching degree meets the preset condition to obtain the matched video frame; and generating the video answer data based on each matched video frame.
[0135] The acquisition method of the key video frame can be referred to the above steps S210 to S240, which will not be repeated here.
[0136] The similarity between two vectors can be measured by calculating the Euclidean distance, and / or inner product, and / or cosine similarity, etc. between the two vectors.
[0137] That is, the text vector similarity corresponding to the key video frame is obtained according to the Euclidean distance, and / or inner product, and / or cosine similarity, etc. between the video text vector and the question text vector; and the multi-modal vector similarity corresponding to the key video frame is obtained according to the Euclidean distance, and / or inner product, and / or cosine similarity, etc. between the video image vector and the multi-modal vector.
[0138] The question matching degree corresponding to each key video frame is calculated respectively according to the text vector similarity and / or the multi-modal vector similarity corresponding to each key video frame.
[0139] If one of the text vector similarity and the multi-modal vector similarity exists, it is directly taken as the question matching degree corresponding to the key video frame.
[0140] If the text vector similarity and the multi-modal vector similarity both exist, any one of the text vector similarity and the multi-modal vector similarity can be selected as the question matching degree corresponding to the key video frame, such as selecting the maximum or minimum value of the text vector similarity and the multi-modal vector similarity to obtain the question matching degree corresponding to the key video frame; or the text vector similarity and / or the multi-modal vector similarity can be weighted and summed to obtain the question matching degree corresponding to the key video frame.
[0141] Exemplarily, the difference between the text vector similarity and the multi-modal vector similarity is calculated to obtain a similarity difference value, if the similarity difference value is greater than a preset difference threshold, the minimum value of the text vector similarity and the multi-modal vector similarity is selected to obtain the question matching degree corresponding to the key video frame, if the similarity difference value is not greater than the preset difference threshold, the maximum value of the text vector similarity and the multi-modal vector similarity is selected to obtain the question matching degree corresponding to the key video frame. Alternatively, the weighting weights corresponding to the text vector similarity and the multi-modal vector similarity are set according to the similarity difference value, and the text vector similarity and the multi-modal vector similarity are weighted and summed according to the weighting weights to obtain the question matching degree corresponding to the key video frame, for example, the greater the similarity difference value, the greater the weighting weight corresponding to the minimum value of the text vector similarity and the multi-modal vector similarity, and the smaller the weighting weight corresponding to the maximum value of the two; the smaller the similarity difference value, the greater the weighting weight corresponding to the maximum value of the text vector similarity and the multi-modal vector similarity, and the smaller the weighting weight corresponding to the minimum value of the two.
[0142] The above embodiments determine the credibility of the similarity calculation result between the same key video frame and the video question data by a similarity difference value. The greater the similarity difference value, the lower the credibility of the similarity calculation result. At this time, the minimum value of the text vector similarity and the multi-modal vector similarity can be selected, or more weighted weights are assigned to the minimum value to obtain the problem matching degree corresponding to the key video frame. Conversely, the smaller the similarity difference value, the higher the credibility of the similarity calculation result. At this time, the maximum value of the text vector similarity and the multi-modal vector similarity can be selected, or more weighted weights are assigned to the maximum value to obtain the problem matching degree corresponding to the key video frame. The accuracy of the problem matching degree calculation is improved, and more video frame image information is retained.
[0143] Then, the key video frame whose problem matching degree meets a preset condition is selected to obtain a matching video frame.
[0144] For example, the key video frame whose problem matching degree is greater than a preset matching degree threshold can be selected to obtain the matching video frame. In addition, each key video frame can be sequentially sorted according to the problem matching degree, and a preset number of key video frames with the largest problem matching degree can be selected to obtain the matching video frame. For example, each key video frame can be sorted in descending order according to the problem matching degree, and the first W key video frames are selected as the matching video frames.
[0145] According to each matching video frame, video answer data is generated.
[0146] In some embodiments, the base library video frame sequence also contains a plurality of normal video frames. Based on each matching video frame, the video answer data is generated, including: selecting a normal video frame adjacent to the matching video frame from the base library video frame sequence, combining the matching video frame and the normal video frame adjacent to the matching video frame to obtain a matching video segment corresponding to the matching video frame; inputting the matching video segment and the question text into a pre-trained question response model to obtain an initial answer of the matching video segment relative to the question text output by the question response model; and generating the video answer data based on the initial answer corresponding to each matching video segment.
[0147] The acquisition method of the adjacent normal video frame can be referred to step S2411, which is not repeated here. The base library video frame sequence in the present application can be a compressed video frame sequence, and the compression step can be referred to steps S2411 to S2415, which are not repeated here.
[0148] The matching video frame and the normal video frame adjacent to the matching video frame are combined in time sequence to obtain a matching video segment corresponding to the matching video frame.
[0149] For example, in a video query service, the matching video segments and the question text are input into a question response model, which is used to analyze the correlation between the matching video segments and the question text to obtain an initial answer. The question response model can be implemented based on a neural network model architecture such as VLM, and / or RNN, and / or CNN, and the like, which is not limited in the present application.
[0150] Taking the question response model as an example, the video frames of each matching video segment and the question text input by the user are combined into a Prompt and input into the VLM in parallel, the VLM is used for reasoning, and finally a plurality of initial answers are generated. For example, the format of the Prompt template is as follows, wherein “XXXX” is replaced by the question text input by the user:
[0151] {{text: The multiple images given below are consecutive video frames corresponding to a matching video segment, which answers the question: XXXX in combination with the image content of the matching video segment. If the image content of the matching video segment is irrelevant to the question, answer: the current matching video segment is irrelevant to the question}
[0152] {image_url: video frame 1 of the matching video segment} ...
[0153] {image_url: video frame r of the matching video segment}
[0154] Wherein, r is the total number of video frames in the input matching video segment, and the initial answer output by the question response model is used to represent whether the matching video segment and the question text are related.
[0155] Of course, in addition to the initial answer representing whether the matching video segment and the question text are related in the above embodiment, the question response model can also input the specific degree of correlation between the matching video segment and the question text, and take the correlation degree as the initial answer, or in other services, the initial answer can also be an image content analysis result, such as the key information obtained by analysis, which is not limited in the present application.
[0156] Then, according to the initial answer corresponding to each matching video segment, video answer data is generated.
[0157] For example, based on the initial answer corresponding to each matching video segment, the video answer data is generated, including: based on the time information of each matching video segment, the initial answer corresponding to each matching video segment is sorted in chronological order to obtain a result sorting sequence; and the result sorting sequence and the question text are input into a pre-trained answer summary model to obtain video answer data output by the answer summary model.
[0158] Specifically, according to the collection time of the key video frame in the matched video segment, the initial answers corresponding to each matched video segment are sequentially sorted, and the initial answers corresponding to the matched video segment with the earlier collection time of the key video frame are sorted earlier, to obtain a result sorting sequence.
[0159] Then, the result sorting sequence and the question text are input into the answer summarization model, the answer summarization model summarizes the input information, and video answer data is output. The video answer data is realized based on a large language model (LLM), and / or a neural network model architecture such as RNN and / or CNN, which is not limited in the present application.
[0160] Taking the problem response model as an example, the result sorting sequence and the question text input by the user are combined to generate an input Prompt input into the LLM, and the LLM is used to summarize and summarize the final answer to generate video answer data. For example, the result sorting sequence is 1.AA; 2.BB; …; r.RR as the initial answer, and the Prompt template format is as follows, wherein “XXXX” is replaced by the question text input by the user:
[0161] {You are an expert in summarizing and summarizing video question and answer answers, and you are given a question about a video and multiple initial answers to the question at different time nodes. Summarize and summarize these initial answers, remove irrelevant answer content, combine related answers, and output video answer data. The video question and answer question and multiple answers are as follows:
[0162] Video question and answer question: XXXX
[0163] Multiple initial answers in time sequence: 1.AA; 2.BB; …; r.RR
[0164] In some embodiments, please refer to Figure 5 , Figure 5 is another example of a flowchart of a video question and answer method according to an example embodiment of the present application, as Figure 5 shown, comprising steps S501 to S506.
[0165] Step S501: Obtain video question data input by a user.
[0166] Step S502: Obtain matched video frames corresponding to the video question data from the bottom library video frame sequence.
[0167] Step S503: Obtain matched video segments corresponding to the matched video frames.
[0168] Step S504: An initial answer of each matched video clip relative to the question text is inferred using the question response model.
[0169] Step S505: Each initial answer is sorted according to time information to obtain a result sorting sequence.
[0170] Step S506: The result sorting sequence is summarized using the answer summary model to obtain video answer data.
[0171] The video question answering method provided in the application improves the correctness of the final video answer by performing feature neighborhood similarity calculation on the video question data and the base library video frame sequence, retrieving matched video frames, obtaining matched video clips corresponding to the matched video frames, generating multiple initial answers of the video question at different time points for each matched video clip, and finally summarizing and summarizing multiple initial answers.
[0172] Figure 6 is a block diagram of a video key information extraction device according to an example embodiment of the application. As shown in Figure 6 , the example video key information extraction device 600 includes:
[0173] The motion calculation module 610 is configured to obtain a video frame sequence, calculate the degree of motion intensity of pixels in the video frame sequence, and obtain a motion amplitude value corresponding to each video frame in the video frame sequence.
[0174] The weight calculation module 620 is configured to calculate a clustering weight corresponding to each video frame based on the motion amplitude value corresponding to each video frame; wherein the motion amplitude value and the clustering weight are positively correlated.
[0175] The weighted clustering module 630 is configured to perform weighted clustering on each video frame based on the clustering weight corresponding to each video frame to obtain a plurality of clustering clusters, and take a video frame corresponding to a cluster center point of each clustering cluster as a centroid video frame.
[0176] The key frame selection module 640 is configured to calculate a first similarity between each video frame and each centroid video frame to obtain a first similarity, and select a video frame satisfying a preset condition to obtain a key video frame.
[0177] The information extraction module 650 is configured to perform image information extraction on each key video frame to obtain key information corresponding to the video frame sequence.
[0178] Figure 7 is a block diagram of a video question answering device according to an example embodiment of the application. As shown in Figure 7 , the example video question answering device 700 includes:
[0179] The data acquisition module 710 is configured to acquire video question data input by a user and acquire key information corresponding to a base library video frame sequence; wherein the key information corresponding to the base library video frame sequence is obtained based on any one of the video key information extraction methods.
[0180] The answer generation module 720 is configured to generate video answer data based on the video question data and the key information corresponding to the base library video frame sequence.
[0181] It should be noted that the video key information extraction apparatus and the video question and answer apparatus provided in the above embodiments belong to the same concept as the video key information extraction method and the video question and answer method provided in the above embodiments, and the specific manner in which each module and unit performs operations has been described in detail in the method embodiments, which will not be described here. The video key information extraction apparatus and the video question and answer apparatus provided in the above embodiments can allocate the above functions to different functional modules to complete all or part of the functions described above according to actual application, i.e., the internal structure of the apparatus is divided into different functional modules to complete all or part of the functions described above, which is not limited herein.
[0182] Please refer to Figure 8 , Figure 8 is a structural schematic diagram of an embodiment of an electronic device of the present application. The electronic device 800 includes a memory 801 and a processor 802, and the processor 802 is configured to execute program instructions stored in the memory 801 to implement the steps in any of the above video key information extraction method or video question and answer method embodiments. In a specific implementation scenario, the electronic device 800 can include but is not limited to: a microcomputer, a server, in addition, the electronic device 800 can also include a notebook computer, a tablet computer and other mobile devices, which are not limited herein.
[0183] Specifically, the processor 802 is configured to control itself and the memory 801 to implement the steps in any of the above-described video key information extraction method or video question answering method embodiments. The processor 802 can also be referred to as a central processing unit (CPU). The processor 802 can be an integrated circuit chip including a processing core. The processor 802 can also be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. In addition, the processor 802 can be implemented by a combination of integrated circuit chips.
[0184] Please refer to Figure 9 , Figure 9 is a structural schematic diagram of an embodiment of the computer readable storage medium of the present application. The computer readable storage medium 900 stores program instructions 910 capable of being executed by a processor, and the program instructions 910 are used to implement the steps in any of the above-described video key information extraction method or video question answering method embodiments.
[0185] In some embodiments, the apparatus provided by the embodiments of the present disclosure has functions or includes modules that can be used to execute the methods described in the above method embodiments, and the specific implementation can refer to the description of the above method embodiments. For brevity, details are not repeated here.
[0186] The above description of various embodiments tends to emphasize the differences between various embodiments, and the same or similar parts can be mutually referred to. For brevity, details are not repeated here.
[0187] In several embodiments provided in the present application, it should be understood that the disclosed methods and apparatuses can be implemented in other ways. For example, the above-described apparatus implementation is only schematic, for example, the division of modules or units is only a logical function division, and actual implementation can have another division manner, for example, a unit or component can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed mutual elements can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.
[0188] In addition, each of the function units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit. When the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application, essentially or in the form of a contribution to the prior art, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to execute all or part of the steps of the methods in each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and various media that can store program codes.
Claims
1. A video question-answering method, characterized in that, The method includes: The system obtains video problem data input by the user and acquires key information corresponding to the video frame sequence in the base database. The key information includes the video text vector and the video image vector corresponding to each key video frame, and the video problem data contains problem text. The steps for obtaining key information corresponding to the base video frame sequence include: acquiring the video frame sequence; calculating the intensity of pixel motion in the video frame sequence to obtain the motion amplitude value corresponding to each video frame in the video frame sequence; calculating the clustering weight corresponding to each video frame based on the motion amplitude value corresponding to each video frame; wherein the motion amplitude value and the clustering weight are positively correlated; performing weighted clustering on each video frame based on the clustering weight corresponding to each video frame to obtain multiple clusters, and taking the video frame corresponding to the cluster center point of each cluster as the centroid video frame; compressing the video frame sequence to obtain a compressed video frame sequence; calculating the image similarity between each video frame in the compressed video frame sequence and each centroid video frame to obtain a first similarity; selecting video frames whose first similarity meets a preset condition from each video frame in the compressed video frame sequence to obtain key video frames; and extracting image information from each key video frame to obtain the key information corresponding to the video frame sequence. The question text is input into a pre-trained text feature extraction model to obtain the question text vector output by the text feature extraction model; the question text is input into a pre-trained multimodal feature extraction model to obtain the multimodal vector output by the multimodal feature extraction model. Calculate the similarity between the video text vector of each key video frame and the question text vector to obtain the text vector similarity corresponding to each key video frame; calculate the similarity between the video image vector of each key video frame and the multimodal vector to obtain the multimodal vector similarity corresponding to each key video frame; Calculate the difference between the text vector similarity and the multimodal vector similarity to obtain a similarity difference. Set weights for the text vector similarity and the multimodal vector similarity based on the similarity difference. Perform a weighted sum of the text vector similarity and the multimodal vector similarity based on the weights to obtain the problem matching degree corresponding to the key video frame. Select key video frames that meet the preset conditions for problem matching to obtain matching video frames; Video response data is generated based on each matched video frame.
2. The method according to claim 1, characterized in that, The step of compressing the video frame sequence to obtain a compressed video frame sequence includes: Take any video frame in the video frame sequence as the video frame to be compressed, obtain the video frames adjacent to the video frame to be compressed, and obtain the adjacent video frames corresponding to the video frame to be compressed. A second similarity is obtained by calculating the image similarity between the video frame to be compressed and the adjacent video frames; If the second similarity is greater than the preset similarity threshold, then the video frame to be compressed and the adjacent video frames are fused to obtain a fused video frame. The fused video frame is added as a new video frame to the video frame sequence, and the video frame to be compressed and the adjacent video frames are deleted from the video frame sequence to obtain the compressed video frame sequence.
3. The method according to claim 1, characterized in that, The step of extracting image information from each key video frame to obtain key information corresponding to the video frame sequence includes: From the video frame sequence, obtain the video frame adjacent to the key video frame, and combine the key video frame and the video frame adjacent to the key video frame to obtain the key video segment corresponding to the key video frame. Image information is extracted from the key video segments corresponding to each key video frame to obtain the key information corresponding to the video frame sequence.
4. The method according to claim 3, characterized in that, The step of extracting image information from key video segments corresponding to each key video frame to obtain key information corresponding to the video frame sequence includes: The key video segments are input into a pre-trained video summarization generation model to obtain the video summarization text output by the video summarization generation model. The video summary text is input into a pre-trained text feature extraction model to obtain a video text vector output by the text feature extraction model; and / or, the key video frame corresponding to the key video segment is input into a pre-trained multimodal feature extraction model to obtain a video image vector output by the multimodal feature extraction model. Based on the video text vector and / or video image vector corresponding to each key video frame, the key information corresponding to the video frame sequence is obtained.
5. The method according to claim 1, characterized in that, The base video frame sequence also contains multiple ordinary video frames; the generation of video response data based on each matched video frame includes: Select ordinary video frames adjacent to the matching video frame from the base video frame sequence, and combine the matching video frame and the ordinary video frames adjacent to the matching video frame to obtain the matching video segment corresponding to the matching video frame. The matching video clip and the question text are input into a pre-trained question response model to obtain the initial answer of the matching video clip relative to the question text, which is output by the question response model. Based on the initial answer corresponding to each matched video segment, video response data is generated.
6. The method according to claim 5, characterized in that, The process of generating video response data based on the initial answer corresponding to each matched video segment includes: Based on the time information of each matched video segment, the initial answers corresponding to each matched video segment are sorted in chronological order to obtain the sorted result sequence; The sorted sequence of results and the question text are input into a pre-trained answer summary model to obtain the video answer data output by the answer summary model.
7. An electronic device, characterized in that, The electronic device includes a memory and a processor, the processor being configured to execute program instructions stored in the memory to implement the steps of the video question-answering method as described in any one of claims 1-6.
8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions that can be executed by a processor to implement the steps of the video question-answering method as described in any one of claims 1-6.
Citation Information
Patent Citations
Video question and answer method and device, electronic equipment and storage medium
CN114120166A
Video content description method, medium and electronic equipment
CN119478786A