Video key information extraction method, video question and answer method, equipment and medium

Abstract

The invention discloses a video key information extraction method, a video question and answer method, equipment and a medium, and the video key information extraction method comprises the steps: calculating the motion intensity of pixels in a video frame sequence, and obtaining a motion amplitude value corresponding to each video frame in the video frame sequence; based on the motion amplitude value corresponding to each video frame, calculating a clustering weight corresponding to each video frame; based on the clustering weight corresponding to each video frame, performing weighted clustering on each video frame, and taking the video frame corresponding to the cluster center point of each clustering cluster as a centroid video frame; selecting a video frame of which the image similarity with each centroid video frame meets a preset condition to obtain a key video frame; and carrying out image information extraction on each key video frame to obtain key information corresponding to the video frame sequence. Through clustering, video frames with different importance can be better processed, more accurate and reliable centroid video frames are obtained, and more accurate key information can be extracted.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method for extracting key video information, a video question-and-answer method, a device, and a medium. Background Art

[0002] By analyzing and extracting information from video data, it is easier to implement downstream tasks such as video question answering, video generation, and video recommendation.

[0003] For example, video question answering refers to generating answers to given questions by understanding relevant video content. With the development of neural network models, video question answering is used in more and more scenarios. However, since video content is often dynamic, answering certain questions requires understanding continuous video frames. However, there may only be a small part of the content related to the question in a video. How to obtain the relevant part from the entire video and then analyze the answer is the key point.

[0004] Therefore, how to improve the accuracy of video information analysis and extraction is a technical problem that needs to be solved urgently by those skilled in the art. Summary of the Invention

[0005] In order to solve the above technical problems, the present application at least provides a video key information extraction method, a video question and answer method, a device and a medium.

[0006] In a first aspect, the present application provides a method for extracting key information from a video, the method comprising: obtaining a video frame sequence, calculating the intensity of pixel motion in the video frame sequence, and obtaining a motion amplitude value corresponding to each video frame in the video frame sequence; calculating a clustering weight corresponding to each video frame based on the motion amplitude value corresponding to each video frame; wherein the motion amplitude value and the clustering weight are positively correlated; performing weighted clustering on each video frame based on the clustering weight corresponding to each video frame to obtain multiple cluster clusters, and using the video frame corresponding to the cluster center point of each cluster cluster as the centroid video frame; calculating the image similarity between each video frame and each centroid video frame to obtain a first similarity, selecting a video frame whose first similarity meets a preset condition to obtain a key video frame; performing image information extraction on each key video frame to obtain key information corresponding to the video frame sequence.

[0007] In one embodiment, the method further includes: taking any video frame in the video frame sequence as the video frame to be compressed, obtaining video frames adjacent to the video frame to be compressed, and obtaining adjacent video frames corresponding to the video frame to be compressed; calculating the image similarity between the video frame to be compressed and the adjacent video frames to obtain a second similarity; if the second similarity is greater than a preset similarity threshold, fusing the video frame to be compressed and the adjacent video frame to obtain a fused video frame; adding the fused video frame as a new video frame to the video frame sequence, and deleting the video frame to be compressed and the adjacent video frames in the video frame sequence to obtain a compressed video frame sequence; calculating the image similarity between each video frame and each centroid video frame to obtain a first similarity, selecting a video frame whose first similarity meets a preset condition, and obtaining a key video frame, including: calculating the image similarity between each video frame in the compressed video frame sequence and each centroid video frame to obtain the first similarity; selecting a video frame whose first similarity meets the preset condition from each video frame in the compressed video frame sequence to obtain a key video frame.

[0008] In one embodiment, image information is extracted from each key video frame to obtain key information corresponding to the video frame sequence, including: obtaining video frames adjacent to the key video frame from the video frame sequence, combining the key video frame and the video frames adjacent to the key video frame to obtain key video segments corresponding to the key video frame; and image information is extracted from the key video segments corresponding to each key video frame to obtain key information corresponding to the video frame sequence.

[0009] In one embodiment, image information extraction is performed on the key video clip corresponding to each key video frame to obtain key information corresponding to the video frame sequence, including: inputting the key video clip into a pre-trained video summary generation model to obtain a video summary text output by the video summary generation model; inputting the video summary text into a pre-trained text feature extraction model to obtain a video text vector output by the text feature extraction model; and / or inputting the key video frame corresponding to the key video clip into a pre-trained multimodal feature extraction model to obtain a video image vector output by the multimodal feature extraction model; based on the video text vector and / or video image vector corresponding to each key video frame, the key information corresponding to the video frame sequence is obtained.

[0010] The second aspect of the present application provides a video question-answering method, which includes: obtaining video question data input by a user, and obtaining key information corresponding to a base video frame sequence; wherein the key information corresponding to the base video frame sequence is extracted based on any video key information extraction method; based on the video question data and the key information corresponding to the base video frame sequence, generating video answer data.

[0011] In one embodiment, the base video frame sequence contains multiple key video frames, the key information includes a video text vector corresponding to each key video frame and / or a video image vector corresponding to each key video frame, and the video question data contains question text; based on the video question data and the key information corresponding to the base video frame sequence, video answer data is generated, including: inputting the question text into a pre-trained text feature extraction model to obtain a question text vector output by the text feature extraction model; and / or, inputting the question text into a pre-trained multimodal feature extraction model to obtain a multimodal vector output by the multimodal feature extraction model; calculating the similarity between the video text vector and the question text vector of each key video frame to obtain the text vector similarity corresponding to each key video frame; and / or, calculating the similarity between the video image vector and the multimodal vector of each key video frame to obtain the multimodal vector similarity corresponding to each key video frame; based on the text vector similarity and / or multimodal vector similarity corresponding to each key video frame, respectively calculating the question matching degree corresponding to each key video frame; selecting the key video frame whose question matching degree meets the preset conditions to obtain the matching video frame; and generating video answer data based on each matching video frame.

[0012] In one embodiment, the base video frame sequence also contains multiple ordinary video frames; based on each matching video frame, video answer data is generated, including: selecting ordinary video frames adjacent to the matching video frame from the base video frame sequence, combining the matching video frame and the ordinary video frames adjacent to the matching video frame to obtain a matching video segment corresponding to the matching video frame; inputting the matching video segment and the question text into a pre-trained question response model to obtain the initial answer of the matching video segment relative to the question text output by the question response model; and generating video answer data based on the initial answer corresponding to each matching video segment.

[0013] In one embodiment, video answer data is generated based on the initial answer corresponding to each matching video clip, including: based on the time information of each matching video clip, the initial answer corresponding to each matching video clip is sorted in chronological order to obtain a result sorting sequence; the result sorting sequence and the question text are input into a pre-trained answer summary model to obtain the video answer data output by the answer summary model.

[0014] According to a third aspect of the present application, a device for extracting key information from a video is provided, which includes: a motion calculation module for acquiring a video frame sequence, calculating the intensity of pixel motion in the video frame sequence, and obtaining a motion amplitude value corresponding to each video frame in the video frame sequence; a weight calculation module for calculating a clustering weight corresponding to each video frame based on the motion amplitude value corresponding to each video frame; wherein the motion amplitude value and the clustering weight are positively correlated; a weighted clustering module for performing weighted clustering on each video frame based on the clustering weight corresponding to each video frame to obtain multiple cluster clusters, and using the video frame corresponding to the cluster center of each cluster cluster as the centroid video frame; a key frame selection module for calculating the image similarity between each video frame and each centroid video frame to obtain a first similarity, and selecting video frames whose first similarity meets a preset condition to obtain key video frames; and an information extraction module for performing image information extraction on each key video frame to obtain key information corresponding to the video frame sequence.

[0015] The fourth aspect of the present application provides a video question-and-answer device, which includes: a data acquisition module for acquiring video question data input by a user and acquiring key information corresponding to a base video frame sequence; wherein the key information corresponding to the base video frame sequence is extracted based on any video key information extraction method; and an answer generation module for generating video answer data based on the video question data and the key information corresponding to the base video frame sequence.

[0016] The fifth aspect of the present application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-mentioned video key information extraction method or video question-answering method.

[0017] In a sixth aspect, the present application provides a computer-readable storage medium having program instructions stored thereon. When the program instructions are executed by a processor, the above-mentioned video key information extraction method or video question-and-answer method is implemented.

[0018] The above scheme obtains a video frame sequence, calculates the intensity of pixel motion in the video frame sequence, obtains the motion amplitude value corresponding to each video frame in the video frame sequence, calculates the clustering weight corresponding to each video frame based on the motion amplitude value corresponding to each video frame, and performs weighted clustering on each video frame based on the clustering weight corresponding to each video frame to obtain multiple cluster clusters. The video frame corresponding to the cluster center point of each cluster cluster is used as the centroid video frame, which can better process video frames with different importance, thereby obtaining more accurate and reliable centroid video frames. Then, according to the image similarity between each video frame in the video frame sequence and each centroid video frame, the key video frame is selected, and image information is extracted for each key video frame, so that more accurate key information can be extracted to ensure that the key information is not lost.

[0019] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and, together with the specification, are used to illustrate the technical solutions of the present application.

[0021] Figure 1 is a schematic diagram of a solution implementation environment shown in an exemplary embodiment of the present application; Figure 2 is a flowchart of a method for extracting key video information shown in an exemplary embodiment of the present application; Figure 3 is a flowchart of a method for extracting key video information shown in another exemplary embodiment of the present application; Figure 4 is a flowchart of a video question-answering method shown in an exemplary embodiment of the present application; Figure 5 is a flowchart of a video question-answering method shown in another exemplary embodiment of the present application; Figure 6 is a block diagram of a video key information extraction device shown in an exemplary embodiment of the present application; Figure 7 is a block diagram of a video question-answering device shown in an exemplary embodiment of the present application; Figure 8 is a schematic structural diagram of an electronic device shown in an exemplary embodiment of the present application; Figure 9 It is a schematic diagram of the structure of a computer-readable storage medium shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0022] The following describes the embodiments of the present application in detail with reference to the accompanying drawings.

[0023] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0024] The term "and / or" in this article is merely information describing the association of associated objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0025] The following describes the video key information extraction method and video question-answering method provided in the embodiments of the present application.

[0026] Please refer to Figure 1 , Figure 1 FIG1 is a schematic diagram of an exemplary embodiment of the present application showing a solution implementation environment, wherein the solution implementation environment may include a terminal 110 and a server 120, and the terminal 110 and the server 120 are in communication connection with each other.

[0027] The number of the terminal 110 may be one or more. The terminal 110 may be a camera, a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart watch, etc., but is not limited thereto.

[0028] Server 120 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0029] In one example, the server 120 may perform key information extraction processing on the video frame sequence obtained from the terminal 110 to obtain key information corresponding to the video frame sequence, or the server 120 may perform question answering processing on the video question data obtained from the terminal 110 to obtain video answer data. Of course, the server 120 may store the key information or video answer data corresponding to the video frame sequence locally, transmit it back to the terminal 110, or transmit it to other terminals.

[0030] In one example, terminal 110 may be installed with a client running a target application, such as an application that provides a key information extraction function or a question answering function. The target application is used to extract key information from a video frame sequence to obtain key information corresponding to the video frame sequence, or to answer video question data to obtain video answer data. Server 120 may be a backend server for the target application, configured to provide backend services for the client of the target application.

[0031] In the video key information extraction method or video question-and-answer method provided in the embodiments of the present application, the execution entity of each step can be the terminal 110, such as the client of the target application installed and running in the terminal 110, or the server 120, or the terminal 110 and the server 120 interact and cooperate to execute, that is, part of the steps of the method are executed by the terminal 110 and the other part of the steps are executed by the server 120.

[0032] See also Figure 2 , Figure 2 This is a flowchart of a method for extracting key information from a video according to an exemplary embodiment of the present application. The method for extracting key information from a video can be applied to Figure 1 It should be understood that the method can also be applied to other exemplary implementation environments and be specifically executed by devices in other implementation environments, and this embodiment does not limit the implementation environment to which the method is applicable.

[0033] like Figure 2 As shown, the video key information extraction method includes at least steps S210 to S250, which are described in detail as follows: Step S210: obtaining a video frame sequence, calculating the intensity of pixel motion in the video frame sequence, and obtaining a motion amplitude value corresponding to each video frame in the video frame sequence.

[0034] A video frame sequence consists of multiple video frames sorted in time order.

[0035] The terminal may capture images of a preset scene to generate a video frame sequence, and the server may obtain the video frame sequence that needs to be analyzed from the terminal; or the video frame sequence may be stored in a preset database, and the server may obtain the video frame sequence that needs to be analyzed from the preset database. This application does not limit the method of obtaining the video frame sequence.

[0036] The intensity of pixel motion in the video frame sequence is calculated to obtain the motion amplitude value corresponding to each video frame in the video frame sequence.

[0037] For example, the intensity of pixel motion in a video frame sequence can be calculated by optical flow method, and / or inter-frame difference method, and / or constructing a displacement time image of an object, etc., to obtain the motion amplitude value corresponding to each video frame in the video frame sequence.

[0038] For example, optical flow represents the displacement of pixels in the same scene captured at different times. By calculating the optical flow, the motion trajectory and speed of the object in the image can be obtained. Specifically, the motion vector of the video frame pixels in the video frame sequence is calculated, and the optical flow amplitude representing the motion speed is extracted from the motion vector. The optical flow amplitude is used to represent the intensity of the movement of the pixels in the two adjacent video frames, and the optical flow amplitude is used as the motion amplitude value corresponding to the video frame. Among them, the larger the motion amplitude value, the more intense the motion events, behaviors, etc. that occurred in the two video frames, that is, the more likely it is to contain key information; conversely, the smaller the motion amplitude value, the more stable and static the content of the two video frames is, and the smaller the probability of containing key information.

[0039] For example, the inter-frame difference method calculates the difference in pixel values between adjacent video frames. The greater the difference in pixel values between adjacent video frames, the greater the corresponding motion amplitude value of the video frame; conversely, the smaller the difference in pixel values between adjacent video frames, the smaller the corresponding motion amplitude value of the video frame. The difference in pixel values between adjacent video frames represents the intensity of pixel motion, and the motion amplitude value corresponding to each video frame in the video frame sequence is obtained.

[0040] For example, in an object's displacement-time image, the slope indicates the speed of the object's motion within the video frame. A larger absolute slope indicates faster object movement, resulting in a larger motion amplitude value for the corresponding video frame. Conversely, slower object movement results in a smaller motion amplitude value for the corresponding video frame. By analyzing the displacement-time image, the intensity of pixel motion can be determined, yielding the corresponding motion amplitude value for each frame in the video frame sequence.

[0041] Step S220: Calculating the clustering weight corresponding to each video frame based on the motion amplitude value corresponding to each video frame.

[0042] The motion amplitude value and the clustering weight are positively correlated, that is, the larger the motion amplitude value of a video frame, the larger the clustering weight of the video frame, and the smaller the motion amplitude value of a video frame, the smaller the clustering weight of the video frame.

[0043] For example, a weight calculation formula may be pre-established to describe the mapping relationship between motion amplitude values and clustering weights. By inputting the motion amplitude values into the weight calculation formula, the clustering weights corresponding to the video frames are calculated. The weight calculation formula can be flexibly set according to specific application scenarios, as long as the motion amplitude values and clustering weights in the weight calculation formula are positively correlated. This application does not limit the implementation of the weight calculation formula.

[0044] For example, the motion amplitude value may be compared with a preset threshold value, and different threshold values may correspond to different clustering weights. The clustering weight corresponding to the video frame may be determined based on the size comparison relationship between the motion amplitude value and the threshold value.

[0045] Of course, other weight allocation algorithms can also be used to calculate the clustering weights of video frames under corresponding motion amplitude values, such as comparing the motion amplitude values between each video frame to set the clustering weight corresponding to each video frame. This application does not limit the specific calculation method of the clustering weight.

[0046] Step S230: Based on the clustering weight corresponding to each video frame, weighted clustering is performed on each video frame to obtain multiple clusters, and the video frame corresponding to the cluster center point of each cluster is used as the centroid video frame.

[0047] The weighted clustering algorithm takes into account the importance of each video frame when calculating the cluster center points, and characterizes the importance of the video frame according to the clustering weight of each video frame, so that the video frame with a larger weight has a greater influence when calculating the cluster center points of the cluster.

[0048] The clustering algorithm may be a K-means clustering algorithm (K-means), a hierarchical clustering algorithm, a spectral clustering algorithm, etc. This application does not limit the type of clustering algorithm used.

[0049] To illustrate this, let's use K-means as an example: K video frames are selected as initial cluster centers. Based on the distance from each video frame to each cluster center, the video frames are assigned to the nearest cluster. Based on the clustering weights and coordinates of the video frames in each cluster, new cluster centers are calculated. Video frames with larger clustering weights contribute more to the calculation of cluster centers. Based on the distance from each video frame to the newly calculated cluster center, the clustering weight of the video frame is updated. The above steps are repeated until the stopping condition is met, for example, the preset maximum number of iterations is reached or the cluster centers converge. The resulting K cluster centers and cluster labels for each video frame are the output of the weighted K-means algorithm.

[0050] The video frame corresponding to the cluster center of each cluster is used as the centroid video frame.

[0051] For example, by performing weighted K-means clustering, a total of K centroid video frames are finally obtained. The larger the value of K, the smaller the difference between the centroid video frames, and the smaller the information loss. However, the subsequent redundant information and the resulting computational overhead will also increase. Therefore, the value of K can be set according to the computing resources available in the current application scenario. The more computing resources that can be used, the larger the value of K. Of course, the value of K can also be set based on experience, and this application does not limit this.

[0052] By assigning clustering weights to video frames according to their corresponding motion amplitude values and performing weighted clustering on video frame sequences based on the clustering weights, video frames with different importance can be better processed, thereby obtaining more accurate and reliable clustering results.

[0053] Step S240: Calculate the image similarity between each video frame and each centroid video frame to obtain a first similarity, select the video frame whose first similarity meets a preset condition, and obtain the key video frame.

[0054] After obtaining the centroid video frame, image similarity calculation is performed on each video frame in the video frame sequence and each centroid video frame to obtain a first similarity.

[0055] For example, image feature vectors may be extracted from the video frame and the centroid video frame, and the image similarity between the video frame and the centroid video frame may be calculated by calculating the vector distance between the extracted image feature vectors. Alternatively, the pixel value difference between the video frame and the centroid video frame may be calculated, and the image similarity between the video frame and the centroid video frame may be obtained based on the magnitude of the pixel value difference. The specific method for calculating image similarity can be flexibly selected based on actual application circumstances and is not limited in this application.

[0056] The video frames whose first similarity satisfies a preset condition are selected to obtain key video frames.

[0057] For example, it can be determined whether the first similarity between the video frame and any centroid video frame is greater than a preset similarity threshold. If the first similarity between the video frame and any centroid video frame is greater than the preset similarity threshold, the video frame is retained as a key video frame.

[0058] For another example, the first similarity between the video frame and each centroid video frame may be obtained, and the number of all first similarities corresponding to the video frame that are greater than a preset similarity threshold may be counted. If the number is greater than the preset number threshold, the video frame may be retained as a key video frame.

[0059] For another example, the first similarity between the video frame and each centroid video frame may be obtained, the average of all first similarities corresponding to the video frame may be calculated, and it may be determined whether the average is greater than a preset similarity threshold. If the average is greater than the preset similarity threshold, the video frame may be retained as a key video frame.

[0060] For example, the first similarity between the video frame and each centroid video frame can be obtained, the maximum value of all first similarities corresponding to the video frame can be counted, and it can be determined whether the maximum value is greater than a preset similarity threshold. If the maximum value is greater than the preset similarity threshold, the video frame can be retained as a key video frame.

[0061] Among them, the preset similarity threshold can be pre-set based on experience, or it can be flexibly calculated according to the actual application situation. For example, the preset similarity threshold can be set according to the computing resources available in the current application scenario. The more computing resources that can be used, the smaller the preset similarity threshold, and the more key video frames are ultimately retained. This application does not limit the setting method of the preset similarity threshold.

[0062] Step S250: extracting image information from each key video frame to obtain key information corresponding to the video frame sequence.

[0063] The image information of the key video frames is extracted, and the extraction result corresponding to each key video frame is used as the key information corresponding to the video frame sequence.

[0064] Among them, image information includes but is not limited to scene information, and / or target object information, and / or behavioral event information, and / or time information, and / or weather information, etc. The specific image information can be flexibly selected according to the actual application scenario. This application does not limit the type of image information.

[0065] The key information corresponding to the extracted video frame sequence can be used to implement downstream tasks such as video question answering, video generation, and video recommendation.

[0066] The present application assigns clustering weights to video frames according to the motion amplitude values corresponding to the video frames, and performs weighted clustering on the video frame sequence based on the clustering weights, so as to better process video frames with different importance, thereby obtaining more accurate and reliable centroid video frames. Then, based on the image similarity between each video frame in the video frame sequence and each centroid video frame, key video frames are selected, and image information is extracted for each key video frame, so as to extract more accurate key information and ensure that key information is not lost.

[0067] Next, some embodiments of the present application are described in detail.

[0068] In some embodiments, in step S240, the image similarity between each video frame and each centroid video frame is calculated to obtain a first similarity, and the video frame whose first similarity meets the preset conditions is selected to obtain a key video frame, including steps S241 to S243.

[0069] Step S241: compress the video frame sequence to obtain a compressed video frame sequence.

[0070] Step S242: Calculate the image similarity between each video frame in the compressed video frame sequence and each centroid video frame to obtain a first similarity.

[0071] Step S243: Selecting video frames whose first similarity satisfies a preset condition from the respective video frames in the compressed video frame sequence to obtain key video frames.

[0072] By compressing the video frame sequence, redundant information of the video frame sequence is removed.

[0073] Exemplarily, the step of compressing the video frame sequence includes the following steps S2411 to S2415.

[0074] Step S2411: taking any video frame in the video frame sequence as the video frame to be compressed, obtaining video frames adjacent to the video frame to be compressed, and obtaining adjacent video frames corresponding to the video frame to be compressed.

[0075] The first N video frames and / or the last N video frames of the video frame to be compressed in the video frame sequence may be used as video frames adjacent to the video frame to be compressed. N is a positive integer greater than or equal to 1. The value of N may be preset based on experience or based on available computing resources in the current application scenario. For example, the more computing resources available, the larger the value of N may be. This application does not limit this.

[0076] Step S2412: Calculate the image similarity between the video frame to be compressed and the adjacent video frame to obtain a second similarity.

[0077] The specific calculation method of the image similarity can be found in the above step S240 and will not be described in detail here.

[0078] Step S2413: determine whether the second similarity is greater than a preset similarity threshold; if so, execute steps S2414 and S2415; if not, continue to execute step S2411 until all video frames in the video frame sequence are traversed.

[0079] Step S2414: Fusing the video frame to be compressed and the adjacent video frames to obtain a fused video frame.

[0080] Image fusion methods include pixel-level fusion and feature-level fusion. Pixel-level fusion obtains a fused image by performing weighted averaging or other operations on image pixels. Methods such as pixel-level fusion and wavelet transform are examples. Feature-level fusion extracts features from an image and then fuses them. Methods such as principal component analysis (PCA) and scale-invariant feature transform (SIFT) are examples.

[0081] The video frame to be compressed and the adjacent video frames are subjected to pixel-level fusion and / or feature-level fusion to obtain a fused video frame.

[0082] Step S2415: adding the fused video frame as a new video frame to the video frame sequence, and deleting the video frame to be compressed and adjacent video frames in the video frame sequence to obtain a compressed video frame sequence.

[0083] In some implementations, if the second similarity between the video frame to be compressed and any adjacent video frame is greater than a preset similarity threshold, one of the video frame to be compressed and the adjacent video frame may be deleted.

[0084] By compressing the video frame sequence in the above embodiment, the subsequent calculation cost and time consumption are greatly reduced while ensuring that video information is not lost.

[0085] In some implementations, step S250 extracts image information from each key video frame to obtain key information corresponding to the video frame sequence, including steps S251 to S252.

[0086] Step S251: obtaining video frames adjacent to the key video frame from the video frame sequence, combining the key video frame and the video frames adjacent to the key video frame to obtain a key video segment corresponding to the key video frame.

[0087] The method for obtaining adjacent video frames can be found in step S2411 and will not be described in detail here.

[0088] The key video frame and the video frames adjacent to the key video frame are combined in time sequence to obtain a key video segment corresponding to the key video frame.

[0089] Step S252: extracting image information from the key video segments corresponding to each key video frame to obtain key information corresponding to the video frame sequence.

[0090] The key video clips retain more image information. By extracting image information from each key video clip, the key information corresponding to the video frame sequence is obtained to avoid missing key information.

[0091] Exemplarily, the key video clips are input into a pre-trained video summary generation model to obtain the video summary text output by the video summary generation model; the video summary text is input into a pre-trained text feature extraction model to obtain the video text vector output by the text feature extraction model; and / or, the key video frames corresponding to the key video clips are input into a pre-trained multimodal feature extraction model to obtain the video image vector output by the multimodal feature extraction model; based on the video text vector and / or video image vector corresponding to each key video frame, the key information corresponding to the video frame sequence is obtained.

[0092] Among them, the video summary generation model can be constructed based on visual language models (VLM), and / or convolutional neural networks (CNN), and / or recurrent neural networks (RNN). This application does not limit the implementation method of the video summary generation model.

[0093] Taking the VLM model as an example, we use VLM to generate a summary description for each key video clip. Specifically, the key video clips are composed of the following input prompts and input into the VLM for summary description generation. The input prompt template is as follows: {{text: The images below represent the consecutive video frames corresponding to the key video clips. Analyze the content of the key video clips and output a video summary. Focus on the following aspects of the key video clips: time information, scene / location / environment information, the target objects appearing in the video, and the behavioral events occurring in the video. Output the results according to the following template: {time: Scene / Location / Environment: Target audience information: Behavioral events: }} {image_url: video frame 1 of the key video clip} ... {image_url: video frame M of the key video clip}} Wherein, M is the total number of video frames in the key video segment. When generating the summary, the information that needs to be paid attention to in the key video segment can be flexibly set according to the actual situation.

[0094] Then, the video summary text is input into a pre-trained text feature extraction model to obtain a video text vector output by the text feature extraction model; and / or, the key video frames corresponding to the key video clips are input into a pre-trained multimodal feature extraction model to obtain a video image vector output by the multimodal feature extraction model, and the video text vector and / or video image vector corresponding to each key video frame are combined to obtain the key information corresponding to the video frame sequence.

[0095] The text feature extraction model is used to encode input text, while the multimodal feature extraction model is used to encode input images and / or text, enabling the integrated processing of data from multiple modalities. The text feature extraction model and the multimodal feature extraction model can be constructed based on bidirectional encoder representations from transformers (BERT), CNN, RNN, and other models. This application does not limit their implementation.

[0096] By integrating feature extraction from multiple dimensions of text and images, the representation ability of the extracted key information is improved, facilitating the execution of downstream tasks.

[0097] In some embodiments, see Figure 3 , Figure 3 is a flowchart of a method for extracting key video information shown in another exemplary embodiment of the present application. Figure 3 As shown, it includes steps S301 to S307.

[0098] Step S301: Acquire a video frame sequence.

[0099] Step S302: Calculate the optical flow amplitude corresponding to each video frame in the video frame sequence, and set the clustering weight corresponding to each video frame according to the optical flow amplitude.

[0100] Step S303: performing weighted clustering on each video frame according to the clustering weight corresponding to each video frame to obtain a centroid video frame.

[0101] Step S304: compress the video frame sequence to obtain a compressed video frame sequence.

[0102] Among them, there is no precedence execution relationship between step S304 and step S302 and step S303. They can be executed in parallel, or step S302 and step S303 can be executed first, and then step S304; or step S304 can be executed first, and then step S302 and step S303.

[0103] Step S305: Selecting, from each video frame in the compressed video frame sequence, a video frame whose image similarity with the centroid video frame satisfies a preset condition to obtain a key video frame.

[0104] Step S306: Acquire key video segments corresponding to key video frames, and generate video summary text corresponding to the key video segments.

[0105] Step S307: Encode the video summary text to obtain a video text vector, encode the key video frames corresponding to the key video clips to obtain a video image vector, and obtain key information corresponding to the key video frames.

[0106] The specific implementation of each of the above steps can be found in the above embodiments and will not be described in detail here.

[0107] See also Figure 4 , Figure 4 This is a flowchart of a video question-answering method shown in an exemplary embodiment of the present application. The video question-answering method can be applied to Figure 1 It should be understood that the method can also be applied to other exemplary implementation environments and be specifically executed by devices in other implementation environments, and this embodiment does not limit the implementation environment to which the method is applicable.

[0108] like Figure 4 As shown, the video question-answering method includes at least steps S410 to S420, which are described in detail as follows: Step S410: Obtain the video question data input by the user, and obtain key information corresponding to the underlying video frame sequence.

[0109] The key information corresponding to the base video frame sequence is extracted based on any of the above-mentioned video key information extraction methods.

[0110] Video question data includes but is not limited to text, audio, images, etc. This application does not limit the data type of video question data.

[0111] The video question data may be used to instruct a video query, video content analysis, etc. For example, the video question data may be "query for videos of rainy weather", which is used to instruct the query of video clips related to rainy weather information; the video question data may also be "query for videos of rainy weather".

[0112] Step S420: Generate video answer data based on the video question data and key information corresponding to the underlying video frame sequence.

[0113] According to the key information corresponding to the base library video frame sequence, the video question data is matched with each video frame in the base library video frame sequence, the video frames in the base library video frame sequence that match the video question data are determined, and the video answer data is generated based on the matched video frames.

[0114] For example, if it is a video query question, the video query result can be directly generated based on the matching video frame, and the video query result can be used as the video answer data. If it is a video content analysis business, after obtaining the matching video frame, the image content of the matching video frame is analyzed according to the video question data, such as the time, and / or scene, and / or location, and / or environment, and / or target object information, and / or behavioral events of the matching video frame. The analysis result related to the video question data is obtained, and the analysis result is used as the video answer data. It should be noted that the specific method of the video answer data can be flexibly selected according to the specific business type, the question intention of the video question and answer data, etc., and this application does not limit this.

[0115] Exemplarily, the base video frame sequence contains multiple key video frames, the key information includes the video text vector corresponding to each key video frame, and / or the video image vector corresponding to each key video frame, and the video question data contains the question text; based on the video question data and the key information corresponding to the base video frame sequence, video answer data is generated, including: inputting the question text into a text feature extraction model to obtain the question text vector output by the text feature extraction model; and / or, inputting the question text into a multimodal feature extraction model to obtain the multimodal vector output by the multimodal feature extraction model; calculating the similarity between the video text vector and the question text vector of each key video frame to obtain the text vector similarity corresponding to each key video frame; and / or, calculating the similarity between the video image vector and the multimodal vector of each key video frame to obtain the multimodal vector similarity corresponding to each key video frame; based on the text vector similarity and / or multimodal vector similarity corresponding to each key video frame, respectively calculate the question matching degree corresponding to each key video frame; select the key video frame whose question matching degree meets the preset conditions to obtain the matching video frame; and generate video answer data based on each matching video frame.

[0116] The method for obtaining the key video frame may refer to the above steps S210 to S240 and will not be described in detail here.

[0117] The similarity between two vectors can be measured by calculating the Euclidean distance, inner product, and / or cosine similarity between the two vectors.

[0118] That is, based on the Euclidean distance, and / or inner product, and / or cosine similarity between the video text vector and the question text vector, the text vector similarity corresponding to the key video frame is obtained; based on the Euclidean distance, and / or inner product, and / or cosine similarity between the video image vector and the multimodal vector, the multimodal vector similarity corresponding to the key video frame is obtained.

[0119] According to the text vector similarity and / or multimodal vector similarity corresponding to each key video frame, the question matching degree corresponding to each key video frame is calculated respectively.

[0120] If there is one of the text vector similarity or multimodal vector similarity, it is directly used as the question matching degree corresponding to the key video frame.

[0121] If both text vector similarity and multimodal vector similarity exist, then either one of the text vector similarity and the multimodal vector similarity can be selected as the question matching degree corresponding to the key video frame, such as selecting the maximum or minimum value of the text vector similarity and the multimodal vector similarity to obtain the question matching degree corresponding to the key video frame; the text vector similarity and / or the multimodal vector similarity can also be weighted and summed to obtain the question matching degree corresponding to the key video frame.

[0122] Exemplarily, the difference between the text vector similarity and the multimodal vector similarity is calculated to obtain a similarity difference. If the similarity difference is greater than a preset difference threshold, the minimum value between the text vector similarity and the multimodal vector similarity is selected to obtain the problem matching degree corresponding to the key video frame. If the similarity difference is not greater than the preset difference threshold, the maximum value between the text vector similarity and the multimodal vector similarity is selected to obtain the problem matching degree corresponding to the key video frame. Alternatively, weighted weights corresponding to the text vector similarity and the multimodal vector similarity are set according to the similarity difference, and the text vector similarity and the multimodal vector similarity are weighted and summed according to the weighted weights to obtain the problem matching degree corresponding to the key video frame. For example, the larger the similarity difference, the larger the weighted weight corresponding to the minimum value between the text vector similarity and the multimodal vector similarity, and the smaller the weighted weight corresponding to the maximum value between the two; the smaller the similarity difference, the larger the weighted weight corresponding to the maximum value between the text vector similarity and the multimodal vector similarity, and the smaller the weighted weight corresponding to the minimum value between the two.

[0123] The above embodiment determines the credibility of the similarity calculation result between the same key video frame and video question data through the similarity difference. The larger the similarity difference, the lower the credibility of the similarity calculation result. At this time, the minimum value of the text vector similarity and the multimodal vector similarity can be selected, or more weighted weights can be assigned to the minimum value to obtain the problem matching degree corresponding to the key video frame; conversely, the smaller the similarity difference, the higher the credibility of the similarity calculation result. At this time, the maximum value of the text vector similarity and the multimodal vector similarity can be selected, or more weighted weights can be assigned to the maximum value to obtain the problem matching degree corresponding to the key video frame, so as to improve the accuracy of the problem matching degree calculation and retain more video frame image information.

[0124] Then, the key video frames whose matching degree meets the preset conditions are selected to obtain the matching video frames.

[0125] For example, key video frames whose problem matching degree is greater than a preset matching degree threshold can be selected to obtain matching video frames; each key video frame can also be sorted in sequence according to the problem matching degree, and a preset number of key video frames with the largest problem matching degree can be selected to obtain matching video frames, such as sorting each key video frame in descending order according to the problem matching degree, and selecting the first W key video frames as matching video frames.

[0126] Based on each matching video frame, video answer data is generated.

[0127] In some embodiments, the base video frame sequence also contains multiple ordinary video frames; based on each matching video frame, video answer data is generated, including: selecting ordinary video frames adjacent to the matching video frame from the base video frame sequence, combining the matching video frame and the ordinary video frames adjacent to the matching video frame to obtain a matching video segment corresponding to the matching video frame; inputting the matching video segment and the question text into a pre-trained question response model to obtain the initial answer of the matching video segment relative to the question text output by the question response model; and generating video answer data based on the initial answer corresponding to each matching video segment.

[0128] The method for obtaining adjacent ordinary video frames can be referred to step S2411, which is not described in detail here. Among them, the base video frame sequence in this application can be a compressed video frame sequence, and the compression step can be referred to the above steps S2411 to S2415, which are not described in detail here.

[0129] The matching video frame and the common video frames adjacent to the matching video frame are combined in time sequence to obtain a matching video segment corresponding to the matching video frame.

[0130] For example, in a video query service, matching video clips and question text are input into a question response model, which analyzes the correlation between the matching video clips and the question text to obtain an initial answer. The question response model can be implemented based on a neural network model architecture such as a VLM, and / or RNN, and / or CNN, and this application does not limit this.

[0131] For example, the question-response model (VLM) is used to construct a prompt, combining each matching video frame with the user-entered question text. The prompt is then fed into the VLM, which then performs inference and ultimately generates multiple initial answers. The following is an example of the prompt template format, where "XXXX" is replaced with the user-entered question text: {{text: The multiple images below are consecutive video frames corresponding to a matching video clip. Answer this question based on the image content of the matching video clip: XXXX. If the image content of the matching video clip is irrelevant to the question, answer: The current matching video clip is irrelevant to the question.} {image_url: video frame 1 of the matching video clip} ... {image_url: video frame matching the video clip}} Where r is the total number of video frames in the input matching video clip, and the initial answer output by the question response model is used to characterize whether the matching video clip and the question text are relevant.

[0132] Of course, in addition to the above-mentioned embodiment in which the initial answer is used to characterize whether the matching video clip and the question text are related, the question response model can also input the specific degree of correlation between the matching video clip and the question text, and use the degree of correlation as the initial answer. Alternatively, in other businesses, the initial answer can also be the result of image content analysis, such as key information obtained from the analysis, etc. This application does not limit this.

[0133] Then, video answer data is generated based on the initial answer corresponding to each matching video clip.

[0134] For example, based on the initial answer corresponding to each matching video clip, video answer data is generated, including: based on the time information of each matching video clip, the initial answer corresponding to each matching video clip is sorted in chronological order to obtain a result sorting sequence; the result sorting sequence and the question text are input into a pre-trained answer summary model to obtain the video answer data output by the answer summary model.

[0135] Specifically, according to the acquisition time of the key video frame in the matching video clip, the initial answer corresponding to each matching video clip is sorted in order. The earlier the acquisition time of the key video frame of the matching video clip, the higher the corresponding initial answer is sorted, and the result sorting sequence is obtained.

[0136] The resulting sorted sequence and question text are then input into the answer summarization model, which summarizes the input information and outputs video answer data. The video answer data is implemented based on a neural network model architecture such as a Large Language Model (LLM), and / or RNN, and / or CNN, but this application does not limit this.

[0137] For example, the question-response model (LLM) is used as an example. The result sorting sequence and the user-entered question text are combined to generate an input prompt, which is then fed into the LLM. The LLM then summarizes the final answer and generates the video answer data. For example, the result sorting sequence is 1.AA;2.BB;…;r.RR as the initial answer. The prompt template format is as follows, where "XXXX" is replaced with the user-entered question text: You are an expert at summarizing video Q&A answers. You are given a question about a video and multiple initial answers to the question at different time points. You summarize these initial answers, remove irrelevant answer content, combine and concatenate relevant answers, and output the video answer data. The video Q&A question and multiple answers are as follows: Question for the video Q&A: XXXX Multiple initial answers in time sequence: 1.AA; 2.BB;…; r.RR} In some embodiments, see Figure 5 , Figure 5 is a flowchart of a video question-answering method shown in another exemplary embodiment of the present application. Figure 5 As shown, it includes steps S501 to S506.

[0138] Step S501: Obtain video question data input by the user.

[0139] Step S502: Obtain matching video frames corresponding to the video question data from the underlying video frame sequence.

[0140] Step S503: Obtain the matching video segment corresponding to the matching video frame.

[0141] Step S504: deriving an initial answer to the question text for each matching video clip using the question response model inference.

[0142] Step S505: Sort each initial answer according to the time information to obtain a result sorting sequence.

[0143] Step S506: Use the answer summary model to summarize the answers in the result sorting sequence to obtain video answer data.

[0144] The video question-answering method provided in this application calculates the feature adjacent similarity between the video question data and the underlying video frame sequence, retrieves the matching video frames, and obtains the matching video clips corresponding to the matching video frames. It then infers each matching video clip to generate multiple initial answers to the video questions at different time points, and then finally summarizes and aggregates the multiple initial answers to improve the accuracy of the final video answer.

[0145] Figure 6 FIG. 1 is a block diagram of a video key information extraction device shown in an exemplary embodiment of the present application. Figure 6 As shown, the exemplary video key information extraction device 600 includes: The motion calculation module 610 is used to obtain a video frame sequence, calculate the intensity of pixel motion in the video frame sequence, and obtain a motion amplitude value corresponding to each video frame in the video frame sequence; The weight calculation module 620 is used to calculate the clustering weight corresponding to each video frame based on the motion amplitude value corresponding to each video frame; wherein the motion amplitude value and the clustering weight are positively correlated; A weighted clustering module 630 is configured to perform weighted clustering on each video frame based on the clustering weight corresponding to each video frame to obtain a plurality of clusters, and use the video frame corresponding to the cluster center of each cluster as the centroid video frame; A key frame selection module 640 is configured to calculate the image similarity between each video frame and each centroid video frame to obtain a first similarity, and select a video frame whose first similarity meets a preset condition to obtain a key video frame; The information extraction module 650 is used to extract image information from each key video frame to obtain key information corresponding to the video frame sequence.

[0146] Figure 7 FIG. 1 is a block diagram of a video question-answering device according to an exemplary embodiment of the present application. Figure 7 As shown, the exemplary video question-answering device 700 includes: The data acquisition module 710 is used to obtain the video question data input by the user and obtain the key information corresponding to the underlying video frame sequence; wherein the key information corresponding to the underlying video frame sequence is extracted based on any video key information extraction method; The answer generation module 720 is used to generate video answer data based on the video question data and key information corresponding to the underlying video frame sequence.

[0147] It should be noted that the video key information extraction device and video question-and-answer device provided in the above embodiments are of the same concept as the video key information extraction method and video question-and-answer method provided in the above embodiments, wherein the specific manner in which each module and unit performs operations has been described in detail in the method embodiments and will not be repeated here. In actual applications, the video key information extraction device and video question-and-answer device provided in the above embodiments can allocate the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0148] See also Figure 8 , Figure 8 8 is a schematic diagram of the structure of an embodiment of an electronic device of the present application. Electronic device 800 includes memory 801 and processor 802. Processor 802 is used to execute program instructions stored in memory 801 to implement the steps of any of the above-mentioned video key information extraction methods or video question-answering method embodiments. In a specific implementation scenario, electronic device 800 may include, but is not limited to: a microcomputer, a server, and may also include mobile devices such as laptops and tablet computers, which are not limited here.

[0149] Specifically, the processor 802 is used to control itself and the memory 801 to implement the steps in any of the above-mentioned video key information extraction methods or video question-answering method embodiments. The processor 802 can also be called a central processing unit (CPU). The processor 802 may be an integrated circuit chip with signal processing capabilities. The processor 802 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 802 can be implemented by an integrated circuit chip.

[0150] See also Figure 9 , Figure 9The computer-readable storage medium 900 stores program instructions 910 that can be executed by a processor, and the program instructions 910 are used to implement the steps of any of the above-mentioned video key information extraction methods or video question-answering methods.

[0151] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0152] The above description of the various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced with each other and will not be repeated herein for the sake of brevity.

[0153] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation methods described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. For example, units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0154] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in either hardware or software functional units. If the integrated units are implemented as software functional units and sold or used as standalone products, they may be stored on a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the portion that contributes to the prior art, or all or part of the technical solution, may be embodied in the form of a software product. This computer software product, stored on a storage medium, includes instructions for causing a computer device (such as a personal computer, server, or network device) or a processor to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned storage media include various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

Claims

1. A method for extracting key information from a video, characterized in that: The method comprises: Acquire a video frame sequence, calculate the intensity of pixel motion in the video frame sequence, and obtain a motion amplitude value corresponding to each video frame in the video frame sequence; Calculating clustering weights corresponding to each video frame based on the motion amplitude value corresponding to each video frame; wherein the motion amplitude value and the clustering weight are positively correlated; Based on the clustering weights corresponding to each video frame, weighted clustering is performed on each video frame to obtain a plurality of clusters, and the video frame corresponding to the cluster center of each cluster is used as the centroid video frame; Calculating the image similarity between each video frame and each centroid video frame to obtain a first similarity, and selecting a video frame whose first similarity meets a preset condition to obtain a key video frame; Image information is extracted from each key video frame to obtain key information corresponding to the video frame sequence.

2. The method according to claim 1, characterized in that The method further comprises: Taking any video frame in the video frame sequence as a video frame to be compressed, obtaining video frames adjacent to the video frame to be compressed, and obtaining adjacent video frames corresponding to the video frame to be compressed; Calculating the image similarity between the to-be-compressed video frame and the adjacent video frame to obtain a second similarity; If the second similarity is greater than a preset similarity threshold, fusing the to-be-compressed video frame and the adjacent video frame to obtain a fused video frame; adding the fused video frame as a new video frame to the video frame sequence, and deleting the video frame to be compressed and the adjacent video frame in the video frame sequence to obtain a compressed video frame sequence; The calculating the image similarity between each video frame and each centroid video frame to obtain a first similarity, selecting a video frame whose first similarity meets a preset condition, and obtaining a key video frame includes: Calculating image similarities between each video frame in the compressed video frame sequence and each centroid video frame to obtain a first similarity; From each video frame in the compressed video frame sequence, a video frame whose first similarity meets a preset condition is selected to obtain a key video frame.

3. The method according to claim 1, characterized in that The step of extracting image information from each key video frame to obtain key information corresponding to the video frame sequence includes: Acquire a video frame adjacent to the key video frame from the video frame sequence, and combine the key video frame and the video frames adjacent to the key video frame to obtain a key video segment corresponding to the key video frame; Image information is extracted from the key video segment corresponding to each key video frame to obtain key information corresponding to the video frame sequence.

4. The method according to claim 3, characterized in that The step of extracting image information from a key video segment corresponding to each key video frame to obtain key information corresponding to the video frame sequence includes: Inputting the key video clip into a pre-trained video summary generation model to obtain a video summary text output by the video summary generation model; Inputting the video summary text into a pre-trained text feature extraction model to obtain a video text vector output by the text feature extraction model; and / or inputting the key video frames corresponding to the key video clips into a pre-trained multimodal feature extraction model to obtain a video image vector output by the multimodal feature extraction model; Based on the video text vector and / or video image vector corresponding to each key video frame, key information corresponding to the video frame sequence is obtained.

5. A video question-answering method, characterized in that: The method comprises: Obtaining video question data input by the user, and obtaining key information corresponding to the underlying video frame sequence; wherein the key information corresponding to the underlying video frame sequence is extracted based on the video key information extraction method according to any one of claims 1 to 4; Video answer data is generated based on the video question data and key information corresponding to the underlying video frame sequence.

6. The method according to claim 5, characterized in that The base video frame sequence contains a plurality of key video frames, the key information includes a video text vector corresponding to each key video frame and / or a video image vector corresponding to each key video frame, and the video question data contains question text; The generating of video answer data based on the video question data and key information corresponding to the base library video frame sequence includes: Inputting the question text into a pre-trained text feature extraction model to obtain a question text vector output by the text feature extraction model; and / or inputting the question text into a pre-trained multimodal feature extraction model to obtain a multimodal vector output by the multimodal feature extraction model; Calculating the similarity between the video text vector of each key video frame and the question text vector to obtain the text vector similarity corresponding to each key video frame; and / or calculating the similarity between the video image vector of each key video frame and the multimodal vector to obtain the multimodal vector similarity corresponding to each key video frame; Based on the text vector similarity and / or multimodal vector similarity corresponding to each key video frame, respectively calculating the question matching degree corresponding to each key video frame; Select key video frames whose matching degree meets preset conditions to obtain matching video frames; Based on each matching video frame, video answer data is generated.

7. The method according to claim 6, characterized in that The base video frame sequence also includes a plurality of common video frames; generating video answer data based on each matching video frame includes: Selecting a common video frame adjacent to the matching video frame from the base video frame sequence, combining the matching video frame with the common video frame adjacent to the matching video frame to obtain a matching video segment corresponding to the matching video frame; Inputting the matching video clip and the question text into a pre-trained question response model, and obtaining an initial answer of the matching video clip relative to the question text output by the question response model; Based on the initial answer corresponding to each matching video clip, video answer data is generated.

8. The method according to claim 7, characterized in that Generating video answer data based on the initial answer corresponding to each matching video clip includes: Based on the time information of each matching video clip, the initial answers corresponding to each matching video clip are sorted in chronological order to obtain a result sorting sequence; The result sorting sequence and the question text are input into a pre-trained answer summary model to obtain video answer data output by the answer summary model.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, and the processor is used to execute program instructions stored in the memory to implement the steps in the video key information extraction method as described in any one of claims 1-4 or the video question-answering method as described in any one of claims 5-8.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores program instructions, which can be executed by a processor to implement the steps in the video key information extraction method according to any one of claims 1 to 4 or the video question-answering method according to any one of claims 5 to 8.

Images