Video processing method and device, electronic equipment and storage medium

Abstract

The invention provides a video processing method and device, electronic equipment and a storage medium, and relates to the technical field of video processing, and the method comprises the steps: obtaining a to-be-processed video and a question related to the to-be-processed video; performing frame splitting processing on the to-be-processed video to obtain an initial frame set; removing redundant video frames based on a grey-scale map sequence corresponding to the video frames in the initial frame set to obtain a candidate frame set; performing keyword processing on the text of the questioning question to obtain a keyword set; selecting a key frame from the candidate frame set based on the association degree sequence of the candidate frame set and the keyword set, and taking the key frame as a key frame set; and inputting the key frame set and the questioning question into a pre-constructed multi-mode video understanding large model to obtain a question and answer result corresponding to the questioning question. Therefore, a large number of irrelevant redundant frames are prevented from serving as key frames, meanwhile, loss of key detail frames is avoided, and then the accuracy of key frame extraction is improved.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a video processing method, device, electronic equipment and storage medium. Background Art

[0002] With the increasing use of video in recent years, the accumulation of large amounts of video data across a variety of scenarios and the development of multimodal large-scale model technology have led to the emergence of this technology. This technology primarily uses intelligent technologies such as deep learning to integrate image, text, voice, and time series information to automatically identify and analyze video content. It is commonly used for tasks such as multi-round video question-answering, time-series action recognition, and video classification.

[0003] Currently, a method that limits the total number of key frames and performs fixed-frequency acquisition is often used to extract key frames from videos. Although a set of key frames can be quickly obtained, it is not associated with the actual question-answering questions. Therefore, it is difficult to ensure the relevance of the extracted frames to the dialogue questions, resulting in a large number of redundant frames participating in the calculation, leading to a waste of computing resources. Moreover, due to the interval acquisition, the continuous temporal behavior is incomplete, and key detail frames are easily lost.

[0004] Therefore, when extracting key frames of a video based on the above-mentioned method, redundant frames and key detail frames are easily lost, thereby affecting the accuracy of key frame extraction. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a video processing method, apparatus, electronic device, and storage medium to solve the problem in the prior art that affects the accuracy of key frame extraction.

[0006] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0007] A first aspect of an embodiment of the present invention provides a video processing method, the method comprising:

[0008] Obtaining a video to be processed and questions related to the video to be processed;

[0009] Performing frame decomposition processing on the video to be processed to obtain an initial frame set, where the initial frame set consists of each video frame corresponding to the video to be processed, and each video frame is a frame image;

[0010] removing redundant video frames based on grayscale image sequences corresponding to the video frames in the initial frame set to obtain a candidate frame set;

[0011] Perform keyword processing on the text of the question to obtain a keyword set;

[0012] Selecting key frames from the candidate frame set based on a correlation degree sequence between the candidate frame set and the keyword set, and using the key frames as the key frame set;

[0013] The key frame set and the question are input into a pre-built multimodal video understanding model to obtain a question-and-answer result corresponding to the question. The multimodal video understanding model is trained based on historical videos to be processed and corresponding questions.

[0014] Optionally, removing redundant video frames based on grayscale image sequences corresponding to video frames in the initial frame set to obtain a candidate frame set includes:

[0015] For each video frame in the initial frame set, converting the frame image corresponding to the video frame into a grayscale image sequence, wherein the grayscale image sequence is composed of the grayscale value of each pixel in the corresponding frame image;

[0016] For each adjacent video frame in the initial frame set, processing is performed based on a grayscale image sequence corresponding to the adjacent video frame to obtain an absolute value sequence of the adjacent video frame;

[0017] Redundant video frames in the initial frame set are removed based on absolute value sequences of adjacent video frames to obtain a candidate frame set.

[0018] Optionally, for each adjacent video frame in the initial frame set, processing is performed based on a grayscale image sequence corresponding to the adjacent video frame to obtain an absolute value sequence of the adjacent video frame, including:

[0019] For each adjacent video frame in the initial frame set, determining the grayscale value at the same position based on a grayscale image sequence corresponding to the adjacent video frames;

[0020] Calculating based on the grayscale values at the same position to obtain a grayscale difference sequence;

[0021] Abnormal values in the grayscale difference sequence are replaced by preset grayscale differences to obtain an absolute value sequence of adjacent video frames.

[0022] Optionally, removing redundant video frames in the initial frame set based on an absolute value sequence of adjacent video frames to obtain a candidate frame set includes:

[0023] If it is determined that the absolute value sequence of the adjacent video frames is less than a preset threshold, the next frame of the adjacent video frames is taken as a redundant video frame from the initial frame set and removed;

[0024] After removing redundant video frames from the initial frame set, video frames are obtained and used as the candidate frame set.

[0025] Optionally, keyword processing is performed on the text of the question to obtain a keyword set, including:

[0026] Perform word segmentation and part-of-speech tagging on the text of the question to obtain a text label list;

[0027] Select a word segment with a preset part of speech from the text tag list as a keyword to obtain a keyword list.

[0028] Optionally, selecting key frames from the candidate frame set based on a sequence of association levels between the candidate frame set and the keyword set and using them as the key frame set includes:

[0029] Calculating a corresponding degree of association based on each video frame in the candidate frame set and the keyword set;

[0030] constructing a correlation degree sequence based on the correlation degrees corresponding to the video frames;

[0031] Key frames are selected from the candidate frame set according to the magnitude of the association degree in the association degree sequence, and are used as the key frame set.

[0032] Optionally, calculating the corresponding degree of association based on each video frame in the candidate frame set and the keyword set includes:

[0033] Inputting each video frame in the candidate frame set into a preset entity recognition model, processing each video frame based on the preset entity recognition model to obtain a target list;

[0034] For each video frame, processing is performed based on the target list and the keyword set corresponding to the video frame to obtain a corresponding degree of association.

[0035] A second aspect of an embodiment of the present invention provides a video processing device, comprising:

[0036] An acquisition module, configured to acquire a video to be processed and questions related to the video to be processed;

[0037] a video processing module configured to perform frame decomposition processing on the video to be processed to obtain an initial frame set, wherein the initial frame set is composed of each video frame corresponding to the video to be processed, and each video frame is a frame image; and to remove redundant video frames based on a grayscale image sequence corresponding to the video frames in the initial frame set to obtain a candidate frame set;

[0038] A text analysis module, configured to perform keyword processing on the text of the question to obtain a keyword set;

[0039] a key frame extraction module, configured to select key frames from the candidate frame set based on a sequence of association degrees between the candidate frame set and the keyword set, and use the selected key frames as the key frame set;

[0040] A multimodal video understanding large model is used to input the key frame set and the question into a pre-built multimodal video understanding large model to obtain a question-and-answer result corresponding to the question. The multimodal video understanding large model is trained based on historical videos to be processed and corresponding questions.

[0041] The third aspect of an embodiment of the present invention shows an electronic device, which includes a processor and a memory, wherein the memory is used to store program code and data for video processing, and the processor is used to call the program instructions in the memory to execute any video processing method as described in the first aspect of the embodiment of the present invention.

[0042] A fourth aspect of an embodiment of the present invention shows a storage medium, which includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the video processing method as described in any one of the first aspects of the embodiment of the present invention.

[0043] Based on the above-mentioned embodiments of the present invention, a video processing method, device, electronic device and storage medium are provided. The method includes: obtaining a video to be processed and a question related to the video to be processed; performing frame decomposition processing on the video to be processed to obtain an initial frame set, wherein the initial frame set is composed of each video frame corresponding to the video to be processed, and each video frame is a frame image; removing redundant video frames based on the grayscale image sequence corresponding to the video frames in the initial frame set to obtain a candidate frame set; performing keyword processing on the text of the question to obtain a keyword set; selecting key frames from the candidate frame set based on the correlation degree sequence between the candidate frame set and the keyword set, and using them as the key frame set; inputting the key frame set and the question into a pre-constructed multimodal video understanding large model to obtain a question and answer result corresponding to the question, wherein the multimodal video understanding large model is trained based on historical videos to be processed and corresponding questions. In an embodiment of the present invention, the video to be processed is first processed and split to obtain an initial frame set; then redundant video frames are removed to obtain a candidate frame set; then, based on the actual dialogue question, that is, the question being asked, frames closely related to the dialogue question are selected from the candidate frame set as key frames, and then fed into a multimodal video understanding model to prevent a large number of irrelevant redundant frames from being used as key frames and to avoid the loss of key detail frames, thereby improving the accuracy of key frame extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0045] Figure 1 A schematic flow chart of a video processing method according to an embodiment of the present invention;

[0046] Figure 2 A schematic diagram of video processing according to an embodiment of the present invention;

[0047] Figure 3 The figure is a schematic structural diagram of a video processing device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0049] The terms "first," "second," "third," "fourth," and the like (if any) in the specification and claims of this application and in the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequential sequence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusions, e.g., a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0050] It should be noted that the descriptions of "first", "second", etc. in the present invention are for descriptive purposes only and should not be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various embodiments can be combined with each other, but this must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0051] In this application, the terms "comprises," "comprising," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0052] Among them, the video processing link, that is, the key frame extraction environment in the multimodal video understanding large model, is a key step in the multimodal video understanding large model. It provides a key data foundation for subsequent video understanding and analysis. The number of key frames and their closeness to the conversation content directly affect the effectiveness of video understanding question and answering.

[0053] A video frame is the basic unit of video, representing a single moment in time. It's a still image, typically 24 or 60 frames per second. A key frame is a frame within a video that contains crucial information, such as moments like fighting, smoking, or running.

[0054] This application shows a key frame extraction method in a large multimodal video understanding model. It can combine actual dialogue problems, select frames closely related to the dialogue problems as key frames, and then send them into the multimodal video understanding model, preventing a large number of irrelevant redundant frames from being used as key frames, while avoiding the loss of key detail frames. When using a multimodal video understanding model with a smaller parameter scale, accurate question and answer can also be achieved.

[0055] See also Figure 1 , is a flow chart of a video processing method according to an embodiment of the present invention, the method comprising:

[0056] Step S101: Obtain a video to be processed and questions related to the video to be processed.

[0057] In the specific implementation of step S101 , the video to be processed and the corresponding question input by the user, ie the question asked, are obtained.

[0058] Step S102: performing frame splitting processing on the video to be processed to obtain an initial frame set.

[0059] In step S102 , the initial frame set is composed of each video frame corresponding to the video to be processed, and each video frame is a frame image.

[0060] In the specific implementation of step S102, the video to be processed is split frame by frame to obtain multiple video frames split according to the video sequence, and the frame number of each video frame is set according to the video time sequence, and the multiple video frames split according to the video sequence are combined into an initial frame set.

[0061] The initial frame set is an image sequence MS=[m1, m2, ..., mn] of all video frames of the video to be processed, wherein mi represents the video frame image of the i-th frame, and the corresponding frame number is i.

[0062] Step S103: removing redundant video frames based on grayscale image sequences corresponding to video frames in the initial frame set to obtain a candidate frame set.

[0063] It should be noted that the specific process of implementing step S103 includes the following steps.

[0064] Step S11: for each video frame in the initial frame set, convert the frame image corresponding to the video frame into a grayscale image sequence, where the grayscale image sequence is composed of the grayscale value of each pixel in the corresponding frame image.

[0065] In the specific implementation of step S11, since each pixel in the image is composed of RGB three colors, different weights are assigned to the RGB three color components, and then the grayscale value of each pixel of the frame image corresponding to the video frame is calculated, and the grayscale value of each pixel is combined to obtain a grayscale image sequence corresponding to the video frame.

[0066] The calculation formula for the grayscale value of each pixel shown in this application is shown in formula (1).

[0067] Formula (1):

[0068] Grayscale value = a1*R+a2*G+a3*B.

[0069] It should be noted that a1 is the weight corresponding to red R, which can generally be set to 0.299; a2 is the weight corresponding to green G, which can generally be set to 0.587; a3 is the weight corresponding to blue B, which can generally be set to 0.114.

[0070] For example, suppose the RGB value of each pixel in a video frame IMG_01 is:

[0071] [[[198, 193, 194], [199, 194, 195], [203, 198, 199], ..., [173, 185, 191], [180, 190, 197], [183, 193, 200]], [[199, 194, 195], [198, 193, 194], [201, 196, 197], .. .,[164,176,182],[180,190,197],[189,199,206]],[[201,196,197],[200,195,196],[200,195,196],...,[189,201,207],[193,203,210],[190,200,207]]].

[0072] Wherein, each [] is a set of RGB values of a pixel, such as [198, 193, 194] means R=198, G=193, B=194. Substituting it into formula (1) for calculation, we can get the grayscale value corresponding to the pixel = 0.299*R+0.587*G+0.114*B=194.

[0073] Next, the grayscale value corresponding to each pixel in the video frame is calculated in the above manner, and then the grayscale value of each pixel is used to form a grayscale image sequence.

[0074] The grayscale image sequence corresponding to the video frame IMG_01 can be [[194, 195, 199, ..., 185, 191, 194], ..., [150, 150, 150, ..., 136, 135, 137], [59, 63, 67, ..., 142, 140, 142]].

[0075] Similarly, the grayscale image sequence corresponding to the video frame IMG_02 can be [[194, 196, 199, ..., 186, 197, 196], ..., [150, 150, 150, ..., 136, 135, 137], [59, 63, 67, ..., 142, 140, 142]].

[0076] Step S12: for each adjacent video frame in the initial frame set, processing is performed based on a grayscale image sequence corresponding to the adjacent video frame to obtain an absolute value sequence of the adjacent video frame.

[0077] It should be noted that the specific implementation of step S12 includes the following steps:

[0078] Step S21: for each adjacent video frame in the initial frame set, determine the grayscale value at the same position based on the grayscale image sequence corresponding to the adjacent video frames.

[0079] During step S21 within a specific time period, grayscale values at the same position, ie, grayscale values at the same pixel position, are obtained from grayscale image sequences corresponding to two adjacent video frames.

[0080] Step S22: performing calculation based on the grayscale values at the same position to obtain a grayscale difference sequence.

[0081] In the specific implementation of step S22, first, the grayscale values at the same position are subjected to difference calculation and the absolute value is obtained; and the difference at each position is used as the grayscale difference sequence of the video frame.

[0082] Step S23: replacing abnormal values in the grayscale difference sequence with preset grayscale differences to obtain an absolute value sequence of adjacent video frames.

[0083] In the specific implementation of step S23, in order to avoid minor differences caused by lighting or other conditions, it is necessary to eliminate abnormal values in the grayscale difference sequence. Specifically, it is determined whether there is a grayscale difference less than a preset abnormal threshold in the grayscale difference sequence. If so, the grayscale difference less than the preset abnormal threshold is determined to be an abnormal value; then, the abnormal value is replaced with the preset grayscale difference; if the grayscale difference is greater than or equal to the preset abnormal threshold, it is indicated that the grayscale difference is normal and does not need to be replaced. All grayscale differences in the grayscale difference sequence are processed in sequence, and the grayscale difference sequence at this time is used as the absolute value sequence of the adjacent video frames.

[0084] It should be noted that the preset grayscale difference is set in advance by technicians based on multiple experiments and can generally be set to 0.

[0085] The preset abnormality threshold is also set in advance by technicians based on multiple experiments, and can generally be set to 5.

[0086] In order to better understand the contents shown in the above steps S21 to S23, the following description is made with reference to adjacent video frames IMG_01 and IMG_02.

[0087] Obtain the grayscale values of the same position from the grayscale image sequences corresponding to the adjacent video frames MG_01 and IMG_02, that is, the grayscale values of the same pixel position; perform difference calculation on the grayscale values at the same position and take the absolute value to obtain a grayscale difference sequence, that is, [[0, 1, 0, ..., 1, 6, 2], ..., [0, 0, 0, ..., 0, 0, 0], [0, 0, 0, ..., 0, 0, 0]]; replace the values of the grayscale difference sequence with grayscale differences less than 5 with a preset grayscale difference value of 0 to obtain an absolute value sequence of the adjacent video frames [[0, 0, 0, ..., 0, 6, 0], ..., [0, 0, 0, ..., 0, 0, 0], [0, 0, 0, ..., 0, 0, 0]]

[0088] It should be noted that each group [] represents the grayscale difference corresponding to a pair of adjacent video frames.

[0089] The present application uses the above method, i.e., using the inter-frame difference method, to perform differential operations on two adjacent frames in the grayscale image sequence MS_G, i.e., each pair of adjacent video frames, to obtain the absolute value sequence of the differences between adjacent frames. S=[s1,s2,…,s (n-1) ] to determine the absolute value sequence of each pair of adjacent videos. This allows us to find the motion or changes that occur in the image sequence and remove static frames or redundant frames with high similarity.

[0090] Where n is the total number of video frames.

[0091] Step S13: removing redundant video frames in the initial frame set based on the absolute value sequence of adjacent video frames to obtain a candidate frame set.

[0092] The specific process of implementing step S13 includes the following steps.

[0093] Step S31: determine whether the absolute value sequence of the adjacent video frames is less than a preset threshold. If so, execute step S32; otherwise, retain the video frames corresponding to the absolute value sequence in the initial frame set.

[0094] In the specific implementation of step S31, the absolute value sequence of the adjacent video frames is determined one by one. Whether each grayscale difference in S is not less than a preset threshold thres, if it is, it is considered that the two adjacent video frames in the frame image sequence MS are redundant, and step S14 is executed; otherwise, the video frames corresponding to the absolute value sequence are retained.

[0095] It should be noted that the setting of the preset threshold is determined based on the total number TN (such as 512) of the preset reserved frames. Specifically, The largest value Tn is selected from S as the threshold thres of the absolute value of the difference between adjacent frames.

[0096] Optionally, in addition to the method shown in step S31 above, step S31 can also be implemented in other ways. Specifically, the grayscale differences in the absolute value sequence of the adjacent video frames are sorted in order from large to small; and the video frames corresponding to the first TN grayscale differences are selected in order from large to small as the candidate frame set.

[0097] Step S32: The next frame among the adjacent video frames is treated as a redundant video frame and removed.

[0098] In the specific implementation of step S32, the next frame in the video corresponding to the grayscale difference is selected to complete the removal of redundant frames.

[0099] Step S33: Count the video frames obtained after removing redundant video frames from the initial frame set, and use them as the candidate frame set.

[0100] It should be noted that the candidate frame set , The images in are original video frame images, not grayscale images.

[0101] in, Represents the video frame image after the t-th split, and the candidate frame set stores the video frame images in chronological order.

[0102] Step S104: performing keyword processing on the text of the question to obtain a keyword set.

[0103] The keyword set of this application can be determined by extracting the entity names involved in the question. Specifically, the entity extraction technology in the field of natural language processing is used to identify and extract meaningful entities related to the question from the text of the unstructured question. These entities include people, animals, objects, actions, places, time, etc., and these entities constitute keywords. For example, "What are the people in this video doing?", the entity "people" can be extracted from it. The specific process of entity extraction, that is, the process of specifically implementing step S104, includes the following steps:

[0104] Step S41: performing word segmentation and part-of-speech tagging on the text of the question to obtain a text label list;

[0105] In the specific implementation of step S41, a lexical analyzer is used to segment the text of the question, and each segmented word is tagged with the part of speech to obtain a text label list WL=[("word1","seg1"), ("word2","seg2"), ..., ("wordn","segn")].

[0106] Among them, word is a word and seg is a part-of-speech identifier.

[0107] Step S42: Selecting a participle of a preset part of speech from the text tag list as a keyword to obtain a keyword list.

[0108] In the specific implementation of step S42, each word in the text tag list is traversed to determine the word with the preset part of speech, that is, the word, and the word with the preset part of speech is used as a keyword; then the keywords are combined to obtain a keyword list.

[0109] For example, the keyword list WL_K can be [“word1”, “word2”, …].

[0110] This application uses the entity extraction method in the field of natural language processing to retain only word tags with the part of speech being noun, so as to obtain a set of keywords from the question.

[0111] Step S105: selecting key frames from the candidate frame set based on the association degree sequence between the candidate frame set and the keyword set, and using them as the key frame set.

[0112] It should be noted that the specific implementation of step S105 includes the following steps.

[0113] Step S51: Calculating the corresponding association degree based on each video frame in the candidate frame set and the keyword set.

[0114] It should be noted that the specific implementation of step S51 includes the following steps.

[0115] Step S61: input each video frame in the candidate frame set into a preset entity recognition model, and process each video frame based on the preset entity recognition model to obtain a target list.

[0116] It should be noted that the target list includes the location, category and confidence of each category entity.

[0117] The preset entity recognition model is an open-set detection model that can use open-set detection to assume that the test data may contain categories that have not been seen in the training data. That is to say, it uses hypothetical test data, namely historical videos and manually annotated objects and categories, for training to construct the corresponding preset entity recognition model.

[0118] by First frame For example, the target list Items is {["box":[x0,y0,x1,y1],"lable":"name1","score":sc1],…}, where box represents the location coordinates, which is [x0,y0,x1,y1], lable represents the category label, and score represents the confidence level.

[0119] Optionally, after obtaining the target list for each video frame, first remove the physical objects with a confidence level less than a preset confidence level and the corresponding category labels.

[0120] Step S62: For each video frame, processing is performed based on the target list and the keyword set corresponding to the video frame to obtain a corresponding correlation degree.

[0121] In the specific implementation of step S62, for each video frame, the video frame is calculated. The degree of correlation with the keyword set WL_K Specifically, compare whether the entity name in the target list is the same as the keyword in the keyword set, and count the number of identical keywords, that is, The target corresponding to the physical object is detected and accumulated to obtain the video frame The degree of association with the jth keyword in the keyword set WL_K .

[0122] Among them, SC means that the keywords in the keyword set WL_K appear in the category corresponding to the video frame, that is, in the video frame The target corresponding to the physical object is detected.

[0123] Step S52: constructing a correlation degree sequence based on the correlation degrees corresponding to the video frames.

[0124] In the specific implementation of step S52, the correlation degree is calculated in the manner of step S62 until the correlation degree of all video frames in the candidate frame set MS_C is calculated, and the correlation degree of each video frame is combined into a correlation degree sequence R=[r1,r2,…,r j ].

[0125] Step S53: selecting key frames from the candidate frame set according to the magnitude of the association degree in the association degree sequence, and using them as the key frame set.

[0126] In the specific implementation of step S53 , the correlation degree greater than a preset threshold is determined from the correlation degree sequence, and the video frames corresponding to the correlation degree greater than the preset threshold are used as key frames, thereby forming a key frame set.

[0127] It should be noted that the order of the video frames in the key frame set remains the same as that of the original video frame sequence MS, that is, the order of the video frames in the key frame set is sorted in ascending order of frame number.

[0128] The preset degree threshold is the maximum number of key frames N set in advance by the technician (for example, it can be set to 64), and the first N frames of images with the highest degree of correlation are selected from the correlation degree sequence R, that is, the video frames as the final key frame set MS_T.

[0129] Step S106: input the key frame set and the question into a pre-built multimodal video understanding model to obtain a question-answering result corresponding to the question.

[0130] It should be noted that the question-and-answer result is the text corresponding to the question asked. That is, the multimodal video understanding model interprets the key frame set according to the question asked to output the text corresponding to the question asked.

[0131] The multimodal video understanding model is trained based on historical videos to be processed, questions asked, and corresponding question-and-answer results.

[0132] Optionally, the method based on the above steps S101 to S106 can be described by a schematic diagram, as shown in FIG. Figure 2 shown.

[0133] Optionally, the intervals between the key frames in the key frame set processed by this application are not fixed, which can avoid the calculation of redundant frames; and the video frames finally obtained by this application, that is, the question and answer results do not include every time stage of the video, but may be concentrated in one or several sections of the video, and can focus more on question and answer related segments.

[0134] In an embodiment of the present invention, the video to be processed is first processed and split to obtain an initial frame set; then redundant video frames are removed to obtain a candidate frame set; then, combined with the actual dialogue question, that is, the question asked, frames closely related to the dialogue question are selected from the candidate frame set as key frames, and then sent into the multimodal video understanding model to prevent a large number of irrelevant redundant frames from being used as key frames, and at the same time avoid the loss of key detail frames. When using a multimodal video understanding model with a smaller parameter order, accurate question and answer can also be achieved.

[0135] Based on the method shown in the above embodiment of the present invention, the embodiment of the present invention also shows a video processing device, such as Figure 3 As shown, the device includes:

[0136] An acquisition module 301 is used to acquire a video to be processed and questions related to the video to be processed;

[0137] The video processing module 302 is configured to perform frame decomposition processing on the video to be processed to obtain an initial frame set, wherein the initial frame set is composed of each video frame corresponding to the video to be processed, and each video frame is a frame image; and remove redundant video frames based on the grayscale image sequence corresponding to the video frames in the initial frame set to obtain a candidate frame set;

[0138] A text analysis module 303 is used to perform keyword processing on the text of the question to obtain a keyword set;

[0139] A key frame extraction module 304 is configured to select key frames from the candidate frame set based on a correlation degree sequence between the candidate frame set and the keyword set, and use the selected key frames as the key frame set;

[0140] The multimodal video understanding large model 305 is used to input the key frame set and the question into a pre-built multimodal video understanding large model to obtain a question-and-answer result corresponding to the question. The multimodal video understanding large model is trained based on historical videos to be processed and corresponding questions.

[0141] The specific principles and execution processes of each unit in the video processing device disclosed in the above embodiment of the present invention are the same as the corresponding contents in the video processing method provided in the above embodiment of the present invention. Please refer to the corresponding parts of the video processing method disclosed in the above embodiment of the present invention, and no further details will be given here.

[0142] In an embodiment of the present invention, the video to be processed is first processed and split to obtain an initial frame set; then redundant video frames are removed to obtain a candidate frame set; then, combined with the actual dialogue question, that is, the question asked, frames closely related to the dialogue question are selected from the candidate frame set as key frames, and then sent into the multimodal video understanding model to prevent a large number of irrelevant redundant frames from being used as key frames, and at the same time avoid the loss of key detail frames. When using a multimodal video understanding model with a smaller parameter order, accurate question and answer can also be achieved.

[0143] Optionally, based on the video processing device shown in the embodiment of the present invention, the video processing module 302 removes redundant video frames based on the grayscale image sequence corresponding to the video frames in the initial frame set to obtain the candidate frame set, and is specifically configured to:

[0144] For each video frame in the initial frame set, converting the frame image corresponding to the video frame into a grayscale image sequence, wherein the grayscale image sequence is composed of the grayscale value of each pixel in the corresponding frame image;

[0145] For each adjacent video frame in the initial frame set, processing is performed based on a grayscale image sequence corresponding to the adjacent video frame to obtain an absolute value sequence of the adjacent video frame;

[0146] Redundant video frames in the initial frame set are removed based on absolute value sequences of adjacent video frames to obtain a candidate frame set.

[0147] The process of processing each adjacent video frame in the initial frame set based on a grayscale image sequence corresponding to the adjacent video frames to obtain an absolute value sequence of the adjacent video frames includes:

[0148] For each adjacent video frame in the initial frame set, determining the grayscale value at the same position based on a grayscale image sequence corresponding to the adjacent video frames;

[0149] Calculation is performed based on the grayscale values at the same position to obtain a grayscale difference sequence.

[0150] Abnormal values in the grayscale difference sequence are replaced by preset grayscale differences to obtain an absolute value sequence of adjacent video frames.

[0151] The step of removing redundant video frames from the initial frame set based on an absolute value sequence of adjacent video frames to obtain a candidate frame set includes:

[0152] If it is determined that the absolute value sequence of the adjacent video frames is less than a preset threshold, the next frame of the adjacent video frames is taken as a redundant video frame from the initial frame set and removed;

[0153] After removing redundant video frames from the initial frame set, video frames are obtained and used as the candidate frame set.

[0154] Optionally, based on the video processing device shown in the above embodiment of the present invention, the text analysis module 303 is specifically configured to:

[0155] Perform word segmentation and part-of-speech tagging on the text of the question to obtain a text label list;

[0156] Select a word segment with a preset part of speech from the text tag list as a keyword to obtain a keyword list.

[0157] Optionally, based on the video processing device shown in the above embodiment of the present invention, the key frame extraction module 304 is specifically configured to:

[0158] Calculating a corresponding degree of association based on each video frame in the candidate frame set and the keyword set;

[0159] constructing a correlation degree sequence based on the correlation degrees corresponding to the video frames;

[0160] Key frames are selected from the candidate frame set according to the magnitude of the association degree in the association degree sequence, and are used as the key frame set.

[0161] The step of calculating the corresponding degree of association based on each video frame in the candidate frame set and the keyword set includes:

[0162] Inputting each video frame in the candidate frame set into a preset entity recognition model, processing each video frame based on the preset entity recognition model to obtain a target list;

[0163] For each video frame, processing is performed based on the target list and the keyword set corresponding to the video frame to obtain a corresponding degree of association.

[0164] An embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory is used to store video processing program code and data, and the processor is used to call program instructions in the memory to execute the steps shown in the video processing method in the above embodiment.

[0165] An embodiment of the present invention provides a storage medium, which includes the electronic device provided by the above-mentioned embodiment of the present application, and the electronic device is used to execute the video processing method disclosed in the embodiment of the present application.

[0166] Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple. For relevant parts, refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0167] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0168] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A video processing method, characterized in that: The method comprises: Obtaining a video to be processed and questions related to the video to be processed; Performing frame decomposition processing on the video to be processed to obtain an initial frame set, where the initial frame set consists of each video frame corresponding to the video to be processed, and each video frame is a frame image; removing redundant video frames based on grayscale image sequences corresponding to the video frames in the initial frame set to obtain a candidate frame set; Perform keyword processing on the text of the question to obtain a keyword set; Selecting key frames from the candidate frame set based on a sequence of association levels between the candidate frame set and the keyword set, and using the key frames as the key frame set; The key frame set and the question are input into a pre-built multimodal video understanding model to obtain a question-and-answer result corresponding to the question. The multimodal video understanding model is trained based on historical videos to be processed and corresponding questions.

2. The method according to claim 1, characterized in that Redundant video frames are removed based on grayscale image sequences corresponding to the video frames in the initial frame set to obtain a candidate frame set, including: For each video frame in the initial frame set, converting the frame image corresponding to the video frame into a grayscale image sequence, wherein the grayscale image sequence is composed of the grayscale value of each pixel in the corresponding frame image; For each adjacent video frame in the initial frame set, processing is performed based on a grayscale image sequence corresponding to the adjacent video frame to obtain an absolute value sequence of the adjacent video frame; Redundant video frames in the initial frame set are removed based on absolute value sequences of adjacent video frames to obtain a candidate frame set.

3. The method according to claim 2, characterized in that For each adjacent video frame in the initial frame set, processing is performed based on a grayscale image sequence corresponding to the adjacent video frame to obtain an absolute value sequence of the adjacent video frame, including: For each adjacent video frame in the initial frame set, determining the grayscale value at the same position based on a grayscale image sequence corresponding to the adjacent video frames; Calculating based on the grayscale values at the same position to obtain a grayscale difference sequence; Abnormal values in the grayscale difference sequence are replaced by preset grayscale differences to obtain an absolute value sequence of adjacent video frames.

4. The method according to claim 2, characterized in that Removing redundant video frames from the initial frame set based on an absolute value sequence of adjacent video frames to obtain a candidate frame set, comprising: If it is determined that the absolute value sequence of the adjacent video frames is less than a preset threshold, the next frame of the adjacent video frames is taken as a redundant video frame from the initial frame set and removed; After removing redundant video frames from the initial frame set, video frames are obtained and used as the candidate frame set.

5. The method according to claim 1, wherein Perform keyword processing on the text of the question to obtain a keyword set, including: Perform word segmentation and part-of-speech tagging on the text of the question to obtain a text label list; Select the participles with preset parts of speech from the text tag list as keywords to obtain a keyword list.

6. The method according to claim 1, characterized in that The method further comprises: selecting key frames from the candidate frame set based on a sequence of association degrees between the candidate frame set and the keyword set, and using the key frames as the key frame set. Calculating a corresponding degree of association based on each video frame in the candidate frame set and the keyword set; constructing a correlation degree sequence based on the correlation degrees corresponding to the video frames; Key frames are selected from the candidate frame set according to the magnitude of the association degree in the association degree sequence, and are used as the key frame set.

7. The method according to claim 6, characterized in that Calculating a corresponding degree of association based on each video frame in the candidate frame set and the keyword set includes: Inputting each video frame in the candidate frame set into a preset entity recognition model, processing each video frame based on the preset entity recognition model to obtain a target list, wherein the target list includes the position, category and confidence of each category entity; For each video frame, processing is performed based on the target list and the keyword set corresponding to the video frame to obtain a corresponding degree of association.

8. A video processing device, characterized in that: The device comprises: An acquisition module, configured to acquire a video to be processed and questions related to the video to be processed; a video processing module configured to perform frame decomposition processing on the video to be processed to obtain an initial frame set, wherein the initial frame set is composed of each video frame corresponding to the video to be processed, and each video frame is a frame image; and to remove redundant video frames based on a grayscale image sequence corresponding to the video frames in the initial frame set to obtain a candidate frame set; A text analysis module, configured to perform keyword processing on the text of the question to obtain a keyword set; a key frame extraction module, configured to select key frames from the candidate frame set based on a sequence of association degrees between the candidate frame set and the keyword set, and use the selected key frames as the key frame set; A multimodal video understanding large model is used to input the key frame set and the question into a pre-built multimodal video understanding large model to obtain a question-and-answer result corresponding to the question. The multimodal video understanding large model is trained based on historical videos to be processed and corresponding questions.

9. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory is used to store program codes and data for video processing, and the processor is used to call program instructions in the memory to execute the video processing method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the video processing method according to any one of claims 1 to 7.

Images