Video processing method and video retrieval enhancement method for multi-modal large model

Abstract

The invention provides a video processing method and a video retrieval enhancement method for a multi-modal large model, and relates to the technical field of artificial intelligence. The method comprises the following steps: performing target identification on a to-be-processed video, and obtaining a target identification result of each video frame in the to-be-processed video; segmenting the to-be-processed video into a plurality of video segments according to the target identification result; extracting a key frame sequence in each video clip; obtaining text information corresponding to each video clip; based on the key frame sequence and the text information, determining feature information of each video clip, and storing each video clip and the corresponding feature information in a feature library; wherein the feature library is used for video retrieval enhancement processing of the multi-modal large model. According to the method, key information in the video can be reserved, the video search efficiency can be improved, the video clip can be accurately positioned, and the video question and answer efficiency can be improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a video processing method and a video retrieval enhancement method for a multimodal large model. Background Art

[0002] As pre-trained big model technology gradually forms large-scale practical applications, traditional conversational big model services can no longer meet the personalized needs of users. Rapidly mounting private knowledge bases through retrieval enhancement generation technology and building personalized big model applications have become current research hotspots.

[0003] Currently, in the field of large-scale model retrieval and enhanced generation, video retrieval is mainly based on video titles, tag sets, keywords and text descriptions for related knowledge retrieval. This information is usually labeled manually or by AI, but it is easy to lose a lot of key information in the video, so the retrieval accuracy is low. Summary of the invention

[0004] The embodiments of the present invention provide a video processing method and a video retrieval enhancement method for a multimodal large model, so as to solve the technical problem of low accuracy of video retrieval in the existing large model retrieval enhancement generation field.

[0005] In a first aspect, an embodiment of the present invention provides a video processing method for a multimodal large model, comprising: performing target recognition on a video to be processed, and obtaining a target recognition result for each video frame in the video to be processed; according to the target recognition result, dividing the video to be processed into a plurality of video segments; extracting a key frame sequence in each of the video segments; obtaining text information corresponding to each of the video segments; determining feature information of each of the video segments based on the key frame sequence and the text information, and storing each of the video segments and its corresponding feature information in a feature library; wherein the feature library is used for video retrieval enhancement processing of the multimodal large model.

[0006] In some embodiments, dividing the video to be processed into multiple video segments according to the target recognition result includes: determining the target change rate between the previous and next video frames in the video to be processed according to the target recognition result; and dividing the video to be processed into the multiple video segments based on the target change rate.

[0007] In some embodiments, extracting the key frame sequence in each of the video clips includes: determining, based on the target recognition result, the first video frame in each of the video clips where a target object appears or disappears; determining the first video frame in each of the video clips as the key frame of the corresponding video clip; and determining the key frame sequence in each of the video clips based on the key frames of each of the video clips.

[0008] In some embodiments, determining the key frame of each video segment based on the first video frame in each video segment includes: determining the target displacement rate of other video frames between the first video frame in each video segment based on the target recognition result; determining the second video frame in each video segment based on the target displacement rate; and determining the first video frame and the second video frame in each video segment as key frames of the corresponding video segment.

[0009] In some embodiments, obtaining the text information corresponding to each of the video clips includes: based on a speech recognition algorithm, obtaining first text information corresponding to the speech information in each of the video clips; based on a character recognition algorithm, identifying the text content in each of the video clips to obtain second text information corresponding to each of the video clips; based on the first text information and the second text information, determining the text information corresponding to each of the video clips.

[0010] In some embodiments, the feature information of each of the video clips includes text feature information, image feature information and fusion feature information; determining the feature information of each of the video clips based on the key frame sequence and the text information includes: for a first video clip among the multiple video clips, performing feature extraction on a first key frame sequence in the first video clip to obtain image feature information of the first video clip; performing feature extraction on the text information of the first video clip to obtain text feature information of the first video clip; and fusing the image feature information and the text feature information to obtain fusion feature information of the first video clip.

[0011] In some embodiments, the feature extraction of the first key frame sequence in the first video clip to obtain image feature information of the first video clip includes: determining the number of each type of target objects included in the first key frame sequence based on the target recognition result; separately extracting features for each type of the target objects in the first key frame sequence to obtain image feature information of each type of target objects; and determining the image feature information of the first video clip based on the number of each type of target objects and the image feature information of each type of target objects.

[0012] In a second aspect, an embodiment of the present invention provides a video retrieval enhancement method for a multimodal large model, comprising: obtaining user input information; vectorizing the user input information, and based on the vectorized expression of the user input information, searching from a feature library to obtain a target video clip; wherein the feature library is obtained based on the video processing method for a multimodal large model described in the first aspect above; inputting the user input information and the target video clip into the multimodal large model, so that the multimodal large model generates an answer based on the user input information and the target video clip.

[0013] In a third aspect, an embodiment of the present invention provides a video processing device for a multimodal large model, comprising: a first acquisition module, used to perform target recognition on a video to be processed, and obtain a target recognition result for each video frame in the video to be processed; a segmentation module, used to segment the video to be processed into multiple video segments according to the target recognition result; an extraction module, used to extract a key frame sequence in each of the video segments; a second acquisition module, used to obtain text information corresponding to each of the video segments; a determination module, used to determine feature information of each of the video segments based on the key frame sequence and the text information, and store each of the video segments and its corresponding feature information in a feature library; wherein the feature library is used for video retrieval enhancement processing of the multimodal large model.

[0014] In a fourth aspect, an embodiment of the present invention provides a video retrieval enhancement device for a multimodal large model, comprising: a third acquisition module, used to obtain user input information; a retrieval module, used to vectorize the user input information, and based on the vectorized expression of the user input information, search from a feature library to obtain a target video clip; wherein the feature library is obtained based on the video processing method for a multimodal large model described in the first aspect above; an input module, used to input the user input information and the target video clip into the multimodal large model, so that the multimodal large model generates an answer based on the user input information and the target video clip.

[0015] In a fifth aspect, an embodiment of the present invention provides an electronic device, comprising a processor and a memory storing a computer program, wherein when the processor executes the program, it implements the video processing method for a multimodal large model described in the first aspect, and / or the video retrieval enhancement method for a multimodal large model described in the second aspect.

[0016] In a sixth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the video processing method for a multimodal large model described in the first aspect and / or the video retrieval enhancement method for a multimodal large model described in the second aspect are implemented.

[0017] In the seventh aspect, an embodiment of the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the video processing method for a multimodal large model described in the first aspect, and / or the video retrieval enhancement method for a multimodal large model described in the second aspect.

[0018] The video processing method and video retrieval enhancement method for a multimodal large model provided by an embodiment of the present invention perform target recognition on a video to be processed, and obtain the target recognition result of each video frame in the video to be processed; according to the target recognition result, the video to be processed is divided into multiple video clips; the key frame sequence in each video clip is extracted; the text information corresponding to each video clip is obtained; based on the key frame sequence and the text information, the feature information of each video clip is determined, and each video clip and its corresponding feature information are stored in a feature library; wherein the feature library is used for video retrieval enhancement processing of a multimodal large model. The present invention can not only retain the key information in the video, but also improve the efficiency of video search, accurately locate the video clip, and improve the efficiency of video question and answer by segmenting the video and determining the feature information of the video clip based on the key frame sequence and text information of the video clip. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 One of the flow charts of the multimodal large model video processing method provided by the embodiment of the present invention; Figure 2 A second flow chart of a video processing method for a multimodal large model provided by an embodiment of the present invention; Figure 3 A schematic diagram of a flow chart of a video processing method for a multimodal large model provided by an embodiment of the present invention; Figure 4 A fourth flowchart of a video processing method for a multimodal large model provided by an embodiment of the present invention; Figure 5 A fifth flowchart of a video processing method for a multimodal large model provided by an embodiment of the present invention; Figure 6 A schematic diagram of a flow chart of a multimodal large model video retrieval enhancement method provided by an embodiment of the present invention; Figure 7 A diagram showing the association between a video processing method for a multimodal large model and a video retrieval enhancement method according to an embodiment of the present invention; Figure 8 A schematic diagram of the structure of a video processing device for a multimodal large model provided by an embodiment of the present invention; Fig. 9 A schematic diagram of the structure of a video retrieval enhancement device for a multimodal large model provided by an embodiment of the present invention; Fig.10 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] The present invention provides a video processing method and a video retrieval enhancement method for a multimodal large model.

[0023] Figure 1 One of the flow charts of the multi-modal large model video processing method provided by the embodiment of the present invention. Figure 1 As shown, the method may include the following steps.

[0024] Step 101 , performing target recognition on the video to be processed, and obtaining the target recognition result of each video frame in the video to be processed.

[0025] In some embodiments, the video to be processed may be an original video stored in a video database, or may be an original video acquired in a preset manner.

[0026] In some embodiments, a target recognition algorithm may be used to perform target recognition on each video frame in the processed video to obtain a target recognition result for each video frame. The target recognition algorithm may be a target recognition algorithm in the relevant technology, such as a target recognition algorithm based on deep learning. The target recognition result for each video frame may include the targets identified at each position and the category to which each target belongs.

[0027] Step 102: dividing the video to be processed into multiple video segments according to the target recognition result.

[0028] In some embodiments, target tracking can be performed based on the target recognition results, the same target objects in the previous and next video frames can be marked, and the target change value of each video frame relative to the previous frame can be determined based on the marking results. If the target change value of a video frame is greater than the change value threshold, a segmentation process is performed between the video frame and the previous video frame.

[0029] In other words, segmentation processing can be performed when the video scene changes, so that the target objects contained in each video frame in each video segment are similar, thereby improving the accuracy of retrieval.

[0030] Step 103: extract the key frame sequence in each video clip.

[0031] In some embodiments, the key frame sequence in each video segment refers to key data of the video content in each video segment, and can be used to represent the video content information in the corresponding video segment. Each key frame sequence can include multiple key frames.

[0032] In some embodiments, the key frame sequence in each video clip may be extracted based on a key frame extraction algorithm in the related art, or may be implemented based on a key frame extraction algorithm proposed by a person skilled in the art.

[0033] In some embodiments, for each video clip, based on the target recognition result, the displacement change rate of the target object in each video frame in the video clip relative to the target object in the previous key frame can be determined in turn. If the displacement change rate of a video frame is greater than or equal to the displacement change rate threshold, the video frame is used as the key frame of the video clip, and so on.

[0034] Step 104: Obtain text information corresponding to each video clip.

[0035] In some embodiments, the text information corresponding to each video segment may include voice information in the video segment, and may also include text content in the video frame of each video segment.

[0036] In some embodiments, the process of obtaining text information corresponding to each video clip may include: obtaining first text information corresponding to the voice information in each video clip based on a voice recognition algorithm; recognizing the text content in each video clip based on a character recognition algorithm to obtain second text information corresponding to each video clip; and determining text information corresponding to each video clip based on the first text information and the second text information. The character recognition algorithm may be an OCR (Optical Character Recognition) algorithm. The process of determining text information corresponding to each video clip based on the first text information and the second text information may be to splice the first text information and the second text information to obtain text information corresponding to each video clip.

[0037] Step 105, based on the key frame sequence and the text information, determine the feature information of each video clip, and store each video clip and its corresponding feature information in a feature library; wherein the feature library is used for video retrieval enhancement processing of a multimodal large model.

[0038] In some embodiments, for each video segment, the key frame sequence and text information of the video segment can be mapped to a feature vector, that is, feature information of the video segment, and each video segment and its corresponding feature information can be stored in a feature library. As an example, each video segment in the feature library can include corresponding identification information, and the feature information of each video segment is also corresponding based on the identification information of the video segment.

[0039] In some embodiments, feature extraction can be performed based on the text information and key frame sequence of each video clip to obtain a text feature vector corresponding to the text information and an image feature vector corresponding to the key frame sequence, and the text feature vector and image feature vector of each video clip are stored in a feature library respectively. For example, each video clip is stored in a video clip library in the feature library, the text feature vector of each video clip is stored in a text feature library in the feature library, and the image feature vector of each video clip is stored in an image feature library in the feature library.

[0040] In other embodiments, the text feature vector and the image feature vector of each video clip may be fused to obtain fused feature information of each video clip, and the fused feature information of each video clip may be stored in a feature library.

[0041] In some embodiments, the video processing method of the embodiment of the present invention can be used in a question-answering scenario based on a multimodal large model, and the constructed feature library is equivalent to the knowledge base mounted by the retrieval enhancement generation technology for the multimodal large model. When the content information of the user's question is received, it can be retrieved from the feature library based on the content information of the user's question to obtain the matched video clip, and the obtained video clip is sent to the multimodal large model, so that the multimodal large model can understand the video clip and the content information of the user's question and generate the corresponding answer.

[0042] According to the video processing method for a multimodal large model in an embodiment of the present application, target recognition is performed on the video to be processed, and the target recognition result of each video frame in the video to be processed is obtained; according to the target recognition result, the video to be processed is divided into multiple video clips; the key frame sequence in each video clip is extracted; the text information corresponding to each video clip is obtained; based on the key frame sequence and the text information, the feature information of each video clip is determined, and each video clip and its corresponding feature information are stored in a feature library; wherein the feature library is used for video retrieval enhancement processing of a multimodal large model. The present invention can not only retain the key information in the video, but also improve the efficiency of video search, accurately locate the video clip, and improve the efficiency of video question and answer by segmenting the video and determining the feature information of the video clip based on the key frame sequence and text information of the video clip.

[0043] Figure 2 The second flowchart of the video processing method for a multi-modal large model provided by an embodiment of the present invention. Figure 2 As shown, based on the above embodiments, Figure 1 The implementation process of step 102 in may include the following steps.

[0044] Step 201, according to the target recognition result, determine the target change rate between the previous and next video frames in the video to be processed.

[0045] Step 202: based on the target change rate, the video to be processed is divided into a plurality of video segments.

[0046] In some embodiments, target tracking can be performed based on the target recognition result, the same target objects in each video frame can be marked, and the target change rate can be determined based on the number of the same targets between different video frames. The target change rate of each video frame is used to express the similarity between the previous and next video frames.

[0047] In some embodiments, calculating the target change rate between the previous and next video frames in the video to be processed may be calculating the target change rate between each video frame and its previous video frame, or may be calculating the target change rate between each video frame and the first video frame of the video segment in which it is located. When the target change rate corresponding to a certain video frame is greater than or equal to the target change rate threshold, video segmentation processing is performed at the beginning of the video frame.

[0048] For example, the calculation of the target change rate between video frames and the video segmentation step can be performed alternately. Starting from the second video frame in the video to be processed, the target change rate between the current video frame and the first video frame in the video to be processed is calculated frame by frame, and each target change rate is compared with the target change rate threshold. If the target change rate of the current video frame is greater than or equal to the target change rate threshold, video segmentation is performed at the beginning of the current video frame; then the target change rate between each subsequent video frame and the first video frame in the continuous video after segmentation is calculated until the target change rate of the current video frame is greater than or equal to the target change rate threshold, and video segmentation is performed again; and so on, so that the content between each video frame in each video segment is similar, and the content difference between different video segments is large, thereby improving the retrieval efficiency.

[0049] As an example, according to the target recognition result, the implementation process of determining the target change rate between the previous and next video frames in the video to be processed can be as follows: Formula (1).

[0050] (1); in, represents the target change rate of the t+kth video frame relative to the tth video frame; Represents all targets in the tth video frame; represents all the targets in the t+kth video frame, where the tth video frame may be the first video frame in the continuous video segment where the t+kth video frame is located.

[0051] According to the video processing method for a multimodal large model of an embodiment of the present invention, the target change rate between the previous and next video frames in the video to be processed is determined according to the target recognition result, and the video to be processed is divided into multiple video segments based on the target change rate. The present invention performs video segmentation based on the target change rate, which can not only avoid the influence of factors such as environmental background, but also make the content of the same video segment after segmentation similar, and the content between different video segments is greatly different, thereby improving the calculation efficiency and the retrieval efficiency.

[0052] Figure 3 The flowchart of the video processing method for a multi-modal large model provided by an embodiment of the present invention is as follows. Figure 3As shown, based on the above embodiments, Figure 1 The implementation of step 103 may include: Step 301 : Based on the target recognition result, determine the first video frame in each video segment where a target object is newly added or disappears.

[0053] That is to say, for each video clip, based on the target recognition result, determine whether there are new targets or missing targets in each video frame in the video clip compared with the previous video frame. If there are new targets or missing targets in the i-th video frame compared with the i-1-th video frame, the i-th video frame is determined as the first video frame.

[0054] There may be multiple first video frames in each video segment.

[0055] Step 302: determining a key frame of each video segment based on the first video frame in each video segment.

[0056] It is understandable that in each video clip, there will be multiple consecutive video frames in which the target object has not changed. The content between these video frames does not change much, so they can be ignored. If a video frame has a new or missing target object compared to the previous video frame, it means that the video frame has a content change compared to the previous video frame. The video frame with a changed content can carry the key information in the video clip, so the key frame of each video clip can be determined based on the first video frame in each video clip.

[0057] In some embodiments, the implementation process of determining the key frame of each video segment based on the first video frame in each video segment may include: determining the first video frame in each video segment as the key frame of each video segment.

[0058] In some embodiments, in order to avoid losing key information in the video, key frame extraction may be continued for the video frames between the first video frames. The implementation process of determining the key frame of each video segment based on the first video frame in each video segment may include: determining the target displacement rate of other video frames between the first video frame in each video segment based on the target recognition result; determining the second video frame in each video segment based on the target displacement rate; and determining the first video frame and the second video frame in each video segment as the key frames of the corresponding video segment.

[0059] The target displacement rate of other video frames between the first video frames in each video clip may be the cumulative sum of displacements of the target object in each video frame between the two first video frames relative to the target object in the previous video frame, or the cumulative sum of displacements of the target object in each video frame between the two first video frames relative to the target object in the previous key frame. It can be understood that if the displacement of the target object in the previous and next video frames is large, it can be explained that the content in the previous and next video frames has changed greatly, so the key frame can be determined based on the displacement change rate of each video frame.

[0060] As an example, for each video clip, the target displacement rate of each video frame between the first video frames in the video clip relative to the previous video frame can be determined based on the target recognition result; the target displacement rate is compared with a target displacement rate threshold, and a video frame whose target displacement rate is greater than or equal to the target displacement rate threshold is determined as the second video frame; and the first video frame and the second video frame are determined as key frames.

[0061] As another example, the first video frame can be first determined as a key frame, and the target displacement rate can be calculated for each video frame between two first video frames; the target displacement rate of the video frame between the two first video frames relative to the previous key frame can be calculated one by one, and each target displacement rate can be compared with the target displacement rate threshold in turn; if the target displacement rate of a certain video frame is greater than or equal to the target displacement rate threshold, the video frame can be determined as a key frame, and the target displacement rate calculation for subsequent video frames relative to the key frame can be continued until all video frames have been traversed and all key frames have been obtained.

[0062] For example, based on the target recognition result, a calculation formula for determining the target displacement rate between the previous and next video frames in each video clip is shown in the following formula (2).

[0063] (2); in, is the target displacement rate of the t+kth video frame relative to the tth video frame; M is the number of identical targets in the t+kth video frame and the tth video frame; is the displacement of the i-th target object between the t+k-th video frame and the t-th video frame.

[0064] Step 303: determining a key frame sequence in each video segment based on the key frames of each video segment.

[0065] In some embodiments, the key frame sequence of each video segment includes a plurality of key frames. For each video segment, the key frames of the video segment can be combined into the key frame sequence of the video segment based on the key frames of the video segment.

[0066] According to the video processing method for a multimodal large model of an embodiment of the present invention, based on the target recognition result, the first video frame in which a target object appears or disappears in each video clip is determined; based on the first video frame in each video clip, the key frame of each video clip is determined; based on the key frame of each video clip, the key frame sequence in each video clip is determined. The present invention can determine the key frame by the change of the target object, so as to represent all the key information carried in the video to be processed based on the key frame sequence, which can improve the calculation efficiency on the one hand, and avoid the loss of key information on the other hand.

[0067] Figure 4 The fourth flow chart of the video processing method for a multimodal large model provided by an embodiment of the present invention. In the embodiment of the present invention, the feature information of each video clip includes text feature information, image feature information and fusion feature information. Figure 4 As shown, based on the above embodiments, Figure 1 The implementation process of step 105 in may include the following steps.

[0068] Step 401 : for a first video segment among multiple video segments, extract features from a first key frame sequence in the first video segment to obtain image feature information of the first video segment.

[0069] The first key frame sequence is a key frame sequence of a first video segment, and the first video segment may be any one of a plurality of video segments.

[0070] In some embodiments, feature extraction may be performed on each key frame in the first key frame sequence in the first video clip based on a convolutional neural network, and features extracted from each key frame may be fused to obtain image feature information of the first video clip.

[0071] In some embodiments, Figure 5 As shown, the process of extracting features from the first key frame sequence in the first video clip and obtaining image feature information of the first video clip may include: Step 501: Determine the number of each type of target object included in the first key frame sequence based on the target recognition result.

[0072] In some embodiments, the target recognition result includes targets at various positions in the video frame and the category to which each target belongs. Based on the target recognition result, the targets included in each key frame and the number of each type of target can be determined, and the number of each type of target included in the first key frame sequence can be obtained by counting.

[0073] Step 502 : extract features of each type of target in the first key frame sequence to obtain image feature information of each type of target.

[0074] In some embodiments, features can be extracted for each type of target in the first key frame sequence based on a convolutional neural network. Since there can be multiple targets of each type, multiple image feature maps of each type of target can be obtained, and then the multiple image feature maps of each type of target are fused to obtain image feature information of each target.

[0075] Step 503: Determine the image feature information of the first video clip based on the number of each type of target objects and the image feature information of each type of target objects.

[0076] In some embodiments, the number of each type of target object and the image feature information of each type of target object may be mapped into a two-dimensional vector, namely, the image feature information of the first video clip.

[0077] The embodiment of the present invention can ensure the integrity of key information in the video by obtaining the category and quantity of targets in each key frame sequence and extracting the image feature information of each type of target respectively, so that the obtained image feature information can comprehensively characterize the image features in the video clip, thereby improving the accuracy of video retrieval.

[0078] Step 402: extract features from the text information of the first video clip to obtain text feature information of the first video clip.

[0079] In some embodiments, the text information of the first video clip may be input into a pre-trained embedding model to convert the text information into a vector representation to obtain text feature information of the first video clip.

[0080] Step 403: fuse the image feature information and the text feature information to obtain fused feature information of the first video clip.

[0081] In some embodiments, the image feature information and the text feature information may be spliced ​​to achieve fusion of the two and obtain fused feature information of the first video clip.

[0082] In some embodiments, the image feature information of the video clip can be stored in the image feature sub-library of the feature library, the text feature information of the video clip can be stored in the text feature sub-library of the feature library, and the fusion feature information of the video clip can be stored in the fusion feature sub-library of the feature library. After obtaining the user input information, the modality of the user input information can be identified first. If the user input information is text information, the user input information can be vectorized, and then the target text feature information similar to the vector expression of the user input information can be determined from the text feature sub-library, and the target video segment corresponding to the target text feature information can be determined; if the user input information is image information, feature extraction can be performed on the user input information to obtain the vector expression, and target image feature information similar to the vector expression of the user input information can be determined from the image feature sub-library, and the target video segment corresponding to the target image feature information can be determined; if the user input information includes image information and text information, feature extraction can be performed on the user input information, and the extracted text features and image features can be fused to obtain the corresponding vector expression, and target fusion feature information similar to the vector expression of the user input information can be determined from the fusion feature sub-library, and the target video segment corresponding to the target fusion feature information can be determined, and finally the obtained target video segment is input into the multimodal large model.

[0083] According to the video processing method for a multimodal large model according to an embodiment of the present invention, the feature information of each video clip includes text feature information, image feature information and fusion feature information. Feature extraction is performed on the key frame sequence in each video clip to obtain the image feature information of each video clip; feature extraction is performed on the text information of each video clip to obtain the text feature information of each video clip; the text feature information and the image feature information of each video clip are fused to obtain the fusion feature information of each video clip, and the above feature information is stored in a feature library. This can be applicable to the retrieval of user input information of different modalities, and the feature information of the corresponding modality is matched based on the modality of the user input information, thereby improving the applicability of the feature library and the accuracy of the retrieval.

[0084] In order to implement the above embodiment, the present invention also provides a video retrieval enhancement method for a multimodal large model.

[0085] Figure 6 The flowchart of the multi-modal large model video retrieval enhancement method provided by the embodiment of the present invention is as follows. Figure 6 As shown, the method includes the following steps.

[0086] Step 601, obtaining user input information.

[0087] Step 602: vectorize the user input information, and search from the feature library based on the vectorized expression of the user input information to obtain the target video clip.

[0088] The feature library is obtained based on the video processing method for a multimodal large model according to the above embodiment.

[0089] In some embodiments, the feature library may include an image feature sub-library, a text feature sub-library, a fusion feature sub-library, and a video clip sub-library. The image feature sub-library stores image feature information of each video clip, the text feature sub-library stores text feature information of each video clip, and the fusion feature sub-library stores fusion feature sub-library of each video clip. When vectorizing the user input information, the modality of the user input information may be determined first, and vectorization may be performed in a corresponding manner based on the modality of the user input information, and a search may be performed from the feature sub-library of the corresponding modality to obtain similar feature information, and determine the target video clip corresponding to the similar feature information.

[0090] Step 603: input the user input information and the target video clip into the multimodal large model, so that the multimodal large model generates an answer based on the user input information and the target video clip.

[0091] Figure 7 The figure is a correlation diagram between the video processing method and the video retrieval enhancement method for the multimodal large model of the embodiment of the present invention. After the video to be processed is segmented, multiple video clips are obtained, and then key frame sequence extraction and text information extraction are performed on each video clip, and image feature information, text feature information and fusion feature information of each video clip are obtained through feature extraction and feature fusion, and stored in the feature library. When the user asks a question, the video can be retrieved based on the feature library based on the user input information, and the target video clip can be input into the multimodal large model so that the multimodal large model can generate an answer.

[0092] The following is a description of a video processing device for a multimodal large model provided in an embodiment of the present invention. The video processing device for a multimodal large model described below and the video processing method for a multimodal large model described above can be referenced to each other.

[0093] Figure 8 Schematic diagram of the structure of a video processing device for a multi-modal large model provided by an embodiment of the present invention. Figure 8 As shown, the device includes: a first acquisition module 810, a segmentation module 820, an extraction module 830, a second acquisition module 840 and a determination module 850.

[0094] The first acquisition module 810 is used to perform target recognition on the video to be processed and obtain the target recognition result of each video frame in the video to be processed; the segmentation module 820 is used to segment the video to be processed into multiple video segments according to the target recognition result; the extraction module 830 is used to extract the key frame sequence in each video segment; the second acquisition module 840 is used to obtain the text information corresponding to each video segment; the determination module 850 is used to determine the feature information of each video segment based on the key frame sequence and the text information, and store each video segment and its corresponding feature information in a feature library; wherein the feature library is used for video retrieval enhancement processing of multimodal large models.

[0095] In some embodiments, the segmentation module 820 is specifically used to: determine the target change rate between the previous and next video frames in the video to be processed according to the target recognition result; and segment the video to be processed into multiple video segments based on the target change rate.

[0096] In some embodiments, the extraction module 830 is specifically used to: determine the first video frame in each video clip where a target object appears or disappears based on the target recognition result; determine the key frame of each video clip based on the first video frame in each video clip; and determine the key frame sequence in each video clip based on the key frame of each video clip.

[0097] In some embodiments, the extraction module 830 is also used to: determine the target displacement rate between the first video frame and the previous and next video frames in each video segment based on the target recognition result; determine the second video frame in each video segment based on the target displacement rate; and determine the first video frame and the second video frame in each video segment as key frames of the corresponding video segment.

[0098] In some embodiments, the second acquisition module 840 is specifically used to: based on a speech recognition algorithm, obtain the first text information corresponding to the speech information in each video clip; based on a character recognition algorithm, recognize the text content in each video clip to obtain the second text information corresponding to each video clip; based on the first text information and the second text information, determine the text information corresponding to each video clip.

[0099] In some embodiments, the feature information of each video clip includes text feature information, image feature information and fusion feature information; the determination module 850 is specifically used to: for a first video clip among multiple video clips, perform feature extraction on a first key frame sequence in the first video clip to obtain image feature information of the first video clip; perform feature extraction on the text information of the first video clip to obtain text feature information of the first video clip; fuse the image feature information and the text feature information to obtain fusion feature information of the first video clip.

[0100] In some embodiments, the determination module 850 is also used to: determine the number of each type of target objects included in the first key frame sequence based on the target recognition result; perform feature extraction on each type of target object in the first key frame sequence to obtain image feature information of each type of target object; determine the image feature information of the first video clip based on the number of each type of target objects and the image feature information of each type of target objects.

[0101] It should be noted that the above explanations and descriptions in the embodiment of the video processing method for a multimodal large model are also applicable to the video processing device for a multimodal large model in the embodiment of the present invention, and will not be repeated here.

[0102] Fig. 9 The schematic diagram of the structure of the video retrieval enhancement device for multi-modal large models provided by the embodiment of the present invention. Fig. 9 As shown, the device includes a third acquisition module 910, a retrieval module 920 and an input module 930. The third acquisition module 910 is used to obtain user input information; the retrieval module 920 is used to vectorize the user input information, and based on the vectorized expression of the user input information, search from the feature library to obtain the target video clip; wherein the feature library is obtained based on the video processing method for the multimodal large model of the above embodiment; the input module 930 is used to input the user input information and the target video clip into the multimodal large model, so that the multimodal large model generates an answer based on the user input information and the target video clip.

[0103] It should be noted that the above explanations and descriptions in the embodiment of the video retrieval enhancement method for a multimodal large model are also applicable to the video retrieval enhancement device for a multimodal large model in the embodiment of the present invention, and will not be repeated here.

[0104] Fig.10 An example of a physical structure diagram of an electronic device is shown in FIG. Fig.10 As shown, the electronic device may include: a processor 1010, a communication interface 1020, a memory 1030 and a communication bus 1040, wherein the processor 1010, the communication interface 1020 and the memory 1030 communicate with each other through the communication bus 1040. The processor 1010 may call the computer program in the memory 1030 to execute the steps of the video processing method for a multimodal large model provided in the above embodiments, and / or execute the steps of the video retrieval enhancement method for a multimodal large model provided in the above embodiments.

[0105] For example, a video processing method for a multimodal large model includes: performing target recognition on a video to be processed, and obtaining target recognition results for each video frame in the video to be processed; according to the target recognition results, dividing the video to be processed into multiple video segments; extracting a key frame sequence in each video segment; obtaining text information corresponding to each video segment; determining feature information of each video segment based on the key frame sequence and the text information, and storing each video segment and its corresponding feature information in a feature library; wherein the feature library is used for video retrieval enhancement processing of a multimodal large model.

[0106] For example, a video retrieval enhancement method for a multimodal large model includes: obtaining user input information; vectorizing the user input information, and based on the vectorized expression of the user input information, searching from a feature library to obtain a target video clip; wherein the feature library is obtained based on a video processing method for a multimodal large model; inputting the user input information and the target video clip into the multimodal large model, so that the multimodal large model generates an answer based on the user input information and the target video clip.

[0107] In addition, the logic instructions in the above-mentioned memory 1030 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0108] On the other hand, an embodiment of the present invention further provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the steps of the video processing method for a multimodal large model provided in the above embodiments, and / or execute the steps of the video retrieval enhancement method for a multimodal large model provided in the above embodiments.

[0109] For example, a video processing method for a multimodal large model includes: performing target recognition on a video to be processed, and obtaining target recognition results for each video frame in the video to be processed; according to the target recognition results, dividing the video to be processed into multiple video segments; extracting a key frame sequence in each video segment; obtaining text information corresponding to each video segment; determining feature information of each video segment based on the key frame sequence and the text information, and storing each video segment and its corresponding feature information in a feature library; wherein the feature library is used for video retrieval enhancement processing of a multimodal large model.

[0110] For example, a video retrieval enhancement method for a multimodal large model includes: obtaining user input information; vectorizing the user input information, and based on the vectorized expression of the user input information, searching from a feature library to obtain a target video clip; wherein the feature library is obtained based on a video processing method for a multimodal large model; inputting the user input information and the target video clip into the multimodal large model, so that the multimodal large model generates an answer based on the user input information and the target video clip.

[0111] On the other hand, an embodiment of the present invention further provides a processor-readable storage medium, wherein the processor-readable storage medium stores a computer program, wherein the computer program is used to enable the processor to execute the steps of the video processing method for a multimodal large model provided in the above-mentioned embodiments, and / or execute the steps of the video retrieval enhancement method for a multimodal large model provided in the above-mentioned embodiments.

[0112] For example, a video processing method for a multimodal large model includes: performing target recognition on a video to be processed, and obtaining target recognition results for each video frame in the video to be processed; according to the target recognition results, dividing the video to be processed into multiple video segments; extracting a key frame sequence in each video segment; obtaining text information corresponding to each video segment; determining feature information of each video segment based on the key frame sequence and the text information, and storing each video segment and its corresponding feature information in a feature library; wherein the feature library is used for video retrieval enhancement processing of a multimodal large model.

[0113] For example, a video retrieval enhancement method for a multimodal large model includes: obtaining user input information; vectorizing the user input information, and based on the vectorized expression of the user input information, searching from a feature library to obtain a target video clip; wherein the feature library is obtained based on a video processing method for a multimodal large model; inputting the user input information and the target video clip into the multimodal large model, so that the multimodal large model generates an answer based on the user input information and the target video clip.

[0114] The processor-readable storage medium can be any available medium or data storage device that can be accessed by the processor, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NANDFLASH), solid-state drive (SSD)), etc.

[0115] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0116] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video processing method for a multimodal large model, characterized in that: include: Performing target recognition on the video to be processed, and obtaining target recognition results for each video frame in the video to be processed; According to the target recognition result, the video to be processed is divided into multiple video segments; Extracting a key frame sequence from each of the video clips; Obtaining text information corresponding to each of the video clips; Based on the key frame sequence and the text information, the feature information of each of the video clips is determined, and each of the video clips and its corresponding feature information are stored in a feature library; wherein the feature library is used for video retrieval enhancement processing of the multimodal large model.

2. The method according to claim 1, characterized in that The step of dividing the video to be processed into a plurality of video segments according to the target recognition result includes: Determining the target change rate between the previous and next video frames in the video to be processed according to the target recognition result; Based on the target change rate, the video to be processed is divided into the multiple video segments.

3. The method according to claim 1, characterized in that The step of extracting a key frame sequence from each of the video clips comprises: Based on the target recognition result, determining a first video frame in each of the video clips in which a target object is newly added or a target object disappears; Determining a key frame of each of the video segments based on a first video frame in each of the video segments; Based on the key frames of each of the video segments, a key frame sequence in each of the video segments is determined.

4. The method according to claim 3, characterized in that The step of determining a key frame of each of the video segments based on the first video frame in each of the video segments comprises: Determining target displacement rates of other video frames between the first video frame in each of the video segments based on the target recognition result; Based on the target displacement rate, determining a second video frame in each of the video segments; The first video frame and the second video frame in each of the video segments are determined as key frames of the corresponding video segment.

5. The method according to claim 1, characterized in that The obtaining of text information corresponding to each of the video clips includes: Based on a speech recognition algorithm, obtaining first text information corresponding to the speech information in each of the video clips; Based on a character recognition algorithm, the text content in each of the video clips is recognized to obtain second text information corresponding to each of the video clips; Based on the first text information and the second text information, text information corresponding to each of the video clips is determined.

6. The method according to claim 1, characterized in that The feature information of each video clip includes text feature information, image feature information and fusion feature information; The determining, based on the key frame sequence and the text information, feature information of each of the video clips includes: For a first video segment among the multiple video segments, extract features of a first key frame sequence in the first video segment to obtain image feature information of the first video segment; Extracting features from the text information of the first video clip to obtain text feature information of the first video clip; The image feature information and the text feature information are fused to obtain fused feature information of the first video clip.

7. The method according to claim 6, characterized in that The step of extracting features from the first key frame sequence in the first video clip to obtain image feature information of the first video clip includes: Based on the target recognition result, determining the number of each type of target objects included in the first key frame sequence; Extracting features of each type of target object in the first key frame sequence respectively to obtain image feature information of each type of target object; Based on the number of each type of target object and the image feature information of each type of target object, the image feature information of the first video clip is determined.

8. A video retrieval enhancement method for a multimodal large model, characterized in that: include: Get user input information; Vectorizing the user input information, and based on the vectorized expression of the user input information, searching from a feature library to obtain a target video clip; wherein the feature library is obtained based on the video processing method for a multimodal large model according to any one of claims 1 to 7 above; The user input information and the target video clip are input into the multimodal large model, so that the multimodal large model generates an answer based on the user input information and the target video clip.

9. A video processing device for a multimodal large model, characterized in that: include: The first acquisition module is used to perform target recognition on the video to be processed and obtain the target recognition result of each video frame in the video to be processed; A segmentation module, used for segmenting the to-be-processed video into a plurality of video segments according to the target recognition result; An extraction module, used for extracting a key frame sequence in each of the video clips; A second acquisition module, used to acquire text information corresponding to each of the video clips; A determination module is used to determine the feature information of each of the video clips based on the key frame sequence and the text information, and store each of the video clips and its corresponding feature information in a feature library; wherein the feature library is used for video retrieval enhancement processing of the multimodal large model.

10. A video retrieval enhancement device for a multimodal large model, characterized in that: include: The third acquisition module is used to obtain user input information; A retrieval module, used to vectorize the user input information, and based on the vectorized expression of the user input information, search from a feature library to obtain a target video clip; wherein the feature library is obtained based on the video processing method for a multimodal large model according to any one of claims 1 to 7 above; An input module is used to input the user input information and the target video clip into the multimodal large model, so that the multimodal large model generates an answer based on the user input information and the target video clip.

11. An electronic device comprising a processor and a memory storing a computer program, characterized in that: When the processor executes the computer program, it implements the video processing method for a multimodal large model as described in any one of claims 1 to 7, and / or implements the video retrieval enhancement method for a multimodal large model as described in claim 8.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the video processing method for a multimodal large model as described in any one of claims 1 to 7, and / or implements the video retrieval enhancement method for a multimodal large model as described in claim 8.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the video processing method for a multimodal large model as described in any one of claims 1 to 7, and / or implements the video retrieval enhancement method for a multimodal large model as described in claim 8.

Images