Video processing method and device based on multi-modal information fusion, equipment and medium

Abstract

The invention provides a video processing method and device based on multi-modal information fusion, equipment and a medium, and relates to the field of video processing. According to the method, the voice text in the video is processed through the large language model, the voice information can be accurately recognized and transcribed, the unstructured voice data is converted into the structured text data, and operable text information is provided for subsequent analysis. The image key frames are extracted by using the image features of the video data and the text features of the obtained voice recognition text, the most representative frame with the maximum information amount can be screened out from a large number of video frames, redundant information is reduced, and meanwhile, the synchronism of visual information and voice information is ensured. Based on the time correspondence, the image key frame and the speech recognition text are aligned and fused, and the multi-modal fusion text is output, so that the time sequence association and context information of the video can be captured more accurately, and the accuracy and depth of video understanding are improved.

Description

Technical Field

[0001] The present application relates to the field of video processing, and in particular to a video processing method, device, equipment and medium based on multimodal information fusion. Background Art

[0002] With the explosive growth of multimedia video data in recent years, understanding, retrieving and analyzing these massive multimedia video data has become an important direction for various research institutions and companies. Video understanding is a preliminary step in this process.

[0003] Existing video understanding technologies include video classification, video content description, video question and answer, etc. They receive a video or additional text as input and output text or specific content. Unlike understanding images and text, video data has the characteristics of large data volume, high data redundancy, multi-modal data, and temporal correlation between frames, which greatly increases the difficulty of understanding and analysis.

[0004] Therefore, how to improve the accuracy of video understanding and analysis has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The present application provides a video processing method, apparatus, device and medium based on multimodal information fusion, aiming to improve the accuracy of video understanding and analysis.

[0006] In a first aspect, the present application provides a video processing method based on multimodal information fusion, the video processing method based on multimodal information fusion comprising the following steps:

[0007] Based on the large language model, performing text processing on the speech text in the video data to obtain a first speech recognition text;

[0008] Extracting image key frames in the video data based on image features of the video data and text features of the first speech recognition text;

[0009] Based on the time correspondence, the image key frame and the first speech recognition text are aligned and fused to output a multimodal fused text.

[0010] In a second aspect, the present application further provides a video processing device based on multimodal information fusion, the video processing device based on multimodal information fusion comprising:

[0011] A speech recognition module, used for performing text processing on the speech text in the video data based on the large language model to obtain a first speech recognition text;

[0012] An image key frame extraction module, configured to extract image key frames from the video data based on image features of the video data and text features of the first speech recognition text;

[0013] The multimodal fusion module is used to align and fuse the image key frame and the first speech recognition text based on the time correspondence, and output the multimodal fusion text.

[0014] In a third aspect, the present application also provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the video processing method based on multimodal information fusion as described above are implemented.

[0015] In a fourth aspect, the present application also provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the video processing method based on multimodal information fusion as described above are implemented.

[0016] The present application provides a video processing method, device, computer equipment and storage medium based on multimodal information fusion. The method of the present application includes text processing of the speech text in the video data based on a large language model to obtain a first speech recognition text; extracting the image key frame in the video data based on the image features of the video data and the text features of the first speech recognition text; aligning and fusing the image key frame and the first speech recognition text based on the time correspondence, and outputting the multimodal fusion text. In the above manner, the present application processes the speech text in the video through a large language model, can accurately identify and transcribe the speech information, convert the unstructured speech data into structured text data, and provide operable text information for subsequent analysis. By extracting the image key frame using the image features of the video data and the text features of the obtained speech recognition text, the most representative and informative frames can be screened out from a large number of video frames, reducing redundant information, while ensuring the synchronization of visual information and speech information. Based on the time correspondence, the image key frames and speech recognition texts are aligned and fused to output multimodal fused text, which can combine the visual information and auditory information of the video to form a more comprehensive and rich video content representation, and can more accurately capture the temporal correlation and contextual information of the video, thereby improving the accuracy and depth of video understanding. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 A schematic diagram of a flow chart of a first embodiment of a video processing method based on multimodal information fusion provided by the present application;

[0019] Figure 2 A schematic diagram of the process of voice information recognition in video data provided by an embodiment of the present application;

[0020] Figure 3 A schematic diagram of the process of extracting image key frames provided in an embodiment of the present application;

[0021] Figure 4 It is a structural schematic diagram of a first embodiment of a video processing device based on multimodal information fusion provided by the present application;

[0022] Figure 5 It is a schematic block diagram of the structure of a computer device provided in an embodiment of the present application.

[0023] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0025] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0026] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0027] Please refer to Figure 1 , Figure 1 A schematic flow chart of a first embodiment of a video processing method based on multimodal information fusion provided in the present application.

[0028] like Figure 1 As shown, the video processing method based on multimodal information fusion includes steps S101 to S103.

[0029] S101, performing text processing on the speech text in the video data based on the large language model to obtain a first speech recognition text;

[0030] Generally speaking, existing systems usually directly obtain the text description of the video based on the video image content, but such solutions can only output a rough description, losing a lot of detailed content and voice modality information.

[0031] In this embodiment, speech recognition is first performed on the video to extract the text information of the speaker's speech content. Then, the speech recognition results are sent to the large language model to adjust and segment the recognition results. Using the large language model for adjustment can improve the speech recognition results, obtain a more detailed text description of the video, and can also process videos of unlimited length.

[0032] Optionally, during the speech recognition process, the speech recognition result is optimized, for example, by improving the quality of the speech signal through a denoising module, or by using speaker recognition technology to distinguish the voices of multiple people, thereby improving the accuracy of speech recognition.

[0033] Furthermore, based on a speech recognition algorithm, speech information in the video data is identified to obtain a second speech recognition text; based on the large language model, text error correction and text segmentation processing are performed on the second speech recognition text to obtain the first speech recognition text.

[0034] In one embodiment, a speech recognition algorithm is used to identify and extract audio signals from video data and convert them into text form. The speech recognition algorithm can be speech recognition technologies such as hidden Markov model HMM, deep full sequence convolutional neural network DFCNN, streaming multi-level truncated attention model SMLTA, etc.

[0035] In one embodiment, the recognized text is corrected by a large language model. These models can be optimized for specific language errors through supervised fine-tuning. For example, GrammarGPT is an open source large language model that specifically corrects Chinese grammatical errors. It improves the model's ability to correct grammatical errors by building a mixed dataset containing manual annotations and automatic generation, and using an error-invariant data enhancement method.

[0036] In one embodiment, after completing the text correction, a large language model (such as a model structure such as Cross-segmentBERT, BERT+Bi-LSTM or Hierarchical BERT) can be used to use local or complete context information to determine the segmentation boundaries of the text, and perform text segmentation processing on the second speech recognition text to improve the readability of the text.

[0037] In one embodiment, based on a user identity recognition algorithm, speaker information corresponding to each voice information in the video data is identified; based on the speaker information, each voice text in the second voice recognition text is text-marked to obtain a corresponding relationship between each voice text and the speaker information.

[0038] In one embodiment, if Figure 2 As shown, based on speech recognition, denoising, speaker recognition and other modules can be added according to actual application scenarios to improve the accuracy of speech recognition.

[0039] The second speech recognition text includes the speaker's modal particles, erroneous recognition results, and the recognition results that fail to be organized into segments.

[0040] In one embodiment, a voice profile of a speaker can be configured. Audio samples of the speaker are collected, and these audio samples need to cover different voice features to ensure the accuracy of the voice profile. After the profile is created, the speaker verification function can be used to confirm the identity of the speaker. If it is necessary to process multi-person conversations, a multi-person conversation speech recognition model can be used. Based on voice biometric technology, the unique voice features of the speaker are analyzed to determine the identity information of the speaker for each speech segment.

[0041] In one embodiment, after the speaker is identified, each speech text in the second speech recognition text is marked to establish a corresponding relationship between the speech text and the speaker information.

[0042] This embodiment can ensure more accurate organization of video content by identifying speaker identity information, so that each sentence in the voice text of video data can be correctly attributed to the corresponding speaker, especially for scenes such as conference records and interviews, which can improve the effectiveness of video data organization. According to the speaker identity information, the second voice recognition text is marked, and a specific voice segment can be searched according to the speaker, and then the corresponding video segment can be retrieved, making the video content easier to search and realizing rapid positioning of the video content.

[0043] S102, extracting image key frames in the video data based on image features of the video data and text features of the first speech recognition text;

[0044] It is understandable that in addition to voice and text information, images in videos also contain rich information. However, image frames in videos often have high redundancy. Conventional methods directly perform uniform or random frame sampling on the video for analysis and processing, which may cause some key information to be lost, and may also result in the extraction of lower-quality image frames. The present application proposes a key frame extraction scheme that combines voice, text, and image features: 1) In terms of images, frames with large differences between previous and subsequent frames are selected as candidate key frames based on image features. 2) In terms of text, the corresponding video timestamp is located based on the key information in the text, and the corresponding video frames are extracted as candidate key frames.

[0045] Specifically, Figure 3 As shown in the figure, in terms of images, the difference between the previous and next frames can be obtained based on the optical flow and image structural similarity features SSIM, or other image features can be used. In order to further improve the processing speed and reduce the impact of small changes in the image between frames on feature calculation, the image can be scaled to a lower resolution for processing. In terms of text, on the one hand, a large language model is used to extract text keywords, and on the other hand, the text is segmented, the video frames are clustered according to the number of segments, and the video frames closest to the cluster center are added to the candidate key frames. For example, CLIP can be used to obtain video frame features, and then DPC-KNN and k-means can be used simultaneously to obtain video frame clusters. After obtaining the key frames that combine image and text features, the key frames with low quality and high similarity are further removed to filter out high-quality key frames in the video.

[0046] Exemplarily, image processing techniques such as edge detection, color histogram analysis, texture analysis, etc. are used to extract features from each image frame in the video. A deep learning model such as CNN is used to extract features from image frames to obtain a deep feature representation. Through image feature extraction, the visual content of each frame is determined, and potential key frames are identified, i.e., those frames with significant visual changes or containing important visual information.

[0047] Exemplarily, the first speech recognition text is subjected to text feature extraction, such as keyword extraction, semantic analysis, sentiment analysis, etc. Based on the text features, key moments in the text are identified, such as semantic turning points, emotional peaks, etc. Combined with the features of the speech recognition text, candidate key frames corresponding to the key moments of the text are screened out.

[0048] Exemplarily, the candidate key frames screened based on image features are compared with the candidate key frames screened based on speech recognition text features. The candidate key frames obtained by the two methods are compared to find common frames, which may be the real key frames in the video. For the candidate key frames that appear in both the image feature and text feature screening results, a temporal proximity analysis is performed to remove duplicate frames that are too close in time. Frames that have significant changes in both vision and text or contain important information are retained to ensure the representativeness and diversity of the key frame sequence. The screened candidate key frames are arranged in the time sequence of the video frames to form the final image key frame sequence. This sequence can represent the main content of the video, capture important moments in the video, and reduce redundant information.

[0049] As a result, image key frame sequences can be effectively extracted from video data. These key frames not only capture the main content of the video visually, but also match the speech recognition content textually, providing strong support for the analysis, retrieval and summarization of video content.

[0050] Furthermore, based on an image feature recognition algorithm, feature differences among image frames in the video data are identified to obtain at least one first candidate key frame; based on text segmentation of the first speech recognition text, video frames in the video data are clustered to obtain at least one second candidate key frame; based on a preset key frame extraction algorithm, the first candidate key frame and the second candidate key frame are fused and extracted to obtain the image key frame.

[0051] In one embodiment, in terms of images, the frames with the largest difference between the previous and next frames are selected as candidate key frames based on image features. By comparing the matching degree of feature points between consecutive frames, significant differences between frames are identified, which may indicate scene changes, motion, or the appearance and disappearance of objects. Based on these feature differences, at least one first candidate key frame is screened out, which represents significant visual change points in the video.

[0052] Exemplarily, the difference between the previous and next frames may be obtained by using methods based on optical flow and image structural similarity features SSIM, or other image features may be used.

[0053] Among them, the optical flow method is a method for estimating the motion information of objects in an image. It estimates the speed and direction of motion by tracking feature points or pixels in the image. The basic assumptions of the optical flow method are that the brightness is constant, that is, the brightness of the same object does not change when it moves between different frames; and the displacement between adjacent frames is relatively small. This method can effectively capture dynamic changes in the video and identify key frames.

[0054] SSIM (Structure Similarity Index Measure) is a measure of the similarity between two images. It evaluates image quality by comparing the brightness, contrast, and structural information of the images. The value of the SSIM index ranges from -1 to +1, where a value of +1 indicates that the two images are very similar or identical, and a value of -1 indicates that the two images are very different. In key frame extraction, frames with significant visual changes can be identified by calculating the SSIM value between consecutive frames.

[0055] It is understandable that in addition to optical flow and SSIM, other image features can also be considered, such as histogram features, edge features, texture features, etc. These features can describe the image content from different perspectives and help to more comprehensively evaluate the difference between frames.

[0056] In one embodiment, in terms of text, the corresponding video timestamp is located according to the key information in the text, and the corresponding video frame is extracted as a candidate key frame. The text is segmented using a large language model, and the video frames are clustered according to the number of segments, and the video frame closest to the cluster center is added to the candidate key frame.

[0057] Specifically, the first speech recognition text is segmented to identify semantic breakpoints, such as the end of a sentence, a change of topic, etc. Natural language processing techniques, such as the recognition of punctuation marks such as periods, question marks, exclamation marks, and semantic segmentation models, are used to segment the text. According to the results of the text segmentation, the video frames are matched with the text segments to determine the video frames corresponding to the text changes, and these frames are used as the second candidate key frames. Based on the text segmentation results of the first speech recognition text, the video frames are clustered and analyzed, and the frames corresponding to the same text segment are clustered together. Clustering algorithms, such as K-means, hierarchical clustering, etc., are used to group the video frames, and each group represents a candidate set of key frames.

[0058] In one embodiment, CLIP can be used to obtain video frame features, and then DPC-KNN and k-means can be used simultaneously to obtain video frame clustering. CLIP (Contrastive Language-Image Pre-training) is a visual-language model that can learn rich visual concept representations and is used to extract features from video frames. DPC is a clustering algorithm based on density peaks, which can automatically discover cluster centers and achieve efficient clustering of data of arbitrary shapes. k-means is a classic clustering algorithm that groups video frames by iteratively optimizing cluster centers.

[0059] Exemplarily, the first speech recognition text of the video is deeply analyzed using a large language model to extract key information and themes. Through text analysis, important events, actions or objects in the text are determined, which usually correspond to key visual content in the video. The text is segmented, and each paragraph represents a logical part or event in the video. This can be achieved through the text segmentation capability of the large language model, which can identify natural breakpoints in the text, such as the end of a sentence, the transition of a topic, etc. According to the results of the text segmentation, the video frames are matched with the text paragraphs, and each segment corresponds to a piece of video content. The feature representation of the video frame is obtained using the CLIP model. Clustering algorithms such as DPC-KNN (Density Peak Clustering-K-Nearest Neighbors) and k-means are used to cluster the video frames. According to the clustering results, the frames closest to the center in each cluster are selected as candidate key frames. These frames represent the most typical content of each segment of the video. For each text segment, the corresponding candidate key frames are selected, which are most visually representative of the content of the text segment.

[0060] In one embodiment, based on a preset resolution, each of the image frames is scaled to obtain a scaled image frame; based on the image feature recognition algorithm, the feature similarity between each consecutive scaled image frame and the anchor image frame is calculated; when the feature similarity between the current scaled image frame and the anchor image frame is less than a preset similarity threshold, the current scaled image frame is determined to be the first candidate key frame, and the current scaled image frame is determined to be the anchor image frame for screening the next first candidate key frame.

[0061] In one embodiment, in order to increase the processing speed and reduce the impact of small changes on feature calculation, the image can be scaled to a lower resolution for processing. For example, the longest side of the image is scaled to 112 pixels, thereby reducing the amount of calculation, speeding up the processing speed, and maintaining the relative consistency of the features. Use image feature recognition algorithms, such as SIFT, SURF, ORB, etc., to extract feature points of continuously scaled image frames and calculate their feature descriptors. Select an anchor image frame as a reference, and then calculate the feature similarity between the current scaled image frame and the anchor image frame. This can be achieved by matching feature descriptors and calculating the distance between them. Common distance metrics include Euclidean distance or Hamming distance.

[0062] In one embodiment, a preset similarity threshold may be set to determine whether the similarity between the current zoomed image frame and the anchor image frame is small enough. If the feature similarity between the current zoomed image frame and the anchor image frame is less than the preset similarity threshold, this indicates that the current frame and the anchor frame are visually significantly different.

[0063] Exemplarily, when the similarity is less than a threshold, the current zoomed image frame is determined to be the first candidate key frame. This means that the current frame captures different visual content from the previous key frame and is therefore considered to be an important moment in the video. At the same time, the current zoomed image frame is set as a new anchor image frame for similarity comparison with the next frame. Repeat the above steps to process the remaining frames of the video, iteratively screening out all first candidate key frames until all frames of the video are processed.

[0064] Generally, the key frame selection strategy can be considered based on the correlation between the number of feature points in each image frame and the previous key frame. There needs to be a certain number of identical or similar features between two adjacent key frames, but there cannot be too many repeated features, so as to ensure that the distribution between key frames is neither too dense nor too sparse, thereby ensuring the integrity and continuity of the video information while ensuring the amount of data is as small as possible. Specifically, the key frame selection strategy can be configured according to actual needs and the image frame information distribution of the video data. For example, if the video data is a continuous operation process, it is necessary to ensure that enough key frames are collected for each step of the operation process so that the operation process appears complete and continuous; however, if the user's demand is only for a certain operation step, the number of key frames for other steps can be appropriately reduced, while the number of key frames for the operation step required by the user can be increased.

[0065] Furthermore, based on the time sequence, each of the first candidate key frames and the second candidate key frames are fused to obtain a fused candidate key frame sequence; based on the key frame extraction algorithm, each candidate key frame in the fused candidate key frame sequence is deduplicated to obtain a target key frame sequence, wherein the target key frame sequence includes at least one frame of the image key frame.

[0066] In one embodiment, a fusion extraction is performed by combining the first candidate key frame (based on image feature differences) and the second candidate key frame (based on speech recognition text segmentation). Exemplarily, a preset key frame extraction algorithm, such as a content-based extraction algorithm (e.g., calculating inter-frame differences, histogram comparison, etc.), a deep learning-based method (e.g., using CNN to score the importance of frames), or other heuristic methods, can be used to comprehensively evaluate the two sets of candidate frames. Based on the results of the fusion extraction, the final image key frames are determined. These frames visually capture the main content of the video and textually correspond to the key moments of the speech recognition content.

[0067] Optionally, during the fusion process, a weighting mechanism can be designed to determine the relative importance of image features and text features in the final decision.

[0068] Specifically, all first candidate key frames (key frames extracted based on image features) and second candidate key frames (key frames extracted based on text features) are sorted in the chronological order of their appearance in the video. These candidate frames are merged into a single list to form a fused candidate key frame sequence. This sequence contains all potential key frames in the video. The similarity between frames can be calculated based on pixel comparison, feature matching, or deep learning models, and those frames with too high similarity can be removed to achieve deduplication of the fused candidate key frame sequence. After deduplication, the remaining frames constitute the target key frame sequence.

[0069] This embodiment selects key frames in two ways and fuses the two selected key frame sequences, so as to ensure that high-quality key frames are extracted from the video. These key frames are not only visually representative but also cover all important parts of the video.

[0070] S103: Based on the time correspondence, align and fuse the image key frame and the first speech recognition text, and output a multimodal fused text.

[0071] In general, existing video understanding systems only output text descriptions of videos or key frames in videos, with a single output modality, and do not combine text and key frames to output graphic documents. The embodiment of the present application aligns the segmented text (first speech recognition text) and image key frames obtained above according to the timestamps and text timestamps corresponding to the key frames, inserts key frame images at the end of the text paragraphs at the aligned position, and organizes them into a document output containing graphics and text. And based on the extracted key frame images, the image-text multimodal large model can be further used to form a description of the image as an image title.

[0072] In this embodiment, speech recognition is performed through the video speech content, and the recognition results are corrected and segmented using a large language model, so that a detailed text description of the video can be obtained; the video key frames are obtained by combining the speech text information and image information, so that the key information in the video image can be obtained to the greatest extent, and the omission of key frames caused by only considering a single mode or random selection can be prevented. According to the timestamps of the text and image, the recognized text and key frames are organized into graphic documents for output, which can effectively align and fuse the image key frames and the speech recognition text, and output a multimodal fusion text with rich content and diverse forms, providing users with more comprehensive and in-depth information expression.

[0073] This embodiment provides a video processing method based on multimodal information fusion. The method processes the speech text in the video through a large language model, can accurately identify and transcribe the speech information, convert the unstructured speech data into structured text data, and provide actionable text information for subsequent analysis. By extracting image key frames using the image features of the video data and the text features of the acquired speech recognition text, the most representative and informative frames can be screened out from a large number of video frames, reducing redundant information while ensuring the synchronization of visual information and speech information. Based on the time correspondence, the image key frames and the speech recognition text are aligned and fused, and the multimodal fusion text is output, which can combine the visual information and auditory information of the video to form a more comprehensive and rich video content representation, which can more accurately capture the temporal association and contextual information of the video, thereby improving the accuracy and depth of video understanding.

[0074] See also Figure 4 , Figure 4 It is a structural schematic diagram of a first embodiment of a video processing device based on multimodal information fusion provided in the present application. The video processing device based on multimodal information fusion is used to execute the aforementioned video processing method based on multimodal information fusion.

[0075] like Figure 4 As shown, the video processing device 200 based on multimodal information fusion includes: a speech recognition module 201, an image key frame extraction module 202 and a multimodal fusion module 203.

[0076] The speech recognition module 201 is used to perform text processing on the speech text in the video data based on the large language model to obtain a first speech recognition text;

[0077] An image key frame extraction module 202, configured to extract image key frames from the video data based on image features of the video data and text features of the first speech recognition text;

[0078] The multimodal fusion module 203 is used to align and fuse the image key frame and the first speech recognition text based on the time correspondence, and output a multimodal fusion text.

[0079] In one embodiment, the speech recognition module 201 includes:

[0080] A speech recognition unit, configured to recognize speech information in the video data based on a speech recognition algorithm, and obtain a second speech recognition text;

[0081] A text error correction unit is used to perform text error correction and text segmentation processing on the second speech recognition text based on the large language model to obtain the first speech recognition text.

[0082] In one embodiment, the speech recognition module 201 further includes:

[0083] A speaker information identification unit, used to identify speaker information corresponding to each voice information in the video data based on a user identity recognition algorithm;

[0084] The text marking unit is used to perform text marking on each speech text in the second speech recognition text based on the speaker information, so as to obtain the corresponding relationship between each speech text and the speaker information.

[0085] In one embodiment, the second speech recognition text includes modal particles, erroneous recognition results, and recognition results that fail to be organized into segments.

[0086] In one embodiment, the image key frame extraction module 202 includes:

[0087] A feature difference recognition unit, configured to recognize feature differences of image frames in the video data based on an image feature recognition algorithm, and obtain at least one first candidate key frame;

[0088] a text clustering unit, configured to cluster video frames in the video data based on the text segmentation of the first speech recognition text to obtain at least one second candidate key frame;

[0089] A key frame extraction unit is used to perform fusion extraction on the first candidate key frame and the second candidate key frame based on a preset key frame extraction algorithm to obtain the image key frame.

[0090] In one embodiment, the feature difference recognition unit includes:

[0091] An image scaling subunit, configured to scale each of the image frames based on a preset resolution to obtain a scaled image frame;

[0092] A feature similarity calculation subunit, used for calculating the feature similarity between each of the continuous zoomed image frames and the anchor point image frame based on the image feature recognition algorithm;

[0093] The first candidate key frame screening subunit is used to determine that the current zoomed image frame is the first candidate key frame when the feature similarity between the current zoomed image frame and the anchor image frame is less than a preset similarity threshold, and to determine that the current zoomed image frame is the anchor image frame for screening the next first candidate key frame.

[0094] In one embodiment, the key frame extraction unit includes:

[0095] A key frame fusion subunit, configured to fuse each of the first candidate key frames and the second candidate key frames based on a time sequence to obtain a fused candidate key frame sequence;

[0096] The key frame screening subunit is used to deduplicate each candidate key frame in the fused candidate key frame sequence based on the key frame extraction algorithm to obtain a target key frame sequence, wherein the target key frame sequence includes at least one frame of the image key frame.

[0097] It should be noted that technicians in the relevant field can clearly understand that for the convenience and conciseness of description, the specific working process of the above-described device and each module can refer to the corresponding process in the aforementioned video processing method embodiment based on multimodal information fusion, and will not be repeated here.

[0098] The apparatus provided in the above embodiment may be implemented in the form of a computer program. The computer program may be Figure 5 Runs on the computer device shown.

[0099] See also Figure 5 , Figure 5 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device may be a server.

[0100] See also Figure 5 The computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0101] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any video processing method based on multimodal information fusion.

[0102] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0103] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any video processing method based on multimodal information fusion.

[0104] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0105] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0106] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0107] Based on the large language model, performing text processing on the speech text in the video data to obtain a first speech recognition text;

[0108] Extracting image key frames in the video data based on image features of the video data and text features of the first speech recognition text;

[0109] Based on the time correspondence, the image key frame and the first speech recognition text are aligned and fused to output a multimodal fused text.

[0110] In one embodiment, when the processor performs text processing on the speech text in the video data based on the large language model to obtain the first speech recognition text, it is used to implement:

[0111] Based on a speech recognition algorithm, recognize the speech information in the video data to obtain a second speech recognition text;

[0112] Based on the large language model, text error correction and text segmentation processing are performed on the second speech recognition text to obtain the first speech recognition text.

[0113] In one embodiment, before the processor performs text error correction and text segmentation processing on the second speech recognition text based on the large language model to obtain the first speech recognition text, it is further used to implement:

[0114] Based on a user identification algorithm, identifying speaker information corresponding to each voice information in the video data;

[0115] Based on the speaker information, each voice text in the second voice recognition text is marked to obtain a corresponding relationship between each voice text and the speaker information.

[0116] In one embodiment, the second speech recognition text includes modal particles, erroneous recognition results, and recognition results that fail to be organized into segments.

[0117] In one embodiment, when the processor implements the extraction of image key frames in the video data based on the image features of the video data and the text features of the first speech recognition text, it is configured to implement:

[0118] Based on an image feature recognition algorithm, identifying feature differences among image frames in the video data to obtain at least one first candidate key frame;

[0119] Clustering video frames in the video data based on the text segmentation of the first speech recognition text to obtain at least one second candidate key frame;

[0120] Based on a preset key frame extraction algorithm, the first candidate key frame and the second candidate key frame are fused and extracted to obtain the image key frame.

[0121] In one embodiment, when the processor implements the image feature recognition algorithm to identify feature differences between image frames in the video data and obtain at least one first candidate key frame, it is configured to implement:

[0122] Based on a preset resolution, performing image scaling on each of the image frames to obtain a scaled image frame;

[0123] Based on the image feature recognition algorithm, calculating the feature similarity between each of the continuous zoomed image frames and the anchor point image frame;

[0124] When the feature similarity between the current zoomed image frame and the anchor image frame is less than a preset similarity threshold, the current zoomed image frame is determined to be the first candidate key frame, and the current zoomed image frame is determined to be the anchor image frame for screening the next first candidate key frame.

[0125] In one embodiment, when the processor implements the fusion extraction of the first candidate key frame and the second candidate key frame based on the preset key frame extraction algorithm to obtain the image key frame, it is used to implement:

[0126] Based on the time sequence, each of the first candidate key frames and the second candidate key frames are fused to obtain a fused candidate key frame sequence;

[0127] Based on the key frame extraction algorithm, the candidate key frames in the fused candidate key frame sequence are deduplicated to obtain a target key frame sequence, wherein the target key frame sequence includes at least one frame of the image key frame.

[0128] A computer-readable storage medium is also provided in an embodiment of the present application, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and the processor executes the program instructions to implement any one of the video processing methods based on multimodal information fusion provided in the embodiments of the present application.

[0129] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.

[0130] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A video processing method based on multimodal information fusion, characterized in that: The method comprises: Based on the large language model, performing text processing on the speech text in the video data to obtain a first speech recognition text; Extracting image key frames in the video data based on image features of the video data and text features of the first speech recognition text; Based on the time correspondence, the image key frame and the first speech recognition text are aligned and fused to output a multimodal fused text.

2. The video processing method based on multimodal information fusion according to claim 1, characterized in that: The method of performing text processing on the speech text in the video data based on the large language model to obtain the first speech recognition text includes: Based on a speech recognition algorithm, recognize the speech information in the video data to obtain a second speech recognition text; Based on the large language model, text error correction and text segmentation processing are performed on the second speech recognition text to obtain the first speech recognition text.

3. The video processing method based on multimodal information fusion according to claim 2, characterized in that: The step of performing text error correction and text segmentation processing on the second speech recognition text based on the large language model to obtain the first speech recognition text further includes: Based on a user identification algorithm, identifying speaker information corresponding to each voice information in the video data; Based on the speaker information, each voice text in the second voice recognition text is marked to obtain a corresponding relationship between each voice text and the speaker information.

4. The video processing method based on multimodal information fusion according to claim 2, characterized in that: The second speech recognition text includes modal particles, erroneous recognition results, and recognition results that fail to be organized into segments.

5. The video processing method based on multimodal information fusion according to claim 1, characterized in that: The step of extracting image key frames from the video data based on the image features of the video data and the text features of the first speech recognition text includes: Based on an image feature recognition algorithm, identifying feature differences among image frames in the video data to obtain at least one first candidate key frame; Clustering video frames in the video data based on the text segmentation of the first speech recognition text to obtain at least one second candidate key frame; Based on a preset key frame extraction algorithm, the first candidate key frame and the second candidate key frame are fused and extracted to obtain the image key frame.

6. The video processing method based on multimodal information fusion according to claim 5, characterized in that: The step of identifying feature differences of image frames in the video data based on an image feature recognition algorithm to obtain at least one first candidate key frame includes: Based on a preset resolution, performing image scaling on each of the image frames to obtain a scaled image frame; Based on the image feature recognition algorithm, calculating the feature similarity between each of the continuous zoomed image frames and the anchor point image frame; When the feature similarity between the current zoomed image frame and the anchor image frame is less than a preset similarity threshold, the current zoomed image frame is determined to be the first candidate key frame, and the current zoomed image frame is determined to be the anchor image frame for screening the next first candidate key frame.

7. The video processing method based on multimodal information fusion according to claim 5, characterized in that: The step of fusing and extracting the first candidate key frame and the second candidate key frame based on a preset key frame extraction algorithm to obtain the image key frame includes: Based on the time sequence, each of the first candidate key frames and the second candidate key frames are fused to obtain a fused candidate key frame sequence; Based on the key frame extraction algorithm, the candidate key frames in the fused candidate key frame sequence are deduplicated to obtain a target key frame sequence, wherein the target key frame sequence includes at least one frame of the image key frame.

8. A video processing device based on multimodal information fusion, characterized in that: The video processing device based on multimodal information fusion includes: A speech recognition module, used for performing text processing on the speech text in the video data based on the large language model to obtain a first speech recognition text; An image key frame extraction module, configured to extract image key frames from the video data based on image features of the video data and text features of the first speech recognition text; The multimodal fusion module is used to align and fuse the image key frame and the first speech recognition text based on the time correspondence, and output the multimodal fusion text.

9. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the video processing method based on multimodal information fusion as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the video processing method based on multimodal information fusion as described in any one of claims 1 to 7 are implemented.

Images