Media file barrier-free description generation method based on multi-modal large language model
Audio and video are preprocessed and analyzed through multimodal large language model, and subtitles, text descriptions and speech descriptions are generated, which solves the deviation problem caused by manual intervention in subtitle generation in the prior art, and achieves efficient and accurate barrier-free descriptions , meet the needs of people with disabilities and reduce costs.
Patent Information
- Application Number
- CN202410914279.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-09
- Publication Date
- 2024-11-22
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In the prior art, video file subtitles generation requires a lot of manual intervention, which is prone to subjective deviations, affecting the accuracy and consistency of descriptions.
Using the media file barrier-free description generation method based on the multimodal large language model, the subtitles, the text description of the audio part, the text description of the video part, and the voice description synchronized with the video are automatically generated by pre-processing the audio and video. Pre-trained natural language processing models, image processing models and audio processing models are used for content analysis and correction.
It improves the efficiency of automatic analysis and description generation of media file content, reduces the need for manual intervention, provides more accurate and consistent description content, meets the needs of people with different disabilities, and reduces costs.
Smart Images

Figure CN119003803A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of artificial intelligence, and in particular relates to a method for generating accessible descriptions of media files based on a multimodal large language model. Background Art
[0002] With the rapid development of information technology, media files such as video and audio have been widely used in daily life and work. However, for the blind and low-vision, deaf and hearing-impaired, as well as people with learning disabilities and cognitive limitations, there are still great obstacles to accessing the information in these media files. Although traditional accessibility technologies, such as subtitles and audio descriptions, have alleviated this problem to a certain extent, their generation process usually requires a lot of manual intervention, which is inefficient, expensive, and has poor accuracy.
[0003] In the existing technology, subtitle generation mainly relies on speech recognition technology, but because speech recognition technology performs poorly in noisy environments, the generated subtitles often require a lot of manual post-correction. In addition, the text description of video content and audio content usually needs to be written manually, which is not only time-consuming and labor-intensive, but also prone to subjective bias, affecting the accuracy and consistency of the description. Summary of the invention
[0004] The technical problem to be solved by the present invention is to provide a method for generating accessible descriptions of media files based on a multimodal large language model, which solves the problem in the prior art that manual intervention in the generation of subtitles for video files is prone to subjective bias, affecting the accuracy and consistency of the description.
[0005] The present invention adopts the following technical solutions to solve the above technical problems:
[0006] A method for generating accessible descriptions of media files based on a multimodal large language model uses a multimodal large language model to analyze and preprocess media files, and automatically generates subtitles and video files with subtitles, text description files for the audio part, text description files for the video part, and voice description files for video content synchronized with the video.
[0007] The preprocessing includes preprocessing the audio part and preprocessing the video part, wherein:
[0008] The process of preprocessing the audio part includes audio segmentation, noise processing, and background sound filtering;
[0009] The process of preprocessing the video portion includes selecting a specific number of video frames within a specific duration for analysis; converting the video frames of the video file into grayscale images, adjusting the video frame size, removing video frame noise in the video, and binarizing the video frames.
[0010] Processing the preprocessed audio file includes generating a subtitle file and generating a text description file of the audio part. The specific process is as follows:
[0011] First, the audio part is converted into text and corresponding start and end timestamps; second, the converted text and timestamps are corrected and optimized to generate a subtitle file and a text description file of the audio part; then, the generated subtitles are displayed synchronously with the video content to generate a video file with subtitles.
[0012] The correction and optimization of the converted text and timestamp include: correcting the content of the generated text; correcting the timeline of the generated text; translating the generated text into multiple languages; classifying the generated text according to different speakers; and processing the length and segmentation of the generated text.
[0013] Converting the audio portion into text includes describing the tone and background sounds in the audio.
[0014] The process of processing the preprocessed video file is as follows:
[0015] Select several key frames and corresponding start and end timestamps; give text descriptions to the key frames; and generate a text description file for the video portion.
[0016] The step of selecting a key frame includes
[0017] Calculate the structural similarity index between the current frame and the last saved frame, select the frame whose structural similarity index is less than a certain threshold as the key frame and save it;
[0018] Calculate the color histogram of the current frame, calculate the difference between the color histogram of the current frame and the last saved frame, select the frame whose histogram difference is greater than a specific threshold as the key frame and save it;
[0019] Extract the text content in the current frame, calculate the difference between the text content of the current frame and the last saved frame, select the frame with significant change in text content as the key frame and save it.
[0020] The structural similarity index was calculated using the following formula:
[0021]
[0022] where μ x and μ y are the means of images x and y, σ x and σ y are the variances of images x and y, σ xy is the covariance of images x and y, C 1 and C 2is a constant used for stability.
[0023] The color histogram difference between the current frame and the last saved frame is calculated using the following formula:
[0024]
[0025] Among them, H 1 and H 2 are the color histograms of the current frame and the last saved frame, respectively, and N is the number of colors; the video frame is divided into multiple regions, and the color histogram difference of each region is calculated.
[0026] The text difference is calculated using the following formula:
[0027]
[0028] Among them, T 1 and T 2 are the text contents of the current frame and the last saved frame, respectively. M is the number of characters. δ is an indicator of whether the characters are the same, 0 if they are the same and 1 if they are different.
[0029] The steps of providing a text description for key frames include: using a multimodal large language model to perform context understanding and semantic analysis on the key frames in combination with the preceding and following frames and audio content within a specific time range, and providing a text description for the key frames; providing a text description for the changes occurring between adjacent key frames; matching the generated text description with the timestamp of the key frames; and generating a text description for the video portion.
[0030] The text description file of the video part includes: the text description of the video part; and the start and end timestamps of the key frames corresponding to the text description.
[0031] The specific process of generating a voice description file of video content synchronized with the video is as follows:
[0032] Using the text description of the key frames generated according to claim 11, a text-to-speech system is used to generate an independent voice file for each text description; the generated voice file is matched with the timestamp of the key frame; each voice file is inserted into the time interval between two key frames of the video file independently of the audio of the original video; and a voice description file of the video content synchronized with the video is generated.
[0033] The functions of the voice description file of the video content synchronized with the video include: when the user chooses to play the audio of the text description of the video content at any time in the video, the voice and text description of the key frame corresponding to the time range of the current video time can be played.
[0034] Multimodal large language model, including pre-trained natural language processing model, image processing model and audio processing model, among which,
[0035] The image processing model processes the preprocessed video files to perform contextual understanding and semantic analysis on key frames, combined with the previous and next frames and audio content within a specific time range, and to provide text descriptions of key frames and changes between adjacent key frames;
[0036] The audio processing model processes the pre-processed audio files to convert the audio content into text and to describe the tone and background sound in the audio;
[0037] Pre-trained natural language processing models are used to correct and optimize the content of converted text.
[0038] Compared with the prior art, the present invention has the following beneficial effects:
[0039] 1. Efficiency: By using a multimodal large language model, automatic analysis and description generation of media file content is achieved, which improves processing efficiency and reduces the need for manual intervention.
[0040] 2. Accuracy: The multimodal large language model has high accuracy in generating natural language descriptions and can provide more accurate and consistent description content.
[0041] 3. Diversity: It can not only generate subtitles, but also text descriptions of the audio and video parts and voice descriptions synchronized with the video to meet the needs of people with different disabilities.
[0042] 4. Economical: Through automated processing, labor costs are reduced, making the generation of accessible descriptions more economical and efficient. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 Schematic diagram for converting audio into text and timestamps;
[0044] Figure 2 Diagram corrected and optimized for audio to text conversion and timestamps
[0045] Figure 3 A schematic diagram for generating a video file with subtitles;
[0046] Figure 4 A schematic diagram for generating a text description file for the audio part;
[0047] Figure 5 This is a schematic diagram of the effect of video frame preprocessing;
[0048] Figure 6 Schematic diagram of selecting key frames based on structural similarity index;
[0049] Figure 7 Schematic diagram of selecting key frames based on color histogram differences;
[0050] Figure 8 A schematic diagram of selecting key frames based on changes in text content;
[0051] Fig. 9 A schematic diagram for generating a text description file for a video portion;
[0052] Fig.10 A schematic diagram of generating a voice description file of video content synchronized with the video.
[0053] Fig.11 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION
[0054] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only embodiments of a part of the present application, rather than all of the embodiments.
[0055] Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative work should fall within the scope of protection of this application.
[0056] The present invention proposes a method for generating accessible descriptions of media files based on a multimodal large language model. By analyzing media files such as video or audio, subtitles, text descriptions of the audio part, text descriptions of the video part, and voice descriptions of video content synchronized with the video are automatically generated, thereby improving the accessibility of media files and helping people with disabilities to understand and obtain media content without barriers. This method not only improves the accuracy and efficiency of information transmission, but also greatly reduces the need for manual intervention, making the accessible processing of media content more accurate, efficient, and economical.
[0057] A method for generating accessible descriptions of media files based on a multimodal large language model uses a multimodal large language model to analyze and preprocess media files, and automatically generates subtitles and video files with subtitles, text description files for the audio part, text description files for the video part, and voice description files for video content synchronized with the video.
[0058] Multimodal large language model, including pre-trained natural language processing model, image processing model and audio processing model, among which,
[0059] The image processing model processes the preprocessed video files to perform contextual understanding and semantic analysis on key frames, combined with the previous and next frames and audio content within a specific time range, and to provide text descriptions of key frames and changes between adjacent key frames;
[0060] The audio processing model processes the pre-processed audio files to convert the audio content into text and to describe the tone and background sound in the audio;
[0061] Pre-trained natural language processing models are used to correct and optimize the content of converted text.
[0062] Specific embodiments, such as Figures 1 to 8 As shown,
[0063] A method for generating accessible descriptions of media files based on a multimodal large language model comprises the following steps:
[0064] 1. Receive media files: The system receives video or audio files uploaded by users. The media files can be common media files in any format, such as MP4, AVI, MP3, WAV, etc.
[0065] 2. Select one or more methods to pre-process the audio part: segment the audio file according to preset time periods (such as every 10 minutes) to facilitate subsequent multi-threaded simultaneous processing to improve efficiency; perform noise processing on the audio file to filter out background noise and improve the clarity of the audio signal; further process the audio file to remove unnecessary background noise and make the voice content clearer.
[0066] 3. Convert audio to text and timestamps: Use a multimodal large language model to generate the text content of the audio; the speaker corresponding to the text content; a text description of the tone and background sound in the audio; and the start and end timestamps corresponding to the text content. Figure 1 shown.
[0067] 4. Select one or more methods to correct and optimize the text and timestamps: Correct the grammar and content of the generated text, as well as some special words such as proper nouns, names, formulas, etc. to ensure the accuracy of the text; correct the timeline of the text to ensure that the subtitles are synchronized with the audio; translate the text into multiple languages as needed, such as from Chinese to English or from English to Chinese; correct the speaker and classify the text according to different speakers; process the length and segmentation of the text. The results after correction and optimization are as follows: Figure 2 shown.
[0068] 5. Generate subtitle file: Generate SRT format subtitle file with the corrected and optimized text and timestamp
[0069] 6. Synchronize subtitles and video: Synchronize the generated subtitle file with the video content to generate a video file with subtitles, such as Figure 3 shown.
[0070] 7. Generate a text description file for the audio part: Use a multimodal large language model to generate a text description of the audio part, including: the text content of the audio, the text description of the tone and background sound, the start and end timestamps corresponding to the text content, and the speaker information, and integrate them into a PDF format file, such as Figure 4 shown.
[0071] 8. Select one or more methods to pre-process the video frames (the effects of each step are as follows Figure 5 shown):
[0072] 8.1. Select a specific number of video frames for analysis within a specific duration: one frame per second;
[0073] The steps to select a keyframe include:
[0074] Calculate the structural similarity index between the current frame and the last saved frame, select the frame whose structural similarity index is less than a certain threshold as the key frame and save it;
[0075] Calculate the color histogram of the current frame, calculate the difference between the color histogram of the current frame and the last saved frame, select the frame whose histogram difference is greater than a specific threshold as the key frame and save it;
[0076] Extract the text content in the current frame, calculate the difference between the text content of the current frame and the last saved frame, select the frame with significant change in text content as the key frame and save it.
[0077] The structural similarity index was calculated using the following formula:
[0078]
[0079] where μ x and μ y are the means of images x and y, σ x and σ y are the variances of images x and y, σ xy is the covariance of images x and y, C 1 and C 2 is a constant used for stability.
[0080] The color histogram difference between the current frame and the last saved frame is calculated using the following formula:
[0081]
[0082] Among them, H1 and H 2 are the color histograms of the current frame and the last saved frame, respectively, and N is the number of colors; the video frame is divided into multiple regions, and the color histogram difference of each region is calculated.
[0083] The text difference is calculated using the following formula:
[0084]
[0085] Among them, T 1 and T 2 are the text contents of the current frame and the last saved frame, respectively. M is the number of characters. δ is an indicator of whether the characters are the same, 0 if they are the same and 1 if they are different.
[0086] 8.2. Convert video frames to grayscale images:
[0087] Function: cv2.cvtColor of OpenCV, an open source computer vision and machine learning software library;
[0088] Parameters: frame, cv2.COLOR_BGR2GRAY.
[0089] 8.3. Adjust the size of the video frame: scale up the image by 2 times in both horizontal and vertical directions, using bicubic interpolation.
[0090] Function: OpenCV's cv2.resize,
[0091] Parameters: gray, None, fx=2, fy=2, interpolation=cv2.INTER_CUBIC.
[0092] 8.4. Apply Gaussian blur to the video frame to remove noise: Apply Gaussian blur to the video frame with a kernel size of 5x5 and a standard deviation of 0.
[0093] Function: OpenCV's cv2.GaussianBlur,
[0094] Parameters: resized, (5,5), 0.
[0095] 8.5. Binarize the video frame: the threshold is 150, the maximum value is 255, and the Otsu's Method is used to automatically determine the optimal threshold.
[0096] Function: OpenCV's cv2.threshold,
[0097] Parameters: blurred, 150, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU.
[0098] 9. Choose one or more methods to select key frames: Calculate the structural similarity index (SSIM) between the current frame and the last saved frame, and select frames with a structural similarity index less than 75% as key frames, such as Figure 6 As shown; calculate the color histogram difference, select the frame with a color histogram difference of more than 70% in more than 20 consecutive 10x10 pixel areas as the key frame, as shown Figure 7 As shown; use OCR technology to extract the text content in the frame, and select the frame with text content changes of more than 3 words as the key frame, such as Figure 8 shown.
[0099] 10. Generate a text description file for the video part: Use the multimodal large language model to perform context understanding and semantic analysis on the key frames selected in step 9, generate a text description of the video part, including: a text description of the key frames and a text description of the changes between adjacent key frames; match with the key frame timestamps; generate a text description file for the video part, including: a text description of the video part and the start and end timestamps of the corresponding key frames, and integrate them into a PDF format file, such as Fig. 9 shown.
[0100] 11. Generate a voice description file of the video content synchronized with the video: Use a text-to-speech system (such as OpenAI's open source speech recognition model Whisper) to convert the text description of each key frame into an independent WAV format voice file; match the generated voice file with the timestamp of the key frame, and insert it into the time interval between two key frames of the video file independently of the audio of the original video; generate a voice description file of the video content synchronized with the video. Users can click a button at any time point during video playback to play the voice file of the key frame text description of the corresponding time node, such as Fig.10 For example, if the user stops at 1:20 and there is a key frame corresponding to 1:10-1:30, the audio and text description of this key frame will be played.
[0101] Through the above implementation steps, the present invention provides an efficient and accurate method for generating accessible descriptions of media files based on a multimodal large language model, which can help users in need to better understand and experience multimedia content.
[0102] This method aims to achieve efficient and accurate text and voice description of the content of media files such as video and audio through advanced natural language processing technology and multimodal data processing technology, thereby improving the accessibility of media content, helping people with visual and hearing impairments and cognitive impairments to obtain information equally, and promoting social inclusiveness and fairness.
[0103] Those skilled in the art should understand that they can implement variations by combining the prior art and the above embodiments. Such variations do not affect the essence of the present solution and are not described in detail here.
[0104] It should be understood that the present solution is not limited to the above-mentioned specific implementation methods, and the devices and structures not described in detail should be understood to be implemented in a common manner in the art; any technician familiar with the art can use the above-disclosed methods and technical contents to make many possible changes and modifications to the technical solution of the present solution without departing from the scope of the technical solution of the present solution, or modify it into an equivalent embodiment with equivalent changes, which does not affect the essential content of the present solution. Therefore, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present solution without departing from the content of the technical solution of the present solution still falls within the scope of protection of the technical solution of the present solution.
Claims
1. A method for generating accessible descriptions of media files based on a multimodal large language model, characterized in that: A multimodal large language model is used to analyze and preprocess media files, and subtitles and video files with subtitles, text description files for the audio part, text description files for the video part, and voice description files for video content synchronized with the video are automatically generated.
2. The method for generating accessible description of media files based on a multimodal large language model according to claim 1, characterized in that: The preprocessing includes preprocessing the audio part and preprocessing the video part, wherein: The process of preprocessing the audio part includes audio segmentation, noise processing, and background sound filtering; The process of preprocessing the video portion includes selecting a specific number of video frames within a specific duration for analysis; converting the video frames of the video file into grayscale images, adjusting the video frame size, removing video frame noise in the video, and binarizing the video frames.
3. The method for generating accessible description of media files based on a multimodal large language model according to claim 2, characterized in that: Processing the preprocessed audio file includes generating a subtitle file and generating a text description file of the audio part. The specific process is as follows: First, the audio part is converted into text and corresponding start and end timestamps; second, the converted text and timestamps are corrected and optimized to generate a subtitle file and a text description file of the audio part; then, the generated subtitles are displayed synchronously with the video content to generate a video file with subtitles.
4. The method for generating accessible description of media files based on a multimodal large language model according to claim 3, characterized in that: The correction and optimization of the converted text and timestamp include: correcting the content of the generated text; correcting the timeline of the generated text; translating the generated text into multiple languages; classifying the generated text according to different speakers; and processing the length and segmentation of the generated text.
5. The method for generating accessible description of media files based on a multimodal large language model according to claim 4, characterized in that: Converting the audio portion into text includes describing the tone and background sounds in the audio.
6. The method for generating accessible description of media files based on a multimodal large language model according to claim 2, characterized in that: The process of processing the preprocessed video file is as follows: Select several key frames and corresponding start and end timestamps; give text descriptions to the key frames; and generate a text description file for the video portion.
7. The method for generating accessible description of media files based on a multimodal large language model according to claim 6, characterized in that: The step of selecting a key frame includes Calculate the structural similarity index between the current frame and the last saved frame, select the frame whose structural similarity index is less than a certain threshold as the key frame and save it; Calculate the color histogram of the current frame, calculate the difference between the color histogram of the current frame and the last saved frame, select the frame whose histogram difference is greater than a specific threshold as the key frame and save it; Extract the text content in the current frame, calculate the difference between the text content of the current frame and the last saved frame, select the frame with significant change in text content as the key frame and save it.
8. The method for generating accessible description of media files based on a multimodal large language model according to claim 7, characterized in that: The structural similarity index was calculated using the following formula: where μ x and μ y are the mean of images x and , σ x and σ y are the variances of images x and y, σ xy is the covariance of images x and y, and C1 and C2 are constants used for stabilization.
9. The method for generating accessible description of media files based on a multimodal large language model according to claim 7, characterized in that: The color histogram difference between the current frame and the last saved frame is calculated using the following formula: Where H1 and H2 are the color histograms of the current frame and the last saved frame, respectively, and N is the number of colors; the video frame is divided into multiple regions, and the color histogram difference of each region is calculated.
10. The method for generating accessible description of media files based on a multimodal large language model according to claim 7, characterized in that: The text difference is calculated using the following formula: Among them, T1 and T2 are the text contents of the current frame and the last saved frame respectively, M is the number of characters, and δ is an indicator of whether the characters are the same, 0 for the same and 1 for different.
11. The method for generating accessible description of media files based on a multimodal large language model according to claim 6, characterized in that: The steps of providing a text description for key frames include: using a multimodal large language model to perform context understanding and semantic analysis on the key frames in combination with the preceding and following frames and audio content within a specific time range, and providing a text description for the key frames; providing a text description for the changes occurring between adjacent key frames; matching the generated text description with the timestamp of the key frames; and generating a text description for the video portion.
12. The method for generating accessible description of media files based on a multimodal large language model according to claim 6, characterized in that: The text description file of the video part includes: the text description of the video part; and the start and end timestamps of the key frames corresponding to the text description.
13. The method for generating accessible description of media files based on a multimodal large language model according to claim 3, characterized in that: The specific process of generating a voice description file of video content synchronized with the video is as follows: Using the text descriptions of the key frames generated according to claim 11, using a text-to-speech system to generate an independent voice file for each text description; matching the generated voice file with the timestamp of the key frame; inserting each voice file independently of the audio of the original video into the time interval between two key frames of the video file; Generates a voice description file of the video content that is synchronized with the video.
14. The method for generating accessible description of media files based on a multimodal large language model according to claim 13, characterized in that: The functions of the voice description file of the video content synchronized with the video include: when the user chooses to play the audio of the text description of the video content at any time in the video, the voice and text description of the key frame corresponding to the time range of the current video time can be played.
15. A multimodal large language model for processing media files by calling the method according to any one of claims 1 to 14, characterized in that: Includes pre-trained natural language processing models, image processing models, and audio processing models, among which: The image processing model processes the preprocessed video files to perform contextual understanding and semantic analysis on key frames, combined with the previous and next frames and audio content within a specific time range, and to provide text descriptions of key frames and changes between adjacent key frames; The audio processing model processes the pre-processed audio files to convert the audio content into text and to describe the tone and background sound in the audio; Pre-trained natural language processing models are used to correct and optimize the content of converted text.
Citation Information
Cited By
Method and system for assisting visually-impaired / hearing-impaired person to browse multimedia without barrier
CN119862289A
