Video content management method based on multi-modal model, computer equipment and medium

Abstract

The invention discloses a video content management method based on a multi-modal model, computer equipment and a medium, and belongs to the technical field of multimedia, and the video content management method based on the multi-modal model comprises the following steps: after extracting an original key frame picture from at least one video clip, obtaining a timestamp and a time point of the original key frame picture; according to the method, repeated picture content and invalid picture content of an original key frame picture are removed, then a blank screen picture is removed, vectorization processing is carried out through a CLIP model, vectorized data are obtained, automatic voice recognition is executed, and finally video metadata is generated. According to the video content management method based on the multi-modal model, video content retrieval service based on pictures or characters can be provided, delicacy management of videos is achieved, video content retrieval based on frame levels is provided, the accuracy of video content retrieval is improved, and millisecond-level retrieval of mass video content is supported.

Description

Technical Field

[0001] The present invention belongs to the field of multimedia technology, and in particular relates to a video content management method, computer equipment and medium based on a multimodal model. Background Art

[0002] In recent years, various video forms such as long video, short video, live broadcast, and on-demand have become increasingly popular in people's lives. The content of videos is richer than simple text, and the relationship attributes between video clips are complex, with both time and space attributes. With the increase in the number of videos, it is becoming more and more difficult to find video content and manage historical videos. As time goes by, it is difficult to grasp the detailed content of each video in the video library, especially for self-media people and small and medium-sized video creation companies. If you want to accurately browse and process a video in the video library, it is even more difficult. Therefore, there is an urgent need for a system that can manage video content more conveniently and at a low cost, so as to realize the assetization and daily management of videos, which is conducive to subsequent editing and other production. Summary of the invention

[0003] In order to solve the above technical problems, the present invention provides a video content management method based on a multimodal model, which comprises the following steps:

[0004] S1, performing normalized transcoding processing on the video to be processed to obtain a normalized transcoded video;

[0005] S2, segmenting the normalized transcoded video to obtain at least one video segment, and extracting an original key frame image from the at least one video segment;

[0006] S3. Obtain the timestamp and time point of the original key frame image according to the display timestamp of the original key frame image;

[0007] S4, removing duplicate image content and invalid image content from the original key frame image to obtain a first key frame image;

[0008] S5. Remove the black screen image from the first key frame image to obtain a second key frame image;

[0009] S6. Performing vectorization processing on the second key frame image through the multimodal model to obtain vectorized data, and saving the vectorized data to a vector database;

[0010] S7. Perform automatic speech recognition on the audio corresponding to the video to be processed to obtain a speech recognition text.

[0011] Furthermore, the video content management method based on the multimodal model also includes the following steps:

[0012] S8. Generate video metadata.

[0013] Further, extracting an original key frame image from at least one video clip includes:

[0014] Set the threshold of the sc_threshold parameter for the video encoder, perform automatic scene analysis, and extract original key frame images based on scene changes.

[0015] Furthermore, the threshold of the sc_threshold parameter for the video encoder is set to 60.

[0016] Further, step S4 includes:

[0017] The duplicate picture content of the original key frame pictures is removed through the histogram detection algorithm, the feature point matching and the recognition algorithm. When the detection results of the histogram detection algorithm, the feature point matching and the recognition algorithm are that the similarity of the picture content of two adjacent original key frame pictures is greater than 76%, the two adjacent original key frame pictures are merged to remove the duplicate picture content of the original key frame pictures.

[0018] Further, step S5 includes:

[0019] S5.1. grayscale the first key frame image to obtain a grayscale image;

[0020] S5.2. Calculate the proportion of white pixels in all pixels in the grayscale image. When the proportion of white pixels in all pixels is less than 20%, the grayscale image is determined to be a black screen image.

[0021] S5.3. Delete the black screen image.

[0022] Further, step S7 includes:

[0023] S7.1, normalizing the audio corresponding to the video to be processed to obtain normalized audio;

[0024] S7.2, segmenting the normalized audio using a VAD algorithm to obtain multiple segmented audios, and setting audio timestamps at the in point and out point of each segmented audio;

[0025] S7.3. Automatically perform speech recognition on multiple segmented audios through the Spleeter dry sound extraction model to obtain speech recognition text.

[0026] Furthermore, video metadata includes:

[0027] Basic video data, including video name, video size, video address, video encoding, video bit rate, video frame rate, video resolution, and audio encoding;

[0028] Video segment data, including the first frame address, video segment size, vectorized data of the second key frame image, display timestamp, audio timestamp, and duration;

[0029] Business support data, including unit affiliation, creator information, security access scope, and business usage restrictions.

[0030] The present invention also provides a computer device, which includes a memory, a processor and a computer program stored in the memory. When the computer program is executed by the processor, the aforementioned video content management method based on the multimodal model is implemented.

[0031] The present invention further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the aforementioned video content management method based on a multimodal model is implemented.

[0032] Compared with the prior art, the video content management method based on the multimodal model according to the present invention can provide video content retrieval services based on pictures or texts and enrich the query means of video content, which greatly simplifies the difficulty and cost of video management, realizes the refined management of videos, and provides frame-level video content retrieval, improves the accuracy of video content retrieval, and supports millisecond-level retrieval of massive video content. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A flowchart of a video content management method based on a multimodal model according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0034] The specific implementation of the present invention is described in detail below. It should be understood that the implementation of the present invention is not limited to the embodiments shown in the accompanying drawings, and the protection scope of the present invention is not limited by the specific implementation. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, similar words such as "one", "one" or "the" do not indicate quantity restrictions, but indicate that there is at least one. Unless otherwise expressly indicated, throughout the specification and claims, the term "include" or its changes such as "include" or "include" will be understood to include the stated elements or components, without excluding other elements or other components. Similar words such as "connect" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0035] Unless otherwise defined, the meanings of all technical terms and scientific terms used in the present invention are the same as those commonly understood by ordinary technicians in the field. In addition, the meanings of the technical terms and scientific terms used in the present invention should be interpreted as having the same meaning as the corresponding terms defined in commonly used technical manuals, and should not be interpreted as having idealized or excessively formal meanings, unless the present invention clearly defines it in this way.

[0036] Figure 1 A flowchart of a video content management method based on a multimodal model according to an embodiment of the present invention is shown. Figure 1 , the video content management method based on the multimodal model comprises the following steps:

[0037] S1. Perform normalized transcoding processing on the video to be processed to obtain a normalized transcoded video.

[0038] Among them, the video format of the video to be processed is 720P, and the video format of the normalized transcoded video is 1080P. In order to ensure the consistency of the normalized transcoded video in the subsequent processing process, the frame rate (frames per second, fps) is uniformly set to 25 during the normalized transcoding process.

[0039] S2. Segment the normalized transcoded video to obtain at least one video segment, and extract an original key frame image from the at least one video segment.

[0040] When segmenting videos, the content, integrity, and time accuracy of the video clips must be guaranteed. Specifically, for content, some meaningless black screens, opening and closing subtitles, etc. need to be removed to prevent video pollution; for integrity, the video content clips must be complete, such as being in the same scene or a group of similar pictures; for time accuracy, it is crucial for the later application of the video, and the timestamp must be accurate to the millisecond level in order to locate the video content of a certain frame during use.

[0041] Specifically, a threshold of a sc_threshold parameter of a video encoder is set, automatic scene analysis is performed, and original key frame images are extracted according to scene changes.

[0042] The sc_threshold parameter is an FFmpeg parameter used for the video encoder. It indicates the difference between two frames of images and is mainly used to control the sensitivity of scene change detection. In a video, scene change refers to a significant change in the content of the picture, such as lens switching, a large change in picture brightness, etc., which can be used as an important basis for judging whether a scene change has occurred in the video. The video encoder uses scene change information to optimize encoding and inserts key frames at scene changes to improve video quality and reduce encoding complexity. Therefore, by setting a reasonable threshold of the sc_threshold parameter, the video content of different scenes can be basically distinguished. The value range of the sc_threshold parameter is 0 to 100. A small value indicates that the difference between the two frames is small, and a large value indicates that the difference between the two frames is large.

[0043] Furthermore, according to the results of multiple actual verifications by the inventors, when the sc_threshold parameter is set to 60, the requirements for the content, integrity, and time accuracy of the video clips obtained by video segmentation can be met.

[0044] S3. According to the presentation time stamp (PTS) of the original key frame image, obtain the time stamp and time point of the original key frame image.

[0045] Specifically, the original key frame image is decoded to obtain the playback time point of the original key frame image, that is, the timestamp, and the time point is obtained after conversion, and the time point is accurate to milliseconds.

[0046] In order to extract the timestamp data of the original key frame image, the C++ dedicated library of ffmpeg is used for processing. The library function can load and analyze the content of the video, and extract the data of the original key frame image by decoding the video frame content. The current timestamp is converted according to the time slice data of the original key frame image, and the time point is obtained after the conversion, which is accurate to milliseconds.

[0047] S4. Remove duplicate image content and delete invalid image content from the original key frame image to obtain a first key frame image.

[0048] Among them, removing duplicate image content refers to merging the image content of two adjacent original key frame images; deleting invalid image content refers to deleting the image content of invalid images in the original key frame images, such as the opening image and the ending image.

[0049] Specifically, this embodiment removes duplicate image content from the original key frame image through a histogram detection algorithm and an ORB algorithm (Oriented Fast and RotatedBRIEF, feature point matching and recognition). According to the results of multiple actual verifications by the inventors, when the detection results of the histogram detection algorithm and the ORB algorithm are that the similarity of the image content of two adjacent original key frame images is greater than 76%, it is considered that the two adjacent original key frame images are basically the same, and the two adjacent original key frame images are merged, otherwise the two adjacent original key frame images are retained.

[0050] S5. Remove the black screen image from the first key frame image to obtain a second key frame image.

[0051] Specifically, step S5 includes:

[0052] S5.1. grayscale the first key frame image to obtain a grayscale image;

[0053] S5.2. Calculate the proportion of white pixels in all pixels in the grayscale image. According to the results of multiple actual verifications by the inventors, when the proportion of white pixels in all pixels is less than 20%, the grayscale image is judged to be a black screen image.

[0054] S5.3. Delete the black screen image.

[0055] S6. Perform vectorization processing on the second key frame image through the CLIP (Contrastive Language-Image Pre-Training, multimodal) model to obtain vectorized data, and save it to the vector database.

[0056] The CLIP model vectorizes the second key frame image, reads the obtained vector data in batches, and saves it to the vector database of ElasticSearch for multimodal retrieval in later business.

[0057] Furthermore, the video content management method based on the multimodal model of the embodiment of the present invention further includes the following steps:

[0058] S7. Perform automatic speech recognition (ASR) on the audio corresponding to the video to be processed to obtain speech recognition text.

[0059] In order to improve the video retrieval accuracy of the later business, automatic speech recognition is performed on the audio corresponding to the video to be processed. Specifically, step S7 includes:

[0060] S7.1. Normalize the audio corresponding to the video to be processed to obtain normalized audio.

[0061] Among them, the sampling rate of the normalized audio is unified to 38.4KHz, the number of channels is two, that is, multiple channels are merged into 2, and the format is wmv.

[0062] S7.2. Segment the normalized audio using a VAD (Voice Activity Detection) algorithm to obtain multiple audio segments, and set audio timestamps at the entry point and the exit point of each audio segment.

[0063] Among them, the VAD algorithm can accurately separate audio content and has high segmentation accuracy. It sets audio timestamps at the entry and exit points of each segmented audio generated by the VAD algorithm, which is conducive to calibrating the audio timestamps and controlling the time error of subsequent automatic speech recognition results within milliseconds. Its accuracy meets the requirements for content matching during post-production.

[0064] S7.3. Automatically perform speech recognition on multiple segmented audios through the Spleeter dry sound extraction model to obtain speech recognition text.

[0065] Among them, the Spleeter dry sound extraction model can effectively extract the effective human voice part in the audio, remove background music and background noise, and improve the recognition accuracy by extracting the dry sound content and then performing speech-to-text recognition.

[0066] Furthermore, the video content management method based on the multimodal model of the embodiment of the present invention further includes the following steps:

[0067] S8. Generate video metadata.

[0068] The video metadata includes:

[0069] Basic video data, including video name, video size, video address, video encoding, video bit rate, video frame rate, video resolution, and audio encoding;

[0070] Video segment data, including the first frame address, the size of the video segment, the vectorized data of the second key frame image, the display timestamp, the audio timestamp, and the duration;

[0071] Business support data, including unit affiliation, creator information, security access scope, and business usage restrictions.

[0072] In summary, the present invention provides a video content management method based on a multimodal model, after extracting the original key frame picture from at least one video clip, obtaining the timestamp and time point of the original key frame picture, removing duplicate picture content and invalid picture content from the original key frame picture, and then removing the black screen picture, performing vectorization processing through the CLIP model, obtaining vectorized data, and then performing automatic speech recognition, and finally generating video metadata. The video content management method based on the multimodal model can realize the refined and low-cost management of video content, that is, searching for the corresponding voice and video through text, then tracing back the segmentation information of the video, and finally tracing back to the entire video content, so as to retrieve and process the video more quickly and accurately, and can provide in-depth analysis and management of the video, classify the video content according to dimensions such as date, location, and scene, and can accurately retrieve the video through text, and has good versatility, compatibility, ease of use, manageability, and scalability.

[0073] Furthermore, the video content management method based on the multimodal model provided by the present invention can automatically and efficiently realize automatic analysis and warehousing based on videos through a specialized processing flow, and can provide video content retrieval services based on pictures or texts, and query methods for enriching video content, which greatly simplifies the difficulty of video management and the management cost of videos. It can be used as a basic tool platform for professional video content management, replacing the video management method based on manual cataloging, and be used by media resource management systems, thereby realizing refined management of videos, and can provide frame-level video content retrieval, improve the accuracy of video content retrieval, and support millisecond-level retrieval of massive video content, thereby improving user work efficiency.

[0074] The present invention further provides a computer device, which includes a memory, a processor, and a computer program stored in the memory, and when the computer program is executed by the processor, the above-mentioned video content management method based on the multimodal model is implemented.

[0075] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the multimodal model-based video content management method is implemented.

[0076] The foregoing description of specific exemplary embodiments of the present invention is for the purpose of illustration and demonstration. These descriptions are not intended to limit the present invention to the precise form disclosed, and it is clear that many changes and variations can be made in accordance with the above teachings. The purpose of selecting and describing the exemplary embodiments is to explain the specific principles of the present invention and its practical application, so that those skilled in the art can realize and utilize various different exemplary embodiments of the present invention and various different selections and changes. The scope of protection of the present invention is defined by the claims and their equivalents.

Claims

1. A video content management method based on a multimodal model, characterized in that: The following steps are involved: S1, performing normalized transcoding processing on the video to be processed to obtain a normalized transcoded video; S2, segmenting the normalized transcoded video to obtain at least one video segment, and extracting an original key frame image from the at least one video segment; S3. Obtain the timestamp and time point of the original key frame image according to the display timestamp of the original key frame image; S4, removing duplicate picture contents and invalid picture contents from the original key frame picture to obtain a first key frame picture; S5. Remove the black screen image from the first key frame image to obtain a second key frame image; S6. Performing vectorization processing on the second key frame image through a multimodal model to obtain vectorized data, and saving the vectorized data to a vector database; S7. Perform automatic speech recognition on the audio corresponding to the video to be processed to obtain a speech recognition text.

2. The video content management method based on a multimodal model according to claim 1, characterized in that: The following steps are also included: S8. Generate video metadata.

3. The video content management method based on a multimodal model according to claim 2, characterized in that: In step S2, extracting an original key frame image from the at least one video segment includes: Set the threshold of the sc_threshold parameter for the video encoder, perform automatic scene analysis, and extract original key frame images based on scene changes.

4. The video content management method based on a multimodal model according to claim 3, characterized in that: The threshold of the sc_threshold parameter for the video encoder is set to 60.

5. The video content management method based on multimodal model according to claim 4, characterized in that: The step S4 comprises: The repeated picture content of the original key frame pictures is removed by using a histogram detection algorithm, a feature point matching and a recognition algorithm. When the detection results of the histogram detection algorithm, the feature point matching and the recognition algorithm are that the similarity of the picture content of two adjacent original key frame pictures is greater than 76%, the two adjacent original key frame pictures are merged to achieve the removal of repeated picture content of the original key frame pictures.

6. The video content management method based on multimodal model according to claim 4, characterized in that: The step S5 comprises: S5.

1. grayscale the first key frame image to obtain a grayscale image; S5.2, calculating the proportion of white pixels in all pixels in the grayscale image, and when the proportion of white pixels in all pixels is less than 20%, determining that the grayscale image is a black screen image; S5.

3. Delete the black screen image.

7. The video content management method based on multimodal model according to claim 4, characterized in that: The step S7 comprises: S7.1, normalizing the audio corresponding to the video to be processed to obtain normalized audio; S7.2, segmenting the normalized audio using a VAD algorithm to obtain multiple segmented audios, and setting audio timestamps at the in-point and out-point of each segmented audio; S7.

3. Automatically perform speech recognition on the multiple segmented audios through a Spleeter dry sound extraction model to obtain speech recognition text.

8. The video content management method based on multimodal model according to claim 7, characterized in that: The video metadata includes: Basic video data, including video name, video size, video address, video encoding, video bit rate, video frame rate, video resolution, and audio encoding; Video segment data, including the first frame address, the video segment size, the vectorized data of the second key frame picture, the display timestamp, the audio timestamp, and the duration; Business support data, including unit affiliation, creator information, security access scope, and business usage restrictions.

9. A computer device, comprising a memory, a processor and a computer program stored in the memory, wherein the computer program, when executed by the processor, implements the multimodal model-based video content management method according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the multimodal model-based video content management method according to any one of claims 1 to 8.

Images