Video content enhancement method and system, terminal and storage medium
By performing lens decomposition and speech recognition on narrative videos, using artificial intelligence to generate enhanced images related to video content, the problem of low enhancement efficiency of narrative video content in the prior art is solved, and higher accuracy and success rate are achieved , provides intuitive visual assistance.
Patent Information
- Application Number
- CN202411150732.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-12-06
AI Technical Summary
The prior art is difficult to effectively enhance the content of narrative videos, resulting in inefficiency of users when browsing and positioning. The existing methods rely on local image libraries. Complex or rare search entries are difficult to match related images, and the semantic correlation between the images and text Low.
By performing lens decomposition and speech recognition on the target video, artificial intelligence content generation technology is used to automatically generate enhanced images related to the video content and allocate them to the corresponding lens unit to generate comprehensive and accurate content enhancement images, avoiding local Image gallery dependency.
It improves the accuracy and success rate of video content enhancement, provides intuitive visual clues, enhances users' browsing and searching experience, and overcomes the one-sidedness and matching difficulties of visually enhanced content in the prior art.
Smart Images

Figure CN119094813A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence and video enhancement technology, and in particular to a video content enhancement method, system, terminal and storage medium. Background Art
[0002] The continuous development of digital media and network communication technology has greatly changed the way people communicate and interact. The medium of language communication is changing from traditional single-mode information such as text and voice to a more effective and attractive rich media form that integrates multi-modal information such as text, images, voice, animation, etc. In recent years, a large amount of narrative audio and video data from various activities such as online meetings, distance learning, lectures, speeches, interviews, discussions, dialogues, storytelling, etc. has shown a rapid growth trend on the Internet. Unlike surveillance videos, which are rich in visual information and the video itself carries a large number of visual clues required for people to browse and locate videos, narrative videos are usually composed of a large number of shots and pictures of speakers, and this kind of visual information often plays a very limited role in helping users quickly find content of interest, and they have to spend a lot of time locating content of interest from unstructured voice. How to effectively enhance the content of narrative videos and provide users with intuitive and useful visual clues to assist them in efficient content browsing and positioning is one of the key challenges that need to be solved in the multimedia field.
[0003] Existing content enhancement methods based on the video itself can work well for vision-centric videos (such as surveillance, sports, etc.) because the visual channels of such videos contain a large number of semantic clues related to finding and browsing video content. However, for narrative videos, it cannot meet the needs of fast browsing and positioning of narrative video content. For content enhancement of narrative videos, one of the existing methods is a retrieval-based visual enhancement method, which extracts query text from video subtitles, then retrieves relevant images from local albums or uses Microsoft Bing Image Search API from online repositories and synchronizes them with voice to achieve real-time content enhancement. However, this method relies heavily on existing image libraries. For complex or rare search terms, it is likely that relevant images cannot be matched or the text semantic correlation between images and queries is not high. Summary of the invention
[0004] In response to the above technical problems, the present application provides a video content enhancement method, system, terminal and storage medium, which automatically generate comprehensive and accurate content-enhanced images for videos, thereby improving the accuracy and success rate of video content enhancement.
[0005] In a first aspect, the present application provides a video content enhancement method, comprising:
[0006] Obtain the target video to be enhanced;
[0007] Dividing the target video into a plurality of shot units, each of which includes a plurality of consecutive frames with the same shooting angle;
[0008] Extracting a plurality of subtitle information and a timestamp corresponding to each of the subtitle information from the target video using a preset speech recognition tool;
[0009] Extracting corresponding prompt words from each of the subtitle information;
[0010] Inputting each of the prompt words into a preset image generation model, so that the image generation model generates an enhanced image corresponding to each of the prompt words, wherein the image generation model is constructed based on artificial intelligence generation technology;
[0011] Allocating each of the enhanced images to each of the lens units according to the start and end time of each of the lens units and the timestamp corresponding to each of the subtitle information;
[0012] The various lens units are recombined to obtain enhanced video.
[0013] The embodiment of the present application provides a method for enhancing video content, which automatically generates an enhanced image related to the current video content by decomposing the original video shots and performing voice recognition, and distributes it to the corresponding shots of the original video as a visual enhancement clue to assist video browsing, content search, etc. Compared with the retrieval-based method in the existing video enhancement technology, the embodiment of the present application can generate a more comprehensive and accurate content-enhanced image for the video, and is not constrained by the local image database, effectively overcoming the technical problems such as the one-sidedness or inability to match the visual enhancement content obtained in the existing retrieval-based technical solution, and improving the accuracy and success rate of video content enhancement.
[0014] In a possible implementation manner, dividing the target video into a plurality of shot units includes:
[0015] Sampling the target video at a preset sampling rate and sampling size to obtain a plurality of sampled images;
[0016] Using a preset image saliency detection algorithm to perform saliency detection on each of the sampled images to obtain corresponding grayscale images;
[0017] Performing Gaussian blur and binarization operations on each of the grayscale images to obtain corresponding binary images;
[0018] Comparing the content of each binary image frame with the content of the next binary image frame, and determining a plurality of boundary frames for shot switching according to the comparison results of the binary images;
[0019] The target video is divided into a plurality of shot units according to each boundary frame.
[0020] Furthermore, the step of comparing the content of each binary image frame with the content of the next binary image frame, and determining a plurality of shot switching boundary frames according to the comparison results of the binary images, includes:
[0021] Dividing each of the binary images into a plurality of image blocks according to a first preset pixel size;
[0022] Performing an XOR operation on a number of image blocks in each frame of the binary image and a number of image blocks at the same position in the next frame of the binary image, and summing the results of the XOR operation in units of image blocks to obtain similarity values corresponding to each image block;
[0023] The similarity values of each image block in each binary image are retrieved in turn. If the similarity value corresponding to a current image block is greater than a first threshold, the current image block is marked as an activated block. If the proportion of activated blocks in a current binary image is greater than a second threshold, the current binary image is marked as a boundary frame for the current shot switching.
[0024] The embodiment of the present application provides a method for segmenting a target video. Since the generated visual enhancement image should be placed in the non-significant area of the video frame to avoid blocking the effective content of the video, the embodiment of the present application needs to distinguish the significant area and the non-significant area of each frame in the video. At the same time, considering that the significant area and the non-significant area of each frame in the same lens unit are basically the same, while the significant areas of different lens units may be very different, the embodiment of the present application first needs to segment the target video into different lens units. Subsequently, the video content is enhanced in units of lens units. In the specific process of video segmentation, the target video is sampled at a preset sampling rate and sampling size, and only the sampled images are processed to avoid processing all the frame images in the video, which can effectively improve the efficiency of video segmentation. Then, each sampled image is processed through saliency detection, Gaussian blur and binarization operations to determine the effective content in each sampled image, which can avoid the influence of irrelevant content in the video, and only perform video segmentation based on the effective content, thereby improving the accuracy and rationality of video segmentation. Finally, according to the effective content in each sampled image, the boundary frame of each lens switch is determined, and the segmentation of the target video is achieved based on each boundary frame.
[0025] In a possible implementation, extracting corresponding prompt words from each of the subtitle information includes:
[0026] Replacing all pronouns in each of the subtitle information with corresponding entities to obtain each corresponding processed subtitle information;
[0027] Performing named entity recognition on each of the processed subtitle information to obtain each corresponding named entity;
[0028] Extracting keywords from each of the processed subtitle information to obtain corresponding keyword groups;
[0029] Using a preset imageability evaluation algorithm to evaluate the imageability of each of the keyword groups, and screening out a number of imageable words from each of the keyword groups;
[0030] Each of the named entities and each of the imageable words are converted into a corresponding prompt word according to a preset format.
[0031] The embodiment of the present application provides a method for extracting prompt words from subtitle information, and solves the problem of measuring the imageability of text in the field of video content enhancement technology and the problem of formulating prompt words for artificial intelligence image generation models. Regarding the imageability measurement problem, the imageability refers to a concept in psychology, that is, to what extent a text, whether a single word or a sentence, triggers the reader's mental image. For example, words such as "sunset" and "mountain" or sentences such as "the sun-drenched beach is presented in front of them, and the turquoise sea water gently laps the pearly white sand beach" are easy to associate with vivid images, so they have high visibility. However, words such as "loyalty" and "freedom" or sentences such as "because of her honesty, she is trusted by everyone" are relatively abstract, not easy to visualize, and have low imageability. Although the existing technology has studied word-level visibility databases and calculation methods to automatically estimate the visuality of words, the existing solutions have a limited number of words in the database on the one hand, which cannot meet the needs of practical applications, and on the other hand, there is a lack of imageability evaluation methods at the phrase and sentence levels. The embodiment of the present application replaces each of the subtitle information with a pronoun, extracts the corresponding keyword group from it, and then evaluates the imageability of each of the keyword groups based on the imageability evaluation algorithm, reflects the imageability of the text subtitle through the imageability of the keyword, and realizes the imageability evaluation at the phrase and sentence level. Regarding the problem of formulating prompt words for the artificial intelligence image generation model, the existing text-guided image generation model requires carefully crafted prompt words (Prompt) as model input, and such prompt words are often designed by users based on their own experience or model usage skills. This method is not suitable for the application scenario of the present application, because the present application is used to achieve automatic content enhancement for a large number of videos, and it is often impractical to rely on manual methods to produce a large number of prompt words. If the entire original text is input into the image generation model as a prompt word without modification or refinement, it may bring the problem of lack of focus, that is, the entire sentence may contain unnecessary details, distracting the model from the core concept to be imageable. In addition, compared with written language, the text obtained by voice transcription in the video is often colloquial, words may have multiple meanings depending on the context, and sentences may also lack a clear structure, which may lead to misunderstanding of intent or irrelevant results. Therefore, the present application extracts named entities and imageable words from the processed subtitle information, and converts each of the named entities and each of the imageable words into corresponding prompt words according to a preset format, thereby deepening the understanding of the video subtitle information by the image generation model, and making each enhanced image generated by the model more in line with the corresponding subtitle information. By solving the above two problems, the embodiment of the present application further improves the accuracy and success rate of video content enhancement.
[0032] In a possible implementation, allocating each enhanced image to each lens unit according to the start and end time of each lens unit and the timestamp corresponding to each subtitle information includes:
[0033] Determining the lens unit to which each of the enhanced images belongs according to the start and end time of each of the lens units;
[0034] Determining subtitle information corresponding to each of the enhanced images;
[0035] Determining the duration of each enhanced image in the lens unit according to the timestamp corresponding to each subtitle information;
[0036] Determine the key frame images of each of the lens units according to a preset rule, and determine the significant areas and non-significant areas in the key frame images;
[0037] Determining the display position of each enhanced image in the corresponding lens unit according to the size of each enhanced image and the size of the significant area and the non-significant area in each key frame image;
[0038] According to the duration and display position of each enhanced image, each enhanced image is allocated to each lens unit.
[0039] The embodiment of the present application provides a method for allocating enhanced images to each lens unit, which is used to solve the problem of synthetic display of visually enhanced content and video. In order to enhance the user's experience of browsing and searching for videos, it is necessary to synchronize the generated visually enhanced images with the video content according to the corresponding time. In order not to destroy the integrity of the video content playback, the embodiment of the present application intends to adopt an integrated display solution, that is, embedding the visually enhanced image into the non-significant area of the video frame. At the same time, in order to ensure the readability of the visually enhanced content, each image needs to be set to a certain duration (i.e., the time from the beginning of the video playback to the disappearance of the player screen), which will also result in multiple visually enhanced images at the same time. The embodiment of the present application first allocates all enhanced images to different lens units based on the start and end time of each lens unit and the timestamp corresponding to each subtitle information. Then, the duration and display position of each enhanced image in each lens unit are determined respectively to solve the problem of synthetic display of visually enhanced content and video, and further improve the accuracy of video content enhancement.
[0040] Further, when it is necessary to determine the display position of any first enhanced image among the enhanced images in the corresponding first lens unit, determining the display position of each enhanced image in the corresponding lens unit according to the size of each enhanced image and the size of the significant area and the non-significant area in each key frame image includes:
[0041] According to a second preset pixel size, dividing the first key frame image corresponding to the first lens unit into a plurality of image blocks to be enhanced;
[0042] Determining the significant pixels and the non-significant pixels in each of the image blocks to be enhanced according to the sizes of the significant areas and the non-significant areas of the first key frame image;
[0043] Scaling the first enhanced image at a preset scaling ratio to obtain a first scaled image;
[0044] Calculating a pixel threshold according to the number of pixels of the first zoomed image and a preset scale factor;
[0045] Traversing whether the preset positions of the image blocks to be enhanced meet the display condition, if the preset position of the current image block to be enhanced meets the display condition, determining the preset position of the current image block to be enhanced as the display position of the first zoomed image, and marking the preset position of the current image block to be enhanced as a significant area, wherein the display condition is that the number of significant pixels at the preset position of the image block to be enhanced is not more than the pixel threshold;
[0046] If each of the image blocks to be enhanced does not meet the display condition, the first scaled image is further scaled and the pixel threshold is updated, and then each of the image blocks to be enhanced is traversed again until there is an image block to be enhanced that meets the display condition.
[0047] The embodiment of the present application takes the first enhanced image as an example, and provides a method for determining the display position of the enhanced image. The key frame image of each lens unit is used as the representative picture of the lens unit, and the display position of the enhanced image in the key frame image is used as the display position of the enhanced image in the entire lens unit. Specifically, the key frame image is divided into a number of image blocks to be enhanced, and the enhanced image is limited to only select one of the image blocks to be enhanced as the display position, so that the display of the enhanced image in the video is more regular and orderly; in order to avoid the enhanced image from blocking the effective content of the original video, the present application further divides the pixels in the key frame image into significant pixels and non-significant pixels based on the division of significant areas and non-significant areas, and sets display conditions, so that the enhanced image selects the image blocks to be enhanced with fewer significant pixels as much as possible in the process of selecting the display position, thereby improving the user's viewing experience. Finally, considering that there may be a situation where there are more significant pixels in a key frame image, which will cause all the image blocks to be enhanced to fail to meet the display conditions, when this happens, the first scaled image is further scaled, and the pixel threshold is lowered and re-traversed, so as to avoid occlusion while ensuring that all enhanced images can be displayed in the original video.
[0048] In a possible implementation, when it is necessary to determine the display position of any first enhanced image among the enhanced images in the corresponding first lens unit, determining the display position of each enhanced image in the corresponding lens unit according to the size of each enhanced image and the size of the significant area and the non-significant area in each key frame image includes:
[0049] According to a second preset pixel size, dividing the first key frame image corresponding to the first lens unit into a plurality of image blocks to be enhanced;
[0050] Determining the significant pixels and the non-significant pixels in each of the image blocks to be enhanced according to the sizes of the significant areas and the non-significant areas of the first key frame image;
[0051] Scaling the first enhanced image at a preset scaling ratio to obtain a first scaled image;
[0052] Calculating a pixel threshold according to the number of pixels of the first zoomed image and a preset scale factor;
[0053] Randomly selecting any first image block to be enhanced among the image blocks to be enhanced, confirming whether a preset position of the first image block to be enhanced meets a display condition, and if the display condition is met, determining the preset position of the first image block to be enhanced as a display position of the first zoomed image, and marking the preset position of the first image block to be enhanced as a significant area, wherein the display condition is that the number of significant pixels at the preset position of the image block to be enhanced is not greater than the pixel threshold;
[0054] If, after a preset number of random selections, none of the selected image blocks to be enhanced meet the display condition, the first scaled image is further scaled and the pixel threshold is updated, and then a preset number of random selections are performed again until there is an image block to be enhanced that meets the display condition.
[0055] The embodiment of the present application takes the first enhanced image as an example and provides another method for determining the display position of the enhanced image. The difference is that the embodiment of the present application does not traverse all the image blocks to be enhanced, but randomly selects the image blocks to be enhanced to detect the display conditions, which further improves the execution efficiency of the algorithm.
[0056] In a second aspect, accordingly, the present application provides a video content enhancement system, including an acquisition module, a segmentation module, a speech recognition module, a prompt word extraction module, an image generation module, an image allocation module, and a combination module;
[0057] Wherein, the acquisition module is used to acquire the target video to be enhanced;
[0058] The segmentation module is used to segment the target video into a plurality of shot units, each of which includes a plurality of consecutive frames with the same shooting angle;
[0059] The speech recognition module is used to extract a plurality of subtitle information and a timestamp corresponding to each of the subtitle information from the target video using a preset speech recognition tool;
[0060] The prompt word extraction module is used to extract the corresponding prompt words from each of the subtitle information;
[0061] The image generation module is used to input each of the prompt words into a preset image generation model, so that the image generation model generates an enhanced image corresponding to each of the prompt words, wherein the image generation model is constructed based on artificial intelligence generation technology;
[0062] The image allocation module is used to allocate each enhanced image to each lens unit according to the start and end time of each lens unit and the timestamp corresponding to each subtitle information;
[0063] The combining module is used to recombine the lens units to obtain enhanced video.
[0064] In a third aspect, the present application provides a terminal, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the video content enhancement method described in the present application when executing the computer program.
[0065] In a fourth aspect, the present application provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the video content enhancement method described in the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0066] Figure 1 : It is a schematic diagram of the thumbnail preview function in the prior art.
[0067] Figure 2 : It is a flowchart of the video content summary and quick overview method in the prior art.
[0068] Figure 3 : Schematic diagram of the principle of video content structuring and visual browsing method based on multimodal information.
[0069] Figure 4 : Schematic diagram of the principle of real-time enhancement system of video content based on retrieval.
[0070] Figure 5 : It is a schematic diagram comparing the prior art and the technical solution of the present application.
[0071] Figure 6 : A flow chart of a video content enhancement method provided in an embodiment of the present application.
[0072] Figure 7 : A schematic diagram of the overall process of a video content enhancement method provided in an embodiment of the present application.
[0073] Figure 8 : A schematic diagram of the process of performing image processing on each sampling frame in a video content enhancement method provided in an embodiment of the present application.
[0074] Fig. 9 : A flowchart of extracting subtitle information and performing imageability evaluation in a video content enhancement method provided in an embodiment of the present application.
[0075] Fig.10 : A schematic diagram of the effect of enhancing a video using a video content enhancement method provided in this application.
[0076] Fig.11 : A schematic diagram of a user interface based on a video content enhancement method provided in an embodiment of the present application.
[0077] Fig.12 : A schematic diagram of the effect of a user interface provided in an embodiment of the present application at a certain moment.
[0078] Fig.13 : A structural diagram of a video content enhancement system provided in an embodiment of the present application. DETAILED DESCRIPTION
[0079] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0080] It should be noted that the step numbers in the text are only for the convenience of explanation of the specific embodiments and do not serve to limit the order in which the steps are executed. In the description of this application, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.
[0081] Throughout this specification, the video content enhancement mentioned in this specification refers to the generation of several enhanced images related to the video content by analyzing and summarizing the video content, and combining them with the original video to enhance the content of the original video. For example, most of the current mainstream online video playback platforms provide a thumbnail preview function in the user interface. Figure 1 As shown in , in the interactive content search and browsing scheme based on thumbnails on YouTube video sharing and social media platforms, when the user slides the mouse on the video timeline, the system will display the current video frame in the form of thumbnails, thereby helping the user locate the content of interest and the corresponding time node. In addition, existing methods also summarize and conclude the video content by analyzing the semantic information of the video content, and provide a quick overview of the video in visual forms such as storyboards or content highlights, such as Figure 2 In addition to video visual information, the language information contained in speech is also often used as an analysis object for video content structuring and visualization. For example, by performing topic boundary detection on the text obtained by speech recognition, the video can be divided into a series of semantic units, which can form useful structural and visual clues through visualization methods such as word clouds to assist users in browsing and locating video content. Figure 3The above methods mainly rely on the video content itself to provide visual clues. Figure 1 In the narrative videos shown in , since most of the video images are shots of the narrator, this type of visual content often plays a very limited role in helping users quickly browse and locate the topic being narrated. Therefore, existing research has also explored how to enhance video content with the help of external information. For example, Figure 4 As shown in the figure, Visual Caption introduces a real-time system for video conference content enhancement and proposes a retrieval-based visual enhancement method to improve the efficiency and experience of users browsing videos. The system extracts query text from video captions, then retrieves relevant images from local albums or online repositories using Microsoft Bing Image Search API and synchronizes them with speech to achieve real-time content enhancement, such as Figure 5 (a). However, this method relies heavily on the existing image library. For complex or rare search terms, it is very likely that relevant images cannot be matched or the text semantic correlation between the image and the query is not high. Therefore, the present application provides a video content enhancement method, system, terminal and storage medium, which uses artificial intelligence content generation (AIGC) technology to automatically generate images from the natural language obtained by speech recognition transcription in the video as visual enhancement clues to assist video browsing, content search, etc. Compared with retrieval-based methods, the present application can generate comprehensive and accurate content-enhanced images for videos without being constrained by local image databases, effectively overcoming the one-sidedness or inability to match the visual enhancement content obtained based on retrieval in the prior art solutions. Figure 5 As shown in (b), the images obtained using the generative technique are more semantically consistent with the text prompt words than the images obtained by retrieval.
[0082] Embodiment 1:
[0083] like Figure 6 As shown, embodiment 1 provides a video content enhancement method, including steps S1-S7:
[0084] Step S1, obtaining a target video to be enhanced;
[0085] Step S2, dividing the target video into a plurality of shot units, each of which includes a plurality of consecutive frames with the same shooting angle;
[0086] Step S3, using a preset speech recognition tool to extract a number of subtitle information and a timestamp corresponding to each of the subtitle information from the target video;
[0087] Step S4, extracting corresponding prompt words from each of the subtitle information;
[0088] Step S5, inputting each of the prompt words into a preset image generation model, so that the image generation model generates an enhanced image corresponding to each of the prompt words, wherein the image generation model is constructed based on artificial intelligence generation technology;
[0089] Step S6, allocating each of the enhanced images to each of the lens units according to the start and end time of each of the lens units and the timestamp corresponding to each of the subtitle information;
[0090] Step S7: Recombine the lens units to obtain enhanced video.
[0091] In a preferred embodiment, a video visual information processing module, a natural language processing module, and a content enhanced image generation and assembly module are set to implement steps S2-S7, wherein the video visual information processing module is used to implement step S2, the natural language processing module is used to implement steps S3-S4, and the content enhanced image generation and assembly module is used to implement steps S5-S7. The overall flow chart is as follows: Figure 7 shown.
[0092] The embodiment of the present application provides a method for enhancing video content, which automatically generates an enhanced image related to the current video content by decomposing the original video shots and performing voice recognition, and distributes it to the corresponding shots of the original video as a visual enhancement clue to assist video browsing, content search, etc. Compared with the retrieval-based method in the existing video enhancement technology, the embodiment of the present application can generate a more comprehensive and accurate content-enhanced image for the video, and is not constrained by the local image database, effectively overcoming the technical problems such as the one-sidedness or inability to match the visual enhancement content obtained in the existing retrieval-based technical solution, and improving the accuracy and success rate of video content enhancement.
[0093] In a possible implementation, in step S2, dividing the target video into a plurality of shot units includes:
[0094] Sampling the target video at a preset sampling rate and sampling size to obtain a plurality of sampled images;
[0095] Using a preset image saliency detection algorithm to perform saliency detection on each of the sampled images to obtain corresponding grayscale images;
[0096] Performing Gaussian blur and binarization operations on each of the grayscale images to obtain corresponding binary images;
[0097] Comparing the content of each binary image frame with the content of the next binary image frame, and determining a plurality of boundary frames for shot switching according to the comparison results of the binary images;
[0098] The target video is divided into a plurality of shot units according to each boundary frame.
[0099] Furthermore, the step of comparing the content of each binary image frame with the content of the next binary image frame, and determining a plurality of shot switching boundary frames according to the comparison results of the binary images, includes:
[0100] Dividing each of the binary images into a plurality of image blocks according to a first preset pixel size;
[0101] Performing an XOR operation on a number of image blocks in each frame of the binary image and a number of image blocks at the same position in the next frame of the binary image, and summing the results of the XOR operation in units of image blocks to obtain similarity values corresponding to each image block;
[0102] The similarity values of each image block in each binary image are retrieved in turn. If the similarity value corresponding to a current image block is greater than a first threshold, the current image block is marked as an activated block. If the proportion of activated blocks in a current binary image is greater than a second threshold, the current binary image is marked as a boundary frame for the current shot switching.
[0103] In a preferred embodiment, the specific process of step S2 is:
[0104] First, in order to improve the efficiency of shot segmentation, the given video is sampled at a sampling rate of 2 frames / second. If the frame width is greater than 224 pixels and the height is greater than 128 pixels, the size of the sampled frame is scaled to an image with a width of 224 pixels and a height of 128 pixels;
[0105] Subsequently, the MBD algorithm (or other similar image saliency detection algorithms) is used to perform saliency detection on the sampled frames, and the obtained grayscale image is subjected to a Gaussian blur operation with a kernel size of 9*9. The built-in threshold adaptive method in the OpenCV library is then used to binarize the grayscale image after Gaussian blur. The image processing process for each sampled frame is as follows: Figure 8 As shown;
[0106] Next, the binary image is divided into 16*16 pixel blocks. For a block of the binary image of the current sampling frame, it is XORed with the block at the same position in the binary image of the next frame adjacent to it (the same pixel value is 0, and different pixel values are 1), and then summed. If the sum is greater than 64, the block is marked as an active block. If the proportion of active blocks in all blocks of the current frame is greater than 25%, it is considered that the boundary of the video shot switching is detected, and the next frame is recorded as the first frame of the new shot;
[0107] For all detected shots, the sampled frame in the middle of the shot is extracted as the key frame of the shot.
[0108] The embodiment of the present application provides a method for segmenting a target video. Since the generated visual enhancement image should be placed in the non-significant area of the video frame to avoid blocking the effective content of the video, the embodiment of the present application needs to distinguish the significant area and the non-significant area of each frame in the video. At the same time, considering that the significant area and the non-significant area of each frame in the same lens unit are basically the same, while the significant areas of different lens units may be very different, the embodiment of the present application first needs to segment the target video into different lens units. Subsequently, the video content is enhanced in units of lens units. In the specific process of video segmentation, the target video is sampled at a preset sampling rate and sampling size, and only the sampled images are processed to avoid processing all the frame images in the video, which can effectively improve the efficiency of video segmentation. Then, each sampled image is processed through saliency detection, Gaussian blur and binarization operations to determine the effective content in each sampled image, which can avoid the influence of irrelevant content in the video, and only perform video segmentation based on the effective content, thereby improving the accuracy and rationality of video segmentation. Finally, according to the effective content in each sampled image, the boundary frame of each lens switch is determined, and the segmentation of the target video is achieved based on each boundary frame.
[0109] In a possible implementation, in step S4, extracting corresponding prompt words from each of the subtitle information includes:
[0110] Replacing all pronouns in each of the subtitle information with corresponding entities to obtain each corresponding processed subtitle information;
[0111] Performing named entity recognition on each of the processed subtitle information to obtain each corresponding named entity;
[0112] Extracting keywords from each of the processed subtitle information to obtain corresponding keyword groups;
[0113] Using a preset imageability evaluation algorithm to evaluate the imageability of each of the keyword groups, and screening out a number of imageable words from each of the keyword groups;
[0114] Each of the named entities and each of the imageable words are converted into a corresponding prompt word according to a preset format.
[0115] In a preferred embodiment, the specific process of steps S3-S4 is:
[0116] First, if Fig. 9 As shown, the WhisperX speech recognition tool is used to extract subtitles from the video (such as Fig. 9As shown in the lower left corner, each subtitle contains the corresponding timestamp and text). Then, use the Spacy reference resolution tool to replace all pronouns in the subtitle text with their corresponding entities. Subsequently, named entity recognition is performed on the processed subtitle text. In order to make the generated visual enhancement content relevant to the video theme, this application then uses KBIR or other tools to extract keywords (groups) from each subtitle. Finally, the imageability evaluation algorithm is used to evaluate the imageability of the keywords. If a keyword (group) scores higher than a predetermined threshold (for example, 5), it is regarded as a highly imageable word and is retained for prompt word formulation.
[0117] Then, for the named entities and highly imageable words obtained in the above steps, we use the following format as the prompt word for visually enhanced image generation: "A photo of KW". Among them, KW represents a named entity and a highly imageable word. The image generation model adopted in this application is Stable Diffusion (version: xl-base-1.0). It should be noted that other text-guided image generation models can also be used for visually enhanced image generation. For named entities, an alternative is to directly use the Wikipedia image query API and directly use the cover image of the entry page corresponding to the named entity as the visually enhanced image. At the same time, the keywords used to generate the image will be embedded in the bottom 1 / 4 of the obtained enhanced image to help users browse.
[0118] The embodiment of the present application provides a method for extracting prompt words from subtitle information, and solves the problem of measuring the imageability of text in the field of video content enhancement technology and the problem of formulating prompt words for artificial intelligence image generation models. Regarding the imageability measurement problem, the imageability refers to a concept in psychology, that is, to what extent a text, whether a single word or a sentence, triggers the reader's mental image. For example, words such as "sunset" and "mountain" or sentences such as "the sun-drenched beach is presented in front of them, and the turquoise sea water gently laps the pearly white sand beach" are easy to associate with vivid images, so they have high visibility. However, words such as "loyalty" and "freedom" or sentences such as "because of her honesty, she is trusted by everyone" are relatively abstract, not easy to visualize, and have low imageability. Although the existing technology has studied word-level visibility databases and calculation methods to automatically estimate the visuality of words, the existing solutions have a limited number of words in the database on the one hand, which cannot meet the needs of practical applications, and on the other hand, there is a lack of imageability evaluation methods at the phrase and sentence levels. The embodiment of the present application replaces each of the subtitle information with a pronoun, extracts the corresponding keyword group from it, and then evaluates the imageability of each of the keyword groups based on the imageability evaluation algorithm, reflects the imageability of the text subtitle through the imageability of the keyword, and realizes the imageability evaluation at the phrase and sentence level. Regarding the problem of formulating prompt words for the artificial intelligence image generation model, the existing text-guided image generation model requires carefully crafted prompt words (Prompt) as model input, and such prompt words are often designed by users based on their own experience or model usage skills. This method is not suitable for the application scenario of the present application, because the present application is used to achieve automatic content enhancement for a large number of videos, and it is often impractical to rely on manual methods to produce a large number of prompt words. If the entire original text is input into the image generation model as a prompt word without modification or refinement, it may bring the problem of lack of focus, that is, the entire sentence may contain unnecessary details, distracting the model from the core concept to be imageable. In addition, compared with written language, the text obtained by voice transcription in the video is often colloquial, words may have multiple meanings depending on the context, and sentences may also lack a clear structure, which may lead to misunderstanding of intent or irrelevant results. Therefore, the present application extracts named entities and imageable words from the processed subtitle information, and converts each of the named entities and each of the imageable words into corresponding prompt words according to a preset format, thereby deepening the understanding of the video subtitle information by the image generation model, and making each enhanced image generated by the model more in line with the corresponding subtitle information. By solving the above two problems, the embodiment of the present application further improves the accuracy and success rate of video content enhancement.
[0119] In a possible implementation, in step S5, allocating each enhanced image to each lens unit according to the start and end time of each lens unit and the timestamp corresponding to each subtitle information includes:
[0120] Determining the lens unit to which each of the enhanced images belongs according to the start and end time of each of the lens units;
[0121] Determining subtitle information corresponding to each of the enhanced images;
[0122] Determining the duration of each enhanced image in the lens unit according to the timestamp corresponding to each subtitle information;
[0123] Determine the key frame images of each of the lens units according to a preset rule, and determine the significant areas and non-significant areas in the key frame images;
[0124] Determining the display position of each enhanced image in the corresponding lens unit according to the size of each enhanced image and the size of the significant area and the non-significant area in each key frame image;
[0125] According to the duration and display position of each enhanced image, each enhanced image is allocated to each lens unit.
[0126] In a preferred embodiment, after obtaining the visually enhanced image corresponding to each named entity and highly imageable word, it is necessary to place it on a non-salient area of the video frame and display it synchronously during video playback or content search. As mentioned above, the present application divides the video into a series of lens units, and encapsulates the visually enhanced image with the lens as the basic unit. Specifically, assuming that the start and end times of a lens are Ts and Te respectively, all subtitles whose start times are in the interval [Ts, Te] are first assigned to the lens. Correspondingly, the duration of the visually enhanced image corresponding to each subtitle is from the start time of the subtitle to the start time of the next subtitle containing visually enhanced content. If the start time of the next subtitle containing visually enhanced content is later than the end time of the lens, the end time of the visually enhanced image of the subtitle is set to the end time of the lens. The duration of all visually enhanced images in an imageable subtitle is set to the same, and all imageable subtitles contained in the lens are assembled with the salient area of the key frame of the lens as a constraint.
[0127] The embodiment of the present application provides a method for allocating enhanced images to each lens unit, which is used to solve the problem of synthetic display of visually enhanced content and video. In order to enhance the user's experience of browsing and searching for videos, it is necessary to synchronize the generated visually enhanced images with the video content according to the corresponding time. In order not to destroy the integrity of the video content playback, the embodiment of the present application intends to adopt an integrated display solution, that is, embedding the visually enhanced image into the non-significant area of the video frame. At the same time, in order to ensure the readability of the visually enhanced content, each image needs to be set to a certain duration (i.e., the time from the beginning of the video playback to the disappearance of the player screen), which will also result in multiple visually enhanced images at the same time. The embodiment of the present application first allocates all enhanced images to different lens units based on the start and end time of each lens unit and the timestamp corresponding to each subtitle information. Then, the duration and display position of each enhanced image in each lens unit are determined respectively to solve the problem of synthetic display of visually enhanced content and video, and further improve the accuracy of video content enhancement.
[0128] Further, when it is necessary to determine the display position of any first enhanced image among the enhanced images in the corresponding first lens unit, determining the display position of each enhanced image in the corresponding lens unit according to the size of each enhanced image and the size of the significant area and the non-significant area in each key frame image includes:
[0129] According to a second preset pixel size, dividing the first key frame image corresponding to the first lens unit into a plurality of image blocks to be enhanced;
[0130] Determining the significant pixels and the non-significant pixels in each of the image blocks to be enhanced according to the sizes of the significant areas and the non-significant areas of the first key frame image;
[0131] Scaling the first enhanced image at a preset scaling ratio to obtain a first scaled image;
[0132] Calculating a pixel threshold according to the number of pixels of the first zoomed image and a preset scale factor;
[0133] Traversing whether the preset positions of the image blocks to be enhanced meet the display condition, if the preset position of the current image block to be enhanced meets the display condition, determining the preset position of the current image block to be enhanced as the display position of the first zoomed image, and marking the preset position of the current image block to be enhanced as a significant area, wherein the display condition is that the number of significant pixels at the preset position of the image block to be enhanced is not more than the pixel threshold;
[0134] If each of the image blocks to be enhanced does not meet the display condition, the first scaled image is further scaled and the pixel threshold is updated, and then each of the image blocks to be enhanced is traversed again until there is an image block to be enhanced that meets the display condition.
[0135] The embodiment of the present application takes the first enhanced image as an example, and provides a method for determining the display position of the enhanced image. The key frame image of each lens unit is used as the representative picture of the lens unit, and the display position of the enhanced image in the key frame image is used as the display position of the enhanced image in the entire lens unit. Specifically, the key frame image is divided into a number of image blocks to be enhanced, and the enhanced image is limited to only select one of the image blocks to be enhanced as the display position, so that the display of the enhanced image in the video is more regular and orderly; in order to avoid the enhanced image from blocking the effective content of the original video, the present application further divides the pixels in the key frame image into significant pixels and non-significant pixels based on the division of significant areas and non-significant areas, and sets display conditions, so that the enhanced image selects the image blocks to be enhanced with fewer significant pixels as much as possible in the process of selecting the display position, thereby improving the user's viewing experience. Finally, considering that there may be a situation where there are more significant pixels in a key frame image, which will cause all the image blocks to be enhanced to fail to meet the display conditions, when this happens, the first scaled image is further scaled, and the pixel threshold is lowered and re-traversed, so as to avoid occlusion while ensuring that all enhanced images can be displayed in the original video.
[0136] In a possible implementation, when it is necessary to determine the display position of any first enhanced image among the enhanced images in the corresponding first lens unit, determining the display position of each enhanced image in the corresponding lens unit according to the size of each enhanced image and the size of the significant area and the non-significant area in each key frame image includes:
[0137] According to a second preset pixel size, dividing the first key frame image corresponding to the first lens unit into a plurality of image blocks to be enhanced;
[0138] Determining the significant pixels and the non-significant pixels in each of the image blocks to be enhanced according to the sizes of the significant areas and the non-significant areas of the first key frame image;
[0139] Scaling the first enhanced image at a preset scaling ratio to obtain a first scaled image;
[0140] Calculating a pixel threshold according to the number of pixels of the first zoomed image and a preset scale factor;
[0141] Randomly selecting any first image block to be enhanced among the image blocks to be enhanced, confirming whether a preset position of the first image block to be enhanced meets a display condition, and if the display condition is met, determining the preset position of the first image block to be enhanced as a display position of the first zoomed image, and marking the preset position of the first image block to be enhanced as a significant area, wherein the display condition is that the number of significant pixels at the preset position of the image block to be enhanced is not greater than the pixel threshold;
[0142] If, after a preset number of random selections, none of the selected image blocks to be enhanced meet the display condition, the first scaled image is further scaled and the pixel threshold is updated, and then a preset number of random selections are performed again until there is an image block to be enhanced that meets the display condition.
[0143] The embodiment of the present application takes the first enhanced image as an example and provides another method for determining the display position of the enhanced image. The difference is that the embodiment of the present application does not traverse all the image blocks to be enhanced, but randomly selects the image blocks to be enhanced to detect the display conditions, which further improves the execution efficiency of the algorithm.
[0144] In a preferred embodiment, the display position of each enhanced image in the corresponding lens unit is determined according to the size of each enhanced image and the size of the significant area and the non-significant area in each key frame image, and the specific process is:
[0145] 1) Assume that the size of the lens key frame is 224 pixels wide and 128 pixels high, and divide it into blocks of 16 pixels * 16 pixels, a total of 14 * 8 = 112 macroblocks;
[0146] 2) The width and height of all enhanced images (the original image size is 512 pixels * 512 pixels) are first scaled to 1 / 2 of the key frame height, that is, 64 pixels;
[0147] 3) For the enhanced image with insertion, from left to right and from top to bottom, traverse the upper left corner vertex of the macroblock in turn as the upper left corner vertex position of the enhanced image in the key frame. If the number of significant pixels in the area where the enhanced image is located is no more than 1 / 10 of the number of pixels in the enhanced image, the insertion is considered successful, and the traversal of the remaining macroblocks is stopped, and all pixels in the key frame area where the current enhanced image is located are marked as significant pixels; if it is unsuccessful, the width and height of the enhanced image are reduced by 8 pixels each, and step 3) is repeated until it is successful;
[0148] The above steps are repeated until all visually enhanced images of the duration are encapsulated in the corresponding areas of the video key frames.
[0149] In addition to the traversal method described in step 3) above, an alternative solution is to randomly select one from the 112 macroblocks for insertion. If the loop fails after 20 times, the width and height of the enhanced image are reduced by 8 pixels each, and the steps are repeated until success.
[0150] After completing the above steps, if the current video playback time is within the duration of the visual enhancement image, the system will superimpose the visual enhancement image to the corresponding position area of the video player in the form of a floating layer (when the video screen scale changes, the visual enhancement image and its area will be scaled proportionally). The video content enhancement effect is as follows: Fig.10 shown.
[0151] In a preferred embodiment, Fig.11 As shown, based on the video content enhancement method proposed in the present application, this embodiment also provides a user interface. The user interface is mainly composed of four panels: the upper left area is an embedded video player. When playing a video from a specific time, if there is a visually enhanced image generated, it will be displayed in a non-salient area of the current video frame. The upper right area is a panel displaying video subtitles, in which keywords rated as having high imaging scores are highlighted with red rectangular frames. In the middle of the user interface is a panel for visualizing detected lens units. Each lens is represented by a circle, the radius of the circle is proportional to the length of the lens, and the background color of the circle indicates the imageability of the subtitles associated with the lens (the darker the imageability, the higher the imageability). At the bottom of the user interface is a visual storyboard composed of video key frames. When the user hovers the mouse over a key frame, the corresponding circle in the visualization panel and the related subtitles in the upper right area of the user interface will be highlighted. As shown Fig.12 As shown, when the video is played to 48 minutes and 49 seconds, two enhanced images generated by "physics" and "information theory" are displayed in the corresponding area of the video.
[0152] Embodiment 2:
[0153] Correspondingly, such as Fig.13 As shown, the second embodiment provides a video content enhancement system, including an acquisition module 10, a segmentation module 20, a speech recognition module 30, a prompt word extraction module 40, an image generation module 50, an image allocation module 60 and a combination module 70;
[0154] Wherein, the acquisition module 10 is used to acquire the target video to be enhanced;
[0155] The segmentation module 20 is used to segment the target video into a plurality of shot units, each of which includes a plurality of consecutive frames with the same shooting angle;
[0156] The speech recognition module 30 is used to extract a plurality of subtitle information and a timestamp corresponding to each of the subtitle information from the target video using a preset speech recognition tool;
[0157] The prompt word extraction module 40 is used to extract the corresponding prompt words from each of the subtitle information;
[0158] The image generation module 50 is used to input each of the prompt words into a preset image generation model, so that the image generation model generates an enhanced image corresponding to each of the prompt words, wherein the image generation model is constructed based on artificial intelligence generation technology;
[0159] The image allocation module 60 is used to allocate each enhanced image to each lens unit according to the start and end time of each lens unit and the timestamp corresponding to each subtitle information;
[0160] The combining module 70 is used to recombine the lens units to obtain enhanced video.
[0161] The embodiment of the present application provides a video content enhancement system, which automatically generates an enhanced image related to the current video content by decomposing the original video shots and voice recognition, and distributes it to the corresponding shots of the original video, as a visual enhancement clue to assist video browsing, content search, etc. Compared with the retrieval-based method in the existing video enhancement technology, the embodiment of the present application can generate a more comprehensive and accurate content enhancement image for the video, and is not constrained by the local image database, effectively overcoming the technical problems such as the one-sidedness or inability to match the visual enhancement content obtained in the existing retrieval-based technical solution, and improving the accuracy and success rate of video content enhancement.
[0162] The more detailed working principle and step flow of this embodiment can refer to, but are not limited to, the relevant records of Embodiment 1.
[0163] Embodiment three:
[0164] Embodiment 3 provides a terminal, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the video content enhancement method described in the present application when executing the computer program.
[0165] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the terminal, and various interfaces and lines are used to connect various parts of the entire terminal.
[0166] The memory can be used to store the computer program, and the processor realizes various functions of the terminal by running or executing the computer program stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.
[0167] Embodiment 4:
[0168] Embodiment 4 provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the video content enhancement method described in the present application.
[0169] Wherein, if the module integrated in the video content enhancement method is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc.
[0170] The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the scope of protection of the present application. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of protection of the present application.
Claims
1. A video content enhancement method, characterized in that: include: Obtain the target video to be enhanced; Dividing the target video into a plurality of shot units, each of which includes a plurality of consecutive frames with the same shooting angle; Extracting a plurality of subtitle information and a timestamp corresponding to each of the subtitle information from the target video using a preset speech recognition tool; Extracting corresponding prompt words from each of the subtitle information; Inputting each of the prompt words into a preset image generation model, so that the image generation model generates an enhanced image corresponding to each of the prompt words, wherein the image generation model is constructed based on artificial intelligence generation technology; Allocating each of the enhanced images to each of the lens units according to the start and end time of each of the lens units and the timestamp corresponding to each of the subtitle information; The various lens units are recombined to obtain enhanced video.
2. A video content enhancement method as claimed in claim 1, characterized in that: The step of dividing the target video into a plurality of shot units comprises: Sampling the target video at a preset sampling rate and sampling size to obtain a plurality of sampled images; Using a preset image saliency detection algorithm to perform saliency detection on each of the sampled images to obtain corresponding grayscale images; Performing Gaussian blur and binarization operations on each of the grayscale images to obtain corresponding binary images; Comparing the content of each binary image frame with the content of the next binary image frame, and determining a plurality of boundary frames for shot switching according to the comparison results of the binary images; The target video is divided into a plurality of shot units according to each boundary frame.
3. A video content enhancement method as claimed in claim 2, characterized in that: The step of comparing the content of each binary image frame with the content of the next binary image frame, and determining a plurality of shot switching boundary frames according to the comparison results of the binary images, comprises: Dividing each of the binary images into a plurality of image blocks according to a first preset pixel size; Performing an XOR operation on a number of image blocks in each frame of the binary image and a number of image blocks at the same position in the next frame of the binary image, and summing the results of the XOR operation in units of image blocks to obtain similarity values corresponding to each image block; The similarity values of each image block in each binary image are retrieved in turn. If the similarity value corresponding to a current image block is greater than a first threshold, the current image block is marked as an activated block. If the proportion of activated blocks in a current binary image is greater than a second threshold, the current binary image is marked as a boundary frame for the current shot switching.
4. A video content enhancement method as claimed in claim 1, characterized in that: The extracting corresponding prompt words from each of the subtitle information includes: Replacing all pronouns in each of the subtitle information with corresponding entities to obtain each corresponding processed subtitle information; Performing named entity recognition on each of the processed subtitle information to obtain each corresponding named entity; Extracting keywords from each of the processed subtitle information to obtain corresponding keyword groups; Using a preset imageability evaluation algorithm to evaluate the imageability of each of the keyword groups, and screening out a number of imageable words from each of the keyword groups; Each of the named entities and each of the imageable words are converted into a corresponding prompt word according to a preset format.
5. A video content enhancement method as claimed in claim 1, characterized in that: The allocating each of the enhanced images to each of the lens units according to the start and end time of each of the lens units and the timestamp corresponding to each of the subtitle information comprises: Determining the lens unit to which each of the enhanced images belongs according to the start and end time of each of the lens units; Determining subtitle information corresponding to each of the enhanced images; Determining the duration of each enhanced image in the lens unit according to the timestamp corresponding to each subtitle information; Determine the key frame images of each of the lens units according to a preset rule, and determine the significant areas and non-significant areas in the key frame images; Determining the display position of each enhanced image in the corresponding lens unit according to the size of each enhanced image and the size of the significant area and the non-significant area in each key frame image; According to the duration and display position of each enhanced image, each enhanced image is allocated to each lens unit.
6. A video content enhancement method as claimed in claim 5, characterized in that: When it is necessary to determine the display position of any first enhanced image among the enhanced images in the corresponding first lens unit, determining the display position of each enhanced image in the corresponding lens unit according to the size of each enhanced image and the size of the significant area and the non-significant area in each key frame image includes: According to a second preset pixel size, dividing the first key frame image corresponding to the first lens unit into a plurality of image blocks to be enhanced; Determining the significant pixels and the non-significant pixels in each of the image blocks to be enhanced according to the sizes of the significant areas and the non-significant areas of the first key frame image; Scaling the first enhanced image at a preset scaling ratio to obtain a first scaled image; Calculating a pixel threshold according to the number of pixels of the first zoomed image and a preset scale factor; Traversing whether the preset positions of the image blocks to be enhanced meet the display condition, if the preset position of the current image block to be enhanced meets the display condition, determining the preset position of the current image block to be enhanced as the display position of the first zoomed image, and marking the preset position of the current image block to be enhanced as a significant area, wherein the display condition is that the number of significant pixels at the preset position of the image block to be enhanced is not more than the pixel threshold; If each of the image blocks to be enhanced does not meet the display condition, the first scaled image is further scaled and the pixel threshold is updated, and then each of the image blocks to be enhanced is traversed again until there is an image block to be enhanced that meets the display condition.
7. A video content enhancement method as claimed in claim 5, characterized in that: When it is necessary to determine the display position of any first enhanced image among the enhanced images in the corresponding first lens unit, determining the display position of each enhanced image in the corresponding lens unit according to the size of each enhanced image and the size of the significant area and the non-significant area in each key frame image includes: According to a second preset pixel size, dividing the first key frame image corresponding to the first lens unit into a plurality of image blocks to be enhanced; Determining the significant pixels and the non-significant pixels in each of the image blocks to be enhanced according to the sizes of the significant areas and the non-significant areas of the first key frame image; Scaling the first enhanced image at a preset scaling ratio to obtain a first scaled image; Calculating a pixel threshold according to the number of pixels of the first zoomed image and a preset scale factor; Randomly selecting any first image block to be enhanced among the image blocks to be enhanced, confirming whether a preset position of the first image block to be enhanced meets a display condition, and if the display condition is met, determining the preset position of the first image block to be enhanced as a display position of the first zoomed image, and marking the preset position of the first image block to be enhanced as a significant area, wherein the display condition is that the number of significant pixels at the preset position of the image block to be enhanced is not greater than the pixel threshold; If, after a preset number of random selections, none of the selected image blocks to be enhanced meet the display condition, the first scaled image is further scaled and the pixel threshold is updated, and then a preset number of random selections are performed again until there is an image block to be enhanced that meets the display condition.
8. A video content enhancement system, characterized in that: It includes an acquisition module, a segmentation module, a speech recognition module, a prompt word extraction module, an image generation module, an image allocation module and a combination module; Wherein, the acquisition module is used to acquire the target video to be enhanced; The segmentation module is used to segment the target video into a plurality of shot units, each of which includes a plurality of consecutive frames with the same shooting angle; The speech recognition module is used to extract a plurality of subtitle information and a timestamp corresponding to each of the subtitle information from the target video using a preset speech recognition tool; The prompt word extraction module is used to extract the corresponding prompt words from each of the subtitle information; The image generation module is used to input each of the prompt words into a preset image generation model, so that the image generation model generates an enhanced image corresponding to each of the prompt words, wherein the image generation model is constructed based on artificial intelligence generation technology; The image allocation module is used to allocate each enhanced image to each lens unit according to the start and end time of each lens unit and the timestamp corresponding to each subtitle information; The combining module is used to recombine the lens units to obtain enhanced video.
9. A terminal, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, a video content enhancement method as claimed in any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute a video content enhancement method as described in any one of claims 1 to 7.
Citation Information
Patent Citations
Video key frame extraction method, system and device based on multi-view features
CN110472484A
Multi-engine intelligent question answering system for multi-type knowledge base
CN115238101A
Video content understanding method and system, computer equipment and storage medium
CN116645624A
Video synthesis method, device and equipment, reading kiosk and storage medium
CN117939259A
Camera content anomaly detection method and device, equipment and storage medium
CN118365608A
