Video processing method and device, electronic equipment, storage medium and program product

Abstract

The invention discloses a video processing method and device, electronic equipment, a storage medium and a program product, relates to the technical field of information, and aims to solve the problem of poor interaction between related video processing technologies and users. The method comprises the following steps: acquiring a cartoon work corresponding to a target video; receiving search input of a user for the cartoon works; according to the search information and a description text corresponding to a key frame in a pre-generated target video, analyzing the target video to obtain a video analysis result; and outputting interaction result information of the search information based on the video analysis result. Therefore, the user can perform interactive operations such as information search and question asking for the cartoon works after video conversion, so that the interactivity between the video processing technology and the user is enriched and improved.

Description

Technical Field

[0001] The present application relates to the field of information technology, and in particular to a video processing method, device, electronic device, storage medium and program product. Background Art

[0002] In order to provide viewers with a unique reading experience and enrich the dissemination channels of video content, some technologies have emerged that adapt video content such as movies into comic works. However, related technologies usually only make simple understanding of the video content and comic style transfer processing, and have poor interactivity with users. Summary of the invention

[0003] The embodiments of the present application provide a video processing method, device, electronic device, storage medium and program product to solve the problem of poor interactivity between related video processing technologies and users.

[0004] In a first aspect, an embodiment of the present application provides a video processing method, including:

[0005] Get the comics corresponding to the target video;

[0006] receiving a search input from a user for the comic work, wherein the search information corresponding to the search input is search information or question information for content in the comic work;

[0007] Parsing the target video according to the search information and the pre-generated description text corresponding to the key frames in the target video to obtain a video parsing result;

[0008] Output interactive result information of the search information based on the video analysis result.

[0009] The step of obtaining the comic work corresponding to the target video includes:

[0010] Extracting key frames from the target video to obtain key frames in the target video;

[0011] Perform comic style conversion on the key frame to obtain a comic image corresponding to the key frame;

[0012] Acquire text information corresponding to the key frame, wherein the text information at least includes the description text;

[0013] Generate a comic work corresponding to the target video based on the comic image and text information corresponding to the key frame.

[0014] Optionally, obtaining text information corresponding to the key frame includes:

[0015] Extracting features from target information and the key frame of the target video respectively, wherein the target information includes at least one of the following: subtitle text, commentary text, audio, character name, and character image;

[0016] Performing feature fusion on each feature obtained by the feature extraction to obtain a fused feature;

[0017] Encoding the fused features to obtain global encoding features of the target video;

[0018] The global encoding feature is decoded to generate a description text corresponding to the key frame.

[0019] Optionally, parsing the target video according to the search information and the pre-generated description text corresponding to the key frame to obtain a video parsing result includes:

[0020] The search information and the description text corresponding to the key frame are input into a large language model LLM for natural language understanding to obtain a language understanding result, wherein the video parsing result includes the language understanding result.

[0021] Optionally, parsing the target video according to the search information and the pre-generated description text corresponding to the key frame to obtain a video parsing result further includes:

[0022] The language understanding result and the global encoding feature are input into an image generation model to obtain a target image, wherein the content represented by the target image is related to the content of the target video and the search information, and the video parsing result also includes the target image.

[0023] Optionally, the image generation model and the LLM are trained in the following manner:

[0024] Using the labeled image description text of the key frame to be trained, the information search pair and the image generation data as the training data set, wherein the information search pair includes the preset search information and the corresponding search result;

[0025] Inputting the image description text and the preset search information into an initial LLM for natural language understanding, and obtaining an actual understanding result output by the LLM;

[0026] Inputting the actual understanding result and the encoding features corresponding to the key frame to be trained into an initial image generation model to obtain actual generated image data output by the image generation model;

[0027] Calculating the cross entropy loss of the LLM language understanding and the accuracy loss of the information search according to the actual understanding result and the search result corresponding to the preset search information;

[0028] Calculating the adversarial loss of the image generated by the image generation model according to the actually generated image data and the image generation data;

[0029] The LLM and the image generation model are jointly trained according to the cross entropy loss, the accuracy loss and the adversarial loss to obtain the trained image generation model and the LLM.

[0030] Optionally, receiving a user's search input for the comic work includes:

[0031] Displaying the comic work and the information search box in the first area and the second area of ​​the web page respectively;

[0032] The search input performed by the user through the information search box is received, and the search information is the information input in the information search box.

[0033] Optionally, extracting key frames from the target video includes:

[0034] Eliminate the opening segment and the ending segment in the target video to obtain a first video;

[0035] Performing frame reduction processing on the first video to obtain a second video;

[0036] Comparing feature information of two adjacent video frames in the second video to determine a candidate key frame, wherein the feature information includes at least one of the following: a color histogram, a feature point, and an edge map;

[0037] Calculating a comprehensive evaluation value of the candidate key frame according to a difference between feature information of the candidate key frame and a video frame preceding the candidate key frame;

[0038] When the comprehensive evaluation value of the candidate key frame is higher than a preset threshold, the candidate key frame is determined to be a key frame.

[0039] Optionally, the text information also includes subtitles corresponding to the key frames;

[0040] Generating a comic work corresponding to the target video according to the comic image and text information corresponding to the key frame includes:

[0041] Adding the description text to the comic image corresponding to the key frame, and adding a dialog box including a dialogue text to the corresponding character in the comic image according to the subtitle;

[0042] The comic image after adding the description text and the dialog box is typeset using a preset typeset algorithm to obtain the comic work.

[0043] In a second aspect, an embodiment of the present application further provides a video processing device, including:

[0044] An acquisition module is used to acquire the comic works corresponding to the target video;

[0045] A receiving module, configured to receive a user's search input for the comic work, wherein the search information corresponding to the search input is search information or question information for content in the comic work;

[0046] A parsing module, used to parse the target video according to the search information and the pre-generated description text corresponding to the key frames in the target video to obtain a video parsing result;

[0047] An output module is used to output interaction result information of the search information based on the video analysis result.

[0048] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the video processing method described above when executing the computer program.

[0049] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the video processing method as described above are implemented.

[0050] In a fifth aspect, an embodiment of the present application provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps in the video processing method as described above.

[0051] In the embodiment of the present application, the comic work corresponding to the target video is obtained; the search input of the user for the comic work is received, and the search information corresponding to the search input is the search information or question information for the content in the comic work; according to the search information and the pre-generated description text corresponding to the key frame in the target video, the target video is parsed to obtain the video parsing result; based on the video parsing result, the interactive result information for the search information is output. In this way, the user can perform interactive operations such as information search and questioning for the comic work after the video conversion, thereby enriching and improving the interactivity between the video processing technology and the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0053] Figure 1 is a framework flow chart of the video processing method provided in the embodiment of the present application;

[0054] Figure 2 is a flow chart of a video processing method provided by an embodiment of the present application;

[0055] Figure 3 It is an architecture diagram of a multimodal video understanding model provided in an embodiment of the present application;

[0056] Figure 4 It is a schematic diagram of an interactive scenario in which a user asks questions to a comic work provided by an embodiment of the present application;

[0057] Figure 5 is a structural diagram of a video processing device provided in an embodiment of the present application;

[0058] Figure 6 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0060] In order to make the embodiments of the present application clearer, the relevant technical knowledge involved in the embodiments of the present application is first introduced as follows:

[0061] Existing video processing technologies lack interactivity with users, mainly manifested in the lack of intelligent interactive mode. Users cannot ask questions about difficult-to-understand plots, nor can they perform secondary creation. In addition, there are the following shortcomings:

[0062] 1) Movie key frame extraction has no adaptive function and lacks dynamic adjustment processing strategy according to video content, such as adjusting the key frame extraction frequency according to the dynamics of the scene, so that the final output is more in line with the emotions and dynamic changes of the original content.

[0063] 2) Without identifying and encoding the characters and scenes, it is impossible to ensure the uniformity of characters and styles in all pictures after conversion into animation.

[0064] 3) The technical solution lacks multi-modal video content understanding. For content description generation, only audio content or lines are used. For key frames without lines, no description is added, which affects the coherence of the entire plot.

[0065] 4) The existing technology does not perform special processing on actors, and cannot provide accurate description when generating lines or plot descriptions.

[0066] The embodiment of the present application mainly proposes a targeted solution to the above-mentioned technical problem of lack of interactivity with users, and solves the above-mentioned four existing technical defects respectively through some optional implementation methods. The complete solution framework process of the embodiment of the present application is as follows Figure 1 As shown, it mainly includes the following key steps:

[0067] 1. First, use a method based on the fusion of visual features and deep learning to achieve automatic frame extraction according to the video content.

[0068] 2. Train a multimodal video understanding model, with the input being the extracted key frames, movie network information, actor names and headshot information, subtitle information, etc., and the output being the description content of each key frame.

[0069] 3. Use a diffusion model algorithm that can recognize faces and buildings to convert each frame of animation into a comic style.

[0070] 4. Add corresponding text content to all comic frames and generate web version comics.

[0071] 5. Each movie content is equipped with a pre-trained model. Users have a search box on the web page where they can enter text to search or ask questions about the movie content, and generate new movie-related pictures based on the text content.

[0072] 6. Below the playing video, you can add the corresponding comic link to give users more viewing options.

[0073] The video processing method provided by each embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0074] See also Figure 2 , Figure 2 is a flow chart of a video processing method provided in an embodiment of the present application, such as Figure 2 As shown, the following steps are included:

[0075] Step 201: Obtain the comic work corresponding to the target video.

[0076] Specifically, the target video may be converted into a comic style in advance to generate a corresponding comic work. When needed, such as when a user requests to read the comic work, the comic work may be directly called, or the comic work corresponding to the existing target video may be directly obtained from the network.

[0077] The target video may be any video such as a movie, a documentary, a short video, or a live broadcast replay.

[0078] Optionally, the step 201 includes:

[0079] Extracting key frames from the target video to obtain key frames in the target video;

[0080] Perform comic style conversion on the key frame to obtain a comic image corresponding to the key frame;

[0081] Acquire text information corresponding to the key frame, wherein the text information at least includes the description text;

[0082] Generate a comic work corresponding to the target video based on the comic image and text information corresponding to the key frame.

[0083] In some embodiments, the target video may be subjected to steps such as key frame extraction, key frame understanding, comic style conversion processing, comic image captioning, and typesetting to generate a comic work corresponding to the target video.

[0084] First, key frames may be extracted from the target video, such as by using an intelligent frame extraction algorithm or other key frame extraction algorithms, so as to obtain a number of key frames in the target video. It can be understood that each key frame is a video frame with a large image difference in the target video;

[0085] Next, the image information of each key frame extracted from the target video can be understood. Specifically, it can be combined with other relevant target information of the target video, such as subtitles, commentary, audio, actor information, character information, etc. to understand the content in each key frame, such as understanding what the character is doing and saying in a certain key frame. Specifically, it can be implemented by using a video understanding model, an image recognition model, etc. Through this processing, the description text corresponding to each key frame can be obtained, that is, the description text of the image information and things in each key frame. The text information can include the description text and can also include the subtitles in each key frame.

[0086] The key frames may also be processed by comic style conversion to convert each key frame into a comic style image. For example, a pre-trained style transfer model such as AnimeGAN may be used to convert each key frame into an anime style comic image. It should be noted that a consistency loss may be added during the training of the style transfer model, that is, the consistency loss of two adjacent comic images generated by the model is calculated to ensure smooth style changes in adjacent frames. In addition, facial recognition and tracking technology may be used for two adjacent comic images to ensure consistency of the faces and clothing of the anime characters in adjacent frames.

[0087] Finally, the comic images and text information corresponding to the migrated key frames can be combined, and the final comic work can be generated by adding text and layout to each comic image frame, that is, the comic work generated based on the content of each key frame in the target video.

[0088] Among them, for the generation of descriptive text corresponding to each key frame: first, a pre-trained Transformer model can be used to generate a text description with role identification for each key frame, such as a descriptive text that "XX" handed "XX" a glass of water; and the subtitles corresponding to the key frame can be found in the subtitle file of the target video according to the timestamp of each key frame.

[0089] For the automatic typesetting of the final comic work: a typesetting algorithm based on deep learning can be introduced to automatically determine the layout and framework of the comic. By learning the typesetting styles of a large number of comic books, the typesetting algorithm can optimize the visual flow of each page, ensuring that the reading experience of the comic is both in line with habits and artistically beautiful.

[0090] Through this implementation, it is possible to convert videos such as movies into anime-style comics, and it can be ensured that the comics can reflect the main storyline and content of the target video, that is, the comics have high content quality, and the layout conforms to the user's reading habits, which can enhance the user's reading interest.

[0091] Optionally, extracting key frames from the target video includes:

[0092] Eliminate the opening segment and the ending segment in the target video to obtain a first video;

[0093] Performing frame reduction processing on the first video to obtain a second video;

[0094] Comparing feature information of two adjacent video frames in the second video to determine a candidate key frame, wherein the feature information includes at least one of the following: a color histogram, a feature point, and an edge map;

[0095] Calculating a comprehensive evaluation value of the candidate key frame according to a difference between feature information of the candidate key frame and a video frame preceding the candidate key frame;

[0096] When the comprehensive evaluation value of the candidate key frame is higher than a preset threshold, the candidate key frame is determined to be a key frame.

[0097] In some embodiments, an intelligent frame extraction algorithm may be used to extract key frames from a target video. Taking a movie as an example, the specific steps are as follows:

[0098] Step 1: Identify the opening and ending moments of the movie. Audio features, such as silence, movie theme music, and visual features, such as black screen or title screen, can be used to accurately identify the opening and ending moments. Eliminate the pre-opening screen and useless end credits screen.

[0099] Step 2: According to the movie frame rate, for example, 30fps, firstly reduce the frame rate, for example, extract one frame every 10 frames, which can greatly reduce the number of frames to be processed later. In some embodiments, the interval can also be dynamically adjusted according to the actual content of the movie. For example, for action scenes in the movie, the interval can be reduced to capture more details, while for static scenes, the interval can be increased to reduce redundancy.

[0100] Step 3: Calculate the color histogram of each frame and store it for subsequent comparison. Use Scale-Invariant Feature Transform (SIFT), Oriented Fast and Rotated Brief (ORB), or Convolutional Neural Networks (CNN) models to detect feature points of each frame. You can also use the Canny edge detection algorithm to obtain the edge map of each frame.

[0101] Step 4: Perform preliminary filtering on all preprocessed video frames. Specifically, calculate the color histogram difference between the current frame and the previous frame, and mark it as a candidate key frame when the difference exceeds the threshold; calculate the number of feature point matches between the current frame and the previous frame, and mark it as a candidate key frame when the number of matches is lower than the threshold; calculate the difference between the edge map of the current frame and the previous frame, and mark it as a candidate key frame when the difference exceeds the threshold.

[0102] Step 5: Calculate a comprehensive score for each candidate key frame, integrating color histogram differences, feature point matching changes, and edge differences. When the score is higher than the set threshold T, the picture can be considered a key frame.

[0103] Through this implementation, not only can the key frames in the video be extracted quickly and accurately, but also the quality of the extracted key frames can be guaranteed, thereby ensuring the quality of the subsequent comic works generated based on the extracted key frames.

[0104] Optionally, obtaining text information corresponding to the key frame includes:

[0105] Extracting features from target information and the key frame of the target video respectively, wherein the target information includes at least one of the following: subtitle text, commentary text, audio, character name, and character image;

[0106] Performing feature fusion on each feature obtained by the feature extraction to obtain a fused feature;

[0107] Encoding the fused features to obtain global encoding features of the target video;

[0108] The global encoding feature is decoded to generate a description text corresponding to the key frame.

[0109] In some embodiments, the description text of each key frame can be generated by extracting features, merging features, encoding, decoding, etc. of each key frame and related target information in the target video. In specific implementation, a video encoding-decoding model can be used, such as using Figure 3 The multimodal video understanding model of the Tranformers architecture shown in FIG. 1 is used to understand the image information of each key frame and output a text description of each key frame, that is, the corresponding description text.

[0110] Specifically, first, the target information of the target video, such as subtitle text, commentary text, audio, character names, actor portraits, etc., and each key frame in the target video are taken as input data, and feature extraction is performed respectively through the corresponding feature extraction layers to obtain multimodal features; then the extracted features of each modality can be input into the feature fusion layer for feature interactive fusion to obtain fused features; then the fused features are encoded by the encoder to obtain the global encoding features of the target video, which can also be called global context representation; finally, the global encoding features are input into the decoder to generate a text description of each key frame.

[0111] Through this implementation, the text description of each key frame in the target video can be generated through different feature layers in the model, and the implementation method is simple and easy to implement.

[0112] Optionally, the text information also includes subtitles corresponding to the key frames;

[0113] Generating a comic work corresponding to the target video according to the comic image and text information corresponding to the key frame includes:

[0114] Adding the description text to the comic image corresponding to the key frame, and adding a dialog box including a dialogue text to the corresponding character in the comic image according to the subtitle;

[0115] The comic image after adding the description text and the dialog box is typeset using a preset typeset algorithm to obtain the comic work.

[0116] That is, in some embodiments, the subtitles corresponding to the key frame can be found from the subtitle file of the target video according to the timestamp of each key frame. Then, corresponding description texts can be added to each comic image frame, and a dialog box can be added to the corresponding character in each comic image frame according to the corresponding subtitles, and the subtitle content can be displayed in the dialog box.

[0117] Then, for each comic image frame with corresponding description text and dialog box added, a preset typesetting algorithm can be used for typeset, such as introducing a typesetting algorithm based on deep learning to automatically determine the layout and frame of the comic, and finally generate a typeset comic work. By learning the typesetting styles of a large number of comic books, the typesetting algorithm can optimize the visual flow of each page, ensuring that the reading experience of the comic is both in line with habits and artistically beautiful.

[0118] Step 202: Receive a user's search input for the comic work, where the search information corresponding to the search input is search information or question information about the content in the comic work.

[0119] That is, it can support the user to search, ask questions and other interactive operations for the content in the comic work during the process of reading the comic work or after opening the comic work. Specifically, the user can perform search input for the comic work, such as entering the search text in the search bar provided in the reading interface of the comic work, or the user can directly enter the relevant questions or retrieval of the comic work through a certain trigger operation, and the input search information form is not limited to text, voice, etc. For example, the user inputs similar search information such as "Why did the character "XX" in the movie choose to return to the workplace" for a comic work after power conversion.

[0120] Optionally, step 202 includes:

[0121] Displaying the comic work and the information search box in the first area and the second area of ​​the web page respectively;

[0122] The search input performed by the user through the information search box is received, and the search information is the information input in the information search box.

[0123] In some embodiments, a web page layout may be performed for the generated comic works to facilitate users to read the comic works on the web page, and the web page layout may be designed to display two parts: an interactive area with users and a comic work display area. Figure 4 As shown, the web page layout of the comic work can be designed to display the comic content on the right and the information search box, or question and answer text box, on the left. In this way, while watching the comic work, the user can ask questions or search for information about the comic work content, such as entering some questions about the comic work in the information search box.

[0124] Step 203: parse the target video according to the search information and the pre-generated description text corresponding to the key frames in the target video to obtain a video parsing result.

[0125] In this step, the target video can be parsed by combining the search information with the semantic understanding of the description text corresponding to the key frames in the target video, which can also be called natural language understanding, that is, understanding the content of each key frame in the target video, so as to find the search results for the search information. The video parsing results may include the search result text, and may also include images generated based on the search results, which is equivalent to secondary creation images based on user questions. Exemplarily, the search information is "Which company does "XX" in the film meet with at noon to discuss project cooperation", and accordingly, the answer to the corresponding question can be found through the semantic understanding of the description text of each key frame, and the final video parsing result is "In the film, "XX" meets with "XX" company at noon to discuss project cooperation".

[0126] Among them, the method of generating the description text corresponding to the key frame is specifically described in the above-mentioned implementation method.

[0127] Optionally, step 203 includes:

[0128] The search information and the description text corresponding to the key frame are input into a large language model LLM for natural language understanding to obtain a language understanding result, wherein the video parsing result includes the language understanding result.

[0129] In some embodiments, a large language model (LLM) can be used to implement the language understanding step of the input text, such as Figure 3As shown, the search information and the description text corresponding to the key frame can be input into the large language model LLM for natural language understanding, and the LLM model outputs the language understanding result of the user's question or information search, and the understanding result is a text result, such as a text answer to the user's question, and the text result is related to the comic content. This implementation method can ensure that the model outputs a relatively accurate text understanding result of the user's interactive question or search.

[0130] Furthermore, the step 203 further includes:

[0131] The language understanding result and the global encoding feature are input into an image generation model to obtain a target image, wherein the content represented by the target image is related to the content of the target video and the search information, and the video parsing result also includes the target image.

[0132] In some embodiments, corresponding images can be further expanded based on the text interaction results with the user to realize the concrete display of the interaction results in the form of images, which is equivalent to realizing secondary creation of comic works according to the user interaction method.

[0133] Specifically, Figure 3 As shown, the text language understanding result output by the LLM model can be used together with the global encoding features of the key frames and target information in the target video, that is, the global context display input image generation model, such as a diffusion model, an adversarial generation model, etc., to generate an image, and obtain a target image that can represent the text language understanding result by utilizing the elements in the target video including characters, clothing, speaking style, etc. The target image can also be a comic style image.

[0134] In this way, through this implementation, the video understanding results and user search information can be combined to generate images related to the target video content, further enhancing the interactivity and personalization of the comic viewing experience.

[0135] Combine the following Figure 3 , the main components of the above multimodal video understanding model are described as follows:

[0136] Input Layer:

[0137] Keyframe: Use the pre-trained visual Transformer to extract keyframe features and obtain their vector representation.

[0138] Subtitles and narrations: Use a pre-trained language model to extract text features and obtain their vector representation.

[0139] Audio: Use a pre-trained audio model to extract features and obtain their vector representation.

[0140] Actor faces and names: Use a facial recognition network to extract features and combine them with name information to get their vector representation.

[0141] User input text (i.e. the search information mentioned above): Use a pre-trained language model such as LLM to extract features and obtain its vector representation.

[0142] Feature fusion layer:

[0143] The multi-head self-attention mechanism is used to interactively fuse the features of different modalities extracted from the input layer.

[0144] Encoding layer:

[0145] A multi-layer Transformer encoder is used to encode the fused features to obtain a global context representation.

[0146] Decoding layer:

[0147] Text description generation: Use the Transformer decoder to generate text descriptions of video key frames.

[0148] Image generation: Generate images using a diffusion model combining user input text and encoded global features.

[0149] Optionally, the image generation model and the LLM are trained in the following manner:

[0150] Using the labeled image description text of the key frame to be trained, the information search pair and the image generation data as the training data set, wherein the information search pair includes the preset search information and the corresponding search result;

[0151] Inputting the image description text and the preset search information into an initial LLM for natural language understanding, and obtaining an actual understanding result output by the LLM;

[0152] Inputting the actual understanding result and the encoding features corresponding to the key frame to be trained into an initial image generation model to obtain actual generated image data output by the image generation model;

[0153] Calculating the cross entropy loss of the LLM language understanding and the accuracy loss of the information search according to the actual understanding result and the search result corresponding to the preset search information;

[0154] Calculating the adversarial loss of the image generated by the image generation model according to the actually generated image data and the image generation data;

[0155] The LLM and the image generation model are jointly trained according to the cross entropy loss, the accuracy loss and the adversarial loss to obtain the trained image generation model and the LLM.

[0156] In some embodiments, the LLM model and the image generation model such as the diffusion model in the embodiments of the present application can be trained by jointly training with a multi-dimensional loss function.

[0157] Combination Figure 3 Taking the information search pair as a question-answer pair as an example, the LLM model and the diffusion model can be trained using the labeled text descriptions of the key frames in the training video, the question-answer pairs for the training video content, and the corresponding image generation data. During the training process, the cross entropy loss of the generated text, the accuracy loss of the question-answering, and the adversarial loss of the image generation are calculated and combined according to the model output and the corresponding labeled data, and the two models are jointly optimized. Through the above design, it is possible to generate pictures similar to a person doing something in a movie based on the interactive text input by the user, that is, the above search information, and generate questions and descriptions of the movie content.

[0158] Through this implementation, it can be ensured that the above-mentioned model obtained through joint training has better performance and accuracy.

[0159] Step 204: output interactive result information for the search information based on the video analysis result.

[0160] The above-mentioned interactive result information of the search information output based on the video analysis result can be displayed as the interactive result information of the user on the comic work after the video analysis result is correspondingly laid out, so as to provide the user with the search results of the content in the comic work, or the answer to the question after asking the user about the content of the comic work. For example, Figure 4 As shown, the answers to the questions entered by the user are displayed in the information search box / question and answer text box on the left.

[0161] Based on the above description of the implementation methods, the main advantages of the embodiments of the present application are as follows:

[0162] User interactivity: Allows users to enter text on the web page to ask questions or provide guidance on the generated content, which enhances the interactive and personalized experience;

[0163] Multimodal integration: Fusing visual, audio, and text data in videos, and using these rich information sources to provide more accurate content understanding and style transfer;

[0164] Style uniformity: By adding the identification of characters and scenes, the uniformity of the comic style and the consistency of the characters are ensured, thus improving the quality of the work;

[0165] Dynamic content adjustment: Dynamically adjust the processing strategy according to the video content, such as adjusting the key frame extraction frequency according to the dynamics of the scene, so that the final output is more in line with the emotions and dynamic changes of the original content.

[0166] The video processing method of the embodiment of the present application obtains the comic work corresponding to the target video; receives the user's search input for the comic work, and the search information corresponding to the search input is the search information or question information for the content in the comic work; parses the target video according to the search information and the pre-generated description text corresponding to the key frame in the target video to obtain the video parsing result; and outputs the interactive result information for the search information based on the video parsing result. In this way, the user can perform interactive operations such as information search and questioning for the comic work after the video conversion, thereby enriching and improving the interactivity between the video processing technology and the user.

[0167] The present application also provides a video processing device. Figure 5 , Figure 5 1 is a structural diagram of a video processing device provided in an embodiment of the present application. Since the principle of solving the problem by the video processing device is similar to that of the video processing method in the embodiment of the present application, the implementation of the video processing device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0168] like Figure 5 As shown, the video processing device 500 includes:

[0169] An acquisition module 501 is used to acquire the comic work corresponding to the target video;

[0170] The receiving module 502 is used to receive a search input from a user for the comic work, where the search information corresponding to the search input is search information or question information for the content in the comic work;

[0171] The parsing module 503 is used to parse the target video according to the search information and the pre-generated description text corresponding to the key frames in the target video to obtain a video parsing result;

[0172] The output module 504 is used to output the interactive result information of the search information based on the video analysis result.

[0173] Optionally, the acquisition module 501 includes:

[0174] A key frame extraction unit, used to extract key frames from the target video to obtain key frames in the target video;

[0175] A comic style conversion unit, used to perform comic style conversion processing on the key frame to obtain a comic image corresponding to the key frame;

[0176] An acquisition unit, configured to acquire text information corresponding to the key frame, wherein the text information at least includes the description text;

[0177] A generating unit is used to generate a comic work corresponding to the target video according to the comic image and text information corresponding to the key frame.

[0178] Optionally, the acquiring unit is specifically used for:

[0179] Extracting features from target information and the key frame of the target video respectively, wherein the target information includes at least one of the following: subtitle text, commentary text, audio, character name, and character image;

[0180] Performing feature fusion on each feature obtained by the feature extraction to obtain a fused feature;

[0181] Encoding the fused features to obtain global encoding features of the target video;

[0182] The global encoding feature is decoded to generate a description text corresponding to the key frame.

[0183] Optionally, the parsing module 503 includes:

[0184] LLM is used to perform natural language understanding on the search information and the description text corresponding to the key frame to obtain a language understanding result, wherein the video parsing result includes the language understanding result.

[0185] Optionally, the parsing module 503 further includes:

[0186] An image generation model is used to generate an image based on the language understanding result and the global encoding feature to obtain a target image, wherein the content represented by the target image is related to the content of the target video and the search information, and the video parsing result also includes the target image.

[0187] Optionally, the image generation model and the LLM are trained in the following manner:

[0188] Using the labeled image description text of the key frame to be trained, the information search pair and the image generation data as the training data set, wherein the information search pair includes the preset search information and the corresponding search result;

[0189] Inputting the image description text and the preset search information into an initial LLM for natural language understanding, and obtaining an actual understanding result output by the LLM;

[0190] Inputting the actual understanding result and the encoding features corresponding to the key frame to be trained into an initial image generation model to obtain actual generated image data output by the image generation model;

[0191] Calculate the cross entropy loss of the LLM language understanding and the accuracy loss of the information search according to the actual understanding result and the search result corresponding to the preset search information;

[0192] Calculating the adversarial loss of the image generated by the image generation model according to the actually generated image data and the image generation data;

[0193] The LLM and the image generation model are jointly trained according to the cross entropy loss, the accuracy loss and the adversarial loss to obtain the trained image generation model and the LLM.

[0194] Optionally, the receiving module 502 includes:

[0195] A display unit, used to display the comic work and the information search box in a first area and a second area of ​​a web page, respectively;

[0196] The receiving unit is used to receive the search input performed by the user through the information search box, and the search information is the information input in the information search box.

[0197] Optionally, the key frame extraction unit is specifically used for:

[0198] Eliminate the opening segment and the ending segment in the target video to obtain a first video;

[0199] Performing frame reduction processing on the first video to obtain a second video;

[0200] Comparing feature information of two adjacent video frames in the second video to determine a candidate key frame, wherein the feature information includes at least one of the following: a color histogram, a feature point, and an edge map;

[0201] Calculating a comprehensive evaluation value of the candidate key frame according to a difference between feature information of the candidate key frame and a video frame preceding the candidate key frame;

[0202] When the comprehensive evaluation value of the candidate key frame is higher than a preset threshold, the candidate key frame is determined to be a key frame.

[0203] Optionally, the text information also includes subtitles corresponding to the key frames;

[0204] The generating unit is specifically used for:

[0205] Adding the description text to the comic image corresponding to the key frame, and adding a dialog box including a dialogue text to the corresponding character in the comic image according to the subtitle;

[0206] The comic image after adding the description text and the dialog box is typeset using a preset typeset algorithm to obtain the comic work.

[0207] The video processing device 500 provided in the embodiment of the present application can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated in this embodiment.

[0208] The video processing device 500 of the embodiment of the present application obtains the comic work corresponding to the target video; receives the user's search input for the comic work, and the search information corresponding to the search input is the search information or question information for the content in the comic work; parses the target video according to the search information and the pre-generated description text corresponding to the key frame in the target video to obtain the video parsing result; and outputs the interactive result information for the search information based on the video parsing result. In this way, the user can perform interactive operations such as information search and questioning for the comic work after the video conversion, thereby enriching and improving the interactivity between the video processing technology and the user.

[0209] The present application also provides an electronic device. Since the principle of solving the problem by the electronic device is similar to the video processing method in the present application, the implementation of the electronic device can refer to the implementation of the method, and the repeated parts will not be repeated. Figure 6 As shown, the electronic device of the embodiment of the present application includes:

[0210] The processor 600 is used to read the program in the memory 620 and execute the following process:

[0211] Get the comics corresponding to the target video;

[0212] receiving a search input from a user for the comic work, wherein the search information corresponding to the search input is search information or question information for content in the comic work;

[0213] Parsing the target video according to the search information and the pre-generated description text corresponding to the key frames in the target video to obtain a video parsing result;

[0214] Output interactive result information of the search information based on the video analysis result.

[0215] The transceiver 610 is configured to receive and send data under the control of the processor 600 .

[0216] Among them, Figure 6In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically one or more processors represented by processor 600 and various circuits of memory represented by memory 620 are linked together. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface. The transceiver 610 may be a plurality of components, namely, a transmitter and a transceiver, providing a unit for communicating with various other devices on a transmission medium. For different user devices, the user interface 630 may also be an interface capable of externally and internally connecting required devices, and the connected devices include but are not limited to a keypad, a display, a speaker, a microphone, a joystick, and the like. The processor 600 is responsible for managing the bus architecture and general processing, and the memory 620 may store data used by the processor 600 when performing operations.

[0217] Optionally, the processor 600 is further configured to read a program in the memory 620 and execute the following steps:

[0218] Extracting key frames from the target video to obtain key frames in the target video;

[0219] Perform comic style conversion on the key frame to obtain a comic image corresponding to the key frame;

[0220] Acquire text information corresponding to the key frame, wherein the text information at least includes the description text;

[0221] Generate a comic work corresponding to the target video based on the comic image and text information corresponding to the key frame.

[0222] Optionally, the processor 600 is further configured to read a program in the memory 620 and execute the following steps:

[0223] Extracting features from target information and the key frame of the target video respectively, wherein the target information includes at least one of the following: subtitle text, commentary text, audio, character name, and character image;

[0224] Performing feature fusion on each feature obtained by the feature extraction to obtain a fused feature;

[0225] Encoding the fused features to obtain global encoding features of the target video;

[0226] The global encoding feature is decoded to generate a description text corresponding to the key frame.

[0227] Optionally, the processor 600 is further configured to read a program in the memory 620 and execute the following steps:

[0228] The search information and the description text corresponding to the key frame are input into a large language model LLM for natural language understanding to obtain a language understanding result, wherein the video parsing result includes the language understanding result.

[0229] Optionally, the processor 600 is further configured to read a program in the memory 620 and execute the following steps:

[0230] The language understanding result and the global encoding feature are input into an image generation model to obtain a target image, wherein the content represented by the target image is related to the content of the target video and the search information, and the video parsing result also includes the target image.

[0231] Optionally, the image generation model and the LLM are trained in the following manner:

[0232] Using the labeled image description text of the key frame to be trained, the information search pair and the image generation data as the training data set, wherein the information search pair includes the preset search information and the corresponding search result;

[0233] Inputting the image description text and the preset search information into an initial LLM for natural language understanding, and obtaining an actual understanding result output by the LLM;

[0234] Inputting the actual understanding result and the encoding features corresponding to the key frame to be trained into an initial image generation model to obtain actual generated image data output by the image generation model;

[0235] Calculating the cross entropy loss of the LLM language understanding and the accuracy loss of the information search according to the actual understanding result and the search result corresponding to the preset search information;

[0236] Calculating the adversarial loss of the image generated by the image generation model according to the actually generated image data and the image generation data;

[0237] The LLM and the image generation model are jointly trained according to the cross entropy loss, the accuracy loss and the adversarial loss to obtain the trained image generation model and the LLM.

[0238] Optionally, the processor 600 is further configured to read a program in the memory 620 and execute the following steps:

[0239] Displaying the comic work and the information search box in the first area and the second area of ​​the web page respectively;

[0240] The search input performed by the user through the information search box is received, and the search information is the information input in the information search box.

[0241] Optionally, the processor 600 is further configured to read a program in the memory 620 and execute the following steps:

[0242] Eliminate the opening segment and the ending segment in the target video to obtain a first video;

[0243] Performing frame reduction processing on the first video to obtain a second video;

[0244] Comparing feature information of two adjacent video frames in the second video to determine a candidate key frame, wherein the feature information includes at least one of the following: a color histogram, a feature point, and an edge map;

[0245] Calculating a comprehensive evaluation value of the candidate key frame according to a difference between feature information of the candidate key frame and a video frame preceding the candidate key frame;

[0246] When the comprehensive evaluation value of the candidate key frame is higher than a preset threshold, the candidate key frame is determined to be a key frame.

[0247] Optionally, the text information also includes subtitles corresponding to the key frames;

[0248] The processor 600 is further configured to read the program in the memory 620 and execute the following steps:

[0249] Adding the description text to the comic image corresponding to the key frame, and adding a dialog box including a dialogue text to the corresponding character in the comic image according to the subtitle;

[0250] The comic image after adding the description text and the dialog box is typeset using a preset typeset algorithm to obtain the comic work.

[0251] The electronic device provided in the embodiment of the present application can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated in this embodiment.

[0252] In addition, the computer-readable storage medium of the embodiment of the present application is used to store a computer program, and the computer program can be executed by a processor to implement Figure 1 The various steps in the illustrated embodiment.

[0253] An embodiment of the present application provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, the various processes of the above-mentioned video processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0254] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0255] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may be physically included separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0256] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform some steps of the sending and receiving methods described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0257] The above is a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles described in the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A video processing method, characterized in that: include: Get the comics corresponding to the target video; receiving a search input from a user for the comic work, wherein the search information corresponding to the search input is search information or question information for content in the comic work; Parsing the target video according to the search information and the pre-generated description text corresponding to the key frames in the target video to obtain a video parsing result; Output interactive result information of the search information based on the video analysis result.

2. The method according to claim 1, characterized in that The step of obtaining the comic work corresponding to the target video includes: Extracting key frames from the target video to obtain key frames in the target video; Perform comic style conversion on the key frame to obtain a comic image corresponding to the key frame; Acquire text information corresponding to the key frame, wherein the text information at least includes the description text; Generate a comic work corresponding to the target video based on the comic image and text information corresponding to the key frame.

3. The method according to claim 2, characterized in that The obtaining of text information corresponding to the key frame includes: Extracting features from target information and the key frame of the target video respectively, wherein the target information includes at least one of the following: subtitle text, commentary text, audio, character name, and character image; Performing feature fusion on each feature obtained by the feature extraction to obtain a fused feature; Encoding the fused features to obtain global encoding features of the target video; The global encoding feature is decoded to generate a description text corresponding to the key frame.

4. The method according to claim 3, characterized in that The step of parsing the target video according to the search information and the pre-generated description text corresponding to the key frame to obtain a video parsing result includes: The search information and the description text corresponding to the key frame are input into a large language model LLM for natural language understanding to obtain a language understanding result, wherein the video parsing result includes the language understanding result.

5. The method according to claim 4, characterized in that The step of parsing the target video according to the search information and the pre-generated description text corresponding to the key frame to obtain a video parsing result further includes: The language understanding result and the global encoding feature are input into an image generation model to obtain a target image, wherein the content represented by the target image is related to the content of the target video and the search information, and the video parsing result also includes the target image.

6. The method according to claim 5, characterized in that The image generation model and the LLM are trained in the following manner: Using the labeled image description text of the key frame to be trained, the information search pair and the image generation data as the training data set, wherein the information search pair includes the preset search information and the corresponding search result; Inputting the image description text and the preset search information into an initial LLM for natural language understanding, and obtaining an actual understanding result output by the LLM; Inputting the actual understanding result and the encoding features corresponding to the key frame to be trained into an initial image generation model to obtain actual generated image data output by the image generation model; Calculating the cross entropy loss of the LLM language understanding and the accuracy loss of the information search according to the actual understanding result and the search result corresponding to the preset search information; Calculating the adversarial loss of the image generated by the image generation model according to the actually generated image data and the image generation data; The LLM and the image generation model are jointly trained according to the cross entropy loss, the accuracy loss and the adversarial loss to obtain the trained image generation model and the LLM.

7. The method according to claim 1, characterized in that The receiving a user's search input for the comic work includes: Displaying the comic work and the information search box in the first area and the second area of ​​the web page respectively; The search input performed by the user through the information search box is received, and the search information is the information input in the information search box.

8. The method according to claim 2, characterized in that: The step of extracting key frames from the target video includes: Eliminate the opening segment and the ending segment in the target video to obtain a first video; Performing frame reduction processing on the first video to obtain a second video; Comparing feature information of two adjacent video frames in the second video to determine a candidate key frame, wherein the feature information includes at least one of the following: a color histogram, a feature point, and an edge map; Calculating a comprehensive evaluation value of the candidate key frame according to a difference between feature information of the candidate key frame and a video frame preceding the candidate key frame; When the comprehensive evaluation value of the candidate key frame is higher than a preset threshold, the candidate key frame is determined to be a key frame.

9. The method according to claim 2, characterized in that: The text information also includes subtitles corresponding to the key frames; Generating a comic work corresponding to the target video according to the comic image and text information corresponding to the key frame includes: Adding the description text to the comic image corresponding to the key frame, and adding a dialog box including a dialogue text to the corresponding character in the comic image according to the subtitle; The comic image after adding the description text and the dialog box is typeset using a preset typeset algorithm to obtain the comic work.

10. A video processing device, characterized in that: include: An acquisition module is used to acquire the comic works corresponding to the target video; A receiving module, configured to receive a user's search input for the comic work, wherein the search information corresponding to the search input is search information or question information for content in the comic work; A parsing module, used for parsing the target video according to the search information and the pre-generated description text corresponding to the key frames in the target video to obtain a video parsing result; An output module is used to output interaction result information of the search information based on the video analysis result.

11. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor; wherein the processor is used to read the program in the memory to implement the steps in the video processing method as described in any one of claims 1 to 9.

12. A computer-readable storage medium for storing a computer program, characterized in that: When the computer program is executed by a processor, the steps in the video processing method according to any one of claims 1 to 9 are implemented.

13. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps in the video processing method according to any one of claims 1 to 9.

Images