Video-to-cartoon conversion method based on multi-modal large model

Abstract

The invention discloses a video-to-cartoon method based on a multi-modal large model, and the method comprises the steps: S1, inputting input data into a task distribution agent, transmitting the input data into a video abstract agent, and outputting a key frame set; s2, the task distribution agent transmits the key frame set and the input data to a picture style migration agent, caption removal and style migration are carried out on the key frames, and a target style picture set is output; s3, the task distribution agent audits the picture quality of each target style picture in the target style picture set to obtain an audited set List; and S4, the layout generation agent performs row layout and column layout and fine adjustment of the page according to the set List, and outputs a cartoon book meeting the picture style requirement. According to the method for converting the video into the cartoon, the video is converted into a cartoon book form in a customized mode, the time cost for watching the video is remarkably reduced, a highly personalized cartoon generation service is provided, and diversified requirements are accurately met.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, specifically to a video-to-comic conversion method based on a multimodal large model. Background Technology

[0002] With the explosive growth of video content, users often struggle to quickly extract the information they need from a massive amount of video. Traditional video reading methods are inefficient, requiring users to spend a significant amount of time watching the entire video to extract key information. To address this problem, video summarization technology has emerged, simplifying content by extracting keyframes or segments from videos. However, existing video summarization methods have significant limitations: they typically employ generalized processing methods and fail to fully consider users' personalized needs. For example, when a user wants to extract content on a specific theme (such as a war scene) from a video, existing video summarization methods cannot extract relevant segments specifically, resulting in summarization results that do not match the user's needs.

[0003] Furthermore, with changing multimedia content consumption habits, users' demands for information presentation formats are becoming increasingly diverse. Comics, as an intuitive and efficient visual narrative medium, can present complex content in a concise and vivid form, while enhancing the user's reading experience. However, automatically converting video content into comic format still faces many technical challenges, including semantic understanding of keyframes, transfer of image styles, and design of comic layouts.

[0004] In recent years, the rapid development of multimodal large model and agent technologies has provided new solutions to the aforementioned problems. By integrating multi-dimensional information such as visual and textual data, multimodal large models can more accurately understand video content, providing a technological foundation for personalized summarization. At the same time, agent technologies based on multimodal large models (MLLM) demonstrate powerful task decomposition and invocation tools, enabling efficient planning and invocation of tools to complete tasks.

[0005] Therefore, there is an urgent need for a video-to-comic conversion method based on a multimodal large model to solve the technical challenges in the process of automatically converting video content into comic form. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a video-to-comic conversion method based on a multimodal large model. This method uses a multimodal large model to intelligently analyze and customize the conversion of given video content, ultimately generating a comic book that meets the user's personalized needs.

[0007] Therefore, the present invention adopts the following technical solution:

[0008] A video-to-comic conversion method based on a multimodal large model includes:

[0009] S1, the input data is input into the task allocation agent. The task allocation agent transmits the video and event information from the input data to the video summarizing agent. The video summarizing agent obtains segments based on the video in the input data, and obtains key video sub-segments based on the summary text corresponding to the segments or the event information in the input data. According to the selection rules, keyframes are obtained from the key video sub-segments to form a keyframe set. The video summarizing agent outputs the keyframe set and transmits it to the task allocation agent. The input data includes: a video, event information, and image style requirements. The image style requirements can be a classic black and white comic style or a color comic style.

[0010] S2, the task allocation agent receives the key frame set, and transmits the key frame set and the image style requirements in the input data to the image style transfer agent. The image style transfer agent performs subtitle removal and style transfer on the key frames in the key frame set in sequence. The image style transfer agent outputs the target style image set and transmits it to the task allocation agent.

[0011] S3, the task allocation agent receives the target style image set, reviews the image quality of each target style image in the target style image set, and for target style images with unqualified image quality, the image style transfer agent regenerates the target style image until the image quality of the target style image is qualified or the maximum number of attempts is reached, and obtains the set List of reviewed target style images;

[0012] S4, the task allocation agent transmits the set List to the layout generation agent. The layout generation agent selects the number of panels on the page and the target style images to be placed on the page, and performs the row and column layout of the page. It fine-tunes the target style images on the page to obtain the main image. The above operations are performed on each page until all target style images in the set List have been processed. The main image is scaled and filled into the corresponding position planned by the target style images on each page with a fixed aspect ratio. The output of the layout generation agent is a comic book that meets the image style requirements.

[0013] Step S1 specifically includes:

[0014] S1-1: A video scene detection tool is used to segment the video in the input data into N video segments, and a multimodal large model is used to generate a corresponding video text description for each video segment; an index is set before the first character of the video text description corresponding to each video segment; all video text descriptions with indexes are merged into a paragraph.

[0015] S1-2, Determine if the content of the event information in the input data is empty: if it is empty, execute S1-3; if it is not empty, execute S1-4.

[0016] S1-3, input the paragraph obtained in S1-1 into the multimodal large model, and input prompt word 1. Generate summary text for the paragraph through the multimodal large model, and make the multimodal large model output the index corresponding to the video text description most relevant to the summary text. Based on the index, select the video sub-segment corresponding to the index from N video sub-segments and use it as the key video sub-segment.

[0017] S1-4, Input the paragraph and event information obtained in S1-1 into the multimodal large model, filter out the index of the video text description related to the event information through the multimodal large model, and filter out the video sub-segment corresponding to the index from N video sub-segments according to the index and use it as the key video sub-segment;

[0018] S1-5, in the key video sub-segment, obtain the video frame set according to the selection rules, input the selected video frame set and prompt word 2 into the multimodal large model, and output the numbers of M key frames through the multimodal large model, so that the M key frames with corresponding numbers form a key frame set.

[0019] Step S2 specifically includes:

[0020] S2-1, Traverse the keyframe set, call the subtitle removal tool on the keyframes containing subtitles in the keyframe set, remove the subtitles in the keyframes, and obtain a keyframe set without subtitles;

[0021] S2-2: Check if a diffusion model that matches the image style requirements and has been trained already exists. If it does not exist, retrieve the diffusion model that matches the image style requirements and has been trained. If it exists, proceed directly to S2-3.

[0022] S2-3, The image style transfer agent uses a diffusion model that corresponds to the image style requirements and has been trained to perform style transfer on each keyframe in the set of keyframes without subtitles, to obtain the corresponding target style image and its text description, which together form the target style image set.

[0023] Step S3 specifically includes:

[0024] S3-1: The task allocation agent receives the target style image set output by the image style transfer agent, traverses the target style image set, and checks the image quality of all target style images. If the image quality of the current target style image is qualified, the next target style image is checked. If the image quality of the current target style image is unqualified, the execution of step S3-2 is performed.

[0025] S3-2, the image style transfer agent regenerates the target style image and reviews its quality. The regeneration and review steps are repeated until the image quality of the target style image is qualified or the maximum number of attempts is reached. Then, the last generated target style image is selected as the reviewed target style image.

[0026] S3-3, after reviewing each target style image in the target style image set, we obtain a set of reviewed target style images, List = {p1, p2, ..., p...} i ,…,p M}, M is the number of target style images in the set List, i∈[1,M], p i Let i be the i-th target style image in the collection List.

[0027] Step S4 specifically includes:

[0028] S4-1, randomly select an integer a within the range [4,7], compare the size of integer a and M, and select the smaller value as the number of panels m1 on the page; select the set List = {p1,p2,…,p i ,…,p M In the context of the target style image, p1 to m1 are the first target style images. get

[0029] S4-2, The layout generation agent performs row layout operations on the page based on the number of panels m1:

[0030] Construct prompt text 1; input the prompt text into the multimodal large model, and use the multimodal large model to analyze the target style image. The process is split into rows to obtain the row splitting results. Based on the row splitting results, the average aspect ratio of the target style images in each row is calculated. If the average aspect ratio of the row is not within the preset range, a multimodal large model is used to split the target style images contained in the row into rows until the maximum number of attempts is reached or the average aspect ratio of the row is within the preset range. If the row splitting results still cannot satisfy the requirement that the average aspect ratio of each row is within the preset range after reaching the maximum number of attempts, the result of the last row splitting is used as the row splitting result.

[0031] S4-3, the layout generation agent performs column layout operations on the page based on the number of panels m1 and the row break results:

[0032] Construct prompt text 2; input the prompt text into the multimodal large model, and use the multimodal large model based on the line breaks of this page to select the target style image. Split the columns to obtain the split results;

[0033] Calculate the average aspect ratio of the target style images in each column based on the splitting results. If the average aspect ratio of the target style images in that column is not within the preset range, then re-split the target style images in the row to which that column belongs until the maximum number of attempts is reached or the average aspect ratio of each column is within the preset range. If the splitting results after reaching the maximum number of attempts do not meet the requirement that the average aspect ratio of each column is within the preset range, then the result of the last splitting is taken as the splitting result.

[0034] S4-4, Target style image for this page Make minor adjustments:

[0035] Iterate through all images of the target style on the page. A multimodal large model is used to generate a subject description for the target style image; the target style image and its corresponding subject description are input into the object detection model to generate a coordinate frame of the subject object corresponding to the target style image, and the aspect ratio of the coordinate frame of the subject object is calculated.

[0036] Based on the aspect ratio of the main object coordinate frame and the aspect ratio of the corresponding position planned for the target style image on the page, the target style image where the main object coordinate frame is located is expanded using an image expansion tool.

[0037] The expanded target style image and its corresponding subject description are input into the object detection model to obtain the coordinate frame of the subject object corresponding to the expanded target style image. The width of the coordinate frame is w1 and the height is h1. Let w2 = w1, and calculate the aspect ratio k of the corresponding position planned in the target style image. Let h3 = h1. Based on the aspect ratio k of the corresponding position of the target style image, calculate w3 = k × h3. Determine whether w2 × h2 is greater than w3 × h3. If so, select w2 as the width of the main image and h2 as the height of the main image. Otherwise, select w3 as the width of the main image and h3 as the height of the main image.

[0038] Using the center coordinates of the main object's coordinate frame as the center, extract the main image from the main object's coordinate frame according to the width and height of the main image;

[0039] S4-5, the main image is scaled down and filled into the corresponding position of the target style image on the page under a fixed aspect ratio setting to obtain the comic page;

[0040] S4-6, repeat S4-1 to S4-5 until all target style images in the collection List have been processed, resulting in multiple comic pages, which are then combined to form a personalized comic book that meets the user's needs and output.

[0041] The selection rules in steps S1-5 are as follows: in the key video sub-segment, one video frame is selected every k seconds to obtain a video frame set composed of multiple video frames. The value of k ensures that the upper limit of the number of video frames in the video frame set does not exceed the processing length. The minimum value of k is 3, and the value of k is increased by 0.5 each time it is adjusted.

[0042] The method for obtaining the trained diffusion model that corresponds to the image style requirements in step S2-2 is as follows:

[0043] A large dataset of image text descriptions is obtained. Based on the image style requirements in the input data, a multimodal large model is used to rewrite the image text descriptions in the dataset, thereby generating paired text descriptions. The paired text descriptions include: image text descriptions and image text descriptions related to image style requirements. Based on the paired text descriptions, images corresponding to the image text descriptions and image text descriptions related to image style requirements are generated respectively. A stable diffusion model is trained based on a large number of paired images to obtain a diffusion model that corresponds to the image style requirements and is fully trained.

[0044] In step S4-2, the prompt text 1 includes: the number of target style images in each row does not exceed the number of panels m1, important target style images occupy a large area on the page, and the aspect ratio of the target style images; the importance of the target style images is determined based on the text description of the target style images; the preset range of the average aspect ratio of each row of target style images is 0.6 to 1.7.

[0045] In step S4-3, the prompt text 2 includes: the number of target style images in each column does not exceed the number of panels m1; important target style images occupy a large area on the page; the aspect ratio of the target style images; the importance of the target style images is determined based on the text description of the target style images; the preset range of the average aspect ratio of each column of target style images is 0.6 to 1.7.

[0046] In step S4-4, the coordinate frame diagram of the main object should completely cover the main object, and the main object is the main target object in the main description.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] 1. The video-to-comic conversion method of the present invention can input any video content, and can intelligently analyze the image style requirements in the input data and convert the video into a customized comic book format, providing a highly personalized comic generation service and improving user experience and satisfaction.

[0049] 2. The video-to-comic conversion method of the present invention extracts video summaries in various scenarios through a video summarizing agent. When a user specifies an event they wish to watch, the video summarizing agent can extract keyframes related to the specified event. When the user does not specify a specific event, the video summarizing agent generates a generalized summary and then extracts keyframes related to the generalized summary, which significantly reduces the time cost for users to watch videos.

[0050] 3. The video-to-comic conversion method of the present invention converts keyframes into any specified style by using an image style transfer agent and the image style requirements in the input data, thereby accurately meeting the diverse needs of users.

[0051] 4. The video-to-comic conversion method of the present invention generates a reasonable layout design for a given comic page through a layout generation intelligent agent, which not only fully preserves the key information of the video, but also meets the user's aesthetic needs for comics. Attached Figure Description

[0052] Figure 1 This is a flowchart of the video-to-comic conversion method according to an embodiment of the present invention;

[0053] Figure 2 This is a flowchart of step S1 in an embodiment of the present invention;

[0054] Figure 3 These are the keyframes and corresponding target style images in the target style image set in the embodiments of the present invention, where (a) to (e) are keyframes and (f) to (j) are corresponding target style images;

[0055] Figure 4 This is a screenshot of the video-to-comic conversion result output in step S4 of this embodiment of the invention. Detailed Implementation

[0056] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0057] Example

[0058] like Figure 1 The diagram shows a flowchart of an embodiment of the video-to-comic conversion method based on a multimodal large model according to the present invention. As shown, the method includes the following steps:

[0059] S1, the input data is input into the task allocation agent. The task allocation agent transmits the video and event information from the input data to the video summarizing agent. The video summarizing agent obtains segments based on the video in the input data, and obtains key video sub-segments based on the corresponding summary text of the segments or the event information in the input data. Keyframes are then obtained from these key video sub-segments according to selection rules, forming a keyframe set. The video summarizing agent outputs the keyframe set and transmits it to the task allocation agent. Wherein:

[0060] The input data includes: a video clip, image style requirements, and event information. In this embodiment, the image style requirement is a classic black and white comic book style.

[0061] The task allocation agent is used to assign tasks and transmit data to the video summarizing agent, image style transfer agent, and layout generation agent based on the processing progress of the input data. The video summarizing agent is used to filter out keyframes related to the event information in the video based on the video and event information in the input data. The task allocation agent, video summarizing agent, image style transfer agent, and layout generation agent are all implemented using the GPT-4o model.

[0062] like Figure 2 As shown, this step specifically includes:

[0063] S1-1, the input data is fed into the task allocation agent, which then transmits the video and event information from the input data to the video summarization agent. In the video summarization agent, the scenedetect tool is used to segment the video in the input data into N video segments, and the Gemini 1.5pro model is used to generate a corresponding video text description for each video segment. An index (ID) is set before the first character of the video text description corresponding to each video segment; the index is used to identify the video segment. All video text descriptions with indexes are merged into one paragraph.

[0064] S1-2, the video summarizing agent determines whether the content of the event information in the input data is empty: if it is empty, then execute S1-3; if it is not empty, then execute S1-4.

[0065] S1-3, the video summarizing agent inputs the paragraph obtained in S1-1 into the GPT-4o model and inputs prompt word 1. The GPT-4o model generates a summary text for the paragraph and outputs an index corresponding to the video text description most relevant to the summary text. Based on the index, the video sub-segment corresponding to the index is selected from N video sub-segments and used as the key video sub-segment.

[0066] The form of the paragraph is: [1] video sub-segment description, [2] video sub-segment description, ..., [N] video sub-segment description;

[0067] In this embodiment, when generating the summary text using the GPT-4o model, the input prompt word 1 is: "The input paragraph describes multiple video segments. [ID] represents the index of the video segment, and the following text content is the video text description of that video segment. Please summarize the input paragraph into a concise summary text and output the index (i.e., [ID]) of the video segment most relevant to the summary text. The output format should be JSON format: {{"abstract": "summary content", "segment": [ID related to the summary content]}}";

[0068] S1-4, Input the paragraph and event information obtained in S1-1 into the GPT-4o model, filter out the index of the video text description related to the event information through the GPT-4o model, and filter out the video sub-segment corresponding to the index from N video sub-segments according to the index and use it as the key video sub-segment;

[0069] S1-5, in the key video sub-segment, obtain the video frame set according to the selection rules, input the selected video frame set and prompt word 2 into the GPT-4o model, and output the numbers of M key frames through the GPT-4o model, so that the M key frames with corresponding numbers form a key frame set.

[0070] The selection rule is as follows: in the key video sub-segment, one video frame is selected every k seconds to obtain a set of video frames consisting of multiple video frames. The value of k ensures that the upper limit of the number of video frames in the set does not exceed the processing length of GPT-4o. The minimum value of k is 3, and the value of k can be increased by 0.5 each time it is adjusted.

[0071] In this embodiment, when selecting keyframes using the GPT-4o model, the input prompt word 2 is: "Video summarization refers to the process of extracting and generating concise and refined content from a longer video. By reducing redundancy and selecting the most important segments, it enables users to quickly understand the main content without watching the entire video. You need to select keyframes from the video frame set to represent the core content of the video. The frame numbers in the video frame set start with 0 for the first input video frame and increment by 1 sequentially. Please output the selected keyframe numbers in JSON format, which is: {"key_frames": [selected keyframe numbers]}".

[0072] S2, the task allocation agent receives the keyframe set output by the video summarization agent, and transmits the keyframe set and the image style requirements from the input data to the image style transfer agent. The image style transfer agent sequentially removes captions and performs style transfer on all keyframes in the keyframe set, and outputs the target style image set and transmits it to the task allocation agent. The image style transfer agent is used to perform style transfer on the keyframes.

[0073] The specific steps are as follows:

[0074] S2-1, Traverse the keyframe set, call the subtitle removal tool on the keyframes containing subtitles in the keyframe set, remove the subtitles in the keyframes, and obtain a keyframe set without subtitles;

[0075] In this embodiment, the GroundingDINO model is used to identify the location of the subtitles in the keyframe containing the subtitles, and a mask image is generated in which the subtitle part is white and the rest is black. The keyframe and the mask image are then input into the LaMa model to remove and complete the subtitle part in the keyframe containing the subtitles.

[0076] S2-2: Check if a diffusion model that matches the image style requirements and has been trained already exists. If it does not exist, retrieve the diffusion model that matches the image style requirements and has been trained. If it exists, proceed directly to S2-3.

[0077] The method for obtaining a diffusion model that corresponds to the image style requirements and has been trained is as follows:

[0078] A large dataset of image text descriptions is obtained. Based on the image style requirements in the input data, the GPT-4o model is used to rewrite the image text descriptions in the dataset, thereby generating paired text descriptions. These paired text descriptions include: image text descriptions and image text descriptions related to the image style requirements. In this embodiment, the Prompt2Prompt model is used to generate a large number of paired images based on the paired text descriptions; that is, images corresponding to the image text descriptions and images corresponding to the "image text descriptions related to the image style requirements." A stable diffusion model is then trained based on these large number of paired images to obtain a diffusion model that corresponds to the image style requirements and is fully trained.

[0079] In this embodiment, a picture description can be: a picture of a girl riding a horse; the picture style is required to be classic black and white cartoon style; the paired text descriptions are: a picture of a girl riding a horse, a picture of a girl riding a horse in classic black and white cartoon style.

[0080] S2-3, The image style transfer agent uses a diffusion model that corresponds to the image style requirements and has been trained to perform style transfer on each keyframe in the set of keyframes without subtitles, to obtain the corresponding target style image and its text description, which together form the target style image set.

[0081] Keyframes in a set of keyframes without subtitles, such as Figure 3 As shown in (a) to (e), and Figure 3 The target style images in the set of target style images that correspond one-to-one with (a) to (e) are as follows: Figure 3 As shown in (f)~(j).

[0082] S3, the task allocation agent receives the target style image set and reviews each target style image in the set, obtaining a list of reviewed target style images. The specific steps are as follows:

[0083] S3-1: The task allocation agent receives the target style image set output by the image style transfer agent, traverses the target style image set, and checks the image quality of all target style images. If the image quality of the current target style image is qualified, the next target style image is checked. If the image quality of the current target style image is unqualified, the execution of step S3-2 is performed.

[0084] S3-2, the image style transfer agent regenerates the target style image and reviews its quality. The regeneration and review steps are repeated until the image quality of the target style image is qualified or the maximum number of attempts is reached. Then, the last generated target style image is selected as the reviewed target style image.

[0085] S3-3, after reviewing each target style image in the target style image set, we obtain a set of reviewed target style images, List = {p1, p2, ..., p...} i ,…,p M}, M is the number of target style images in the set List, i∈[1,M], p i Let i be the i-th target style image in the collection List.

[0086] In this embodiment, the maximum number of attempts is 3.

[0087] S4, the task allocation agent transmits the List to the layout generation agent. The layout generation agent performs row and column layout for each page based on the List, fine-tunes the target style images on the page to obtain the main image, scales the main image with a fixed aspect ratio, and fills the corresponding position of the target style image on the page to obtain the comic page. The above operations are performed on each page until all target style images in the List have been processed. The layout generation agent outputs the generated comic pages as a comic book.

[0088] The specific steps are as follows:

[0089] S4-1, randomly select an integer a within the range [4,7], compare the size of integer a and M, and select the smaller value as the number of panels m1 on the page; select the set List = {p1,p2,…,p i ,…,p M In the context of the target style image p1 to the m1st target style image p1, ... m1 ,get

[0090] S4-2, the layout generation agent performs row layout operations on the page based on the number of panels m1. Specifically:

[0091] Construct the prompt text, including: the number of target style images in each line does not exceed the number of panels m1; important target style images occupy a large area on the page; the aspect ratio of the target style images, wherein the importance of the target style images is determined based on the text description of the target style images;

[0092] The prompt text is input into the GPT-4o model, and the GPT-4o model is used to analyze the target style image. Perform line breaks to obtain the line break results, including: the total number of lines on the page, the number of target style images in each line, and the relative height of each line;

[0093] The average aspect ratio of the target style images in each row is calculated based on the row splitting results. If the average aspect ratio of the row is not within the preset range, the GPT-4o model is used to split the target style images in that row until the maximum number of attempts is reached or the average aspect ratio of the row is within the preset range. If the row splitting results still do not meet the requirement that the average aspect ratio of each row is within the preset range after reaching the maximum number of attempts, the result of the last row splitting is used as the row splitting result. The preset range of the average aspect ratio of the target style images in each row is 0.6 to 1.7.

[0094] The maximum number of attempts is 3; if the GPT-4o model is suitable for the target style image The page was split into three rows. The average aspect ratio of the first two rows was within the preset range, but the average aspect ratio of the third row was not within the preset range. Therefore, only the third row was rearranged, for example, by generating an additional row or reducing the number of target style images in that row, while the first two rows remained unchanged.

[0095] S4-3, the layout generation agent performs column layout operations on the page based on the number of panels m1 and the row division results. Details are as follows:

[0096] Construct prompt text, including: the number of target style images in each column does not exceed the number of panels m1; important target style images occupy a large area on the page; the aspect ratio of the target style images, wherein the importance of the target style images is determined based on the text description of the target style images;

[0097] The prompt text is input into the GPT-4o model, which then uses the GPT-4o model to analyze the target style image based on the line breaks of the page. Split the data into columns to obtain the following results: the number of columns in each row, the number of target style images in each column, and the width of each column;

[0098] The average aspect ratio of the target style images in each column is calculated based on the splitting results. If the average aspect ratio of the target style images in that column is not within the preset range, the target style images in the corresponding rows are split again until the maximum number of attempts (3 times) is reached or the average aspect ratio of each column is within the preset range. If the splitting results after reaching the maximum number of attempts still fail to meet the requirement that the average aspect ratio of each column is within the preset range, the result of the last splitting is used as the final splitting result. The maximum number of attempts is 3, and the preset range for the average aspect ratio of the target style images in each column is 0.6 to 1.7.

[0099] S4-4, Target style image for this page Make minor adjustments:

[0100] Iterate through all images of the target style on the page. The GPT-4o model is used to generate a subject description for the target style image. The target style image and its corresponding subject description are then input into the GroundingDINO model to generate a coordinate frame diagram of the subject object corresponding to the target style image, and the aspect ratio of the coordinate frame diagram is calculated. The coordinate frame diagram of the subject object should completely cover the subject object, which is the main target object in the subject description.

[0101] Based on the aspect ratio of the main object coordinate frame and the aspect ratio of the corresponding position planned for the target style image on the page (calculated based on the page's line break and column break results), the target style image where the main object coordinate frame is located is expanded using the outpainting tool;

[0102] The expanded target style image and its corresponding subject description are input into the GroundingDINO model to obtain the coordinate frame diagram of the subject object corresponding to the expanded target style image. The width of this coordinate frame diagram is w1, and the height is h1. Let w2 = w1, and calculate the aspect ratio k of the corresponding position planned in the target style image. Let h3 = h1. Based on the aspect ratio k of the corresponding position of the target style image, calculate w3 = k × h3. Determine whether w2 × h2 is greater than w3 × h3. If so, select w2 as the width of the main image and h2 as the height of the main image. Otherwise, select w3 as the width of the main image and h3 as the height of the main image.

[0103] Using the center coordinates of the main object's coordinate frame (corresponding to the expanded target style image) as the center, extract the main image from the target style image based on the width and height of the main image.

[0104] S4-5, the main image is scaled down and filled into the corresponding position of the target style image on the page under a fixed aspect ratio setting to obtain the comic page.

[0105] S4-6, repeat S4-1 to S4-5 until all target style images in the collection List have been processed, resulting in multiple comic pages, which are then combined to form a personalized comic book that meets the user's needs and output.

[0106] In this embodiment, a page of a personalized comic book is like... Figure 4 As shown in the image, the key character and scene information from the five input keyframes was successfully extracted and naturally integrated into a complete comic strip. The overall style is consistent, the scene transitions are smooth, and it presents the unique artistic feel of comics.

[0107] The present invention has been described above by way of example. It should be noted that any simple modifications, alterations or other equivalent substitutions that can be made by those skilled in the art without creative effort without departing from the core of the present invention fall within the protection scope of the present invention.

Claims

1. A video-to-comic conversion method based on a multimodal large model, characterized in that, include: S1, Input data is input into the task allocation agent. The task allocation agent transmits the video and event information in the input data to the video summarization agent. The video summarization agent obtains segments based on the video in the input data, and obtains key video sub-segments based on the summary text corresponding to the segments or the event information in the input data. Key frames are obtained from the key video sub-segments according to the selection rules, forming a key frame set. The video summarization agent outputs the set of keyframes to the task-assigning agent; S2, the task allocation agent transmits the received keyframe set and the image style requirements in the input data to the image style transfer agent; The image style transfer agent performs subtitle removal and style transfer on the keyframes in the keyframe set in sequence to obtain the target style image set, and outputs the target style image set to the task allocation agent; S3, the task allocation agent reviews the image quality of each target style image in the received target style image set in turn. If the image quality is not up to standard, the image style transfer agent regenerates the target style image until its quality is up to standard or the maximum number of attempts is reached, and obtains the set List of reviewed target style images. S4, the task allocation agent transmits the collection List to the layout generation agent. The layout generation agent selects the number of panels on the page and the target style image to be placed on the page, and performs the row and column layout of the page. It fine-tunes the target style image on the page to obtain the main image. The main image is scaled and filled into the corresponding position of the target style image on the page under a fixed aspect ratio setting to obtain the comic page. Perform the above operation on each page until all target style images in the collection List have been processed; The layout generation agent outputs multiple comic pages as a comic book.

2. The video-to-comic conversion method according to claim 1, characterized in that, Step S1 specifically includes: S1-1, Input data is input into the task allocation agent. The task allocation agent transmits the video and event information in the input data to the video summarization agent. In the video summarization agent, a video scene detection tool is used to segment the video in the input data into N video sub-segments, and a multimodal large model is used to generate a corresponding video text description for each video sub-segment. An index is set before the first character of the video text description corresponding to each video sub-segment. All video text descriptions with indexes are merged into one paragraph. S1-2, the video summarizing agent determines whether the content of the event information in the input data is empty: if it is empty, then execute S1-3; if it is not empty, then execute S1-4. S1-3, the video summarizing agent inputs the paragraph obtained in S1-1 into the multimodal big model and inputs prompt word 1. The multimodal big model generates a summary text for the paragraph and outputs the index corresponding to the video text description most relevant to the summary text. Based on the index, the video sub-segment corresponding to the index is selected from N video sub-segments and used as the key video sub-segment. S1-4, Input the paragraph and event information obtained in S1-1 into the multimodal large model, filter out the index of the video text description related to the event information through the multimodal large model, and filter out the video sub-segment corresponding to the index from N video sub-segments according to the index and use it as the key video sub-segment; S1-5, in the key video sub-segment, obtain the video frame set according to the selection rules, input the selected video frame set and prompt word 2 into the multimodal large model, and output the numbers of M key frames through the multimodal large model, so that the M key frames with corresponding numbers form a key frame set.

3. The video-to-comic conversion method according to claim 1, characterized in that, Step S2 specifically includes: S2-1, Traverse the keyframe set, call the subtitle removal tool on the keyframes containing subtitles in the keyframe set, remove the subtitles in the keyframes, and obtain a keyframe set without subtitles; S2-2: Check if a diffusion model that matches the image style requirements and has been trained already exists. If it does not exist, retrieve the diffusion model that matches the image style requirements and has been trained. If it exists, proceed directly to S2-3. S2-3, The image style transfer agent uses a diffusion model that corresponds to the image style requirements and has been trained to perform style transfer on each keyframe in the set of keyframes without subtitles, to obtain the corresponding target style image and its text description, which together form the target style image set.

4. The video-to-comic conversion method according to claim 1, characterized in that, Step S3 specifically includes: S3-1: The task allocation agent receives the target style image set output by the image style transfer agent, traverses the target style image set, and checks the image quality of all target style images. If the image quality of the current target style image is qualified, the next target style image is checked. If the image quality of the current target style image is unqualified, S3-2 is executed. S3-2, the image style transfer agent regenerates the target style image and reviews its quality. The regeneration and review steps are repeated until the image quality of the target style image is qualified or the maximum number of attempts is reached. Then, the last generated target style image is selected as the reviewed target style image. S3-3, after reviewing each target style image in the target style image set, we obtain a set of reviewed target style images, List = {p1, p2, ..., p...} i ,…,p M }, M is the number of target style images in the set List, i∈[1,M], p i Let i be the i-th target style image in the collection List.

5. The video-to-comic conversion method according to claim 1, characterized in that, Step S4 specifically includes: S4-1, randomly select an integer a within the range [4,7], compare the size of integer a and M, and select the smaller value as the number of panels m1 on the page; select the set List = {p1,p2,…,p i ,…,p M In the context of the target style image, p1 to m1 are the first target style images. get S4-2, The layout generation agent performs row layout operations on the page based on the number of panels m1: Construct prompt text 1; input the prompt text into the multimodal large model, and use the multimodal large model to analyze the target style image. The process is split into rows to obtain the row splitting results. Based on the row splitting results, the average aspect ratio of the target style images in each row is calculated. If the average aspect ratio of the row is not within the preset range, a multimodal large model is used to split the target style images contained in the row into rows until the maximum number of attempts is reached or the average aspect ratio of the row is within the preset range. If the row splitting results still cannot satisfy the requirement that the average aspect ratio of each row is within the preset range after reaching the maximum number of attempts, the result of the last row splitting is used as the row splitting result. S4-3, the layout generation agent performs column layout operations on the page based on the number of panels m1 and the row break results: Construct prompt text 2; input the prompt text into the multimodal large model, and use the multimodal large model based on the line breaks of this page to select the target style image. Split the columns to obtain the split results; Calculate the average aspect ratio of the target style images in each column based on the splitting results. If the average aspect ratio of the target style images in that column is not within the preset range, then re-split the target style images in the row to which that column belongs until the maximum number of attempts is reached or the average aspect ratio of each column is within the preset range. If the splitting results after reaching the maximum number of attempts do not meet the requirement that the average aspect ratio of each column is within the preset range, then the result of the last splitting is taken as the splitting result. S4-4, Target style image for this page Make minor adjustments: Iterate through all images of the target style on the page. A multimodal large model is used to generate a subject description for the target style image; the target style image and its corresponding subject description are input into the object detection model to generate a coordinate frame of the subject object corresponding to the target style image, and the aspect ratio of the coordinate frame of the subject object is calculated. Based on the aspect ratio of the main object coordinate frame and the aspect ratio of the corresponding position planned for the target style image on the page, the target style image where the main object coordinate frame is located is expanded using an image expansion tool. The expanded target style image and its corresponding subject description are input into the object detection model to obtain the coordinate frame of the subject object corresponding to the expanded target style image. The width of the coordinate frame is w1 and the height is h1. Let w2 = w1, and calculate the aspect ratio k of the corresponding position planned in the target style image. Let h3 = h1. Based on the aspect ratio k of the corresponding position of the target style image, calculate w3 = k × h3. Determine whether w2 × h2 is greater than w3 × h3. If so, select w2 as the width of the main image and h2 as the height of the main image. Otherwise, select w3 as the width of the main image and h3 as the height of the main image. Using the center coordinates of the main object's coordinate frame as the center, extract the main image from the main object's coordinate frame according to the width and height of the main image; S4-5, the main image is scaled down and filled into the corresponding position of the target style image on the page under a fixed aspect ratio setting to obtain the comic page; S4-6, repeat S4-1 to S4-5 until all target style images in the collection List have been processed, resulting in multiple comic pages, which are then combined to form a personalized comic book that meets the user's needs and output.

6. The video-to-comic conversion method according to claim 2, characterized in that, The selection rules in steps S1-5 are as follows: in the key video sub-segment, one video frame is selected every k seconds to obtain a video frame set composed of multiple video frames. The value of k ensures that the upper limit of the number of video frames in the video frame set does not exceed the processing length. The minimum value of k is 3, and the value of k is increased by 0.5 each time it is adjusted.

7. The video-to-comic conversion method according to claim 3, characterized in that, The method for obtaining the trained diffusion model that corresponds to the image style requirements in step S2-2 is as follows: A large dataset of image text descriptions is obtained. Based on the image style requirements in the input data, a multimodal large model is used to rewrite the image text descriptions in the dataset, thereby generating paired text descriptions. The paired text descriptions include: image text descriptions and image text descriptions related to image style requirements. Based on the paired text descriptions, images corresponding to the image text descriptions and image text descriptions related to image style requirements are generated respectively. A stable diffusion model is trained based on a large number of paired images to obtain a diffusion model that corresponds to the image style requirements and is fully trained.

8. The video-to-comic conversion method according to claim 5, characterized in that, In step S4-2, the prompt text 1 includes: the number of target style images in each row does not exceed the number of panels m1, important target style images occupy a large area on the page, and the aspect ratio of the target style images; the importance of the target style images is determined based on the text description of the target style images; the preset range of the average aspect ratio of each row of target style images is 0.6 to 1.

7.

9. The video-to-comic conversion method according to claim 5, characterized in that, In step S4-3, the prompt text 2 includes: the number of target style images in each column does not exceed the number of panels m1; important target style images occupy a large area on the page; the aspect ratio of the target style images; the importance of the target style images is determined based on the text description of the target style images; the preset range of the average aspect ratio of each column of target style images is 0.6 to 1.

7.

10. The video-to-comic conversion method according to claim 5, characterized in that, In step S4-4, the coordinate frame diagram of the main object should completely cover the main object, which is the main target object in the main description.

Images