Multi-mode convergence media content auxiliary creation method based on AI technology

Abstract

The invention discloses a multi-mode convergence media content auxiliary creation method based on an AI technology, and the method comprises the steps: extracting key features according to a creation demand text inputted by a user, carrying out the retrieval in a knowledge base through employing a knowledge graph retrieval mode according to the key features, obtaining a creation material, and generating a first draft; carrying out cross-modal feature mapping on the first draft by adopting a multi-modal alignment model, aligning text-image-video embedded vectors through contrastive learning, and dynamically adjusting the correlation of multi-modal contents by utilizing an attention mechanism to obtain a multi-modal content packet; duplicate checking is carried out on the multi-modal content packet in multiple modes, and the multi-modal content packet after duplicate checking is sent to a user for manual editing. Through multi-mode processing methods such as AI auxiliary writing, AI illustration and video generation and whole-network duplicate checking and propagation value evaluation, the content production efficiency and quality are improved, the propagation effect is optimized, and the original content is protected.

Description

Technical Field

[0001] The present invention belongs to the field of converged media technology, and specifically relates to a multimodal converged media content assisted creation method based on AI technology. Background Art

[0002] Currently, converged media content creation relies primarily on manual labor, which is inefficient and unable to meet large-scale content demands. While some AI-assisted tools exist on the market (such as AI writing tools and AI image generation tools), these tools typically have limited functionality and struggle to achieve the collaborative production of multimodal content. Furthermore, existing technologies lack scientific evaluation of the dissemination value of content, making it difficult for some content to achieve the desired dissemination effect. Furthermore, plagiarism and duplicate content remain widespread, creating an urgent need for a tool that can instantly perform online duplication checks at the time of content creation to avoid copyright disputes.

[0003] Currently, mainstream converged media content-assisted creation technology solutions fall into two categories: Single-modal AI generation tools, such as Transformer-based text generation models that independently write articles or Generative Adversarial Networks (GANs), lack data interaction and semantic associations between modal tools. Traditional content duplication detection systems primarily employ a text comparison scheme combining TF-IDF and cosine similarity, with some systems also employing local sensitive hashing (LSH) technology for image duplication detection. However, these solutions lack cross-modal collaboration, predicting communication value, and detecting complex plagiarism. Summary of the Invention

[0004] In response to the above-mentioned shortcomings, the present invention discloses a multimodal integrated media content assisted creation method based on AI technology. Through AI-assisted writing, AI illustration, and multimodal processing methods for video generation, as well as full-network duplication checking and dissemination value assessment, the present invention solves the technical problems of traditional content production methods such as low content production efficiency, unstable content quality, poor content dissemination effect, plagiarism and duplicate content problems, improves content production efficiency and quality, optimizes dissemination effects, and protects original content.

[0005] The present invention is achieved by adopting the following technical solutions: A multimodal integrated media content assisted creation method based on AI technology, comprising the following steps: (1) Based on the creative demand text input by the user, the creative demand text is parsed based on the intention recognition model of natural language processing (NLP) to obtain key features, which include keywords and regional features; based on the key features, the creative materials are retrieved from the knowledge base using the knowledge graph retrieval method, which include one or more of pictures, audio, video, text, and local language; A draft is generated based on the creative material using a draft generation model. Specifically, a draft text is generated based on the text in the creative material using a pre-trained language model of Transformer; a draft image is generated based on the pictures and text in the creative material using a conditional generative adversarial network (cGAN) or a diffusion model; a storyboard is generated based on the pictures and text in the creative material using a preset narrative template, and then a draft video is synthesized using a time-series action generation model; (2) Based on one or more of the creative materials, draft text, draft image, and draft video described in step (1), a multimodal alignment model (CLIP) is used to perform cross-modal feature mapping, and by learning to align the embedding vectors of text, image, and video through comparison, the relevance of multimodal content is dynamically adjusted using the attention mechanism (Cross-Attention) to obtain a multimodal content package; (3) Check the multimodal content package obtained in step (2) for duplicates. First, compare the content package with the data in the corresponding region in the knowledge base based on the regional characteristics, and then perform a multi-level check for duplicates with all the data in the knowledge base and the cloud data connected to the network. The multi-level duplication check includes a first-level duplication check and a second-level duplication check. The first-level duplication check includes text duplication check, image duplication check, and video duplication check. The text duplication check extracts semantic fingerprints from the text and then compares the text in the knowledge base based on the semantic fingerprints. The image duplication check uses a pre-trained CNN to extract image feature encoding and compares it with the image library in the knowledge base through cosine similarity calculation. The video duplication check extracts the key frame sequence of the video, uses a time series model (LSTM) to encode the relationship between frames based on the key frame sequence, and compares the spatiotemporal features of the video library. The secondary duplication check is used to detect cross-modal plagiarism, compare the text semantic fingerprint with the video keyframes through multi-modal embedding, and construct a cross-modal association graph through a graph neural network (GNN) to identify potential plagiarism paths; (4) The multimodal content package that has been checked for duplicate content in step (3) is sent to the user for manual editing. Content features are extracted from the manually edited multimodal content package. The obtained content features are fed back into the model through reinforcement learning (RL) to optimize the draft generation model described in step (1).

[0006] Furthermore, in step (1), the pictures in the creative material include pictures of patterns and logos of different places, iconic attractions and buildings.

[0007] Furthermore, in step (3), the text duplication check is to extract semantic fingerprints from the text and then compare the texts in the knowledge base based on the semantic fingerprints, which specifically includes the following steps: S11. Preprocess the text by segmenting and removing stop words, and then extract the core features; S12. Generate context-aware text vectors based on core features using a pre-trained language model (RoBERTa); S13, after PCA dimensionality reduction is applied to the text vector, a 64-bit fingerprint code is generated through local sensitive hashing (LSH); S14. Calculate the Hamming distance between the target text and the fingerprints in the library based on the fingerprint code. If the distance is within a preset threshold, it is determined to be a duplicate.

[0008] Furthermore, in step (3), the image duplication check is to extract image feature encoding using a pre-trained CNN and compare it with the image library in the knowledge base through cosine similarity calculation, specifically including the following steps: S21. Use the pre-trained Inception-v3 model to extract global image features; S22. Based on the global features of the image, after strengthening the regional characteristic elements through the self-attention mechanism, the global features of the image are vectorized to obtain the feature vector; S23. Binarize the feature vector into a hash code and use it in combination with an inverted index to accelerate retrieval and comparison.

[0009] Furthermore, in step (3), the video duplication check is to extract the key frame sequence of the video, use the time series model (LSTM) to encode the relationship between frames according to the key frame sequence, and compare the spatiotemporal features of the video library, which specifically includes the following steps: S31, extracting representative frames based on shot boundary detection to form a key frame sequence; S32. Use 3D-CNN to extract spatiotemporal features from key frame sequences and combine it with temporal position encoding (Transformer) to model temporal relationships. S33. Use the dynamic time warping (DTW) algorithm to match the similarity of video sequences and prioritize them based on the data in the knowledge base.

[0010] Furthermore, in step (3), the secondary duplication check is used to detect whether it is cross-modal plagiarism, and the text semantic fingerprint is compared with the video key frame through multimodal embedding, wherein the text and video key frame are processed by the multimodal alignment model to obtain high-dimensional feature vectors of the text and video key frame, and then the similarity is calculated based on the high-dimensional feature vector. If the similarity is within the preset threshold range, a duplication alarm is prompted.

[0011] Compared with the existing technology, this technical solution has the following beneficial effects: 1. This invention uses AI-assisted multimodal content collaborative creation as its core, constructing an intelligent generation system covering text, images, and videos, and achieving content consistency output of text, images, and audio through cross-modal semantic alignment technology. Its integrated communication value prediction module generates a quantitative evaluation and optimization strategy for communication effects based on user behavior analysis, content feature extraction, and environmental parameter modeling, combined with the historical communication case library and regional user portraits in the knowledge base. At the same time, a multi-level, full-network duplication detection mechanism is established, and cross-modal plagiarism behavior can be accurately identified through technologies such as text semantic fingerprinting, image feature encoding, and video frame sequence analysis.

[0012] 2. The present invention realizes the automatic integration of content generation and regional cultural characteristics by extracting regional element features from the knowledge base, and at the same time connects the public data of the entire network with the private copyright library of the knowledge base to establish a regional content priority protection channel. Moreover, based on the content dissemination feedback data, the regional hot spot map and copyright feature library in the knowledge base are continuously updated. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is a flowchart of the multimodal integrated media content assisted creation method based on AI technology described in Example 1. DETAILED DESCRIPTION

[0014] The present invention is further illustrated by the following examples, which are not intended to limit the present invention. Specific experimental conditions and methods not specified in the following examples are conventional methods well known to those skilled in the art.

[0015] The scheme described in the following embodiments is applied in combination with the knowledge base of the Guangxi Media Convergence Three-Level Interconnection Platform established by the applicant of the present invention. The knowledge base of the Guangxi Media Convergence Three-Level Interconnection Platform integrates the technical platform resources at the Guangxi Zhuang Autonomous Region level, accommodates the media history and newly added manuscript resources of the entire Guangxi region, and integrates the distributed knowledge graph of the provincial-municipal-district-county level integrated media platform data through technical means, wherein independent hot topic libraries, local cultural keyword libraries and historical high-quality content narrative mode libraries are set up to provide support for copywriting. At the same time, the collection of user data and integrated media publishing content is carried out with the authorization of users and publishers, and the knowledge base of the Guangxi Media Convergence Three-Level Interconnection Platform is equipped with a security protection mechanism.

[0016] Example 1: A multimodal integrated media content assisted creation method based on AI technology, comprising the following steps: (1) Based on the creative demand text input by the user, the creative demand text is parsed based on the intention recognition model of natural language processing (NLP) to obtain key features, which include keywords and regional features; based on the key features, the creative materials are retrieved from the knowledge base of the Guangxi Media Convergence Three-Level Platform using a knowledge graph search method, which includes one or more of pictures, audio, video, text, and local language; the pictures in the creative materials include pictures of patterns and logos of different places, iconic attractions, and pictures of buildings; A draft is generated based on the creative material using a draft generation model. Specifically, a draft text is generated based on the text in the creative material using a pre-trained language model of Transformer; a draft image is generated based on the pictures and text in the creative material using a conditional generative adversarial network (cGAN) or a diffusion model; a storyboard is generated based on the pictures and text in the creative material using a preset narrative template, and then a draft video is synthesized using a time-series action generation model; (2) Based on one or more of the creative materials, draft text, draft image, and draft video described in step (1), a multimodal alignment model (CLIP) is used to perform cross-modal feature mapping, and by learning to align the embedding vectors of text, image, and video through comparison, the relevance of multimodal content is dynamically adjusted using the attention mechanism (Cross-Attention) to obtain a multimodal content package; (3) Check the multimodal content package obtained in step (2) for duplicates. First, compare and check the data in the knowledge base with the data of the corresponding region based on regional characteristics. The copyright resource library of the Guangxi Media Integration Three-Level Knowledge Base can be connected to give priority to matching the feature coding of the autonomous region-level media registration to improve the recall rate of regional original content. Then, perform multi-level duplicate checking with all the data in the knowledge base and the cloud data connected to the network. The multi-level duplication check includes a first-level duplication check and a second-level duplication check. The first-level duplication check includes text duplication check, image duplication check, and video duplication check. The text duplication check extracts semantic fingerprints from the text and then compares the text in the knowledge base based on the semantic fingerprints. The image duplication check uses a pre-trained CNN to extract image feature encoding and compares it with the image library in the knowledge base through cosine similarity calculation. The video duplication check extracts the key frame sequence of the video, uses a time series model (LSTM) to encode the relationship between frames based on the key frame sequence, and compares the spatiotemporal features of the video library. The text duplication check is to extract semantic fingerprints from the text and then compare the texts in the knowledge base based on the semantic fingerprints. It specifically includes the following steps: S11. Preprocess the text by segmenting and removing stop words, and then extract the core features; S12. Generate context-aware text vectors based on core features using a pre-trained language model (RoBERTa); S13, after PCA dimensionality reduction is applied to the text vector, a 64-bit fingerprint code is generated through local sensitive hashing (LSH); S14, calculating the Hamming distance between the target text and the fingerprints in the library based on the fingerprint code, and determining it as a duplicate if it is within a preset threshold range; The image duplication check uses a pre-trained CNN to extract image feature encoding and compares it with the image library in the knowledge base through cosine similarity calculation, specifically including the following steps: S21, use the pre-trained Inception-v3 model to extract the global features of the image, which are output by the pooling layer; S22. Based on the global features of the image, after strengthening the regional characteristic elements through the self-attention mechanism, such as the Guangxi bronze drum patterns annotated in the knowledge base, the global features of the image are then vectorized to obtain the feature vector; S23, binarize the feature vector into a hash code, and use an inverted index method to accelerate retrieval and comparison; The video duplication check is to extract the key frame sequence of the video, use the time series model (LSTM) to encode the relationship between frames according to the key frame sequence, and compare the spatiotemporal features of the video library. Specifically, it includes the following steps: S31, extracting representative frames based on shot boundary detection to form a key frame sequence, for example, by using color histogram mutation analysis; S32. Use 3D-CNN to extract spatiotemporal features from key frame sequences and combine it with temporal position encoding (Transformer) to model temporal relationships. S33, using the dynamic time warping (DTW) algorithm to match the similarity of video sequences and prioritize them based on the data in the knowledge base; The secondary duplicate check is used to detect whether it is cross-modal plagiarism. The text semantic fingerprint is compared with the video key frame in a multimodal embedding. The text and video key frame are processed by a multimodal alignment model to obtain high-dimensional feature vectors of the text and video key frame. The similarity is then calculated based on the high-dimensional feature vector. If the similarity is within a preset threshold, a duplication alarm is prompted. A cross-modal association map is constructed through a graph neural network (GNN) to identify potential plagiarism paths. (4) The multimodal content package that has been checked for duplicate content in step (3) is sent to the user for manual editing. Content features are extracted from the manually edited multimodal content package. The obtained content features are fed back into the model through reinforcement learning (RL) to optimize the draft generation model described in step (1).

[0017] By creating converged media content according to the scheme described in this embodiment, the converged media content production cycle can be greatly shortened, cross-modal consistency is good, the propagation prediction model has a high AUC value in regional scenarios, and has good classification accuracy. In addition, by connecting to the three-level interconnected knowledge base, the recall rate of content with regional characteristics and the accuracy of intelligent recommendation of local cultural elements are further improved.

[0018] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A multimodal integrated media content assisted creation method based on AI technology, characterized by: The following steps are involved: (1) Based on the creation requirement text input by the user, the intention recognition model based on natural language processing is used to parse the creation requirement text to obtain key features, wherein the key features include keywords and regional features; Retrieve creative materials from the knowledge base using a knowledge graph retrieval method based on key features, where the creative materials include one or more of images, audio, video, text, and local languages; Generate a draft based on the creative material using a draft generation model. Specifically, generate a draft text based on the text in the creative material using a pre-trained language model of Transformer; generate a draft image based on the pictures and text in the creative material using a conditional generative adversarial network or a diffusion model; generate a storyboard based on the pictures and text in the creative material using a preset narrative template, and then synthesize the storyboard to obtain a draft video using a time-series action generation model; (2) Based on one or more of the creative materials, draft text, draft image, and draft video described in step (1), a multimodal alignment model is used to perform cross-modal feature mapping, and the embedding vectors of the aligned text-image-video are learned by comparison, and the relevance of the multimodal content is dynamically adjusted using the attention mechanism to obtain a multimodal content package; (3) Check the multimodal content package obtained in step (2) for duplicates. First, compare the content package with the data in the corresponding region in the knowledge base based on the regional characteristics, and then perform a multi-level check for duplicates with all the data in the knowledge base and the cloud data connected to the network. The multi-level duplication check includes a first-level duplication check and a second-level duplication check, wherein the first-level duplication check includes text duplication check, image duplication check and video duplication check; the text duplication check extracts semantic fingerprints from the text and then compares the text in the knowledge base based on the semantic fingerprints; the image duplication check uses a pre-trained CNN to extract image feature encoding and compares it with the image library in the knowledge base through cosine similarity calculation; the video duplication check extracts the key frame sequence of the video, uses a temporal model to encode the relationship between frames based on the key frame sequence, and compares the spatiotemporal features of the video library; The secondary duplication check is used to detect cross-modal plagiarism, perform multimodal embedding comparison between text semantic fingerprints and video key frames, and construct a cross-modal association graph through a graph neural network to identify potential plagiarism paths; (4) The multimodal content package that has been checked for duplicate content in step (3) is sent to the user for manual editing, and content features are extracted from the manually edited multimodal content package. The obtained content features are fed back into the reinforcement learning model to optimize the draft generation model described in step (1).

2. The AI-based multimodal media content assisted creation method according to claim 1 is characterized by: In step (1), the pictures in the creative material include pictures of patterns and logos of different places, iconic attractions and buildings.

3. The AI-based multimodal media content assisted creation method according to claim 1 is characterized by: In step (3), the text duplication check is to extract semantic fingerprints from the text and then compare the texts in the knowledge base based on the semantic fingerprints, which specifically includes the following steps: S11. Preprocess the text by segmenting and removing stop words, and then extract the core features; S12. Generate context-aware text vectors based on core features using a pre-trained language model; S13, after PCA dimensionality reduction is performed on the text vector, a 64-bit fingerprint code is generated through local sensitive hashing; S14. Calculate the Hamming distance between the target text and the fingerprints in the library based on the fingerprint code. If the distance is within a preset threshold, it is determined to be a duplicate.

4. The AI-based multimodal media content assisted creation method according to claim 1 is characterized by: In step (3), the image duplication check is to extract image feature encoding using pre-trained CNN and compare it with the image library in the knowledge base through cosine similarity calculation, which specifically includes the following steps: S21. Use the pre-trained Inception-v3 model to extract global image features; S22. Based on the global features of the image, after strengthening the regional characteristic elements through the attention mechanism, the global features of the image are vector-converted to obtain the feature vector; S23. Binarize the feature vector into a hash code and use it in combination with an inverted index to accelerate retrieval and comparison.

5. The AI-based multimodal media content assisted creation method according to claim 1 is characterized by: In step (3), the video duplication check is to extract the key frame sequence of the video, encode the relationship between frames using a temporal model based on the key frame sequence, and compare the spatiotemporal features of the video library, which specifically includes the following steps: S31, extracting representative frames based on shot boundary detection to form a key frame sequence; S32. Use 3D-CNN to extract spatiotemporal features from key frame sequences and combine them with time position coding to model temporal relationships. S33. Use dynamic time warping algorithm to match the similarity of video sequences and prioritize them based on the data in the knowledge base.

6. The AI-based multimodal media content assisted creation method according to claim 1 is characterized by: In step (3), the secondary duplication check is used to detect whether it is cross-modal plagiarism, and the text semantic fingerprint is compared with the video key frame through multimodal embedding, wherein the text and video key frame are processed by the multimodal alignment model to obtain the high-dimensional feature vectors of the text and video key frame, and then the similarity is calculated based on the high-dimensional feature vector. If the similarity is within the preset threshold range, a duplication alarm is prompted.

Images