Content creation and multi-mode display platform based on AIGC

Abstract

The invention discloses a content creation and multi-mode display platform based on AIGC. The system comprises six modules of user interaction, content creation intelligent collaboration, AIGC auxiliary creation, data management, multi-modal deep fusion display and user feedback. The user interaction module supports various inputs and provides a creation template; the content creation intelligent collaboration module analyzes the intention and intelligently distributes tasks; the AIGC auxiliary creation module provides multi-mode creation assistance; the data management module stores various data; the multi-modal deep fusion display module is used for mining modal connection to realize deep fusion display; and the user feedback module collects and feeds back optimization contents and algorithms. Through cooperative work of the modules, the problems that an existing platform is difficult in creation cooperation, single in content generation, insufficient in multi-mode display and the like are effectively solved, creation efficiency and content quality are improved, immersive experience is brought to users, and innovative development of the field of content creation and display is promoted.

Description

Technical Field

[0001] The present invention relates to computer-related fields, and specifically to an AIGC-based content creation and multimodal display platform. Background Art

[0002] Current content creation and display platforms face challenges such as difficulty in creator collaboration, a single content generation model, and a lack of deep integration in multimodal presentation. Creators face high communication costs and struggle to efficiently manage progress when collaborating. Furthermore, existing AIGC tools are mostly single-function, unable to intelligently collaborate based on creator intent. Multimodal presentations often consist of a simple list of images, text, audio, and video, failing to fully explore the connections between different modalities. Summary of the Invention

[0003] The purpose of the present invention is to provide a content creation and multimodal display platform based on AIGC to solve the problems raised in the above background technology.

[0004] To achieve the above objectives, the present invention provides the following technical solutions: an AIGC-based content creation and multimodal display platform, comprising a user interaction module, an intelligent content creation collaboration module, an AIGC-assisted creation module, a data management module, a multimodal deep fusion display module, and a user feedback module. The user interaction module provides an operation interface, allowing creators to clearly define their creative intent through text input or voice input, and also provides several creative templates for users to apply.

[0005] The content creation intelligent collaboration module uses semantic understanding and task allocation algorithms. When the creator enters the creative intention, it uses natural language processing technology to deeply analyze the keywords in the creative intention.

[0006] The AIGC assisted creation module integrates a text generation model, an image generation model, and an audio generation model to provide assistance to users during their creation process based on their creative needs.

[0007] The data management module is responsible for storing and managing creator information, creative materials, creative history, and user feedback data. Specifically, it uses a distributed database architecture to classify and manage creator information, including personal information, areas of expertise, and creative achievements, for task allocation in the content creation intelligent collaboration module. At the same time, creative materials are managed in a tagged manner.

[0008] The multimodal deep fusion display module collects different modal contents generated by the content creation intelligent collaboration module and the AIGC auxiliary creation module, including text, pictures, video clips, and audio, and mines the intrinsic connections between the contents of each modality through cross-modal feature extraction and fusion algorithms;

[0009] The user feedback module sets a feedback entrance in the multimodal content display module, and users express their opinions on the content through ratings, text comments, and voice feedback; the user feedback module collects feedback data in real time and feeds it back to the creator and the data management module. The creator optimizes the content based on the feedback, providing a basis for subsequent content creation direction adjustments and algorithm optimization.

[0010] Preferably, the user interaction module specifically includes an intention input unit, a template calling unit and an auxiliary function unit, and its specific contents are as follows:

[0011] The intention input unit integrates a text input subunit and a voice input subunit. The text input subunit provides the creator with a standard text input box. At the same time, the input box can perform grammar checking in real time, mark incorrect grammar with wavy lines, and provide modification suggestions to improve the accuracy of the input text. The voice input subunit integrates a high-precision voice recognition engine. After the creator clicks the voice input button, he directly speaks the creative intention through the microphone, and the voice is quickly and accurately converted into text.

[0012] The template calling unit integrates a template display subunit and a template screening and search subunit, wherein the template display subunit displays various creative templates in the form of pictures and texts, wherein the creative templates are classified according to the content type; the template screening and search subunit is used for creators to search for templates by keywords, template popularity, creation time, and applicable fields;

[0013] The auxiliary function unit is integrated with a history record sub-unit and a help and feedback sub-unit, wherein the history record sub-unit automatically records each creative intention input record of the creator, including text input content, text converted by voice recognition and the corresponding creation time. The creator views past inputs in the history record list and clicks on the record to refill the content into the intention input unit for reusing similar creative needs; the help and feedback sub-unit sets a help button, which pops up a detailed platform usage guide after clicking it, covering the entire process from creative intention input to content publishing, and is equipped with graphic demonstrations and video tutorial links. At the same time, a feedback entrance is provided. If the creator encounters any problems or has improvement suggestions during use, he or she can submit feedback in the form of text or screenshots. The platform operator receives feedback information in real time, processes and replies to the creator in a timely manner, and continuously optimizes the platform usage experience.

[0014] Preferably, the specific implementation logic of the content creation intelligent collaboration module is as follows:

[0015] Creative Intention Analysis: When the creator inputs the creative intention through the user interaction module and converts it into text, the content creation intelligent collaboration module uses natural language processing technology and lexical analysis to split the input text into individual words or phrases, clarifying the part of speech of each word. It then uses syntactic analysis and semantic analysis technology to understand the grammatical relationship and semantic association between words, determine the subject-verb-object structure of the sentence, and grasp the core creative needs.

[0016] Creator information retrieval: Based on the keywords in the analyzed creative intent, a search is performed in the creator information database stored in the data management module. The database records in detail the labels of each creator's expertise. The labels are based on the information filled out when the creator registers and the platform's analysis of their past works. This is used to screen out a list of creators with corresponding expertise labels. For the screened creators, the number of works created, quality ratings, and average time to complete tasks are analyzed to assess the creator's creative ability and experience level. The real-time online status of the above creators is then obtained so that online creators can receive and process tasks in a timely manner, thereby improving creative efficiency.

[0017] Task allocation decision-making: Based on the creator's field of expertise, past creative data, and real-time online status, the task allocation algorithm is used to generate the optimal task allocation plan. After the task allocation plan is determined, the platform's message push system sends a task notification to the selected creator. The notification content details the task requirements, including creative intent, task deadline, and links to relevant reference materials;

[0018] Creation progress monitoring: key progress tracking nodes are set for each task allocation plan. Creators mark and confirm each node task on the platform after completing it.

[0019] Collaboration during the creative process: During the creative process, if creators have questions about task requirements or other creators' work, or need to communicate and collaborate, they can communicate through the built-in instant messaging function of the Content Creation Intelligent Collaboration Module. The instant messaging function supports text and voice messaging, as well as file transfers. The Content Creation Intelligent Collaboration Module automatically records the communication content between creators, including message logs and file transfer records, and stores the communication content in the data management module for subsequent review and experience summary.

[0020] Integration and delivery of task results: When all creators have completed their respective tasks, the intelligent content creation collaboration module automatically collects the task results and organizes them according to pre-set formats and specifications; the organized multimodal task results are then delivered to the multimodal deep fusion display module for subsequent cross-modal feature extraction and fusion processing, and finally presented to users.

[0021] Preferably, the specific working steps of the multimodal deep fusion display module are as follows:

[0022] Step 1: Multimodal content collection: The multimodal deep fusion display module establishes a data transmission connection with the content creation intelligent collaboration module and the AIGC auxiliary creation module. When the creator completes content creation, the creative results are automatically transmitted to this module. At the same time, the multimodal content of the creative results transmitted in different formats is format converted and uniformly processed;

[0023] Step 2: Cross-modal feature extraction:

[0024] Text feature extraction: Using natural language processing technology, key descriptive features are extracted for the text content in step 1, that is, a text feature vector is constructed through lexical analysis, syntactic analysis, and semantic understanding; Image feature extraction: Using edge detection, color analysis, and texture recognition algorithms, the visual features of the image in step 1 are extracted and converted into corresponding image feature vectors; Video feature extraction: For the video content in step 1, a spatiotemporal feature extraction algorithm is used to analyze the changes in different frames in the video in the time dimension; in the spatial dimension, the visual features of each frame of the image are extracted, and the time dimension and space dimension features are integrated to generate a video feature vector; Audio feature extraction: For the audio data in step 1, audio signal processing technology is used to extract frequency, amplitude, and timbre features and convert them into audio feature vectors;

[0025] Step 3: Cross-modal feature fusion: The feature vectors extracted from different modalities in step 2 are fused through the fusion algorithm of the attention mechanism. During the fusion process, different weights are assigned according to the degree of correlation between the features.

[0026] Step 4: Display logic construction: Based on the fused feature vectors, formulate the display and interaction rules of multimodal content according to customer needs;

[0027] Step 5: Content display and user interaction: When users access platform content, the multimodal deep fusion display module presents multimodal content in real time according to the display logic established in step 4.

[0028] Preferably, the specific working logic of cross-modal feature fusion in step 3 is as follows:

[0029] Step 31: Initialize the deep learning model: Select the Transformer architecture model for initialization and set the model's hyperparameters, including the number of layers, hidden layer dimensions, and number of attention heads. Adjust the model parameters based on the characteristics and scale of different modal data to ensure that the model can effectively learn the relationship between different modal features.

[0030] Step 32: Feature vector input and preprocessing: The text, image, video, and audio feature vectors extracted in step 2 are organized to ensure that their dimensions and formats meet the input requirements of the deep learning model. The feature vectors are normalized so that feature vectors of different modalities have similar scales and distributions, thus obtaining preprocessed feature vectors.

[0031] Step 33: Cross-modal feature correlation analysis: The preprocessed feature vectors are input into the deep learning model. The model uses a self-attention mechanism and a multi-layer neural network to parallelize and interactively learn the feature vectors of different modalities. In the self-attention mechanism, the model calculates the attention score between each feature vector element and all other elements, thereby capturing the potential correlation between features of different modalities. Through the nonlinear transformation of the multi-layer neural network, the complex relationship between feature vectors is further explored, the feature representation capability is enhanced, and the model can more accurately identify and understand the intrinsic connection between features of different modalities.

[0032] Step 34: Start the attention mechanism fusion algorithm: Based on the cross-modal feature correlation results obtained by the deep learning model in step 33, the results are input into the attention mechanism fusion algorithm to calculate the attention weight of each modal feature vector in the fusion process. The attention weight reflects the degree of correlation between different modal feature vectors and other modal feature vectors.

[0033] Step 35, feature vector fusion calculation: Based on the attention weights calculated in step 34, perform weighted fusion on the feature vectors of different modalities. Multiply the feature vector of each modality by its corresponding attention weight, and then sum or concatenate the weighted feature vectors to obtain a fused feature vector.

[0034] Step 36: Post-processing of fusion results: The fused feature vector is post-processed using a regularization method to further optimize feature representation, prevent model overfitting, and improve the generalization ability of the fused feature vector.

[0035] Compared with the existing technology, the beneficial effects of the present invention are: improving creative efficiency: through the intelligent collaboration module for content creation, the platform can accurately analyze creative intent and intelligently assign tasks based on the creator's areas of expertise, past data, and online status. This reduces communication costs and task allocation time, allowing each creator to focus on their areas of expertise. The creative progress monitoring function can promptly identify problems and improve overall creative efficiency. The AIGC auxiliary creation module provides real-time assistance, providing sentence suggestions during copywriting and generating sketches during image design, further accelerating the creative process.

[0036] Optimizing content quality: The multimodal deep integration display module explores the inherent connections between text, images, videos, and audio, making content presentation more logical and coherent. In tourism promotions, clicking on a text waterfall description highlights the corresponding area in the image, and specific video scenes enhance relevant audio, making the content more immersive and engaging. The data management module manages creative materials through labeling, making it easier for creators to obtain high-quality materials and enrich content details.

[0037] Improved user experience: The user interaction module offers convenient operation. Voice and text input, combined with grammar checking and voice recognition, facilitates the input of creative intent. The template call unit displays categorized content and allows for filtered search of templates, lowering the threshold for creative creation. The auxiliary function unit's history record reuse and help feedback features enhance ease of use and satisfaction. The multimodal deep fusion display module provides real-time interactive display, meeting diverse user needs and providing an immersive experience.

[0038] Driving the platform's sustainable development: The user feedback module collects ratings, comments, and voice feedback, allowing creators to optimize their content. The data management module analyzes this feedback and optimizes algorithms to further enhance platform functionality and better meet user needs. The use of a distributed database architecture ensures the secure storage and efficient management of creator information, creative materials, and other data, providing data support for the platform's long-term and stable development. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic diagram of the overall structure of the present invention;

[0040] Figure 2 This is a schematic diagram of the workflow of the content creation intelligent collaboration module of the present invention. DETAILED DESCRIPTION

[0041] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0042] See also Figure 1-2 The present invention provides a technical solution: an AIGC-based content creation and multimodal display platform, comprising a user interaction module, a content creation intelligent collaboration module, an AIGC-assisted creation module, a data management module, a multimodal deep fusion display module, and a user feedback module. The user interaction module provides an operation interface, and creators use text input and voice input to clarify their creative intent. At the same time, several creative templates are set for users to apply. The user interaction module specifically includes an intent input unit, a template calling unit, and an auxiliary function unit. The specific contents are as follows:

[0043] The intent input unit integrates a text input sub-unit and a voice input sub-unit. The text input sub-unit provides creators with a standard text input box. At the same time, the input box can perform real-time grammar checking, mark incorrect grammar with wavy lines, and provide modification suggestions to improve the accuracy of the input text. The voice input sub-unit integrates a high-precision speech recognition engine. After the creator clicks the voice input button, he directly speaks the creative intention through the microphone, and the speech is quickly and accurately converted into text.

[0044] The template calling unit is integrated with a template display sub-unit and a template screening and search sub-unit, wherein the template display sub-unit displays various creative templates in the form of pictures and texts, wherein the creative templates are classified according to the content type, such as video templates are subdivided into promotional videos, teaching videos, short dramas, etc.; graphic templates include posters, tweets, brochures, etc. Each template display area is equipped with a concise description of the template's use and sample pictures, allowing creators to quickly understand the template style and applicable scenarios. For example, the tourism promotion tweet template display area will present a tweet style that has been filled with sample content, highlighting replaceable parts such as attraction names, picture placeholders, etc.; the template screening and search sub-unit is used for creators to search for templates by keywords, template popularity, creation time, and applicable fields;

[0045] The auxiliary function unit is integrated with a history record sub-unit and a help and feedback sub-unit. The history record sub-unit automatically records each creative intention input record of the creator, including the text input content, the text converted by voice recognition, and the corresponding creation time. The creator can view past inputs in the history record list and click on the record to refill the content into the intention input unit for reusing similar creative needs. For example, if a creator has previously created a product promotion copy requirement, when he wants to create similar product promotion content later, he can directly retrieve it from the history record and modify some key information, saving input time; the help and feedback sub-unit sets a help button, which pops up a detailed platform usage guide after clicking it, covering the entire process from creative intention input to content publishing, and is equipped with graphic demonstrations and video tutorial links. At the same time, it provides a feedback portal. If the creator encounters any problems or has improvement suggestions during use, he can submit feedback in the form of text or screenshots. The platform operators receive feedback information in real time, process and reply to the creators in a timely manner, and continuously optimize the platform usage experience.

[0046] The intelligent content creation collaboration module uses semantic understanding and task allocation algorithms. When the creator enters their creative intent, it uses natural language processing technology to deeply analyze the keywords within the creative intent. The specific implementation logic is as follows:

[0047] Analysis of creative intent: When the creator inputs the creative intent through the user interaction module and converts the creative intent into text, the content creation intelligent collaboration module calls natural language processing technology and uses lexical analysis to split the input text into individual words or phrases, and clarify the part of speech of each word; then uses syntactic analysis and semantic analysis technology to understand the grammatical relationship and semantic association between words. Determine the subject-verb-object structure of the sentence and grasp the core creative needs. For example, in the sentence "Create a graphic tweet to promote tourist attractions, focusing on displaying natural scenery and special food", accurately identify the nouns "tourist attractions", "graphic tweet", "natural scenery", "special food", and verbs "create", "display", etc., and make it clear that the core is to create graphic tweets around tourist attractions, and focus on displaying natural scenery and special food. At the same time, use the pre-trained language model, combined with the context, to eliminate the ambiguity of words and accurately interpret the creative intent;

[0048] Creator information retrieval: Based on the keywords in the analyzed creative intent, the creator information database stored in the data management module is searched, where the database records in detail the labels of each creator's expertise. The labels are based on the information filled out when the creator joins the platform and the platform's analysis of their past works, so as to filter out a list of creators with corresponding expertise labels; for the selected creators, the number of works created, quality ratings, and average time to complete tasks are analyzed to evaluate the creator's creative ability and experience level. For example, for the creation of travel copywriting, the performance of the travel copywriting written by the creator in the past in terms of reading volume, number of likes, conversion rate, etc. is checked to evaluate their creative ability and experience level; then the real-time online status of the above creators is obtained, so that the online creators can receive and process tasks in a timely manner, thereby improving creative efficiency;

[0049] Task allocation decisions: Based on the matching degree of the creator's expertise, past creative data, and real-time online status, a task allocation algorithm is used to generate the optimal task allocation plan. For example, for more complex tasks, priority is given to creators with rich creative experience and high-quality past works. For urgent tasks, they are assigned to creators who are currently online and have the shortest expected completion time. Taking the creation of travel photo and text tweets as an example, the copywriting task is assigned to creator A, who has rich creative experience in the field of travel copywriting, has a high readership of recent works, and is currently online. The image shooting and processing tasks are assigned to creator B, who is proficient in photography and image processing, has previously processed travel photos that have received high reviews, and is online. After the task allocation plan is determined, a task notification is sent to the selected creator through the platform's message push system. The notification content details the task requirements, including the creative intent, the task deadline, and links to relevant reference materials. For example, the notification received by creator A will clearly state that the copywriting theme is to promote a tourist attraction, and that it must highlight natural scenery and special food, as well as word count requirements, style preferences, and other information. At the same time, links to some basic introduction documents about the tourist attraction are attached.

[0050] Creative progress monitoring: Set key progress tracking nodes for each task assignment. For example, for copywriting tasks, set nodes such as outline completion, first draft completion, and final draft completion; for image shooting and processing tasks, set nodes such as shooting plan formulation, material shooting completion, and image retouching completion. After completing each node task, the creator will mark it for confirmation on the platform;

[0051] Collaboration during the creative process: During the creative process, if creators have questions about task requirements or other creators' achievements or need to communicate and collaborate, they can communicate through the instant messaging function built into the content creation intelligent collaboration module. The instant messaging function supports text and voice messaging, as well as file transfer. For example, when creator B is taking pictures based on the copy, if he or she does not understand the description of the characteristics of a certain scenic spot in the copy, he or she can directly ask creator A via instant messaging. Creator A will respond promptly and provide a more detailed explanation, and can also transmit some auxiliary pictures or documents. The content creation intelligent collaboration module automatically records the communication content between creators, including message records and file transfer records. The communication content is stored in the data management module to facilitate subsequent review and experience summary. For example, after the project is completed, managers or creators can review the communication records, analyze the problems encountered during the creative process and the solutions, provide reference for similar projects in the future, and continuously optimize task allocation and collaboration processes.

[0052] Integration and delivery of task results: After all creators have completed their respective tasks, the intelligent content creation collaboration module automatically collects the task results, such as documents, images, and video clips. The results are organized according to pre-set formats and specifications, such as standardizing the formatting of documents and adjusting image resolutions. The organized multimodal task results are then delivered to the multimodal deep fusion display module for subsequent cross-modal feature extraction and fusion processing, and finally presented to users.

[0053] The AIGC auxiliary creation module integrates text generation models, image generation models, and audio generation models. It provides assistance according to the creation needs of users during the creation process, and specifically provides targeted assistance according to the creation needs of creators during the creation process. For example, when creators write copy, they can call the text generation model to obtain relevant sentences and paragraph suggestions; when designing images, they can use the image generation model to generate creative sketches or perform style conversion on existing materials; in the audio creation process, they can use the audio generation model to generate environmental sound effects, background music samples, etc. This module works closely with the content creation intelligent collaboration module to provide creators with all-round creative support;

[0054] The data management module is responsible for storing and managing creator information, creative materials, creative history, and user feedback data. Specifically, it uses a distributed database architecture to categorize creator information, including personal information, areas of expertise, and creative achievements, for task allocation in the content creation intelligent collaboration module. At the same time, creative materials are managed in a tagged manner.

[0055] The multimodal deep fusion display module collects content from different modalities generated by the content creation intelligent collaboration module and the AIGC auxiliary creation module, including text, images, video clips, and audio. It uses cross-modal feature extraction and fusion algorithms to explore the inherent connections between the modal content. The specific working steps of this module are as follows:

[0056] Step 1. Multimodal content collection: The multimodal deep fusion display module establishes a data transmission connection with the content creation intelligent collaboration module and the AIGC auxiliary creation module. When the creator completes the content creation, the creative results are automatically transmitted to this module. Taking the tourism promotion project as an example, after the content creation intelligent collaboration module assigns the task, the tourist attraction introduction copy completed by creator A, the scenery and food pictures taken and processed by creator B, and the related audio clips (such as the characteristic sound effects of the scenic spot) and video clips (such as short scenic scenery videos) that may be generated by the AIGC auxiliary creation module, will be aggregated to the multimodal deep fusion display module through the established data channel; at the same time, the multimodal content of the creative results transmitted in different formats will be format converted and unified, such as unifying various types of pictures into a common JPEG or PNG format, converting videos into common formats such as MP4, and standardizing audio into MP3 format. At the same time, preliminary classification and sorting are carried out according to the type and theme of the content, such as classifying all pictures, audio and video about the natural scenery of the scenic spot into one category, and the content related to characteristic food into another category, to facilitate subsequent in-depth processing;

[0057] Step 2: Cross-modal feature extraction:

[0058] Text feature extraction: Natural language processing technology is used to extract key descriptive features from the text content in step 1. That is, through lexical analysis, syntactic analysis, and semantic understanding, a text feature vector is constructed. For example, for the text describing a waterfall, "The waterfall is majestic, the water flows like a galloping horse, and the white water curtain cascades down," lexical analysis, syntactic analysis, and semantic understanding are used to extract key descriptive words and phrases such as "waterfall," "majestic," "galloping horses," "white water curtain," and "cascading down." These feature vectors are used to construct text feature vectors. These feature vectors represent the focus of the text's description of a specific object.

[0059] Image feature extraction: Through edge detection, color analysis, and texture recognition algorithms, the visual features of the image in step 1 are extracted and converted into corresponding image feature vectors. For example, for a waterfall image, edge detection, color analysis, texture recognition and other algorithms are used to extract visual features such as the waterfall's shape, water flow pattern, color distribution, and surrounding environment. For example, the dynamic texture of the waterfall flow, the proportion and distribution area of ​​the white tones of the waterfall body in the image, etc. are identified and converted into corresponding image feature vectors.

[0060] Video feature extraction: For the video content in step 1, a spatiotemporal feature extraction algorithm is used to analyze the changes in different frames in the video in the temporal dimension, such as the continuity and rhythm of the dancers' movements in the scenic spot's characteristic dance scene. In the spatial dimension, the visual features of each frame are extracted, such as the stage layout of the dance scene, the color and pattern of the dancers' costumes, etc., and the temporal and spatial dimension features are combined to generate a video feature vector. Audio feature extraction: For the audio data in step 1, audio signal processing technology is used to extract frequency, amplitude, and timbre features and convert them into audio feature vectors.

[0061] Step 3: Cross-modal feature fusion: The feature vectors extracted from different modalities in step 2 are fused through the fusion algorithm of the attention mechanism. During the fusion process, different weights are assigned according to the degree of correlation between the features.

[0062] Step 4. Construction of display logic: Based on the fused feature vectors, formulate interactive display rules for multimodal content according to customer needs. For example, when a user is browsing a travel picture and text tweet, and clicks on the description of a waterfall in the text, based on the previously established feature association, the system can accurately locate the area corresponding to the waterfall in the picture and automatically highlight it. Conversely, when the user clicks on the waterfall in the picture, the relevant description in the text will also be highlighted. Similar rules are also formulated for video and audio modalities. For example, when a video plays a characteristic dance scene at a scenic spot, the corresponding folk music audio element playback is automatically enhanced through feature association to enhance the user's immersive experience.

[0063] Step 5. Content display and user interaction: When users access platform content, the multimodal deep fusion display module presents multimodal content in real time according to the display logic established in step 4. For example, when users browse tourism promotion pages, the system dynamically loads and displays the integrated graphics, text, audio and video content. According to user operations (such as clicks, scrolling pages, etc.), it immediately responds and executes the corresponding interaction rules to achieve flexible and intelligent display of multimodal content. In addition, during the content display process, the user feedback collection function is simultaneously enabled. If the user is not satisfied with the display effect, he or she can express his or her opinions through the ratings, text comments, voice feedback, etc. provided by the user feedback module. These feedback data will be transmitted back to the multimodal deep fusion display module and the data management module in real time to provide a basis for the optimization of subsequent modules.

[0064] The specific working logic of cross-modal feature fusion in step 3 is as follows:

[0065] Step 31: Initialize the deep learning model: Select the Transformer architecture model for initialization and set the model's hyperparameters, including the number of layers, hidden layer dimensions, and number of attention heads. Adjust the model parameters based on the characteristics and scale of different modal data to ensure that the model can effectively learn the relationship between different modal features.

[0066] Step 32: Feature vector input and preprocessing: Arrange the text, image, video, and audio feature vectors extracted in step 2 to ensure that their dimensions and formats meet the input requirements of the deep learning model. Standardize the feature vectors so that feature vectors of different modalities have similar scales and distributions to obtain preprocessed feature vectors. For example, normalize text feature vectors and standardize and scale image feature vectors. If the dimensions of the feature vectors are inconsistent, use methods such as padding or dimensionality reduction to make all feature vectors have the same dimensions so that the model can perform unified calculations and analysis.

[0067] Step 33, cross-modal feature association analysis: The pre-processed feature vectors are input into the deep learning model. The model uses a self-attention mechanism and a multi-layer neural network to parallelize and interactively learn the feature vectors of different modalities. In the self-attention mechanism, the model calculates the attention score between each feature vector element and all other elements, thereby capturing the potential associations between features of different modalities. For example, the connection between the features of an object described in the text and the visual features of the corresponding object in the image. Through the nonlinear transformation of the multi-layer neural network, the complex relationship between feature vectors is further explored, the feature representation ability is enhanced, and the model can more accurately identify and understand the intrinsic connection between features of different modalities;

[0068] Step 34: Start the attention mechanism fusion algorithm: Based on the cross-modal feature correlation results obtained by the deep learning model in step 33, the results are input into the attention mechanism fusion algorithm to calculate the attention weight of each modal feature vector in the fusion process. The attention weight reflects the degree of correlation between different modal feature vectors and other modal feature vectors. For example, if the image features and text features are highly correlated when describing a specific scene, then the image feature vector will receive a higher weight in the fusion process.

[0069] Step 35, feature vector fusion calculation: Based on the attention weights calculated in step 34, perform weighted fusion on the feature vectors of different modalities. Multiply the feature vector of each modality by its corresponding attention weight, and then sum or concatenate the weighted feature vectors to obtain a fused feature vector.

[0070] Step 36: Post-processing of fusion results: The fused feature vector is post-processed by regularization method to further optimize the feature representation, prevent the model from overfitting, and improve the generalization ability of the fused feature vector.

[0071] The user feedback module sets up a feedback entrance in the multimodal content display module, and users express their opinions on the content through ratings, text comments, and voice feedback; the user feedback module collects feedback data in real time and feeds it back to the creators and the data management module. The creators optimize the content based on the feedback, providing a basis for subsequent content creation direction adjustments and algorithm optimization.

[0072] The platform of the present invention integrates multiple core functional modules. The user interaction module supports text and voice input of creative intent, provides rich creative templates, and has functions such as grammar checking, intention record reuse, and operation guide query, which is convenient for creators to use. The content creation intelligent collaboration module uses natural language processing technology to analyze creative intent, intelligently assign tasks based on the creator's expertise, creative ability and online status, monitor progress in real time and promote collaborative communication among creators. The AIGC auxiliary creation module integrates multiple generation models to provide creators with assistance in text, images, audio and other aspects during the creative process. The data management module adopts a distributed database architecture to classify and manage creator information, materials, historical records and user feedback data to provide support for task allocation and creation. The multimodal deep fusion display module collects content in different modalities, and through feature extraction, fusion and display logic construction, it realizes the deep fusion display of multimodal content and enhances user experience. The user feedback module collects user ratings, comments and voice feedback to help creators optimize content and provide a basis for platform algorithm optimization.

[0073] Cross-modal feature fusion technology is a major innovation of this invention. It uses the Transformer architecture model and attention mechanism to effectively mine and fuse the features of content in different modalities, and enhance the relevance and immersion of content presentation. The intelligent collaborative creation model breaks the limitations of traditional creation. Through semantic understanding and task allocation algorithms, it enables efficient collaboration between creators, improving creation efficiency and quality. The platform is also highly integrated and interactive. The modules work closely together to provide one-stop services for creators and users. At the same time, it pays attention to user feedback, forming a good ecological cycle and promoting the continuous optimization and development of the platform.

[0074] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. The AIGC-based content creation and multimodal display platform includes a user interaction module, an intelligent content creation collaboration module, an AIGC-assisted creation module, a data management module, a multimodal deep fusion display module, and a user feedback module. Its features include: The user interaction module provides an operation interface, and the creator can clearly express his or her creative intention through text input or voice input, and also sets several creative templates for users to apply; The content creation intelligent collaboration module uses semantic understanding and task allocation algorithms. When the creator enters the creative intention, it uses natural language processing technology to deeply analyze the keywords in the creative intention. The AIGC assisted creation module integrates a text generation model, an image generation model, and an audio generation model to provide assistance to users during their creation process based on their creative needs. The data management module is responsible for storing and managing creator information, creative materials, creative history, and user feedback data. Specifically, it uses a distributed database architecture to classify and manage creator information, including personal information, areas of expertise, and creative achievements, for task allocation in the content creation intelligent collaboration module. At the same time, creative materials are managed in a tagged manner. The multimodal deep fusion display module collects different modal contents generated by the content creation intelligent collaboration module and the AIGC auxiliary creation module, including text, pictures, video clips, and audio, and mines the intrinsic connections between the contents of each modality through cross-modal feature extraction and fusion algorithms; The user feedback module sets a feedback entry in the multimodal content display module, and users express their opinions on the content through ratings, text comments, and voice feedback; The user feedback module collects feedback data in real time and feeds it back to the creators and the data management module. The creators optimize the content based on the feedback, providing a basis for subsequent content creation direction adjustments and algorithm optimization.

2. The AIGC-based content creation and multimodal display platform according to claim 1, characterized in that: The user interaction module specifically includes an intention input unit, a template calling unit, and an auxiliary function unit, and its specific contents are as follows: The intention input unit integrates a text input subunit and a voice input subunit. The text input subunit provides the creator with a standard text input box. At the same time, the input box can perform grammar checking in real time, mark incorrect grammar with wavy lines, and provide modification suggestions to improve the accuracy of the input text. The voice input subunit integrates a high-precision voice recognition engine to convert the creator's voice into text. The template calling unit integrates a template display subunit and a template screening and search subunit, wherein the template display subunit displays various creative templates in the form of pictures and texts, wherein the creative templates are classified according to the content type; the template screening and search subunit is used for creators to search for templates by keywords, template popularity, creation time, and applicable fields; The auxiliary function unit is integrated with a history record sub-unit and a help and feedback sub-unit, wherein the history record sub-unit automatically records each creative intention input record of the creator, including the text input content, the text converted by voice recognition, and the corresponding creation time. The creator can view past inputs in the history record list and click on the record to refill the content into the intention input unit for reusing similar creative needs; The help and feedback sub-unit is provided with a help button, which pops up a detailed platform usage guide when clicked, covering the entire process from inputting creative intentions to publishing content, and is accompanied by graphic demonstrations and video tutorial links. At the same time, a feedback portal is provided. If creators encounter any problems or have improvement suggestions during use, they can submit feedback in the form of text or screenshots. Platform operators will receive feedback information in real time, process and reply to creators in a timely manner, and continuously optimize the platform usage experience.

3. The AIGC-based content creation and multimodal display platform according to claim 1, characterized in that: The specific implementation logic of the content creation intelligent collaboration module is as follows: Creative Intention Analysis: When the creator inputs the creative intention through the user interaction module and converts it into text, the content creation intelligent collaboration module uses natural language processing technology and lexical analysis to split the input text into individual words or phrases, clarifying the part of speech of each word. It then uses syntactic analysis and semantic analysis technology to understand the grammatical relationship and semantic association between words, determine the subject-verb-object structure of the sentence, and grasp the core creative needs. Creator information retrieval: Based on the keywords in the analyzed creative intent, a search is performed in the creator information database stored in the data management module. The database records in detail the labels of each creator's expertise. The labels are based on the information filled out when the creator registers and the platform's analysis of their past works. This is used to screen out a list of creators with corresponding expertise labels. For the screened creators, the number of works created, quality ratings, and average time to complete tasks are analyzed to assess the creator's creative ability and experience level. The real-time online status of the above creators is then obtained so that online creators can receive and process tasks in a timely manner, thereby improving creative efficiency. Task allocation decision-making: Based on the creator's field of expertise, past creative data, and real-time online status, the task allocation algorithm is used to generate the optimal task allocation plan. After the task allocation plan is determined, the platform's message push system sends a task notification to the selected creator. The notification content details the task requirements, including creative intent, task deadline, and links to relevant reference materials; Creation progress monitoring: key progress tracking nodes are set for each task allocation plan. Creators mark and confirm each node task on the platform after completing it. Collaboration during the creative process: During the creative process, if creators have questions about task requirements or other creators' work, or need to communicate and collaborate, they can communicate through the built-in instant messaging function of the Content Creation Intelligent Collaboration Module. The instant messaging function supports text and voice messaging, as well as file transfers. The Content Creation Intelligent Collaboration Module automatically records the communication content between creators, including message logs and file transfer records, and stores the communication content in the data management module for subsequent review and experience summary. Integration and delivery of task results: When all creators have completed their respective tasks, the intelligent content creation collaboration module automatically collects the task results and organizes them according to pre-set formats and specifications; the organized multimodal task results are then delivered to the multimodal deep fusion display module for subsequent cross-modal feature extraction and fusion processing, and finally presented to users.

4. The AIGC-based content creation and multimodal display platform according to claim 1, characterized in that: The specific working steps of the multimodal deep fusion display module are as follows: Step 1: Multimodal content collection: The multimodal deep fusion display module establishes a data transmission connection with the content creation intelligent collaboration module and the AIGC auxiliary creation module. When the creator completes content creation, the creative results are automatically transmitted to this module. At the same time, the multimodal content of the creative results transmitted in different formats is format converted and uniformly processed; Step 2: Cross-modal feature extraction: Text feature extraction: Using natural language processing technology, key descriptive features are extracted from the text content in step 1. That is, a text feature vector is constructed through lexical analysis, syntactic analysis, and semantic understanding. Image feature extraction: Using edge detection, color analysis, and texture recognition algorithms, the visual features of the image in step 1 are extracted and converted into corresponding image feature vectors. Video feature extraction: Based on the video content in step 1, a spatiotemporal feature extraction algorithm is used to analyze the changes between different frames in the video in the temporal dimension. In the spatial dimension, the visual features of each frame of the image are extracted, and the temporal and spatial dimension features are integrated to generate a video feature vector. Audio feature extraction: For the audio data in step 1, use audio signal processing technology to extract frequency, amplitude, and timbre features and convert them into audio feature vectors; Step 3: Cross-modal feature fusion: The feature vectors extracted from different modalities in step 2 are fused through the fusion algorithm of the attention mechanism. During the fusion process, different weights are assigned according to the degree of correlation between the features. Step 4: Display logic construction: Based on the fused feature vectors, formulate interactive rules for displaying multimodal content according to customer needs; Step 5: Content display and user interaction: When users access platform content, the multimodal deep fusion display module presents multimodal content in real time according to the display logic established in step 4.

5. The AIGC-based content creation and multimodal display platform according to claim 4, characterized in that: In step 3, the Transformer architecture model is used to fuse the feature vectors of different modalities in the cross-modal feature fusion. The specific working logic is as follows: Step 31: Initialize the deep learning model: Select the Transformer architecture model for initialization and set the model's hyperparameters, including the number of layers, hidden layer dimensions, and number of attention heads. Adjust the model parameters based on the characteristics and scale of different modal data to ensure that the model can effectively learn the relationship between different modal features. Step 32: Feature vector input and preprocessing: The text, image, video, and audio feature vectors extracted in step 2 are organized to ensure that their dimensions and formats meet the input requirements of the deep learning model. The feature vectors are normalized so that feature vectors of different modalities have similar scales and distributions, thus obtaining preprocessed feature vectors. Step 33: Cross-modal feature association analysis: The preprocessed feature vectors are input into the deep learning model. The model uses a self-attention mechanism and a multi-layer neural network to perform parallel processing and interactive learning on the feature vectors of different modalities. In the self-attention mechanism, the model calculates the attention score between each feature vector element and all other elements, determines the association data between features of different modalities, and calculates the association data between feature vectors through nonlinear transformation of the multi-layer neural network, thereby enhancing the feature vector representation ability and enabling the model to identify the connection between features of different modalities. Step 34: Start the attention mechanism fusion algorithm: Based on the cross-modal feature correlation results obtained by the deep learning model in step 33, the results are input into the attention mechanism fusion algorithm to calculate the attention weight of each modal feature vector in the fusion process. The attention weight reflects the degree of correlation between different modal feature vectors and other modal feature vectors. Step 35, feature vector fusion calculation: Based on the attention weights calculated in step 34, perform weighted fusion on the feature vectors of different modalities. Multiply the feature vector of each modality by its corresponding attention weight, and then sum or concatenate the weighted feature vectors to obtain a fused feature vector. Step 36: Post-processing of fusion results: The fused feature vector is post-processed using a regularization method to further optimize the feature representation and prevent overfitting of the model.

Images