AI multi-modal copywriting assistance system for cultural media

Abstract

The application discloses a culture media-oriented AI multi-modal copywriting creation auxiliary system and relates to the technical field of digital content generation.The system comprises a multi-modal feature alignment and fusion module, which is used for extracting and aligning multi-modal material features to obtain global semantic representation; a media knowledge enhancement and hot spot retrieval module, which is used for retrieving and acquiring field knowledge and hot spot labels; a multi-dimensional copywriting generation module, which is used for generating copywriting containing aligned labels in combination with knowledge and constraints; and a cross-modal linkage editing module, which is used for calculating semantic difference of copywriting editing and reversely mapping the semantic difference to materials to provide editing suggestions.The application realizes multi-modal deep alignment and fusion, field knowledge and hot spot enhancement, cross-modal linkage editing of text and materials, and online self-adaptive fine-tuning of the model, and greatly improves the creation efficiency and professional degree of media copywriting.

Description

Technical Field

[0001] This invention relates to the field of digital content generation technology, and in particular to an AI-powered multimodal copywriting assistance system for cultural media. Background Technology

[0002] With the rapid development of mobile internet and social media, the cultural and media industry has placed higher demands on the efficiency and quality of content production. In daily media creation, creators often need to write copy that adapts to the styles of different platforms based on original materials in multiple modalities such as images, videos, audio, and text. Traditional content creation mainly relies on manual work, which is time-consuming, labor-intensive, and highly dependent on personal experience.

[0003] In recent years, text generation technology based on large language models has made significant progress, but the following technical challenges still exist in its practical application in the cultural and media fields: 1) Insufficient depth of multimodal fusion: Existing copy generation systems are mostly based on single text or simple image-text pairing, which makes it difficult to fully explore and integrate the deep semantic relationships between dynamic visuals, audio rhythms and text metadata in videos, resulting in a disconnect between the generated copy and the original multimodal materials.

[0004] 2) Lack of domain knowledge and integration with trending topics: The general model lacks professional knowledge in the cultural and media field (such as platform style guidelines, prohibited word database, and popular content structure), and it is difficult to integrate with trending internet topics in real time, resulting in the generated copy often being "off" or lacking appeal.

[0005] 3) Difficulty in text-element linkage editing: When creators modify text generated by artificial intelligence (AI), existing systems cannot reverse-map the modification intentions back to the original material. For example, if a user deletes a sentence in the text, the system cannot intelligently suggest cutting the corresponding video segment, resulting in "text modified but image / video not modified," increasing the cost of post-editing.

[0006] 4) The model lacks adaptive evolution capability: After the existing system generates copy, it cannot perform closed-loop feedback learning based on the user's final adoption rate and modification range, which makes it impossible for the model to continuously optimize for the style and preferences of specific users. Summary of the Invention

[0007] This invention provides an AI-powered multimodal copywriting assistance system for cultural media, aiming to solve the problems of insufficient multimodal fusion depth, lack of domain knowledge and hot topic integration, difficulty in text element linkage editing, and lack of adaptive evolution capabilities in existing technologies.

[0008] In a first aspect, embodiments of the present invention provide an AI-powered multimodal copywriting assistance system for cultural media, the system comprising: The module includes a multimodal feature alignment and fusion module, a media knowledge enhancement and hot topic retrieval module, a multi-dimensional copywriting generation module, and a cross-modal collaborative editing module. The multimodal feature alignment and fusion module is used to acquire a set of multimodal materials input by the user, extract cross-modal features from the set of multimodal materials, and perform feature alignment and fusion in the shared latent space to obtain a global multimodal semantic representation. The media knowledge enhancement and hotspot retrieval module is used to retrieve related domain knowledge enhancement data and real-time hotspot tags from a pre-set cultural media knowledge base and a real-time hotspot database based on the global multimodal semantic representation. The multi-dimensional copywriting generation module is used to input the global multimodal semantic representation, domain knowledge augmentation data, real-time hot topic tags and user-defined creative constraints into a pre-trained multimodal large language model to generate multi-dimensional candidate copywriting that is adapted to the style of a specific media platform and includes multimodal alignment tags. The cross-modal linkage editing module is used to respond to the user's editing operation on the candidate text, obtain the edited candidate text, calculate the semantic difference between the candidate text and the edited candidate text, back-map it to the related fragments in the multimodal material set, and provide editing suggestions for the material.

[0009] The technical solutions provided in this application have at least the following beneficial effects: By independently encoding visual, auditory, and textual information and aligning and fusing them in a shared latent space, the system can comprehensively capture the global semantics of multimodal materials, avoiding information loss and making the generated copy more aligned with the original meaning of the material, thus achieving deep cross-modal semantic fusion. It introduces a cultural media-specific knowledge base and a real-time trending search database, using hybrid retrieval and rearrangement algorithms to ensure that the copy not only conforms to the platform's tone and compliance requirements but also leverages trending topics, achieving a dual-drive of professional knowledge and real-time trending topics. It breaks down the barriers of traditional "one-way generation," achieving a linked editing experience of "editing text is the same as editing images / videos" through multimodal alignment tags and semantic difference reverse mapping, greatly improving creation efficiency. Based on an online fine-tuning mechanism using reinforcement learning and knowledge distillation, it transforms user adoption rates and modification amounts into feedback rewards, making the model smarter and more user-aware with use.

[0010] In one optional implementation, the multimodal feature alignment and fusion module includes an image visual coding unit, an audio auditory coding unit, a text semantic coding unit, and a cross-modal alignment and fusion unit; The image visual encoding unit is used to acquire a multimodal material set input by the user, and use a visual transformer network to encode the image and video keyframes in the multimodal material set, capture the spatial layout and object features of the screen, and obtain visual semantic features. The audio auditory coding unit is used to encode the audio track data in the multimodal material set using an audio spectrum analysis model, extract acoustic spectrum features and speech text content, and obtain auditory rhythm features. The text semantic encoding unit is used to use a text embedding model to vectorize and encode text information containing titles, subtitles, and metadata in a multimodal material set to obtain text metadata features. The cross-modal alignment and fusion unit is used to map the visual semantic features, auditory rhythm features, and text metadata features to a shared latent space for alignment based on a contrastive learning mechanism. It also employs a multi-head cross-modal attention mechanism, using text metadata features as query vectors and visual semantic features and auditory rhythm features as key and value vectors for fusion, to obtain a global multimodal semantic representation.

[0011] In one optional implementation, the media knowledge enhancement and hotspot retrieval module includes a feature vectorization retrieval unit, a domain knowledge matching unit, and a real-time hotspot aggregation unit. The feature vectorization retrieval unit is used to use the global multimodal semantic representation as a query vector to perform nearest neighbor retrieval in a pre-set cultural and media knowledge base to obtain candidate knowledge entries. The domain knowledge matching unit is used to extract data from the candidate knowledge items that correspond to the media platform style guidelines, prohibited word library and popular copywriting structure template, as domain knowledge enhancement data. The real-time hotspot aggregation unit is used to match the most relevant tag in the preset real-time hotspot library based on the core semantic tags in the global multimodal semantic representation, and use it as the real-time hotspot tag.

[0012] In one optional implementation, the multi-dimensional copywriting generation module includes a constraint splicing and prompt construction unit, a large model reasoning generation unit, and a multi-dimensional alignment tag embedding unit. The constraint splicing and prompt construction unit is used to structurally splice the global multimodal semantic representation, domain knowledge enhancement data, real-time hot topic tags, and user-defined creative constraints to construct input prompt words for the multimodal large language model; the creative constraints include target platform constraints, word count limits, and audience constraints. The large model reasoning and generation unit is used to input the input prompt words into a pre-trained multimodal large language model and reason to generate initial text content including title, body text and layout marks. The multi-dimensional alignment tag embedding unit is used to automatically embed timestamps and modal source markers corresponding to the timeline or spatial region of the original multimodal material into the sentences of the generated initial copy content, so as to generate multi-dimensional candidate copy containing multimodal alignment tags.

[0013] In one optional implementation, the cross-modal linkage editing module includes an editing difference calculation unit, a reverse mapping positioning unit, and a linkage modification suggestion unit; The editing difference calculation unit is used to capture the user's modification operations on the candidate text in the editing interface in real time, and use the text embedding model to calculate the difference in text semantic vectors before and after modification. The reverse mapping localization unit is used to search for the nearest non-text feature clusters in the shared latent space based on the difference of the text semantic vector and combined with the multimodal alignment tags embedded in the candidate text, to locate the corresponding time segment or spatial region in the original multimodal material. The linked modification suggestion unit is used to generate editing suggestions for the original material based on the semantic direction of the user's modification and the located original multimodal material fragment, and to highlight the suggestions in the editing interface; the editing suggestions include synchronous cropping suggestions and replacement suggestions.

[0014] In one optional implementation, the linked modification suggestion unit is specifically used for: When it detects that a user has deleted any description in the candidate text, the corresponding segment located in the original multimodal material through reverse mapping is automatically highlighted, and an interactive prompt suggesting that the segment be cropped is generated, i.e., a cropping suggestion is generated simultaneously. When it is detected that a user modifies the first description in the candidate text to a second description with opposite semantics, the difference between the modified semantic vectors of the first and second descriptions is calculated. Based on the difference between the modified semantic vectors, a new material that matches the semantics of the second description is retrieved from the preset material library, and an interactive prompt is generated to replace the corresponding segment located in the original multimodal material with the new material, i.e., a replacement suggestion.

[0015] In one alternative implementation, the system further includes an online adaptive fine-tuning module; The online adaptive fine-tuning module is used to collect logs of the final adoption rate and modification range of the candidate copy generated by the system after the user completes the copy editing and exports. The final adoption rate and modification magnitude logs are converted into feedback reward signals, and low-rank adaptive fine-tuning technology is used to incrementally update the parameters of the multimodal large language model in order to optimize the system's copywriting generation strategy for subsequent multimodal materials.

[0016] In one optional implementation, the online adaptive fine-tuning module is specifically used for: Construct an evaluation function for the quality of generated copy, wherein the variables of the evaluation function include candidate copy text fluency score, platform style matching score, user editing distance, and final adoption rate; The evaluation function is weighted and summed to obtain the feedback reward signal; Based on the feedback reward signal, a cross-entropy loss function for model fine-tuning is constructed. The gradient of the low-rank adaptation matrix in the multimodal large language model is calculated through the backpropagation algorithm, and only the parameters of the low-rank adaptation matrix are updated, keeping the basic weights of the multimodal large language model frozen.

[0017] In one optional implementation, the system further includes a multimodal consistency verification module; The multimodal consistency verification module is used to perform multimodal semantic consistency verification on the modified text and the replaced material after the user performs cross-modal collaborative editing and confirms the modification suggestions; Calculate the cosine similarity between the text semantic vector of the modified copy and the visual and / or auditory feature vector of the replaced material in the shared latent space; If the cosine similarity is lower than the preset consistency threshold, a warning message indicating audio-visual-text mismatch will be issued to the user, and the linkage modification suggestion unit will be triggered again to generate a new replacement suggestion.

[0018] In one optional implementation, the multimodal consistency verification module is specifically used for: Extract the sentence-level text vector of each sentence in the modified candidate copy, and the modal feature vector of the corresponding segment in the replaced material; Calculate the local cosine similarity between the sentence-level text vector and the corresponding modal feature vector for each statement in turn; For target statements with a local cosine similarity lower than a preset local threshold, it is determined that the target statement has a semantic conflict with the corresponding material segment; Generate semantic conflict identifiers for the target statement, visualize the semantic conflict identifiers on the editing interface, and re-retrieve matching materials based on the semantics of the target statement to overwrite the replaced materials.

[0019] A second aspect of this invention provides an electronic device, which includes: At least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by at least one processor, such that the at least one processor can execute the system proposed in the first aspect of the present invention.

[0020] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the system as described in the first aspect of the present invention. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the electronic device structure of the hardware operating environment involved in the embodiments of the present invention; Figure 2 This is a structural block diagram of an AI multimodal copywriting assistance system for cultural media provided by an embodiment of the present invention. Detailed Implementation

[0022] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0023] The present invention will be further described below with reference to the accompanying drawings.

[0024] Reference Figure 1 , Figure 1 This is a schematic diagram of the electronic device structure of the hardware operating environment involved in the embodiments of the present invention.

[0025] like Figure 1 As shown, the electronic device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0026] Those skilled in the art will understand that Figure 1The structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0027] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a data storage module, a network communication module, a user interface module, and an electronic program for an AI multimodal copywriting assistance system for cultural media.

[0028] exist Figure 1 In the illustrated electronic device, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the electronic device of the present invention can be set in the electronic device. The electronic device calls the electronic program of the AI ​​multimodal copywriting assistance system for cultural media stored in the memory 1005 through the processor 1001, and executes the AI ​​multimodal copywriting assistance system for cultural media provided in the embodiment of the present invention.

[0029] Reference Figure 2 The present invention provides an AI multimodal copywriting assistance system 200 for cultural media, the system comprising: Multimodal feature alignment and fusion module 201, media knowledge enhancement and hot topic retrieval module 202, multi-dimensional copywriting generation module 203, and cross-modal collaborative editing module 204; The multimodal feature alignment and fusion module is used to acquire a set of multimodal materials input by the user, extract cross-modal features from the set of multimodal materials, and perform feature alignment and fusion in the shared latent space to obtain a global multimodal semantic representation. The media knowledge enhancement and hotspot retrieval module is used to retrieve related domain knowledge enhancement data and real-time hotspot tags from a pre-set cultural media knowledge base and a real-time hotspot database based on the global multimodal semantic representation. The multi-dimensional copywriting generation module is used to input the global multimodal semantic representation, domain knowledge augmentation data, real-time hot topic tags and user-defined creative constraints into a pre-trained multimodal large language model to generate multi-dimensional candidate copywriting that is adapted to the style of a specific media platform and includes multimodal alignment tags. The cross-modal linkage editing module is used to respond to the user's editing operation on the candidate text, obtain the edited candidate text, calculate the semantic difference between the candidate text and the edited candidate text, back-map it to the related fragments in the multimodal material set, and provide editing suggestions for the material.

[0030] The technical solutions provided in this application have at least the following beneficial effects: By independently encoding visual, auditory, and textual information and aligning and fusing them in a shared latent space, the system can comprehensively capture the global semantics of multimodal materials, avoiding information loss and making the generated copy more aligned with the original meaning of the material, thus achieving deep cross-modal semantic fusion. It introduces a cultural media-specific knowledge base and a real-time trending search database, using hybrid retrieval and rearrangement algorithms to ensure that the copy not only conforms to the platform's tone and compliance requirements but also leverages trending topics, achieving a dual-drive of professional knowledge and real-time trending topics. It breaks down the barriers of traditional "one-way generation," achieving a linked editing experience of "editing text is the same as editing images / videos" through multimodal alignment tags and semantic difference reverse mapping, greatly improving creation efficiency. Based on an online fine-tuning mechanism using reinforcement learning and knowledge distillation, it transforms user adoption rates and modification amounts into feedback rewards, making the model smarter and more user-aware with use.

[0031] In one optional implementation, the multimodal feature alignment and fusion module includes an image visual coding unit, an audio auditory coding unit, a text semantic coding unit, and a cross-modal alignment and fusion unit; The image visual encoding unit is used to acquire a multimodal material set input by the user, and use a visual transformer network to encode the image and video keyframes in the multimodal material set, capture the spatial layout and object features of the screen, and obtain visual semantic features. In this embodiment, the visual transformer network is constructed based on the 3D spatiotemporal vision algorithm of the Swing Transformer, and the formula is: In the formula, Visual semantic features; For the feature encoding function of the visual transformer network; Keyframes of images or videos within a multimodal material set; The audio auditory coding unit is used to encode the audio track data in the multimodal material set using an audio spectrum analysis model, extract acoustic spectrum features and speech text content, and obtain auditory rhythm features. In this embodiment, the audio spectrum analysis model is constructed based on Whisper's audio spectrum analysis algorithm, and the formula is as follows: In the formula, It is an auditory rhythmic feature; For the feature encoding function of the audio spectrum analysis model; Audio track data from a multimodal material set; The text semantic encoding unit is used to use a text embedding model to vectorize and encode text information containing titles, subtitles, and metadata in a multimodal material set to obtain text metadata features. In this embodiment, the text embedding model is constructed based on the text embedding algorithm of the Robustly Optimized BERT Pretraining Approach (RoBERTa), and the formula is as follows: In the formula, It is an auditory rhythmic feature; For the feature encoding function of the text embedding model; This is a textual information set containing titles, subtitles, and metadata for multimodal content. The cross-modal alignment and fusion unit is used to map the visual semantic features, auditory rhythm features and text metadata features to a shared latent space for alignment based on a contrastive learning mechanism, and to use a multi-head cross-modal attention mechanism to fuse the text metadata features as query vectors and the visual semantic features and auditory rhythm features as key vectors and value vectors to obtain a global multimodal semantic representation. In this embodiment, the formula for the contrast loss function of the alignment process is: In the formula, To compare the loss values; It is a normalized latent space representation of visual semantic features, auditory rhythm features, and text metadata features; For sample index; Batch size; This refers to temperature hyperparameters. This is the function for calculating cosine similarity. It is an exponential function; In this embodiment, the specific formula for the fusion process is as follows: In the formula, For global multimodal semantic representation; These include visual semantic features, auditory rhythm features, and textual metadata features; Symbols for splicing operations; It is the transpose symbol; The weight parameters are the query matrix composed of query vectors, the key matrix composed of key vectors, and the value matrix composed of value vectors. This is the scaling factor; This is the normalization function.

[0032] In one optional implementation, the media knowledge enhancement and hotspot retrieval module includes a feature vectorization retrieval unit, a domain knowledge matching unit, and a real-time hotspot aggregation unit. The feature vectorization retrieval unit is used to use the global multimodal semantic representation as a query vector to perform nearest neighbor retrieval in a pre-set cultural and media knowledge base to obtain candidate knowledge entries. In this embodiment, an attention-pooling-based dimensionality reduction algorithm is used to compress the global multimodal semantic representation to obtain the global query vector, as shown in the formula: In the formula, This is the global query vector; This is the attention weight matrix; Based on the global query vector, an approximate nearest neighbor search algorithm based on Facebook AISimilarity Search (FAISS) is used to retrieve candidate knowledge entries from a pre-built vector database (cultural and media knowledge base). The formula is as follows: In the formula, This is the index set of the retrieved candidate knowledge items; For vector database feature matrix; To sort, take the first K Operation functions; K The preset total number of candidate knowledge items; The domain knowledge matching unit is used to extract data from the candidate knowledge items that correspond to the media platform style guidelines, prohibited word library and popular copywriting structure template, as domain knowledge enhancement data. In this embodiment, a reordering algorithm based on a hybrid of Best Match 25 (BM25) and semantic similarity is used to extract the corresponding text content from the candidate knowledge item index set and calculate the reordering score. The formula is as follows: In the formula, The reordering score of the current candidate knowledge entry in the candidate knowledge entry index set; This is a mixed weighting factor, and its value range is [0,1]. The BM25 algorithm score for the current candidate knowledge item; This is the vector of the current candidate knowledge entries; Based on the re-ranking scores, domain knowledge that matches the media platform's style guidelines, prohibited word database, and viral copywriting structure templates is selected as domain knowledge augmentation data. The real-time hotspot aggregation unit is used to match the most relevant tag in the preset real-time hotspot library based on the core semantic tags in the global multimodal semantic representation, and use it as the real-time hotspot tag. In this embodiment, based on the global query vector, a tag matching algorithm based on maximum inner product search is used to match the tags with the highest relevance in the real-time crawled social platform hot search and popular phrase database (real-time hot topic database) as real-time hot topic tags.

[0033] In one optional implementation, the multi-dimensional copywriting generation module includes a constraint splicing and prompt construction unit, a large model reasoning generation unit, and a multi-dimensional alignment tag embedding unit. The constraint splicing and prompt construction unit is used to structurally splice the global multimodal semantic representation, domain knowledge enhancement data, real-time hot topic tags, and user-defined creative constraints to construct input prompt words for the multimodal large language model; the creative constraints include target platform constraints, word count limits, and audience constraints. The large model reasoning and generation unit is used to input the input prompt words into a pre-trained multimodal large language model and reason to generate initial text content including title, body text and layout marks. The multi-dimensional alignment tag embedding unit is used to automatically embed timestamps and modal source tags corresponding to the timeline or spatial region of the original multimodal material into the sentences of the generated initial copy content, so as to generate multi-dimensional candidate copy containing multimodal alignment tags. In this embodiment, the formula for generating alignment tags is: In the formula, For the generated first l Multimodal alignment tags for individual text statements; l Index of copywriting statements; To generate the first The mean of the cross-attention weight matrix for visual, auditory, and text modal features at each tag is used to select the modality source with the largest average weight and the corresponding timestamp of the material as the [weight]. The content is used for subsequent cross-modal reverse localization; For tag index; For modal feature indexing; For the first l The set of word indexes corresponding to each text statement in the generated sequence; Modal symbols representing visual semantic features, auditory rhythm features, and text metadata features; In one optional implementation, the cross-modal linkage editing module includes an editing difference calculation unit, a reverse mapping positioning unit, and a linkage modification suggestion unit; The editing difference calculation unit is used to capture the user's modification operations on the candidate text in the editing interface in real time, and use the text embedding model to calculate the difference in text semantic vectors before and after modification. In this embodiment, a semantic vector difference algorithm based on Sentence-Transformer-based Bidirectional Encoder Representations from Transformers (BERT) is used to calculate the semantic vector of the candidate text before modification and the semantic vector of the candidate text after modification, and the difference is obtained by subtracting the two to obtain the difference in the semantic vector of the text before and after modification. The reverse mapping localization unit is used to search for the nearest non-text feature clusters in the shared latent space based on the difference of the text semantic vector and combined with the multimodal alignment tags embedded in the candidate text, to locate the corresponding time segment or spatial region in the original multimodal material. In this embodiment, a latent space nearest neighbor search algorithm based on alignment tag constraints is used. By combining the multimodal alignment tags embedded in the candidate text with the semantic vector difference, the corresponding time segment or spatial region in the original multimodal material is located in the shared latent space. The linked modification suggestion unit is used to generate editing suggestions for the original material based on the semantic direction of the user's modification and the located original multimodal material fragment, and to highlight the suggestions in the editing interface; the editing suggestions include synchronous cropping suggestions and replacement suggestions; In this embodiment, a recommendation algorithm based on a rule engine and semantic direction matching is used to determine the semantic direction modified by the user and the associated material fragments located.

[0034] In one optional implementation, the linked modification suggestion unit is specifically used for: When it detects that a user has deleted any description in the candidate text, the corresponding segment located in the original multimodal material through reverse mapping is automatically highlighted, and an interactive prompt suggesting that the segment be cropped is generated, i.e., a cropping suggestion is generated simultaneously. When it is detected that a user modifies the first description in the candidate text to a second description with opposite semantics, the difference between the modified semantic vectors of the first and second descriptions is calculated. Based on the difference between the modified semantic vectors, a new material that matches the semantics of the second description is retrieved from the preset material library, and an interactive prompt is generated to replace the corresponding segment located in the original multimodal material with the new material, i.e., a replacement suggestion.

[0035] In one alternative implementation, the system further includes an online adaptive fine-tuning module; The online adaptive fine-tuning module is used to collect logs of the final adoption rate and modification range of the candidate copy generated by the system after the user completes the copy editing and exports. The final adoption rate and modification magnitude logs are converted into feedback reward signals, and low-rank adaptive fine-tuning technology is used to incrementally update the parameters of the multimodal large language model in order to optimize the system's copywriting generation strategy for subsequent multimodal materials. In this embodiment, a reinforcement learning algorithm based on near-end policy optimization is used to transform the final adoption rate and modification magnitude logs into feedback reward signals, and a fine-tuning loss function is constructed, the formula of which is: In the formula, To fine-tune the loss value; The symbol represents the mathematical expectation (mean), indicating the average of the results calculated over all sampled data; A sequence of multi-dimensional candidate text tags generated for the model; For the currently being trained, with parameters of A multimodal large language model; For the first The dominant function value of the step; The time step index for generating the sequence; Importance sampling ratio measures the effectiveness of the new and old strategies in generating the same word. The probability ratio above; For the new strategy (the model currently being updated), given the context and prompt words Below, the probability of generating words; This represents the probability of generating words under the same conditions using the old strategy (a snapshot of the model when this batch of data was collected); For the model in the first The actual words generated; For the model in the first All words generated before the previous step; The function is for finding the minimum value; This is the clipping function; This is the clipping hyperparameter (usually set to 0.1 or 0.2). This is the scalar reward value that is ultimately input into the reinforcement learning algorithm; The final adoption rate is the percentage of model-generated copy that users ultimately use (a value between 0 and 1), which is a very strong positive reward metric. To modify the magnitude log, the number of modifications the user made to the candidate text in the editing interface (such as adding or deleting words, usually measured by edit distance) indicates that the more modifications are made, the worse the generation is, which is a negative indicator; The value function is calculated by an internal evaluator of the model, representing the value of a given prompt word. In this case, the model expects to receive an average reward (i.e., the baseline level). The input prompt word sequence (including multimodal features, knowledge, hot topics, constraints, etc. generated by the preceding modules) is fed into the large model. This is the normalization function; This is a positive adoption weighting coefficient, used to control the level of reward for user adoption; To negatively modify the weighting coefficient, the severity of the penalty for user modifications is controlled. The low-rank adaptive fine-tuning algorithm is used to optimize the fine-tuning loss function and update the parameters of the low-rank adaptation matrix in the multimodal large language model, so as to optimize the copywriting generation strategy of the system for subsequent multimodal materials.

[0036] In one optional implementation, the online adaptive fine-tuning module is specifically used for: Based on a reward calculation algorithm that uses a multi-indicator weighted fusion, an evaluation function for the quality of copywriting generation is constructed. The variables of the evaluation function include the candidate copywriting text fluency score, the platform style matching score, the user's editing distance, and the final adoption rate. The evaluation function is weighted and summed to obtain the feedback reward signal, as shown in the formula: In the formula, The feedback reward signal obtained from the detailed reward function; The selected copywriting scores include text fluency, platform style matching, user editing distance, and final adoption rate. The weighting coefficients for the evaluation indicators; Based on the feedback reward signal, a cross-entropy loss function for model fine-tuning is constructed. The gradient of the low-rank adaptation matrix in the multimodal large language model is calculated through the backpropagation algorithm. Only the parameters of the low-rank adaptation matrix are updated, keeping the basic weights of the multimodal large language model frozen. In this embodiment, a joint fine-tuning algorithm incorporating knowledge distillation is used to construct the cross-entropy loss function for model fine-tuning based on the feedback reward signal. The formula is as follows: In the formula, This represents the cross-entropy loss value. For the first The actual target words at each position; For word position indices in the sequence; The length of the target text sequence in the current training sample; For the first All words preceding the given word (i.e., the already generated preceding context); The sequence of prompt words is input to the large model; To predict the first [number]th [item] given the context of the model. Each word element is The probability of; The total loss function is obtained by combining the fine-tuned loss function, and the formula is as follows: In the formula, This represents the total loss value. To enhance the learning balance coefficient; This is the distillation loss function; This is the distillation loss coefficient; This is the KL divergence, also called relative entropy; The output probability distribution of the student model; This represents the output probability distribution of the teacher model.

[0037] In one optional implementation, the system further includes a multimodal consistency verification module; The multimodal consistency verification module is used to perform multimodal semantic consistency verification on the modified text and the replaced material after the user performs cross-modal collaborative editing and confirms the modification suggestions; Calculate the cosine similarity between the text semantic vector of the modified copy and the visual and / or auditory feature vector of the replaced material in the shared latent space; If the cosine similarity is lower than the preset consistency threshold, a warning message indicating audio-visual-text mismatch will be issued to the user, and the linkage modification suggestion unit will be triggered again to generate a new replacement suggestion.

[0038] In one optional implementation, the multimodal consistency verification module is specifically used for: Extract the sentence-level text vector of each sentence in the modified candidate copy, and the modal feature vector of the corresponding segment in the replaced material; Calculate the local cosine similarity between the sentence-level text vector and the corresponding modal feature vector for each statement in turn; For target statements with a local cosine similarity lower than a preset local threshold, it is determined that the target statement has a semantic conflict with the corresponding material segment; Generate semantic conflict identifiers for the target statement, visualize the semantic conflict identifiers on the editing interface, and re-retrieve matching materials based on the semantics of the target statement to overwrite the replaced materials.

[0039] Based on the same inventive concept, another embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus. Memory, used to store computer programs; The processor, when executing the program stored in the memory, implements the AI ​​multimodal copywriting assistance system for cultural media of the present invention.

[0040] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EI) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. The communication interface is used for communication between the aforementioned terminal and other devices. The memory can include Random Access Memory (RAM), or non-volatile memory, such as at least one disk storage device. Optionally, the memory can also be at least one storage device located remotely from the aforementioned processor.

[0041] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0042] Furthermore, to achieve the above objectives, embodiments of the present invention also propose a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the AI ​​multimodal copywriting assistance system for cultural media of the present invention.

[0043] Those skilled in the art will understand that embodiments of the present invention can be provided as systems, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable hardware devices (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0044] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of systems, terminal devices (apparatus), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0045] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0046] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0047] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. "And / or" indicates that either one or both can be chosen. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, system, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, system, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, system, article, or terminal device that includes the element.

[0048] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An AI-powered multimodal copywriting assistance system for cultural media, characterized in that: The system includes: The module includes a multimodal feature alignment and fusion module, a media knowledge enhancement and hot topic retrieval module, a multi-dimensional copywriting generation module, and a cross-modal collaborative editing module. The multimodal feature alignment and fusion module is used to acquire a set of multimodal materials input by the user, extract cross-modal features from the set of multimodal materials, and perform feature alignment and fusion in the shared latent space to obtain a global multimodal semantic representation. The media knowledge enhancement and hotspot retrieval module is used to retrieve related domain knowledge enhancement data and real-time hotspot tags from a pre-set cultural media knowledge base and a real-time hotspot database based on the global multimodal semantic representation. The multi-dimensional copywriting generation module is used to input the global multimodal semantic representation, domain knowledge augmentation data, real-time hot topic tags and user-defined creative constraints into a pre-trained multimodal large language model to generate multi-dimensional candidate copywriting that is adapted to the style of a specific media platform and includes multimodal alignment tags. The cross-modal linkage editing module is used to respond to the user's editing operation on the candidate text, obtain the edited candidate text, calculate the semantic difference between the candidate text and the edited candidate text, back-map it to the related fragments in the multimodal material set, and provide editing suggestions for the material.

2. The AI ​​multimodal copywriting assistance system for cultural media as described in claim 1, characterized in that, The multimodal feature alignment and fusion module includes an image visual coding unit, an audio auditory coding unit, a text semantic coding unit, and a cross-modal alignment and fusion unit; The image visual encoding unit is used to acquire a multimodal material set input by the user, and use a visual transformer network to encode the image and video keyframes in the multimodal material set, capture the spatial layout and object features of the screen, and obtain visual semantic features. The audio auditory coding unit is used to encode the audio track data in the multimodal material set using an audio spectrum analysis model, extract acoustic spectrum features and speech text content, and obtain auditory rhythm features. The text semantic encoding unit is used to use a text embedding model to vectorize and encode text information containing titles, subtitles, and metadata in a multimodal material set to obtain text metadata features. The cross-modal alignment and fusion unit is used to map the visual semantic features, auditory rhythm features, and text metadata features to a shared latent space for alignment based on a contrastive learning mechanism. It also employs a multi-head cross-modal attention mechanism, using text metadata features as query vectors and visual semantic features and auditory rhythm features as key and value vectors for fusion, to obtain a global multimodal semantic representation.

3. The AI ​​multimodal copywriting assistance system for cultural media as described in claim 2, characterized in that, The media knowledge enhancement and hot topic retrieval module includes a feature vectorization retrieval unit, a domain knowledge matching unit, and a real-time hot topic aggregation unit. The feature vectorization retrieval unit is used to use the global multimodal semantic representation as a query vector to perform nearest neighbor retrieval in a pre-set cultural and media knowledge base to obtain candidate knowledge entries. The domain knowledge matching unit is used to extract data from the candidate knowledge items that correspond to the media platform style guidelines, prohibited word library and popular copywriting structure template, as domain knowledge enhancement data. The real-time hotspot aggregation unit is used to match the most relevant tag in the preset real-time hotspot library based on the core semantic tags in the global multimodal semantic representation, and use it as the real-time hotspot tag.

4. The AI-powered multimodal copywriting assistance system for cultural media as described in claim 3, characterized in that, The multi-dimensional copywriting generation module includes a constraint splicing and prompt construction unit, a large model reasoning generation unit, and a multi-dimensional alignment tag embedding unit. The constraint splicing and prompt construction unit is used to structurally splice the global multimodal semantic representation, domain knowledge enhancement data, real-time hot topic tags, and user-defined creative constraints to construct input prompt words for the multimodal large language model; the creative constraints include target platform constraints, word count limits, and audience constraints. The large model reasoning and generation unit is used to input the input prompt words into a pre-trained multimodal large language model and reason to generate initial text content including title, body text and layout marks. The multi-dimensional alignment tag embedding unit is used to automatically embed timestamps and modal source markers corresponding to the timeline or spatial region of the original multimodal material into the sentences of the generated initial copy content, so as to generate multi-dimensional candidate copy containing multimodal alignment tags.

5. The AI ​​multimodal copywriting assistance system for cultural media as described in claim 4, characterized in that, The cross-modal linkage editing module includes an editing difference calculation unit, a reverse mapping positioning unit, and a linkage modification suggestion unit; The editing difference calculation unit is used to capture the user's modification operations on the candidate text in the editing interface in real time, and use the text embedding model to calculate the difference in text semantic vectors before and after modification. The reverse mapping localization unit is used to search for the nearest non-text feature clusters in the shared latent space based on the difference of the text semantic vector and combined with the multimodal alignment tags embedded in the candidate text, to locate the corresponding time segment or spatial region in the original multimodal material. The linked modification suggestion unit is used to generate editing suggestions for the original material based on the semantic direction of the user's modification and the located original multimodal material fragment, and to highlight the suggestions in the editing interface. The editing suggestions include simultaneous cropping suggestions and replacement suggestions.

6. The AI-powered multimodal copywriting assistance system for cultural media as described in claim 5, characterized in that, The linked modification suggestion unit is specifically used for: When it detects that a user has deleted any description in the candidate text, the corresponding segment located in the original multimodal material through reverse mapping is automatically highlighted, and an interactive prompt suggesting that the segment be cropped is generated, i.e., a cropping suggestion is generated simultaneously. When it is detected that a user modifies the first description in the candidate text to a second description with opposite semantics, the difference between the modified semantic vectors of the first and second descriptions is calculated. Based on the difference between the modified semantic vectors, a new material that matches the semantics of the second description is retrieved from the preset material library, and an interactive prompt is generated to replace the corresponding segment located in the original multimodal material with the new material, i.e., a replacement suggestion.

7. The AI ​​multimodal copywriting assistance system for cultural media as described in claim 6, characterized in that, The system also includes an online adaptive fine-tuning module; The online adaptive fine-tuning module is used to collect logs of the final adoption rate and modification range of the candidate copy generated by the system after the user completes the copy editing and exports. The final adoption rate and modification magnitude logs are converted into feedback reward signals, and low-rank adaptive fine-tuning technology is used to incrementally update the parameters of the multimodal large language model in order to optimize the system's copywriting generation strategy for subsequent multimodal materials.

8. The AI ​​multimodal copywriting assistance system for cultural media as described in claim 7, characterized in that, The online adaptive fine-tuning module is specifically used for: Construct an evaluation function for the quality of generated copy, wherein the variables of the evaluation function include candidate copy text fluency score, platform style matching score, user editing distance, and final adoption rate; The evaluation function is weighted and summed to obtain the feedback reward signal; Based on the feedback reward signal, a cross-entropy loss function for model fine-tuning is constructed. The gradient of the low-rank adaptation matrix in the multimodal large language model is calculated through the backpropagation algorithm, and only the parameters of the low-rank adaptation matrix are updated, keeping the basic weights of the multimodal large language model frozen.

9. The AI ​​multimodal copywriting assistance system for cultural media as described in claim 8, characterized in that, The system also includes a multimodal consistency verification module; The multimodal consistency verification module is used to perform multimodal semantic consistency verification on the modified text and the replaced material after the user performs cross-modal collaborative editing and confirms the modification suggestions; Calculate the cosine similarity between the text semantic vector of the modified copy and the visual and / or auditory feature vector of the replaced material in the shared latent space; If the cosine similarity is lower than the preset consistency threshold, a warning message indicating audio-visual-text mismatch will be issued to the user, and the linkage modification suggestion unit will be triggered again to generate a new replacement suggestion.

10. The AI ​​multimodal copywriting assistance system for cultural media as described in claim 9, characterized in that, The multimodal consistency verification module is specifically used for: Extract the sentence-level text vector of each sentence in the modified candidate copy, and the modal feature vector of the corresponding segment in the replaced material; Calculate the local cosine similarity between the sentence-level text vector and the corresponding modal feature vector for each statement in turn; For target statements with a local cosine similarity lower than a preset local threshold, it is determined that the target statement has a semantic conflict with the corresponding material segment; Generate semantic conflict identifiers for the target statement, visualize the semantic conflict identifiers on the editing interface, and re-retrieve matching materials based on the semantics of the target statement to overwrite the replaced materials.

Images