A method, system, and medium for persona style alignment for multi-modal content

Abstract

This invention discloses a method, system, and medium for aligning persona styles for multimodal content, relating to the field of artificial intelligence technology. The method includes: acquiring and preprocessing a set of text, cover images, and accompanying images, then extracting features and mapping each feature to a unified persona space to obtain text alignment features, cover alignment features, and accompanying image alignment features; using the text alignment features as queries and the cover and accompanying image alignment features as keys and values, performing weighted aggregation through a cross-attention mechanism to output alignment enhancement features; based on a supervised training consistency generation model, calculating the persona consistency probability according to the text alignment features and the alignment enhancement features, and outputting a release decision based on the comparison result of the persona consistency probability with a preset threshold; this method achieves automated extraction of deep persona features and strong consistency constraints.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, system, and medium for aligning character styles for multimodal content. Background Technology

[0002] With the explosive growth of self-media, short videos, and social platforms, content production has fully entered the era of multimodal integration. Creators typically need to use a combination of media formats, such as headlines, body text, cover posters, and embedded images, to convey information. In this context, maintaining consistency in "persona style" has become crucial for brand operation and content distribution—a stable and consistent visual and linguistic tone can significantly enhance user brand recognition and strengthen trust and engagement. However, in the actual creative process, because different modalities such as text and images are often generated by different tools or produced manually in stages, the lack of a unified style control center easily leads to perceptual dissonance issues such as "mismatch between content and title" or "visual style jumps."

[0003] Persona style alignment refers to the technical process of ensuring that the textual tone and visual style of a collection of text content, cover images, and accompanying images within the same account, brand, or content theme meet preset consistency conditions in the feature space. This is achieved through cross-modal feature extraction, unified spatial mapping, attention-weighted aggregation, and consistency probability judgment. Currently, most persona style alignment methods rely on a single data source or method and can be mainly divided into the following three categories: Rule-based or template-based static alignment methods: This method primarily uses preset fixed layouts, standard terminology, or high-frequency word lists for rule matching. This approach is simple and fast.

[0004] A hierarchical verification method based on independent discriminators: This method uses style classification models for text and images respectively to perform independent style evaluation and correction for each modality.

[0005] A method for association alignment based on general semantic mapping: This method uses a pre-trained cross-modal model to map images and text to a general shared semantic space, and achieves association by calculating similarity. Its core lies in solving "semantic relevance" (such as recognizing that both images and text describe "forest").

[0006] Current technical solutions for character style alignment mainly suffer from the following problems: First, rule-driven static alignment lacks semantic flexibility. Existing methods based on preset rules, fixed layout templates, or standard terminology libraries are based on hard mapping. Because such solutions lack the ability to dynamically perceive the deep logic and context of content, the resulting alignment is often too mechanical and rigid. They cannot flexibly capture and present the subtle and varied style characteristics of a specific persona according to different creative themes, and thus cannot meet the needs of personalized brand building.

[0007] Second, the isolated modality verification mechanism lacks cross-modal feature coupling. Existing methods based on independent classification models perform single-modal style evaluation on text and images separately. Since the alignment criteria between each modality are isolated in the feature space and lack deep interactive connections, although the output results may conform to style tags in a single dimension (such as pure text or pure image), when displayed as a whole, it is very easy to have problems such as mismatch between visual tone and linguistic connotation, aesthetic conflict, and other user perception disjointedness.

[0008] Third, alignment based on a general semantic space lacks the ability to extract style-dimensional features. Existing methods based on pre-trained cross-modal large models (such as CLIP) mainly anchor feature alignment on the semantic association of physical attributes such as objects and scenes. Because the general semantic space has insufficient precision in representing high-order abstract features such as artistic style and personality tone, it cannot achieve style coordination between the main text, cover visuals, and embedded images in a deeper personality space, making it difficult to support long-term consistency across the account dimension.

[0009] Therefore, how to automatically extract deep character traits and impose strong consistency constraints in multimodal content creation is a core problem that urgently needs to be solved in the current technology field. Summary of the Invention

[0010] Based on the technical problems existing in the background technology, this invention proposes a method, system and medium for aligning character styles for multimodal content, which realizes the automatic extraction of deep character features and strong consistency constraints.

[0011] This invention proposes a method for aligning character styles for multimodal content, comprising: After obtaining and preprocessing the text, cover image, and illustration set, feature extraction is performed. Each feature is mapped to a unified persona space to obtain text alignment features, cover alignment features, and illustration alignment features. The illustration set is a collection of one or more visual materials embedded in addition to the cover image. Using the text alignment feature as the query, and the cover alignment feature and the image alignment feature as the key and value, a weighted aggregation is performed through a cross-attention mechanism to output the alignment enhancement feature; The supervised training-based consistency generation model calculates the persona consistency probability based on the text alignment features and the alignment enhancement features, and outputs a release decision based on the comparison result of the persona consistency probability and a preset threshold.

[0012] Furthermore, the mapping of each feature to a unified persona space yields text alignment features, cover alignment features, and image alignment features, specifically including: The text features extracted by the trainingable first linear projection layer are mapped to the target dimension to obtain the text alignment features; The visual features of the cover image obtained by feature extraction are mapped to the target dimension using a trainable second linear projection layer to obtain the cover alignment features; The second linear projection layer is used to map the visual features of each image in the image set, obtained by feature extraction, to the target dimension to obtain the image alignment features. The first linear projection layer and the second linear projection layer are trained based on cross-modal contrastive learning loss. The cross-modal contrastive learning loss is used to reduce the feature distance between the text alignment features and cover alignment features of the same content sample in the unified persona space, and to increase the feature distance between the text alignment features and cover alignment features of different content samples in the unified persona space.

[0013] Furthermore, the step of using the text alignment feature as the query, and the cover alignment feature and the accompanying image alignment feature as the key and value, respectively, and performing weighted aggregation through a cross-attention mechanism to output alignment enhancement features, specifically includes: The query is obtained by performing a linear transformation on the text alignment features; A linear transformation is performed on the combined feature formed by splicing the cover alignment feature and the image alignment feature to obtain the key and the value; Calculate the scaled dot product similarity between the query and the key, and process it with a normalized exponential function to obtain the attention weight; The values ​​are weighted and summed using the attention weights to generate the alignment enhancement feature.

[0014] Furthermore, the consistency generation model includes a feature forward computation link and a consistency discrimination head; The feature forward computation link is used to perform feature extraction, mapping to a unified persona space, and cross-attention mechanism processing on the preprocessed text, cover image, and image set to generate the text alignment feature and the alignment enhancement feature; wherein, the preprocessed text, cover image, and image set correspond to the main text content, cover visual material, and one or more image visual materials embedded in the main text, respectively. The consistency discriminant head is used to calculate the cosine similarity between the text alignment feature and the alignment enhancement feature, and to map the cosine similarity to the persona consistency probability.

[0015] Furthermore, the consistency generation model is trained under supervision based on binary cross-entropy loss. The training sample library includes positive samples and negative samples. The positive samples are combinations of text and images published historically by the same account that satisfy the persona consistency. The negative samples are combinations of text and images formed by randomly combining the text of the account with the image sets of other accounts. The text and image combinations include text, cover image, and accompanying image set.

[0016] Furthermore, in the process of obtaining and preprocessing the text, cover image, and illustration set for feature extraction, the feature extraction is performed by a parallel feature extraction network, which includes a language sub-model and a visual sub-model. The language sub-model is trained based on masked language modeling loss and is used to extract text feature vectors from preprocessed text. The visual sub-model is trained based on contrastive learning loss and is used to extract visual feature vectors from the preprocessed cover image and the accompanying image set.

[0017] Furthermore, the language sub-model adopts a Transformer encoder structure, and the text features are obtained by pooling the output of the Transformer encoder; The visual sub-model adopts a Vision Transformer encoder structure, and the visual features are obtained by encoding the input image through the Vision Transformer encoder.

[0018] Further, the text, cover image, and accompanying illustrations are acquired and preprocessed, the preprocessing including: The text is subjected to noise removal and sequence length normalization. Each image in the cover image and accompanying image set is subjected to scale reshaping and pixel value normalization.

[0019] A persona style alignment system for multimodal content includes: At least one processor; and A memory that is communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method described above.

[0020] A computer-readable storage medium storing a plurality of computer programs for being invoked by a processor and executing the method described above.

[0021] The advantages of the persona style alignment method, system, and medium provided by this invention for multimodal content are as follows: For text-image integrated content, it extracts the visual / linguistic features of the main text, cover image, and accompanying images, maps them to a unified persona space, and forces the three to maintain stylistic consistency through an attention mechanism. This ensures that different content works under the same account maintain a high degree of stylistic consistency, avoiding fragmented user perception. Through this series of designs, it aims to provide a more intelligent cross-modal persona style collaborative alignment solution, providing technical support for building a highly consistent digital brand image. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the process of the present invention; Figure 2 This is a flowchart of the training process for the language sub-model. Figure 3 The training flowchart for the visual sub-model; Figure 4 The training flowchart for the dimension alignment and projection mapping model; Figure 5 Flowchart for training the cross-attention alignment model; Figure 6 A flowchart of the training process for generating a consistency model. Detailed Implementation

[0023] The technical solution of the present invention will now be described in detail through specific embodiments. Many specific details are set forth in the following description to provide a thorough understanding of the invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0024] like Figures 1 to 6 As shown, the present invention proposes a method for aligning character styles for multimodal content, comprising: Step 1: Obtain the text, cover image, and illustration set, preprocess them, and then extract features. Map each feature to a unified persona space to obtain text alignment features, cover alignment features, and illustration alignment features. Step 2: Using the text alignment feature as the query, and the cover alignment feature and the accompanying image alignment feature as the key and value, perform weighted aggregation through a cross-attention mechanism to output the alignment enhancement feature; Step 3: Obtain a supervised training-based consistency generation model, calculate the persona consistency probability based on the text alignment features and the alignment enhancement features, and output a release decision based on the comparison result of the persona consistency probability and the preset threshold.

[0025] The core idea of ​​this embodiment is to construct a unified persona space that can simultaneously represent visual and linguistic features. A feature extraction network maps the main text, cover image, and accompanying images to this persona space. An attention-based forced alignment constraint algorithm is introduced to dynamically adjust the contribution weight of each modal feature to style consistency, ultimately achieving a high degree of stylistic coherence across different content works under the same account. Specifically, this embodiment has the following advantages: 1. Construct a comprehensive alignment framework that can integrate cross-modal heterogeneous features and unify the representation of style attributes, overcoming the limitations of isolated alignment standards and lack of interaction in traditional methods.

[0026] 2. Based on the stability requirements of persona style and the mutual perception characteristics of multimodal features, dynamic optimization of feature weights and strong consistency constraints are achieved to reduce perceptual bias.

[0027] 3. Establish a high-level feature extraction and correction mechanism for "persona tone" to solve the problem of insufficient accuracy in general semantic space representation.

[0028] Step 1: Multimodal data acquisition and preprocessing; The system first receives raw multimodal datasets from external input and a database. The data is then standardized to provide a standardized input for subsequent feature extraction networks. This process includes the following steps: (a1) Acquisition and preprocessing of raw data stream; The system acquires the multimodal dataset to be processed. ,in. This indicates the text content to be aligned; This represents the cover image to be aligned; This represents the set of images to be aligned. Includes one or more visual images embedded in the text. For the first One accompanying image, This refers to the number of images included.

[0029] The cover image refers to the image displayed as the main visual element in the content to be published at the publishing entrance, cover position, or list display position. The accompanying image set refers to the collection of one or more visual materials embedded in the main text content, excluding the cover image.

[0030] Language data preprocessing, specifically for the main text Remove noise characters to obtain a simplified word sequence. Sequence normalization: based on a preset threshold for the length of the user-defined space. The abbreviated word sequence is truncated or padded to ensure that the input sequence length is consistent, resulting in preprocessed text. .

[0031] Visual data preprocessing, specifically for the cover image With accompanying image collection Perform a uniform image transformation operation. Scale reshaping: Adjust all original images to a preset resolution. Tensor normalization: Normalizes pixel values; the calculation formula is as follows: (in These are the original pixel values. The average pixel value. (where is the standard deviation) maps the data distribution to a specified interval to accelerate algorithm convergence. To simplify parameters, the cover image used subsequently... With accompanying image collection Both refer to images that have undergone visual data preprocessing.

[0032] Step 2: Parallel feature extraction network; The system will obtain the preprocessed text (a1). Cover image With accompanying image collection Parallel input of language and vision sub-models, respectively extracting text feature vectors. With visual feature vectors And summarize them to form a multimodal original feature set. This provides standardized feature inputs for subsequent cross-modal alignment and fusion.

[0033] For language sub-models, such as Figure 2 As shown, this is used to extract text from a preprocessed sequence. Extracting global semantic representation from the text yields text feature vectors. The training and execution process is as follows (b1) to (b4): (b1) Training data construction: The system uses preprocessed text Construct training samples. The input sequence is given by masking a subset of words according to a preset masking ratio. And using the original lexical units at the masked positions as the supervision targets, a definite training pair is constructed. .

[0034] (b2) Model Structure: The language sub-model adopts a Transformer encoder structure. The system will... After word embedding and positional encoding, the result is input into the Transformer encoder to obtain the sequence hidden representation; its global text features are extracted using the pooling operator. ; in, This refers to the text feature vectors extracted from the text using a language sub-model. For Transformer encoders, For pooling operators (take) (The vector serves as a global representation). Simultaneously, the hidden vectors at each position output by the encoder are used for masked word prediction.

[0035] (b3) Loss Function: The language sub-model is trained using masked language modeling loss. Let the set of masked locations be... ,Location The real lexical units are The vocabulary distribution predicted by the model for this position is as follows: The loss is: ; in, The loss function for training the language sub-model. Input sequence to the model after a given mask Under the condition of predicting location Real lexical The probability of.

[0036] By minimizing The language sub-model is forced to learn deep dependencies between lexical units, thereby enabling it to encode the input text sequence into feature vectors containing rich semantic information for subsequent persona space mapping.

[0037] (b4) Model output: After training is completed, the language sub-model receives input during the inference phase. (No masking required) Output text feature vector .

[0038] For visual sub-models, such as Figure 3 As shown, used from the cover image With accompanying image collection Extract global visual features and output a visual feature vector, which includes the cover visual feature vector. Visual feature vectors of the accompanying images The training and execution of the visual sub-model are detailed in (c1) to (c4): (c1) Training data construction: The system uses the preprocessed cover image from step (a1) as the training data. With accompanying image collection Each of the accompanying images is used as a training sample, and each input image is processed accordingly. Apply two preset image enhancement transformations to obtain two views of the same image. Both enhancement transformations are fixed-definition combination transformations used to construct positive sample pairs. And within the same batch, different image views are used as negative samples.

[0039] (c2) Model Structure: The visual sub-model employs a Vision Transformer (ViT) encoder structure. The input image view is divided into fixed-size patches, which are then linearly projected and embedded before being input into the ViT encoder to obtain visual feature vectors. For any view... Its characteristics are represented as ,in For ViT encoder, Indicates the view index.

[0040] (c3) Loss Function: The visual sub-model is trained using contrastive learning loss (InfoNCE). Assume a batch contains... Each image generates two views and yields features. Define the similarity function as cosine similarity: ,by As the anchor point, its positive samples are Then one-way comparison loss for: ; in, For image indexes within a batch, This is a temperature coefficient used to adjust the smoothness of the similarity distribution. For the first The feature vector of the first enhanced view of the image. For the first The feature vector of the second enhanced view of the image. For the first The feature vector of the second enhanced view of the image, namely , This is the cosine similarity function.

[0041] This one-way contrast loss The core objective is to train the visual sub-model to learn discriminative features that are invariant to image enhancement transformations. Specifically: Positive sample pairs: the same image obtained after undergoing two different enhancement transformations. This reduces its characteristic distance within a unified persona space. (The molecule's...) Encourage the pair to have the highest possible similarity in features.

[0042] Negative sample pairs: Increase anchor points Other image views Feature distance within a unified persona space. The denominator includes all... The sum of similarities forces the visual submodel to set the anchor point with a significantly higher similarity to positive samples than with any negative samples.

[0043] Symmetry: The formula shows the one-way loss. (Using the first view as the anchor point) In actual training, the loss in the symmetric direction will also be calculated. (Using the second view as the anchor point).

[0044] The final visual sub-model's total loss is symmetric. .

[0045] (c4) Model output: After training is complete, the visual sub-model receives input during the inference phase. and Output cover features and the set of illustration features .

[0046] Step 3: Dimensional alignment and projection mapping model; The system constructs a trainable projection mapping model to address the differences in dimensionality and distribution space of different modal features, thereby mapping text feature vectors. With visual feature vectors (including and The unified persona space is mapped to a pre-defined unified persona mapping space to obtain a same-dimensional feature representation that can be directly used for cross-modal similarity calculation and consistency verification. Both the unified persona space and the general cross-modal shared semantic space are used to map textual and visual features to comparable same-dimensional feature spaces, but their optimization objectives differ: the general cross-modal shared semantic space mainly represents the correspondence between text and images at the general semantic level, such as objects, scenes, and events; the unified persona space, on the other hand, further constrains the language expression, visual tone, composition style, and overall content tone of the account, brand, or content subject, representing the persona consistency relationship between the text language tone and the visual style tone. Therefore, the unified persona space is a cross-modal style representation space oriented towards persona style consistency tasks, rather than a shared semantic space used only for general semantic matching.

[0047] Specifically, it involves taking the text feature vector obtained above... and visual feature vectors Mapped to a unified dimension through a learnable linear projection transformation. Persona space The values ​​can be powers of 2 such as 512, 1024, etc., and the output is the aligned feature set. Including text alignment features Cover alignment features and image alignment features .

[0048] By analyzing text feature vectors Its corresponding visual feature vector Dimensional alignment and projection mapping are performed to establish a unified persona space, providing a unified feature foundation for subsequent cross-modal similarity calculation, consistency verification, and style alignment.

[0049] like Figure 4 As shown, the training and execution process of the dimension alignment and projection mapping model is as follows: (d1) Construction of training data; The training data comes directly from the multimodal original feature set obtained above: The system uses the text feature vector within the same sample Its corresponding visual feature vector Construct positive sample pairs, using the text feature vectors between different samples. Its corresponding visual feature vector Negative sample pairs are constructed to form a sample set for cross-modal alignment training. This construction method does not rely on additional uncertain labels; the sample correspondences are naturally determined by the original data.

[0050] (d2) Model structure; The system constructs separate linear projection layers for different modalities to eliminate dimensional differences and map them to a unified dimension. (Optional). Text projection layer parameters are: The parameters of the visual projection layer are The alignment mapping is calculated as follows: Text feature projection: Cover image feature projection: Image feature set projection: For each image feature set... Obtained by projection respectively And construct the set of image projections: After the above mapping, the system summarizes and obtains a unified set of character space features: .

[0051] in, The text projection weight matrix, The visual projection weight matrix, This is the text projection bias vector. This is the visual projection bias vector. For the image index, For the first Alignment features of each accompanying image For text alignment features, For cover alignment features, This is for image alignment features.

[0052] (d3) Loss function; To ensure consistency of the projected cross-modal representations within a unified space, the system employs cross-modal contrastive learning loss to adjust the projection parameters. Perform training. Assume a batch contains B samples, and the... The text alignment features of the sample are Its cover alignment features are Define the cosine similarity function. .

[0053] With text Contrast loss in the direction of the cover for: ; Contrast loss in the direction of cover → text for: ; in, For temperature coefficient, This is the sample index.

[0054] The final loss function takes a symmetric form: By minimizing The system achieves feature alignment between text and images within the same sample in a unified persona space.

[0055] (d4) Model output; After training, the dimension alignment and projection mapping model receives input during the inference phase. Output the unified persona spatial feature set after alignment: text alignment features Cover alignment features and image alignment features ;Summarized It is used for subsequent similarity calculation, consistency verification and downstream module calls.

[0056] Step 4: Cross-attention alignment model; Based on the unified character space feature set obtained above We construct a cross-attention alignment model, using text alignment features within a unified dimensional space. Anchor point alignment feature for cover and image alignment features Dynamic weighted alignment is performed to obtain alignment enhancement features. This is to enhance the consistency of visual style with text semantics.

[0057] Specifically, the cross-attention alignment model is used to align features with text. As a query, with cover alignment feature and image alignment features As key / value pairs, cross-modal attention weights are calculated and weighted converged to output aligned enhancement features. This enables optimal alignment of multiple images and suppression of style conflicts.

[0058] like Figure 5 As shown, the training and execution process of the cross-attention alignment model is as follows (e1) to (e4): (e1) Construction of training data; The training data comes from the unified persona space features mentioned above. For the same sample, the system extracts its text alignment features. Corresponding visual alignment feature set Construct positive sample pairs; construct negative sample pairs for the text alignment feature and visual alignment feature sets between different samples. This forms a cross-modal alignment training batch, which is used to learn the trainable parameters in cross-attention, enabling the model to improve the consistency between homologous text and vision and suppress heterologous matching within a unified persona space.

[0059] (e2) Model structure; The system provides unified character space features ( A linear transformation is applied to construct the query matrix Q, the key matrix K, and the value matrix V. The query Q is determined by text alignment features. Generate, keys and values ​​are aligned by cover features and image alignment features Generated after splicing. Let the learnable parameters be... ,but: ; ; .

[0060] The system calculates the scaled dot product similarity between the query and the key, and then obtains the attention weights using Softmax. : ; in, To unify the dimensions of character design features (and) Consistent).

[0061] The value matrix is ​​then weighted and converged to obtain the alignment enhancement features: This alignment enhancement feature It can automatically increase the weight of visual components that are consistent with the text's persona in the visual candidate set, and suppress visual components that conflict with the text's style.

[0062] This embodiment employs a cross-attention structure using text alignment features as the query and cover alignment features and accompanying image alignment features as keys and values. The reason for this is that in multimodal content to be published, text is typically a single semantic input, reflecting the theme and tone of the current content; while the visual side usually includes a cover image and multiple accompanying images within the text, forming multiple candidate visual materials. Therefore, using text alignment features as the query allows the text's tone to serve as an alignment anchor, matching, filtering, and weighting the visual features of the cover image and each accompanying image. This increases the weight of visual components consistent with the text's style and suppresses the weight of visual components inconsistent with the text's style.

[0063] (e3) Loss function; To enhance the alignment of cross-attention output features Alignment features with source text To achieve greater consistency within a unified persona space, the system employs cross-modal contrastive learning loss to train learnable parameters in cross-attention. In a containing In the batch of samples, the first The text alignment features of the sample are Its alignment enhancement feature is Define the cosine similarity function. .

[0064] Then text The contrastive loss for alignment enhancement features is: .

[0065] Alignment Enhancement Features The contrast loss of the text is: .

[0066] Final loss function Take the symmetrical form: ,in The temperature coefficient is minimized. The system learns a stable cross-modal alignment weight allocation strategy.

[0067] (e4) Model output; After training, the cross-attention alignment model receives input during the inference phase. Output alignment enhancement features .in, This represents a unified visual alignment representation of the cover and accompanying image features after weighted convergence under text anchor constraints; simultaneously, attention weights... It can serve as an interpretable result of the alignment process, used to characterize the contribution of each visual element to the final alignment representation.

[0068] Step 5: Consistency generation model; The consistency generation model is used to supervise the learning of the matching relationship between text and vision within a unified persona space, and outputs consistency scores and release decisions based on alignment enhancement features during the inference stage, thereby realizing an integrated generation process of training-selection-output.

[0069] like Figure 6 As shown, the training and execution process of the consistency generation model is as follows (f1) to (f4): (f1) Construction of training data; The system constructs a training sample library for persona consistency. Each sample consists of a "text-image set" and is given a binary consistency label. Among them: positive samples ( ): A combination of images and text posted historically by the same account and manually confirmed to be consistent with the persona; negative samples ( The text-image combination is obtained by randomly combining the account's text with image sets from other accounts. This combination is used to simulate style conflict samples. The text-image combination includes text, a cover image, and a set of accompanying images. For any training sample, the text input is the preprocessed text. Image input is and The feature extraction and alignment process follows the parallel feature extraction network from step 2 to the cross-attention alignment model from step 4, resulting in the corresponding... .

[0070] (f2) Model structure; The consistency generation model consists of a feature forward computation link and a consistency discrimination head. The feature forward computation link reuses the calculation results from steps 2 to 4, specifically for performing feature extraction, mapping to a unified persona space, and cross-attention mechanism processing on the preprocessed text, cover image, and accompanying image set to generate the text alignment features and the alignment enhancement features. The preprocessed text, cover image, and accompanying image set correspond to the main text content, cover visual material, and one or more accompanying image visual materials embedded in the main text, respectively. Parallel feature extraction yields… ; Projection mapping yields Cross-attention alignment yields aligned enhancement features. Based on this, the system will and Input a consistency discrimination header and calculate the image-text consistency score. This header uses cosine similarity as a metric. And map the similarity to a probability output: ; in, This indicates the predicted probability that the current combination of text and images satisfies the consistency of the character profile.

[0071] (f3) Loss function; The system uses binary cross-entropy loss to supervise the training of the consistency generation model. Let the training batch contain B samples, and the i-th... The true label for each sample is The model outputs a consistency probability of 1. Then the loss function for: . The system minimizes through backpropagation. This makes the consistency probability of positive samples higher and the consistency probability of negative samples lower, thereby solidifying the consistency discrimination boundary in the unified persona space.

[0072] (f4) Model output; After training, the consistency generation model receives a set of text and images to be published during the inference phase. Obtained by following steps 2 to 4 and Output a consistency score: As a result of the prediction of consistency between the persona and the character.

[0073] Decision to be issued: With preset threshold In comparison, when When the combination of images and text is determined to be consistent with the persona, the publishing process is allowed. The system will detect any deviation from the character's established persona and trigger a notification or replacement suggestion.

[0074] Alignment Enhancement Feature: Output As a feature input for subsequent content generation or layout combination.

[0075] In summary, this embodiment has the following advantages: First, this embodiment achieves cross-modal alignment between text semantics and cover / illustration visual style through a unified persona space modeling mechanism of dimension alignment projection + cross attention alignment. This reduces the risk of text-image style conflict and persona deviation, and improves the overall consistency and stability of multimodal content.

[0076] Second, this embodiment introduces a discriminative training and threshold screening mechanism based on supervised consistency samples, outputting a quantifiable probability of persona consistency and a clear release decision, making the alignment effect measurable and interpretable, and facilitating automated review and quality control in the content production chain.

[0077] Third, the alignment enhancement features and attention weights output in this embodiment can be used as inputs for downstream content generation, material recommendation, layout combination and risk warning, so as to realize the reusability and scalability of alignment capabilities, reduce the subsequent system integration costs and improve the overall content production efficiency.

[0078] Based on the above description of the embodiments, those skilled in the art will understand that the persona style alignment method, system, and medium for multimodal content described in this embodiment can be implemented in pure software or deployed and run on a general-purpose or dedicated computing hardware platform. Based on this essence, the technical solution of this embodiment can be specifically implemented in the form of a software product containing program instructions. This software product can be stored on various non-volatile storage media or directly deployed as a local or cloud service. The program instructions are used to cause computer devices with processing capabilities—including but not limited to personal computers, server clusters, mobile terminals, or other network devices—to execute the steps described in this embodiment.

[0079] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for persona style alignment for multi-modal content, the method comprising: include: After obtaining and preprocessing the text, cover image, and illustration set, feature extraction is performed. Each feature is mapped to a unified persona space to obtain text alignment features, cover alignment features, and illustration alignment features. The illustration set is a collection of one or more visual materials embedded in addition to the cover image. Using the text alignment feature as the query, and the cover alignment feature and the image alignment feature as the key and value, a weighted aggregation is performed through a cross-attention mechanism to output the alignment enhancement feature; The supervised training-based consistency generation model calculates the persona consistency probability based on the text alignment features and the alignment enhancement features, and outputs a release decision based on the comparison result of the persona consistency probability and a preset threshold.

2. The method of claim 1, wherein, The process of mapping each feature to a unified persona space yields text alignment features, cover alignment features, and image alignment features, specifically including: The text features extracted by the trainingable first linear projection layer are mapped to the target dimension to obtain the text alignment features; The visual features of the cover image obtained by feature extraction are mapped to the target dimension using a trainable second linear projection layer to obtain the cover alignment features; The second linear projection layer is used to map the visual features of each image in the image set, obtained by feature extraction, to the target dimension to obtain the image alignment features. The first linear projection layer and the second linear projection layer are trained based on cross-modal contrastive learning loss. The cross-modal contrastive learning loss is used to reduce the feature distance between the text alignment features and cover alignment features of the same content sample in the unified persona space, and to increase the feature distance between the text alignment features and cover alignment features of different content samples in the unified persona space.

3. The method of claim 1, wherein, The process of using the text alignment feature as the query, and the cover alignment feature and the accompanying image alignment feature as the key and value, respectively, and performing weighted aggregation through a cross-attention mechanism to output alignment enhancement features, specifically includes: The query is obtained by performing a linear transformation on the text alignment features; A linear transformation is performed on the combined feature formed by splicing the cover alignment feature and the image alignment feature to obtain the key and the value; Calculate the scaled dot product similarity between the query and the key, and process it with a normalized exponential function to obtain the attention weight; The values ​​are weighted and summed using the attention weights to generate the alignment enhancement feature.

4. The method of claim 1, wherein, The consistency generation model includes a feature forward computation link and a consistency discrimination head; The feature forward computation link is used to perform feature extraction, mapping to a unified persona space, and cross-attention mechanism processing on the preprocessed text, cover image, and illustration set to generate the text alignment feature and the alignment enhancement feature; The consistency discriminant head is used to calculate the cosine similarity between the text alignment feature and the alignment enhancement feature, and to map the cosine similarity to the persona consistency probability.

5. The method of claim 4, wherein, The consistency generation model is trained under supervision based on binary cross-entropy loss. The training sample library includes positive samples and negative samples. The positive samples are combinations of text and images published historically by the same account that satisfy the persona consistency. The negative samples are combinations of text and images formed by randomly combining the text of the account with the image sets of other accounts. The text and image combinations include text, cover image, and a set of accompanying images.

6. The method of claim 1, wherein, In the process of obtaining and preprocessing the text, cover image and illustration set and then performing feature extraction, the feature extraction is performed by a parallel feature extraction network, which includes a language sub-model and a visual sub-model. The language sub-model is trained based on masked language modeling loss and is used to extract text feature vectors from preprocessed text. The visual sub-model is trained based on contrastive learning loss and is used to extract visual feature vectors from the preprocessed cover image and the accompanying image set.

7. The method of claim 6, wherein, The language sub-model adopts a Transformer encoder structure, and the text features are obtained by pooling the output of the Transformer encoder. The visual sub-model adopts a Vision Transformer encoder structure, and the visual features are obtained by encoding the input image through the Vision Transformer encoder.

8. The method of claim 1, wherein, Obtain the text, cover image, and accompanying images, and preprocess them. The preprocessing includes: The text is subjected to noise removal and sequence length normalization. Each image in the cover image and accompanying image set is subjected to scale reshaping and pixel value normalization.

9. A persona style alignment system for multi-modal content, the system comprising: include: At least one processor; as well as A memory that is communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of computer programs, which are used to be invoked by a processor and to execute the method as described in any one of claims 1-8.

Images