Content generation data processing method and system based on multi-agent cooperation

Abstract

The invention discloses a content generation data processing method and system based on multi-agent collaboration, and the method comprises the steps: carrying out the semantic deconstruction of a multi-modal content demand inputted by a user through a task analysis agent, and generating a structured task description vector containing a theme dimension, a style element and an emotional tendency; generating a multi-modal content draft based on the structured task description vector; inputting the multi-modal content draft into a quality evaluation agent network, and generating a quality evaluation matrix containing semantic coherence scores and style matching degrees; and dynamically adjusting a content generation strategy through a decision coordination agent according to the quality evaluation matrix, optimizing an agent cooperation weight by adopting a reinforcement learning mechanism, and outputting a final optimized multi-modal content work. By utilizing the embodiment of the invention, high-quality and high-consistency parallel generation and deep fusion of multi-modal contents such as texts, images and audios can be realized, and the creativity and semantic coherence of final works and the matching degree with user requirements are improved.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, and in particular to a content generation data processing method and system based on multi-agent collaboration. Background Technology

[0002] In the current digital media field, multimodal content generation technology is receiving widespread attention. Existing methods typically employ a serial processing flow, first generating one modality of content (such as text), and then using it as a basis to generate other modalities of content (such as images and audio). This pipeline approach easily leads to problems such as weak correlation and semantic inconsistencies between different modal elements; for example, the generated images may not match the text description, or the audio atmosphere may not match the visual emotion. Furthermore, traditional single-model approaches often lack flexibility when dealing with complex creative needs that combine stylization and emotional expression, making it difficult to balance content quality with creative efficiency. Although existing technologies have attempted to integrate multiple generation modules, they generally lack an effective collaborative decision-making and global quality control mechanism, resulting in unsatisfactory overall quality of the final work and the degree of integration between different modalities. Summary of the Invention

[0003] The purpose of this invention is to provide a content generation data processing method and system based on multi-agent collaboration to overcome the shortcomings of the prior art. It can achieve high-quality, highly consistent parallel generation and deep fusion of multimodal content such as text, images, and audio, thereby improving the creativity, semantic coherence, and matching degree of the final work with user needs.

[0004] One embodiment of this application provides a content generation data processing method based on multi-agent collaboration, the method comprising:

[0005] The task parsing agent performs semantic deconstruction of the multimodal content requirements input by the user, generating a structured task description vector containing topic dimensions, style elements, and sentiment tendencies;

[0006] Based on the structured task description vector, candidate content fragments are generated in parallel by a cluster of content generation agents, and a cross-modal attention mechanism is used to perform spatiotemporal alignment and fusion of text, image and audio fragments to generate a multimodal content draft.

[0007] The multimodal content draft is input into the quality assessment agent network, and a quality assessment matrix containing semantic coherence score and style matching degree is generated through a multi-dimensional consistency detection algorithm and a creativity index quantification model.

[0008] Based on the quality assessment matrix, the content generation strategy is dynamically adjusted by the decision-coordinating agent, and the agent's collaborative weights are optimized using a reinforcement learning mechanism to output the final optimized multimodal content.

[0009] Optionally, the step of semantically deconstructing the multimodal content requirements input by the user through a task parsing agent to generate a structured task description vector containing topic dimensions, style elements, and sentiment tendencies includes:

[0010] The system receives user-provided text descriptions, reference images, and audio samples through a multimodal input interface, and uses a data normalization algorithm to unify the format and sampling rate of different modal data to generate standardized multimodal input data.

[0011] Semantic features are extracted from standardized multimodal input data. A pre-trained language model is used to parse the semantics of the text, a visual model is used to extract image style features, and an audio model is used to analyze emotional tone and generate a multimodal semantic feature set.

[0012] Semantic association analysis is performed based on multimodal semantic feature sets. A dependency graph between topics, styles and sentiments is constructed using graph neural networks to generate a semantic association graph.

[0013] The semantic association graph is encoded into a fixed-dimensional numerical vector. An embedding layer technique is used to map the topic dimension, style elements, and sentiment to a continuous vector space, ultimately generating a structured task description vector.

[0014] Optionally, based on the structured task description vector, candidate content fragments are generated in parallel by a cluster of content generation agents, and a cross-modal attention mechanism is used to perform spatiotemporal alignment and fusion of text, image, and audio fragments to generate a multimodal content draft, including:

[0015] The structured task description vector is distributed to the text generation agent, image generation agent, and audio generation agent. Each agent creates content fragments in parallel based on a dedicated generation model, generating an initial set of candidate content fragments.

[0016] Spatiotemporal feature encoding is performed on the initial candidate content fragment set. The temporal encoder is used to process the time series of audio fragments, the spatial encoder is used to process the visual layout of image fragments, and the text encoder is used to process the language sequence, generating a multimodal feature sequence.

[0017] Based on multimodal feature sequences, a cross-modal attention mechanism is applied to calculate the visual attention weights of text features and image features, and the temporal attention weights of text features and audio features, generating a cross-modal attention map.

[0018] The feature fusion process is guided by cross-modal attention maps. By integrating feature representations from different modalities through weighted summation, and decoding, an aligned multimodal content draft is generated.

[0019] Optionally, the step of inputting the multimodal content draft into the quality assessment agent network, and generating a quality assessment matrix including semantic coherence score and style matching degree through a multi-dimensional consistency detection algorithm and a creativity index quantification model, includes:

[0020] Feature separation is performed on the multimodal content draft to extract text semantic embeddings, image visual embeddings, and audio acoustic embeddings, generating separated modal feature vectors;

[0021] Multi-dimensional consistency detection is performed based on the separated modal feature vectors, calculating the text-image semantic consistency score, the text-audio sentiment consistency score, and the image-audio spatiotemporal synchronization score, and generating consistency detection results;

[0022] A creativity index quantification model is used to evaluate the novelty of content. By comparing the uniqueness and innovation index of content fragments with a pre-trained knowledge base, a creativity score is generated.

[0023] By integrating consistency test results and creativity scores, a multidimensional scoring matrix is ​​constructed and normalized to finally generate a quality assessment matrix that includes semantic coherence scores and style matching degree.

[0024] Optionally, the step of dynamically adjusting the content generation strategy through a decision-coordinating agent based on the quality assessment matrix, optimizing the agent's collaborative weights using a reinforcement learning mechanism, and outputting the final optimized multimodal content work includes:

[0025] Analyze the scores in the quality assessment matrix, identify content weaknesses and optimization directions using a decision tree algorithm, and generate strategy adjustment suggestions.

[0026] Based on policy adjustment suggestions, a reinforcement learning reward function is designed, with the quality assessment matrix score as the optimization objective. The reward signal for agent cooperation is defined, and a reinforcement learning environment configuration is generated.

[0027] The reinforcement learning training process is executed, and the collaborative weight parameters of the task parsing agent, content generation agent and quality assessment agent are updated through the policy gradient method to generate an optimized agent weight set.

[0028] The content generation process is re-executed using the optimized agent weight set, and the multimodal content draft is iteratively optimized until the quality threshold is met, and the final optimized multimodal content work is output.

[0029] Another embodiment of this application provides a content generation data processing system based on multi-agent collaboration, the system comprising:

[0030] The deconstruction module is used to semantically deconstruct the multimodal content requirements input by the user through the task parsing agent, and generate a structured task description vector containing topic dimensions, style elements and sentiment tendencies.

[0031] The fusion module is used to generate candidate content fragments in parallel through a cluster of content generation agents based on the structured task description vector, and to perform spatiotemporal alignment and fusion of text, image and audio fragments using a cross-modal attention mechanism to generate a multimodal content draft.

[0032] The generation module is used to input the multimodal content draft into the quality assessment agent network, and generate a quality assessment matrix containing semantic coherence score and style matching degree through a multi-dimensional consistency detection algorithm and a creativity index quantification model.

[0033] The optimization module is used to dynamically adjust the content generation strategy through the decision-coordination agent based on the quality assessment matrix, optimize the agent collaboration weights using a reinforcement learning mechanism, and output the final optimized multimodal content work.

[0034] Another embodiment of this application provides a storage medium storing a computer program, wherein the computer program is configured to execute the method described in any of the preceding claims when running.

[0035] Another embodiment of this application provides an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the method described in any of the preceding claims.

[0036] Compared with existing technologies, this invention provides a content generation data processing method based on multi-agent collaboration. It uses a task parsing agent to semantically deconstruct the multimodal content requirements input by the user, generating a structured task description vector containing topic dimensions, style elements, and sentiment tendencies. Based on the structured task description vector, a multimodal content draft is generated. The multimodal content draft is input into a quality assessment agent network to generate a quality assessment matrix containing semantic coherence scores and style matching degrees. Based on the quality assessment matrix, a decision-making coordination agent dynamically adjusts the content generation strategy, and a reinforcement learning mechanism is used to optimize the agent collaboration weights, outputting the final optimized multimodal content work. This enables high-quality, highly consistent parallel generation and deep fusion of multimodal content such as text, images, and audio, improving the creativity, semantic coherence, and matching degree of the final work to user needs. Attached Figure Description

[0037] Figure 1 A hardware structure block diagram of a computer terminal for a content generation data processing method based on multi-agent collaboration provided in an embodiment of the present invention;

[0038] Figure 2 A flowchart illustrating a content generation data processing method based on multi-agent collaboration provided in an embodiment of the present invention;

[0039] Figure 3 This is a schematic diagram of the structure of a content generation data processing system based on multi-agent collaboration, provided in an embodiment of the present invention. Detailed Implementation

[0040] The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0041] This invention first provides a content generation data processing method based on multi-agent collaboration, which can be applied to electronic devices, such as computer terminals, specifically ordinary computers.

[0042] The following detailed explanation uses a computer terminal as an example. Figure 1 This is a hardware structure block diagram of a computer terminal for a content generation data processing method based on multi-agent collaboration, provided as an embodiment of the present invention. Figure 1 As shown, the computer device includes a processor, memory, and network interface connected via a system bus, wherein the memory may include non-volatile storage media and internal memory.

[0043] Non-volatile storage media can store operating systems and computer programs. These computer programs include program instructions that, when executed, cause the processor to perform any content generation data processing method based on multi-agent cooperation.

[0044] The processor provides computing and control capabilities, supporting the operation of the entire computer device.

[0045] Internal memory provides an environment for the execution of computer programs in non-volatile storage media. When executed by a processor, the computer program enables the processor to perform any content generation data processing method based on multi-agent cooperation.

[0046] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 1 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0047] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0048] See Figure 2 The embodiments of the present invention provide a content generation data processing method based on multi-agent collaboration, which may include the following steps:

[0049] S201, through the task parsing intelligent agent, semantically deconstructs the multimodal content requirements input by the user, and generates a structured task description vector containing topic dimensions, style elements and sentiment tendencies;

[0050] Specifically, it can receive user-provided text descriptions, reference images, and audio samples through a multimodal input interface, and use a data normalization algorithm to unify the format and sampling rate of different modal data to generate standardized multimodal input data;

[0051] This step is the fundamental entry point for processing multimodal content requirements. Its core is to break down the format barriers between different modalities of data, transforming fragmented inputs into a unified standard dataset, providing a consistent data foundation for subsequent semantic parsing. The specific implementation method is as follows:

[0052] The multimodal input interface adopts a "modular adaptation + real-time verification" design, supporting parallel reception of three core modalities: text, image, and audio. The interface is compatible with mainstream data formats: text descriptions support TXT, DOCX, and rich text formats; reference images support JPG, PNG, and TIFF formats; and audio samples support WAV, MP3, and FLAC formats. The interface has a built-in format recognition module. When a user uploads data, it automatically detects the format type and triggers the corresponding parsing process. For example, when receiving a user-uploaded DOCX text titled "Warm Family Dinner," three JPG reference images (including table setting and interactive scenes), and one MP3 background audio clip (a light piano piece), the interface first verifies the file integrity (no corruption, no encryption) before extracting the core content.

[0053] The data normalization algorithm designs differentiated processing rules according to the characteristics of different modalities. The core goal is to unify the data dimensions, formats, and numerical ranges to ensure the compatibility of subsequent feature extraction. The normalization of text data adopts the strategy of "encoding unification + length standardization": First, all texts are converted to UTF-8 encoding to avoid garbled problems; then, through sentence segmentation and stop word removal (such as meaningless words like "de", "le", etc.), and then the text length is unified to 512 characters by truncation or padding (covering more than 95% of user input scenarios). If the user's text description is "The whole family had a dinner in the yard on the weekend. The sun was just right. Everyone was chatting and laughing with light music in the background." (a total of 38 characters), then the placeholder "[PAD]" is filled at the end to 512 characters; at the same time, the text is converted into an ID sequence, and each character is mapped to a unique integer ID in the predefined dictionary (such as "weekend" is mapped to 1025, "dinner" is mapped to 2341).

[0054] The normalization of image data focuses on the "resolution + color space + pixel range" unification: The resolution of all reference images is uniformly adjusted to 512×512 pixels by using the bilinear interpolation algorithm (taking into account both detail retention and computational efficiency). For example, for a high-definition dinner image uploaded by the user with a resolution of 1920×1080, the resolution is reduced to 512×512 after interpolation, and the pixel loss is controlled within 3%; the color space is converted from RGB to CIELab (which is closer to human visual perception), and the pixel values are normalized from 0 - 255 to the 0 - 1 interval. The calculation method is pixel normalization value = original pixel value / 255. For example, the RGB value of a warm-color pixel is (255, 200, 150), and the Lab space value after normalization is (89.2, 15.3, 32.1), and the pixel range is unified to 0 - 1.

[0055] The normalization of audio data focuses on "sampling rate + duration + amplitude": The audio sampling rate is uniformly set to 44.1kHz by using the resampling algorithm (the standard sampling rate for audio processing). If the original sampling rate of the piano piece uploaded by the user is 22kHz, it is increased to 44.1kHz through linear interpolation to ensure that the sound quality is not significantly distorted; the audio duration is unified to 30 seconds. For audio shorter than 30 seconds, silent segments are filled at the end, and for audio longer than 30 seconds, the core segment is intercepted (the 30 seconds with the highest volume are extracted through the energy detection algorithm); the audio amplitude is normalized to the [-1, 1] interval. The calculation method is amplitude normalization value = original amplitude value / maximum amplitude value. For example, the maximum amplitude of an audio segment is 32767 (16-bit audio), and the amplitude of a certain sampling point is 16384, and the normalized value is 0.5.

[0056] The generated standardized multimodal input data is stored in a structured manner according to "modal type-data identifier-standardized content". For example: "text modality, ID: T001, content: [1025,2341,...512 ID sequences]; image modality, ID: I001, content: 512×512×3 (Lab channel) 0-1 pixel matrix; audio modality, ID: A001, content: 44.1kHz, 30 seconds, [-1,1] amplitude audio sequence". The three types of data are associated through a unified session ID to ensure the relevance of subsequent processing.

[0057] Semantic features are extracted from standardized multimodal input data. A pre-trained language model is used to parse the semantics of the text, a visual model is used to extract image style features, and an audio model is used to analyze emotional tone and generate a multimodal semantic feature set.

[0058] This step is crucial for extracting core semantic information from the raw data. It involves using a dedicated pre-trained model to extract features from each modality, forming a multi-dimensional feature set to provide a basis for subsequent correlation analysis. The specific implementation method is as follows:

[0059] Text semantic extraction employs a pre-trained BERT-base model, which contains a 12-layer Transformer structure with a hidden layer dimension of 768, possessing powerful contextual semantic understanding capabilities. The model input is a sequence of standardized text IDs. An embedding layer converts these IDs into 768-dimensional word embedding vectors, followed by multiple layers of self-attention mechanisms to capture dependencies between words, ultimately outputting a global semantic feature vector (768 dimensions). For example, in the user text "A warm family dinner, the sun is shining brightly," the core semantic features extracted by the model include the topic keyword "family dinner" (corresponding dimension value of 0.89 in the feature vector), the scene description "sunshine" (0.72), and the emotional tendency "warm" (0.85), while outputting the semantic labels "life scene - family interaction - positive emotion." To improve feature accuracy, a masked language model (MLM) is used to predict and validate the core words in the text, ensuring that the semantic parsing error is less than 5%.

[0060] Image style feature extraction utilizes a pre-trained VGG-19 model, which, through 16 convolutional layers and 3 fully connected layers, effectively captures stylistic information such as texture, color, and composition of images. The model input is a standardized 512×512 Lab image matrix. The first 10 convolutional layers extract low-order style features (texture, edges), and the last 6 convolutional layers extract high-order style features (color tone, composition style). Finally, global average pooling yields a 2048-dimensional image style feature vector. For example, in a user-uploaded reference image of a dinner party, the model extracts low-level features including "the curved edges of the round table" (high value in the 120-150 dimension range, approximately 0.78) and "the regular arrangement texture of the tableware" (value in the 320-350 dimension range, 0.65); high-level features include "warm color scheme" (value in the 800-850 dimension range, 0.92) and "the compositional style of focusing on people in the foreground" (value in the 1500-1600 dimension range, 0.81). The model also outputs the style label "documentary style - warm color scheme - shallow depth of field." During feature extraction, batch normalization is used to accelerate convergence, ensuring a feature extraction speed of 2 images per second.

[0061] Audio sentiment and intonation analysis employs a pre-trained VGGish model, specifically designed for audio scenarios. The input is a standardized audio sequence, and the output is a 128-dimensional acoustic feature vector. Mel-frequency cepstral coefficients (MFCCs) are then used to extract sentiment-related features. First, the 30-second audio clip is segmented into 1200 frames with a stride of 10 milliseconds each. For each frame, a 40-dimensional MFCC feature (reflecting the spectral envelope of the audio and directly related to sentiment and intonation) is calculated. The MFCC features are then input into the VGGish model, which extracts acoustic features through eight convolutional layers. Finally, a 256-dimensional audio sentiment feature vector, fusing MFCCs and model features, is output. For example, when a user provides a lively piano piece, the model extracts features including "high frequency dominance (2000-4000Hz)" (feature dimension 50-80 range, value 0.88), "short note continuity (brisk rhythm)" (dimensional 120-150 range, value 0.91), and "stable volume (fluctuation amplitude ≤0.2)" (dimensional 200-220 range, value 0.76), corresponding to the emotional tags "cheerful-relaxed-pleasant," with an emotional classification accuracy of 92%.

[0062] The generated multimodal semantic feature set integrates feature vectors and label information from three modalities, organized in the format of "modality-feature vector-label-confidence". For example: "Text modality: 768-dimensional vector [0.89, 0.72, ...], label [family dinner, warm], confidence 0.96; Image modality: 2048-dimensional vector [0.78, 0.65, ...], label [warm color tone, documentary life], confidence 0.93; Audio modality: 256-dimensional vector [0.88, 0.91, ...], label [cheerful, relaxed], confidence 0.95". The confidence of each modality feature is calculated using the softmax probability output by the model to ensure the reliability of the features.

[0063] Semantic association analysis is performed based on multimodal semantic feature sets. A dependency graph between topics, styles and sentiments is constructed using graph neural networks to generate a semantic association graph.

[0064] This step is crucial for uncovering the intrinsic connections between multimodal features. It uses graph neural networks to link disparate features into a structured graph, clarifying the interactions between themes, styles, and emotions. The specific implementation is as follows:

[0065] The core of semantic association analysis is to identify three core elements: "topic, style, and sentiment" and the strength of association between these elements. First, core elements are extracted from the multimodal semantic feature set: topic elements come from text semantic tags and image content recognition (such as "family dinner", "yard scene", "sunny environment"); style elements come from image style tags and audio rhythm features (such as "warm color tone", "life documentary style", "upbeat rhythm"); and sentiment elements come from text sentiment tags and audio sentiment tags (such as "warm", "cheerful", "pleasant"). A total of 8 core elements are extracted as graph nodes, and each node is associated with a corresponding feature vector and confidence level.

[0066] The graph neural network uses a Graph Convolutional Network (GCN) to construct an undirected weighted graph as a dependency graph. The nodes of the graph represent eight core elements, and the weights of the edges represent the strength of the association between two elements. The association strength is calculated using cosine similarity, with the formula: Association Strength = (Feature Vector of Element A ・ Feature Vector of Element B) / (||Feature Vector of Element A|| × ||Feature Vector of Element B||). The value ranges from 0 to 1, with larger values ​​indicating a stronger association. For example, the cosine similarity of the feature vectors of "Family Dinner" (theme element) and "Warmth" (emotional element) is 0.92, so the association strength is set to 0.92; the cosine similarity of "Warm Color Tone" (style element) and "Warmth" (emotional element) is 0.87, so the association strength is set to 0.87; and the cosine similarity of "Upbeat Rhythm" (style element) and "Courtyard Scene" (theme element) is 0.45, so the association strength is set to 0.45, reflecting a weak association.

[0067] The training and inference process of GCN focuses on optimizing the accuracy of association strength calculation. The input is a node feature matrix (8 nodes × 256-dimensional features, the feature vector is the dimensionality reduction result of the original features of each element) and an adjacency matrix (8×8 matrix, the elements are association strengths). The node features are updated through two layers of graph convolutional layers. The activation function of each convolution is ReLU, and the learning rate is set to 0.001. The model converges after 50 iterations. During the inference phase, the model outputs an optimized adjacency matrix, correcting the deviation of the initial association strength. For example, the initial calculated association strength between "sunny environment" and "warm color tone" is 0.78, which is improved to 0.83 after GCN optimization, which is more in line with the actual semantic association between "sunny" and "warm color tone".

[0068] The generated semantic association graph is presented in the form of "node-edge-attribute". Node attributes include feature type (theme / style / emotion), feature vector, and confidence level. Edge attributes represent the optimized association strength. An example of the graph structure is as follows: the theme node "family dinner" (confidence 0.96) is connected to the emotion node "warmth" (0.95) through an edge of strength 0.92; the style node "warm color scheme" (0.93) is connected to "warmth" through an edge of strength 0.87; the style node "upbeat rhythm" (0.95) is connected to the emotion node "cheerful" (0.94) through an edge of strength 0.81; and the theme node "yard scene" (0.91) is connected to the style node "documentary style" (0.92) through an edge of strength 0.75. The graph is visualized as a network structure, with core elements (such as "family dinner" and "warmth") located at the center. Edges with high association strength are represented by thick lines, intuitively demonstrating the inherent associations of multimodal semantics.

[0069] The semantic association graph is encoded into a fixed-dimensional numerical vector. An embedding layer technique is used to map the topic dimension, style elements, and sentiment to a continuous vector space, ultimately generating a structured task description vector.

[0070] This step is crucial for transforming the structured graph into numerical vectors that the algorithm can process. It uses embedding layer technology to condense and standardize semantic information, providing unified task instructions for subsequent content generation agents. The specific implementation is as follows:

[0071] The semantic association graph encoding adopts a "node embedding + edge weight fusion" strategy. First, the eight core nodes in the graph are embedded and encoded. The Node2Vec algorithm is used to map each node into a 128-dimensional node embedding vector. This algorithm generates node sequences through random walks. Then, the Skip-gram model is used to learn the contextual representation of the nodes, ensuring that closely related nodes are closer in the vector space. For example, the cosine similarity of the node embedding vectors of "family dinner" and "warm" is 0.89, and the similarity of "warm color tone" and "warm" is 0.85, consistent with the association strength, reflecting the preservation of the graph's semantics.

[0072] Edge weight fusion is achieved through an attention mechanism, which sums the embedding vector of each node with the weights of all associated edges of that node to obtain the node feature vector that incorporates associated information. The formula is: Fusion feature vector = Node embedding vector × (1 + Σ associated edge weights) / (1 + Number of associated edges). For example, the "Warm" node has 3 associated edges (strengths of 0.92, 0.87, and 0.81), so the fusion feature vector = "Warm" node embedding × (1 + 0.92 + 0.87 + 0.81) / (1 + 3) = node embedding × 3.6 / 4 = node embedding × 0.9. This preserves the node's own features while incorporating associated information.

[0073] The embedding layer technique employs a three-layer fully connected network structure. The input is a 1024-dimensional vector resulting from the concatenation of eight fused feature vectors (each 128-dimensional). The first fully connected layer compresses the 1024 dimensions to 512 dimensions using ReLU activation. The second layer compresses to 256 dimensions using LeakyReLU activation (to avoid gradient vanishing). The third layer outputs a fixed-dimensional 128-dimensional vector, serving as the final output of the embedding layer. Training of the embedding layer aims to preserve semantic similarity using a contrastive loss function to ensure that semantically similar graphs are encoded with short vector distances. The training data consists of 100,000 labeled graph-vector pairing samples. After training, the semantic retention rate of the encoded vectors reaches 94%.

[0074] The structured task description vector is a 128-dimensional continuous numerical vector. The vector is divided into fixed intervals based on "theme dimension (first 40 dimensions) - style elements (middle 48 dimensions) - sentiment tendency (last 40 dimensions)," with each interval's numerical distribution corresponding to the intensity of different elements. For example, in the first 40 dimensions of the theme dimension, "family dinner" corresponds to a value of 0.92, and "yard scene" corresponds to a value of 0.81, reflecting the primary and secondary relationships of the theme. In the middle 48 dimensions of the style elements, "warm colors" corresponds to a value of 0.93, and "light rhythm" corresponds to 0.88, reflecting the core characteristics of the style. In the last 40 dimensions of the sentiment tendency, "warmth" corresponds to 0.95, and "cheerfulness" corresponds to 0.91, clearly indicating the dominant direction of the emotion. The numerical range of the vector is normalized to the 0-1 interval for easy subsequent agent parsing. For example, the first 5 values ​​of the complete vector are [0.92, 0.81, 0.75, 0.68, 0.52], corresponding to the theme intensities of "family dinner," "yard scene," "sunny environment," "tableware arrangement," and "character interaction," respectively.

[0075] The generated structured task description vector is accompanied by a vector description document, which marks the core elements, numerical meanings, and confidence levels of each dimension. For example, "Dimensions 1-5: Thematic element strength, Dimension 1 = Family dinner (0.92, confidence level 0.96); Dimensions 41-60: Style element strength, Dimension 41 = Warm color tone (0.93, confidence level 0.93); Dimensions 89-100: Emotional element strength, Dimension 89 = Warmth (0.95, confidence level 0.95)", ensuring that the content generation intelligent agent can accurately analyze task requirements and provide clear guidance for subsequent parallel creation.

[0076] S202, based on the structured task description vector, candidate content fragments are generated in parallel by a content generation agent cluster, and a cross-modal attention mechanism is used to perform spatiotemporal alignment and fusion of text, image and audio fragments to generate a multimodal content draft;

[0077] Specifically, the structured task description vector can be distributed to text generation agents, image generation agents, and audio generation agents. Each agent can create content fragments in parallel based on a dedicated generation model to generate an initial set of candidate content fragments.

[0078] This step is the core initiation stage of multimodal content creation. Its core is to enable specialized intelligent agents to accurately respond to needs and quickly produce single-modal fragments that conform to the theme, style, and emotion through task distribution and parallel generation mechanisms. This provides basic materials for subsequent fusion. The specific implementation method is as follows:

[0079] The distribution of structured task description vectors adopts an "agent registration-demand matching-load balancing" mechanism. The system has a built-in agent management module that records the online status, computing load (current number of tasks / maximum number of tasks), and areas of expertise of three types of agents: text, image, and audio. For example, when receiving a 128-dimensional task vector for "warm family dinner" (theme dimension "family dinner" intensity 0.92, style dimension "warm color" 0.93, emotion dimension "warm" 0.95), the management module prioritizes matching agents with a load below 50% and those skilled in creating "life scenes." The vector is then fully distributed via the RPC (Remote Procedure Call) protocol, with transmission latency controlled within 50ms to ensure parallel startup.

[0080] Each agent's dedicated generative model is customized and optimized for modal characteristics, with parameter configurations deeply bound to task vectors:

[0081] The text generation agent uses a fine-tuned GPT-4o model, which includes a 12-layer decoder, 1024 hidden layer dimensions, and 16 attention heads. It was fine-tuned with 100,000 samples for "life scene descriptions." The input consists of a 128-dimensional task vector and text modal labels. The generation parameters are set to max_length=512 (maximum 512 characters per segment), temperature=0.7 (balancing determinism and diversity), and top_p=0.9 (selecting high-probability words). The generated text segments focus on "detailed scenes of family gatherings," such as: "In the courtyard on a weekend afternoon, a beige tablecloth covered the wooden table. Mom brought out freshly steamed crabs, Dad poured rice wine for Grandpa, and my five-year-old niece ran over with a juice cup. Sunlight filtered through the sycamore leaves onto everyone's smiling faces, and the aroma of food wafted in the wind." The semantics align with the emotion of "warmth," and the details echo the theme of "family gatherings."

[0082] The image generation agent uses the MidJourneyV6 model, with style weights optimized for a "warm-toned documentary style". The input is a task vector and an image style label. The generation parameters are set to a resolution of 1024×1024, a stylization coefficient of 0.8 (close to the reference style), and a Chaos value of 10 (low randomness). Three candidate images are generated in parallel: the first focuses on a close-up of the dining table (warm yellow light, crabs and a yellow wine glass), the second shows people interacting (father and grandfather clinking glasses, and a little niece snuggling against her mother), and the third is a panoramic view of the yard (a dining table and smiling faces under a sycamore tree). All three images meet the style requirements of "warm-toned + documentary style".

[0083] The audio generation agent uses the MusicGen-large model, fine-tuned with the "light and warm piano music" dataset. The input is the task vector and audio emotion label. The generation parameters are set to a duration of 30 seconds, a sampling rate of 44.1kHz, a BPM (beats per minute) of 85 (light and not hurried), and an instrument combination of "piano + violin" (to enhance the warmth). The generated audio clips are mainly in C major, with a light piano introduction at the beginning, a violin accompaniment in the middle, and a gradual fading at the end, matching the emotional tendency of "joyful and pleasant".

[0084] The generated initial candidate content fragment set is structured according to "modality-fragment ID-generation parameters-confidence score", and contains 1 text fragment (ID: T001, confidence score 0.94), 3 images (ID: I001-I003, confidence scores 0.92, 0.95 and 0.91 respectively), and 1 audio fragment (ID: A001, confidence score 0.93). The confidence score is calculated through the generation probability output by the model to ensure the reliability of the matching between each fragment and the task requirements, and to provide high-quality input for the subsequent encoding stage.

[0085] Spatiotemporal feature encoding is performed on the initial candidate content fragment set. The temporal encoder is used to process the time series of audio fragments, the spatial encoder is used to process the visual layout of image fragments, and the text encoder is used to process the language sequence, generating a multimodal feature sequence.

[0086] This step is crucial for transforming raw content fragments into machine-recognizable features. The core of this process involves extracting key spatiotemporal information from each modality using a dedicated encoder, forming standardized feature sequences that provide a unified feature carrier for cross-modal association. The specific implementation method is as follows:

[0087] The temporal encoder is designed specifically for audio time series and uses the TransformerEncoder architecture (6-layer encoder, 512 hidden layer dimensions, 8 attention heads). The input is the Mel frequency cepstral coefficient (MFCC) features of the audio segment. First, the 30-second audio is divided into 2999 frames with a frame length of 25ms and a step length of 10ms. 40-dimensional MFCC features are extracted from each frame to form a 2999×40 two-dimensional feature matrix. The 40-dimensional MFCC features are mapped to a 512-dimensional vector through an embedding layer, and position encoding (reflecting the temporal order of the frames, position encoding value = sin(position / (10000^(2i / 512))), where i is the feature dimension) is added. The vector is then input into the TransformerEncoder layer, which captures the inter-frame dependencies through a self-attention mechanism (such as the association between the piano sounds in the first 100 frames and the violin sounds in the last 500 frames). Finally, a 2999×512 audio temporal feature sequence is output. The features at each time step correspond to one frame of the audio. For example, in the feature vector of the 1000th frame (corresponding to 10 seconds), the dimension value of the "high-frequency piano component" is 0.82, reflecting the audio characteristics at that moment.

[0088] The spatial encoder extracts features for the visual layout of the image, employing a SwingTransformer model (4 stages, each containing 2 Transformer blocks). The input is a 1024×1024 RGB image. First, the image is divided into 16×16 image patches, totaling 4096 patches. The 3-channel pixel values ​​of each patch are flattened into a 768-dimensional vector. Linear projection maps the 768 dimensions to 1024 dimensions, and relative position encoding (reflecting the spatial relationship of the patches, such as the positional difference between the center patch of the dining table and the edge tableware patches) is added. After processing through 4 stages (each layer captures local and global spatial relationships through window attention and shifted window attention), the final output is a 4096×1024 image spatial feature sequence. Each feature corresponds to a patch. For example, in the feature vector of the center patch of the dining table, the "warm yellow tone" dimension value is 0.91, and the "circular outline" dimension value is 0.85, accurately reflecting the core information of the visual layout.

[0089] The text encoder focuses on the semantic associations of language sequences, employing a BERT-based model (12 Transformer layers, 768 hidden layer dimensions, 12 attention heads). The input is a sequence of IDs for text segments (512 IDs). The IDs are mapped to 768-dimensional vectors through a word embedding layer, incorporating segment encoding (identifying single text segments) and positional encoding (reflecting word order). The Transformer layer's self-attention mechanism captures inter-word dependencies (e.g., the verb-object association between "crab" and "steamed" and "served"). The final output is a 512×768 text feature sequence, with each feature corresponding to a single word in the text. For example, in the feature vector corresponding to "warmth," the "positive emotion" dimension value is 0.95, and the "interpersonal relationship" dimension value is 0.92, closely aligning with the semantic core.

[0090] The generated multimodal feature sequences contain three types of subsequences, labeled according to "modality-sequence length-feature dimension": audio temporal feature sequence (2999×512), image spatial feature sequence (4096×1024), and text feature sequence (512×768). To facilitate subsequent attention calculation, the feature dimension of the image and audio sequences is uniformly adjusted to 768 (consistent with the text features) through linear projection. The image sequence is transformed from 4096×1024 to 1024×768 through global average pooling, and the audio sequence is transformed from 2999×512 to 1024×768 through downsampling. Finally, standardized feature sequences with a length of 1024 and a dimension of 768 are formed, ensuring cross-modal computation compatibility.

[0091] Based on multimodal feature sequences, a cross-modal attention mechanism is applied to calculate the visual attention weights of text features and image features, and the temporal attention weights of text features and audio features, generating a cross-modal attention map.

[0092] This step is the core of establishing multimodal feature associations. It quantifies the association strength of different modal features through an attention mechanism, clarifies the matching relationship between "text-image" and "text-audio", and provides precise guidance for feature fusion. The specific implementation method is as follows:

[0093] The cross-modal attention mechanism adopts the ScaledDot-ProductAttention architecture. The core logic is to obtain the attention weight by calculating the similarity of features of different modalities. The larger the weight value, the stronger the relationship between the two features. The formula is Attention(Q,K,V)=Softmax(QK^T / √dk)V, where Q (query), K (key), and V (value) are the feature vectors of different modalities, dk is the dimension of K (768), and √dk (27.7) is the scaling factor used to avoid the Softmax gradient from disappearing due to the large value of QK^T.

[0094] The calculation of text-image visual attention weights uses text features as Q and image features as K and V, focusing on the "matching of text descriptions with image visual elements." For example, the Q vector (768 dimensions) corresponding to "a wooden dining table covered with a beige tablecloth" in the text feature sequence is multiplied by the K vectors of all 1024 features in the image feature sequence. After dividing by 27.7 and normalizing using Softmax, 1024 attention weight values ​​are obtained. The weight corresponding to the "dining table patch" is 0.89, the weight corresponding to the "person patch" is 0.05, and the weight corresponding to the "background patch" is 0.06, clearly demonstrating the strong correlation between text semantics and image visual elements. By traversing the 1024 Q vectors of the text feature sequence, a 1024×1024 visual attention weight matrix is ​​finally generated. The matrix element values ​​range from 0 to 1, with rows representing text feature positions and columns representing image feature positions.

[0095] The text-audio temporal attention weight calculation uses text features as Q and audio features as K and V, with the core being "matching the sentiment and rhythm of text sentences with audio time segments." Text features are segmented into 12 semantic units (e.g., "in the yard on a weekend afternoon," "Mom brought over freshly steamed crabs"), with each semantic unit corresponding to 85 text feature positions (1024 / 12≈85). Audio features are segmented into 12 segments (approximately 2.5 seconds each), with each segment corresponding to 85 audio feature positions (1024 / 12≈85). The similarity between the Q vector of each text semantic unit and the K vector of the audio segment is calculated. For example, the Q vector corresponding to "my little niece ran over with a juice cup" has the highest similarity to the K vector of the upbeat 5th segment (10-12.5 seconds) in the audio, with a weight of 0.92, while the weight is only 0.15 with the slower-paced 10th segment, achieving alignment between text semantics and audio temporal sequence. Finally, a 1024×1024 temporal attention weight matrix is ​​generated, where rows represent text feature positions and columns represent audio feature positions, reflecting the correlation strength in the time dimension.

[0096] The generated cross-modal attention map comprises two heatmaps: a visual heatmap and a temporal heatmap. Weights are represented by grayscale values ​​from 0 to 255 (255 for weight 1, 0 for weight 0). In the visual attention heatmap, the regions corresponding to the text "dining table" and the image "dining table patch" are dark black (grayscale value 248), while other regions are light gray (grayscale value ≤ 50). In the temporal attention heatmap, the regions corresponding to the text "joyful interaction" and the audio segment "upbeat rhythm" are dark black (grayscale value 250), visually demonstrating the focus of cross-modal feature association. The attention map also stores a numerical weight matrix, providing precise weighting criteria for subsequent feature fusion.

[0097] The feature fusion process is guided by cross-modal attention maps. By integrating feature representations from different modalities through weighted summation, and decoding, an aligned multimodal content draft is generated.

[0098] This step is the final stage of multimodal content integration. Its core is to achieve precise feature fusion based on attention weights, and then generate a spatiotemporally aligned and semantically consistent content draft through decoding, completing the transformation from features to content. The specific implementation method is as follows:

[0099] Feature fusion employs a "two-stage weighted integration" strategy. The first stage completes feature fusion for text-image and text-audio features separately. The second stage integrates the two types of fused features to generate the final multimodal features. Text-image feature fusion is based on text features, summing the vector at each text feature location with the image feature sequence according to visual attention weights. The formula is: Fusion Feature 1 = Text Feature Vector + Σ(Image Feature Vector × Visual Attention Weight). For example, the text "wooden dining table" feature vector (768 dimensions) is superimposed with the product of all image feature vectors and their corresponding weights (dining table patch weight 0.89, others ≤ 0.06). After fusion, the "dining table visual attribute" dimension value increases to 0.92, preserving both text semantics and incorporating image visual information. Text-audio feature fusion uses the same logic, with the formula: Fusion Feature 2 = Text Feature Vector + Σ(Audio Feature Vector × Temporal Attention Weight). The text "cheerful interaction" feature vector is superimposed with the weight of the upbeat audio segment (0.92), resulting in a "emotional rhythm" dimension value of 0.91 after fusion, reflecting the emotional impact of the audio. In the second stage, the fused feature 1 and fused feature 2 are summed with a weight of 5:5 to generate a final multimodal feature sequence of 1024×768. This sequence simultaneously contains text semantics, image vision, audio emotion and spatiotemporal correlation information.

[0100] Feature decoding employs dedicated decoders for different modalities, and the decoding process synchronously references the spatiotemporal correlation of the attention map to ensure output alignment.

[0101] The text decoder uses the GPT-4o decoder module (12-layer Transformer, 768 hidden layer dimensions). The input is the final multimodal feature sequence. The decoding parameters are set to temperature=0.5 (to improve text coherence) and repetition_penalty=1.2 (to avoid repetition). The generated text supplements the visual and audio related details on the basis of the initial fragment. For example: "In the courtyard on a weekend afternoon, the sycamore leaves sifted out warm yellow light. The wooden dining table was covered with a beige tablecloth. Mom brought out freshly steamed crabs, their bright red shells reflecting her smiling face. Dad was pouring yellow wine for Grandpa. My five-year-old niece ran over with an orange juice cup. Her laughter and the light piano music drifted away. The wind was filled with the aroma of food." The text echoes the "warm yellow light" in the image and the "piano music" in the audio, making the semantics more coherent.

[0102] The image decoder uses the VQ-GAN model, with the input being the image association features extracted from the fusion features. Combined with the focal region of the visual attention map, it optimizes and fuses the initial three candidate images, preserving the details of the "dining table close-up" and the scene of "human interaction," generating a 1024×1024 fused image: the center of the image is a wooden dining table covered with a beige tablecloth, with crabs and yellow wine cups neatly arranged. On the right, the father and grandfather are clinking glasses, and on the left, the mother is holding her little niece holding a juice cup. The background is a courtyard with sycamore leaves fluttering in the wind, and warm yellow light covers the entire image, perfectly matching the text description.

[0103] The audio decoder uses the WaveRNN model, with audio association features from the fusion features as input. Referring to the alignment relationship of the temporal attention map, the rhythm of the initial audio is adjusted: at 10-12.5 seconds corresponding to the text "my little niece ran over", the violin volume is increased to enhance the cheerful feeling; at 15-18 seconds corresponding to "laughter", a slight human laughter sound effect (extracted from the audio material library and blended with the piano sound) is added to generate 30 seconds of optimized audio, with the rhythm completely synchronized with the emotional changes of the text scene.

[0104] The generated multimodal content draft comprises a three-in-one structure of text, image, and audio: a coherent 380-character text describing a complete family dinner scene; a warm-toned image that closely matches the text details; and a 30-second audio clip featuring a piano melody accompanied by violin and laughter sound effects, conveying a warm and cheerful mood. The draft's spatiotemporal alignment was verified: the text "running with a juice glass" corresponds to the dynamic posture of the niece in the image, and to the rhythmic climax of the audio from 10-12.5 seconds; the text "clinking glasses" corresponds to the actions of the father and grandfather in the image, and to the volume increase of the audio from 18-20 seconds, achieving comprehensive synergy across semantics, vision, and hearing. The draft also includes a fusion log, recording the application process of attention weights and decoding parameters, providing a traceable basis for subsequent quality assessment.

[0105] S203, the multimodal content draft is input into the quality assessment agent network, and a quality assessment matrix containing semantic coherence score and style matching degree is generated through a multi-dimensional consistency detection algorithm and a creativity index quantification model.

[0106] Specifically, feature separation can be performed on multimodal content drafts to extract text semantic embeddings, image visual embeddings, and audio acoustic embeddings, generating separated modal feature vectors;

[0107] This step is the foundational data processing stage for quality assessment. Its core is to extract the key features of each single modality from the fused multimodal content, and then extract high-dimensional embedding vectors using a specialized model. This provides a precise feature carrier for subsequent consistency detection and creativity assessment. The specific implementation method is as follows:

[0108] The multi-modal content draft exists in the form of a trinity of "text-image-audio". Taking the theme of "warm family dinner" as an example, it includes 380 characters of descriptive text (such as "In the yard on a weekend afternoon, the sycamore leaves filter down the warm yellow light, and the wooden dining table is covered with a milky white tablecloth"), 1 warm-toned image of 1024×1024 (focusing on the interaction between the dining table and people), and 1 piece of 30-second piano and violin ensemble audio (with a brisk and warm tone). Feature separation adopts the process of "modal recognition-exclusive extraction-dimensional unification" to ensure that each modal feature not only retains its own attributes but also has comparability.

[0109] For text semantic embedding extraction, a pre-trained BERT-base model is selected. This model contains 12 layers of Transformer structures, with a hidden layer dimension of 768 and 12 attention heads, and is fine-tuned and optimized for life-scene texts. First, the text is segmented into 392 tokens (including start and end symbols [CLS] and separator [SEP]) at the character level and converted into a corresponding sequence of vocabulary IDs; through the word embedding layer, the IDs are mapped to 768-dimensional vectors, and position encoding is added (reflecting the order of tokens in the text, such as the position relationship between "sycamore leaves" and "in the yard"); the self-attention mechanism of the Transformer layer is used to capture semantic associations (such as the scene association between "warm yellow light" and "afternoon"), and finally, the output vector corresponding to the [CLS] token is extracted as the text global semantic embedding, with a dimension of 768. In the example, the 120th to the 150th dimensions of the text semantic embedding correspond to the "warm-toned scene" feature, and the values are concentrated between 0.75 and 0.88. The 200th to the 230th dimensions correspond to the "family interaction" feature, and the values are 0.82 to 0.91, accurately reflecting the core semantics of the text.

[0110] For image visual embedding extraction, a ResNet-50 model is used. This model contains 50 layers of convolutional and pooling layers and can effectively capture visual features such as the texture, color, and object contours of the image. First, the 1024×1024 image is Resized to 224×224 (to adapt to the model input) and normalized (the pixel values are subtracted by the mean [123.68, 116.77, 103.93] and divided by the standard deviation [58.39, 57.12, 57.37]); after inputting into the model, low-level features (such as the milky white texture of the tablecloth and the edge contour of the sycamore leaves) and high-level features (such as the scene layout of people interaction) are extracted through the first 49 convolutional layers, and finally, the output of the 50th global average pooling layer is taken as the visual embedding vector, with a dimension of 2048. For example, the values of the feature dimensions (the 800th to the 850th dimensions) corresponding to "warm yellow light" in the image are 0.89 to 0.94, and the values of the dimensions corresponding to "round wooden dining table" (the 1200th to the 1250th dimensions) are 0.81 to 0.87, which highly match the visual elements described in the text.

[0111] The audio acoustic embedding extraction uses the VGGish model, specifically designed for audio feature extraction, and includes 8 convolutional layers and 2 fully connected layers. First, the 30-second audio is divided into 2999 frames with a 25ms frame length and a 10ms stride. 40-dimensional Mel-frequency cepstral coefficients (MFCCs) are extracted from each frame, forming a 2999×40 feature matrix. After being input into the model, the convolutional layers capture the audio's spectral features (such as the high-frequency components of a piano and the mid-frequency overtones of a violin). The fully connected layers then output a 128-dimensional acoustic embedding vector, which incorporates the audio's emotional and rhythmic information. In the example, the feature dimensions (30-50) for the audio "light piano melody" are 0.85-0.92, and the feature dimensions (70-90) for "violin accompaniment" are 0.78-0.83, consistent with the emotional tendency of "warm and cheerful."

[0112] The generated feature vectors of the separated modalities are stored in a structured manner according to "modality-feature ID-dimension-confidence". They are text semantic embedding (ID: TE001, 768 dimensions, confidence 0.94), image visual embedding (ID: VE001, 2048 dimensions, confidence 0.93), and audio acoustic embedding (ID: AE001, 128 dimensions, confidence 0.92). The confidence is calculated by the probability output of the features extracted by the model to ensure the reliability of the features.

[0113] Multi-dimensional consistency detection is performed based on the separated modal feature vectors, calculating the text-image semantic consistency score, the text-audio sentiment consistency score, and the image-audio spatiotemporal synchronization score, and generating consistency detection results;

[0114] This step is the core of quality assessment. By quantifying the degree of matching between different modalities, it determines whether multimodal content is "semantically consistent, sentiment-coordinated, and spatiotemporally aligned." The specific implementation method is as follows:

[0115] Multi-dimensional consistency detection employs a dual logic of "feature matching + label verification." First, it calculates similarity using feature vectors, and then combines this with manually labeled semantic / sentiment tags for verification, ensuring accurate scoring. The detection metrics are divided into three categories, each using a scoring range of 0-1. Higher scores indicate stronger consistency, while scores below 0.6 are considered "inconsistent."

[0116] The text-image semantic consistency score is calculated based on the semantic matching between the text description and the image content. It employs a weighted calculation of "feature cosine similarity + semantic label overlap," with weights of 0.7 and 0.3 respectively. The formula is: Semantic consistency score = 0.7 × cos(text embedding, image embedding projection) + 0.3 × (number of overlapping labels / total number of labels). First, the 2048-dimensional image visual embedding is reduced to 768 dimensions (consistent with the text semantic embedding dimension) through linear projection. The cosine similarity between the two is calculated (range -1 to 1, normalized to 0 to 1). Then, semantic labels are extracted from the text (e.g., "wooden dining table, warm yellow light, sycamore leaves, human interaction"), and visual labels are extracted from the image (identified using the YOLOv8 object detection model, e.g., "wooden dining table, warm color tone, sycamore leaves, 3 people interacting"), and the number of overlapping labels is counted. In the example, the cosine similarity between the text and image embeddings is 0.86 (after normalization), the number of overlapping labels is 3 ("wooden dining table, warm yellow light, human interaction"), and the total number of labels is 4. Therefore, the semantic consistency score = 0.7 × 0.86 + 0.3 × (3 / 4) = 0.602 + 0.225 = 0.827, which is judged as "highly consistent".

[0117] The core of text-audio sentiment consistency scoring is the "synergy between text sentiment tendency and audio sentiment tone." First, the sentiment of both is quantified into a vector of a unified dimension, and then the vector distance is calculated. Text sentiment is extracted through the sentiment analysis layer of the BERT model, outputting a 5-dimensional sentiment vector (corresponding to "pleasure, sadness, anger, calmness, and surprise"). The sentiment vector of the example text is [0.92, 0.03, 0.01, 0.02, 0.02], with "pleasure" as the dominant sentiment. Audio sentiment is extracted using the librosa library to extract sentiment-related features (such as BPM, spectral centroid, and volume fluctuation), which are input into the sentiment classification model to output a sentiment vector of the same dimension. The sentiment vector of the example audio is [0.89, 0.02, 0.01, 0.05, 0.03], with "pleasure" as the dominant sentiment. Emotional consistency score is calculated using "1 - Euclidean distance / √5" (√5 is the maximum possible distance between 5-dimensional vectors). In the example, the Euclidean distance between the two vectors is √[(0.92-0.89)²+(0.03-0.02)²+...]≈0.032, and the score is 1-0.032 / 2.236≈0.986, which is judged as "completely congruent".

[0118] The image-audio spatiotemporal synchronicity score targets the alignment between audio temporal changes and image visual focus. A 30-second audio clip is divided into 10 3-second segments, corresponding to 10 visual regions of the image (divided by attention heatmaps, such as the central figure area, the dining table area, and the background sycamore area). The Pearson correlation coefficient (ranging from -1 to 1, normalized to 0 to 1) is calculated between the acoustic activity (quantified by volume and spectral change rate, higher values ​​indicate greater activity) of each audio segment and the visual salience (calculated using an image saliency model, higher values ​​indicate greater prominence) of the corresponding visual region. In the example, the 5th audio segment (12-15 seconds, with violin accompaniment added, activity 0.85) corresponds to the central figure interaction area of ​​the image (salience 0.91), and the 8th segment (21-24 seconds, with decreased volume, activity 0.32) corresponds to the background sycamore area (salience 0.28), with a correlation coefficient of 0.84. After normalization, the spatiotemporal synchronicity score is 0.84, which is considered "good alignment."

[0119] The generated consistency test results are presented in the format of "Indicator Name-Score-Judgment Result". Specifically, the text-image semantic consistency score is 0.827 (highly consistent), the text-audio emotional consistency score is 0.986 (fully consistent), and the image-audio spatiotemporal synchronization score is 0.84 (good alignment). All three scores are higher than 0.6, indicating excellent overall consistency performance.

[0120] A creativity index quantification model is used to evaluate the novelty of content. By comparing the uniqueness and innovation index of content fragments with a pre-trained knowledge base, a creativity score is generated.

[0121] This step focuses on "whether the content is novel and unique." By comparing it with a large-scale knowledge base, the originality of the content is quantified to avoid generating homogenized content. The specific implementation method is as follows:

[0122] The creativity index quantification model adopts a "knowledge base comparison + combinatorial novelty assessment" architecture. The pre-trained knowledge base contains 3 million multimodal content items from daily life scenarios (1 million texts, 1 million images, and 1 million audio clips), stored in categories of "theme-style-emotion," and indexed for rapid retrieval. The assessment index is divided into two categories: "uniqueness" and "innovation," both using a 0-1 rating system. The final creativity score is obtained by weighted summation, with weights of 0.5 and 0.5 respectively.

[0123] The uniqueness evaluation focuses on "the degree of difference between unimodal content and the knowledge base" and is calculated separately for text, images, and audio. The text uniqueness is statistically analyzed through the repetition rate of N-grams (2-grams and 3-grams). 2-gram phrases (such as "filtered through the parasol leaves" and "warm yellow light") and 3-gram phrases (such as "the wooden dining table is covered with beige") are extracted from the text and compared with the text library of the same theme "family dinner" in the knowledge base to count the proportion of non-repeated phrases. A total of 25 2-grams and 18 3-grams are extracted from the example text, of which 18 and 13 respectively do not appear in the knowledge base. Text uniqueness = (18 / 25 + 13 / 18) / 2 ≈ (0.72 + 0.722) / 2 ≈ 0.721. Image uniqueness is compared through visual feature hash values. The visual embedding of the image is converted into a 64-bit hash value and compared with the image hash library of the same style "warm-toned life documentary style" in the knowledge base to calculate the Hamming distance (the number of different bits). For example, the "mottled light and shadow formed by the warm yellow light passing through the parasol leaves on the tablecloth" in the image is rare in the knowledge base, and the Hamming distance from the most similar image is 32. Image uniqueness = 32 / 64 = 0.5. Audio uniqueness is calculated through spectral feature similarity. The acoustic embedding of the audio is compared with the audio library of the same emotion "warm and cheerful" in the knowledge base. The cosine similarity of the most similar audio is 0.65. Audio uniqueness = 1 - 0.65 = 0.35. The average value of unimodal uniqueness = (0.721 + 0.5 + 0.35) / 3 ≈ 0.524.

[0124] The innovation evaluation focuses on "the novelty of the combination method of multimodal content". The core is to judge whether the combination of "text-image-audio" is a rare and reasonable combination in the knowledge base. First, the core feature tags of each modality are extracted: text tags "parasol leaf light and shadow + family dinner", image tags "warm yellow mottled light and shadow + wooden dining table", audio tags "piano + violin + slight vocal laughter", and a combined tag set is constructed; the multimodal content containing this combined tag is retrieved in the knowledge base, and the number of retrieval results is counted. In the example, the number of retrieval results is only 8 (a total of 100,000 combinations of the same theme in the knowledge base). Innovation = 1 - (8 / 100000) = 0.9992, approximately 1. To avoid misjudging "unreasonable combinations" as innovative, a rationality check is required. It is verified through a pre-trained multimodal combination judgment model (accuracy 98%). The rationality score of the example combination is 0.96. Therefore, innovation = 0.9992 × 0.96 ≈ 0.959.

[0125] The creativity score = 0.5 × average value of unimodal uniqueness + 0.5 × innovation = 0.5 × 0.524 + 0.5 × 0.959 ≈ 0.7415. The score is higher than 0.7, so it is judged as "excellent novelty", which not only avoids homogenization but also ensures the rationality of the combination.

[0126] By integrating consistency detection results and creativity scores, a multidimensional scoring matrix is ​​constructed and normalized to finally generate a quality assessment matrix that includes semantic coherence scores and style matching degree.

[0127] This step is the final step in the quality assessment. It integrates various scores into a structured matrix, eliminates dimensional differences through normalization, and provides a clear quality basis for subsequent decisions. The specific implementation method is as follows:

[0128] In addition to the consistency test (3 items) and creativity score (1 item) mentioned above, the integrated scoring dimensions include two core indicators: "textual semantic coherence" and "style matching degree", forming 6 evaluation dimensions that comprehensively cover the quality requirements of multimodal content. Text semantic coherence was calculated using N-gram perplexity, with the GPT-2 model used to evaluate perplexity. Lower perplexity indicates stronger coherence. The example text had a perplexity of 18.5 (the average perplexity for everyday life scenarios was 35), and the normalized score was 0.5 (formula: 1 - (perplexity - 5) / (50 - 5), where 5 is the lowest perplexity and 50 is the highest; substituting 18.5 gives 1 - 13.5 / 45 = 0.7). Style matching was evaluated to assess the degree of matching between each modality and the task requirement of "warm-toned documentary style + warm and cheerful". The style matching scores for text, image, and audio were 0.93, 0.96, and 0.91, respectively, with a mean of 0.933.

[0129] Construct an initial multidimensional scoring matrix. Rows represent evaluation dimensions, and columns represent "unimodal / overall". Specific dimensions include: text semantic coherence, text-image semantic consistency, text-audio sentiment consistency, image-audio spatiotemporal synchronicity, creativity score, and style fit, totaling 6 rows. Columns include text, image, audio, and multimodal overall, totaling 4 columns. An initial scoring example is as follows: Text semantic coherence (text column 0.7, other columns none), text-image semantic consistency (overall column 0.827), text-audio sentiment consistency (overall column 0.986), image-audio spatiotemporal synchronicity (overall column 0.84), creativity score (overall column 0.7415), style fit (text 0.93, image 0.96, audio 0.91, overall 0.933). Items with no data are filled with "-".

[0130] Normalization was performed using Min-Max standardization, mapping all scores to the 0-1 range. The formula is: Normalized Score = (Original Score - Minimum Dimension) / (Maximum Dimension - Minimum Dimension). The minimum and maximum values ​​for each dimension were determined based on 100,000 historical evaluation data points. For example, the minimum value for text semantic coherence is 0.2, and the maximum value is 0.9. The normalized score of 0.7 for the example text is (0.7-0.2) / (0.9-0.2)≈0.714. The minimum value for the creativity score is 0.1, and the maximum value is 0.9. The normalized score of 0.7415 is (0.7415-0.1) / (0.9-0.1)≈0.802.

[0131] The final quality assessment matrix is ​​a 6×4 normalized matrix, with the following core scores: text semantic coherence (text column 0.714), text-image semantic consistency (overall column 0.827), text-audio sentiment consistency (overall column 0.986), image-audio spatiotemporal synchronization (overall column 0.84), creativity score (overall column 0.802), and style fit (overall column 0.933). The matrix clearly presents the quality performance of each dimension, with text-audio sentiment consistency and style fit performing best, and the creativity score at a good level, providing clear guidance for subsequent strategy adjustments.

[0132] S204, based on the quality assessment matrix, the content generation strategy is dynamically adjusted through the decision-coordination agent, the agent collaboration weights are optimized using a reinforcement learning mechanism, and the final optimized multimodal content work is output.

[0133] Specifically, it can analyze the scores in the quality assessment matrix, identify content weaknesses and optimization directions through decision tree algorithms, and generate strategy adjustment suggestions;

[0134] This step is a prerequisite for dynamic adjustment. Its core is to use decision tree algorithms to deeply analyze the quality assessment matrix, accurately pinpoint the quality weaknesses of multimodal content, clarify optimization directions, and provide targeted objectives for subsequent reinforcement learning. The specific implementation method is as follows:

[0135] The quality assessment matrix is ​​a 6×4 normalized matrix. The core scoring dimensions include textual semantic coherence, text-image semantic consistency, text-audio sentiment consistency, image-audio spatiotemporal synchronization, creativity score, and style fit. Each dimension has a maximum score of 1 point. The overall quality score is the weighted average of all dimensions (weights are allocated according to importance: consistency 0.2 each, creativity 0.15, style fit 0.15, textual semantic coherence 0.1). The overall quality score of the example matrix is ​​(0.714×0.1+0.827×0.2+0.986×0.2+0.84×0.2+0.802×0.15+0.933×0.15)≈0.856. The decision tree algorithm used is the C4.5 algorithm. This algorithm selects the optimal splitting feature through information gain, effectively handles continuous scoring data, and is suitable for hierarchical analysis of quality dimensions.

[0136] The decision tree is constructed with the "overall quality score" as the root node, and the scores of each dimension are the splitting features. The splitting thresholds are determined based on historical data statistics (e.g., text semantic coherence threshold 0.8, creativity score threshold 0.85, style matching threshold 0.9). First, the information entropy of the root node is calculated. In the example, the overall quality score distribution is "Excellent (≥0.9): 0%, Good (0.8-0.9): 100%, Average (0.7-0.8): 0%, Poor (<0.7): 0%", and the information entropy H = -(1×log21) = 0. Then, the information gain of each feature is calculated. The information gain of text semantic coherence is the highest (0.12), so it becomes the first splitting feature, splitting the data into two child nodes: "≥0.8" and "<0.8". In the example, the score of this dimension is 0.714 (<0.8), so it enters the corresponding child node. This child node continues to split by the creativity score (information gain 0.09). In the example, the score is 0.802 (<0.85), so it enters the next level child node. Finally, the leaf node outputs the weakness identification result of "insufficient text semantic coherence + creativity needs to be improved".

[0137] The optimization directions are generated based on the mapping relationship between weaknesses and agent responsibilities: text semantic coherence is dominated by the text generation agent and associated with the semantic deconstruction accuracy of the task parsing agent. The optimization direction is to "improve the semantic coherence modeling of the text generation agent and enhance the deconstruction weight of the task parsing agent on 'scene details'"; creativity score is related to text, image and audio generation agents. The optimization direction is to "increase the detailed originality output of the text generation agent, improve the novelty of the visual element combination of the image generation agent, and enrich the melodic variation of the audio generation agent".

[0138] The generated strategy adjustment suggestions are structured text, comprising four parts: "Quality Overview, Weakness Identification, Agent Adjustment Direction, and Parameter Optimization Suggestions." The quality overview shows an overall multimodal content quality score of 0.856 (good), with excellent text-audio sentiment consistency (0.986) and style matching (0.933). The main weaknesses are text semantic coherence (0.714) and creativity score (0.802). Weakness identification indicates insufficient semantic logical coherence in the text description and a lack of original details in the multimodal content. The agent adjustment direction... 1. Text generation agent: Enhance semantic coherence modeling and increase original output of scene details; 2. Task parsing agent: Increase the deconstruction weight of the 'detail description' dimension; 3. Image generation agent: Optimize the novelty of visual element combination; Parameter optimization suggestions: Reduce the temperature parameter of the text generation agent from 0.7 to 0.6 (to improve coherence), increase the detail deconstruction weight of the task parsing agent from 0.1 to 0.15, and increase the Chaos value of the image generation agent from 10 to 15 (to increase visual novelty)".

[0139] Based on policy adjustment suggestions, a reinforcement learning reward function is designed, with the quality assessment matrix score as the optimization objective. The reward signal for agent cooperation is defined, and a reinforcement learning environment configuration is generated.

[0140] This step is the core design element of reinforcement learning. It transforms the quality objective into a signal that the agent can perceive through the reward function, and simultaneously configures the reinforcement learning environment to provide a framework for weight optimization. The specific implementation method is as follows:

[0141] The core elements of a reinforcement learning environment include agents, state space, action space, and reward function. The agents consist of five collaborative agents: a task parsing agent, a text generation agent, an image generation agent, an audio generation agent, and a quality assessment agent. The state space is defined as "the 6-dimensional core score of the quality assessment matrix + the current collaborative weights of the five agents," with 11 dimensions. For example, a state might be [0.714, 0.827, 0.986, 0.84, 0.802, 0.933, 0.1, 0.3, 0.3, 0.2, 0.1] (the first 6 are scores, and the last 5 are the weights of each agent: task parsing 0.1, text 0.3, image 0.3, audio 0.2, quality assessment 0.1). The action space is defined as "the adjustment amount of each agent's collaborative weight," with each weight adjusted within the range of [-0.05, +0.05] to ensure smooth weight adjustments and avoid large fluctuations.

[0142] The reward function design follows the principle of "goal-oriented + collaborative incentive," with the core objective of maximizing the overall quality score and improving weaker dimensions. The formula is: R = R_base + R_coop - R_penalty. Here, R_base is the base reward, directly linked to the overall quality score: R_base = 10 when the score is ≥ 0.9, R_base = 5 when the score is 0.8 ≤ score < 0.9, R_base = 0 when the score is 0.7 ≤ score < 0.8, and R_base = -5 when the score is < 0.7. R_coop is the collaborative reward, calculated as the sum of the products of each agent's weight and the corresponding dimension score, i.e., R_coop = ω1×S1 + ω2×S2 + ω3×S3 + ω4×S4 + ω5×S5 (ω1-ω5 are the agent weights, S1-S5 are the corresponding dimension scores: task parsing corresponds to text semantic coherence, text generation corresponds to creativity). The scores are calculated as follows: gender score, semantic consistency between image generation and text-image, emotional consistency between audio generation and text-audio, and overall score stability of quality assessment. In this example, R_coop = 0.1 × 0.714 + 0.3 × 0.802 + 0.3 × 0.827 + 0.2 × 0.986 + 0.1 × 0.856 ≈ 0.0714 + 0.2406 + 0.2481 + 0.1972 + 0.0856 ≈ 0.8429; R_penalty is the penalty term, which deducts 2 points for each weak dimension that fails to meet the standard. In this example, 4 points are deducted for 2 weak dimensions, and the final R = 5 + 0.8429 - 4 ≈ 1.8429.

[0143] The reward function parameters have been specifically optimized: the score threshold for the basic reward is determined based on the rating distribution of historical high-quality content (the average score for high-quality content is 0.88); the weight coefficient for the collaborative reward ensures that the contribution of each agent matches its own responsibility; and the penalty for the penalty item can both incentivize improvement of weaknesses and avoid excessively suppressing overall performance. The reinforcement learning environment configuration also includes training parameters: 100 iterations, 10 tracks sampled per round (each track corresponds to a complete content generation-evaluation process), discount factor γ=0.9 (emphasizing future rewards), exploration rate ε=0.1 (10% probability of randomly adjusting weights to ensure exploratory nature), and the environment is encapsulated using the Gym framework, supporting real-time interaction between agent actions and states.

[0144] The reinforcement learning training process is executed, and the collaborative weight parameters of the task parsing agent, content generation agent and quality assessment agent are updated through the policy gradient method to generate an optimized agent weight set.

[0145] This step is the execution phase of weight optimization. Its core is to maximize cumulative rewards through the policy gradient method, dynamically adjusting the collaborative weights of each agent to allocate resources towards improving weaknesses. The specific implementation is as follows:

[0146] The REINFORCE algorithm is chosen as the policy gradient method. This algorithm directly optimizes the expected reward of the policy function and is suitable for weight adjustment scenarios in discrete action spaces. The policy network adopts a two-layer fully connected neural network structure. The input is an 11-dimensional state vector, the hidden layer dimension is 256, the activation function is ReLU, and the output layer is the probability distribution of weight adjustment amounts for five agents (each adjustment amount corresponds to an action). The network parameters are initialized using a Xavier normal distribution, and the learning rate is set to 0.001 (to balance convergence speed and stability).

[0147] The training process is executed in a loop of "sampling - reward calculation - parameter update": In each training round, the agent samples 10 trajectories according to the current policy. Each trajectory contains T=5 steps (corresponding to 5 weight adjustments - content generation - evaluation processes). For each trajectory, the cumulative discount reward G_t=R_t+γ×R_{t+1}+γ²×R_{t+2}+...+γ^{T-1}×R_T is calculated. In the example, the reward sequence of a certain trajectory is [1.84,2.32,3.15,4.02,5.18], and the cumulative discount reward G_1=1.84+0.9×2.32+0.9²×3.15+0.9³×4.02+0.9 4 ×5.18≈1.84+2.088+2.5605+2.905+3.33≈12.7235; then calculate the policy gradient ∇θJ(θ)=(1 / T)×Σ(G_t×∇θlogπ_θ(a_t|s_t)), where π_θ(a_t|s_t) is the probability of choosing action a_t in state s_t. Update the network parameters θ through stochastic gradient ascent to maximize the expected reward.

[0148] The initial weights for agent collaboration are set based on the importance of each agent's role, with the initial weight set being [ω1=0.1 (task parsing), ω2=0.3 (text generation), ω3=0.3 (image generation), ω4=0.2 (audio generation), ω5=0.1 (quality assessment)]. After 100 rounds of training, the weights are dynamically adjusted: the weight of the text generation agent is increased from 0.3 to 0.35 (due to the core weakness of text semantic coherence, its resource allocation needs to be increased), the weight of the task parsing agent is increased from 0.1 to 0.12 (to strengthen detailed deconstruction), the weight of the image generation agent is slightly adjusted from 0.3 to 0.28 (style matching is already excellent, so the weight is appropriately reduced), the weight of the audio generation agent remains at 0.2 (emotional consistency is excellent), and the weight of the quality assessment agent is increased to 0.15 (to strengthen the guidance of assessment accuracy for optimization).

[0149] The optimized agent weight set is [0.12, 0.35, 0.28, 0.2, 0.15], with the sum of weights remaining at 1, ensuring the rationality of resource allocation. The effectiveness of the weight adjustment is verified using a validation set: in the content draft generated after adjustment, the text semantic coherence score increased to 0.82, the creativity score increased to 0.85, and the overall quality score reached 0.89, an improvement of 0.034 from the initial score, proving that the policy gradient method can effectively optimize collaborative weights.

[0150] The content generation process is re-executed using the optimized agent weight set, and the multimodal content draft is iteratively optimized until the quality threshold is met, and the final optimized multimodal content work is output.

[0151] This step is the final stage of the iterative optimization. Its core is to continuously improve weaknesses in the duplicate content generation process based on the optimized weight set until the overall quality meets the standards and a satisfactory multimodal work is output. The specific implementation method is as follows:

[0152] The quality threshold is set based on the industry's high-quality standards for multimodal content, defined as "overall quality score ≥ 0.85", while requiring no single dimension score to be lower than 0.75 (to avoid serious weaknesses in any one dimension). The iterative optimization process is consistent with the initial content generation process (task parsing - parallel generation - cross-modal fusion - quality assessment), but the parameters of each agent are adjusted according to the optimized weight set, and resource allocation is tilted towards improving weaknesses.

[0153] First iteration optimization: After increasing the weight of the text generation agent to 0.35, the generation parameters were adjusted: temperature was reduced from 0.7 to 0.6 (to reduce randomness and improve semantic coherence), an n-gram penalty was added (penalty_alpha=0.6 to avoid repeated phrases), and the generation of "scene details" was strengthened (the weight of the parsing agent's detail deconstruction was increased to 0.15, increasing the strength of the "detail description" dimension in the task vector from 0.7 to 0.85). The generated new text fragment is: "In the courtyard on a weekend afternoon, freshly steamed bright red crabs were laid out on the off-white tablecloth woven by Grandma. Dad poured Grandpa a full glass of amber-colored yellow wine. My five-year-old niece rushed into Mom's arms with an orange-yellow juice cup. Warm, dappled light filtered through the sycamore leaves. Laughter, accompanied by the ensemble of piano and violin, drifted far away. The wind was filled with the aroma of food and the freshness of grass and trees." The semantic coherence score increased to 0.82, and the creativity score increased to 0.85 due to original details such as "Grandma's tablecloth" and "amber-colored yellow wine." The image generation agent's weights were fine-tuned to 0.28, and the Chaos value was increased from 10 to 15. The generated images, while retaining the core scene, added novel visual elements such as "embroidered patterns on the tablecloth" and "fallen sycamore leaves on the ground." The text-image semantic consistency score remained at 0.83. The audio generation agent maintained its parameters, and the text-audio sentiment consistency remained at 0.98. The overall quality score after quality assessment was 0.89, meeting the quality threshold, but a second iteration was initiated for further optimization.

[0154] The second iteration of optimization focused on improving the creativity score. The stylization coefficient of the image generation agent was increased from 0.8 to 0.85, and the BPM of the audio generation agent was fine-tuned from 85 to 88 (to add melodic variation). The text generation agent incorporated personalized dialogue details (e.g., the niece saying "Grandpa, cheers"). The newly generated text fragment achieved a semantic coherence score of 0.84 and a creativity score of 0.88. The image now features a cartoon image of the niece's juice cup, and a slight clinking sound was added to the audio at 15 seconds. The overall quality score reached 0.91, with all dimensions scoring ≥0.82, meeting the "excellent standard."

[0155] The iteration termination condition is "the overall quality score improvement between two consecutive iterations is ≤0.02". The second iteration improved by 0.02 compared to the first, and the score after the third iteration was 0.92, an improvement of 0.01, thus meeting the termination condition. The final output multimodal content work includes: a 420-character text (semantically coherent and with original details), a 1024×1024 image (visually novel and highly matched with the text), and a 30-second audio (emotionally collaborative and spatiotemporally synchronized), along with a quality assessment report (overall score of 0.92, detailed scores for each dimension) and an agent collaboration weight log. The work can be directly used for dissemination or secondary editing, fully meeting the user's multimodal content needs.

[0156] As can be seen, by using a task parsing agent to semantically deconstruct the multimodal content requirements input by the user, a structured task description vector containing topic dimensions, style elements, and sentiment tendencies is generated. Based on the structured task description vector, a multimodal content draft is generated. The multimodal content draft is then input into a quality assessment agent network to generate a quality assessment matrix containing semantic coherence scores and style matching degrees. Based on the quality assessment matrix, the content generation strategy is dynamically adjusted by a decision-coordinating agent, and a reinforcement learning mechanism is used to optimize the agent's collaborative weights. The final optimized multimodal content is then output, thereby enabling high-quality, highly consistent parallel generation and deep fusion of multimodal content such as text, images, and audio, improving the creativity, semantic coherence, and matching degree of the final work with user needs.

[0157] Another embodiment of the present invention provides a content generation data processing system based on multi-agent collaboration, see [link to relevant documentation]. Figure 3 The system may include:

[0158] The deconstruction module 301 is used to semantically deconstruct the multimodal content requirements input by the user through the task parsing agent, and generate a structured task description vector containing topic dimensions, style elements and sentiment tendencies.

[0159] The fusion module 302 is used to generate candidate content fragments in parallel through a cluster of content generation agents based on the structured task description vector, and to perform spatiotemporal alignment and fusion of text, image and audio fragments using a cross-modal attention mechanism to generate a multimodal content draft.

[0160] The generation module 303 is used to input the multimodal content draft into the quality assessment agent network and generate a quality assessment matrix containing semantic coherence score and style matching degree through a multi-dimensional consistency detection algorithm and a creativity index quantification model.

[0161] The optimization module 304 is used to dynamically adjust the content generation strategy through the decision-coordination agent based on the quality assessment matrix, optimize the agent collaboration weights using a reinforcement learning mechanism, and output the final optimized multimodal content work.

[0162] This invention also provides a storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when running.

[0163] This invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0164] Specifically, the aforementioned electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the aforementioned processor, and the input / output device is connected to the aforementioned processor.

[0165] The above description, based on the embodiments shown in the figures, details the structure, features, and effects of the present invention. The above description is only a preferred embodiment of the present invention, but the present invention is not limited to the scope of implementation shown in the figures. Any changes made in accordance with the concept of the present invention, or equivalent embodiments modified to have equivalent changes, that do not exceed the spirit covered by the specification and figures, should be within the protection scope of the present invention.

Claims

1. A content generation data processing method based on multi-agent cooperation, characterized in that, The method includes: The task parsing agent performs semantic deconstruction of the multimodal content requirements input by the user, generating a structured task description vector containing topic dimensions, style elements, and sentiment tendencies; Based on the structured task description vector, candidate content fragments are generated in parallel by a cluster of content generation agents, and a cross-modal attention mechanism is used to perform spatiotemporal alignment and fusion of text, image and audio fragments to generate a multimodal content draft. The multimodal content draft is input into the quality assessment agent network, and a quality assessment matrix containing semantic coherence score and style matching degree is generated through a multi-dimensional consistency detection algorithm and a creativity index quantification model. Based on the quality assessment matrix, the content generation strategy is dynamically adjusted by the decision-coordinating agent, and the agent's collaborative weights are optimized using a reinforcement learning mechanism to output the final optimized multimodal content.

2. The method of claim 1, wherein, The process involves semantically deconstructing the multimodal content requirements input by the user through a task parsing agent, generating a structured task description vector containing topic dimensions, style elements, and sentiment tendencies, including: The system receives user-provided text descriptions, reference images, and audio samples through a multimodal input interface, and uses a data normalization algorithm to unify the format and sampling rate of different modal data to generate standardized multimodal input data. Semantic features are extracted from standardized multimodal input data. A pre-trained language model is used to parse the semantics of the text, a visual model is used to extract image style features, and an audio model is used to analyze emotional tone and generate a multimodal semantic feature set. Semantic association analysis is performed based on multimodal semantic feature sets. A dependency graph between topics, styles and sentiments is constructed using graph neural networks to generate a semantic association graph. The semantic association graph is encoded into a fixed-dimensional numerical vector. An embedding layer technique is used to map the topic dimension, style elements, and sentiment to a continuous vector space, ultimately generating a structured task description vector.

3. The method of claim 2, wherein, Based on the structured task description vector, candidate content fragments are generated in parallel by a cluster of content generation agents. A cross-modal attention mechanism is then used to spatiotemporally align and fuse text, image, and audio fragments to generate a multimodal content draft, including: The structured task description vector is distributed to the text generation agent, image generation agent, and audio generation agent. Each agent creates content fragments in parallel based on a dedicated generation model, generating an initial set of candidate content fragments. Spatiotemporal feature encoding is performed on the initial candidate content fragment set. The temporal encoder is used to process the time series of audio fragments, the spatial encoder is used to process the visual layout of image fragments, and the text encoder is used to process the language sequence, generating a multimodal feature sequence. Based on multimodal feature sequences, a cross-modal attention mechanism is applied to calculate the visual attention weights of text features and image features, and the temporal attention weights of text features and audio features, generating a cross-modal attention map. The feature fusion process is guided by cross-modal attention maps. By integrating feature representations from different modalities through weighted summation, and decoding, an aligned multimodal content draft is generated.

4. The method of claim 3, wherein, The multi-modal content draft quality evaluation agent network is input, a multi-dimensional consistency detection algorithm and a creativity index quantification model are used to generate a quality evaluation matrix containing semantic coherence scores and style matching degrees, including: The multi-modal content draft is subjected to feature separation to extract text semantic embedding, image visual embedding and audio acoustic embedding, and separate modal feature vectors are generated; Based on the separate modal feature vectors, multi-dimensional consistency detection is performed to calculate text-image semantic consistency scores, text-audio emotional consistency scores and image-audio space-time synchronization scores, and consistency detection results are generated; The creativity index quantification model is used to evaluate the novelty of the content, and the uniqueness and innovation index of the content segment are calculated by comparing the pre-trained knowledge base to generate a creativity score; The consistency detection results and the creativity score are integrated to construct a multi-dimensional score matrix and normalized to finally generate a quality evaluation matrix containing semantic coherence scores and style matching degrees.

5. The method of claim 4, wherein, The content generation strategy is dynamically adjusted according to the quality evaluation matrix through the decision coordination agent, the agent cooperation weight is optimized using the reinforcement learning mechanism, and the final optimized multi-modal content work is output, including: Analyzing each score in the quality evaluation matrix, identifying content weaknesses and optimization directions through a decision tree algorithm, and generating strategy adjustment suggestions; Based on the strategy adjustment suggestions, a reinforcement learning reward function is designed, the quality evaluation matrix score is taken as the optimization target, the reward signal of agent cooperation is defined, and a reinforcement learning environment configuration is generated; The reinforcement learning training process is executed, the cooperation weight parameters of the task analysis agent, the content generation agent and the quality evaluation agent are updated through the policy gradient method, and the optimized agent weight set is generated; The content generation process is re-executed using the optimized agent weight set, and the multi-modal content draft is iteratively optimized until the quality threshold is met, and the final optimized multi-modal content work is output.

6. A content generation data processing system based on multi-agent collaboration, characterized by, The system comprises: A deconstruction module for performing semantic deconstruction on the multi-modal content requirements input by the user through a task analysis agent to generate a structured task description vector containing theme dimensions, style elements and emotional tendencies; A fusion module for generating candidate content segments in parallel through a content generation agent cluster based on the structured task description vector, and performing space-time alignment fusion on text, image and audio segments using a cross-modal attention mechanism to generate a multi-modal content draft; A generation module for inputting the multi-modal content draft into a quality evaluation agent network, generating a quality evaluation matrix containing semantic coherence scores and style matching degrees through a multi-dimensional consistency detection algorithm and a creativity index quantification model; An optimization module for dynamically adjusting the content generation strategy according to the quality evaluation matrix through a decision coordination agent, optimizing the agent cooperation weight using a reinforcement learning mechanism, and outputting the final optimized multi-modal content work.

7. The system of claim 6, wherein, The deconstruction module is specifically configured to: Receive the text description, reference image and audio sample provided by the user through a multi-modal input interface, and use a data normalization algorithm to unify the format and sampling rate of different modal data to generate standardized multi-modal input data; The semantic feature extraction is performed on the standardized multi-modal input data, a pre-trained language model is used to analyze the text semantics, a visual model is used to extract image style features, and an audio model is used to analyze emotional intonation, and a multi-modal semantic feature set is generated; The semantic correlation analysis is performed based on the multi-modal semantic feature set, a dependency graph among the theme, style and emotion is constructed through a graph neural network, and a semantic correlation graph is generated; The semantic correlation graph is encoded into a fixed-dimensional numerical vector, an embedding layer technology is used to map the theme dimension, style elements and emotional tendency to a continuous vector space, and finally a structured task description vector is generated.

8. The system of claim 7, wherein, The fusion module is specifically used for: The structured task description vector is distributed to a text generation agent, an image generation agent and an audio generation agent, each agent creates a content segment in parallel based on a special generation model, and an initial candidate content segment set is generated; The initial candidate content segment set is subjected to spatio-temporal feature encoding, a time series encoder is used to process the time series of the audio segment, a spatial encoder is used to process the visual layout of the image segment, and a text encoder is used to process the language sequence, and a multi-modal feature sequence is generated; A cross-modal attention mechanism is applied based on the multi-modal feature sequence, visual attention weights of the text features and the image features, and time series attention weights of the text features and the audio features are calculated, and a cross-modal attention graph is generated; The cross-modal attention graph is used to guide the feature fusion process, the feature representations of different modalities are integrated through weighted summation, and an aligned multi-modal content draft is generated through decoding.

9. A storage medium, characterized by The storage medium stores a computer program, and the computer program is configured to execute the method of any one of claims 1-5 when running.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method of any one of claims 1-5 by running the computer program.

Images