Intelligently-started artificial intelligence multi-modal content generation system

Abstract

The invention relates to the technical field of artificial intelligence and multi-modal content generation, and discloses an intelligent starting artificial intelligence multi-modal content generation system which comprises a multi-modal input module, a modal encoder, a joint representation construction module, a user control condition embedding module, a semantic consistency optimization module, a modal distribution alignment module and a multi-modal decoder. Feature extraction, joint representation construction and user control condition dynamic adjustment are performed on multi-modal input data, and semantic consistency optimization and modal distribution alignment technologies are utilized to generate diversified and high-quality multi-modal target content with high semantic consistency and controllable user. The problems that in the prior art, multi-modal information fusion is difficult, the generated content is uncontrollable, and modal distribution is inconsistent are solved, and the quality and practicability of multi-modal content generation and the user experience effect are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence and multimodal content generation, and specifically to a Zhiqi artificial intelligence multimodal content generation system. Background Art

[0002] Multimodal content generation technology is a hot research direction in the current field of artificial intelligence and is widely used in scenarios such as intelligent creation, virtual reality, and cross-modal retrieval. This technology achieves the generation of multimodal content in complex scenarios through deep understanding and fusion of multimodal data. However, in the existing technology, multimodal content generation still faces the following deficiencies, which limit the performance and scalability of the technology in practical applications.

[0003] Existing multimodal content generation technologies have problems such as difficulty in multimodal information fusion, insufficient user control ability, excessive redundant information in the generated results, and insufficient alignment of modal distribution. The existence of these problems directly affects the semantic consistency, controllability and generation quality of the generated content, limiting the practical application value of multimodal generation technology; in view of these deficiencies in the prior art, the present invention proposes a new multimodal content generation method by innovatively designing a modal encoder, a joint representation construction module, a user control condition embedding module, a semantic consistency optimization module and a modal distribution alignment module, which realizes the deep fusion and high-quality generation of multimodal information, and significantly improves the semantic consistency, user control ability and reliability of the generated content of the system. Summary of the invention

[0004] In response to the shortcomings of the prior art, the present invention provides a Zhiqi artificial intelligence multimodal content generation system. In response to the shortcomings of the prior art, a multimodal content generation method is proposed through innovative design of a modal encoder, a joint representation construction module, a user control condition embedding module, a semantic consistency optimization module and a modal distribution alignment module, which realizes the deep fusion and high-quality generation of multimodal information, and significantly improves the system's semantic consistency, user control capability and the reliability of generated content.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: a Zhiqi artificial intelligence multimodal content generation system, comprising:

[0006] A multimodal input module, used to receive and process multimodal input data, wherein the multimodal input data includes text modality, image modality and audio modality;

[0007] A modality encoder, used to encode multimodal input data into corresponding modality latent representations;

[0008] The joint representation construction module is used to fuse the latent representations of each modality to generate a joint latent representation;

[0009] A user-controlled condition embedding module, which is used to dynamically adjust the joint latent representation according to the control conditions provided by the user;

[0010] A multimodal decoder for generating target modality content from the adjusted joint latent representation;

[0011] The semantic consistency optimization module is used to optimize the joint latent representation through the variational information bottleneck theory to remove redundant information and enhance the semantic consistency between modalities;

[0012] The modal distribution alignment module is used to align the input modal distribution and the generated modal distribution through optimal transmission theory to reduce the semantic deviation between modalities.

[0013] Preferably, the modal encoder includes a text modal encoder, an image modal encoder and an audio modal encoder, and the specific implementation steps include:

[0014] The text modal encoder takes a text modal input X T Processing is performed to embed the text input into a low-dimensional semantic representation space, which specifically includes the following steps:

[0015] Enter X for text T Perform word segmentation and map the word segmentation results to the word embedding matrix through the pre-trained word vector model;

[0016] Use stacked multi-layer bidirectional Transformer encoders to extract text semantic features Z T , the calculation process is:

[0017]

[0018] Where E(X T ) represents the output of the word embedding operation, represents the text modality encoder, Z T is the potential representation of the text modality;

[0019] The image modality encoder takes the image modality input X T Processing is performed to map the feature information of the image to the potential representation space, which specifically includes the following steps:

[0020] For image input X T Perform normalization processing;

[0021] The image features are extracted through the convolutional neural network, and the output F of the convolution operation I It is expressed as:

[0022] F I =CNN(X I )

[0023] The image features are mapped to the latent space through the fully connected layer to generate the latent representation Z of the image modality I :

[0024] Z I =W I F I +b I

[0025] Where W I and b I are the weights and biases of the fully connected layer, Z I is the potential representation of the image modality;

[0026] The audio modality encoder is used to generate the audio modality input X A Processing is performed to extract the spectral features of the audio and map them to the latent representation space, specifically including the following steps:

[0027] For audio signal X A Perform preprocessing, including sampling, denoising and framing;

[0028] Perform Fourier transform on the preprocessed audio signal to generate the spectrum feature S A :

[0029] S A =FFT(X A )

[0030] The feature F of the spectrogram is extracted by stacking convolutional neural networks A ;

[0031] Use recurrent neural networks to capture the time series information of audio features;

[0032] Map the extracted spectral features into the latent representation Z of the audio modality A :

[0033] Z A =Bi-LSTM(F A )

[0034] The above modality encoders are used to generate the latent representations Z of text, image and audio modalities respectively. T ,Z I ,Z A , and provide input to the subsequent joint latent representation building module.

[0035] Preferably, the joint representation construction module fuses the latent representations of each modality into a unified joint latent space representation through a joint latent representation network, and provides a shared semantic representation for the subsequent generation of target modality content.

[0036] Preferably, the user control condition embedding module embeds the control conditions input by the user into the joint latent representation to adjust the style, modality priority or diversity of the generated content and customize the generated content according to user needs.

[0037] Preferably, the semantic consistency optimization module realizes semantic alignment between the input modality and the generated modality through a variational information bottleneck optimization objective, and the optimization objectives include:

[0038] Reduce the amount of redundant information in the joint latent representation;

[0039] Enhancing the semantic relevance between latent representation and generated content.

[0040] Preferably, the modal distribution alignment module uses an optimal transmission method to achieve modal alignment based on the Wasserstein distance between the input modal distribution and the generated modal distribution, and minimizes the semantic deviation between the modalities.

[0041] Preferably, the multimodal decoder generates target modal content, including text, image and audio modalities, based on the joint latent representation, and improves the quality of the generated modal content through a semantic consistency optimization module.

[0042] Preferably, the user-controlled condition embedding module dynamically adjusts the semantic compactness parameter and the modal distribution alignment weight according to different user-controlled parameters, so as to balance the diversity and semantic consistency of the generated content.

[0043] Preferably, the optimization process of the system adopts a joint loss function, including semantic consistency loss, modal distribution alignment loss and inter-modal similarity loss, and the joint loss function is used to simultaneously optimize the compactness of the latent representation, the semantic alignment of the generated modal content and the semantic consistency between modalities.

[0044] Preferably, the present invention also provides a method for generating multimodal content using intelligent artificial intelligence, characterized in that it comprises the following steps:

[0045] Receive multimodal input data and encode it into a latent representation of the corresponding modality through a modality encoder;

[0046] The joint representation construction module is used to fuse the potential representations of each modality to generate a joint potential representation.

[0047] Dynamically adjust the joint latent representation according to the control conditions input by the user;

[0048] The joint latent representation is optimized through the semantic consistency optimization module to remove redundant information and enhance semantic consistency;

[0049] Align the input modal distribution with the generated modal distribution through the modal distribution alignment module;

[0050] The target modality content, which includes text, image, and audio, is generated from the optimized joint latent representation through a multimodal decoder.

[0051] The present invention provides a Zhiqi artificial intelligence multimodal content generation system. It has the following beneficial effects:

[0052] 1. The present invention adopts a technical solution combining a modal encoder and a joint representation building module, and achieves the technical effect of highly integrating multimodal semantic information in a unified space by extracting features from text, image and audio modalities and fusing them to generate a joint potential representation. Compared with the technical solution in the prior art that processes multimodal features independently and lacks semantic alignment, it solves the problems of semantic mismatch and information isolation between modalities in the process of multimodal information fusion.

[0053] 2. The present invention uses a user-controlled conditional embedding module, an embedding network and a dynamic adjustment mechanism to perform user-directed adjustment on the joint potential representation, thereby achieving the technical effect of generating personalized and stylized content. Compared with the technical solution in the prior art that generates content lacks user demand drive and the generated results are uncontrollable, this solves the problem that multimodal generated content cannot meet specific needs and lacks diversity and controllability.

[0054] 3. The present invention uses the variational information bottleneck theory to remove redundant information in the joint potential representation through the semantic consistency optimization module, and at the same time enhances the semantic consistency between the input modality and the generated modality, achieving the technical effect of accurate semantics of generated content and concise information expression. Compared with the problems of insufficient semantic relevance and information redundancy of generated content in the prior art, the shortcomings of semantic ambiguity, information loss and low quality of generated content in the existing generation model are solved.

[0055] 4. The present invention adopts a modal distribution alignment module, calculates the deviation between the input modal distribution and the generated modal distribution based on the optimal transmission theory, achieves the alignment effect by optimizing the distribution of the generated modality, and ensures the strong correlation and consistency between the generated content and the input modality. Compared with the problem of semantic deviation of generated content caused by the lack of inter-modal distribution alignment mechanism in the prior art, this solves the problem of inconsistency between the generated modal distribution and the input modal distribution, which in turn affects the quality and reliability of the generated content. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a system architecture diagram of the present invention;

[0057] Figure 2 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0058] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] Please refer to the attached Figure 1 , an embodiment of the present invention provides a Zhiqi artificial intelligence multimodal content generation system, including:

[0060] A multimodal input module, used to receive and process multimodal input data, wherein the multimodal input data includes text modality, image modality and audio modality;

[0061] In the Zhiqi artificial intelligence multimodal content generation system of the present invention, the multimodal input module is used to receive and preprocess multimodal input data, including text modality, image modality and audio modality. In general, due to the large differences in data characteristics and structures of different modalities, it is necessary to design specific preprocessing strategies for each modality to ensure that the input data can be efficiently processed by the subsequent modal encoder and extract semantic information. As a necessary function, the preprocessing results of this step directly affect the performance of subsequent modal encoding and the overall quality of the generation task.

[0062] Specifically, this step converts the input text, image and audio data into a format suitable for processing by the modality encoder through a series of standardization, structuring and feature conversion operations. The following describes the technical implementation of this embodiment in detail in combination with the text modality, image modality and audio modality.

[0063] In this embodiment, the text mode input data X T The system is usually input in the form of paragraphs, sentences or word sequences. In order to adapt to the processing requirements of the modal encoder, the text data is first segmented and embedded.

[0064] In general, the word sequence after word segmentation needs to be further mapped to a low-dimensional vector space. Specifically, a pre-trained word vector model can be used to convert the word segmentation result into a word embedding matrix E(X T ), which is expressed as:

[0065] E(X T )={e1,e2,…,e N}

[0066] Where: e i Represents the embedding vector of each word or subword; N represents the length of the input text after tokenization.

[0067] In this embodiment, the image modality input data X TUsually it is input into the system in the form of a two-dimensional pixel matrix. First, the input image is standardized, including normalization of pixel values, size adjustment, and data format conversion.

[0068] Specifically, the image normalization process includes the following steps:

[0069] First, the image pixel values ​​are normalized and their range is limited to [0, 1] or [-1, 1] to adapt to the input requirements of the convolutional neural network;

[0070] Then adjust the image size and scale all input images to a fixed size to ensure data consistency;

[0071] Finally, the resized image data is formatted into a tensor representation X′ I , this tensor form is convenient for subsequent modality encoder processing.

[0072] In this embodiment, the audio mode input data X A Usually it is input into the system in the form of a one-dimensional time series signal. Generally speaking, since the original audio signal contains a lot of redundant information and has a high dimension, it needs to be preprocessed to extract key information.

[0073] The audio signal is first sampled, framed and denoised:

[0074] The sampling operation resamples the audio signal to a uniform frequency to ensure that the format of the input data is consistent;

[0075] The framing operation decomposes the audio signal into several frames of equal length, providing a basis for subsequent frequency domain feature extraction;

[0076] De-noising can remove background noise and enhance the quality of the signal through filters or noise suppression algorithms.

[0077] Then, the framed audio signal is subjected to a fast Fourier transform (FFT) to convert the time domain signal into a frequency domain representation. The calculation formula is:

[0078] S A =FFT(X A )

[0079] Where: S A It is the spectrum graph feature of the audio, which is used to describe the frequency components of the signal; FFT is the fast Fourier transform operation.

[0080] The audio spectrogram can be further transformed into a Mel-spectrogram by logarithmic transformation. This operation can enhance the auditory-related features of the audio signal.

[0081] In this embodiment, by independently preprocessing the text modality, image modality, and audio modality, the system obtains standardized input data adapted for subsequent modality encoder processing.

[0082] In order to facilitate efficient processing in subsequent steps, multimodal input data can be organized into a unified batch format. For example, the processed text input X T ′、Image input X I ′ and audio input X A 'Package into multimodal batch data {X T ′,X I ′,X A '};

[0083] These preprocessed multimodal data will be directly passed to the modality encoder and serve as the basic input for subsequent latent representation extraction.

[0084] A modality encoder, used to encode multimodal input data into corresponding modality latent representations;

[0085] In the Zhiqi artificial intelligence multimodal content generation system of the present invention, the modal encoder is used to extract features and encode the potential representation of the multimodal input data preprocessed by the multimodal input module. This step uses a specially designed modal encoder to extract the adapted potential semantic representation Z according to the data structure and characteristics of different modalities. T , Z I and Z A These latent representations are the direct input to the subsequent joint representation building module, ensuring that multimodal data are semantically consistent and universal.

[0086] The text modality has the property of time series, the image modality has the property of two-dimensional space, and the audio modality has both time series and spectrum characteristics. Therefore, in this step, different encoder structures are designed according to the data characteristics of each modality to achieve feature extraction and semantic compression of multimodal input.

[0087] In this embodiment, the text modality encoder is used to convert the text modality input X T Mapped to its potential representation Z T Specifically, the text modality input is usually a word sequence or subword sequence after word segmentation, with a length of N, and each word or subword is represented as an embedding vector e i The embedding matrix for text modal input is:

[0088] E(X T )={e1,e2,…,e N}

[0089] where e i Embedding vector representing a word or subword.

[0090] The text modality encoder is designed based on the Transformer structure. First, the embedding matrix E(X T ) is input into a multi-layer bidirectional Transformer encoder to capture the global and local semantic relationships in the text. Its calculation formula is:

[0091]

[0092] in: represents the text modality encoder, with parameter θ; Transformer(·) is a multi-layer Transformer structure; Z T It is the latent semantic representation of text modality.

[0093] In order to enhance the modeling ability of text order information, positional encoding can be added to the input layer. The formula is:

[0094]

[0095] Where i represents the position of the word in the sequence and d represents the embedding dimension.

[0096] An L2 regularization operation can be added after the Transformer encoder output to ensure the numerical stability of the latent representation.

[0097] In this embodiment, the image modality encoder is used to convert the image modality input X I Mapped to its potential representation Z I In general, the image modality is input into the encoder in the form of a two-dimensional pixel matrix. In order to extract the multi-level features of the image, this step adopts an encoder structure based on a convolutional neural network (CNN).

[0098] Specifically, a series of convolution operations are first performed on the input image to extract local features. The output of the convolution operation is the feature tensor F I , the formula is:

[0099] F I =CNN(X I )

[0100] Where: CNN(·) represents the convolution operation, which is used to extract local features of the image; F I is the output feature tensor of the convolutional layer.

[0101] In order to map the image features to a latent space of fixed dimension, this step uses a fully connected layer to convolution output F I Perform linear transformation to generate the potential representation Z of the image modality I :

[0102] Z I =W I ·F I +b I

[0103] Where: W I and b I are the weight matrix and bias term of the fully connected layer respectively; Z I is the potential representation of the image modality.

[0104] The classic Res Net or Efficient Net structure can be used as a convolutional encoder to improve the robustness of image feature extraction.

[0105] A global average pooling (GAP) operation can be added after the convolutional layer to reduce the feature dimension while retaining the global semantic information.

[0106] In this embodiment, the audio modality encoder is used to convert the audio modality input X A Mapped to its potential representation Z A Generally, the audio modality is input into the encoder in the form of a one-dimensional time series signal, and its feature representation is extracted through frequency domain transformation.

[0107] Specifically, firstly, the audio signal is subjected to a fast Fourier transform (FFT) to convert the time domain signal into a frequency domain representation and generate a spectrum graph S A :

[0108] S A =FFT(X A )

[0109] Where: S A is the frequency spectrum of the audio; FFT(·) represents the fast Fourier transform operation.

[0110] The spectrogram can be further converted into a Mel-spectrogram to enhance the auditory properties of the audio.

[0111] Then, the audio modality encoder extracts the spectrogram features F through a convolutional neural network. A , and combined with a bidirectional recurrent neural network (such as Bi-LSTM) to extract time series features:

[0112] Z A =Bi-LSTM(CNN(S A ))

[0113] Where: CNN(·) represents the spectral features extracted by the convolutional network; Bi-LSTM(·) represents the bidirectional recurrent neural network, which is used to capture the time series dependency; ZA is the potential representation of the audio modality.

[0114] In order to improve the expressiveness of audio features, data augmentation operations can be performed on the Mel-spectrogram, such as randomly masking frequencies or time blocks (Spec Augment).

[0115] The joint representation construction module is used to fuse the latent representations of each modality to generate a joint latent representation;

[0116] In the intelligent AI multimodal content generation system of the present invention, the joint representation construction module combines the multimodal potential representation Z from the modality encoder through the joint representation construction module. T , Z I and Z A The multimodal latent representations are fused to generate a shared joint semantic representation Z. In general, direct fusion may lead to semantic mismatch or information loss due to the different distribution characteristics of the latent representations of text, image and audio modalities. Therefore, this module maps the multimodal latent representations to a unified latent semantic space through specific nonlinear transformations, inter-modal weight modeling, and alignment operations, thereby achieving efficient integration of cross-modal information.

[0117] In this embodiment, the joint representation construction module receives the multimodal potential representation Z from the modality encoder T , Z I and Z A These latent representations are generated by the text modality encoder, image modality encoder, and audio modality encoder, respectively, where feature extraction and dimension normalization have been completed.

[0118] To ensure that the latent representations of different modalities can be effectively fused semantically, the input representation can be further dimensioned. Specifically, the latent representations of different modalities can be aligned to the same dimension d through linear transformation. The formula is:

[0119] Z′ T =W T ·Z T +b T ,Z I ′=W I ·Z I +b I ,Z′ A =W A ·Z A +b A

[0120] Where: W T , W I , W A are the weight matrices of linear transformation respectively; b T , bI , b A is the corresponding bias term; Z′ T , Z I ′、Z A ′ is the potential representation after dimension alignment.

[0121] The adjusted representations may be normalized to ensure that the representations of each modality are within the same numerical range.

[0122] In this embodiment, the joint representation construction module combines the potential representations Z of different modalities through a nonlinear fusion method. T ′、Z I ′、Z A ′ is integrated into a shared joint semantic representation Z. In one possible implementation, a multi-layer perceptron (MLP) is used to perform nonlinear fusion of the modal representations, and the calculation formula is:

[0123] Z=g φ ([Z′ T ,Z′ I ,Z′ A ])

[0124] Where: g φ Construct a network for joint representation with parameter φ; [Z T ′,Z I ′,Z A ′] represents the concatenation operation of the potential representations of each modality; Z is the final joint semantic representation.

[0125] In order to enhance the nonlinear expression ability of the network, a nonlinear activation function, such as ReLU or Tanh activation function, can be introduced after each layer of MLP. In another possible implementation, the attention mechanism can be used to perform weighted fusion of the representations of each modality, thereby dynamically adjusting the importance of each modality in the joint representation. Specifically, the attention weight α of each modality representation is first calculated. T ,α I ,α A :

[0126] α T =softmax(W T ·Z T ′),α I =softmax(W I ·Z I ′),

[0127] α A =softmax(W A ·Z A )

[0128] The modality representations are then weighted summed according to the attention weights to generate the final joint representation:

[0129] Z=α T ·Z′ T +α I ·Z′ I +α A ·Z′ A

[0130] In order to ensure that the generated joint representation Z has good distribution characteristics and robustness, the joint representation is regularized in this embodiment. Generally, L2 norm regularization can be used to constrain the numerical range of the joint representation, and its regularization objective function is:

[0131]

[0132] The joint representation can be normalized in combination with Batch Normalization to further improve its numerical stability and accelerate the optimization process of subsequent modules. As an option, the hidden layer output of the joint representation can also be randomly masked through the Dropout technique to reduce the risk of overfitting.

[0133] In this embodiment, in order to further improve the modality consistency of the joint semantic representation, the joint representation can be optimized through a cross-modal contrastive learning method. Specifically, by calculating the semantic similarity of the potential representations of different modalities, their distribution in the semantic space is ensured to be consistent. The loss function is:

[0134]

[0135] Where: sim(·,·) represents the similarity measure between latent representations, such as cosine similarity; The image potential representation of other samples is used for comparative learning. The above optimization process can significantly improve the semantic consistency of the joint representation between modalities and provide a higher quality semantic representation for the subsequent user-controlled conditional embedding module.

[0136] A user-controlled condition embedding module, which is used to dynamically adjust the joint latent representation according to the control conditions provided by the user;

[0137] In the Zhiqi artificial intelligence multimodal content generation system of the present invention, the user control condition embedding module dynamically adjusts the joint semantic representation Z generated from the joint representation construction module through the user control condition embedding module. In general, the control conditions U input by the user include but are not limited to style parameters, modal priorities, diversity adjustment factors and other information, which can make directional adjustments to the joint semantic representation, thereby generating target modal content with user-specified characteristics and requirements.

[0138] This module embeds the user control conditions into the joint representation through a nonlinear mapping method, ensuring that the adjusted semantic representation Z′ retains the original semantic information while meeting the user's customized needs for generated content. The following is a detailed description of the specific implementation of this module in combination with the technical implementation method.

[0139] In this embodiment, the user control condition U is a control parameter input by the user through an interactive interface, which may include a continuous value or a discrete value. In order to facilitate subsequent processing, the input control condition U generally needs to be standardized.

[0140] For continuous control conditions, they can be mapped to a fixed range, such as [0,1], by normalization. The formula is:

[0141]

[0142] Among them: U min and U max are the minimum and maximum values ​​of the control conditions respectively; U′ is the normalized control condition vector. For discrete control conditions (such as style options), the One-Hot Encoding method can be used to convert them into high-dimensional vector form. For example, if the style selected by the user is "realistic", the encoding result can be represented as a vector [1,0,0].

[0143] The control condition U input by the user may also be processed dynamically, for example, by selecting different control parameter ranges or encoding methods according to specific task requirements.

[0144] In this embodiment, in order to embed the user control condition U′ into the joint representation Z, the control condition embedding module designs an embedding network h(U′) for mapping U′ into a vector E(U′) consistent with the dimension of the joint representation Z. Generally, the embedding network adopts a multi-layer perceptron (MLP) structure, and its calculation formula is:

[0145] E(U′)=h(U′)=MLP(U′)

[0146] Where: h(U′) is the embedding function of the control condition; E(U′) is the vector output by the embedding network, which is consistent with the dimension of the joint semantic representation Z.

[0147] In this embodiment, the control condition vector E(U′) mapped by the embedding network will be fused with the joint representation Z to generate a dynamically adjusted joint representation Z′. In general, the fusion method can adopt a weighted summation or an attention mechanism.

[0148] The weighted summation method is used to model the influence of the control condition on the joint representation in the form of weight α, which is formulated as follows:

[0149] Z′=Z+α·E(U′)

[0150] Where: α is the fusion weight specified by the user or dynamically learned through the training process; Z′ is the adjusted joint semantic representation.

[0151] The control condition embedding can be dynamically weighted through the attention mechanism. Specifically, the similarity weight β between the joint representation and the control condition embedding is first calculated:

[0152]

[0153] Then, the joint representation is weighted updated according to the weight β:

[0154] Z′=β·Z+(1-β)·E(U′)

[0155] In order to ensure that the adjusted joint representation Z′ has good distribution characteristics, it is regularized in this embodiment. Generally, L2 norm regularization can be used to constrain the numerical range of the joint representation, and its regularization objective function is:

[0156]

[0157] The adjusted representation can also be normalized in combination with Batch Normalization to further improve numerical stability.

[0158] In this embodiment, the user control condition embedding module supports multiple types of control parameters. For example:

[0159] Style control: Users can adjust the parameters U style Specify the generation style of the target modality content (such as realistic or abstract style);

[0160] Priority control: Users can set the priority of the priority Specify the importance of different modalities (text, image, audio);

[0161] Diversity control: Users can set the diversity Modulate the diversity of generated content, for example by increasing the randomness of the latent space to generate more diverse content.

[0162] In this embodiment, the joint representation Z′ adjusted by the user control condition embedding module will be used as the input of the multimodal decoder. Generally, the adjusted semantic representation Z′ not only retains the global semantic information of the original joint representation, but also explicitly embeds the user control condition, so as to generate the target modality content that meets the user's needs.

[0163] The adjusted joint representation can be further dimensionalized to match the input requirements of the multimodal decoder. For example, the dimension of Z′ can be set to 512 or 1024 to adapt to common decoder structures.

[0164] A multimodal decoder for generating target modality content from the adjusted joint latent representation;

[0165] In the Zhiqi artificial intelligence multimodal content generation system of the present invention, the multimodal decoder decodes the adjusted joint representation Z' from the user control condition embedding module through the multimodal decoder to generate the target modal content. In general, the role of the decoder is to convert the high-dimensional semantic representation Z' into a specific modal output while ensuring that the output content can accurately reflect the input semantics and the control conditions specified by the user.

[0166] As an option, this module designs independent decoders for different modal generation tasks, such as text generation decoder, image generation decoder, and audio generation decoder. Through these dedicated decoders, the system can efficiently complete the generation task of multimodal content. The technical implementation of this step is described in detail below.

[0167] In this embodiment, the adjusted joint representation Z′ is used as the input of the multimodal decoder. Generally, Z′ is a unified semantic representation that contains the semantic information of the multimodal input and the user's control conditions. During the decoding process, Z′ needs to be mapped to the target modal space to generate content related to a specific modality.

[0168] The joint representation Z′ is decoded into outputs of text, image, and audio modalities through independent decoding networks. Specifically:

[0169]

[0170] in: Decoders for text, image, and audio modalities, respectively; Y T ,Y I ,Y A are the decoded text, image and audio modal contents respectively; ψ is the parameter of the decoder.

[0171] In this embodiment, the text modality decoder is used to map the joint representation Z′ into a natural language text Y TGenerally speaking, text modality decoders adopt a model structure based on sequence generation.

[0172] Specifically, the text modality decoder can be implemented using a conditional autoregressive generation model (such as a Transformer decoder). The generation process is divided into multiple steps:

[0173] Each step generates a word or subword y t , and use it as input for the next step;

[0174] By maximizing the conditional probability P(Y T |Z′) realizes the decoding process, and its conditional probability formula is:

[0175]

[0176] Where: T is the length of the generated sequence, y t Represents the tth word generated.

[0177] In this embodiment, the image modality decoder is used to map the joint representation Z′ into the image modality content Y I Typically, image modality decoders adopt structures based on generative adversarial networks (GANs) or diffusion models.

[0178] Specifically, the image modality decoder consists of a generator G and a discriminator D. The generator is responsible for generating images from Z′, and the discriminator is used to determine whether the generated image is real. The generation process is:

[0179] Y I =G(Z′)

[0180] The goal of the discriminator is to maximize the following adversarial loss:

[0181]

[0182] Where: Y real represents the real image; Y I Indicates the generated image.

[0183] As an option, the feature capture capability of the generator can be enhanced through a U-Net or Vision Transformer structure to improve the quality of the generated images.

[0184] In this embodiment, the audio modality decoder is used to map the joint representation Z′ into the audio modality content Y A Generally, audio modality decoders are implemented based on time domain or frequency domain generation methods.

[0185] The audio modality decoder first generates a Mel-spectrogram M from Z′ A, and then the spectrogram is converted into an audio signal Y through a vocoder A The calculation formula is:

[0186] Y A =Vocoder(M A )

[0187] in: represents the spectrum graph generation network; Vocoder(·) represents the vocoder model; Y A The generated audio signal.

[0188] As an option, the time alignment of the generated audio can be optimized in conjunction with a dynamic time warping (DTW) algorithm to ensure that the output signal is time synchronized with the input semantics.

[0189] In this embodiment, in order to improve the generation performance of the decoder, it is generally necessary to regularize the decoding process. For example, the distribution of the generation results can be constrained to ensure that the generated content has high diversity and semantic accuracy.

[0190] In this embodiment, after being processed by the text, image and audio modality decoders, the system generates the target modality content Y T , Y I and Y A These generated results will be used as the final output to meet the user's multimodal content generation needs.

[0191] The semantic consistency optimization module is used to optimize the joint latent representation through the variational information bottleneck theory to remove redundant information and enhance the semantic consistency between modalities;

[0192] In the Zhiqi artificial intelligence multimodal content generation system of the present invention, the semantic consistency optimization module further optimizes the multimodal decoding results or joint representations from the multimodal decoder through the semantic consistency optimization module, aiming to eliminate potential redundant information while improving the semantic consistency of the generated content. In general, due to the characteristics of different modalities and potential information loss in the decoding process, the generated results may have a certain degree of semantic deviation. This step uses the variational information bottleneck (VIB) theory to constrain the semantic relevance of the joint representation and the generated content to ensure the semantic consistency of the generated content with the input modality and user control conditions.

[0193] As an option, this module introduces a variational information bottleneck loss function to optimize the potential representation, including the compression of redundant information, the enhancement of semantic preservation, and the correction of semantic consistency. The following is a detailed description of the technical implementation of the semantic consistency optimization module.

[0194] In this embodiment, the semantic consistency optimization module receives the adjusted joint representation Z′ generated by the user control condition embedding module and the multimodal decoding result Y generated by the multimodal decoder. T , Y I and Y A The main goal of the module is to optimize Z' and generate the result Y (including Y T , Y I , Y A ), in order to maximize the semantic consistency between it and the input modality X.

[0195] In this embodiment, in order to optimize the mutual information objective function, an approximate method of a variational lower bound is used for calculation.

[0196] In general, it is difficult to directly calculate mutual information, so we optimize it by following the following steps:

[0197] First, the mutual information I(Z′;X) and I(Z′;Y) are decomposed into tractable loss terms. The variational lower bound of the mutual information I(Z′;X) is:

[0198]

[0199] q φ (Z′|X) represents the conditional distribution from the input modality X to the joint representation Z′; p(Z′) is the prior distribution of the joint representation, which is generally assumed to be a standard normal distribution; D KL represents the Kullback-Leibler divergence, which is used to measure q φ The difference in distribution between (Z′|X) and p(Z′).

[0200] For the mutual information I(Z′;Y), it can be expressed as the reconstruction error of the generated target by the following variational upper bound:

[0201]

[0202] Where: p θ (Y|Z′) is the conditional distribution from Z′ to the generated modal content Y. Combining the above objective functions, the optimized total loss function is:

[0203]

[0204] In this embodiment, in order to further reduce the interference of redundant information on semantic consistency, it is generally necessary to regularize the joint representation Z′. For example, the distribution of Z′ can be constrained by the L2 norm, and its regularization objective function is:

[0205]

[0206] The above method can effectively enhance the stability of the joint representation under different input conditions and provide a more reliable foundation for subsequent modal distribution alignment.

[0207] In this embodiment, in order to further ensure the generation of content Y T , Y I , Y A The semantic consistency of can be used to compare the distribution between the generated modality and the input modality. For example, by calculating the similarity between the feature representation of the generated modality and the feature representation of the input modality, the following loss function is optimized:

[0208]

[0209] Where: sim(·,·) represents the feature similarity measure, such as cosine similarity; f(Y) and f(X) are the feature representations of the generated modality and the input modality, respectively.

[0210] In this embodiment, after being processed by the semantic consistency optimization module, the joint representation Z′ and the generated content Y T , Y I , Y A The semantic consistency of is significantly improved. These optimized representations and contents will be passed as input to the modality distribution alignment module to further optimize the distribution alignment between modalities.

[0211] The modal distribution alignment module is used to align the input modal distribution and the generated modal distribution through the optimal transmission theory to reduce the semantic deviation between modalities;

[0212] In the Zhiqi artificial intelligence multimodal content generation system of the present invention, the modal distribution alignment module uses the modal distribution alignment module to align the joint representation Z′ and the generated content Y (including Y T , Y I , Y A ) to align the modal distribution to further reduce the distribution deviation between the input modality and the generated modality. In general, due to the differences in the characteristics of different modalities and the complexity of the generation process, there may be a certain deviation in the semantic distribution between the input modality X and the generated modality Y. To this end, this step uses the optimal transport (OT) theory to achieve distribution alignment between modalities to ensure the semantic consistency of the generated content and the high correlation between modalities.

[0213] As an option, this module strictly measures the deviation between distributions based on Wasserstein distance and minimizes the distribution difference between the input mode and the generated mode by optimizing the transfer mapping function. The following is a detailed description of the specific implementation of the modal distribution alignment module in combination with the technical implementation method.

[0214] In this embodiment, the input of the modal distribution alignment module includes the optimized joint representation Z′, the input modal distribution P X and generate the modal distribution P Y The main goal of the module is to minimize P X and P Y The distribution difference between them makes the distribution of the generated modality content Y closer to the distribution of the input modality X. In one possible implementation, the module constructs a joint distribution γ∈Π(P X ,P Y ), where Π(P X ,P Y ) indicates P X and P Y The joint distribution set of , and the deviation between distributions is measured by Wasserstein distance:

[0215]

[0216] Where: c(x,y) is the cost function between the input mode x∈X and the generated mode y∈Y, and the commonly used Euclidean distance c(x,y) = ||xy|| 2 Indicates; W c (P X ,P Y ) is the Wasserstein distance, which is used to measure the deviation between distributions.

[0217] In this embodiment, in order to efficiently calculate W c (P X ,P Y ) and achieve distribution alignment, using an optimization method based on Sinkhorn distance. Sinkhorn distance is a regularized Wasserstein distance with higher computational efficiency, which is defined as:

[0218]

[0219] Where: λ is the regularization parameter, which is used to balance the transmission cost and distribution entropy; KL(·) represents the Kullback-Leibler divergence, which is used to add regularization constraints to the joint distribution γ; represents independent distributions of input and generated modalities.

[0220] The above method can converge quickly while ensuring the alignment of the generated modal distribution with the input modal distribution.

[0221] In this embodiment, in order to further ensure that the generated modal content Y maintains high quality and diversity after distribution alignment, distribution constraints can be imposed on the generated results.Y A Gaussian distribution constraint is imposed to make it consistent with the target distribution. In one possible implementation, the distribution of the generated modalities is optimized via maximum likelihood estimation (MLE):

[0222]

[0223] Among them: p(y|Z′) represents the conditional distribution of the generated modality; Z′ is the joint representation.

[0224] As an option, adversarial learning methods (such as GAN) can be combined to further optimize the generated distribution. For example, by introducing a discriminator D to determine whether the generated mode Y comes from the target distribution P X , and achieves distribution alignment by optimizing the following adversarial loss:

[0225]

[0226] In this embodiment, in order to ensure the semantic consistency of the generated results of different modalities, the relationship between the generated modal contents can be further aligned. For example, by optimizing the following contrastive learning loss, it is ensured that the generated results of text, image, and audio modalities share the same semantic information:

[0227]

[0228] Among them: sim(·,·) represents the similarity measure between modal features, such as cosine similarity; f(Y T ) and f(Y I ) represent the feature representations of text and image modalities respectively.

[0229] The above method can significantly enhance the semantic consistency between modalities and provide higher content relevance for the final generation task.

[0230] In this embodiment, after being processed by the modal distribution alignment module, the generated multimodal content Y T , Y I , Y A The distribution of the input modality X is highly consistent, and at the same time, it has stronger semantic consistency and relevance between modalities. These optimized results will serve as the final output of the system, providing high-quality multimodal generated content.

[0231] Please refer to the attached Figure 2 The present invention also provides a method for generating multimodal content using intelligent artificial intelligence, comprising the following steps:

[0232] Receive multimodal input data, and perform feature extraction and representation encoding on different modal data (including text, image and audio) through modal encoders, and map the input data into latent semantic representations of the corresponding modalities;

[0233] The extracted latent representations of each modality are fused through the joint representation construction module, and the joint latent representation is generated using nonlinear transformation methods or attention mechanisms. The joint representation integrates different modal information in the semantic space to ensure semantic consistency among multiple modalities.

[0234] Dynamically adjust the joint latent representation according to the control conditions input by the user. After vectorizing the control conditions using an embedding network, the joint representation is adjusted in a targeted manner through weighted fusion or attention mechanism to meet the user's personalized needs for generated content.

[0235] The joint latent representation is optimized through the semantic consistency optimization module, and the variational information bottleneck theory is used to remove irrelevant redundant information, maximize the retention of the key semantics of the input data, and ensure the stability of the latent representation through regularization technology;

[0236] The modal distribution alignment module aligns the input modal distribution with the generated modal distribution based on the optimal transmission theory, minimizes the distribution deviation between the modalities, and improves the correlation between the generated modal content and the input modal content.

[0237] The target modal content is generated from the optimized joint latent representation through a multimodal decoder. The decoding process uses independent text, image and audio decoders to generate the target content respectively. The generated modal content not only meets user needs, but also reflects the semantic consistency and diversity among multiple modalities.

[0238] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. Zhiqi artificial intelligence multimodal content generation system, characterized by: include: A multimodal input module, used to receive and process multimodal input data, wherein the multimodal input data includes text modality, image modality and audio modality; A modality encoder, used to encode multimodal input data into corresponding modality latent representations; The joint representation construction module is used to fuse the latent representations of each modality to generate a joint latent representation; A user-controlled condition embedding module, which is used to dynamically adjust the joint latent representation according to the control conditions provided by the user; A multimodal decoder for generating target modality content from the adjusted joint latent representation; The semantic consistency optimization module is used to optimize the joint latent representation through the variational information bottleneck theory to remove redundant information and enhance the semantic consistency between modalities; The modal distribution alignment module is used to align the input modal distribution and the generated modal distribution through optimal transmission theory to reduce the semantic deviation between modalities.

2. The Zhiqi artificial intelligence multimodal content generation system according to claim 1 is characterized in that: The modal encoder includes a text modal encoder, an image modal encoder and an audio modal encoder, and the specific implementation steps include: The text modal encoder takes a text modal input X T Processing is performed to embed the text input into a low-dimensional semantic representation space, which specifically includes the following steps: Enter X for text T Perform word segmentation and map the word segmentation results to the word embedding matrix through the pre-trained word vector model; Use stacked multi-layer bidirectional Transformer encoders to extract text semantic features Z T , the calculation process is: Where E(X T ) represents the output of the word embedding operation, represents the text modality encoder, Z T is the potential representation of the text modality; The image modality encoder takes the image modality input X T Processing is performed to map the feature information of the image to the potential representation space, which specifically includes the following steps: For image input X T Perform normalization processing; The image features are extracted through the convolutional neural network, and the output F of the convolution operation I It is expressed as: F I =CNN(X I ) The image features are mapped to the latent space through the fully connected layer to generate the latent representation Z of the image modality I : Z I =W I F I +b I Where W I and b I are the weights and biases of the fully connected layer, Z I is the potential representation of the image modality; The audio modality encoder is used to generate the audio modality input X A Processing is performed to extract the spectral features of the audio and map them to the latent representation space, specifically including the following steps: For audio signal X A Perform preprocessing, including sampling, denoising and framing; Perform Fourier transform on the preprocessed audio signal to generate the spectrum feature S A : S A =FFT(X A ) The feature F of the spectrogram is extracted by stacking convolutional neural networks A ; Use recurrent neural networks to capture the time series information of audio features; Map the extracted spectral features into the latent representation Z of the audio modality A : Z A =Bi-LSTM(F A ) The above modality encoders are used to generate the latent representations Z of text, image and audio modalities respectively. T ,Z I ,Z A , and provide input to the subsequent joint latent representation building module.

3. The Zhiqi artificial intelligence multimodal content generation system according to claim 1 is characterized in that: The joint representation construction module fuses the potential representations of each modality into a unified joint latent space representation through a joint potential representation network, and provides a shared semantic representation for the subsequent generation of target modality content.

4. The Zhiqi artificial intelligence multimodal content generation system according to claim 1 is characterized in that: The user control condition embedding module embeds the control conditions input by the user into the joint latent representation to adjust the style, modality priority or diversity of the generated content and customize the generated content according to user needs.

5. The Zhiqi artificial intelligence multimodal content generation system according to claim 1 is characterized in that: The semantic consistency optimization module realizes semantic alignment between the input modality and the generated modality through the variational information bottleneck optimization objective. The optimization objectives include: Reduce the amount of redundant information in the joint latent representation; Enhancing the semantic relevance between latent representation and generated content.

6. The Zhiqi artificial intelligence multimodal content generation system according to claim 1 is characterized in that: The modality distribution alignment module uses the optimal transfer method to achieve modality alignment based on the Wasserstein distance between the input modality distribution and the generated modality distribution, and minimizes the semantic deviation between the modalities.

7. The Zhiqi artificial intelligence multimodal content generation system according to claim 1 is characterized in that: The multimodal decoder generates target modality content, including text, image and audio modalities, based on the joint latent representation, and improves the quality of the generated modality content through a semantic consistency optimization module.

8. The Zhiqi artificial intelligence multimodal content generation system according to claim 1 is characterized in that: The user-controlled condition embedding module dynamically adjusts the semantic compactness parameters and modal distribution alignment weights according to different user-controlled parameters to balance the diversity and semantic consistency of generated content.

9. The Zhiqi artificial intelligence multimodal content generation system according to claim 1, characterized in that: The optimization process of the system adopts a joint loss function, including semantic consistency loss, modal distribution alignment loss and inter-modal similarity loss. The joint loss function is used to simultaneously optimize the compactness of the latent representation, the semantic alignment of the generated modal content, and the semantic consistency between modalities.

10. A method for generating multimodal content using intelligent artificial intelligence, applied to a system for generating multimodal content using intelligent artificial intelligence as claimed in any one of claims 1 to 9, characterized in that: The following steps are involved: Receive multimodal input data and encode it into a latent representation of the corresponding modality through a modality encoder; The joint representation construction module is used to fuse the potential representations of each modality to generate a joint potential representation. Dynamically adjust the joint latent representation according to the control conditions input by the user; The joint latent representation is optimized through the semantic consistency optimization module to remove redundant information and enhance semantic consistency; Align the input modal distribution with the generated modal distribution through the modal distribution alignment module; The target modality content, which includes text, image, and audio, is generated from the optimized joint latent representation through a multimodal decoder.

Images