Content creation and multi-mode display platform based on AIGC

Abstract

The invention discloses a content creation and multi-mode display platform based on AIGC, and relates to the technical field of data processing. The platform comprises an interaction management module, a user behavior analysis module and a content creation module which are in communication connection. The interaction management module is used for acquiring a standard instruction word vector and a first retrieval enhancement data set according to the current input information, and generating standard intention data according to the standard instruction word vector and the context information; the user behavior analysis module is used for acquiring a second retrieval enhancement data set according to the historical content description word vector of the user; and the content creation module is used for generating multi-modal content data and corresponding content description word vectors according to the standard intention data, the first retrieval enhancement data set and the second retrieval enhancement data set. According to the method and the device, the accuracy of user intention interpretation is improved, highly related multi-modal content data can be generated for individual users, personalized content generation is provided, and the calculation burden of content creation is reduced.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a content creation and multimodal display platform based on AIGC. Background Art

[0002] With the explosive growth of digital information, users have higher demands for diversified and personalized content creation methods. Traditional methods rely on manual editing and single-mode (such as pure text or pictures) content generation. The content generation process is time-consuming and labor-intensive, with low efficiency and high dependence on professional experience.

[0003] To meet these challenges, in recent years, researchers and companies have begun to explore content generation technologies based on AIGC (generative artificial intelligence), which extracts useful information from massive data through machine learning and natural language processing technologies, and automatically generates high-quality, multimodal content to meet the diverse needs of users.

[0004] However, existing technologies still have some deficiencies in intent recognition and personalized recommendations. In particular, in the understanding of complex intents, existing systems often rely on fixed user input parsing patterns that are difficult to adjust flexibly, resulting in limited system understanding capabilities. In addition, the accuracy of personalized content generation is also limited by computing resources. The huge amount of retrieved information increases the computational complexity, resulting in slow system response and generation of personalized content in real time, which affects the quality of recommendations and user experience. Summary of the invention

[0005] The present invention provides an AIGC-based content creation and multimodal display platform to solve at least one of the problems mentioned in the above background technology.

[0006] The specific technical solutions provided by this application are as follows: An AIGC-based content creation and multimodal display platform, including a communication connection interaction management module, a user behavior analysis module, and a content creation module; The interaction management module is used to obtain a standard instruction word vector and a first retrieval enhancement data set according to current input information, and generate standard intent data according to the standard instruction word vector and context information; The user behavior analysis module is used to obtain a second retrieval enhancement data set based on the user's historical content description word vector; The content creation module is used to generate multimodal content data and its corresponding content description word vector according to the standard intent data, the first retrieval enhancement data set and the second retrieval enhancement data set; The interaction management module includes an input processing module, a context management module and an intention recognition module; the input processing module is used to process the current input information into a standard instruction word vector and a first retrieval enhanced data set; the context management module is used to generate context information based on the dialogue history instruction word vector and the content description word vector; the intention recognition module is used to generate standard intention data based on the standard instruction word vector and the context information.

[0007] As a preferred solution, the input processing module includes an instruction input module and a retrieval enhancement input module; the instruction input module is used to obtain the original operation instruction based on the current input information, and process the original operation instruction into a standard instruction word vector; the retrieval enhancement input module is used to obtain a first retrieval enhancement data set based on the current input information, construct and store the retrieval keywords corresponding to the first retrieval enhancement data set.

[0008] As a preferred embodiment, the original operation instruction is a text instruction or a voice instruction; the instruction input module includes a speech-to-text conversion module and a first word vector extraction module; the speech-to-text conversion module is used to convert the voice instruction into a text instruction through a modal conversion strategy when the original operation instruction is a voice instruction; the first word vector extraction module is used to process the text instruction through a word vector extraction strategy to obtain a standard instruction word vector of the original operation instruction.

[0009] As a preferred solution, the modal conversion strategy specifically includes: The voice command is divided into several audio frames, and each audio frame is pre-processed to obtain corresponding spectrum data; The spectrum data is passed through a Mel filter to obtain Mel spectrum data, a cepstrum analysis is performed based on the Mel spectrum data to obtain Mel frequency cepstrum coefficients, and the Mel frequency cepstrum coefficients are integrated into the current audio features; The current audio features are converted into text instructions through the speech recognition model.

[0010] As a preferred solution, the Mel spectrum data is expressed as: , , in, represents the mth Mel spectrum data, where m is the number of the Mel filter; k represents the number of the frequency point, and N represents the total number of frequency points; Represents the power spectrum of the kth frequency point of the frequency domain signal; Represents the value of the mth Mel filter at frequency point k; Indicates the center position of the mth Mel filter on the linear spectrum; The Mel-frequency cepstral coefficients are expressed as: , Where M represents the number of Mel filters; n represents the number of Mel frequency cepstrum coefficients; Represents the nth Mel-frequency cepstral coefficient value.

[0011] As a preferred solution, the word vector extraction strategy specifically includes: Use a word segmentation algorithm to process text instructions to obtain several instruction word fragments; The several instruction word fragment representations are preprocessed and then input into the Skip-gram model to obtain the standard instruction word vectors of the original operation instructions.

[0012] As a preferred solution, the method of generating standard intent data according to the standard instruction word vector and context information includes the following steps: Integrate the standard instruction word vector and context information into a number of continuous word vector sequences and word vector type sequences according to the data timestamp; the word vector sequence consists of a start identifier, an end identifier and a number of word segmentation IDs; Input the word vector sequence and the word vector type sequence into several layers of Transformer to generate a context representation of each word segmentation ID; extract the start identifier from the last layer of Transformer as a comprehensive feature of the word vector sequence; The comprehensive features are passed through the fully connected layer and the Softmax layer to output the intention recognition result with the highest probability as the standard intention data.

[0013] As a preferred solution, the user behavior analysis module includes an intention data management module, a portrait analysis module and a data matching module; the intention data management module is used to store the standard intention data and generate the user's historical intention data based on the standard intention data; the portrait analysis module is used to generate user preference features based on the user's historical content description word vectors; the data matching module is used to store the retrieval enhancement data and its corresponding feature index, and obtain a second retrieval enhancement data set by calculating the matching degree between the user preference feature and the feature index.

[0014] As a preferred solution, the portrait analysis module stores a preference word vector set, and the preference word vector set stores a number of preference word vectors; the user preference features are generated based on the user's historical content description word vectors, specifically, the user preference features are generated based on the preference word vector set and the user's historical content description word vectors; the user preference features store a number of groups of preference word vectors and their corresponding frequencies.

[0015] As a preferred solution, the content creation module includes a feature extraction module, a feature screening module and a content generation module; the feature extraction module is used to obtain data features of the first retrieval enhanced data set and the second retrieval enhanced data set; the feature screening module is used to remove parts of the data features of the first retrieval enhanced data set and the second retrieval enhanced data set that are irrelevant to the standard intent data, and obtain an intent-related feature set; the content generation module encodes the standard intent data and the intent-related feature set into the same vector space through a Transformer encoder, and generates content data of the corresponding modality and its corresponding content description word vector through an integrated number of generation models.

[0016] Compared with the prior art, the present invention has the following beneficial effects: This application obtains the standard instruction word vector according to the current input information through the interaction management module, and generates the standard intent data according to the standard instruction word vector and context information, so that the system can understand different user instructions more easily and improve the accuracy of user intent interpretation; the user behavior analysis module obtains the second retrieval enhancement data set that meets the user's preferences according to the user's historical content description word vector; the content creation module generates multimodal content data and its corresponding content description word vector according to the standard intent data, the currently input first retrieval enhancement data set and the second retrieval enhancement data set that meets the user's preferences. This application improves the accuracy of user intent interpretation, can generate highly relevant multimodal content data for individual users, to provide personalized content generation, and reduces the computational burden of content creation.

[0017] This application captures the complex dependencies between words through the Transformer architecture and self-attention mechanism, thereby improving the depth of understanding and prediction accuracy of the dialogue system in intent recognition; through the fully connected layer, feature combination and dimensionality reduction are performed, and the Softmax layer outputs the intent recognition result with the highest probability as the standard intent data, so that the system can effectively handle complex semantic tasks, and ultimately provide more accurate intent parsing and a better user interaction experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] Figure 1A schematic diagram of the structure of an AIGC-based content creation and multimodal display platform provided in one embodiment of the present invention; Figure 2 A schematic diagram of the structure of an interaction management module provided by an embodiment of the present invention; Figure 3 A schematic diagram of a process for generating standard intent data based on standard instruction word vectors and context information provided in an embodiment of the present invention; Figure 4 A schematic diagram of the structure of a user behavior analysis module provided by an embodiment of the present invention; Figure 5 A schematic diagram of the structure of a content creation module provided in one embodiment of the present invention. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0023] In addition, the descriptions of "first", "second", etc. in the present invention are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least the feature. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0024] With the explosive growth of digital information, users have higher demands for diversified and personalized content creation methods. Traditional methods rely on manual editing and single-mode (such as pure text or pictures) content generation. The content generation process is time-consuming and labor-intensive, with low efficiency and high dependence on professional experience.

[0025] To meet these challenges, in recent years, researchers and companies have begun to explore content generation technologies based on AIGC (generative artificial intelligence), which extracts useful information from massive data through machine learning and natural language processing technologies, and automatically generates high-quality, multimodal content to meet the diverse needs of users.

[0026] However, existing technologies still have some deficiencies in intent recognition and personalized recommendations. In particular, in the understanding of complex intents, existing systems often rely on fixed user input parsing patterns that are difficult to adjust flexibly, resulting in limited system understanding capabilities. In addition, the accuracy of personalized content generation is also limited by computing resources. The huge amount of retrieved information increases the computational complexity, resulting in slow system response and generation of personalized content in real time, which affects the quality of recommendations and user experience.

[0027] To this end, the present invention provides a content creation and multimodal display platform based on AIGC, and the specific implementation process of the present invention will be described in detail below.

[0028] See also Figure 1 , the present invention provides a content creation and multimodal display platform based on AIGC, including a communication connection interaction management module, a user behavior analysis module and a content creation module; The interaction management module is used to obtain a standard instruction word vector and a first retrieval enhancement data set according to current input information, and generate standard intent data according to the standard instruction word vector and context information; The user behavior analysis module is used to obtain a second retrieval enhancement data set based on the user's historical content description word vector; The content creation module is used to generate multimodal content data and its corresponding content description word vector according to the standard intent data, the first retrieval enhancement data set and the second retrieval enhancement data set; Among them, the multimodal content data can be text data, image data, audio data and video data. The first retrieval augmented data set and the second retrieval augmented data set are both used in the content creation module to implement retrieval-augmented generation (Retrieval-Augmented Generation). Retrieval-augmented generation is a model architecture that combines information retrieval and generation, and is used to improve the performance and accuracy of generative language models. Compared to relying solely on large-scale retrieval generators, the present application reduces the computational burden of the content creation module by using the first retrieval augmented data set and the second retrieval augmented data set, and provides users with more personalized multimodal content data.

[0029] This application obtains standard instruction word vectors according to current input information through the interaction management module, and generates standard intent data according to the standard instruction word vectors and context information, so that the content creation module can understand different user instructions more easily and improve the accuracy of user intent interpretation; the user behavior analysis module obtains a second retrieval enhancement data set that meets the user's preferences based on the user's historical content description word vectors; the content creation module generates multimodal content data and its corresponding content description word vectors based on the standard intent data, the currently input first retrieval enhancement data set, and the second retrieval enhancement data set that meets the user's preferences, so as to generate highly relevant multimodal content data for individual users, provide personalized content generation, and reduce the computational burden of content creation.

[0030] For further information, see Figure 2 The interaction management module includes an input processing module, a context management module and an intention recognition module. The input processing module is used to process the current input information into a standard instruction word vector and a first retrieval enhancement data set; the context management module is used to generate context information based on the dialogue history instruction word vector and the content description word vector; the intention recognition module is used to generate standard intention data based on the standard instruction word vector and the context information.

[0031] The dialogue history instruction word vector is all the standard instruction word vectors generated by the input processing module in this round of dialogue. Among them, this round of dialogue refers to the continuous question-answering or instruction exchange process between the user and the system in the same interaction. This round of dialogue starts from the first time the user inputs an instruction until the user ends the current session or stops inputting. In this round of dialogue, all standard instruction word vectors generated by the input processing module constitute the dialogue history instruction word vector.

[0032] In the interaction management module of this embodiment, the input processing module is responsible for receiving and processing the information currently input by the user, and processing the information input by the user (such as text, voice, etc.) into a standard instruction word vector and a first retrieval enhancement data set. The standard instruction word vector is an identifier of the user's specific intention, and the first retrieval enhancement data set provides rich and relevant background information. The context management module is responsible for managing the context of the conversation, summarizing all historical word vector instructions and generated multimodal content data in the current conversation round to form complete context information, providing a data basis for the system to understand the evolution of the user's intention in long-term conversations. The intention recognition module is responsible for the generation of intention data. It uses a set algorithm or model to generate standard intention data based on the standard instruction word vector and context information provided by the input processing module to accurately represent the specific needs and intentions of the current user.

[0033] Furthermore, the input processing module includes an instruction input module and a retrieval enhancement input module; the instruction input module is used to obtain original operation instructions based on current input information, and process the original operation instructions into standard instruction word vectors; the retrieval enhancement input module is used to obtain a first retrieval enhancement data set based on current input information, construct and store retrieval keywords corresponding to the first retrieval enhancement data set.

[0034] Among them, the instruction input module obtains the original operation instructions in the current input information based on the preset instruction interface, and then identifies the original operation instructions input by the user and outputs the standard instruction word vector; the output result is the standard instruction word vector to ensure that the subsequent processing modules can consistently understand and use this information. The retrieval enhancement input module obtains the first retrieval enhancement data set in the current input information based on the preset file interface, and constructs its corresponding retrieval keywords according to the modality, file name, and content keywords of the first retrieval enhancement data set, and then transmits the first retrieval enhancement data set and its corresponding retrieval keywords to the designated storage system or cache for quick access. The first retrieval enhancement data set can be data in the form of text, voice, image, video, etc.

[0035] In one embodiment, the original operation instruction is a text instruction or a voice instruction; the instruction input module includes a voice-to-text conversion module and a first word vector extraction module. The voice-to-text conversion module is used to convert the voice instruction into a text instruction through a modal conversion strategy when the original operation instruction is a voice instruction; the first word vector extraction module is used to process the text instruction through the word vector extraction strategy to obtain a standard instruction word vector of the original operation instruction.

[0036] Furthermore, the modal conversion strategy specifically includes: A1. Divide the voice command into several audio frames, and pre-process each audio frame to obtain corresponding spectrum data; The voice command is divided into several audio frames, specifically, the continuous voice command is divided into several short time periods, each of which is recorded as an audio frame. In one embodiment, the length of each frame is 25 milliseconds. Each audio frame is preprocessed, including denoising, normalization and fast Fourier transform operations. Denoising and normalization are performed to improve signal quality and reduce environmental noise and other interference. Fast Fourier transform converts the time domain signal of each audio frame into a frequency domain signal to obtain the spectrum data corresponding to each audio frame.

[0037] A2. Obtain Mel frequency spectrum data through a Mel filter, perform cepstrum analysis based on the Mel frequency spectrum data to obtain Mel frequency cepstrum coefficients, and integrate the Mel frequency cepstrum coefficients into the current audio features; The spectrum data is passed through a set number of Mel filters and output as Mel spectrum data, which represents the energy distribution of the spectrum on the Mel scale. Then, the Mel spectrum is subjected to discrete cosine transform (DCT) to obtain Mel frequency cepstral coefficients (MFCCs), which represent the characteristics of the speech signal and remove correlations for subsequent processing. Finally, the Mel frequency cepstral coefficients corresponding to all audio frames are integrated into the current audio feature. The current audio feature is represented in the form of a vector, which contains the main information that can describe the voice command.

[0038] The Mel spectrum data is expressed as: , , in, represents the mth Mel spectrum data, where m is the number of the Mel filter; k represents the number of the frequency point, and N represents the total number of frequency points; Represents the power spectrum of the kth frequency point of the frequency domain signal; Represents the value of the mth Mel filter at frequency point k; Indicates the center position of the mth Mel filter on the linear spectrum.

[0039] The Mel-frequency cepstral coefficients are expressed as: , Where M represents the number of Mel filters; n represents the number of Mel frequency cepstrum coefficients; Represents the nth Mel frequency cepstral coefficient value; A3. Convert the current audio features into text instructions through the speech recognition model; The audio features are processed using a speech recognition model (e.g., a deep neural network) deployed in the command input module. The speech recognition model has been trained with a large amount of voice and text data and is able to recognize and convert different phonemes. The current audio features are input into the speech recognition model, and the model outputs the corresponding text sequence, which is then integrated into a text command.

[0040] Furthermore, the word vector extraction strategy specifically includes: B1. Use a word segmentation algorithm to process text instructions to obtain several instruction word fragments; wherein the text instruction can be the original operation instruction obtained according to the current input information, or it can be the text instruction obtained by converting the voice instruction by the voice-to-text conversion module through the modal conversion strategy. The word segmentation algorithm can be based on the existing Jieba, NLTK and other word segmentation methods to perform word segmentation processing on the text instructions to obtain instruction word fragments. Among them, after using the word segmentation algorithm to process the text instruction, operations such as removing punctuation and stop words can also be included.

[0041] B2. The plurality of instruction word fragments are pre-processed and input into the Skip-gram model to obtain the standard instruction word vector of the original operation instruction.

[0042] Among them, the Skip-gram model is an implementation method of the lightweight neural network model Word2Vec, which is used to predict the context word vector based on a given word and is suitable for generating word vector representation. The Skip-gram model includes an input layer, a hidden layer, and an output layer, where the size of the input layer is set to a one-hot vector of the vocabulary length, the dimension of the hidden layer is a dense vector of embedding_size, which is used for word vector representation, and the size of the output layer is set to a softmax layer of the vocabulary length. Using the Skip-gram model in the input processing stage can effectively generate word vectors that can represent the semantics of words, thereby improving the semantic understanding and similarity calculation of words.

[0043] See also Figure 3 In one embodiment, the intent recognition module generates standard intent data according to the standard instruction word vector and context information, including the steps of: C1. Integrate the standard instruction word vector and context information into several continuous word vector sequences and word vector type sequences according to the data timestamp; the word vector sequence is composed of a start mark, an end mark and several segmentation IDs; based on the above content, the context information is generated by the conversation history instruction word vector and the content description word vector, so the standard instruction word vector and the context information can be integrated into a word vector sequence in a unified format in this step. Among them, the word vector sequence can correspond to a standard instruction word vector, a conversation history instruction word vector or a content description word vector. Specifically, the word vector sequence is composed of a start mark, an end mark and several segmentation IDs. For example, the text content corresponding to the word vector sequence is "Generate a dynamic picture about climate change based on the text generated in the previous article, and the output format needs to be GIF", and the word vector sequence is expressed as [101, 3306, 736, ... , 102], 101 corresponds to the start mark, 102 corresponds to the end mark, and the numbers in the word vector sequence are the preset segmentation IDs. Based on the aforementioned text content, the word vector type sequence is used to mark the sentence type to which the word segmentation belongs. For example, user input is recorded as type 0 and system output is recorded as type 1.

[0044] In this step, by integrating the standard instruction word vectors and context information into a unified format of word vector sequences and type sequences, the module can process input data from different sources and ensure that the information is processed under the same structure to simplify subsequent processing, enhance the model's understanding of information types, and help improve the accuracy of intent recognition.

[0045] C2, input the word vector sequence and word vector type sequence into several layers of Transformer to generate a context representation of each word segmentation ID; extract the start identifier from the last layer of Transformer as a comprehensive feature of the word vector sequence; In an exemplary embodiment, the word vector sequence and the word vector type sequence are input into the BERT model, and several Transformer layers in the BERT model capture the relationship between each word and other words in the input word vector sequence through the self-attention mechanism, so that the model can pay attention to the contextual information in the input sequence. Through several Transformer layers, each word segmentation ID finally generates a contextual representation, that is, the embedding vector of the word, which contains the contextual information of the entire sentence or paragraph. Since the contextual representation of the start mark in the BERT model is designed to contain the semantic information of the entire sequence, the contextual representation of the start mark (denoted as [CLS] or 101) is extracted from the Transformer output of the last layer as a comprehensive feature of the entire input sequence or sentence.

[0046] C3. Output the highest probability intention recognition result of the comprehensive features through the fully connected layer and the Softmax layer as the standard intention data.

[0047] In this step, the fully connected layer further processes and reduces the dimensions of the comprehensive features through linear transformation and nonlinear activation function. The Softmax layer is used to normalize the output of the fully connected layer, convert the result into the probability distribution of several intent recognition results, and then use the intent recognition result with the highest probability as the standard intent data.

[0048] This embodiment captures the complex dependencies between words through the Transformer architecture and self-attention mechanism, thereby improving the understanding depth and prediction accuracy of the dialogue system in intent recognition. By combining features and reducing dimensions through the fully connected layer and outputting the highest probability intent recognition result as standard intent data through the Softmax layer, the system can effectively handle complex semantic tasks and ultimately provide more accurate intent parsing and a better user interaction experience.

[0049] For further information, see Figure 4 The user behavior analysis module includes an intention data management module, a portrait analysis module and a data matching module; the intention data management module is used to store the standard intention data and generate the user's historical intention data according to the standard intention data; the portrait analysis module is used to generate user preference features according to the user's historical content description word vector; the data matching module is used to store the retrieval enhancement data and its corresponding feature index, and obtain the second retrieval enhancement data set by calculating the matching degree between the user preference feature and the feature index.

[0050] In this embodiment, the intent data management module stores the standard intent data generated by the interaction management module, and creates and maintains the historical intent data corresponding to each user based on the standard intent data to provide reliable data storage and management support, ensuring that all user intent data is effectively recorded and updated. The portrait analysis module stores the user's historical content description word vector, that is, the content description word vector corresponding to the multimodal content data provided to the user by the content creation module in all the interaction data generated by the user and this platform, and extracts the user preference features based on the historical content description word vector. The user preference features can be used as user portraits, so that the platform can better personalize content recommendations. The data matching module facilitates fast query and matching by storing and managing retrieval enhancement data and its feature index; and calculates the matching degree between the user preference features and the retrieval data feature index, and selects the most matching second retrieval enhancement data set.

[0051] In one embodiment, the portrait analysis module stores a preference word vector set, and the preference word vector set stores a plurality of preference word vectors; the user preference features are generated based on the user's historical content description word vectors, specifically, the user preference features are generated based on the preference word vector set and the user's historical content description word vectors; the user preference features store a plurality of groups of preference word vectors and their corresponding frequencies.

[0052] Among them, the preference word vector set is stored in the database of the portrait analysis module. Each preference word vector in the preference word vector set represents a potential user interest or preference. The word vector may be extracted from the content description word vector output by the content creation module for all user groups through a set statistical algorithm, which can capture typical user tendencies or preferences. User preference features are recorded in the form of preference word vectors and their corresponding frequencies to help the system understand the number of times a preference is selected or reflected by the user's historical behavior, thereby measuring the significance of these preferences and helping to make more accurate decisions when recommending personalized content.

[0053] In one embodiment, the step of obtaining the second search enhancement data set by calculating the matching degree between the user preference feature and the feature index comprises the following steps: The user preference features and feature indexes are processed by a filling method. In this embodiment, both the user preference features and feature indexes can be represented in the form of vectors. Since the vector dimensions of the user preference features and feature indexes may be different, it is necessary to fill the dimensions of the shorter vectors in the user preference features and feature indexes to be consistent with the dimensions of the longer vectors.

[0054] Calculate the cosine similarity between the user preference feature and each feature index; in this step, the matching degree is measured by cosine similarity, that is, the cosine of the angle between two vectors is calculated to measure the degree of directional similarity between the two vectors.

[0055] The retrieval enhancement data corresponding to several feature indexes with the highest cosine similarity are obtained and integrated into a second retrieval enhancement data set.

[0056] This embodiment processes user preference features and feature indexes through the filling method, so that this solution is not limited to the scenario of fixed feature dimensions; by calculating the cosine similarity between the user preference features and each feature index, several retrieval enhancement data that best matches the user preference features can be found, thereby improving the correlation between the user preference features and the matched second retrieval enhancement data set, providing a data basis for personalized content generation.

[0057] In one embodiment, please refer to Figure 5 The content creation module includes a feature extraction module, a feature screening module and a content generation module; the feature extraction module is used to obtain data features of the first retrieval enhanced data set and the second retrieval enhanced data set; the feature screening module is used to remove the parts of the data features of the first retrieval enhanced data set and the second retrieval enhanced data set that are irrelevant to the standard intent data, and obtain an intent-related feature set; the content generation module encodes the standard intent data and the intent-related feature set into the same vector space through a Transformer encoder, and generates content data of the corresponding modality and its corresponding content description word vector through an integrated number of generation models.

[0058] Based on the above embodiment, the first retrieval enhancement data set and the second retrieval enhancement data set can be retrieval enhancement data of different modalities; the feature extraction module can extract features from the first retrieval enhancement data set and the second retrieval enhancement data set through a preset model. For example, image feature data is extracted through a convolutional neural network, image feature data of a 3D convolutional neural network video, text feature data is extracted through models such as Word2Vec and GPT, and audio feature data is extracted by combining Mel frequency cepstral coefficients with convolutional neural networks. The feature screening module, based on the features extracted by the above feature extraction module, retains the features related to the standard intent data through alignment analysis and correlation measurement to obtain an intent-related feature set. The content generation module, based on the Transformer encoder, independently encodes the features in the standard intent data and the intent-related feature set, and then merges the encoding results into the same vector space; in the same vector space, the target modality of the content data is selected according to the standard intent data, the preset generation model is determined according to the target modality, and then the content data of the corresponding modality and its corresponding content description word vector are generated through the generation model. Among them, the target modality can be one or more, and each target modality corresponds to a generation model.

[0059] This embodiment obtains data features of retrieval enhancement data through a feature extraction module, and removes data features that are irrelevant to standard intent data through a feature screening module, so as to reduce noise and irrelevant features, while improving the model's ability to accurately capture and understand intent; the features are encoded into the same vector space through a content generation module, and then content data of the corresponding modality and its corresponding content description word vector are generated based on several integrated generation models, thereby achieving efficient, accurate and unified multimodal content data generation and ensuring the quality and accuracy of the content data.

[0060] In the embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the modules can be electrical, mechanical or other forms.

[0061] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0062] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0063] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

[0064] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A content creation and multimodal display platform based on AIGC, characterized by: It includes an interactive management module for communication connections, a user behavior analysis module, and a content creation module; The interaction management module is used to obtain a standard instruction word vector and a first retrieval enhancement data set according to current input information, and generate standard intent data according to the standard instruction word vector and context information; The user behavior analysis module is used to obtain a second retrieval enhancement data set based on the user's historical content description word vector; The content creation module is used to generate multimodal content data and its corresponding content description word vector according to the standard intent data, the first retrieval enhancement data set and the second retrieval enhancement data set; The interaction management module includes an input processing module, a context management module and an intention recognition module; the input processing module is used to process the current input information into a standard instruction word vector and a first retrieval enhancement data set; The context management module is used to generate context information based on the dialogue history instruction word vector and the content description word vector; The intent recognition module is used to generate standard intent data based on standard instruction word vectors and context information.

2. The AIGC-based content creation and multimodal display platform according to claim 1, characterized in that: The input processing module includes an instruction input module and a search enhancement input module; the instruction input module is used to obtain the original operation instruction according to the current input information, and process the original operation instruction into a standard instruction word vector; The search enhancement input module is used to obtain a first search enhancement data set according to current input information, construct and store search keywords corresponding to the first search enhancement data set.

3. The AIGC-based content creation and multimodal display platform according to claim 2, characterized in that: The original operation instruction is a text instruction or a voice instruction; the instruction input module includes a speech-to-text conversion module and a first word vector extraction module; the speech-to-text conversion module is used to convert the voice instruction into a text instruction through a modal conversion strategy when the original operation instruction is a voice instruction; the first word vector extraction module is used to process the text instruction through a word vector extraction strategy to obtain a standard instruction word vector of the original operation instruction.

4. The AIGC-based content creation and multimodal display platform according to claim 3, characterized in that: The modal conversion strategy specifically includes: The voice command is divided into several audio frames, and each audio frame is pre-processed to obtain corresponding spectrum data; The spectrum data is passed through a Mel filter to obtain Mel spectrum data, a cepstrum analysis is performed based on the Mel spectrum data to obtain Mel frequency cepstrum coefficients, and the Mel frequency cepstrum coefficients are integrated into the current audio features; The current audio features are converted into text instructions through the speech recognition model.

5. The AIGC-based content creation and multimodal display platform according to claim 4, characterized in that: The Mel spectrum data is represented as: ， ， in, represents the mth Mel spectrum data, where m is the number of the Mel filter; k represents the number of the frequency point, and N represents the total number of frequency points; Represents the power spectrum of the kth frequency point of the frequency domain signal; Represents the value of the mth Mel filter at frequency point k; Indicates the center position of the mth Mel filter on the linear spectrum; The Mel-frequency cepstral coefficients are expressed as: ， Where M represents the number of Mel filters; n represents the number of Mel frequency cepstrum coefficients; Represents the nth Mel-frequency cepstral coefficient value.

6. The AIGC-based content creation and multimodal display platform according to claim 3, characterized in that: The word vector extraction strategy specifically includes: Use a word segmentation algorithm to process text instructions to obtain several instruction word fragments; The several instruction word fragment representations are preprocessed and then input into the Skip-gram model to obtain the standard instruction word vectors of the original operation instructions.

7. The AIGC-based content creation and multimodal display platform according to claim 1, characterized in that: The method of generating standard intent data according to the standard instruction word vector and context information comprises the following steps: Integrate the standard instruction word vector and context information into a number of continuous word vector sequences and word vector type sequences according to the data timestamp; the word vector sequence consists of a start identifier, an end identifier and a number of word segmentation IDs; Input the word vector sequence and the word vector type sequence into several layers of Transformer to generate a context representation of each word segmentation ID; extract the start identifier from the last layer of Transformer as a comprehensive feature of the word vector sequence; The comprehensive features are passed through the fully connected layer and the Softmax layer to output the intention recognition result with the highest probability as the standard intention data.

8. The AIGC-based content creation and multimodal display platform according to claim 1, characterized in that: The user behavior analysis module includes an intention data management module, a portrait analysis module and a data matching module; the intention data management module is used to store the standard intention data and generate the user's historical intention data based on the standard intention data; the portrait analysis module is used to generate user preference features based on the user's historical content description word vectors; the data matching module is used to store the retrieval enhancement data and its corresponding feature index, and obtain a second retrieval enhancement data set by calculating the matching degree between the user preference feature and the feature index.

9. The AIGC-based content creation and multimodal display platform according to claim 8, characterized in that: The portrait analysis module stores a set of preference word vectors, and the preference word vector set stores a number of preference word vectors; the user preference features are generated based on the user's historical content description word vectors, specifically, the user preference features are generated based on the preference word vector set and the user's historical content description word vectors; the user preference features store a number of groups of preference word vectors and their corresponding frequencies.

10. The AIGC-based content creation and multimodal display platform according to claim 1, characterized in that: The content creation module includes a feature extraction module, a feature screening module and a content generation module; the feature extraction module is used to obtain data features of the first retrieval enhancement data set and the second retrieval enhancement data set; The feature screening module is used to remove the parts of the data features of the first retrieval enhanced data set and the second retrieval enhanced data set that are irrelevant to the standard intent data, so as to obtain an intent-related feature set; The content generation module encodes the standard intent data and intent-related feature set into the same vector space through a Transformer encoder, and generates content data of the corresponding modality and its corresponding content description word vector through an integrated number of generation models.

Images