A text generation method and device based on multi-modal data, equipment and medium
By receiving multimodal data to generate user profile information, the problem that traditional text generation systems cannot meet users' personalized needs is solved, realizing more personalized text generation that meets user needs and improving user experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA UNITED NETWORK COMM GRP CO LTD
- Filing Date
- 2024-11-27
- Publication Date
- 2026-06-05
AI Technical Summary
Traditional text generation systems cannot effectively capture and reflect users' personal preferences and needs, resulting in generated text content that fails to meet users' personalized requirements.
By receiving multimodal data (including text, images, and voice data) to generate sentiment feature vectors, fusing and processing the multimodal data and sentiment feature vectors to generate user profile information, and using a large language model to generate personalized text.
The generated personalized text better meets user needs, improving the user experience and the intelligence level of personalized services.
Smart Images
Figure CN122154893A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device and medium for text generation based on multimodal data. Background Technology
[0002] With the rapid development of internet and artificial intelligence technologies, personalized services have gradually become a core requirement for various applications. Especially in the field of text generation, how to provide users with text content that meets their personalized needs has become an important research topic.
[0003] Traditional text generation systems often rely on fixed templates or generic models, failing to effectively capture and reflect users' individual preferences and needs, resulting in generated text content that falls short of user expectations. Therefore, providing users with text content that meets their personalized needs is a pressing issue that needs to be addressed. Summary of the Invention
[0004] This application provides a text generation method, apparatus, device, and medium based on multimodal data to solve the problem of being unable to provide users with text content that meets their personalized needs.
[0005] Firstly, this application provides a text generation method based on multimodal data, including:
[0006] Receive multimodal data from users; wherein, multimodal data includes at least one of text data, image data, and voice data;
[0007] Based on multimodal data, an emotional feature vector is obtained; whereby the emotional feature vector represents the user's emotional tendency.
[0008] Multimodal data and sentiment feature vectors are fused to obtain user profile information; the profile information represents the user's habits and style.
[0009] Personalized text is generated based on the user's profile information; the personalized text is text content that the user is interested in.
[0010] Optionally, the method described above can be used to fuse multimodal data and sentiment feature vectors to obtain user profile information, including:
[0011] Multimodal data and sentiment feature vectors are fused to obtain fused features; whereby the fused features represent both multimodal data and sentiment feature vectors.
[0012] Based on the fusion characteristics, determine the user's profile information.
[0013] Optionally, the method described above can be used to fuse multimodal data and sentiment feature vectors to obtain fused features, including:
[0014] Feature extraction is performed on the multimodal data to obtain the multimodal vectors corresponding to the multimodal data;
[0015] Multiple target vectors are obtained by filtering and processing the multimodal vectors and sentiment feature vectors;
[0016] Multiple target vectors are fused to obtain fused features.
[0017] Optionally, the method described above can be used to filter the multimodal vectors and sentiment feature vectors to obtain multiple target vectors, including:
[0018] Both multimodal data and sentiment feature vectors were identified as vectors to be screened.
[0019] Determine the attention coefficient between two vectors to be screened; where the attention coefficient characterizes the correlation between the two vectors.
[0020] The target vector is determined from all the vectors to be selected based on all the attention coefficients.
[0021] Optionally, the above method can be used to fuse multiple target vectors to obtain fused features, including:
[0022] Determine the attention coefficient between two target vectors;
[0023] Based on the attention coefficient between the two target vectors and the two target vectors themselves, determine the update features corresponding to the two target vectors;
[0024] All updated features are processed using a fully connected layer to obtain fused features.
[0025] Optionally, the above method determines user profile information based on fused features, including:
[0026] Based on multiple preset image labels, the fused features are classified to obtain the feature vector corresponding to each image label.
[0027] Based on the feature vectors corresponding to each profile tag, the user's profile information is determined.
[0028] Optionally, the above method generates personalized text based on the user's profile information, including:
[0029] The user's profile information is input into a pre-defined large language model to obtain personalized output text.
[0030] Alternatively, the method described above may also include:
[0031] Receive user feedback on personalized text; where the feedback information represents the user's level of satisfaction with the personalized text;
[0032] Based on the feedback information, the user profile information is updated to obtain new profile information;
[0033] New personalized text is generated based on the new profile information.
[0034] Secondly, this application provides a text generation apparatus based on multimodal data, comprising:
[0035] The receiving module is used to receive multimodal data from users; wherein the multimodal data includes at least one of text data, image data, and voice data.
[0036] The first module is used to obtain the sentiment feature vector based on multimodal data; wherein the sentiment feature vector represents the user's sentiment tendency.
[0037] The second module is used to fuse multimodal data and sentiment feature vectors to obtain user profile information; whereby the profile information represents the user's habits and style.
[0038] The generation module is used to generate personalized text based on the user's profile information; the personalized text is text content that the user is interested in.
[0039] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;
[0040] The memory stores the instructions that the computer executes;
[0041] The processor executes computer execution instructions stored in memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.
[0042] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.
[0043] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.
[0044] This application provides a text generation method, apparatus, device, and medium based on multimodal data. It receives multimodal data from users, including at least one of text, image, and voice data. By collecting multimodal data, the data types are enriched, resulting in more accurate user profile information. Based on the multimodal data, a sentiment feature vector is obtained, representing the user's emotional tendency. Sentiment tendency analysis of the multimodal data increases the analytical dimensions, further improving the accuracy of the user profile information. The multimodal data and sentiment feature vector are fused to obtain user profile information, representing the user's habits and style. Personalized text is generated based on the user profile information, containing text content that interests the user. Through the fusion of multimodal data and sentiment tendency, the obtained user profile information is more comprehensive and richer, making the generated personalized text more in line with user needs and improving the user experience. Attached Figure Description
[0045] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0046] Figure 1 A flowchart illustrating a text generation method based on multimodal data provided in this application embodiment;
[0047] Figure 2 A flowchart illustrating a text generation method based on multimodal data provided in this application embodiment;
[0048] Figure 3 A schematic diagram of a fusion feature provided in an embodiment of this application;
[0049] Figure 4 A schematic diagram of the structure of a text generation device based on multimodal data provided in this application embodiment;
[0050] Figure 5 This is a schematic diagram of the electronic device structure provided in an embodiment of this application.
[0051] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation
[0052] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0053] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.
[0054] Most current text generation technologies do not consider user profiling, typically employing generic models for generation tasks. A few methods do consider building user profiles, but these rely on static user data, failing to update and reflect changes in user interests in real time. This results in generated text content that struggles to keep pace with users' latest needs and preferences. Furthermore, due to the generalization of user profiles and the universality of models, the generated text often lacks depth and personalization, failing to fully reflect users' unique interests and behavioral habits. Moreover, while using big data technologies to enhance personalization, it can easily lead to a decline in the quality of generated text, resulting in logical inconsistencies or poor expression, negatively impacting user experience.
[0055] This application provides a text generation method, apparatus, device, and medium based on multimodal data. It receives multimodal data from users, including at least one of text, image, and voice data. By collecting multimodal data, the data types are enriched, resulting in more accurate user profile information. Based on the multimodal data, a sentiment feature vector is obtained, representing the user's emotional tendency. Sentiment tendency analysis of the multimodal data increases the analytical dimensions, further improving the accuracy of the final user profile information. The multimodal data and sentiment feature vector are fused to obtain user profile information, representing the user's habits and style. Personalized text is generated based on the user profile information, containing content of interest to the user. Through the fusion of multimodal data and sentiment tendency, the obtained user profile information is more comprehensive and richer, making the generated personalized text more aligned with user needs and improving the user experience.
[0056] First, let me explain the terms used in this application:
[0057] Multimodal data refers to data from multiple different modes or sources, which can include text, images, audio, video, sensor data, etc.
[0058] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.
[0059] Figure 1 This application provides a text generation method based on multimodal data. For example... Figure 1 As shown, the subject of this method can be an electronic device; however, this embodiment does not impose any special limitations on it. Figure 1 As shown, the method includes:
[0060] S101. Receive multimodal data from the user; wherein the multimodal data includes at least one of text data, image data, and voice data.
[0061] This system can acquire user-related data in real-time or at set intervals, including multimodal data such as text, images, and voice. Text data may include the user's current input, text from historical interactions, and text from basic user information. Image data may include the user's current image input and facial recognition data. Voice data may include the user's current voice input.
[0062] S102. Based on the multimodal data, obtain the sentiment feature vector; whereby the sentiment feature vector represents the user's sentiment tendency.
[0063] The emotional feature vector uses 0-7 to represent eight categories, ranging from positive to negative emotions. Specific categories include happiness, surprise, sadness, anger, calmness, anticipation, fear, and disgust. By analyzing data from multiple modalities, information related to a user's emotions can be determined, thus obtaining the emotional feature vector.
[0064] For example, for text data, a multimodal large model can be used to classify the text data, identify sentiment tendencies, and obtain sentiment features. For speech data, the speech data is converted into text data, and then a multimodal large model is used to classify the text data and identify sentiment tendencies. For image data involving faces, the FER (Facial Expression Recognition) model is used for sentiment classification to obtain sentiment features. Alternatively, the three types of multimodal data can be combined, and the final sentiment feature vector can be determined based on the pre-defined weights of each data modality. For example, the emotion identified from facial images can be the primary factor, supplemented by the emotions identified from text and speech, to obtain the final sentiment feature vector. Sentiment features are vectorized using One-Hot (One-Hot Encoding), divided into eight categories, from positive to negative emotions: happiness, surprise, sadness, anger, calmness, anticipation, fear, and disgust. One-Hot encoding is a technique for converting categorical data into numerical data, commonly used in input processing for machine learning and deep learning models. Its main purpose is to convert discrete categorical variables into a numerical format that can be input into the model.
[0065] A multimodal large model refers to an artificial intelligence model capable of processing and generating multiple types of data (such as text, images, and audio). By combining data from different modalities, multimodal large models can achieve more complex and comprehensive tasks, such as text and image generation, video understanding, and speech recognition. In this embodiment, the model structure of the multimodal large model is not specifically limited.
[0066] S103. The multimodal data and sentiment feature vectors are fused to obtain user profile information; the profile information represents the user's habits and style.
[0067] In this application, user profiles are represented by a user profile graph. All user profiles are based on this graph. The adjacency matrix can be a form of graph representation. The features (i.e., nodes) in the user profile graph represent user profile information. If historical user profile data exists, it is necessary to read the historical user profile data and use it as a feature (i.e., node) in the user profile graph to obtain more accurate user profile information.
[0068] User profile information includes: user interests and preferences, emotional state, language style, etc.
[0069] By assigning different weights to various multimodal data and sentiment feature vectors, the multimodal data and sentiment feature vectors are fused to obtain fused features. User profile information is then derived based on these fused features. The model can then extract features from the fused features to obtain the user profile information.
[0070] In this embodiment of the application, multimodal data and sentiment feature vectors are fused to obtain user profile information, including:
[0071] Multimodal data and sentiment feature vectors are fused to obtain fused features; whereby the fused features represent both multimodal data and sentiment feature vectors.
[0072] Based on the fusion characteristics, determine the user's profile information.
[0073] The fusion feature is a high-dimensional vector that combines multimodal data and sentiment feature vectors.
[0074] By assigning different weights to different multimodal data and sentiment feature vectors, and then using the weights corresponding to the multimodal data and sentiment feature vectors, a fused feature is obtained. For example, a weighted summation process can be performed. Feature extraction is then performed on the fused feature using a pre-set model to obtain user profile information. The pre-set model can be based on a pre-trained Transformer model. The Transformer model is a deep learning model architecture.
[0075] The advantage of this setup is that by fusing multimodal data and sentiment feature vectors, a more comprehensive and accurate user profile can be created.
[0076] In this embodiment of the application, multimodal data and sentiment feature vectors are fused to obtain fused features, including:
[0077] Feature extraction is performed on the multimodal data to obtain the multimodal vectors corresponding to the multimodal data;
[0078] Multiple target vectors are obtained by filtering and processing the multimodal vectors and sentiment feature vectors;
[0079] Multiple target vectors are fused to obtain fused features.
[0080] The filtering process refers to determining the two target vectors that need to be connected.
[0081] The target vector represents all target vector pairs that need to be connected.
[0082] All multimodal vectors and sentiment feature vectors are bidirectionally connected to obtain an initial adjacency matrix. A node in the initial adjacency matrix is one of the multimodal data and the sentiment feature vector (i.e., any one of text data, image data, language data, and sentiment feature vector). The edges in the initial adjacency matrix represent that any node is connected to all other node vectors in a bidirectional manner. The initial adjacency matrix is then pruned, retaining only the corresponding edges that meet the preset conditions for each multimodal vector and sentiment feature vector (i.e., determining the target vectors to be connected). Then, multiple target vectors are fused to obtain a fused feature, in which the nodes are unidirectionally connected. Meeting the preset conditions can mean that the two edges with the highest attention coefficients (the values can be preset) for each node (referring to multimodal vectors or sentiment feature vectors) are selected. For example, given nodes A, B, C, and D, the attention coefficient between nodes A and B is X, between nodes A and C is Y, and between nodes A and D is Z, where X > Y > Z. In this case, the edges between A and C, and between A and B, are retained. Meeting the preset conditions can also mean that the attention coefficient threshold is greater than a preset value. For example, if the attention coefficient X between nodes A and B is greater than the threshold, the edge between nodes A and B is retained, and the value of the edge between nodes A and B is the attention coefficient X. For example, given four features: speech feature 1, text feature 2, emotion feature 3, and image feature 4, speech feature 1 needs to be bidirectionally fully connected with the other three features. The same applies to text feature 2, emotion feature 3, and image feature 4. The attention coefficient A between speech feature 1 and text feature 2, the attention coefficient B between speech feature 1 and emotion feature 3, and the attention coefficient C between speech feature 1 and image feature 4 are calculated. If attention coefficient A > attention coefficient C > attention coefficient B, then speech feature 1 is connected to text feature 2, speech feature 1 is connected to text feature 2, and speech feature 1 is connected to image feature 4. The same applies to text feature 2, emotion feature 3, and image feature 4. Finally, multiple target vectors are obtained. These target vectors are then fused to obtain the fused feature. The fusion process can refer to weighted summation, followed by a fully connected operation on all the results of the weighted summation.
[0083] The advantage of this setup is that by effectively integrating multimodal data and sentiment features, the generated profile can be more accurate.
[0084] In this embodiment of the application, the multimodal vectors and sentiment feature vectors are filtered to obtain multiple target vectors, including:
[0085] Both multimodal data and sentiment feature vectors were identified as vectors to be screened.
[0086] Determine the attention coefficient between two vectors to be screened; where the attention coefficient characterizes the correlation between the two vectors.
[0087] The target vector is determined from all the vectors to be selected based on all the attention coefficients.
[0088] In this process, the vectors to be selected are those for which attention coefficients need to be calculated. Multimodal data and sentiment feature vectors are combined in pairs, and each pair constitutes a vector to be selected. The corresponding target vectors are also vectors formed by combining pairs of vectors.
[0089] The formula for obtaining the attention coefficient is:
[0090]
[0091] Where, α ij This represents the attention coefficient of the one-way edge between node i and node j. Represents the eigenvalue of node i. Represents the eigenvalues of node j. W represents the feature values of the neighboring nodes of node i. A node represents any one of the text vector, speech vector, image vector, and sentiment feature vector. W is a projection matrix (shared weight matrix). It is a learnable parameter vector (feedforward neural network parameters). W represents the adjacent nodes (i.e., connected nodes) of node i. The LeakyReLU (Leaky Rectified Linear Unit) is a commonly used activation function in deep learning. It is an improvement on the traditional ReLU (Rectified Linear Unit) function, and exp represents the natural exponential function.
[0092] Choose any two vectors from the multimodal vectors and sentiment feature vectors as nodes i and j. Identify nodes i and j (i.e., the vectors to be selected). Calculate the attention coefficients of nodes j and i using the formula for obtaining the attention coefficient. When the attention coefficient is greater than a preset threshold, confirm nodes i and j as target vectors. Then, calculate the attention coefficients of all pairs of multimodal vectors and sentiment feature vectors to obtain all target vectors. Alternatively, calculate the attention coefficients X1, X2, ..., Xn of node i and all other nodes. From these attention coefficients X1, X2, ..., Xn, identify the five largest attention coefficients (values can be customized): Xmax1, Xmax2, Xmax3, Xmax4, and Xmax5. The nodes a and b, a and b, a and c, a and d, and a and e corresponding to Xmax1, Xmax2, Xmax3, Xmax4, and Xmax5 are the target vectors.
[0093] The advantage of this setup is that by using an attention mechanism to filter multimodal and sentiment feature vectors, more efficient and accurate feature extraction and selection can be achieved in complex data environments, thereby improving the overall system performance and application effectiveness.
[0094] In this embodiment of the application, multiple target vectors are fused to obtain fused features, including:
[0095] Determine the attention coefficient between two target vectors;
[0096] Based on the attention coefficient between the two target vectors and the two target vectors themselves, determine the update features corresponding to the two target vectors;
[0097] All updated features are processed using a fully connected layer to obtain fused features.
[0098] The process involves obtaining multiple pairs of target vectors for updating features. These two target vectors must satisfy certain conditions, such as an attention coefficient between them exceeding a preset threshold. For example, two vectors are randomly selected from the multimodal vectors and sentiment feature vectors as nodes i and j. Nodes i and j (i.e., the vectors to be selected) are then identified. The attention coefficient between nodes j and i is calculated using the formula for obtaining the attention coefficient. If the attention coefficient exceeds the preset threshold, nodes i and j are identified as target vectors. This process is repeated for all pairs of multimodal vectors and sentiment feature vectors to obtain all target vectors. The condition can also be met if the attention coefficients between two vectors are among the largest at a certain node. Alternatively, the attention coefficients X1, X2, ..., Xn of node i and all other nodes can be calculated. From these X1, X2, ..., Xn, the five largest attention coefficients (which can be customized) Xmax1, Xmax2, Xmax3, Xmax4, and Xmax5 are identified. The corresponding nodes i-node a, node i-node b, node i-node c, node i-node d, and node i-node e are then determined as the target vectors. This yields the target vectors associated with node i, and so on, for all other nodes.
[0099] The formula for obtaining the updated features is:
[0100]
[0101] Where σ(·) represents a nonlinear activation function, and W is a projection matrix (shared weight matrix). Let i represent the eigenvalues of the neighboring nodes of node i. This represents the adjacent nodes (i.e., connected nodes) of node i. To update features, α ij This represents the attention coefficient between node i and node j.
[0102] The formula for fully connected processing is:
[0103]
[0104] Among them, f final Indicates fusion characteristics, To update features.
[0105] The attention coefficients are obtained by substituting the two target vector values into the formula for acquiring the attention coefficients. The weighted sum of the attention coefficients (i.e., weights) between the two target vectors is calculated, and then all weighted sums are added together to obtain the updated features. All updated features are then processed through a fully connected layer to obtain the fused features. For example: calculate the attention coefficients a and b between node i and node a, c and d between node i and node c, d and f between node i and node e. Then calculate the weighted sum of attention coefficients a, b, c, d, and f. This yields the updated features related to node i through all the target vectors related to node i. This process is repeated for all nodes, and all updated features are then processed through a fully connected layer to obtain the fused features.
[0106] The advantage of this setup is that it achieves efficient feature fusion through attention mechanisms and fully connected processing, providing more accurate and effective feature representations in multimodal data processing, thereby improving the overall system performance and application effectiveness.
[0107] In this embodiment of the application, the user's profile information is determined based on the fusion features, including:
[0108] Based on multiple preset image labels, the fused features are classified to obtain the feature vector corresponding to each image label.
[0109] Based on the feature vectors corresponding to each profile tag, the user's profile information is determined.
[0110] The preset profile tags include interests and preferences (e.g., technology, health, travel, entertainment, food), emotional state (e.g., positive, neutral, negative), and language style (e.g., calm, concise, humorous, creative).
[0111] By inputting the fused features into a pre-defined classification model, the fused features can be classified to obtain the feature vector corresponding to each profile label. Through each profile label and its corresponding feature vector, user profile information is obtained. The pre-defined model can be based on a pre-trained Transformer model, and user information can include: interests: technology, emotional state: negative, and language style: calm. Simultaneously, real-time behavioral data can be received, and user profiles can be updated in real-time using Exponentially Weighted Moving Average (EWMA). EWMA is a statistical method for analyzing data sequences, particularly in time series analysis, used to smooth data and detect trends. By assigning higher weights to more recent data points, EWMA can more sensitively reflect the latest changes.
[0112] The advantage of this setup is that by integrating feature classification processing, accurate determination of user profile information is achieved, enabling more efficient and personalized services in various application scenarios, and improving the overall system's intelligence level and user experience.
[0113] S104. Generate personalized text based on the user's profile information; wherein, the personalized text is text content that the user is interested in.
[0114] In this embodiment of the application, personalized text is generated based on the user's profile information, including:
[0115] The user's profile information is input into a pre-defined large language model to obtain personalized output text.
[0116] The preset large language model is based on Llama.
[0117] Personalized text refers to text customized for users that matches their interests, emotional inclinations, and language style.
[0118] User profile information, multimodal data, and sentiment feature vectors are input into a pre-defined large language model to output personalized text.
[0119] The advantage of this setup is that by combining user profile information and a large language model, personalized text generation is achieved, enabling more intelligent and personalized services in multiple application areas and significantly improving the user experience.
[0120] In this embodiment of the application, it also includes:
[0121] Receive user feedback on personalized text; where the feedback information represents the user's level of satisfaction with the personalized text;
[0122] Based on the feedback information, the user profile information is updated to obtain new profile information;
[0123] New personalized text is generated based on the new profile information.
[0124] The feedback information reflects the user's level of satisfaction with the personalized text.
[0125] Based on feedback, user profile information is updated to obtain new user profiles. For example, if a user is dissatisfied with their interests (content), the interests section of the user profile is updated. Then, the new user profile information, multimodal data, and sentiment feature vectors are input into a large language model to generate new personalized text. Real-time facial expression recognition can also be used to update user profile information and personalize the text accordingly. Simultaneously, the large language model can be updated, and A / B testing can be introduced to optimize it. A / B testing is a method used to compare two or more versions of a product or service to determine which performs better on a specific metric. Personalized text can be used for text on social media accounts, chat tools, etc.
[0126] The advantage of this setup is that user feedback allows for a more accurate understanding of user preferences and needs. This feedback mechanism enables the system to continuously adjust and optimize user profiles, thereby generating more personalized text that better meets user expectations.
[0127] This application provides a text generation method based on multimodal data. By fusing multimodal data and sentiment feature vectors, the user profile information is obtained more accurately, thereby making the personalized text obtained based on the user profile information more accurate and able to provide users with text content that meets their personalized needs.
[0128] Figure 2 Another text generation method based on multimodal data is provided for embodiments of this application. For example... Figure 2 As shown, the subject of this method can be an electronic device; however, this embodiment does not impose any special limitations on it. Figure 2 As shown, the method includes:
[0129] S201. Collect multimodal data.
[0130] Methods for collecting multimodal data:
[0131] Collect real-time user data (text, images, voice, and other multimodal data) and behavioral data (basic user information and historical interaction records). Behavioral data includes text, images, and other multimodal data.
[0132] S202. Use a multimodal large model to extract features from multimodal data and extract sentiment features.
[0133] The emotional characteristics are represented by 0-7, corresponding to eight emotions: happiness, sadness, anxiety, excitement, fear, anger, surprise, and disgust.
[0134] The multimodal large model is a model trained based on the Llama3 model.
[0135] Multimodal large models are used to extract features from multimodal data. The method for extracting sentiment features is as follows:
[0136] The voice data input is converted into text format to achieve the voice feature extraction function.
[0137] The speech data is input into a text format and the text data is input into a multimodal model to obtain the emotional features of the speech data and the text data.
[0138] For image data involving human faces, the FER model is used for sentiment classification to obtain the sentiment features of the image data.
[0139] S203. Input multimodal features (text, image, speech, emotion) as nodes into the graph attention model (GAT) to obtain fused features.
[0140] The method of using multimodal features (text, image, speech) and sentiment features as nodes, inputting them into a graph attention model (GAT), and obtaining the fused features is as follows:
[0141] Connect all features to obtain the adjacency matrix.
[0142] Calculate the adaptive attention coefficients between the features of the adjacency matrix:
[0143]
[0144] Where, α ij The attention coefficient represents the one-way edge between node i and node j. Represents the eigenvalue of node i. Let W represent the feature value of node j. A node represents any one of the text vector, speech vector, image vector, and sentiment feature vector. W is a projection matrix (shared weight matrix). It is a learnable parameter vector (feedforward neural network parameters). This represents the adjacent nodes (i.e., connected nodes) of node i. W, and The LeakyReLU (Leaky Rectified Linear Unit) is a commonly used activation function in deep learning. It is an improvement on the traditional ReLU (Rectified Linear Unit) function, and exp represents the natural exponential function.
[0145] Weighted summation yields the updated features:
[0146]
[0147] Where σ(·) represents a nonlinear activation function, and W is a projection matrix (shared weight matrix). Let represent the eigenvalues of the neighboring nodes of node i. This represents the adjacent nodes (i.e., connected nodes) of node i. To update features, α ij This represents the attention coefficient between node i and node j.
[0148] The updated features are processed using a fully connected layer to obtain the fused features:
[0149]
[0150] Among them, f final Indicates fusion characteristics, To update features.
[0151] Figure 3 This application provides a schematic diagram of a fusion feature structure, such as... Figure 3 As shown, the fused feature includes text feature 1, text feature 2, text feature 3, image feature 1, image feature 2, speech feature 1, and sentiment feature 1. The nodes of the fused feature are unidirectionally connected. It should be noted that the nodes of the initial adjacency matrix are bidirectionally connected. After being filtered by the GAT model (i.e., after calculating the adaptive attention coefficients between the features of the adjacency matrix), the adjacency matrix becomes unidirectionally connected. Then, the unidirectionally connected adjacency matrix is fully connected to obtain the fused feature.
[0152] S204. Use a pre-trained multi-profile classification model to obtain user profiles based on fused features.
[0153] The user profile is composed of preset user profile tags, which include interest preferences, emotional tendencies, and language style.
[0154] The trained multi-image classification model is based on a pre-trained Transformer model.
[0155] The method for obtaining user profiles using a pre-trained multi-profile classification model based on fused features is as follows:
[0156] It receives dynamic behavioral data in real time and updates user profiles using Exponentially Weighted Moving Average (EWMA). EWMA is a statistical method for analyzing data series, particularly in time series analysis, used to smooth data and detect trends. By assigning higher weights to more recent data points, EWMA is able to more sensitively reflect the latest changes.
[0157] The fused features are input into a pre-trained multi-profile classification model to obtain user profiles.
[0158] S205. Input user profiles and multimodal features into a large language model based on Llama to generate personalized text.
[0159] S206. Update personalized text based on user feedback.
[0160] Based on user feedback, the method for updating personalized text is as follows:
[0161] The user profile is updated based on real-time user behavior and emotional changes. The updated user profile is then input into a large language model based on Llama to obtain updated personalized text.
[0162] Using user feedback (satisfaction, usage frequency), the Llama-based large language model is fine-tuned, and A / B testing is conducted to evaluate and update the Llama large language model, resulting in an updated Llama large language model. The updated Llama large language model is then used to update personalized text.
[0163] This application provides another text generation method based on multimodal data. It fuses user-provided text, image, speech, and emotional features using a graph attention network (GAT), and introduces a pre-trained multi-profile classification model and preset user profile labels to quickly generate user profiles. For the fused user features, the multi-profile classification model labels the features, quickly locating a specific user profile. Furthermore, it achieves accurate description and dynamic adjustment by receiving and analyzing the user's dynamic behavior data in real time. This makes the personalized text obtained based on the user's profile information more accurate, providing users with text content that meets their individual needs.
[0164] Figure 4This is a schematic diagram of a text generation device based on multimodal data, provided as an embodiment of this application. The device in this embodiment can be in the form of software and / or hardware. For example... Figure 4 As shown in the figure, an embodiment of this application provides a text generation device 40 based on multimodal data. The device includes: a receiving module 401, a first obtaining module 402, a second obtaining module 403, and a generating module 404, wherein:
[0165] The receiving module 401 is used to receive multimodal data from the user; wherein the multimodal data includes at least one of text data, image data, and voice data;
[0166] The first module 402 is used to obtain an emotional feature vector based on multimodal data; wherein the emotional feature vector represents the user's emotional tendency.
[0167] The second module 403 is used to fuse multimodal data and sentiment feature vectors to obtain user profile information; wherein, the profile information represents the user's habits and style;
[0168] The generation module 404 is used to generate personalized text based on the user's profile information; wherein, the personalized text is text content that the user is interested in.
[0169] In one example, the second obtained module 403 includes:
[0170] The unit is used to fuse multimodal data and sentiment feature vectors to obtain fused features; whereby the fused features represent multimodal data and sentiment feature vectors.
[0171] The determining unit is used to determine the user's profile information based on the fusion features.
[0172] In one possible implementation, the unit is obtained, specifically for:
[0173] Feature extraction is performed on the multimodal data to obtain the multimodal vectors corresponding to the multimodal data;
[0174] Multiple target vectors are obtained by filtering and processing the multimodal vectors and sentiment feature vectors;
[0175] Multiple target vectors are fused to obtain fused features.
[0176] In one possible implementation, the unit is obtained, specifically for:
[0177] Both multimodal data and sentiment feature vectors were identified as vectors to be screened.
[0178] Determine the attention coefficient between two vectors to be screened; where the attention coefficient characterizes the correlation between the two vectors.
[0179] The target vector is determined from all the vectors to be selected based on all the attention coefficients.
[0180] In one possible implementation, the unit is obtained, specifically for:
[0181] Determine the attention coefficient between two target vectors;
[0182] Based on the attention coefficient between the two target vectors and the two target vectors themselves, determine the update features corresponding to the two target vectors;
[0183] All updated features are processed using a fully connected layer to obtain fused features.
[0184] In one possible implementation, the defined unit is specifically used for:
[0185] Based on multiple preset image labels, the fused features are classified to obtain the feature vector corresponding to each image label.
[0186] Based on the feature vectors corresponding to each profile tag, the user's profile information is determined.
[0187] In one instance, module 404 is generated, including:
[0188] The receiving unit is used to input the user's profile information into a preset large language model and obtain the output personalized text.
[0189] In one example, receiving module 401 includes:
[0190] The receiving unit is used to receive user feedback information on personalized text; wherein, the feedback information represents the user's level of satisfaction with the personalized text.
[0191] The update unit is used to update the user's profile information based on feedback information to obtain new profile information;
[0192] The generation unit is used to generate new personalized text based on the new profile information.
[0193] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 5 As shown, the electronic device 50 includes:
[0194] The electronic device 50 may include one or more processors 501 with processing cores, one or more memory 502s of computer-readable storage media, communication components 503, and other components. The processor 501, memory 502, and communication components 503 are connected via a bus 504.
[0195] In the specific implementation process, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to execute the above-described method for text generation based on multimodal data.
[0196] In the above Figure 5 In the illustrated embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.
[0197] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.
[0198] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.
[0199] In some embodiments, a computer program product is also provided, comprising a computer program or instructions that, when executed by a processor, implement the steps in any of the above-described text generation methods based on multimodal data.
[0200] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.
[0201] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.
[0202] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.
[0203] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.
[0204] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.
[0205] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.
[0206] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as these combinations of technical features do not contradict each other, they should be considered within the scope of this specification.
[0207] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.
[0208] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.
Claims
1. A text generation method based on multimodal data, characterized in that, The method includes: Receive multimodal data from users; wherein the multimodal data includes at least one of text data, image data, and voice data; Based on the multimodal data, an emotional feature vector is obtained; wherein, the emotional feature vector represents the user's emotional tendency; The multimodal data and the sentiment feature vector are fused to obtain the user's profile information; wherein, the profile information represents the user's habits and style. Personalized text is generated based on the user's profile information; wherein, the personalized text is text content that the user is interested in.
2. The method according to claim 1, characterized in that, The process of fusing the multimodal data and the sentiment feature vector to obtain the user's profile information includes: The multimodal data and the sentiment feature vector are fused to obtain a fused feature; wherein the fused feature represents the multimodal data and the sentiment feature vector. Based on the fusion features, the user's profile information is determined.
3. The method according to claim 2, characterized in that, The process of fusing the multimodal data and the sentiment feature vector to obtain fused features includes: Feature extraction processing is performed on the multimodal data to obtain the multimodal vector corresponding to the multimodal data; The multimodal vectors and the sentiment feature vectors are filtered to obtain multiple target vectors; The multiple target vectors are fused to obtain the fused feature.
4. The method according to claim 3, characterized in that, The filtering process of the multimodal vectors and the sentiment feature vectors yields multiple target vectors, including: Both the multimodal data and the sentiment feature vectors are determined as vectors to be screened; Determine the attention coefficient between two vectors to be screened; wherein the attention coefficient characterizes the correlation between the two vectors; The target vector is determined from all the vectors to be filtered based on all the attention coefficients.
5. The method according to claim 3, characterized in that, The process of fusing the multiple target vectors to obtain the fused feature includes: Determine the attention coefficient between two target vectors; Based on the attention coefficient between the two target vectors and the two target vectors themselves, the updated features corresponding to the two target vectors are determined. All updated features are processed using a fully connected method to obtain the fused features.
6. The method according to claim 2, characterized in that, Determining the user's profile information based on the fusion features includes: Based on a variety of preset image tags, the fused features are classified to obtain the feature vector corresponding to each image tag. The user's profile information is determined based on the feature vectors corresponding to each profile tag.
7. The method according to claim 1, characterized in that, The step of generating personalized text based on the user's profile information includes: The user's profile information is input into a preset large language model to obtain personalized output text.
8. The method according to any one of claims 1-7, characterized in that, Also includes: Receive feedback information from the user regarding the personalized text; wherein the feedback information represents the user's level of satisfaction with the personalized text; Based on the feedback information, the user's profile information is updated to obtain new profile information; Based on the new profile information, new personalized text is generated.
9. A text generation device based on multimodal data, characterized in that, include: A receiving module is used to receive multimodal data from users; wherein the multimodal data includes at least one of text data, image data, and voice data; The first obtaining module is used to obtain an emotional feature vector based on the multimodal data; wherein the emotional feature vector represents the user's emotional tendency; The second obtaining module is used to fuse the multimodal data and the emotional feature vector to obtain the user's profile information; wherein, the profile information represents the user's habits and style; The generation module is used to generate personalized text based on the user's profile information; wherein the personalized text is text content that the user is interested in. The generation module is used to generate personalized text based on the user's profile information; wherein the personalized text is text content that the user is interested in.
10. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 8.
11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 8.
12. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-8.