Content generation method and device, computer equipment and storage medium

Through the multi-head attention mechanism, the multi-modal data characteristics of financial products are integrated to generate rich and diverse publicity content, solving the monotonous problems caused by the generation of content in a single modal category, and improving user experience and publicity effects.

CN119089011AActive Publication Date: 2024-12-06PING AN TECH (SHENZHEN) CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411204738.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-29
Publication Date
2024-12-06
Estimated Expiration
2044-08-29

AI Technical Summary

Technical Problem

The promotional content of financial products is displayed in a single modal category, resulting in monotonous content generation, affecting user experience and publicity effects, and lacking richness.

Method used

By acquiring multimodal data, the multi-head attention mechanism is used to calculate the data features of different modal categories, the fused features generate target data features, and the content generation is performed based on the target content generation model.

Benefits of technology

It realizes multimodal fusion of content generation, improves the richness and accuracy of content, and enhances the attractiveness of financial product publicity.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119089011A_ABST
    Figure CN119089011A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a content generation method and device, computer equipment and a storage medium, and belongs to the technical field of financial science and technology. The method comprises the following steps: acquiring multi-modal data; wherein the multi-modal data comprises at least three original data features and at least three original modal categories, and each original data feature has an original modal category; selecting a selected data feature from the at least three original data features; performing attention calculation on the selected data features through a preset multi-head attention mechanism and the reference data features to obtain a target attention weight set of the selected data features; wherein the reference modal category of the reference data feature is different from the original modal category of the selected data feature; performing feature fusion on the at least three original data features according to the target attention weight set to obtain a target data feature; and performing content generation on the target data features based on a preset target content generation model. According to the embodiment of the invention, content generation can be richer.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of financial technology, and in particular to a content generation method and apparatus, computer equipment and storage medium. Background Art

[0002] In order to improve the quality of financial services, financial applications will be set up to display promotional content of financial products to introduce financial products through promotional content. In related technologies, the promotional content of financial products is displayed in the form of a single modal category, and the promotional content is also generated after learning the content of a single modal category. Therefore, the generated promotional content is relatively monotonous, which affects the user experience when viewing the promotional content, thereby affecting the promotional effect of financial products. Therefore, improving the richness of content generation has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a content generation method and apparatus, a computer device and a storage medium, aiming to make content generation richer.

[0004] To achieve the above objective, a first aspect of an embodiment of the present application provides a content generation method, the method comprising:

[0005] Acquire multimodal data; wherein the multimodal data includes: at least three original data features and at least three original modal categories, each of the original data features having one of the original modal categories;

[0006] Selecting a selected data feature from the at least three original data features;

[0007] Attention calculation is performed on the selected data features through a preset multi-head attention mechanism and reference data features to obtain a target attention weight set of the selected data features; wherein the reference modality category of the reference data features is different from the original modality category of the selected data features;

[0008] Performing feature fusion on the at least three original data features according to the target attention weight set to obtain a target data feature;

[0009] Content is generated for the target data features based on a preset target content generation model.

[0010] In some embodiments, the performing attention calculation on the selected data features by using a preset multi-head attention mechanism and reference data features to obtain a target attention weight set for the selected data features includes:

[0011] Splitting the selected data features according to a preset number of attention heads to obtain candidate data features;

[0012] Performing attention calculation on the candidate data features through the multi-head attention mechanism and the reference data features to obtain a preliminary attention weight;

[0013] The preliminary attention weights are concatenated to obtain the target attention weight set for the selected data features.

[0014] In some embodiments, the preliminary attention weights include: query weights, key weights, and value weights; the target attention weight set includes: query weight matrix, key weight matrix, and value weight matrix; the concatenation of the preliminary attention weights to obtain the target attention weight set for the selected data features includes:

[0015] Concatenate the query weights to obtain a query weight matrix;

[0016] The key weights are concatenated to obtain a key weight matrix;

[0017] The value weights are concatenated to obtain a value weight matrix;

[0018] The target attention weight set is constructed according to the query weight matrix, the key weight matrix and the value weight matrix.

[0019] In some embodiments, the at least three original data features include: original text features, original image features, and original audio features; and the step of fusing the at least three original data features according to the target attention weight set to obtain the target data features includes:

[0020] Performing attention processing on the original text features, the original image features, and the original audio features according to the target attention weight set to obtain selected data features;

[0021] The selected data features are merged according to a preset weight matrix to obtain the target data features.

[0022] In some embodiments, if the selected data feature is a selected text feature; performing attention processing on the original text feature, the original image feature, and the original audio feature according to the target attention weight set to obtain the selected data feature includes:

[0023] Performing attention processing on the original text features according to the query weight matrix to obtain text query features;

[0024] Performing attention processing on the original image features according to the key weight matrix to obtain text key features;

[0025] Performing attention processing on the original audio features according to the value weight matrix to obtain text value features;

[0026] The text query feature, the text key feature and the text value feature are normalized to obtain the selected text feature.

[0027] In some embodiments, merging the selected data features according to a preset weight matrix to obtain the target data features includes:

[0028] Performing a linear transformation on the selected data features according to the weight matrix to obtain transformed data features;

[0029] The transformed data features are spliced ​​to obtain the target data features.

[0030] In some embodiments, after generating content for the target data feature based on a preset target content generation model, the method further includes:

[0031] Acquire target content generated by the target content generation model;

[0032] Calculate the loss according to the preset verification content and the target content to obtain target loss data;

[0033] The target content generation model is updated according to the target loss data.

[0034] To achieve the above objective, a second aspect of an embodiment of the present application provides a content generation device, the device comprising:

[0035] A data acquisition module, used to acquire multimodal data; wherein the multimodal data includes: at least three original data features and at least three original modal categories, each of the original data features having one of the original modal categories;

[0036] A feature selection module, used to select a selected data feature from the at least three original data features;

[0037] an attention calculation module, configured to perform attention calculation on the selected data feature through a preset multi-head attention mechanism and reference data features to obtain a target attention weight set of the selected data feature; wherein the reference modality category of the reference data feature is different from the original modality category of the selected data feature;

[0038] A feature fusion module, used for fusing the at least three original data features according to the target attention weight set to obtain a target data feature;

[0039] The content generation module is used to generate content for the target data features based on a preset target content generation model.

[0040] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes a computer device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the method described in the first aspect when executing the computer program.

[0041] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0042] The content generation method and device, computer equipment and storage medium proposed in the present application obtain at least three original data features, and each original data feature has an original modal category, and then select a selected data feature from the at least three original data features, perform attention calculation on the selected data feature according to the multi-head attention mechanism and the reference data feature, and calculate the correlation between the selected data feature and the reference data feature with different modal categories to obtain a target attention weight set. Therefore, based on the target attention weight set, at least three original data features are fused into a target data feature, and data features covering multiple modal categories are constructed. Then, content generation based on the target data feature can achieve richer content generation, thereby improving the richness of the content. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a flow chart of a content generation method provided by an embodiment of the present application;

[0044] Figure 2 yes Figure 1 Flow chart of step S103 in FIG.

[0045] Figure 3 yes Figure 2 Flow chart of step S203 in FIG.

[0046] Figure 4 yes Figure 3 Flow chart of step S302 in FIG.

[0047] Figure 5 yes Figure 4 Flow chart of step S401 in FIG.

[0048] Figure 6 yes Figure 4 Flow chart of step S402 in FIG.

[0049] Figure 7is a flow chart of a content generation method provided by another embodiment of the present application;

[0050] Figure 8 is a schematic diagram of the structure of a content generation device provided in an embodiment of the present application;

[0051] Fig. 9 It is a schematic diagram of the hardware structure of the computer device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0053] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0055] First, some nouns involved in this application are analyzed:

[0056] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0057] Natural language processing (NLP): NLP uses computers to process, understand and apply human languages ​​(such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text understanding. Natural language processing is often used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0058] Multimodal deep learning: is a subfield of artificial intelligence that can simultaneously process and learn models of multiple types of data, such as text, images, audio, video, and sensor data, to improve the performance and generalization ability of machine learning models. This technology is particularly suitable for handling complex tasks such as video analysis, sentiment analysis, and environmental perception for autonomous vehicles.

[0059] Multi-Head Attention: It is an important technique in deep learning, especially in the field of natural language processing (NLP), and is widely used in models such as Transformer. The core idea of ​​this mechanism is to distribute the attention of input data to different representation subspaces so that the model can focus on different aspects of the input sequence at the same time.

[0060] Generative model: It is a type of model in the field of deep learning. Its goal is to learn how to generate the distribution of data so that it can generate new samples similar to real data.

[0061] As computer technology is integrated into the financial field, a large number of financial applications have emerged, and the promotional content of financial products is displayed through financial applications to achieve the promotion and display of financial products. In related technologies, the promotional content of financial products is displayed in a single modal category, such as text, pictures or audio, so the display of promotional content is too single. Moreover, the promotional content of the same modal category is only learned and generated in the content of one modal category, and the characteristics of other modal categories cannot be learned. The generated promotional content is single, which affects the promotional effect of financial products.

[0062] Based on this, the embodiments of the present application provide a content generation method and apparatus, a computer device, and a storage medium, by obtaining at least three original data features of different original modal categories, and then selecting a selected data feature from the at least three original data features, and performing attention calculation on the selected data feature and other reference data features of different modal categories through a multi-head attention mechanism, and obtaining a target attention weight set for the selected data feature, so as to fuse the three original data features into a target data feature according to the target attention weight set. Therefore, the target data feature fuses at least three data features of different modal categories, and the fusion is to obtain the target data feature according to the attention weight fusion, which can obtain a more accurate fusion feature. Therefore, the target content generation model generates content with the target data feature, generates content covering multiple modalities, and makes the content generation richer.

[0063] The content generation method and apparatus, computer device, and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the content generation method in the embodiments of the present application is described.

[0064] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0065] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0066] The content generation method provided in the embodiment of the present application relates to the field of financial technology. The content generation method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the content generation method, etc., but is not limited to the above forms.

[0067] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer computer devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0068] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0069] Figure 1 is an optional flowchart of the content generation method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S105.

[0070] Step S101, obtaining multimodal data; wherein the multimodal data includes: at least three original data features and at least three original modal categories, each original data feature having an original modal category;

[0071] Step S102, selecting a selected data feature from at least three original data features;

[0072] Step S103, performing attention calculation on the selected data feature through a preset multi-head attention mechanism and the reference data feature to obtain a target attention weight set of the selected data feature; wherein the reference modality category of the reference data feature is different from the original modality category of the selected data feature;

[0073] Step S104, performing feature fusion on at least three original data features according to the target attention weight set to obtain target data features;

[0074] Step S105 , generating content for the target data features based on a preset target content generation model.

[0075] In the steps S101 to S105 shown in the embodiment of the present application, by obtaining original data features of different at least three original modal categories, and selecting selected data features from at least three original data features, attention calculation is performed on the selected data features and the reference data features based on the multi-head attention mechanism to analyze the correlation between the selected data features and data features of different other modal categories, and obtain a target attention weight set that characterizes the correlation between the selected data features and the reference data features. Then, at least three original data features are fused into target data features based on the target attention weight set, so that the target data features can cover data features of at least three original modal categories. Therefore, content generation is performed on the target data features based on the target content generation model, and then the generated content covers information of different modal categories based on the preset target content generation model for the target data features, so as to obtain more information-rich content.

[0076] In step S101 of some embodiments, the content generation method can be applied to the generation of promotional content for financial applications, and the generated promotional content can cover information of multiple modal categories. If applied to the generation of promotional content for financial applications, it is necessary to collect multimodal data of financial products in advance and store the multimodal data in a preset database in advance. When it is necessary to generate promotional content for financial products, the multimodal data is extracted from the database.

[0077] It should be noted that the multimodal data includes at least three original data features and at least three original modal categories, and each original data feature has an original modal category. Among them, the at least three original modal categories can be image categories, text categories, audio categories, and sensor categories, etc., and there is no specific limitation on the at least three original modal categories.

[0078] Specifically, at least three original data features are obtained by extracting data features from at least three original data, and one original data has an original modal category. In this embodiment, the original data of each original modal category is subjected to feature extraction by a pre-trained feature extraction model to obtain original data features, and the feature extraction model may be a BERT model, a Vi T model, or a Wave2Vec model. It should be noted that the BERT model can perform feature extraction based on context, the ViT model can perform feature extraction on original data of text categories and image categories, and the Wave2Vec model can perform feature extraction on original data of speech categories. Therefore, different feature extraction models are used to perform targeted feature extraction on original data based on different original modal categories.

[0079] In step S102 of some embodiments, a selected data feature is selected from at least three original data features. Specifically, original data features are selected one by one from at least three original data features as selected data features, and a target attention weight set of each selected data feature is calculated one by one.

[0080] See also Figure 2 In some embodiments, step S103 may include but is not limited to steps S201 to S203:

[0081] Step S201, splitting the selected data features according to a preset number of attention heads to obtain candidate data features;

[0082] Step S202, performing attention calculation on the candidate data features through the multi-head attention mechanism and the reference data features to obtain a preliminary attention weight;

[0083] Step S203, concatenate the preliminary attention weights to obtain a target attention weight set for the selected data features.

[0084] In step S201 of some embodiments, the number of attention heads is the number of attention heads in the multi-head attention mechanism, and also represents the number of self-attention components set, each of which is used to perform attention calculation on the candidate data feature. It should be noted that the selected data feature is split according to the number of attention heads to obtain multiple candidate data features. For example, if the number of attention heads is H, the selected data feature is split into H parts.

[0085] In step S202 of some embodiments, the number of reference data features is the same as the number of original data features, and the reference data features are also at least three original data features. The multi-head attention mechanism and the reference data features are used to perform attention calculation on the candidate data features, that is, to calculate the feature correlation between the candidate data features and the reference data features, and to calculate the correlation between the selected data features and the data features of the same and different modality categories, and obtain the attention weight representing the correlation between the candidate data features and the reference data features as the preliminary attention weight.

[0086] In step S203 of some embodiments, there are multiple candidate data features and at least three reference data features, so there are multiple calculated preliminary attention weights. Specifically, if the number of reference data features is 3 and the number of candidate data features is H, then the calculated preliminary attention weights are 3*H. By splicing the preliminary attention weights into a target attention weight set, the target attention weight set contains 3*H preliminary attention weights.

[0087] For example, if the original modality category of the selected data feature is the text category, and the reference modality category of the reference data feature can be the image category and the audio category, the selected data feature of the text category is defined as T2, and the reference data feature of the image category is defined as I C , the reference data feature of the audio category is A C , and the target attention weight set is calculated and represented in matrix form.

[0088] In steps S201 to S203 shown in the present embodiment, the selected data features are split into multiple candidate data features according to the number of attention heads, and the correlation between each candidate data feature and the reference data feature is calculated through a multi-head attention mechanism to obtain a preliminary attention weight. Finally, the preliminary attention weights are spliced ​​into a target attention weight set, and the feature correlation between each selected data feature and the reference data feature is determined. Then, the target attention weight set is used to fuse multiple original data features to obtain accurate and rich target data features.

[0089] In some embodiments, the preliminary attention weights include query weights, key weights, and value weights, and the target attention weight set includes: a query weight matrix, a key weight matrix, and a value weight matrix. Specifically, the query weight matrix is The key weight matrix is The value weight matrix is And h = 1,…,H; where h is the sequence number of the attention head and H is the total number of attention heads.

[0090] In some embodiments, please refer to Figure 3Step S203 may include but is not limited to steps S301 to S304:

[0091] Step S301, concatenating the query weights to obtain a query weight matrix;

[0092] Step S302, concatenating the key weights to obtain a key weight matrix;

[0093] Step S303, concatenating the value weights to obtain a value weight matrix;

[0094] Step S304, construct a target attention weight set according to the query weight matrix, the key weight matrix and the value weight matrix.

[0095] In steps S301 to S303 of some embodiments, the concatenation process is to combine multiple weights into a matrix form, combine multiple query weights into a query weight matrix, combine multiple key weights into a key weight matrix, and combine value weights into a value weight matrix. For example, if the query weight is W Q , combined into a query weight matrix: The key weight is W K , combined into a key weight matrix The value weight is W V , combined into a value weight matrix Where Q is the query, K is the key, V is the value, and W is the weight.

[0096] In step S304 of some embodiments, the query weight matrix, the key weight matrix and the value weight matrix are concatenated into a target attention weight set, that is, the target attention weight includes the query weight matrix, the key weight matrix and the value weight matrix.

[0097] In steps S301 to S304 shown in this embodiment, the weights are concatenated one by one into a matrix, and then multiple weight matrices are combined into a target attention weight set to obtain an attention weight set that characterizes the feature correlation between the selected data features and the reference data features, and the target attention weight set is easy to construct.

[0098] In some embodiments, the at least three original data features include: original text features, original image features, and original audio features, and the at least three original data features may also include original video features, original structured features, etc. The original text features, original image features, and original audio features are used as examples for illustration, and no specific limitation is made on the data features specifically included in the at least three original data features.

[0099] See also Figure 4 In some embodiments, step S104 may include but is not limited to steps S401 to S402:

[0100] Step S401, performing attention processing on the original text features, the original image features, and the original audio features according to the target attention weight set to obtain selected data features;

[0101] Step S402: Merge the selected data features according to a preset weight matrix to obtain target data features.

[0102] In step S401 of some embodiments, attention processing is performed on the original text features, the original image features, and the original audio features, mainly to allow the original text features, the original image features, and the original audio features to learn from each other's data features. It should be noted that the selected data features are features that integrate multiple information of images, texts, and audios, and have richer information.

[0103] In step S402 of some embodiments, the selected data features are multimodal representations that integrate image, text, and audio features, and the selected data features are specifically selected text features, selected image features, and selected audio features. In order to obtain richer data features, a weight matrix is ​​used to merge the selected data features into target data features. Then, the target data features are data features that integrate multimodal categories, and the represented content is richer.

[0104] For example, in the scenario of generating promotional content for financial products, the target data features include text, images, and audio. Then the promotional content generated based on the target data features can integrate the content of the three modal categories of images, text, and audio, which can promote financial products more richly.

[0105] In steps S401 to S402 shown in this embodiment, the original text features, original image features and original audio features are subjected to attention processing according to the target attention weight set to generate selected data features covering multiple modal categories of text, image and audio, and then at least three selected data features are merged into target data features. In this way, the target data features more fully represent the relationship between the three modal categories of image, text and audio.

[0106] See also Figure 5 In some embodiments, if the selected data feature is a selected text feature, step S401 may also include but is not limited to steps S501 to S504:

[0107] Step S501, performing attention processing on the original text features according to the query weight matrix to obtain text query features;

[0108] Step S502, performing attention processing on the original image features according to the key weight matrix to obtain text key features;

[0109] Step S503, performing attention processing on the original audio features according to the value weight matrix to obtain text value features;

[0110] Step S504, normalizing the text query feature, the text key feature, and the text value feature to obtain the selected text feature.

[0111] It should be noted that when generating each selected data feature, it is necessary to learn the relationship between features from the original text features, the original image features and the original audio features to generate selected data features covering text features, image features and audio features.

[0112] In step S501 of some embodiments, if the selected data feature is a selected text feature, the query weight matrix is ​​used to characterize the contextual relationship between text features, so the query weight matrix is ​​used to perform attention processing on the original text features to learn the contextual relationship between the original text features to obtain the text query feature.

[0113] Specifically, if the original text feature is T1, the query weight matrix is The text query feature generation process is:

[0114] In step S502 of some embodiments, if the selected data feature is a selected text feature, the key weight represents the correlation between the text feature and the image feature, so attention processing is performed based on the key weight matrix and the original image feature to learn the original image feature and obtain the text key feature containing the image feature.

[0115] Specifically, if the original image feature is I1, the key weight matrix is The text key feature generation process is:

[0116]

[0117] In step S503 of some embodiments, if the selected data feature is a selected text feature, the value weight represents the correlation between the text feature and the original audio feature, so the original audio feature is subjected to attention processing according to the value weight matrix to learn the audio feature and obtain the text value feature containing the audio feature.

[0118] Specifically, if the original audio feature is A1, the value weight matrix is The text value feature generation process is:

[0119]

[0120] In step S504 of some embodiments, after the text query feature, text key feature and text value feature are generated, the data features including text features, image features and audio features need to be merged into the final selected text feature. Therefore, the text query feature, text key feature and text value feature need to be normalized to obtain the selected text feature including the three modal categories of text, image and audio.

[0121] Specifically, the process of generating selected text features is as follows:

[0122]

[0123] Where, R, d k is a constant for normalization configuration, and softmax is a normalization function.

[0124] In steps S501 to S504 shown in this embodiment, when generating selected text features, the text features, image features and audio features are first learned using their respective corresponding weight matrices to obtain text query features, text key features and text value features. Finally, the text query features, text key features and text value features are normalized to generate selected text features covering text, image and audio features to obtain selected text features with richer content.

[0125] Specifically, if the selected data feature is a selected image feature, the query weight matrix represents the contextual relationship of the image feature, the key weight matrix represents the correlation between the image feature and the text feature, and the value weight matrix represents the correlation between the image feature and the audio feature. Therefore, the original image features are processed according to the query weight matrix to obtain the image query features, the original text features are processed according to the key weight matrix to obtain the image key features, and then the original audio features are processed according to the value weight matrix to obtain the image value features. Finally, the image query features, image key features and image value features are normalized to obtain the selected image features. It should be noted that the process of generating the selected image features is:

[0126]

[0127] Where, R, d k is the constant of the normalized configuration, is the image query feature, is the image key feature, is the image value feature, and For selected image features.

[0128] In some embodiments, if the selected data feature is a selected audio feature, the query weight matrix represents the contextual relationship of the original audio feature, the key weight matrix represents the correlation between the audio feature and the text feature, and the value weight matrix represents the correlation between the audio feature and the image feature. Therefore, the original audio feature is processed according to the query weight matrix to obtain the audio query feature, the original text feature is processed according to the key weight matrix to obtain the audio key feature, and then the original image feature is processed according to the value weight matrix to obtain the audio value feature. Finally, the audio query feature, the audio key feature and the audio value feature are normalized to obtain the selected audio feature. It should be noted that the process of generating the selected audio feature is:

[0129]

[0130] Where, R, d k is the constant of the normalized configuration, is the audio query feature, is the audio key feature, is the audio value feature, and softmax is the normalization function.

[0131] See also Figure 6 In some embodiments, step S402 includes but is not limited to steps S601 to S602:

[0132] Step S601, performing linear transformation on the selected data features according to the weight matrix to obtain transformed data features;

[0133] Step S602, concatenating the transformed data features to obtain target data features.

[0134] It should be noted that the operation of merging the selected data features is mainly divided into linear transformation and concatenation, and the linear transformation operation is mainly to transform the selected data features back to the same space as the original data features, that is, the same dimension and length as the original data features.

[0135] In step S601 of some embodiments, the selected data feature may be any one of a selected text feature, a selected image feature, and a selected audio feature, and the weight matrix is ​​another pre-set weight matrix, and the weight matrix corresponding to each selected data feature is different, so the weight matrix corresponding to the selected data feature is first selected as the selected weight feature, and then the selected weight matrix and the selected data feature are multiplied to complete the linear transformation of the selected data feature.

[0136] Specifically, the weight matrix is Wherein, O represents the linear transformation space. If the selected data feature is the selected text feature, the transformed data feature is the transformed text feature, and the transformed text feature is represented by the following formula:

[0137]

[0138] In the formula, is the selected text feature, T fused To transform text features, H is the number of attention heads and h is the order of the attention heads.

[0139] If the selected data feature is the selected image feature, the transformed data feature is the transformed image feature, and the transformed image feature is represented by the following formula:

[0140]

[0141] In the formula, To select image features, I fused is the transformed image feature.

[0142] If the selected data feature is the selected audio feature, the transformed data feature is the transformed audio feature, and the transformed audio feature is represented by the following formula:

[0143]

[0144] In the formula, To select audio features, A fused To transform audio features.

[0145] In step S602 of some embodiments, the transformation data features may be concatenated in any one of direct connection, averaging, summing, and weighted summing. This embodiment does not impose any specific limitation on the concatenation method of the transformation data features.

[0146] Specifically, if the splicing method is direct connection, there are at least three transformed data features, and the at least three transformed data features include: transformed text features, transformed image features and transformed audio features. The target data features are obtained by directly connecting the transformed text features, the transformed image features and the transformed audio features, and the implementation of directly connecting and constructing the target data features is shown in the following formula:

[0147] M final =concat(T fused ,I fused ,A fused );

[0148] In some embodiments, if the concatenation method is to take an average value, the target data feature is obtained by averaging the transformed text feature, the transformed image feature, and the transformed audio feature. Then, the target data feature is realized as shown in the following formula:

[0149]

[0150] Where Mfinal is the target data feature.

[0151] In steps S601 to S602 shown in this embodiment, the selected data features are linearly transformed to obtain transformed data features with the same feature space as the original data features, and the transformed data features are then spliced ​​into target data features, making the target data feature construction operation easier and integrating data features of multiple modal categories.

[0152] In step S105 of some embodiments, the target content generation model is obtained by pre-training a preset original content generation model, and the training process is also based on training the original content generation model based on the fusion of sample data features of multiple modal categories and preset verification content. Therefore, the target content generation model can learn data features of multiple modal categories to generate content covering multiple modal categories.

[0153] For example, in the scenario of generating promotional content for financial products, content generation based on features including text, images, and audio can generate promotional content that meets user needs. The promotional content covers text, images, and audio content, which can more richly display financial products and increase users' interest in understanding financial products.

[0154] As disclosed above, the target data features can be applied to content generation, data classification, data regression processing, etc. Specifically, the target data features integrate the features of text, image, and audio. If in a data classification scenario, such as image classification, the information covered by the image is richer, the accuracy of image classification can be improved accordingly.

[0155] See also Figure 7 In some embodiments, after step S105, the content generation method may further include but is not limited to steps S701 to S703:

[0156] Step S701, obtaining target content generated by a target content generation model;

[0157] Step S702, performing loss calculation according to preset verification content and target content to obtain target loss data;

[0158] Step S703: update the target content generation model according to the target loss data.

[0159] In step S702 of some embodiments, the verification content is pre-set content, which is used to verify the accuracy of content generation by the target content generation model. The target content is the content generated in real time by the target content generation model, and the target content and the verification content have the same data characteristics. The target loss data is obtained by performing loss calculation on the verification content and the target content, and the target loss data characterizes the accuracy of content generation by the target content generation model. It should be noted that the loss calculation can be implemented using a loss function, and the loss function can be any one of a cross entropy loss function, a mean square error loss function, a logarithmic loss function, an absolute value loss function, and a square root loss function. This embodiment does not make specific restrictions on the loss function.

[0160] In step S703 of some embodiments, the model update also involves adjusting the model parameters of the target content generation model until the updated target loss data converges, completing the update of the target content generation model to obtain a content generation model with more accurate content generation.

[0161] In steps S701 to S703 shown in this embodiment, after the target content generation model generates rich target content based on the target data features, loss calculation is performed based on the verification content and the target content to obtain target loss data, so as to determine the generation accuracy of the target content generation model through the target loss data, and then optimize the target content generation model based on the target loss data to obtain a content generation model with more accurate content generation.

[0162] The embodiment of the present application sets the original text data, original image data and original audio features in advance, extracts features from the original text data through an encoder to obtain original text features, extracts features from the original image data through an encoder to obtain original image features, and extracts features from the original audio data through an encoder to obtain original audio features. Therefore, three original data features with different modal categories are set.

[0163] For the original data features of each modal category, a multi-head attention mechanism is used to calculate the attention calculation of each original data feature and the original data features of other modal categories to analyze the relationship between the original text features, the original image features and the original audio features. Specifically, taking the original text features of the text category as an example, the original text features are first split into multiple candidate text features according to the number of attention heads H, and then each attention head calculates the relationship between each candidate text feature and the original text feature, the original image feature and the original audio feature to obtain the preliminary attention weight. Among them, the preliminary attention weight includes query weight, key weight and value weight, and H query weights are spliced ​​into a query weight matrix, H key weights are spliced ​​into a key weight matrix, and then H value weights are spliced ​​into a value weight matrix. It should be noted that the query weight matrix, key weight matrix and value weight matrix generated by the original data features of other categories are the same, which will not be repeated here.

[0164] For the candidate text features of the text category, attention processing is first performed based on the query weight matrix and the candidate text features to obtain the text query features, and attention processing is performed based on the key weight matrix and the candidate image features to obtain the text key features. Attention processing is performed based on the value weight matrix and the candidate audio features to obtain the text value features. Finally, the text query features, text key features, and text value features are normalized to obtain the selected text features. At the same time, for the candidate image features of the image category and the candidate audio features of the audio category, the selected image features and selected audio features are also generated according to the above method, which will not be repeated here.

[0165] When the selected text features, selected image features, and selected audio features are constructed, the selected text features are linearly transformed according to the preset weight matrix to obtain the transformed text features, the selected image features are linearly transformed according to the weight matrix to obtain the transformed image features, and the selected audio features are linearly transformed according to the weight matrix to obtain the transformed audio features. Finally, the transformed text features, transformed image features, and transformed audio features are directly connected or averaged to form the target data features. Therefore, the target data features cover the features of text, images, and audio, and capture the complex relationships between text categories, image categories, and audio categories, generating richer multimodal representations. Therefore, generating content for the target data features through the target content generation model can generate richer content.

[0166] See also Figure 8 The embodiment of the present application further provides a content generation device, which can implement the above-mentioned content generation method, and the device includes:

[0167] The data acquisition module 801 is used to acquire multimodal data; wherein the multimodal data includes: at least three original data features and at least three original modal categories, each original data feature having one original modal category;

[0168] A feature selection module 802 is used to select a selected data feature from at least three original data features;

[0169] The attention calculation module 803 is used to perform attention calculation on the selected data features through a preset multi-head attention mechanism and reference data features to obtain a target attention weight set for the selected data features; wherein the reference modality category of the reference data features is different from the original modality category of the selected data features;

[0170] A feature fusion module 804 is used to fuse at least three original data features according to a target attention weight set to obtain a target data feature;

[0171] The content generation module 805 is used to generate content for target data features based on a preset target content generation model.

[0172] The specific implementation of the content generation device is substantially the same as the specific implementation of the above-mentioned content generation method, and will not be described in detail herein.

[0173] The embodiment of the present application also provides a computer device, the computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the above content generation method when executing the computer program. The computer device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0174] See also Fig. 9 , Fig. 9 The hardware structure of a computer device according to another embodiment is shown, and the computer device includes:

[0175] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0176] The memory 902 may be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 may store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the content generation method of the embodiment of this application;

[0177] Input / output interface 903, used to implement information input and output;

[0178] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WI FI, Bluetooth, etc.);

[0179] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0180] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0181] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned content generation method is implemented.

[0182] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0183] The content generation method and apparatus, computer equipment and storage medium provided in the embodiments of the present application first select the selected data features from at least three original data features of different original modal categories, and then use the multi-head attention mechanism and reference data features to perform attention calculation on the selected data features to obtain a target attention weight set that characterizes the correlation between the selected data features and reference data features of different modal categories, and then fuse at least three original data features into a target data feature with the target attention weight set, so that the target data feature fuses the features of multiple modal categories. Therefore, content generation based on target data features can generate richer content and improve the readability of generated content.

[0184] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0185] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0186] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0187] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0188] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0189] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0190] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0191] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0192] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0193] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0194] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A content generation method, characterized in that: The method comprises: Acquire multimodal data; wherein the multimodal data includes: at least three original data features and at least three original modal categories, each of the original data features having one of the original modal categories; Selecting a selected data feature from the at least three original data features; Attention calculation is performed on the selected data features through a preset multi-head attention mechanism and reference data features to obtain a target attention weight set of the selected data features; wherein the reference modality category of the reference data features is different from the original modality category of the selected data features; Performing feature fusion on the at least three original data features according to the target attention weight set to obtain a target data feature; Content is generated for the target data features based on a preset target content generation model.

2. The method according to claim 1, characterized in that The performing attention calculation on the selected data features through a preset multi-head attention mechanism and reference data features to obtain a target attention weight set for the selected data features includes: Splitting the selected data features according to a preset number of attention heads to obtain candidate data features; Performing attention calculation on the candidate data features through the multi-head attention mechanism and the reference data features to obtain a preliminary attention weight; The preliminary attention weights are concatenated to obtain the target attention weight set for the selected data features.

3. The method according to claim 2, characterized in that The preliminary attention weights include: query weights, key weights and value weights; the target attention weight set includes: query weight matrix, key weight matrix and value weight matrix; the preliminary attention weights are concatenated to obtain the target attention weight set of the selected data features, including: Concatenate the query weights to obtain a query weight matrix; The key weights are concatenated to obtain a key weight matrix; The value weights are concatenated to obtain a value weight matrix; The target attention weight set is constructed according to the query weight matrix, the key weight matrix and the value weight matrix.

4. The method according to claim 3, characterized in that The at least three original data features include: original text features, original image features, and original audio features; the at least three original data features are subjected to feature fusion according to the target attention weight set to obtain the target data features, including: Performing attention processing on the original text features, the original image features, and the original audio features according to the target attention weight set to obtain selected data features; The selected data features are merged according to a preset weight matrix to obtain the target data features.

5. The method according to claim 4, characterized in that If the selected data feature is a selected text feature; performing attention processing on the original text feature, the original image feature, and the original audio feature according to the target attention weight set to obtain the selected data feature includes: Performing attention processing on the original text features according to the query weight matrix to obtain text query features; Performing attention processing on the original image features according to the key weight matrix to obtain text key features; Performing attention processing on the original audio features according to the value weight matrix to obtain text value features; The text query feature, the text key feature and the text value feature are normalized to obtain the selected text feature.

6. The method according to claim 4, characterized in that The step of merging the selected data features according to a preset weight matrix to obtain the target data features includes: Performing a linear transformation on the selected data features according to the weight matrix to obtain transformed data features; The transformed data features are spliced ​​to obtain the target data features.

7. The method according to any one of claims 1 to 6, characterized in that: After generating content for the target data feature based on the preset target content generation model, the method further includes: Acquire target content generated by the target content generation model; Calculate the loss according to the preset verification content and the target content to obtain target loss data; The target content generation model is updated according to the target loss data.

8. A content generating device, characterized in that: The device comprises: A data acquisition module, used to acquire multimodal data; wherein the multimodal data includes: at least three original data features and at least three original modal categories, each of the original data features having one of the original modal categories; A feature selection module, used to select a selected data feature from the at least three original data features; an attention calculation module, configured to perform attention calculation on the selected data feature through a preset multi-head attention mechanism and reference data features to obtain a target attention weight set of the selected data feature; wherein the reference modality category of the reference data feature is different from the original modality category of the selected data feature; A feature fusion module, used for fusing the at least three original data features according to the target attention weight set to obtain a target data feature; The content generation module is used to generate content for the target data features based on a preset target content generation model.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the content generation method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the content generating method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Method for solving video question and answer task based on multi-mode progressive attention model

    CN113688296A

  • Text generation method and device and computing equipment

    CN118364877A