Content generation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a content generation method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a query image and a query text, carrying out the coding of the query text, obtaining a first text feature, carrying out the pooling of various granularities of an output feature of a feedforward network in the coding process of the query text, obtaining various pooling features, and carrying out the processing of the pooling features. The multiple pooling features are fused to obtain fused pooling features, a first text feature is obtained based on the sum of the fused pooling features and the output features, the query image is encoded to obtain visual features, the visual features and the first text feature are fused to obtain query features, and a reference text is retrieved according to the query features; and calling the visual large language model to generate content based on the reference text, the query image and the query text to obtain the target content, so that the content generation accuracy can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a content generation method, device, electronic device and storage medium. Background Art

[0002] With the rapid growth of Internet data such as text and image content, the application of visual big language models has become more and more widespread. Visual big language models usually have the ability to understand the input query text and images to generate corresponding content. In related technologies, when using visual big language models for content generation, the retrieval augmented generation (RAG) method can generally be adopted. In the retrieval stage, the visual big language model is assisted in reasoning by retrieving text related to the input. However, in the retrieval stage, the ability to understand the input text is often insufficient, resulting in a decrease in the correlation between the retrieved text and the input, thereby reducing the accuracy of content generation. Summary of the invention

[0003] The following is a summary of the subject matter of the detailed description of the present disclosure. This summary is not intended to limit the scope of the claims.

[0004] The embodiments of the present disclosure provide a content generation method, device, electronic device, and storage medium, which can improve the accuracy of content generation.

[0005] In one aspect, an embodiment of the present disclosure provides a content generation method, comprising:

[0006] Acquire a query image, encode the query image, and obtain visual features;

[0007] Acquire a query text, encode the query text to obtain a first text feature, wherein, in the process of encoding the query text, perform pooling of multiple granularities on the output features of the feedforward network to obtain multiple pooling features, fuse the multiple pooling features to obtain a fused pooling feature, sum the fused pooling feature and the output feature, and obtain the first text feature according to the summation result;

[0008] fusing the visual feature with the first text feature to obtain a query feature, and retrieving reference text related to the query text according to the query feature;

[0009] The visual large language model is called to generate content based on the reference text, the query image and the query text to obtain target content.

[0010] On the other hand, the present disclosure also provides a content generation device, including:

[0011] A first encoding module, used to obtain a query image, encode the query image, and obtain visual features;

[0012] A second encoding module is used to obtain a query text, encode the query text to obtain a first text feature, wherein, in the process of encoding the query text, the output features of the feedforward network are pooled at multiple granularities to obtain multiple pooling features, the multiple pooling features are fused to obtain a fused pooling feature, the fused pooling feature is summed with the output feature, and the first text feature is obtained according to the summation result;

[0013] A retrieval module, configured to fuse the visual feature with the first text feature to obtain a query feature, and retrieve reference texts related to the query text according to the query feature;

[0014] The generation module is used to call the visual large language model to generate content based on the reference text, the query image and the query text to obtain target content.

[0015] Furthermore, the second encoding module is also used for:

[0016] Performing local maximum pooling on the output features of the feedforward network to obtain a first pooling feature, and performing segmented maximum pooling on the output features to obtain a second pooling feature;

[0017] The first pooling feature and the second pooling feature are fused to obtain a third pooling feature, and the third pooling feature and the output feature are summed to obtain a fused pooling feature.

[0018] Furthermore, the second encoding module is also used for:

[0019] Obtaining a first weight corresponding to the first pooling feature according to the output feature and the first learnable matrix, and obtaining a second weight corresponding to the second pooling feature according to the output feature and the second learnable matrix;

[0020] According to the first weight and the second weight, the first pooling feature and the second pooling feature are weightedly summed to obtain a third pooling feature.

[0021] Furthermore, the second encoding module is also used for:

[0022] Obtaining a third weight according to the output feature and a third learnable matrix, and weighting the second pooling feature according to the second weight and performing a Hadamard product with the third weight;

[0023] After weighting the first pooling feature according to the first weight, summing the result with the Hadamard product obtains the third pooling feature.

[0024] Furthermore, the second encoding module is also used for:

[0025] Obtaining a filling feature according to an average value of the output feature of the feedforward network, and splicing the filling features at the beginning and the end of the output feature respectively;

[0026] Based on the sliding window, sliding sampling is performed in the output feature after the filling feature is spliced, and maximum pooling is performed on the features obtained by each sliding sampling to obtain the first pooling sub-feature corresponding to each sliding sampling;

[0027] The plurality of the first pooling sub-features are concatenated to obtain a first pooling feature.

[0028] Furthermore, the second encoding module is also used for:

[0029] Dividing the output feature into a plurality of segmented sub-features based on a preset interval;

[0030] For any of the segmentation sub-features, perform maximum pooling on the segmentation sub-feature to obtain a second pooling sub-feature corresponding to the segmentation sub-feature, copy the second pooling sub-feature according to the size of the segmentation sub-feature, and concatenate the multiple copied second pooling sub-features to obtain a third pooling sub-feature corresponding to the segmentation sub-feature;

[0031] A plurality of the third pooling sub-features are concatenated to obtain a second pooling feature.

[0032] Furthermore, the first encoding module is also used for:

[0033] aggregating the visual feature based on the first text feature to align the visual feature with the first text feature;

[0034] A query feature is obtained according to the average of the first text feature and the aligned visual feature.

[0035] Furthermore, the first encoding module is also used for:

[0036] Determining a similarity score between the first text feature and the visual feature, and normalizing the similarity score;

[0037] Obtaining an aggregation weight according to a ratio of the normalized similarity score to the temperature coefficient;

[0038] The visual features are aggregated according to the aggregation weights to align the visual features with the first text features.

[0039] Furthermore, the generation module is also used to:

[0040] Summarizing the reference text to obtain a summary text;

[0041] The summary text, the query image and the query text are input into a visual large language model to generate content, thereby obtaining target content.

[0042] Furthermore, the generation module is also used to:

[0043] Constructing a prompt text based on the reference text, the image title text and the query text, wherein the prompt text is used to prompt the large language model to summarize the reference text according to the image title text and the query text;

[0044] The prompt text is input into the large language model for text generation to obtain a summary text.

[0045] Furthermore, the first encoding module is also used for:

[0046] Performing object detection on the query image to obtain a sub-image corresponding to the visual object, and constructing an image set based on the query image and the sub-image;

[0047] The image set is encoded to obtain visual features.

[0048] Furthermore, the search module is also used to:

[0049] Encode each reference text in the knowledge base to obtain a second text feature;

[0050] According to the similarity between the query feature and the second text feature, the reference text related to the query text is retrieved from the plurality of knowledge bases.

[0051] Furthermore, the search module is also used to:

[0052] Acquire a plurality of first sample texts in the same training batch and second sample texts related to each of the first sample texts;

[0053] Encoding the first sample text by the first text encoder to obtain a first sample feature;

[0054] Encoding the second sample text related to the first sample text by the second text encoder to obtain a second sample feature;

[0055] Encoding the second sample text in the training batch that is unrelated to the first sample text by the second text encoder to obtain a third sample feature;

[0056] Determining a first similarity between the first sample feature and the second sample feature, and a second similarity between the first sample feature and the third sample feature;

[0057] A target loss is determined according to the first similarity and the second similarity, and the first text encoder and the second text encoder are jointly trained based on the target loss, wherein the target loss is used to maximize the first similarity and minimize the second similarity.

[0058] Furthermore, the second encoding module is also used for:

[0059] For the query image, extracting label text of visual objects and image title text related to the query text;

[0060] The label text and the image title text are combined with the query text, and the combined query text is encoded to obtain a first text feature.

[0061] Furthermore, the generation module is also used to:

[0062] For each of the reference texts, the calling of the visual large language model generates content based on the reference text, the query image and the query text to obtain a generation result corresponding to each of the reference texts;

[0063] Voting is performed on a plurality of the generated results to obtain target content.

[0064] On the other hand, an embodiment of the present disclosure further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned content generation method when executing the computer program.

[0065] On the other hand, an embodiment of the present disclosure further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is executed by a processor to implement the above-mentioned content generation method.

[0066] On the other hand, the embodiment of the present disclosure further provides a computer program product, which includes a computer program, and the computer program is stored in a computer-readable storage medium. The processor of the computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the above-mentioned content generation method.

[0067] The disclosed embodiment includes at least the following beneficial effects: by acquiring a query image and a query text, the query text is encoded to obtain a first text feature, wherein, in the process of encoding the query text, the output features of the feedforward network are pooled at multiple granularities to obtain multiple pooling features, the multiple pooling features are fused to obtain a fused pooling feature, the fused pooling feature is summed with the output feature, and the first text feature is obtained according to the summation result, and the fine-grained understanding ability of the query text can be improved by pooling at different granularities, thereby making the first text feature more accurate, and subsequently the query image is encoded to obtain visual features, and the visual features are fused with the first text features to obtain query features, and when a reference text related to the query text is retrieved according to the query features, since the first text features are more accurate, the correlation between the reference text and the query text can be made higher, therefore, when the visual large language model is called to generate content based on the reference text, the query image and the query text to obtain the target content, the accuracy of the target content can be effectively improved.

[0068] Other features and advantages of the present disclosure will be set forth in the following description, and in part will be apparent from the description, or may be learned by practicing the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] The accompanying drawings are used to provide further understanding of the technical solution of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solution of the present disclosure and do not constitute a limitation on the technical solution of the present disclosure.

[0070] Figure 1 An optional schematic diagram of a content generation method provided in an embodiment of the present disclosure;

[0071] Figure 2 A schematic diagram of an optional implementation environment provided for an embodiment of the present disclosure;

[0072] Figure 3 An optional flow chart of the content generation method provided by the embodiment of the present disclosure;

[0073] Figure 4 An optional schematic diagram of encoding a query text provided in an embodiment of the present disclosure;

[0074] Figure 5 The embodiments of the present disclosure provide Figure 4 An alternative schematic diagram of a multi-granularity pooling layer.

[0075] Figure 6 The embodiments of the present disclosure provide Figure 5 An alternative schematic diagram of the local maximum pooling layer in ;

[0076] Figure 7 The embodiments of the present disclosure provide Figure 5 An alternative schematic diagram of the segmented max pooling layer in ;

[0077] Figure 8 An optional schematic diagram of text-oriented gated aggregation provided by an embodiment of the present disclosure;

[0078] Fig. 9 An optional schematic diagram of a prompt text template provided in an embodiment of the present disclosure;

[0079] Fig.10 An optional overall flow chart of the content generation method provided in the embodiment of the present disclosure;

[0080] Fig.11 An optional schematic diagram of a knowledge searcher provided in an embodiment of the present disclosure;

[0081] Fig.12 A schematic diagram of an optional structure of a content generation device provided in an embodiment of the present disclosure;

[0082] Fig.13 A partial structural block diagram of a terminal provided in an embodiment of the present disclosure;

[0083] Fig.14 A partial structural block diagram of a server provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0084] In order to make the purpose, technical solution and advantages of the present disclosure more clear, the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not used to limit the present disclosure.

[0085] It should be noted that in various specific embodiments of the present disclosure, when it comes to the need to perform relevant processing based on data related to the characteristics of the target object such as the target object attribute information or attribute information set, the permission or consent of the target object will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. Among them, the target object can be a user. In addition, when the embodiment of the present disclosure needs to obtain the attribute information of the target object, the separate permission or separate consent of the target object will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the separate permission or separate consent of the target object, the necessary target object-related data used to enable the normal operation of the embodiment of the present disclosure will be obtained.

[0086] In the embodiments of the present disclosure, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0087] To facilitate understanding of the technical solution provided by the embodiments of the present disclosure, some key terms used in the embodiments of the present disclosure are explained here:

[0088] Visual Large Language Model (VLM): is a deep learning model that combines computer vision and natural language processing technologies. It can process image and text data simultaneously and generate relevant natural language descriptions or instructions. The Visual Large Language Model is mainly composed of a visual encoder and a large language model, and is mostly used in image caption generation, visual question answering and other fields.

[0089] Pooling: It is an important operation in convolutional neural networks. It generates a smaller output feature map by applying a certain aggregation function (such as maximum value, average value, etc.) to the local area of ​​the input feature map. It is mainly used to reduce the dimension of the feature map output by the convolutional layer while retaining important feature information.

[0090] Hadamard product: also known as element-wise product, is a special multiplication operation between matrices or vectors. It obtains a new vector by multiplying the elements at corresponding positions of two vectors of the same size.

[0091] With the rapid growth of Internet data such as text and image content, the application of visual language models has become more and more widespread. At present, content generation using visual language models usually includes two steps: retrieval and generation. Figure 1 , Figure 1 An optional schematic diagram of a content generation method provided in an embodiment of the present disclosure, when performing a search, such as Figure 1 As shown in (a) in the figure, the query text is generally input into a text-based retriever to obtain knowledge text. However, the content generation method based on the text retriever lacks the assistance of world knowledge, and the visual language model may generate hallucinations or lose visual information when extracting visual information. In addition, the retrieval performance of the retriever often depends on the open world knowledge source. When the open world knowledge source is not selected properly, the relevance of the retrieved text will be insufficient, thereby affecting the accuracy of the generated content.

[0092] Based on this, the embodiments of the present disclosure provide a content generation method, device, electronic device and storage medium, such as Figure 1 As shown in (b) in the figure, by combining the query image and the query text, a fine-grained multimodal retriever is introduced to retrieve the relevant knowledge text, and then the retrieved knowledge text, query image and query text are combined for content generation, which can effectively improve the accuracy of the target content.

[0093] Reference Figure 2 , Figure 2 A schematic diagram of an optional implementation environment provided for an embodiment of the present disclosure, the implementation environment includes a terminal 201 and a server 202, wherein the terminal 201 and the server 202 are connected via a communication network.

[0094] Exemplarily, terminal 201 obtains a query image and a query text, and sends the query image and the query text to server 202. Server 202 encodes the query image to obtain visual features, and encodes the query text to obtain a first text feature. In the process of encoding the query text, the output features of the feedforward network are pooled at multiple granularities to obtain multiple pooling features, the multiple pooling features are fused to obtain a fused pooling feature, the fused pooling feature is summed with the output feature, and the first text feature is obtained according to the summation result. The visual feature is fused with the first text feature to obtain the query feature, and a reference text related to the query text is retrieved according to the query feature. The visual large language model is called to generate content based on the reference text, the query image and the query text to obtain the target content, and the generated target content is sent to terminal 201.

[0095] It is understandable that the above content generation process may also be performed only in the terminal 201 .

[0096] The server 202 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. In addition, the server may also be a node server in a blockchain network.

[0097] The terminal 201 may be a mobile phone, a computer, an intelligent voice interaction device, an intelligent wearable device, an intelligent home appliance, a vehicle-mounted terminal, etc., but is not limited thereto. The terminal 201 and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present disclosure.

[0098] Reference Figure 3 , Figure 3 An optional flowchart of a content generation method provided in an embodiment of the present disclosure. The content generation method can be executed by a terminal, or can also be executed by a server, or can also be executed by a terminal and a server in cooperation. The content generation method includes but is not limited to the following steps S301 to S304.

[0099] Step S301: obtaining a query image, encoding the query image, and obtaining visual features;

[0100] The query image is a complete image containing visual query information, and there is at least one visual object in the query image. The visual object is an instance object existing in the query image, and the visual object can be an animal, a plant, a building, a tool, etc. The visual feature is a feature containing the visual information of the query image and each sub-image, which can be an aggregate feature, such as the visual feature G, or a feature set or feature sequence containing the image features corresponding to the query image and the image features corresponding to each sub-image, such as the visual feature G = {g1, g2, ..., g n}, g1 can be the visual feature corresponding to the query image, g2,...,g n It can be the image features corresponding to each sub-image; the query feature integrates the visual feature and the text feature, and the knowledge text related to the query text can be retrieved based on the query feature.

[0101] In a possible implementation, in the process of encoding the query image and obtaining the visual features, the query image may be subjected to object detection to obtain a sub-image corresponding to the visual object, an image set may be constructed based on the query image and the sub-image, and the image set may be encoded to obtain the visual features. A sub-image is an image containing only one visual object. For example, if the query image is a dog holding a frisbee on the grass, and there is a row of houses behind the dog, then the visual objects may be a dog, a frisbee, a grass, and a house, and the sub-image may be an image containing only a dog or an image containing only a house.

[0102] Specifically, object detection is performed on the query image, visual objects in the query image are identified and bounding boxes are drawn around them, and the visual objects are cropped out of the query image through the bounding boxes to obtain sub-images corresponding to the visual objects. An image set is constructed based on the query image and each sub-image, and the image set is encoded by an image encoder to obtain visual features corresponding to the image set, wherein the visual encoder is used to encode images and can be a CLIP image encoder. For example, based on query image i, the visual objects {obj1,...,obj n}, where n is the number of visual objects recognized, based on the query image i and the visual objects {obj1,...,obj n}Construct an image set I = {i, obj1, ..., obj n}, the image set I and each image in the image set I are encoded by the visual encoder to obtain the visual feature sequence corresponding to the image set I By encoding the query image and the sub-images corresponding to each visual object in the query image to obtain visual features, the global visual information of the query image and the local visual information corresponding to each sub-image can be obtained. The local visual information provides more accurate visual information by capturing subtle visual information and uniqueness in the sub-image, and the global visual information further strengthens the contextual relationship between the visual objects corresponding to each sub-image, so that the obtained visual features contain rich and clear deep visual information, which helps to improve the accuracy of alignment of the visual features with the first text features in subsequent steps.

[0103] Step S302: Acquire a query text, and encode the query text to obtain a first text feature.

[0104] Among them, the query text is a query question directly related to the query image, which is used to guide the retrieval direction of the reference text and the generation direction of the target content; in the embodiment of the present disclosure, only the query text can be introduced for encoding. In addition, when encoding the query text, the label text of the visual object and the image title text related to the query text can be extracted for the query image, and the label text and the image title text are merged with the query text. The merged query text is encoded to obtain the first text feature. The label text is a short text that defines or accurately describes the visual object. The label text can be a noun, a verb or a phrase; the image title text is a text that summarizes and describes the overall information of the query image. Specifically, the query image can be input into the artificial intelligence model for object recognition and feature extraction, image features are extracted for the recognized visual object, and label text is assigned to the visual object according to the image features, and finally the image title text related to the query text is generated. For example, the query image is "a dog is holding a Frisbee in the grass, and there is a row of houses behind the dog", and the query question is "Which cat has fur similar to these animals?". Based on the query image, the visual objects obtained can be dog, Frisbee, and grass. Then the label text of the visual object "dog" can be "dog" or "animal", the label text of the visual object "Frisbee" can be "Frisbee" or "toy", and the label text of the visual object "grass" can be "grass" or "grass". Since the main query direction of the query question is animals, the generated image title text can be "a dog holding a Frisbee in its mouth", and the visual object "grass" does not need to be considered. The label text and image title text of the visual object are obtained based on the query image and query text, so that the label text and image title text are highly correlated with the query image and query text, which helps to extract highly correlated visual features and text features, and is more accurate in subsequent retrieval.

[0105] Among them, since the query text is a text collection containing rich text information, in the process of encoding the query text, the output features of the feedforward network are pooled at multiple granularities to obtain multiple pooling features, the multiple pooling features are fused to obtain fused pooling features, the fused pooling features are summed with the output features, and the first text feature is obtained according to the summation result; the feedforward network is used for feature extraction, which can be a fully connected neural network (Fully-connected neural network, FCNN) or a multilayer perceptron (Multilayer Perceptron, MLP), and the feature dimension of the output feature of the feedforward network can be L represents the sequence length of the output feature; the fused pooling feature is a feature obtained by fusing the pooling feature, and the first text feature is a text feature obtained based on the fused pooling feature.

[0106] Specifically, in the process of encoding the query text to obtain the first text feature, referring to Figure 4 , Figure 4 An optional schematic diagram of encoding a query text provided by an embodiment of the present disclosure. Specifically, a self-attention operation is performed on the query text q to obtain a self-attention feature z1, the self-attention feature z1 is added to the query text and then normalized to obtain an intermediate feature f, the intermediate feature f is input into a feedforward network to obtain an output feature X of the feedforward network, the intermediate feature X is pooled at multiple granularities to obtain a fused pooled feature F, and finally, the fused pooled feature F is added to the intermediate feature and then normalized to obtain a first text feature Alternatively, the fusion pooling feature F is added to the output feature X and then normalized to obtain the first text feature Since the query text carries rich text information, pooling at different granularities can improve the ability to understand the query text in a fine-grained manner, so that the first text feature obtained can express the query text more accurately.

[0107] In a possible implementation, in the process of performing pooling of multiple granularities on the output features of the feedforward network to obtain pooling features of multiple granularities, and fusing the multiple pooling features to obtain fused pooling features, the output features of the feedforward network may be subjected to local maximum pooling to obtain a first pooling feature, the output features may be subjected to segmented maximum pooling to obtain a second pooling feature, the first pooling feature and the second pooling feature may be fused to obtain a third pooling feature, and the third pooling feature and the output feature may be summed to obtain a fused pooling feature. The first pooling feature is a feature obtained after local maximum pooling, and the second pooling feature is a feature obtained after segmented maximum pooling.

[0108] Specifically, the maximum pooling operation is performed on some features of the output features of the feedforward network in sequence to obtain the first pooling feature; the output features of the feedforward network are divided into multiple sub-features, and each sub-feature is subjected to segmented maximum pooling to obtain the second pooling feature. The first pooling feature is fused with the second pooling feature to obtain the third pooling feature, wherein the feature fusion can be performed by adding the first pooling feature to the second pooling feature at the element level. Finally, the third pooling feature is added to the output feature to obtain the fused pooling feature. According to the above description, the process of obtaining the fused pooling feature X~ can be expressed by the following formula,

[0109]

[0110] Where X represents the output feature of the feedforward network; Represents a multi-granularity pooling operation, and the output is the third pooling feature. By fusing the pooling features of multiple granularities, it is possible to capture text detail information at different levels and granularities. This fusion process improves the feature expression ability of the fused pooling features, thereby enhancing the modeling ability of text information.

[0111] In addition, the output features of the feedforward network can also be average pooled to reduce the noise and fluctuation of the output features; or, the output features of the feedforward network can be mixed pooled by combining average pooling and maximum pooling. The mixed pooling can adaptively adjust the pooling strategy to better adapt to the output features of different feature distributions; or, the output features of the feedforward network can be spatially pyramid pooled. The spatial pyramid pooling can process output features of different sequence lengths, which helps to improve the generalization ability of processing data. In addition, pooling methods such as overlapping pooling and adaptive pooling can also be applied to the output features of the feedforward network.

[0112] In a possible implementation, in the process of fusing the first pooling feature with the second pooling feature to obtain the third pooling feature, specifically, the first weight corresponding to the first pooling feature is obtained according to the output feature and the first learnable matrix, the second weight corresponding to the second pooling feature is obtained according to the output feature and the second learnable matrix, and the first pooling feature and the second pooling feature are weighted and summed according to the first weight and the second weight to obtain the third pooling feature. Among them, the first learnable matrix is ​​used to control and adjust the importance of each part in the first pooling feature, and the first weight is used to adjust the contribution of the first pooling feature in the fused pooling feature; the second learnable matrix is ​​used to control and adjust the importance of each part in the second pooling feature, and the second weight is used to adjust the contribution of the second pooling feature in the fused pooling feature.

[0113] Specifically, the output feature is matrix multiplied with the first learnable matrix to obtain the first weight corresponding to the first pooling feature, and the output feature is matrix multiplied with the second learnable matrix to obtain the second weight corresponding to the second pooling feature. Then, the product result of the first pooling feature is weighted and summed based on the first weight and the second weight, and the product result of the second weight and the second pooling feature is added to obtain the third pooling feature. By introducing a learnable matrix to weight the pooling feature, the pooling features of different granularities can be combined with different weights to balance the contribution of the pooling features of different granularities, so as to better fuse the pooling features of multiple granularities and enhance the interpretability of the third pooling feature. In addition, by adjusting the weights of the pooling features of different granularities for content generation, the influence of the pooling features of different granularities on the generation results can be clearly understood, and the learnable matrix can be further optimized based on the generation results to adjust the weights of the pooling features of different granularities, so that the content generation process can be adjusted and controlled more accurately, which helps to improve the accuracy of the generated target content.

[0114] In addition to weighting the pooling results based on the learnable matrix, the pooling input can also be weighted based on the learnable matrix. Specifically, the output features of the feedforward network are weighted using the first learnable matrix to highlight the important parts of the output features and weaken the less important parts of the output features. The output features are matrix multiplied by the first learnable matrix to obtain the first weight corresponding to the output features, and the first weight is locally maximum pooled to obtain the first pooled feature. Similarly, the output features are matrix multiplied by the second learnable matrix to obtain the second weight corresponding to the output features, and the second weight is segmented maximum pooled to obtain the second pooled feature. The first pooled feature is added to the second pooled feature to obtain a fused pooled feature. By weighting the pooled input, the pooling process can better focus on the key information of the input features, thereby obtaining more accurate pooled features, which helps to improve the accuracy of generating target content.

[0115] In a possible implementation, in the process of weighted summing the first pooling feature and the second pooling feature according to the first weight and the second weight to obtain the third pooling feature, the third weight may be obtained according to the output feature and the third learnable matrix, the second pooling feature is weighted according to the second weight and then Hadamard product is performed with the third weight, and the first pooling feature is weighted according to the first weight and then summed with the result of the Hadamard product to obtain the third pooling feature. The third learnable matrix is ​​used to adjust and update the third weight, and the third weight is used to capture the global information of the output feature.

[0116] Specifically, the third weight is obtained according to the product of the output feature and the third learnable matrix, the second pooling feature is weighted according to the second weight to obtain a weighted second pooling feature, and the weighted second pooling feature is Hadamard producted with the third weight to obtain a Hadamard product result. Next, the first pooling feature is weighted according to the first weight to obtain a weighted first pooling feature, and the weighted first pooling feature and the Hadamard product result are summed to obtain the third pooling feature. The third pooling feature φ is obtained above. POA The process can be expressed by the following formula:

[0117]

[0118] Among them, X is the output feature of the feedforward network, W L is the first learnable matrix, is the first weight, W S is the second learnable matrix, is the second weight, W G is the third learnable matrix, is the third weight; φ LMP (·) is the local maximum pooling operation, φ LMP (X) The feature obtained is the first pooling feature, φ sMP (·) is the segmented maximum pooling operation, φ SMP (X) The feature obtained is the second pooling feature.

[0119] In addition, when the pooled input is weighted based on the learnable matrix, that is, the output features of the feedforward network are weighted, the third pooled feature φ is obtained. POA The process can be expressed by the following formula.

[0120]

[0121] By fusing the third weight and the weighted second weight through the Hadamard product, the information interaction between the two features at the corresponding positions can be strengthened, so as to capture the intrinsic connection between the two features. In addition, since the computational complexity of the Hadamard product is relatively low, feature fusion through the Hadamard product can reduce the consumption of computing resources, improve the efficiency of feature fusion, and thus speed up the generation of target content.

[0122] Since the third weight is mainly used to capture the global information of the output features, when the current task does not need to pay attention to the global information of the output features, the third pooling feature φ is obtained POA The process can be expressed by the following formula.

[0123]

[0124] According to the above description, the process of fusing multiple pooling features to obtain fused pooling features can be referred to Figure 5 , Figure 5 The embodiments of the present disclosure provide Figure 4 An optional schematic diagram of a multi-granularity pooling layer in . Figure 5 The input feature X in is the output feature of the feedforward network. The input feature X is input into the local maximum pooling layer for local maximum pooling operation to obtain the pooling feature f1. The input feature X is input into the segmented maximum pooling layer for segmented maximum pooling operation to obtain the pooling feature f2. The input feature X is input into the first neural network for global average pooling to obtain the pooling feature f3. The pooling feature f2 is Hadamard-producted with the pooling feature f3 to obtain the fused feature f4. The fused feature f4, the pooling feature f1 and the input feature X are added to obtain the fused pooling feature F.

[0125] In a possible implementation, in the process of performing local maximum pooling on the output features of the feedforward network to obtain the first pooling feature, specifically, the padding feature can be obtained according to the average value of the output features of the feedforward network, the padding feature is spliced ​​at the beginning and end of the output feature, and sliding sampling is performed in the output features after splicing the padding feature based on the sliding window, and the features obtained by each sliding sampling are respectively subjected to maximum pooling to obtain the first pooling sub-feature corresponding to each sliding sampling, and multiple first pooling sub-features are spliced ​​to obtain the first pooling feature. Among them, the padding feature is an additional value or vector added to the edge of the output feature, which is used to adjust the size of the output feature to ensure that the obtained first pooling feature is consistent with the size of the output feature.

[0126] Specifically, the mean of all values ​​in the output feature of the feedforward network is calculated, and the calculated mean is used as the value of the filling feature. The filling features are spliced ​​at the beginning and end of the output feature to obtain the output feature after splicing the filling features. For example, assuming that the output feature of the feedforward network can be represented as [1,5,10,8,21] in the form of a vector, and the average value is 9, then the value of the filling feature is 9, and a layer of filling features is spliced ​​at the beginning and end of the output feature. The output feature after splicing the filling features can be [9,1,5,10,8,21,9]. Then, a sliding window is used to perform sliding sampling in the output feature after splicing the filling features according to a preset sliding step size, and the features in the sliding window area during sliding sampling are maximized pooled to obtain the first pooled sub-feature corresponding to the sliding sampling. After the sampling is completed, multiple first pooled sub-features are spliced ​​to obtain the first pooled feature consistent with the size of the output feature. By performing local maximum pooling on the output features of the feedforward network through a sliding window, the surrounding subsequence information can be effectively used to enhance the expressiveness of the current subsequence features, thereby generating sub-pooled features that retain the key features of the current subsequence. At the same time, the method of performing local maximum pooling reduces the complexity and consumption of calculations and improves the efficiency of query text encoding. In addition, using the mean of the output features as the value of the filling feature can ensure that the edge features are treated similarly to the internal features during the pooling process, thereby reducing the loss of text information.

[0127] Reference Figure 6 , Figure 6 The embodiments of the present disclosure provide Figure 5 An alternative diagram of the local maximum pooling layer in , Figure 6 The input features in have 5 subsequence features x1, x2, x3, x4, x5. Use a sliding window of size 3 with a sliding step of 1 to perform maximum pooling. For example, perform maximum pooling on the subsequence features x1, x2, x3 to obtain the pooled subfeatures Perform maximum pooling on subsequence features x2, x3, and x4 to obtain pooled sub-features Perform maximum pooling on subsequence features x3, x4, and x5 to obtain pooled sub-features In order to ensure that the size of the obtained pooled features is consistent with that of the input features, padding features x0 and x6 can be added before the subsequence feature x1 and after the subsequence feature x5, respectively. At this time, the subsequence features x0, x1, and x2 are pooled to obtain the pooled sub-features. Perform maximum pooling on subsequence features x4, x5, and x6 to obtain pooled sub-features Pooling sub-features Concatenate and obtain the first pooling feature.

[0128] In a possible implementation, in the process of performing segmented maximum pooling on the output feature to obtain the second pooling feature, the output feature can be specifically divided into multiple segmented sub-features based on a preset interval, and for any segmented sub-feature, the segmented sub-feature is subjected to maximum pooling to obtain the second pooling sub-feature corresponding to the segmented sub-feature, and the second pooling sub-feature is copied according to the size of the segmented sub-feature, and the multiple copied second pooling sub-features are spliced ​​to obtain the third pooling sub-feature corresponding to the segmented sub-feature, and the multiple third pooling sub-features are spliced ​​to obtain the second pooling feature. Among them, the preset interval can be regarded as segmenting the granularity, and can be a fixed sequence length of the segmented sub-features set by itself, and the number of segmented sub-features can be calculated based on the sequence length of the output feature and the preset interval.

[0129] Specifically, the output feature is divided into multiple segmentation sub-features along the sequence dimension based on a preset interval, and each segmentation sub-feature contains text information of different granularity. For any segmentation sub-feature, the segmentation sub-feature is subjected to maximum pooling, and each sequence in the segmentation sub-feature is aggregated to retain the key features in each segmentation sequence, and the second pooling sub-feature corresponding to the segmentation sub-feature is obtained. For example, the feature dimension of the output feature is The output features are divided along the sequence dimension based on the preset interval to obtain the segmentation sequence S = {S 1 ,S 2 ,...,S D}, where S 1 ,S 2 ,...,S D is a segmentation sub-feature. When the preset interval is t, the number D of segmentation sub-features can be For any segment sub-feature S m , and its corresponding sub-division sequence can be expressed as in, is the sub-feature S of the segment m The t subsequence features contained in . m Perform maximum pooling to obtain the second pooling sub-feature Get the second pooling sub-feature s m The process can be expressed by the following formula, where: Represents a max pooling operation.

[0130]

[0131] Next, the second pooling sub-feature is copied according to the size of the segmentation sub-feature, and the multiple copied second pooling sub-features are sequentially spliced ​​to obtain the third pooling sub-feature corresponding to the segmentation sub-feature. At this time, the size of the third pooling sub-feature is the same as the size of the segmentation sub-feature. Then, the third pooling sub-features corresponding to each segmentation sub-feature are spliced ​​to obtain the second pooling feature that is consistent with the output feature size of the feedforward network. Based on the above example, any segmentation sub-feature If the sequence length is t, then t second pooling sub-features need to be copied and t second pooling sub-features are concatenated to obtain the third pooling sub-feature Finally, the third pooling sub-features corresponding to each segment sub-feature To splice,

[0132] Get the second pooling feature By performing segmented maximum pooling on the output features of the feedforward network, it focuses on the maximum value of each segmented sub-feature and is insensitive to other values, which can effectively resist the interference of noise or data deformation, thereby improving the quality and reliability of query text encoding. In addition, segmented maximum pooling can perform independent maximum pooling operations on each segmented sub-feature, effectively improving the efficiency of query text encoding.

[0133] Reference Figure 7 , Figure 7 The embodiments of the present disclosure provide Figure 5 An optional schematic diagram of the segmented maximum pooling layer in , which pools the input features based on a preset interval of 4 Split into segment sub-features along the sequence dimension and segment sub-features For each segment feature S 1 and segment sub-feature S 2 Perform maximum pooling to obtain segment sub-feature S 1 The corresponding pooling sub-features and segment sub-feature S 2 The corresponding pooling sub-features Then the pooled sub-feature s 1 Copy the 4 pooled sub-features s 1 Splice to get pooling features Similarly, the four pooled sub-features s obtained by copying 2 Splice to get pooling features

[0134] Step S303: The visual feature is integrated with the first text feature to obtain a query feature, and reference texts related to the query text are retrieved according to the query feature.

[0135] In a possible implementation, in the process of fusing the visual feature with the first text feature to obtain the query feature, the visual feature can be aggregated based on the first text feature to align the visual feature with the first text feature, and the query feature is obtained according to the mean of the first text feature and the aligned visual feature.

[0136] Specifically, the first text feature and each image feature in the visual feature sequence are calculated separately, and the calculation results are aggregated to align the visual feature with the first text feature. Then, the first text feature and the aligned visual feature are fused by calculating the mean between the first text feature and the aligned visual feature to obtain the query feature. The process of fusion of the first text feature and the aligned visual feature to obtain the query feature can be expressed by the following formula:

[0137]

[0138] in, represents the feature fusion operation, Represents the first text feature, and its feature dimension is Represents the aligned visual features, and its feature dimension is By taking the average of the first text feature and the aligned visual feature for feature fusion, the difference between the first text feature and the aligned visual feature can be effectively smoothed, which helps to reduce the impact of extreme values ​​on the query feature and also reduces the noise and uncertainty that may be introduced in the process of extracting the first text feature and the aligned visual feature, thereby improving the stability and reliability of the query feature. In addition, the query feature obtained after taking the average value usually has a smaller fluctuation range, which further reduces the occurrence of extreme values ​​and outliers, thereby simplifying the difficulty and computational complexity of subsequent processing and helping to improve the accuracy of the target content.

[0139] In addition to achieving feature fusion by calculating the mean of the first text feature and the aligned visual feature, the first text feature and the aligned visual feature can also be added element by element to retain more text information and visual information. Alternatively, the weights of the first text feature and the aligned visual feature can be calculated through an attention mechanism, and the two features can be weightedly fused according to the calculated weights, which can effectively highlight important features and suppress minor features, and help capture key text information or visual information in the first text feature and the aligned visual feature, thereby obtaining a query feature that can accurately represent the query image and query text.

[0140] In a possible implementation, in the process of aggregating the visual features based on the first text feature so as to align the visual features with the first text feature, specifically, a similarity score between the first text feature and the visual feature is determined, the similarity score is normalized, an aggregation weight is obtained according to a ratio of the normalized similarity score to a temperature coefficient, and the visual features are aggregated according to the aggregation weight so as to align the visual features with the first text feature. The temperature coefficient is used to adjust the probability distribution coefficient of the similarity score.

[0141] Specifically, image features are taken out from the visual feature sequence, and for any image feature, the image feature is transposed, and the transposed image feature is sequentially dot-multiplied with the first text feature to obtain the similarity score between each image feature and the first text feature. After the similarity score is normalized, the aggregation weight is obtained according to the ratio of the normalized similarity score to the temperature coefficient, and the image feature is aligned with the first text feature according to the product result of the aggregation weight and the image feature. Finally, the product results corresponding to each image feature are aggregated to align the visual feature with the first text feature. The above feature alignment process can be expressed by the following formula,

[0142]

[0143] in, Represents the aligned visual features, which are global visual features and have a feature dimension of Represents the first text feature, and its feature dimension is g i Represents the i-th image feature in the visual feature sequence, and its feature dimension is π represents the temperature coefficient. When the value of the temperature coefficient is very small, it is possible to focus more on extracting the most significant and important visual features, such as the image features corresponding to the sub-image (local visual features). When the value of the temperature coefficient is large, it is usually more focused on extracting broader and more comprehensive visual features, such as the image features corresponding to the query image (global visual features). By calculating each image feature in the visual feature sequence with the first text feature, it is helpful to dig out deeper and more diverse feature information. This fine-grained feature information provides an accurate information basis for the generation of target content. Furthermore, the various calculation results are aggregated to align the visual features with the first text features, which not only strengthens the correlation between the visual features and the first text features, but also improves the accuracy of the alignment of the visual features with the first text features, which helps to improve the accuracy of generating target content.

[0144] Further, the method of aggregating visual features based on the first text feature to align the visual features with the first text feature is based on a text-guided gating aggregation strategy, referring to Figure 8 , Figure 8 which is an optional schematic diagram of the text-guided gating aggregation provided by the embodiments of the present disclosure. The dot product operation is performed on the visual features and the first text feature to obtain the dot product result a. The dot product result a is converted into a probability distribution μ through the Softmax function, and the weight fusion is performed on the probability distribution μ and the visual features to align the visual features with the first text feature.

[0145] Among them, the reference text is a knowledge text related to the query text obtained from an open-world knowledge source, and the knowledge text can be a text paragraph; the open-world knowledge source can be a search engine or a specified knowledge base.

[0146] In a possible implementation manner, in the process of retrieving the reference text related to the query text according to the query feature, specifically, each reference text in the knowledge base can be encoded to obtain the second text feature, and the reference text related to the query text is retrieved from multiple knowledge bases according to the similarity between the query feature and the second text feature. Among them, the knowledge base is constructed by information such as theoretical knowledge or factual data in a specific domain.

[0147] Specifically, multiple reference texts are obtained from the knowledge base, and these reference texts are used as candidate texts. The candidate text set can be represented as P = {p1, p2,... p N}, where p1, p2,... p N represent N candidate texts. Each candidate text in the candidate text set is text-encoded to obtain the second text feature corresponding to each candidate text. Among them, the second text feature can be a low-dimensional feature. Then, the second text feature corresponding to each candidate text is sequentially calculated for similarity with the query feature to obtain the similarity score corresponding to each second text feature. The similarity scores are arranged in descending order, and the first k (k << N) candidate texts corresponding to the candidate text features with the highest similarity scores are selected as the reference text. The process of calculating the similarity can be represented by the following formula,

[0148]

[0149] Among them, represents the similarity calculation operation, and the similarity calculation can be dot product similarity. E q (·) represents the sequence encoder, and the sequence encoder encodes based on the query image and the query text to generate the query feature. i represents the query image, t represents the query text, and E p(·) represents the second text encoder, which is used to encode the candidate text set. i ∈P represents the candidate text. Since the knowledge base has a clear knowledge orientation, by obtaining reference texts in the knowledge base, the reference texts can be obtained in the right direction, making the reference texts obtained after similarity calculation more reliable.

[0150] Furthermore, the above formula for calculating the similarity between the query feature and the second text feature can also be expressed as:

[0151]

[0152] Among them, q represents the query text, I represents the image set, p represents the candidate text, and E t (·) represents the text encoder, E t The output of (q) is the first text feature, E t The output of (p) is the second text feature, E i (·) represents the image encoder, E i The output of (I) is visual features; It represents a feature fusion layer, which is used to fuse the first text feature with the visual feature to obtain the query feature.

[0153] In a possible implementation, the first text feature is obtained by a first text encoder, the second text feature is encoded by a second text encoder, and the first text encoder and the second text encoder are trained through the following steps: multiple first sample texts in the same training batch and second sample texts related to each first sample text are obtained, the first sample text is encoded by the first text encoder to obtain a first sample feature, the second sample text related to the first sample text is encoded by the second text encoder to obtain a second sample feature, the second sample text in the training batch that is not related to the first sample text is encoded by the second text encoder to obtain a third sample feature, a first similarity between the first sample feature and the second sample feature, and a second similarity between the first sample feature and the third sample feature are determined, a target loss is determined according to the first similarity and the second similarity, and the first text encoder and the second text encoder are jointly trained based on the target loss. Among them, the first sample text can be regarded as an input sample, the second sample text is a sample text related to the first sample text, which contains a potential answer corresponding to the first sample text, and the second sample text containing the potential answer to the first sample can be regarded as a positive sample of the first sample text. Since there are multiple first sample texts in the same training batch, each first sample text has a corresponding second sample text. Therefore, for any current first sample text, the corresponding second sample text can be used as a positive sample of the current first sample text, and the second sample texts corresponding to other first sample texts can be used as negative samples of the current first sample text (that is, the second sample text that is irrelevant to the current first sample text in the training batch); the target loss is used to maximize the first similarity and minimize the second similarity to optimize the training of the first text encoder and the second text encoder.

[0154] Specifically, multiple first sample texts in the same training batch and second sample texts related to each first sample text are obtained, and the second sample texts include positive samples and negative samples of the first sample text. The first sample text is encoded by the first text encoder to obtain the first sample feature, and the second sample text related to the first sample text is encoded by the second text encoder, that is, the positive sample in the second sample text is encoded to obtain the second sample feature. Then, the second sample text that is unrelated to the first sample text is encoded by the second text encoder, that is, the negative sample in the second sample text is encoded to obtain the third sample feature. Then, the first similarity between the first sample feature and the second sample feature is calculated by dot product similarity, and the second similarity between the first sample feature and the third sample feature is calculated. The first similarity and the second similarity are exponentially calculated, and then the logarithm of the ratio of the first similarity after exponential calculation to the second similarity after exponential calculation is taken to determine the target loss. Finally, the first text encoder and the second text encoder are jointly trained based on the target loss. According to the above description, the target loss can be obtained by the following formula,

[0155]

[0156] Among them, q is the first sample text, p * is a positive sample in the second sample text, N(q) is a training sample text in the same batch including the first sample text, r is a negative sample in the second sample text, s(·) is a similarity calculation operation, and exp(·) is an exponential operation. The target loss is determined by the similarity between the input sample and the positive sample and the similarity between the input sample and the negative sample, so that the similarity between the input sample and the positive sample is maximized to more accurately identify the text with high relevance, and the similarity between the input sample and the negative sample is minimized to better distinguish the input sample from the text with low relevance, which helps to improve the ability of the first text encoder and the second text encoder to distinguish positive and negative samples, thereby improving the accuracy of retrieving reference text.

[0157] Step S304: calling the visual large language model to generate content based on the reference text, the query image and the query text to obtain the target content.

[0158] According to the above steps, a query image and a query text are obtained, and the label text of the visual object and the image title text related to the query text are extracted for the query image. The query text, the label text and the image title text are merged into the query text, and the query text is encoded to obtain the first text feature. In the process of encoding the query text, the output features of the feedforward network are subjected to segmented maximum pooling, local maximum pooling and global pooling to obtain pooling features of multiple granularities, and the multiple pooling features are subjected to feature fusion to obtain fused pooling features, and the first text feature is obtained based on the fused pooling features. Next, a sub-image corresponding to the visual object is extracted from the query image, and the query image and the sub-image are encoded to obtain visual features. The visual features are aligned with the first text features through similarity calculation, and the aligned visual features are aggregated to obtain query features. According to the query features, a reference text related to the query text is retrieved from the knowledge base, and the visual large language model is called to generate content based on the reference text, the query image and the query text to obtain the target content. Among them, the visual large language model is used to generate content related to the query text, which can be InstructBLIP; the target content is a representative and accurate answer generated by the visual large language model.

[0159] In a possible implementation, the visual large language model may include a visual encoder, a bridge module, and a large language model. Before calling the visual large language model to generate content based on reference text, query image, and query text to obtain target content, the visual large language model is trained. Specifically, the network parameters of the visual encoder and the bridge module are frozen, and a low-rank matrix is ​​introduced into the large language model to fine-tune the parameters. In the process of fine-tuning the visual large language model, the cross entropy loss can be used to guide the model training, and the weight parameters of the visual processing layer and the low-rank matrix can be optimized by maximizing the generated text in an autoregressive manner. Since the number of parameters of the low-rank matrix is ​​much less than the number of parameters of the original weight matrix of the visual large language model, by introducing a low-rank matrix to fine-tune the parameters, the number of training parameters and memory usage can be reduced, the computational consumption is effectively reduced, and the training efficiency is improved. At the same time, this training method can fine-tune and optimize the visual large language model by changing the training data while retaining the original performance of the visual large language model, so that the visual large language model can adapt to content generation in various scenarios, and improve the generalization ability of the visual large language model.

[0160] In a possible implementation, in the process of obtaining the target content, the reference text may be summarized to obtain the summary text, and the summary text, the query image and the query text may be input into the visual large language model for content generation to obtain the target content. Specifically, the summary text is the text obtained after content extraction of the reference text, and the number of words in the summary text is much smaller than the number of words in the reference text. Since the reference text usually contains some redundant or irrelevant information, when processing the reference text, the reference text is content extracted, and the key text information is extracted to form a text summary, and the summary text is obtained based on the text summary. For example, the reference text is "On a sunny afternoon, a lively and cute puppy ran happily on the vast grass. It sometimes excitedly chased the circle formed by its tail, and sometimes... recorded its carefree and free good time." After summarization, the reference text can be compressed into "a puppy playing on the grass." Next, the summary text, query image, and query text are input into the visual big language model, and the knowledge about the open world is supplemented by the summarized reference text in the summary text to generate content and obtain the target content. By summarizing the reference text, the computational cost in the process of generating content results can be reduced, and the burden of the visual big language model to identify effective information from the summary text can be reduced, thereby effectively improving the accuracy and speed of generating target content.

[0161] In a possible implementation, in the process of summarizing the reference text to obtain the summary text, a prompt text may be constructed based on the reference text, the image title text and the query text, and the prompt text may be input into the large language model for text generation to obtain the summary text. The prompt text is used to prompt the large language model to summarize the reference text based on the image title text and the query text, and the prompt text includes an image description item, a knowledge text item and a question item; the large language model may be a text compression model, such as an LLaMA model.

[0162] Specifically, the prompt text is constructed based on the image description item, knowledge text item and question item, referring to Fig. 9 , Fig. 9 An optional schematic diagram of a prompt text template provided in an embodiment of the present disclosure, wherein reference text is filled into a knowledge text item, image title text is filled into an image description item, and query text is filled into a question item, wherein the reference text filled into the knowledge text item can be represented as p i =P k ={p1,p2,...,p k}, P KIndicates the k candidate texts with the highest similarity scores in the candidate text set. Input the prompt text of the filled content into the large language model, summarize the reference text through the large language model, extract the key text information from the reference text, replace the content of the knowledge text item in the prompt text with the key text information, and obtain the summary text. Since one reference text corresponds to one summary text, the summary text set generated by the large language model can be expressed as S = {s1, s2, ..., s k By constructing prompt text to associate image title text, query text, and reference text, it helps the large language model understand the reference text from multiple angles and levels, thereby improving the accuracy of the large language model in generating summary text content and providing a solid foundation for the subsequent generation of target content.

[0163] In a possible implementation, there are multiple reference texts. In the process of calling the visual large language model to generate content based on the reference text, query image and query text to obtain the target content, specifically, for each reference text, the visual large language model is called to generate content based on the reference text, query image and query text to obtain the generation result corresponding to each reference text, and the multiple generation results are voted to obtain the target content. Specifically, the visual large language model is called to generate content based on the reference text, query image and query text to obtain the generation result corresponding to each reference text. Since the similarity between the query feature and the second text feature cannot fully measure the confidence of the generation result, the multiple generation results are voted, and the generation result with the most occurrences of the same content in the generation results is used as the target content. For example, there are five reference texts p1, p2, p3, p4, and p5. The result of content generation based on reference text p1 is "BBB", the result of content generation based on reference text p2 is "AABBBB", the result of content generation based on reference text p3 is "BBB", the result of content generation based on reference text p4 is "BBB", and the result of content generation based on reference text p5 is "AABBBB". Among the five generated results, the generated results of reference texts p1, p3, and p4 are "BBB", which means that the generated result "BBB" is voted for 3 times, and the generated results of reference texts p2 and p5 are "AABBBB", which means that the generated result "AABBBB" is voted for 2 times. At this time, the generated result "BBB" has 3 votes, and the generated result "AABBBB" has 2 votes, so the target content is "BBB". By voting on the generated results, the target content is determined according to the frequency of occurrence of the same generated results. The universality and consistency of the generated results are taken into account, avoiding the singleness and limitations of relying solely on similarity as the only judgment criterion, making the generated target content more accurate.

[0164] Alternatively, weights for voting can also be generated based on the similarity score between the query feature and the second text feature. Specifically, the similarity score is divided into multiple similarity intervals according to a fixed interval, and each similarity interval corresponds to a different voting weight. The generation result is weighted and voted according to the interval in which the similarity score between the query feature and the second text feature is located. At this time, the voting result can be a floating point number. Calculate all the voting results, and take the generation result with the highest final number of votes as the target content. For example, the weight for the similarity score in the interval [98, 100] is w1, and the weight for the similarity score in the interval [96, 98) is w2. There are 5 reference texts p1, p2, p3, p4, p5. The similarity score between the second text feature of the reference text p1 and the query feature is 99, and the generation result obtained by generating content based on the reference text p1 is "BBB". The similarity score between the second text feature of the reference text p2 and the query feature is 98.5, and the generation result obtained by generating content based on the reference text p2 is "AABBBB". The similarity score between the second text feature of the reference text p3 and the query feature is 97, and the generation result obtained by generating content based on the reference text p3 is "BBB". The similarity score between the second text feature of the reference text p4 and the query feature is 96.5, and the generation result obtained by generating content based on the reference text p4 is "BBB". The similarity score between the second text feature of the reference text p5 and the query feature is 98.7, and the generation result obtained by generating content based on the reference text p5 is "AABBBB". At this time, the result of weighted voting for the generation result "BBB" can be A1 = 99w1 + 97w2 + 96.5w2, and the result of weighted voting for the generation result "AABBBB" can be A2 = 98.5w1 + 98.7w1. When A1 > A2, the target content is "BBB"; when A1 < A2, the target content is "AABBBB". By performing weighted voting on the obtained generation results, the frequency of the same generation result can be associated with the similarity scores of each reference text, taking into account not only the repeatability of the generation results but also the matching degree between the query feature corresponding to the reference text and the second text feature, making the obtained target content more accurate and reliable.

[0165] Refer to Fig.10 , Fig.10 FIG. is an optional overall flowchart of the content generation method provided by the embodiments of the present disclosure. The principle of the speech enhancement method in the embodiments of the present disclosure will be described in detail below from an overall perspective:

[0166] The content generation method provided in the embodiment of the present application may be composed of a knowledge retriever, a knowledge compressor, and a knowledge enhancement generator, wherein the knowledge retriever is composed of a sequence encoder and a text paragraph encoder, the sequence encoder includes a first text encoder and an image encoder, and the text paragraph encoder is a second text encoder, referring to Fig.11 , Fig.11 An optional schematic diagram of a knowledge retriever provided for an embodiment of the present disclosure; the knowledge compressor may be a large language model; and the knowledge enhancement generator may be a visual large language model.

[0167] First, before generating the target content, the knowledge retriever, knowledge compressor and knowledge enhancement generator need to be trained. In the process of training the retriever, the first text encoder and the second text encoder are mainly trained. Specifically, sample text, positive samples related to the sample text and negative samples unrelated to the sample text are obtained, the first similarity between the sample text and the positive sample is calculated, the second similarity between the sample text and the negative sample is calculated, the target loss is determined according to the first similarity and the second similarity, and the first text encoder and the second text encoder are trained according to the target loss to maximize the first similarity and minimize the second similarity. In the process of training the knowledge enhancement generator, the original parameters of the knowledge enhancement generator are frozen, and a low-rank matrix is ​​introduced as a learnable parameter to achieve fine tuning of the parameters. The cross entropy loss function is used to optimize the parameters of the visual processing layer in the knowledge enhancement generator, and the generated text is maximized in an autoregressive manner.

[0168] After training, the query image and query text are obtained, object recognition and feature extraction are performed on the query image, image features are extracted for the recognized visual objects, and label text is assigned to the visual objects based on the image features, and image title text related to the query text is generated, and the query text, label text and image title text are merged. At the same time, the visual objects in the query image are identified and bounding boxes are drawn around them. The visual objects are cropped out of the query image through the bounding boxes to obtain the sub-images corresponding to the visual objects, and an image set is constructed based on the query image and each sub-image.

[0169] Next, the merged query text is input into the first text encoder for encoding, and a self-attention operation is performed on the query text to obtain a self-attention feature. The self-attention feature is added to the query text and then normalized to obtain an intermediate feature. The intermediate feature is input into the feedforward network to obtain the output feature of the feedforward network. The output feature of the feedforward network is subjected to local maximum pooling operation, segmented maximum pooling operation, and global pooling operation to obtain pooling features of different granularities. The three pooling features of different granularities are subjected to feature fusion to obtain a fused pooling feature. The fused pooling feature is added to the output feature of the feedforward network to obtain the first text feature.

[0170] Next, the image set is input into the visual encoder for encoding to obtain a visual feature sequence corresponding to the image set. The visual feature sequence contains multiple image features. The similarity between each image feature and the first text feature is calculated and then feature fusion is performed. The feature fusion results corresponding to each image feature are subjected to text-oriented gated aggregation to align the visual feature with the first text feature to obtain an aligned visual feature. Then, the mean between the first text feature and the aligned visual feature is calculated to perform feature fusion on the first text feature and the aligned visual feature to obtain the query feature.

[0171] Next, multiple reference texts are obtained from the knowledge base, and the reference texts are encoded by the second text encoder to obtain the second text features. The second text features corresponding to each reference text are similarly calculated with the query features to obtain the similarity scores corresponding to each second text feature. The similarity scores are arranged in descending order, and the first k reference texts with the highest similarity scores are selected and input into the knowledge compressor.

[0172] Next, a prompt text is constructed based on the reference text, image title text, and query text. The prompt text is input into the large language model. The reference text is summarized by the large language model, and key text is extracted from the reference text. The number of words in the key text is much smaller than that in the reference text. A summary text is obtained based on the key text. The summary text, query image, and query text are input into the knowledge enhancement generator. The summary text is used to supplement the knowledge enhancement generator with open world knowledge, and multiple generation results are obtained. Votes are cast for multiple generation results, and the generation result with the highest number of votes is used as the target content. At this point, the generation of the target content is completed.

[0173] The disclosed embodiment obtains the target content through three steps: knowledge retrieval, knowledge compression, and content generation. In the process of knowledge retrieval, a multi-granularity pooling operation is used to fuse the pooled features of multiple granularities to obtain a first text feature that can accurately express the query text information, thereby improving the first text encoder's ability to understand multi-granularity features. Next, each image feature in the visual feature is respectively similarly calculated with the first text feature and then aggregated to align the visual feature with the first text feature, and then the aligned visual feature is feature-fused with the first text feature to obtain a query feature containing rich visual information and text information. Based on the query feature, a reference text related to the query text is retrieved from the knowledge base, so that the retrieved reference text is highly correlated with the query text, and the reference text is compressed to reduce computational consumption. On this basis, content generation is performed based on the compressed reference text, query image, and query text to obtain reliable and accurate target content.

[0174] In one possible implementation, the content generation method provided by the embodiment of the present disclosure can be applied to visual question-answering tasks in a large model. Specifically, the user inputs the query image and the query text into the large model, and the large model encodes the query image and the query text respectively and then performs feature fusion to obtain query features containing rich image information and query text information. Next, the large model obtains multiple knowledge texts from the database, encodes the knowledge texts to obtain knowledge text features, and selects knowledge texts for content generation based on the similarity between the query features and the knowledge text features. Finally, the answer corresponding to the query image and the query text is generated based on the selected knowledge text, and the answer is returned to the user for viewing and analysis by the user.

[0175] It is to be understood that, although the steps in the above-mentioned flowcharts are sequentially displayed according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in the present embodiment, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the above-mentioned flowcharts may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the steps or stages in other steps.

[0176] Reference Fig.12 , Fig.12 This is an optional structural diagram of a content generation device provided in an embodiment of the present disclosure. The content generation device 1200 can execute the content generation method in any of the above embodiments. The device includes:

[0177] The first encoding module 1201 is used to obtain a query image, encode the query image, and obtain visual features;

[0178] The second encoding module 1202 is used to obtain a query text, encode the query text to obtain a first text feature, wherein, in the process of encoding the query text, the output features of the feedforward network are pooled at multiple granularities to obtain multiple pooling features, the multiple pooling features are fused to obtain a fused pooling feature, the fused pooling feature is summed with the output feature, and the first text feature is obtained according to the summation result;

[0179] A retrieval module 1203 is used to fuse the visual feature with the first text feature to obtain a query feature, and retrieve reference text related to the query text according to the query feature;

[0180] The generation module 1204 is used to call the visual large language model to generate content based on the reference text, the query image and the query text to obtain the target content.

[0181] Further, the second encoding module 1202 is also used for:

[0182] Perform local maximum pooling on the output features of the feedforward network to obtain the first pooling features, and perform segmented maximum pooling on the output features to obtain the second pooling features;

[0183] The first pooling feature is fused with the second pooling feature to obtain the third pooling feature, and the third pooling feature is summed with the output feature to obtain the fused pooling feature.

[0184] Further, the second encoding module 1202 is also used for:

[0185] Obtain a first weight corresponding to the first pooling feature according to the output feature and the first learnable matrix, and obtain a second weight corresponding to the second pooling feature according to the output feature and the second learnable matrix;

[0186] According to the first weight and the second weight, the first pooling feature and the second pooling feature are weightedly summed to obtain a third pooling feature.

[0187] Further, the second encoding module 1202 is also used for:

[0188] Obtain a third weight according to the output feature and the third learnable matrix, weight the second pooling feature according to the second weight, and then perform a Hadamard product with the third weight;

[0189] After weighting the first pooling feature according to the first weight, the third pooling feature is obtained by summing the result of the Hadamard product.

[0190] Further, the second encoding module 1202 is also used for:

[0191] The padding feature is obtained according to the average value of the output feature of the feedforward network, and the padding feature is concatenated at the beginning and end of the output feature respectively;

[0192] Based on the sliding window, sliding sampling is performed in the output features after splicing and filling features, and the features obtained by each sliding sampling are respectively subjected to maximum pooling to obtain the first pooling sub-feature corresponding to each sliding sampling;

[0193] Multiple first pooling sub-features are concatenated to obtain the first pooling feature.

[0194] Further, the second encoding module 1202 is also used for:

[0195] Dividing the output feature into multiple segmented sub-features based on a preset interval;

[0196] For any segmentation sub-feature, perform maximum pooling on the segmentation sub-feature to obtain a second pooling sub-feature corresponding to the segmentation sub-feature, copy the second pooling sub-feature according to the size of the segmentation sub-feature, and concatenate the multiple copied second pooling sub-features to obtain a third pooling sub-feature corresponding to the segmentation sub-feature;

[0197] Multiple third pooling sub-features are concatenated to obtain the second pooling feature.

[0198] Furthermore, the first encoding module 1201 is also used for:

[0199] Aggregating the visual feature based on the first text feature to align the visual feature with the first text feature;

[0200] The query feature is obtained according to the mean of the first text feature and the aligned visual feature.

[0201] Furthermore, the first encoding module 1201 is also used for:

[0202] determining a similarity score between the first text feature and the visual feature, and normalizing the similarity score;

[0203] The aggregation weight is obtained according to the ratio of the normalized similarity score to the temperature coefficient;

[0204] The visual features are aggregated according to an aggregation weight to align the visual features with the first text feature.

[0205] Furthermore, the generating module 1204 is also used for:

[0206] Summarize the reference text to obtain a summary text;

[0207] The summary text, query image, and query text are input into the visual large language model for content generation to obtain the target content.

[0208] Furthermore, the generating module 1204 is also used for:

[0209] Constructing a prompt text based on the reference text, the image title text and the query text, wherein the prompt text is used to prompt the large language model to summarize the reference text according to the image title text and the query text;

[0210] The prompt text is input into the large language model for text generation to obtain the summary text.

[0211] Further, the first encoding module 1201 is also used for:

[0212] Perform object detection on the query image to obtain sub-images corresponding to the visual objects, and construct an image set based on the query image and the sub-images;

[0213] Encode the image collection to obtain visual features.

[0214] Furthermore, the search module 1203 is also used for:

[0215] Encode each reference text in the knowledge base to obtain a second text feature;

[0216] According to the similarity between the query feature and the second text feature, reference texts related to the query text are retrieved from multiple knowledge bases.

[0217] Furthermore, the search module 1203 is also used for:

[0218] Acquire multiple first sample texts in the same training batch and second sample texts related to each of the first sample texts;

[0219] Encoding the first sample text by a first text encoder to obtain a first sample feature;

[0220] Encoding a second sample text related to the first sample text by a second text encoder to obtain a second sample feature;

[0221] Encoding the second sample text in the training batch that is unrelated to the first sample text by a second text encoder to obtain a third sample feature;

[0222] Determining a first similarity between the first sample feature and the second sample feature, and a second similarity between the first sample feature and the third sample feature;

[0223] A target loss is determined according to the first similarity and the second similarity, and the first text encoder and the second text encoder are jointly trained based on the target loss, wherein the target loss is used to maximize the first similarity and minimize the second similarity.

[0224] Further, the second encoding module 1202 is also used for:

[0225] For the query image, extract the label text of the visual object and the image title text related to the query text;

[0226] The label text and the image title text are merged with the query text, and the merged query text is encoded to obtain a first text feature.

[0227] Furthermore, the generating module 1204 is also used for:

[0228] For each reference text, call the visual language model to generate content based on the reference text, query image and query text to obtain the generation result corresponding to each reference text;

[0229] Vote on multiple generated results to obtain the target content.

[0230] In summary, the content generation device 1200 in the embodiment of the present disclosure obtains a query image and a query text, encodes the query text to obtain a first text feature, wherein, in the process of encoding the query text, the output features of the feedforward network are pooled at multiple granularities to obtain multiple pooling features, the multiple pooling features are fused to obtain a fused pooling feature, the fused pooling feature is summed with the output feature, and the first text feature is obtained according to the summation result. Pooling at different granularities can improve the fine-grained understanding ability of the query text, thereby making the first text feature more accurate. Subsequently, the query image is encoded to obtain visual features, and the visual features are fused with the first text features to obtain query features. When a reference text related to the query text is retrieved according to the query features, since the first text feature is more accurate, the correlation between the reference text and the query text can be made higher. Therefore, when the visual large language model is called to generate content based on the reference text, the query image and the query text to obtain the target content, the accuracy of the target content can be effectively improved.

[0231] The electronic device for executing the above content generation method provided in the embodiment of the present disclosure may be a terminal. Fig.13 , Fig.13 This is a partial structural block diagram of a terminal provided in an embodiment of the present disclosure, and the terminal includes: a camera assembly 1310, a first memory 1320, an input unit 1330, a display unit 1340, a sensor 1350, an audio circuit 1360, a wireless fidelity (WiFi) module 1370, a first processor 1380, and a first power supply 1390. Those skilled in the art can understand that Fig.13 The terminal structure shown in the figure does not constitute a limitation on the terminal, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0232] The camera assembly 1310 can be used to capture images or videos. Optionally, the camera assembly 1310 includes a front camera and a rear camera. Typically, the front camera is disposed on the front panel of the terminal, and the rear camera is disposed on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtua l Reality) shooting function or other fusion shooting functions.

[0233] The first memory 1320 may be used to store software programs and modules. The first processor 1380 executes various functional applications and data processing of the terminal by running the software programs and modules stored in the first memory 1320 .

[0234] The input unit 1330 may be used to receive input digital or character information and generate key signal input related to the terminal's settings and function control. Specifically, the input unit 1330 may include a touch panel 1331 and other input devices 1332 .

[0235] The display unit 1340 may be used to display input information or provided information and various menus of the terminal. The display unit 1340 may include a display panel 1341 .

[0236] The audio circuit 1360 , the speaker 1361 , and the microphone 1362 may provide an audio interface.

[0237] The first power source 1390 may be alternating current, direct current, a disposable battery, or a rechargeable battery.

[0238] The number of sensors 1350 may be one or more, and the one or more sensors 1350 include but are not limited to: acceleration sensors, gyroscope sensors, pressure sensors, optical sensors, etc. Among them:

[0239] The acceleration sensor can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal. For example, the acceleration sensor can be used to detect the components of gravity acceleration on the three coordinate axes. The first processor 1380 can control the display unit 1340 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor. The acceleration sensor can also be used for collecting game or user motion data.

[0240] The gyroscope sensor can detect the body direction and rotation angle of the terminal, and the gyroscope sensor can cooperate with the acceleration sensor to collect the user's 3D actions on the terminal. The first processor 1380 can implement the following functions based on the data collected by the gyroscope sensor: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0241] The pressure sensor can be set in the side frame of the terminal and / or the lower layer of the display unit 1340. When the pressure sensor is set in the side frame of the terminal, the user's holding signal of the terminal can be detected, and the first processor 1380 performs left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor. When the pressure sensor is set in the lower layer of the display unit 1340, the first processor 1380 controls the operability controls on the UI interface according to the user's pressure operation on the display unit 1340. The operability control includes at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0242] The optical sensor is used to collect the ambient light intensity. In one embodiment, the first processor 1380 can control the display brightness of the display unit 1340 according to the ambient light intensity collected by the optical sensor. Specifically, when the ambient light intensity is high, the display brightness of the display unit 1340 is increased; when the ambient light intensity is low, the display brightness of the display unit 1340 is decreased. In another embodiment, the first processor 1380 can also dynamically adjust the shooting parameters of the camera assembly 1310 according to the ambient light intensity collected by the optical sensor.

[0243] In this embodiment, the first processor 1380 included in the terminal can execute the content generating method of the previous embodiment.

[0244] The electronic device for executing the above-mentioned content generation method provided in the embodiment of the present disclosure may also be a server. Fig.14 , Fig.14 Partial structural block diagram of a server provided in an embodiment of the present disclosure. The server may have relatively large differences due to different configurations or performances, and may include one or more second processors 1410 and a second memory 1430, and one or more storage media 1440 (e.g., one or more mass storage devices) storing application programs 1443 or data 1442. Among them, the second memory 1430 and the storage medium 1440 may be short-term storage or persistent storage. The program stored in the storage medium 1440 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the second processor 1410 may be configured to communicate with the storage medium 1440 to execute a series of instruction operations in the storage medium 1440 on the server.

[0245] The server may also include one or more second power supplies 1420, one or more wired or wireless network interfaces 1450, one or more input and output interfaces 1460, and / or one or more operating systems 1441, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0246] The second processor 1410 in the server may be configured to execute the content generating method.

[0247] The embodiments of the present disclosure further provide a computer-readable storage medium, which is used to store a computer program, and the computer program is used to execute the content generation method of each of the above embodiments.

[0248] The embodiment of the present disclosure also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the above-mentioned content generation method.

[0249] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate to describe the embodiments of the present disclosure, such as being able to be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0250] It should be understood that in the present disclosure, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0251] It should be understood that in the description of the embodiments of the present disclosure, the meaning of multiple (or multiple items) is more than two, greater than, less than, exceed, etc. are understood to not include the number, and above, below, within, etc. are understood to include the number.

[0252] In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0253] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0254] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0255] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0256] It should also be understood that the various implementations provided in the embodiments of the present disclosure can be combined arbitrarily to achieve different technical effects.

[0257] The above is a specific description of the preferred implementation of the present disclosure, but the present disclosure is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions under the shared conditions without violating the spirit of the present disclosure. These equivalent deformations or substitutions are all included in the scope defined by the claims of the present disclosure.

Claims

1. A content generation method, characterized in that: include: Acquire a query image, encode the query image, and obtain visual features; Acquire a query text, encode the query text to obtain a first text feature, wherein, in the process of encoding the query text, perform pooling of multiple granularities on the output features of the feedforward network to obtain multiple pooling features, fuse the multiple pooling features to obtain a fused pooling feature, sum the fused pooling feature and the output feature, and obtain the first text feature according to the summation result; fusing the visual feature with the first text feature to obtain a query feature, and retrieving reference text related to the query text according to the query feature; The visual large language model is called to generate content based on the reference text, the query image and the query text to obtain target content.

2. The content generation method according to claim 1, characterized in that: The output features of the feedforward network are pooled at various granularities to obtain various pooling features, and the various pooling features are fused to obtain fused pooling features, including: Performing local maximum pooling on the output features of the feedforward network to obtain a first pooling feature, and performing segmented maximum pooling on the output features to obtain a second pooling feature; The first pooling feature and the second pooling feature are fused to obtain a third pooling feature, and the third pooling feature and the output feature are summed to obtain a fused pooling feature.

3. The content generation method according to claim 2, characterized in that: The fusing the first pooling feature and the second pooling feature to obtain a third pooling feature includes: Obtaining a first weight corresponding to the first pooling feature according to the output feature and the first learnable matrix, and obtaining a second weight corresponding to the second pooling feature according to the output feature and the second learnable matrix; According to the first weight and the second weight, the first pooling feature and the second pooling feature are weightedly summed to obtain a third pooling feature.

4. The content generation method according to claim 3, characterized in that: The step of performing weighted summation of the first pooling feature and the second pooling feature to obtain a third pooling feature according to the first weight and the second weight includes: Obtaining a third weight according to the output feature and a third learnable matrix, and weighting the second pooling feature according to the second weight and performing a Hadamard product with the third weight; After weighting the first pooling feature according to the first weight, summing the result with the Hadamard product obtains the third pooling feature.

5. The content generation method according to claim 2, characterized in that: The performing local maximum pooling on the output features of the feedforward network to obtain the first pooling features includes: Obtaining a filling feature according to an average value of the output feature of the feedforward network, and splicing the filling features at the beginning and the end of the output feature respectively; Based on the sliding window, sliding sampling is performed in the output feature after the filling feature is spliced, and maximum pooling is performed on the features obtained by each sliding sampling to obtain the first pooling sub-feature corresponding to each sliding sampling; The plurality of the first pooling sub-features are concatenated to obtain a first pooling feature.

6. The content generation method according to claim 2, characterized in that: The performing segmented maximum pooling on the output feature to obtain a second pooling feature includes: Dividing the output feature into a plurality of segmented sub-features based on a preset interval; For any of the segmentation sub-features, perform maximum pooling on the segmentation sub-feature to obtain a second pooling sub-feature corresponding to the segmentation sub-feature, copy the second pooling sub-feature according to the size of the segmentation sub-feature, and concatenate the multiple copied second pooling sub-features to obtain a third pooling sub-feature corresponding to the segmentation sub-feature; A plurality of the third pooling sub-features are concatenated to obtain a second pooling feature.

7. The content generation method according to claim 1, characterized in that: The step of fusing the visual feature with the first text feature to obtain a query feature includes: aggregating the visual feature based on the first text feature to align the visual feature with the first text feature; A query feature is obtained according to the average of the first text feature and the aligned visual feature.

8. The content generation method according to claim 7, characterized in that: The step of aggregating the visual feature based on the first text feature so as to align the visual feature with the first text feature comprises: Determining a similarity score between the first text feature and the visual feature, and normalizing the similarity score; Obtaining an aggregation weight according to a ratio of the normalized similarity score to the temperature coefficient; The visual features are aggregated according to the aggregation weights to align the visual features with the first text features.

9. The content generation method according to claim 1, characterized in that: The calling of the visual large language model to generate content based on the reference text, the query image and the query text to obtain target content includes: Summarizing the reference text to obtain a summary text; The summary text, the query image and the query text are input into a visual large language model to generate content, thereby obtaining target content.

10. The content generation method according to claim 9, characterized in that: The step of summarizing the reference text to obtain a summary text includes: Constructing a prompt text based on the reference text, the image title text and the query text, wherein the prompt text is used to prompt the large language model to summarize the reference text according to the image title text and the query text; The prompt text is input into the large language model for text generation to obtain a summary text.

11. The content generation method according to claim 1, characterized in that: The step of encoding the query image to obtain visual features includes: Performing object detection on the query image to obtain a sub-image corresponding to the visual object, and constructing an image set based on the query image and the sub-image; The image set is encoded to obtain visual features.

12. The content generation method according to claim 1, characterized in that: The retrieving reference text related to the query text according to the query feature includes: Encode each reference text in the knowledge base to obtain a second text feature; According to the similarity between the query feature and the second text feature, the reference text related to the query text is retrieved from the plurality of knowledge bases.

13. The content generation method according to claim 12, characterized in that: The first text feature is obtained by encoding through a first text encoder, and the second text feature is obtained by encoding through a second text encoder, and the first text encoder and the second text encoder are trained by the following steps: Acquire a plurality of first sample texts in the same training batch and second sample texts related to each of the first sample texts; Encoding the first sample text by the first text encoder to obtain a first sample feature; Encoding the second sample text related to the first sample text by the second text encoder to obtain a second sample feature; Encoding the second sample text in the training batch that is unrelated to the first sample text by the second text encoder to obtain a third sample feature; Determining a first similarity between the first sample feature and the second sample feature, and a second similarity between the first sample feature and the third sample feature; A target loss is determined according to the first similarity and the second similarity, and the first text encoder and the second text encoder are jointly trained based on the target loss, wherein the target loss is used to maximize the first similarity and minimize the second similarity.

14. The content generation method according to claim 1, characterized in that: The step of encoding the query text to obtain a first text feature includes: For the query image, extracting label text of visual objects and image title text related to the query text; The label text and the image title text are combined with the query text, and the combined query text is encoded to obtain a first text feature.

15. The content generation method according to claim 1, characterized in that: The number of the reference texts is multiple, and the calling of the visual large language model generates content based on the reference texts, the query image and the query text to obtain target content, including: For each of the reference texts, the calling of the visual large language model generates content based on the reference text, the query image and the query text to obtain a generation result corresponding to each of the reference texts; Voting is performed on a plurality of the generated results to obtain target content.

16. A content generating device, characterized in that: include: A first encoding module, used to obtain a query image, encode the query image, and obtain visual features; A second encoding module is used to obtain a query text, encode the query text to obtain a first text feature, wherein, in the process of encoding the query text, the output features of the feedforward network are pooled at multiple granularities to obtain multiple pooling features, the multiple pooling features are fused to obtain a fused pooling feature, the fused pooling feature is summed with the output feature, and the first text feature is obtained according to the summation result; A retrieval module, configured to fuse the visual feature with the first text feature to obtain a query feature, and retrieve reference texts related to the query text according to the query feature; The generation module is used to call the visual large language model to generate content based on the reference text, the query image and the query text to obtain target content.

17. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the content generating method according to any one of claims 1 to 14 is implemented.

18. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the content generation method according to any one of claims 1 to 14 is implemented.

19. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the content generation method according to any one of claims 1 to 14 is implemented.

Images