Meme figure text matching generation method and device with image sub-figure number adaptability

Abstract

The application provides a meme picture text matching method and device with image subgraph number adaptability, which has the following characteristics: step S1, constructing a Chinese meme dataset according to existing multiple meme pictures, the Chinese meme dataset including multiple pictures and texts; step S2, training an existing large language model according to the Chinese meme dataset to obtain a final large language model and a reinforcement learning model; step S3, inputting a specified image into the existing multi-modal large model to obtain a specified humor chain and specified image features; step S4, inputting the specified humor chain and specified image features into the final large language model to obtain an initial text; and step S5, inputting the initial text into the reinforcement learning model to obtain a final text. In summary, the method can generate a humor text that matches a meme picture better.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of machine high-order cognition, and particularly relates to a meme image subgraph quantity adaptive meme image caption generation method and device. BACKGROUND

[0002] Humor is a fundamental human trait, deeply rooted in the understanding of language, culture, and social norms. For machines, understanding the essence of "humor" is particularly challenging. In the pursuit of more natural and empathetic human-machine interactions, endowing machines with a sense of humor has become a key research direction. It is worth noting that, although humor in the field of natural language processing (NLP) is interesting, humor in the real world is often resonated in a multi-modal framework, combining visual and textual elements. This is typically embodied in memes, which originated from the Greek word "Mimema" and were popularized by Richard Dawkins in his 1976 book "The Selfish Gene." Memes are not just an internet phenomenon; they also capture the spirit of the times, reflect social views, and become a key part of today's popular culture, especially due to the rise of social media.

[0003] Currently, multi-modal humor research is expanding. There are works that explore the dialogues of television sitcoms, while there are works that focus on "The Big Bang Theory" for humor detection. There are also works that provide humor datasets from television and memes, respectively. There are works that consider laughter in robot interactions. There are works that discuss humor theories. There are works that demonstrate models for multi-modal humor understanding and labeling. There are works that delve into humor and emotion in television programs. Regarding humor generation, there are works that focus on robot humor. In terms of meme generation, there are works that provide tools, datasets, and systems for memes. Text humor generation aims to produce comedic content. Templates often involve using tools like WordNet for lexical variation, as demonstrated by works in Japanese comedy and others in puns. However, they can be too formulaic. In contrast, neural models promise more originality. For example, there are works that use such models to create puns.

[0004] In addition, one aspect that is less explored by current research is the number of images in memes. The distinction between single-image and multi-image memes can be of great significance. Single-image memes associate key visual information with their corresponding textual parts, while multi-image memes require a nuanced understanding of the relationships between images and related texts, which are overlooked by existing methods.

[0005] In summary, there is still a lot of room for improvement in the existing technology for generating humor text, i.e., captions, for different meme images. SUMMARY

[0006] The present application is carried out in order to solve the above problems, and aims to provide a memetic graph caption generation method and device with image subgraph quantity adaptability.

[0007] The present application provides a memetic graph caption generation method with image subgraph quantity adaptability, which has the following characteristics: step S1, constructing a Chinese memetic dataset according to existing multiple memetic graphs, the Chinese memetic dataset including multiple pictures and captions; step S2, training an existing large language model according to the Chinese memetic dataset to obtain a final large language model and a reinforcement learning model; step S3, inputting a specified image into the existing multi-modal large model to obtain a specified humor chain and specified image features; step S4, inputting the specified humor chain and specified image features into the final large language model to obtain an initial caption; and step S5, inputting the initial caption into the reinforcement learning model to obtain a final caption, wherein step S2 includes the following sub-steps: step S2-1, obtaining humor chains and image features corresponding to each picture in the Chinese memetic dataset according to the multi-modal large model; step S2-2, performing supervised fine-tuning on the large language model according to the humor chains and image features to calculate a supervised fine-tuning loss; step S2-3, inputting each caption in the Chinese memetic dataset into the large language model to obtain corresponding text features; step S2-4, performing adaptive conversion and attention mechanism strengthening according to the text features and image features to calculate a prior loss; step S2-5, calculating a supervised learning total loss according to the prior loss and the supervised fine-tuning loss; step S2-6, updating the large language model according to the supervised learning total loss to obtain a trained large language model as the final large language model; step S2-7, inputting the humor chains and specified image features corresponding to each picture in the Chinese memetic dataset into the final large language model to obtain corresponding training captions; step S2-8, constructing a reward model according to all the training captions and captions; and step S2-9, performing reinforcement learning on the final large language model according to the reward model and all the training captions to obtain the reinforcement learning model.

[0008] In the memetic graph caption generation method with image subgraph quantity adaptability provided by the present application, the memetic graph can include a single-graph memetic graph and a multi-graph memetic graph, the single-graph memetic graph including a picture containing only one subgraph image and a corresponding caption, and the multi-graph memetic graph including a picture containing multiple subgraph images and a corresponding caption.

[0009] In the memetic graph caption generation method with image subgraph quantity adaptability provided by the present application, the calculation formula of the prior loss in step S2-4 can be: In the formula, and is the prior loss, SSFT S is the similarity between the predicted image and the caption for the large language model, I S is the similarity between the image and the caption calculated based on prior knowledge, S is the similarity score between the i-th image region of picture I and the j-th wordpiece of the caption of picture I obtained by supervised fine-tuning, S is the similarity between the i-th image region of picture I and the j-th wordpiece of the caption of picture I calculated based on prior knowledge (such as global and wordpiece-level similarity), i.e., wordpiece-level attention, g and t are trainable weights, KL is the KL divergence, S I and wordpiece-level attention The calculation expression of S In the formula, and are trainable weight matrices, and are corresponding bias terms, is the image feature corresponding to the i-th image region of picture I, T I is the text feature corresponding to the j-th wordpiece of the caption corresponding to picture I, W Q and W K are the weight matrix of the query and the weight matrix of the key in the self-attention mechanism, respectively, d k is the dimension of the key vector, is the attention score before normalization, S I,j S is the global attention of picture I, and N is the number of sub-images in picture I.

[0010] In the meme image caption generation method with image subgraph number adaptability provided by the present application, the contrast loss function can also have the following features: the contrast loss function is constructed according to the global attention and the wordpiece-level attention, which is used to maximize the similarity between the picture and the related caption and minimize the similarity between the picture and the unrelated caption, and the calculation expression of the contrast loss function is: In the formula, τ is a temperature parameter, L I is the loss between the i-th image region of picture I and the j-th wordpiece of the caption of picture I, and N' is the number of wordpieces of the caption of picture I

[0011] In the meme image caption generation method with image subgraph number adaptability provided by the present application, the calculation expression of the total loss of supervised learning in step S2-5 can also have the following features: In the formula, is the total loss of supervised learning, is the supervised fine-tuning loss, and λSFT-ori , λ t , and λ g are trainable weights of the corresponding items, respectively.

[0012] In the meme graph caption generation method with image subgraph number adaptability provided by the application, the step S2-8 can further include the following sub-steps: step S2-8-1, selecting a% of the training captions from all the training captions and performing evaluation ranking to obtain a first caption sequence; step S2-8-2, modifying the existing large language model with a softmax layer into a linear layer as an initial reward model; step S2-8-3, training the initial reward model according to the first caption sequence and the captions, and calculating a first loss L r ; step S2-8-4, updating the initial reward model according to the first loss L r to obtain a reward model.

[0013] In the meme graph caption generation method with image subgraph number adaptability provided by the application, the first loss L r can be calculated according to the following expression: wherein k is the number of training captions in the first caption sequence, x is a caption, D is a set of ranking pairs constructed according to the training captions in the first caption sequence, and the ranking pair contains a training caption y w with a higher ranking and a training caption y l with a lower ranking.

[0014] In the meme graph caption generation method with image subgraph number adaptability provided by the application, the step S2-9 can further include the following sub-steps: step S2-9-1, taking the final large language model as an initial reinforcement learning model; step S2-9-2, inputting the training text into the initial reinforcement learning model to obtain a corresponding optimized caption; step S2-9-3, calculating a second loss L RL according to the training text and the corresponding optimized caption in combination with the reward model; and step S2-9-4, updating the initial reinforcement learning model according to the second loss L RL to obtain a trained initial reinforcement learning model as a reinforcement learning model.

[0015] In the meme graph caption generation method with image subgraph number adaptability provided by the application, the second loss L RL can be calculated according to the following expression: wherein is an optimized caption generated by the initial reinforcement learning model, π SFT (y / x) is a training caption corresponding to the optimized caption, and are trainable weights, r θ (x,y) is a score, i.e., a reward, computed by the reward model from the training caption x and the optimized caption y.

[0016] The application also provides a memetic graph caption generation device with image subgraph quantity adaptability, which is used for generating a corresponding final caption for a specified image of a user, and has the characteristics that the device comprises a memetic graph input module for inputting the specified image of the user, a data extraction module comprising an existing multi-modal large model, which is used for extracting a specified humor chain and specified image features from the specified image, an initial caption generation module comprising a final large language model, which is used for generating an initial caption according to the specified humor chain and the specified image features, and a final caption generation module comprising a reinforcement learning model, which is used for generating a final caption according to the initial caption, wherein the final large language model and the reinforcement learning model are obtained by training an existing large language model by using a Chinese memetic dataset, and the Chinese memetic dataset is constructed according to existing multiple memetic graphs and comprises multiple pictures and captions, and the construction process of the final large language model and the reinforcement learning model comprises the following steps: step S2-1, obtaining the humor chain and the image features corresponding to each picture in the Chinese memetic dataset according to the multi-modal large model; step S2-2, performing supervised fine-tuning on the large language model according to the humor chain and the image features, and calculating a supervised fine-tuning loss; step S2-3, inputting each caption in the Chinese memetic dataset into the large language model to obtain corresponding text features; step S2-4, performing adaptive conversion and attention mechanism strengthening according to the text features and the image features, and calculating a priori loss; step S2-5, calculating a supervised learning total loss according to the priori loss and the supervised fine-tuning loss; step S2-6, updating the large language model according to the supervised learning total loss to obtain a trained large language model as the final large language model; step S2-7, inputting the humor chain and the specified image features corresponding to each picture in the Chinese memetic dataset into the final large language model to obtain corresponding training captions; step S2-8, constructing a reward model according to all the training captions and the captions; and step S2-9, performing reinforcement learning on the final large language model according to the reward model and all the training captions to obtain the reinforcement learning model.

[0017] Effects of the application

[0018] According to the meme image caption generation method and device with image subgraph number adaptability, when the large language model is supervised and fine-tuned and reinforced learning is performed, global attention and word segmentation level attention are used to consider the global and local similarities between the vision and the text, and a reward model obtained through ranking is combined to obtain the final large language model and the reinforcement learning model, so that the final caption with humor is generated for the specified image. Therefore, the meme image caption generation method and device with image subgraph number adaptability can generate a humorous caption that matches the meme image better. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 is a block diagram of a meme image caption generation device in an embodiment of the present application;

[0020] Figure 2 is a flowchart of a process of constructing a final large language model and a reinforcement learning model in an embodiment of the present application;

[0021] Figure 3 is a flowchart of a process of constructing a final large language model and a reinforcement learning model in an embodiment of the present application;

[0022] Figure 4 is a schematic diagram of a single-image meme image in the embodiment;

[0023] Figure 5 is a schematic diagram of a multi-image meme image in the embodiment. DETAILED DESCRIPTION

[0024] In order to make the technical means, creative features, purposes and effects achieved by the present application easy to understand, the following embodiments will be specifically described in combination with the drawings.

[0025] The embodiment provides a meme image caption generation device with image subgraph number adaptability, which is used to generate a corresponding final caption for a specified image of a user.

[0026] Figure 1 is a block diagram of a meme image caption generation device in an embodiment of the present application.

[0027] As shown in Figure 1 , the meme image caption generation device 100 of the embodiment includes a meme image input module 10, a data extraction module 20, an initial caption generation module 30, a final caption generation module 40 and a display module 50.

[0028] The meme image input module 10 is used for a user to input a specified image.

[0029] The data extraction module 20 comprises an existing multimodal large model for extracting a specified humor chain and a specified image feature from a specified image.

[0030] The initial caption generation module 30 comprises a final large language model for generating an initial caption according to the specified humor chain and the specified image feature.

[0031] The final caption generation module 40 comprises a reinforcement learning model for generating a final caption according to the initial caption.

[0032] The final large language model and the reinforcement learning model are trained from a Chinese meme dataset on an existing large language model, and in this embodiment, the existing large language model is baichuan2-13B.

[0033] The Chinese meme dataset is constructed from existing multiple meme images, including multiple pictures and captions, and in this embodiment, the Chinese meme dataset is constructed from 18,110 meme images from open source platforms and various Chinese meme image websites.

[0034] The meme images include single-image meme images and multi-image meme images. The single-image meme images include pictures containing only one sub-image and one corresponding caption, and the multi-image meme images include pictures containing multiple sub-images and one corresponding caption, emphasizing the relationship between individual sub-images and the caption.

[0035] In this embodiment, all meme images are further divided into four emotional types: self-praise, praise of others, self-mockery, and mockery of others according to the emotions of the meme images. In order to make the model trained from the Chinese meme dataset have a relatively balanced performance, the proportions of single-image meme images and multi-image meme images in the Chinese meme dataset are 54% and 46% respectively, and the proportions of the four emotional types of self-praise, praise of others, self-mockery, and mockery of others are 21%, 23%, 29%, and 27% respectively.

[0036] Figure 2 is a flowchart of constructing the final large language model and the reinforcement learning model in the embodiments of the present application.

[0037] As shown in Figure 2 , the final large language model and the reinforcement learning model are constructed in this embodiment by the Chinese meme dataset, combined with adaptive transformation and attention guidance, including the following steps:

[0038] Step S2-1, according to the multimodal large model, obtain the humor chain and image feature corresponding to each picture in the Chinese meme dataset.

[0039] In this embodiment, the sub-image extraction is performed on the picture containing multiple sub-images, and then data augmentation is performed on all sub-images in the Chinese meme dataset, and finally data extraction is performed through a multi-modal large model.

[0040] The process of sub-image extraction is to use OpenCV-Python to accurately identify each sub-image and its boundary in the picture containing multiple sub-images, and capture their coordinates by selecting ROIs. In this process, a mouse click function is declared to outline the boundary of each sub-image.

[0041] The process of data augmentation is to convert each sub-image into a corresponding augmented version using AutoAugment. The conversion process includes applying techniques such as cropping and rotating, which are customized according to the specific characteristics of each sub-image.

[0042] In data extraction, the multi-modal large model acts as a visual encoder to capture unique detailed features in each sub-image, such as object shape and color gradient, from each sub-image and its corresponding augmented version, thereby extracting deep features as image features. On the other hand, it generates a description for each sub-image and converts it into structured text using the "humor chain" template.

[0043] In this embodiment, both the humor chain and the specified humor chain are constructed through the "humor chain" template, which includes creating a narrative containing core concepts (e.g., main objects in sub-images), emotions (e.g., surprise), events (e.g., sharing photos), consequences (e.g., discussing photos), and humor elements (e.g., using anthropomorphism techniques).

[0044] Step S2-2, according to the humor chain and the image feature, the large language model is supervised fine-tuned, and the supervised fine-tuning loss is calculated.

[0045] Step S2-3, each caption in the Chinese meme dataset is input into the large language model to obtain the corresponding text feature.

[0046] Step S2-4, according to the text feature and the image feature, adaptive conversion and attention mechanism strengthening are performed, and the prior loss is calculated, wherein the calculation formula of the prior loss is:

[0047]

[0048]

[0049]

[0050]

[0051]

[0052]

[0053] wherein and is the prior loss, S SFT is the similarity rate predicted by the large language model, S I is the similarity between the image and the caption calculated based on the prior knowledge, is the similarity score of the caption obtained by supervised fine-tuning. This score is calculated based on the similarity between the caption predicted by supervised fine-tuning and the real caption, is the similarity between the i-th sub-image of the picture I and the j-th wordpiece of the caption of the picture I, i.e., the wordpiece-level attention, λ g and λ t are trainable weights, and KL is the KL divergence.

[0054] S I and the attention weight is calculated as:

[0055]

[0056]

[0057]

[0058]

[0059]

[0060] wherein and are both trainable weight matrices, and are both corresponding bias terms, is the image feature corresponding to the i-th sub-image of the picture I, T I is the text feature corresponding to the j-th wordpiece of the caption of the picture I, W Q and W K are respectively the weight matrix of the query and the weight matrix of the key in the self-attention mechanism, d k is the dimension of the key vector, is the attention score before normalization, S I,j is the global attention of the picture I, N is the number of sub-images in the picture I, and the prior knowledge in this embodiment includes the global similarity, i.e., the global attention, and the word-level similarity, i.e., the wordpiece-level attention.

[0061] Wherein, the contrast loss function is constructed according to the global attention and the segmentation-level attention, used for maximizing the similarity between the picture and the related caption while minimizing the similarity between the picture and the unrelated caption, and the calculation expression of the contrast loss function is:

[0062]

[0063] Wherein, τ is a temperature parameter, L I is the loss between the i-th sub-image of the picture I and the j-th segmentation of the caption of the picture I, and N' is the number of segmentations of the caption of the picture I.

[0064] Step S2-5, the supervised learning total loss is calculated according to the prior loss and the supervised fine-tuning loss, and the calculation expression of the supervised learning total loss is:

[0065]

[0066] Wherein is the supervised learning total loss, is the supervised fine-tuning loss, λ SFT-ori , λ t and λ g are the trainable weights of the corresponding items respectively.

[0067] In this embodiment, the global attention and the segmentation-level attention are constructed by the text features and the image features, which is the process of adaptive transformation, and the supervised learning total loss is constructed by the global attention and the segmentation-level attention, which is the process of attention-guided supervised fine-tuning, so as to improve the performance of the final large language model in generating captions.

[0068] Step S2-6, the large language model is updated according to the supervised learning total loss, and the trained large language model is obtained as the final large language model, and in this embodiment, the prediction text of the trained large language model is the initial caption and the similarity between the picture and the corresponding caption in the Chinese meme data set, i.e. the global attention and the segmentation-level attention can form good alignment.

[0069] Step S2-7, the humor chain corresponding to each picture in the Chinese meme data set and the specified image features are input into the final large language model to obtain the corresponding training captions.

[0070] Step S2-8, the reward model is constructed according to all the training captions and the captions.

[0071] Wherein, step S2-8 includes the following sub-steps:

[0072] Step S2-8-1, selecting a % of the training texts from all the training texts, and performing evaluation ranking to obtain a first text sequence, in the embodiment, a = 1, and the evaluation ranking is that a % of the training texts corresponding to the picture are manually evaluated, and the first text sequence is obtained by ranking the training texts according to the matching degree from small to large. The first text sequence is represented as {c1, c2,..., c k-1 k}.

[0073] Step S2-8-2, modifying the existing large language model with a softmax layer to a linear layer as an initial reward model.

[0074] Step S2-8-3, training the initial reward model according to the first text sequence and the text, and calculating a first loss L r .

[0075] The calculation expression of the first loss L r is as follows:

[0076]

[0077] In the formula, k is the number of training texts in the first text sequence, x is the text, D is a set of ranking pairs constructed according to the training texts in the first text sequence, and the ranking pair contains a training text y w with a higher ranking and a training text y l with a lower ranking.

[0078] Step S2-8-4, updating the initial reward model according to the first loss L r to obtain a reward model, in the embodiment, the reward model obtained by updating according to the first loss L r can assign a higher score, i.e., a reward, to an excellent training text, and assign a lower score, i.e., a penalty, to a poor training text, thereby skillfully reflecting the human preference, i.e., manual ranking, and the preference of the large language model.

[0079] Step S2-9, performing reinforcement learning on a final large language model according to the reward model and all the training texts to obtain a reinforcement learning model.

[0080] Step S2-9 includes the following sub-steps:

[0081] Step S2-9-1, taking the final large language model as an initial reinforcement learning model.

[0082] Step S2-9-2, inputting the training text into the initial reinforcement learning model to obtain a corresponding optimized text.

[0083] ​Step S2-9-3, according to the training text and the corresponding optimized text, combined with the reward model, the second loss L is calculated RL , wherein the second loss L RL The calculation expression is:

[0084]

[0085]

[0086]

[0087] In the formula The optimized text generated by the initial reinforcement learning model is π SFT (y / x) is the training text corresponding to the optimized text, and are trainable weights, r θ (x,y) is the score, i.e. reward, calculated by the reward model according to the training text x and the optimized text y.

[0088] Step S2-9-4, according to the second loss L RL Update the initial reinforcement learning model to obtain a trained initial reinforcement learning model as the reinforcement learning model.

[0089] The display module 50 is used to display the final text corresponding to the specified image generated according to the specified image to the user.

[0090] The following describes the process of the meme image text generation method with image subgraph number adaptability using the meme image text generation device with image subgraph number adaptability in the embodiment in combination with the drawings.

[0091] Figure 3 is the flowchart of the meme image text generation method with image subgraph number adaptability in the embodiment of the application.

[0092] As shown in Figure 3 , the meme image text generation method with image subgraph number adaptability in the embodiment includes the following steps:

[0093] Step S1, construct a Chinese meme data set according to existing multiple meme images.

[0094] Step S2, train an existing large language model according to the Chinese meme data set to obtain a final large language model and a reinforcement learning model.

[0095] Step S3, input the specified image received by the meme image input module 10 into the data extraction module 20 containing a multi-modal large model to obtain a specified humor chain and a specified image feature.

[0096] Step S4, inputting the specified humor chain and the specified image feature into the initial caption generation module 30 containing the final large language model to obtain an initial caption.

[0097] Step S5, inputting the initial caption into the final caption generation module 40 containing the reinforcement learning model to obtain a final caption.

[0098] Through performance testing on existing data sets, compared with the existing optimal MemeIFY method, the final caption generated by the meme image caption generation method with image subgraph number adaptability in the embodiment improves by 76.8% in the single-image meme image comprehensive index and by 72.8% in the multi-image meme image comprehensive index.

[0099] Figure 4 is a schematic diagram of a single-image meme image in the embodiment.

[0100] Figure 5 is a schematic diagram of a multi-image meme image in the embodiment.

[0101] As shown in Figure 4 , Figure 5 , the final caption generated by the meme image caption generation method with image subgraph number adaptability in the embodiment for the single-image meme image is "You look like your mom after you make a mistake", and the final caption generated for the multi-image meme image is "Once the best friends, always good friends", which shows that the meme image caption generation method with image subgraph number adaptability in the embodiment can generate a final caption that is very suitable for the specified image and has a sense of humor.

[0102] Effects of the embodiment

[0103] According to the meme image caption generation method and device with image subgraph number adaptability, when the large language model is supervised and fine-tuned and reinforced learning is performed, the global and local similarities between the vision and the text are considered through global attention and word-level attention, and the reward model obtained through ranking is combined to obtain the final large language model and the reinforcement learning model, so that a final caption with a sense of humor is generated for the specified image. In summary, the method can generate a humorous caption that matches the meme image better.

[0104] The above embodiments are preferred cases of the present application and do not limit the protection scope of the present application.

Claims

1. A memetic image caption generation method with image subgraph number adaptivity, used for generating a corresponding final caption for a specified image, characterized in that, The method comprises the following steps: Step S1, constructing a Chinese meme dataset according to existing multiple meme graphs, the Chinese meme dataset comprising multiple pictures and captions; Step S2, training an existing large language model according to the Chinese meme dataset to obtain a final large language model and a reinforcement learning model; Step S3, inputting the specified image into an existing multi-modal large model to obtain a specified humor chain and a specified image feature; Step S4, inputting the specified humor chain and the specified image feature into the final large language model to obtain an initial caption; Step S5, inputting the initial caption into the reinforcement learning model to obtain the final caption, wherein the step S2 comprises the following sub-steps: Step S2-1, obtaining a humor chain and an image feature corresponding to each picture in the Chinese meme dataset according to the multi-modal large model; Step S2-2, performing supervised fine-tuning on the large language model according to the humor chain and the image feature to calculate a supervised fine-tuning loss; Step S2-3, inputting each caption in the Chinese meme dataset into the large language model to obtain a corresponding text feature; Step S2-4, performing adaptive conversion and attention mechanism strengthening according to the text feature and the image feature to calculate a prior loss; wherein in the step S2-4, the calculation formula of the prior loss is: In the formula and is the prior loss, S SFT is the similarity between the image and the caption predicted by the large language model, S I is the similarity between the image and the caption calculated based on prior knowledge, is the similarity score between the i-th image region of the picture I and the j-th word piece of the caption of the picture I obtained by supervised fine-tuning, is the similarity between the i-th image region of the picture I and the j-th word piece of the caption of the picture I calculated based on prior knowledge, i.e., the word piece level attention, λ g and λ t are trainable weights, and KL is the KL divergence. The word segmentation level attention The calculation expression is: wherein and are trainable weight matrices, and are respective bias terms, is the image feature corresponding to the i-th image region of the image I, T I is the text feature corresponding to the j-th wordpiece of the caption corresponding to the image I, W Q and W K are the weight matrix of the query and the weight matrix of the key in the self-attention mechanism, respectively, d k is the dimension of the key vector, is the attention score before normalization, S I,j is the global attention of the image I, and N is the number of sub-images in the image I; Step S2-5, calculating a supervised learning total loss according to the prior loss and the supervised fine-tuning loss; Step S2-6, updating the large language model according to the supervised learning total loss to obtain a trained large language model as the final large language model; Step S2-7, inputting the humor chain and the specified image feature corresponding to each picture in the Chinese meme dataset into the final large language model to obtain a corresponding training caption; Step S2-8, constructing a reward model according to all the training captions and the captions; Step S2-9, performing reinforcement learning on the final large language model according to the reward model and all the training captions to obtain the reinforcement learning model.

2. The meme graph caption generation method with image subgraph number adaptability according to claim 1, wherein: wherein the meme graph comprises a single-image meme graph and a multi-image meme graph, the single-image meme graph comprises the picture containing only one image region and the corresponding one caption, the multi-image meme graph comprises the picture containing multiple image regions and the corresponding one caption.

3. The meme graph caption generation method with image subgraph number adaptability according to claim 1, wherein: wherein a contrast loss function is constructed according to the global attention and the word-level attention, which is used to maximize the similarity between the picture and the related caption wordings while minimizing the similarity between the picture and the unrelated caption wordings, the calculation expression of the contrast loss function is: where τ is a temperature parameter, L I is the loss between the i-th image region of picture I and the j-th segmented word of the caption of picture I, and N' is the number of segmented words of the caption of picture I.

4. The meme graph caption generation method with image subgraph number adaptability according to claim 1, wherein: wherein, In the step S2-5, the calculation expression of the total loss of the supervised learning is: wherein is the supervised learning total loss, is the supervised fine-tuning loss, λ SFT-ori , λ t , and λ g are trainable weights for the respective terms.

5. The meme image-text generation method with image sub-graph number adaptability according to claim 2, wherein: wherein, The step S2-8 comprises the following sub-steps: Step S2-8-1, select a% of the training texts from all the training texts, and perform evaluation sorting to obtain a first text sequence; Step S2-8-2, modify the existing large language model with a softmax layer to a linear layer as an initial reward model; Step S2-8-3, training the initial reward model according to the first text sequence and the text, to calculate a first loss L r ; Step S2-8-4, updating the first loss L r updating the initial reward model to obtain the reward model.

6. The meme image-text generation method with image sub-graph number adaptability according to claim 5, wherein: wherein The first loss L r The calculation expression is: where k is the number of training sequences in the first sequence of sequences, x is the sequence, D is a set of ranked pairs constructed from the training sequences in the first sequence of sequences, the ranked pairs comprising a higher ranked training sequence y w and a lower ranked training sequence y l .

7. The meme image-text generation method with image sub-graph number adaptability according to claim 2, wherein: wherein The step S2-9 comprises the following sub-steps: Step S2-9-1, use the final large language model as an initial reinforcement learning model; Step S2-9-2, input the training text into the initial reinforcement learning model to obtain a corresponding optimized text; Step S2-9-3, according to the training text and the corresponding optimization text, combined with the reward model, the second loss L is calculated RL ; Step S2-9-4, updating the second loss L RL updating the initial reinforcement learning model to obtain a trained initial reinforcement learning model as the reinforcement learning model.

8. The meme image-text generation method with image sub-graph number adaptability according to claim 7, wherein: wherein, The second loss L RL The calculation expression is: In the formula The optimized policy generated for the initial reinforcement learning model, π SFT (y / x) is the training policy corresponding to the optimized policy, And Are trainable weights, r θ (x, y) is the score, i.e. the reward, calculated by the reward model according to the training policy x and the optimized policy y.

9. A memetic graph caption generation apparatus with image subgraph number adaptivity, configured to generate a corresponding final caption for a specified image of a user, characterized in that, It comprises: A meme image input module for the user to input the specified image; A data extraction module comprising an existing multi-modal large model for extracting a specified humor chain and a specified image feature from the specified image; An initial text generation module comprising a final large language model for generating an initial text according to the specified humor chain and the specified image feature; A final text generation module comprising a reinforcement learning model for generating the final text according to the initial text, Wherein, the final large language model and the reinforcement learning model are trained from a Chinese meme dataset on an existing large language model, The Chinese meme dataset is constructed from existing multiple meme images, comprising multiple pictures and texts, The construction process of the final large language model and the reinforcement learning model comprises the following steps: Step S2-1, according to the multi-modal large model, obtain the humor chain and image feature corresponding to each picture in the Chinese meme dataset; Step S2-2, according to the humor chain and the image feature, perform supervised fine-tuning on the large language model to calculate the supervised fine-tuning loss; Step S2-3, input each text in the Chinese meme dataset into the large language model to obtain the corresponding text feature; Step S2-4, according to the text feature and the image feature, perform adaptive conversion and attention mechanism strengthening to calculate the prior loss; Wherein, in the step S2-4, the calculation formula of the prior loss is: wherein and is the prior loss, S SFT is the similarity between the image and the caption predicted by the large language model, S I is the similarity between the image and the caption calculated based on prior knowledge, is the similarity score between the i-th image region of the picture I and the j-th wordpiece of the caption of the picture I obtained by supervised fine-tuning, is the similarity between the i-th image region of the picture I and the j-th wordpiece of the caption of the picture I calculated based on prior knowledge, i.e., the wordpiece-level attention, λ g and λ t are trainable weights, and KL is the KL divergence, The word segmentation level attention The calculation expression is: wherein and are trainable weight matrices, and are respective bias terms, is the image feature corresponding to the i-th image region of the image I, T I is the text feature corresponding to the j-th wordpiece of the caption corresponding to the image I, W Q and W K are the weight matrix of the query and the weight matrix of the key in the self-attention mechanism, d k is the dimension of the key vector, is the attention score before normalization, S I,j is the global attention of the image I, and N is the number of sub-images in the image I; Step S2-5, calculate the total loss of the supervised learning according to the prior loss and the supervised fine-tuning loss; Step S2-6, update the large language model according to the total loss of the supervised learning to obtain a trained large language model as the final large language model; Step S2-7, input the humor chain and the specified image feature corresponding to each picture in the Chinese meme dataset into the final large language model to obtain the corresponding training text; Step S2-8, constructing a reward model according to all the training texts and the texts; Step S2-9, performing reinforcement learning on the final large language model according to the reward model and all the training texts, to obtain the reinforcement learning model.

Images