Method and apparatus for generating a caption generator and method and apparatus for outputting a caption - Patents.com

Abstract

The embodiment of the present disclosure discloses a method and apparatus for generating a caption generator, and a method and apparatus for outputting a caption. A specific embodiment of the method includes the steps of obtaining a sample image set, inputting the sample image set to an image encoder of a sentence generator and outputting an object set, grouping the object set into a first object set that is an object set included in a predetermined object set and a second object set that is an object set excluded from the predetermined object set, inputting the object set output by the image encoder to a sentence decoder of the sentence generator, and in the decoding step, performing a beam search using the first object set and the second object set as constraints to generate a set of pseudo image-sentence pairs, and training a sentence generator using the set of pseudo image-sentence pairs as a sample set to obtain a caption generator.

Description

[Technical field]

[0001] <CROSS REFERENCE TO RELATED APPLICATIONS> This disclosure claims priority to a Chinese patent application filed on March 30, 2021, bearing application number 202110338045.X and entitled "Method and Apparatus for Generating a Caption Generator, and Method and Apparatus for Outputting a Caption," the entire contents of which are incorporated herein by reference.

[0002] TECHNICAL FIELD Embodiments of the present disclosure relate to the field of computer technology, and in particular to a method and apparatus for generating a caption generator, and a method and apparatus for outputting a caption. [Background technology]

[0003] Image captioning, the technique of automatically describing images in natural language text, is a new and rapidly growing research topic.

[0004] In the related art, most caption generators are trained on annotated image-sentence pairs and operate according to the codec paradigm, which first encodes the input image with a convolutional neural network and then decodes the sentence with a recurrent neural network. A series of works have upgraded the attention mechanism of image caption to enhance the cross-domain basis between visual content and natural sentences.

[0005] Related techniques rely heavily on a large number of training image-sentence pairs; on the one hand, obtaining these training image-sentence pairs is extremely expensive and time-consuming; on the other hand, excessive reliance on training image-sentence pairs hinders the widespread application of caption generators. Summary of the Invention

[0006] SUMMARY OF THE DISCLOSURE Embodiments of the present disclosure provide a method and apparatus for generating a caption generator, as well as a method and apparatus for outputting a caption.

[0007] An embodiment of the present disclosure provides a method for generating a caption generator, including the steps of obtaining a sample image set, inputting the sample image set into an image encoder of a sentence generator and outputting an object set, grouping the object set into a first object set which is an object set included in a predetermined object set and a second object set which is an object set excluded from the predetermined object set, inputting the object set output by the image encoder into a sentence decoder of the sentence generator, and in a decoding step, performing a beam search using the first object set and the second object set as constraints to generate a set of pseudo image-sentence pairs, and training the sentence generator using the set of pseudo image-sentence pairs as a sample set to obtain a caption generator.

[0008] In some embodiments, the method further includes optimizing the caption generator by at least one of the following methods: optimizing the caption generator by adversarially training the caption generator with a sentence identifier; optimizing the caption generator by the inclusion degree of objects identified by the caption generator in sentences output from the caption generator; and optimizing the caption generator by the semantic correlation between image triplets including a query image, a positive image, and a negative image and the corresponding generated sentences.

[0009] In some embodiments, a method for optimizing a caption generator by adversarially training the caption generator with a sentence classifier includes the steps of: extracting a pre-established first sample set consisting of a first sample including an image and a corresponding real sentence; extracting a pre-created generative adversarial network including a caption generator for image-coding an input image and then performing sentence decoding to obtain a pseudo-sentence; and a sentence classifier for determining whether the input sentence is a pseudo-sentence output from the caption generator; selecting a first sample from the first sample set based on a machine learning method, inputting the image in the selected first sample to the caption generator to output the pseudo-sentence; inputting the pseudo-sentence and the real sentence in the selected first sample to the sentence classifier, and determining whether the pseudo-sentence is a pseudo-sentence output from the caption generator. Out The method includes a step of executing a first training step, which includes a step of inputting a classification result based on the output classification result, a step of calculating the accuracy rate of the sentence classifier based on the output classification result, and a step of determining that the training of the caption generator is completed when the accuracy rate reaches a predetermined value.

[0010] In some embodiments, the method further includes the steps of: if the accuracy rate does not reach the predetermined value, calculating an adversarial loss for the sentence classifier, adjusting associated parameters of the sentence classifier so that the adversarial loss is smaller, reselecting the first sample from the first sample set, and continuing to perform the first training step.

[0011] In some embodiments, the method further includes calculating an adversarial reward for the caption generator if the accuracy rate does not reach a predetermined value, adjusting associated parameters of the caption generator to increase the adversarial reward, reselecting the first sample from the first sample set, and continuing to perform the first training step.

[0012] In some embodiments, a method for optimizing a caption generator based on the inclusion degree of objects identified by the caption generator in sentences output from the caption generator includes performing a second training step including the steps of extracting a second predetermined sample set consisting of second samples including images, selecting samples from the second sample set based on a machine learning method, inputting images in the selected second samples to an image encoder of the caption generator, and outputting a sample object set, inputting the sample object set to a sentence decoder of the caption generator, and outputting pseudo-sentences, calculating an average confidence score that the pseudo-sentences contain sample objects in the sample object set as an object inclusion reward for the pseudo-sentence, and determining that training of the caption generator is completed when the object inclusion reward reaches a predetermined inclusion reward threshold.

[0013] In some embodiments, the method further includes, if the object inclusion reward has not reached a predetermined inclusion reward threshold, adjusting associated parameters of the caption generator to increase the object inclusion reward, reselecting a second sample from the second sample set, and continuing to perform the second training step.

[0014] In some embodiments, a method for optimizing a caption generator according to a semantic correlation between an image triplet and a corresponding generated sentence includes: extracting a third preset sample set consisting of third samples including a query image, a positive image, and a negative image, where the positive image shares at least two objects with the query image, and the negative image has no object in common with the query image; selecting a third sample from the third sample set based on a machine learning method, inputting the query image, the positive image, and the negative image in the selected third sample into a caption generator, respectively, and outputting the query sentence, the positive sentence, and the negative sentence; performing a third training step including: calculating a first semantic similarity between the query sentence and the positive sentence, calculating a second semantic similarity between the query sentence and the negative sentence, calculating a self-supervised triplet loss based on the first semantic similarity and the second semantic similarity, and determining that the training of the caption generator is completed if the self-supervised triplet loss is less than a predetermined loss threshold.

[0015] In some embodiments, the method further includes, if the self-supervised triplet loss is greater than or equal to a predetermined loss threshold, adjusting associated parameters of the caption generator such that the self-supervised triplet loss is smaller, reselecting a third sample from the third sample set, and continuing to perform the third training step.

[0016] In some embodiments, the step of calculating a first semantic similarity between the query sentence and the positive sentence and calculating a second semantic similarity between the query sentence and the negative sentence includes the steps of: calculating object-based probability distributions of each word in the sentence for the query sentence, the positive sentence, and the negative sentence, respectively, and performing a max pooling operation to obtain features of the query sentence, the positive sentence, and the negative sentence, respectively; calculating the first semantic similarity between the features of the query sentence and the features of the positive sentence, and calculating the second semantic similarity between the features of the query sentence and the features of the negative sentence.

[0017] In some embodiments, the method further includes adjusting associated parameters of the caption generator such that if a weighted sum of the adversarial reward, the object inclusion reward, and the self-supervised triplet loss is greater than a predetermined target value, the weighted sum is reduced.

[0018] In some embodiments, the image encoder includes a two-layer LSTM with a domain-level attention mechanism, where the first layer LSTM acts as a top-down attention module to compute object-level attention based on context information, and the second layer LSTM is a language model to generate sentences.

[0019] An embodiment of the present disclosure further provides a method for outputting a caption, comprising the steps of obtaining an image to be processed, inputting the image to a caption generator generated by the above method for generating a caption generator, and outputting a caption corresponding to the image.

[0020] An embodiment of the present disclosure further provides an apparatus for generating a caption generator, comprising: an acquisition unit configured to acquire a sample image set; an encoding unit configured to input the sample image set to an image encoder of the sentence generator and output an object set; a grouping unit configured to group the object set into a first object set which is an object set included in a predetermined object set and a second object set which is an object set excluded from the predetermined object set; a decoding unit configured to input the object set output by the image encoder to a sentence decoder of the sentence generator, and in a decoding step, perform a beam search using the first object set and the second object set as constraints to generate a set of pseudo image-sentence pairs; and a training unit configured to train the sentence generator using the set of pseudo image-sentence pairs as a sample set to obtain a caption generator.

[0021] In some embodiments, the apparatus further comprises an optimization unit configured to optimize the caption generator by at least one of the following methods: optimizing the caption generator by adversarially training the caption generator with a sentence identifier; optimizing the caption generator by a degree of inclusion of objects identified by the caption generator in sentences output from the caption generator; and optimizing the caption generator by a semantic correlation between an image triplet including a query image, a positive image, and a negative image and the corresponding generated sentence.

[0022] In some embodiments, the optimization unit further includes the steps of: extracting a pre-established first sample set, which is composed of a first sample including an image and a corresponding real sentence; extracting a pre-created generative adversarial network, which includes a caption generator for image-coding an input image and then decoding the sentence to obtain a pseudo-sentence; and a sentence identifier for determining whether the input sentence is a pseudo-sentence output from the caption generator; selecting a first sample from the first sample set based on a machine learning method, inputting the image in the selected first sample to the caption generator to output the pseudo-sentence; inputting the pseudo-sentence and the real sentence in the selected first sample to the sentence identifier, and determining whether the pseudo-sentence is a pseudo-sentence output from the caption generator. Out The method is configured to execute a first training step, which includes the steps of: inputting a classification result based on the classification result; calculating the accuracy rate of the sentence classifier based on the output classification result; and determining that training of the caption generator is completed when the accuracy rate reaches a predetermined value.

[0023] In some embodiments, the optimization unit is further configured to: if the accuracy rate does not reach the predetermined value, calculate an adversarial loss of the sentence classifier, adjust related parameters of the sentence classifier so that the adversarial loss is smaller, reselect the first sample from the first sample set, and continue to perform the first training step.

[0024] In some embodiments, the optimization unit is further configured to calculate an adversarial reward for the caption generator if the accuracy rate does not reach a predetermined value, adjust associated parameters of the caption generator so that the adversarial reward increases, reselect the first sample from the first sample set, and continue performing the first training step.

[0025] In some embodiments, the optimization unit is further configured to perform a second training step including the steps of: extracting a second predetermined sample set consisting of second samples including images; selecting a sample from the second sample set based on a machine learning method, inputting images in the selected second sample to an image encoder of the caption generator and outputting a sample object set; inputting the sample object set to a sentence decoder of the caption generator and outputting a pseudo-sentence; calculating an average confidence score that the pseudo-sentence contains a sample object in the sample object set as an object inclusion reward for the pseudo-sentence; and determining that training of the caption generator is completed when the object inclusion reward reaches a predetermined inclusion reward threshold.

[0026] In some embodiments, the optimization unit is further configured to adjust associated parameters of the caption generator so that the object inclusion reward is increased if the object inclusion reward has not reached a predetermined inclusion reward threshold, reselect a second sample from the second sample set, and continue performing the second training step.

[0027] In some embodiments, the optimization unit is further configured to perform the steps of: extracting a third preset sample set consisting of third samples including a query image, a positive image, and a negative image, where the positive image shares at least two objects with the query image, and the negative image has no object in common with the query image; selecting a third sample from the third sample set based on the machine learning device, inputting the query image, the positive image, and the negative image in the selected third sample to the caption generator, respectively, and outputting the query sentence, the positive sentence, and the negative sentence; calculating a first semantic similarity between the query sentence and the positive sentence, calculating a second semantic similarity between the query sentence and the negative sentence, calculating a self-supervised triplet loss based on the first semantic similarity and the second semantic similarity, and determining that training of the caption generator is completed if the self-supervised triplet loss is less than a predetermined loss threshold.

[0028] In some embodiments, the optimization unit is further configured to, when the self-supervised triplet loss is greater than or equal to a predetermined loss threshold, adjust associated parameters of the caption generator so that the self-supervised triplet loss is smaller, reselect a third sample from the third sample set, and continue to perform the third training step.

[0029] In some embodiments, the optimization unit is further configured to perform the steps of: calculating, for the query sentence, the positive sentence, and the negative sentence, an object-based probability distribution of each word in the sentence, respectively, and performing a max pooling operation to obtain features of the query sentence, features of the positive sentence, and features of the negative sentence, respectively; calculating a first semantic similarity between the features of the query sentence and the features of the positive sentence, and calculating a second semantic similarity between the features of the query sentence and the features of the negative sentence.

[0030] In some embodiments, the optimization unit is further configured to adjust relevant parameters of the caption generator such that a weighted sum of the adversarial reward, the object inclusion reward and the self-supervised triplet loss is smaller when the weighted sum is greater than a predetermined target value.

[0031] In some embodiments, the image encoder includes a two-layer LSTM with a domain-level attention mechanism, where the first layer LSTM acts as a top-down attention module to compute object-level attention based on context information, and the second layer LSTM is a language model to generate sentences.

[0032] An embodiment of the present disclosure provides an apparatus for outputting a caption, comprising: an acquisition unit configured to acquire an image to be processed; and an output unit configured to input the image to a caption generator generated by the above method for generating a caption generator, and output a caption corresponding to the image.

[0033] An embodiment of the present disclosure provides an electronic device comprising at least one processor and a memory in which at least one computer program is stored, wherein when the at least one computer program is executed by the at least one processor, the at least one processor implements the method for generating the above-mentioned caption generator.

[0034] An embodiment of the present disclosure provides a computer readable medium having stored thereon a computer program which, when executed by a processor, implements the method for generating a caption generator as described above.

[0035] An embodiment of the present disclosure provides a computer program which, when executed by a processor, implements the method for generating a caption generator as described above.

[0036] Other features, objects and advantages of the present disclosure will become more apparent upon reading the detailed description of the non-limiting embodiments that follow, with reference to the drawings, in which: [Brief description of the drawings]

[0037] [Figure 1] 1 is an exemplary system architecture to which an embodiment of the present disclosure can be applied. [Diagram 2] 1 is a flow chart of one embodiment of a method for generating a caption generator according to the present disclosure. [Diagram 3] 1 is a schematic diagram of an application scene of a method for generating a caption generator according to the present disclosure; FIG. [Figure 4] 1 is a flow chart of one embodiment of a method for outputting captions according to the present disclosure. [Diagram 5] 1 is a structural schematic diagram of an embodiment of an apparatus for generating a caption generator according to the present disclosure; [Figure 6] FIG. 2 is a structural schematic diagram of an embodiment of an apparatus for outputting captions according to the present disclosure; [Figure 7] FIG. 1 is a structural schematic diagram of a computer system suitable for implementing an electronic device according to one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0038] The present disclosure will be described in more detail below in combination with the accompanying drawings. It should be understood that the specific embodiments described in this specification are merely for interpreting related technical means, and are not limited to the technical means. For convenience of explanation, only the parts related to the technical means are shown in the drawings.

[0039] It should be understood that the embodiments and features of the embodiments in the present disclosure can be combined with each other unless they are inconsistent. Hereinafter, the present disclosure will be described in detail in combination with the embodiments with reference to the accompanying drawings.

[0040] FIG. 1 shows an exemplary system architecture 100 to which the method for generating a caption generator, the apparatus for generating a caption generator, the method for outputting a caption, or the apparatus for outputting a caption of the embodiments of the present disclosure can be applied.

[0041] 1, system architecture 100 may include terminals 101, 102, a network 103, a database server 104, and a server 105. Network 103 is used as a medium for providing communications links between terminals 101, 102, database server 104, and server 105. Network 103 may include various connection types, such as, for example, wired, wireless communication links, or fiber optic cables.

[0042] Users 110 can use terminals 101, 102 to interact with server 105 over network 103 to send and receive messages, etc. Various client applications can be installed on terminals 101, 102, such as model training applications, caption generation applications, image processing applications, shopping applications, payment applications, web browsers and instant messaging tools.

[0043] The terminals 101, 102 may be hardware or software. If the terminals 101, 102 are hardware, they may be various electronic devices having a display, including but not limited to smartphones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), laptop computers and desktop computers. If the terminals 101, 102 are software, they may be installed in the electronic devices listed above. They may be implemented as multiple software or software modules (e.g., used to provide distributed services) or as a single software or software module. There is no particular limitation here.

[0044] When the terminals 101 and 102 are hardware, they may be equipped with image capture devices. The image capture devices may be various devices capable of realizing image capture functions, such as cameras, sensors, etc. The user 110 can use the image capture devices on the terminals 101 and 102 to capture images of various scenes.

[0045] The database server 104 may be a database server that provides various services. For example, a sample set may be stored in the database server. The sample set includes a large number of samples. The samples may include sample images and sentences corresponding to the sample images. In this manner, the user 110 may select a sample from the sample set stored in the database server 104 via the terminals 101 and 102.

[0046] Server 105 may also be a server providing various services, for example a back-end server supporting various applications displayed on terminals 101 and 102. The back-end server may train an initial model using samples in the sample set sent by terminals 101 and 102, and send the training results (e.g., a generated caption generator) to terminals 101 and 102. In this way, a user may apply the generated caption generator to generate a caption for an image.

[0047] The database server 104 and the server 105 here may be hardware or software. If they are hardware, they may be implemented as a distributed server cluster consisting of multiple servers, or as a single server. If they are software, they may be implemented as multiple software programs or software modules (e.g., used to provide distributed services), or as a single software program or software module. There is no particular limitation here.

[0048] It should be noted that the method for generating a caption generator or the method for outputting a caption provided by the embodiment of the present disclosure are generally performed by the server 105. Therefore, the device for generating a caption generator or the device for outputting a caption are generally provided in the server 105.

[0049] It should be noted that if the server 105 is capable of implementing the relevant functionality of the database server 104, the system architecture 100 may not include the database server 104.

[0050] It should be understood that the number of terminals, networks, database servers and servers in Figure 1 are merely examples, and any number of terminals, networks, database servers and servers may be included depending on the needs of the implementation.

[0051] 2, a flow 200 of one embodiment of a method for generating a caption generator according to the present disclosure is shown. The method for generating a caption generator may include the following steps.

[0052] In step 201, a sample image set is obtained.

[0053] In this embodiment, the execution entity of the method for generating a caption generator (for example, the server shown in FIG. 1) can obtain a set of sample images stored in advance from a database server, and can obtain images taken by a terminal as sample images from the terminal.

[0054] In step 202, the sentence generator's image encoder inputs a set of sample images and outputs a set of objects.

[0055] In this embodiment, the sentence generator is an initial caption generator, which is a neural network that converts input images into sentences. The sentence generator may include an image encoder and a sentence decoder.

[0056] The image encoder generates an intermediate representation for each input image. In this disclosure, the most common object detection model (Faster R-CNN) is used as the image encoder to detect objects in the image, although other image encoders can be used in practical applications. For each image I i A set of prominent image regions

number

[0057] In step 203, the object set is grouped into a first object set and a second object set.

[0058] In this embodiment, the first object set is a set of objects included in the predetermined object set, and the second object set is a set of objects excluded from the predetermined object set. i Given a given object,detection model (such as Faster R-CNN) can

number

[0059] In step 204, the object set output by the image encoder is input to the sentence decoder of the sentence generator, and in the decoding step, a beam search is performed using the first object set and the second object set as constraints to generate a set of pseudo image-sentence pairs.

[0060] In this embodiment, given the intermediate representation generated by the image encoder, the output sentence is decoded verbatim by the sentence decoder. Referring to the bottom-up and top-down attention model, the sentence decoder can be implemented as a two-layer LSTM (Long Short-Term Memory) network with a domain-level attention mechanism. The first layer LSTM (LSTM 1 ) acts as a top-down attention module, which calculates object-level attention based on context information, and the second layer, LSTM ( 2 ) is a language model for generating sentences.

number

[0061] A natural way to generate pseudo image-sentence pairs using a pre-trained caption generator is to use beam search, which searches for a beam B at each decoding step. t It is a heuristic search algorithm that maintains the b most likely sentences and contains a part of the b most likely sentences. However, the semantic correlation between the input image and the output sentence is not fully utilized for sentence generation during inference. To alleviate this problem, the present disclosure designs a semantic constraint beam search in which the beam search is restructured to include the identified objects and exclude irrelevant objects.

[0062] Technically, the input image I i Given a given object,detection model (such as Faster R-CNN) can

number

number

number

number

[0063] Each pseudo image-sentence pair in the set of pseudo image-sentence pairs includes an image and a sentence, and the image and sentence may not be paired.

[0064] In step 205, a sentence generator is trained using the set of pseudo image-sentence pairs as a sample set to obtain a caption generator.

[0065] In this embodiment,

number

number

number

[0066] In some optional embodiments of this embodiment, the method further includes optimizing the caption generator by at least one of the following methods: optimizing the caption generator by adversarially training the caption generator with a sentence identifier; optimizing the caption generator by the inclusion degree of objects identified by the caption generator in sentences output from the caption generator; and optimizing the caption generator by the semantic correlation between image triplets including a query image, a positive image, and a negative image and the corresponding generated sentences.

[0067] The caption generator can be optimized in any of the above ways, or it can be optimized by a combination of any two of the ways, or it can be optimized by a combination of all three ways.

[0068] In some optional embodiments of this embodiment, a method for optimizing the caption generator by adversarially training the caption generator with a sentence classifier includes the steps of: extracting a pre-established first sample set consisting of a first sample including an image and a corresponding real sentence; extracting a pre-created generative adversarial network including a caption generator for image-coding an input image and then decoding the sentence to obtain a pseudo-sentence; and a sentence classifier for determining whether the input sentence is a pseudo-sentence output from the caption generator; selecting a first sample from the first sample set based on a machine learning method, inputting the image in the selected first sample to the caption generator and outputting a pseudo-sentence; inputting the pseudo-sentence and the real sentence in the selected first sample to the sentence classifier and determining whether the pseudo-sentence is a pseudo-sentence output from the caption generator. Out and executing a first training step including the steps of: inputting a classification result based on the output classification result; calculating a accuracy rate of the sentence classifier based on the output classification result; and determining that training of the caption generator is completed when the accuracy rate reaches a predetermined value.

[0069] If the accuracy rate does not reach a predetermined value, calculate an adversarial loss of the sentence classifier, adjust related parameters of the sentence classifier so that the adversarial loss is smaller, reselect a first sample from the first sample set, and continue to perform the first training step.

[0070] If the accuracy rate does not reach a predetermined value, calculate an adversarial reward for the caption generator, adjust related parameters of the caption generator so that the adversarial reward increases, reselect a first sample from the first sample set, and continue performing the first training step.

[0071] The structure of the sentence classifier is shown in Figure 3. The sentence classifier and the caption generator (including the image encoder and sentence decoder) constitute a generative adversarial network. The sentence classifier is used to identify whether the input sentence is a real sentence in the unpaired sentence dataset or a pseudo-sentence generated by the caption generator. In sentence modeling based on recurrent neural networks (RNNs), LSTM can be used to encode word sequences into a sentence-level representation format over context to identify real / generated sentences.

number

[0072] In each training process, the sentence classifier judges whether the input sentence is a real sentence or a pseudo-sentence generated by the caption generator. It statistically checks whether the classification result is correct, and if the accuracy rate reaches a predetermined value (for example, 0.5), it means that the pseudo-sentence of the caption generator is effective and can fool the sentence classifier, and the training is terminated. If not, it is necessary to adjust the network parameters of the caption generator and the sentence classifier and train again. First, fix the parameters of the caption generator, adjust the parameters of the sentence classifier and train, then fix the parameters of the sentence classifier, adjust the parameters of the caption generator and train. The parameters of the sentence classifier and the caption generator are adjusted alternately, and finally the training of the sentence classifier and the caption generator is completed. It is the caption generator that is actually used.

[0073] Adversarial reward is used to generate sentences that are indistinguishable from human-written captions, and in this disclosure we use adversarial training and sentence-level adversarial reward to ensure that the generated sentence distributions match those manually described.

number

[0074] In some optional embodiments of this embodiment, the method of optimizing the caption generator according to the degree of inclusion of objects identified by the caption generator in sentences output from the caption generator includes: Extracting a second preset sample set of second samples including images; selecting a sample from the second sample set based on a machine learning method; The method includes a step of performing a second training step, the second training step including the steps of inputting an image in the selected second sample to an image encoder of the caption generator and outputting a sample object set, inputting the sample object set to a sentence decoder of the caption generator and outputting a pseudo-sentence, calculating an average confidence score that the pseudo-sentence contains a sample object in the sample object set and setting it as an object inclusion reward for the pseudo-sentence, and determining that training of the caption generator is completed when the object inclusion reward reaches a predetermined inclusion reward threshold.

[0075] If the object inclusion reward has not reached a predetermined inclusion reward threshold, relevant parameters of the caption generator are adjusted so that the object inclusion reward is increased, a second sample is reselected from the second sample set, and the second training step is continued.

[0076] Object Inclusion Reward has no clear concept of explaining the semantic correlation between image content and generated sentences, because the adversarial reward only enhances the caption generator to generate more realistic sentences. Therefore, the present disclosure further takes the inclusion degree of the identified object in the output sentence as an additional self-supervised object (i.e., object inclusion reward) to encourage the caption generator to explain the inclusion of the identified object in the generated sentence. This method emphasizes the semantic correlation between the two and improves the quality of the generated caption.

number

[0077] In some optional embodiments of this embodiment, a method for optimizing the caption generator based on semantic correlation between image triplets and corresponding generated sentences includes the steps of: extracting a third preset sample set consisting of third samples including a query image, a positive image, and a negative image, where the positive image shares at least two objects with the query image, and the negative image does not have any common object with the query image; selecting a third sample from the third sample set based on a machine learning method, inputting the query image, the positive image, and the negative image in the selected third sample to the caption generator, respectively, and outputting a query sentence, a positive sentence, and a negative sentence, respectively; performing a third training step including the steps of calculating a first semantic similarity between the query sentence and the positive sentence, calculating a second semantic similarity between the query sentence and the negative sentence, calculating a self-supervised triplet loss based on the first semantic similarity and the second semantic similarity, and determining that training of the caption generator is completed if the self-supervised triplet loss is less than a predetermined loss threshold.

[0078] If the self-supervised triplet loss is greater than or equal to a predetermined loss threshold, adjust relevant parameters of the caption generator so that the self-supervised triplet loss is smaller, reselect a third sample from the third sample set, and continue performing the third training step.

[0079] In some optional embodiments of this embodiment, the step of calculating a first semantic similarity between the query sentence and the positive sentence and calculating a second semantic similarity between the query sentence and the negative sentence includes the steps of calculating object-based probability distributions of each word in the sentence for the query sentence, the positive sentence and the negative sentence, respectively, and performing a max pooling operation to obtain features of the query sentence, the positive sentence and the negative sentence, respectively; and calculating the first semantic similarity between the features of the query sentence and the features of the positive sentence and calculating the second semantic similarity between the features of the query sentence and the features of the negative sentence.

[0080] Self-supervised Triplet Loss uses the semantic correlation between each image and the corresponding generated sentence independently in optimizing with object inclusion reward, regardless of the similar or dissimilar relationship between the images. From the idea of ​​exploring the relative relationship, the present disclosure designs a self-supervised triplet loss that semantically constrains the learning of the caption generator in a triplet manner to preserve the relative semantic order between the sentences. Based on the visual objects identified in the images, each image triplet (composed of a query image, a positive image, and a negative image) is constructed. The positive image shares at least two objects with the query image, while the negative image and the query image do not have any objects in common. Given such an image triplet, the caption generator is optimized such that the generated sentence of the query image is more similar to the generated sentence of the positive image than the generated sentence of the negative image.

number

number

[0081] Optionally, finally the whole model training can combine self-critical sequence training with Adversarial Reward, Object Inclusion Reward and Self-supervised Triplet Loss, and the overall objective gradient formula is

number

number

[0082] Continuing to refer to FIG. 3, FIG. 3 is a schematic diagram of an application scene of the method for generating a caption generator according to the present embodiment. In the application scene of FIG. 3, a query image, a positive image and a negative image are input into the image encoder Faster R-CNN of the caption generator to obtain an object set {tree, man, bench, grass, dog...}. After grouping the object set based on the predetermined object set, input it into the sentence decoder (the two-layer LSTM structure at the bottom of FIG. 3) of the caption generator, and perform beam search decoding based on the meaning of the object set to generate pseudo-sentences such as "a man sitting on a bench near a tree". These pseudo-sentences and corresponding images are used as a set of pseudo-image-sentence pairs to train the caption generator (the two-layer LSTM structure at the top of FIG. 3 represents the sentence decoder). For convenience, the parameters of the image encoder can be fixed and only the sentence decoder can be trained, or the image encoder can be trained after the sentence decoder is trained, and the image encoder and sentence decoder are trained alternately to obtain the best-performing caption generator. The cross-entropy method is used in the training process. The parameters of the upper two-layer LSTM structure obtained can be shared with the lower two-layer LSTM structure.

[0083] The caption generator can be further optimized to introduce adversarial reward, object inclusion reward and self-supervised triplet loss. 1. Optimization of adversarial reward: The real sentence "a cow stands in the back of a large truck" is input to the sentence classifier together with the pseudo-sentence generated by the caption generator to be classified. If the classification accuracy rate is less than 0.5, the parameters of the sentence classifier are adjusted to minimize the adversarial loss, and then the parameters of the caption generator are adjusted to maximize the adversarial reward. The caption generator can be optimized by alternately training (adjusting) the sentence classifier and the caption generator. 2. Optimizing object inclusion reward: Calculate the inclusion degree of the identified objects in the pseudo-sentences generated by the caption generator. For example, the identified objects include tree, man, and bench. If sentence 1 contains tree (confidence 0.9) and sentence 2 contains tree (confidence 0.8) and man (confidence 0.7), the object inclusion reward of sentence 2 is higher than sentence 1. The goal of training is to improve the object inclusion reward as much as possible, and each time the parameters are adjusted, the object inclusion reward is improved. 3. Optimization of self-supervised triplet loss: The input sample in Figure 3 may be a triplet including a query image, a positive image, and a negative image. Different images can generate different pseudo-sentences, and the self-supervised triplet loss is determined by comparing the semantic similarity between the query sentence, the positive sentence, and the negative sentence. The training goal is to reduce the self-supervised triplet loss so that the positive sentence is semantically close to the query sentence, and the negative sentence is not semantically related to the query sentence.

[0084] In the training stage, the present disclosure adopts a self-learning mode, alternating between the two processes of generating pseudo-caption pairs and retraining the caption generator, thereby achieving the goal of optimizing the entire model and iteratively improving the caption generator.

[0085] This disclosure proposes a self-learning framework based on semantic constraints and deeply studies the self-learning idea of ​​unpaired image captioning. The problem is considered from the perspective of constructing pseudo-sentence generation and iterative optimization, and the quality of sentence generation is gradually improved. Furthermore, semantic constraints are successfully integrated into the model, making full use of the meaning of objects in images to guide the training of the caption generator, and an advanced unsupervised caption generation method is obtained.

[0086] 4, a flow 400 of another embodiment of a method for outputting a caption is shown. The flow 400 of the method for outputting a caption includes the following steps.

[0087] In step 401, an image to be processed is obtained.

[0088] In this embodiment, the electronic device in which the method for outputting captions is executed (e.g., the server shown in FIG. 1) can receive the image to be processed from the terminal where the user performs caption editing via a wired or wireless connection. The image to be processed may be a single image or a video file, and the server divides the video into frames to obtain the image to be processed.

[0089] In step 402, an image is input to a caption generator, which outputs a caption corresponding to the image.

[0090] In this embodiment, the caption generator is trained by the method of steps 201-205. The caption generator can automatically add a caption to an image. The caption can be output directly to the image, or a separate file can be generated and returned to the terminal, which can format the caption according to the user's needs and output it to the image. The caption generator can generate a caption Out Not only can it input the object classification information, but it can also output the objects identified by the image encoder, which can be used for semantic constraints during the training process.

[0091] Steps 401-402 may be performed alternately with steps 201-205. The captions generated in steps 401-402 may be used as training samples for steps 201-205.

[0092] As can be seen from Fig. 4, compared with the embodiment corresponding to Fig. 2, the flow 400 of the method for outputting a caption in this embodiment embodies the application step of a caption generator. In this way, the technical means described in this embodiment can optimize the caption generator and improve the generation accuracy of the caption by generating training samples through the caption generator, reusing them for training the caption generator, and alternating between generating captions and retraining the caption generator.

[0093] Further referring to FIG. 5, as an implementation of the method shown in each of the above figures, the present disclosure provides an embodiment of an apparatus for generating a caption generator, which embodiment of the apparatus corresponds to the embodiment of the method shown in FIG. 2, and which is applicable to various electronic devices.

[0094] As shown in Fig. 5, the apparatus 500 for generating a caption generator of this embodiment includes an acquisition unit 501, an encoding unit 502, a grouping unit 503, a decoding unit 504, and a training unit 505. The acquisition unit 501 is configured to acquire a sample image set, the encoding unit 502 is configured to input the sample image set to an image encoder of a sentence generator and output an object set, the grouping unit 503 is configured to group the object set into a first object set which is an object set included in a predetermined object set and a second object set which is an object set excluded from the predetermined object set, the decoding unit 504 is configured to input the object set output by the image encoder to a sentence decoder of the sentence generator, and in a decoding step, perform a beam search using the first object set and the second object set as constraints to generate a set of pseudo image-sentence pairs, and the training unit 505 is configured to train the sentence generator using the set of pseudo image-sentence pairs as a sample set to obtain a caption generator.

[0095] In this embodiment, the specific processing of the acquisition unit 501, the encoding unit 502, the grouping unit 503, the decoding unit 504 and the training unit 505 of the apparatus 500 for generating a caption generator may refer to steps 201, 202, 203, 204 and 205 in the corresponding embodiment of FIG. 2.

[0096] In some optional embodiments of this embodiment, the apparatus further comprises an optimization unit (not shown) configured to optimize the caption generator by at least one of the following methods: optimizing the caption generator by adversarially training the caption generator with a sentence identifier; optimizing the caption generator by the inclusion degree of objects identified by the caption generator in sentences output from the caption generator; and optimizing the caption generator by the semantic correlation between image triplets including a query image, a positive image, and a negative image and the corresponding generated sentences.

[0097] In some optional embodiments of this embodiment, the optimization unit further includes the steps of: extracting a pre-established first sample set consisting of a first sample including an image and a corresponding real sentence; extracting a pre-created generative adversarial network including a caption generator for image-coding an input image and then decoding the sentence to obtain a pseudo-sentence; and a sentence identifier for determining whether the input sentence is a pseudo-sentence output from the caption generator; selecting a first sample from the first sample set based on a machine learning device, inputting the image in the selected first sample to the caption generator to output the pseudo-sentence; inputting the pseudo-sentence and the real sentence of the selected first sample to the sentence identifier, and determining whether the pseudo-sentence is a pseudo-sentence output from the caption generator. OutThe method is configured to execute a first training step including the steps of: inputting a classification result based on the classification result; calculating the accuracy rate of the sentence classifier based on the output classification result; and determining that training of the caption generator is completed when the accuracy rate reaches a predetermined value.

[0098] In some optional embodiments of this embodiment, the optimization unit is further configured to: if the accuracy rate does not reach the predetermined value, calculate an adversarial loss for the sentence classifier, adjust related parameters of the sentence classifier so that the adversarial loss is smaller, reselect the first sample from the first sample set, and continue to perform the first training step.

[0099] In some optional embodiments of this embodiment, the optimization unit is further configured to calculate an adversarial reward for the caption generator if the accuracy rate does not reach a predetermined value, adjust associated parameters of the caption generator so that the adversarial reward increases, reselect the first sample from the first sample set, and continue performing the first training step.

[0100] In some optional embodiments of this embodiment, the optimization unit is further configured to perform a second training step including the steps of extracting a predetermined second sample set consisting of second samples including images; selecting a sample from the second sample set based on the machine learning device, inputting images in the selected second sample to an image encoder of the caption generator, and outputting a sample object set; inputting the sample object set to a sentence decoder of the caption generator, and outputting a pseudo-sentence; calculating an average confidence score of the sample objects of the sample object set included in the pseudo-sentence, as an object inclusion reward for the pseudo-sentence; and determining that training of the caption generator is completed when the object inclusion reward reaches a predetermined inclusion reward threshold.

[0101] In some optional embodiments of this embodiment, the optimization unit is further configured to adjust associated parameters of the caption generator so that the object inclusion reward is increased if the object inclusion reward has not reached a predetermined inclusion reward threshold, reselect a second sample from the second sample set, and continue performing the second training step.

[0102] In some optional embodiments of this embodiment, the optimization unit is further configured to perform the steps of: extracting a third predetermined sample set consisting of third samples including a query image, a positive image, and a negative image, where the positive image shares at least two objects with the query image and the negative image has no objects in common with the query image; selecting a third sample from the third sample set based on the machine learning device, inputting the query image, the positive image, and the negative image in the selected third sample to the caption generator, respectively, and outputting the query sentence, the positive sentence, and the negative sentence; calculating a first semantic similarity between the query sentence and the positive sentence, calculating a second semantic similarity between the query sentence and the negative sentence, calculating a self-supervised triplet loss based on the first semantic similarity and the second semantic similarity, and determining that training of the caption generator is completed if the self-supervised triplet loss is less than a predetermined loss threshold.

[0103] In some optional embodiments of this embodiment, the optimization unit is further configured to, if the self-supervised triplet loss is greater than or equal to a predetermined loss threshold, adjust associated parameters of the caption generator so that the self-supervised triplet loss is smaller, reselect a third sample from the third sample set, and continue to perform the third training step.

[0104] In some optional embodiments of this embodiment, the optimization unit is further configured to perform the steps of: calculating, for the query sentence, the positive sentence, and the negative sentence, an object-based probability distribution for each word in the sentence, respectively, and performing a max pooling operation to obtain features of the query sentence, the positive sentence, and the negative sentence, respectively; calculating a first semantic similarity between the features of the query sentence and the features of the positive sentence, and calculating a second semantic similarity between the features of the query sentence and the features of the negative sentence.

[0105] In some optional embodiments of this embodiment, the optimization unit is further configured to adjust relevant parameters of the caption generator such that when a weighted sum of the adversarial reward, the object inclusion reward and the self-supervised triplet loss is greater than a predetermined target value, the weighted sum is reduced.

[0106] In some optional embodiments of this embodiment, the image encoder includes a two-layer LSTM with a domain-level attention mechanism, where the first layer LSTM acts as a top-down attention module to calculate object-level attention based on context information, and the second layer LSTM is a language model for generating sentences.

[0107] Further referring to FIG. 6, as an implementation of the method shown in each of the above figures, the present disclosure provides an embodiment of an apparatus for outputting captions, which embodiment of the apparatus corresponds to the embodiment of the method shown in FIG. 4, and which is applicable to various electronic devices.

[0108] As shown in FIG. 6, the apparatus 600 for outputting a caption in this embodiment includes an acquisition unit 601 configured to acquire an image to be processed, and an output unit 602 configured to input the image to a caption generator generated by the apparatus 500 and output a caption corresponding to the image.

[0109] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device and a readable storage medium.

[0110] FIG. 7 illustrates a schematic block diagram of an exemplary electronic device 700 that may be used to implement embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, mobile phones, smartphones, wearable devices, and other similar computing devices. The components, their connections and relationships, and their functions illustrated herein are merely examples and are not intended to limit the practice of the present disclosure as described and / or claimed herein.

[0111] 7, the device 700 includes a computing unit 701 that can perform various appropriate operations and processes depending on a computer program stored in a read only memory (ROM) 702 or loaded from a storage unit 708 into a random access memory (RAM) 703. The RAM 703 may store various programs and data necessary for the operation of the device 700. The computing unit 701, the ROM 702 and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0112] The components in device 700 are connected to an I / O interface 705, including an input unit 706 such as a keyboard, a mouse, etc., an output unit 707 such as various types of displays, speakers, etc., a storage unit 708 such as a magnetic disk, an optical disk, etc., and a communication unit 709 such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 enables device 700 to exchange information / data with other devices, such as via the Internet's computer network and / or various telecommunication networks.

[0113] The computing unit 701 may be various general-purpose and / or special-purpose processing components having processing and computation capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, computing units that execute various machine learning model algorithms, a digital signal processor (DSP) and any suitable processor, controller, microcontroller, etc. The computing unit 701 executes each of the methods and processes described above, such as the method for generating a caption generator. For example, in some embodiments, the method for generating a caption generator may be implemented as a computer software program tangibly included in a machine-readable medium, such as the storage unit 708. In some embodiments, some or all of the computer program may be loaded and / or installed into the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the method for generating a caption generator described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the method for generating the caption generator by any other suitable means (eg, by firmware).

[0114] The method and apparatus for generating a caption generator and the method and apparatus for outputting a caption provided by the embodiments of the present disclosure aim to provide an unsupervised solution to image captioning. Unlike conventional image captioning methods that rely heavily on a large number of image-sentence pairs for training, the present disclosure eliminates this dependency by training a caption generator in a self-learning manner. The caption generator can be trained with unpaired image and sentence data in pursuit of more realistic scenes.

[0115] Various embodiments of the systems and techniques described above may be implemented in digital electronic circuitry systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may be embodied in one or more computer programs that may be executed and / or interpreted by a programmable system that includes at least one programmable processor, which may be a special purpose or general purpose programmable processor, and may include a processor that is capable of receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.

[0116] The program codes for implementing the methods of the present disclosure can be programmed employing any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general purpose computer, a special purpose computer or other programmable data processing apparatus, such that when the program codes are executed by the processor or controller, the functions / operations shown in the flowcharts and / or block diagrams are performed. The program codes may be entirely executed on a machine, partly executed on a machine, partly executed on a machine as a separate software package, partly executed on a remote machine, or entirely executed on a remote machine or server.

[0117] In the context of this disclosure, a machine-readable medium may be a tangible medium, and may contain or store a program used by or in combination with an instruction execution system, apparatus or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, apparatus or device, or any suitable combination of the above. More specific examples of machine-readable storage media include one or more wire-based electrical connections, portable computer disks, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), optical fibers, portable compact disk read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the above described.

[0118] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user, and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other types of devices can also be used to provide interaction with a user. For example, feedback provided to the user can be any form of sensing feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and can receive input from the user in any form, including acoustic, speech, or tactile input.

[0119] The systems and techniques described herein may be implemented in a computing system that includes background components (e.g., as a data server) or a computing system that includes middleware components (e.g., an application server) or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or an Internet browser through which a user may interact with embodiments of the systems and techniques described herein) or any combination of such background, middleware, or front-end components. The components of the system may be connected to each other by any form or medium of digital data communication (e.g., a communication network). Exemplary communication networks include a local area network (LAN), a wide area network (WAN), and an internetwork.

[0120] The computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The relationship of a client and a server is created by computer programs running on corresponding computers and having a client-server relationship to each other. The server may be a server of a distributed system or a server combined with a blockchain. The server may be a cloud server, an intelligent cloud computing server with artificial intelligence technology, or an intelligent cloud host. The server may be a server of a distributed system or a server combined with a blockchain. The server may be a cloud server, an intelligent cloud computing server with artificial intelligence technology, or an intelligent cloud host.

[0121] It should be understood that steps can be re-sorted, added or removed using the various forms of flow shown above. For example, each step described in this disclosure can be performed in parallel, sequentially, or in a different order, and is not limited herein as long as the technical solutions disclosed in this disclosure can achieve the desired results.

[0122] The above specific embodiments do not limit the scope of protection of the present disclosure. It should be understood that those skilled in the art can make various modifications, combinations, subcombinations and substitutions according to design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principle of the present disclosure should be included within the scope of protection of the present disclosure.

Claims

1. 1. A method for generating a caption generator, comprising: obtaining a sample image set; inputting the sample image set into an image encoder of a sentence generator and outputting a set of objects; grouping said object sets into a first object set, the first object set being the set of objects included in the predefined object set, and a second object set, the second object set being the set of objects excluded from the predefined object set; inputting the object set output by the image encoder to a sentence decoder of a sentence generator, and in a decoding step, performing a beam search using the first object set and the second object set as constraints to generate a set of pseudo image-sentence pairs; training the sentence generator using the set of pseudo image-sentence pairs as a sample set to obtain a caption generator; A method for generating a caption generator, comprising:

2. a method for optimizing the caption generator by adversarially training the caption generator with a sentence classifier; a method for optimizing the caption generator according to the degree of inclusion of objects identified by the caption generator in sentences output from the caption generator; A method for optimizing the caption generator based on semantic correlations between image triplets including a query image, a positive image, and a negative image, and corresponding generated sentences; 2. The method for generating a caption generator of claim 1, further comprising optimizing the caption generator by at least one of:

3. A method for optimizing the caption generator by adversarially training the caption generator with a sentence classifier includes: Extracting a first sample set including a first sample including a predetermined image and a corresponding real text; A step of extracting a pre-created generative adversarial network, the generative adversarial network including a caption generator for image-coding an input image and then decoding the sentence to obtain a pseudo-sentence, and a sentence identifier for determining whether the input sentence is a pseudo-sentence output from the caption generator; and selecting a first sample from the first sample set based on a machine learning method to perform a first training step; The first training step comprises: inputting the image in the first selected sample to the caption generator and outputting a pseudo-sentence; inputting the pseudo-sentence and the selected real sentence in the first sample to the sentence classifier, and outputting a classification result; calculating a correct answer rate of the sentence classifier based on the output classification result; and determining that training of the caption generator is complete when the accuracy rate reaches a predetermined value.

3. A method for generating a caption generator as claimed in claim 2.

4. 4. The method for generating a caption generator of claim 3, further comprising the steps of: if the accuracy rate does not reach the predetermined value, calculating an adversarial loss for the sentence classifier, adjusting associated parameters of the sentence classifier so that the adversarial loss is smaller, reselecting a first sample from the first sample set, and continuing to perform the first training step.

5. 4. The method for generating a caption generator of claim 3, further comprising the steps of: if the accuracy rate does not reach the predetermined value, calculating an adversarial reward for the caption generator, adjusting associated parameters of the caption generator so that the adversarial reward increases, reselecting a first sample from the first sample set, and continuing to perform the first training step.

6. A method for optimizing the caption generator according to a degree of inclusion of an object identified by the caption generator in a sentence output from the caption generator, comprising: Extracting a second sample set of a second set of samples including a predetermined image; and selecting samples from the second sample set based on a machine learning method to perform a second training step; The second training step comprises: inputting the selected second sample image into an image encoder of the caption generator and outputting a set of sample objects; inputting the set of sample objects into a sentence decoder of a caption generator to output pseudo-sentences; calculating an average confidence score that the pseudo-sentence contains a sample object of the sample object set, and setting the average confidence score as an object inclusion reward for the pseudo-sentence; determining that training of the caption generator is complete when the object inclusion reward reaches a predetermined inclusion reward threshold.

3. A method for generating a caption generator as claimed in claim 2.

7. The method for generating a caption generator as described in claim 6, further comprising the steps of adjusting associated parameters of the caption generator so that the object inclusion reward increases if the object inclusion reward has not reached the predetermined inclusion reward threshold, reselecting a second sample from the second sample set, and continuing to perform the second training step.

8. A method for optimizing the caption generator based on semantic correlation between image triplets and corresponding generated sentences includes: Extracting a third preset sample set, each of the third samples including a query image, a positive image and a negative image, the positive images sharing at least two objects with the query image, and the negative images having no objects in common with the query image; and selecting a third sample from the third sample set based on a machine learning method to perform a third training step; The third training step includes: inputting a query image, a positive image, and a negative image in the selected third sample into the caption generator, respectively, and outputting a query sentence, a positive sentence, and a negative sentence; calculating a first semantic similarity between the query sentence and the positive sentences, and calculating a second semantic similarity between the query sentence and the negative sentences; calculating a self-supervised triplet loss based on the first semantic similarity and the second semantic similarity; and determining that training of the caption generator is complete if the self-supervised triplet loss is below a predetermined loss threshold.

9. 9. The method for generating a caption generator of claim 8, further comprising the steps of: if the self-supervised triplet loss is greater than or equal to the predetermined loss threshold, adjusting associated parameters of the caption generator such that the self-supervised triplet loss is smaller, reselecting a third sample from the third sample set, and continuing to perform the third training step.

10. The step of calculating a first semantic similarity between the query sentence and the positive sentence and calculating a second semantic similarity between the query sentence and the negative sentence includes: For the query sentence, the positive sentence and the negative sentence, respectively, calculating an object-based probability distribution of each word in the sentence, and performing a max pooling operation to obtain the features of the query sentence, the features of the positive sentence and the features of the negative sentence; 9. The method for generating a caption generator of claim 8, comprising the steps of: calculating a first semantic similarity between features of the query sentence and features of the positive sentence; and calculating a second semantic similarity between features of the query sentence and features of the negative sentence.

11. The method for generating a caption generator according to any one of claims 2 to 10, further comprising the step of adjusting relevant parameters of the caption generator such that a weighted sum of the adversarial reward, the object inclusion reward and the self-supervised triplet loss is smaller if the weighted sum is larger than a predetermined target value.

12. The method for generating a caption generator according to any one of claims 1 to 10, wherein the image encoder includes a two-layer LSTM with a region-level attention mechanism, the first-layer LSTM acts as a top-down attention module and calculates object-level attention based on context information, and the second-layer LSTM is a language model for generating sentences.

13. 1. An apparatus for generating a caption generator, comprising: an acquisition unit configured to acquire a sample image set; an encoding unit configured to input the sample image set to an image encoder of a sentence generator and output a set of objects; a grouping unit configured to group said object set into a first object set, the first object set being a set of objects included in a predefined object set, and a second object set, the second object set being a set of objects excluded from the predefined object set; a decoding unit configured to input the object set output by the image encoder to a sentence decoder of a sentence generator, and in a decoding step, to perform a beam search using the first object set and the second object set as constraints to generate a set of pseudo image-sentence pairs; a training unit configured to train the sentence generator using the set of pseudo image-sentence pairs as a sample set to obtain a caption generator.

14. An electronic device comprising at least one processor and a memory in which at least one computer program is stored, Electronic device, wherein said at least one computer program, when executed by said at least one processor, causes said at least one processor to implement the method according to any one of claims 1 to 12.

15. A computer readable medium having stored thereon a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 12.

16. A computer program which, when executed by a processor, implements the method according to any one of claims 1 to 12.

Images