Model training method, image description generation method and device, equipment, medium

Abstract

The embodiment of the present application provides a model training method, image description generation method and device, equipment, and medium, which belongs to the field of artificial intelligence technology. The method includes: obtaining a sample image; performing target detection on the sample image to obtain a sample image category label; encoding the sample image category label through a preset reference state factor to obtain a target category label sequence; traversing a preset image description corpus, decoding the target category label sequence, and obtaining a pseudo description text; constructing a sample feature pair based on the sample image and the pseudo description text; training a preset neural network model based on the sample feature pair to obtain an image description generation model, and the image description generation model is used to generate a target image description text. The embodiment of the present application can improve the training effect of the model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a model training method, an image description generation method and apparatus, equipment, and medium. Background Art

[0002] At present, most image descriptions are often generated using supervised learning methods, that is, the basic features of the image are extracted through a supervised learning model, and corresponding description sentences are generated based on the basic features to obtain image description text. The training of supervised learning models often requires a large amount of training corpus, including training images and corresponding description sentences. However, it is difficult to obtain a large amount of high-quality training corpus, which affects the training effect of the model. Therefore, how to improve the training effect of the model has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a model training method, image description generation method and device, equipment, and medium, aiming to improve the training effect of the model.

[0004] To achieve the above objectives, a first aspect of an embodiment of the present application proposes a model training method, the method comprising:

[0005] Get a sample image;

[0006] Performing target detection on the sample image to obtain a sample image category label;

[0007] Encoding the sample image category labels through a preset reference state factor to obtain a target category label sequence;

[0008] Traversing a preset image description corpus, decoding the target category label sequence to obtain a pseudo description text;

[0009] Constructing a sample feature pair according to the sample image and the pseudo description text;

[0010] A preset neural network model is trained according to the sample features to obtain an image description generation model, and the image description generation model is used to generate a target image description text.

[0011] In some embodiments, the step of performing target detection on the sample image to obtain a sample image category label includes:

[0012] Input the sample image into a preset target detection model, wherein the target detection model includes a convolutional layer, an RPN layer, a RoI pooling layer and a fully connected layer;

[0013] Extracting features of the sample image through the convolution layer to obtain a sample feature map;

[0014] Performing target detection on the sample feature map through the RPN layer and preset coordinate information to obtain an initial detection frame;

[0015] Performing feature fusion processing on the sample feature map and the initial detection frame through the preset scaling parameters of the RoI pooling layer to obtain a sample fusion feature map;

[0016] The classification probability of the sample fusion feature map is calculated by using the prediction function of the fully connected layer and the preset image category label to obtain a classification probability value, and the preset image category label is screened according to the classification probability value to obtain the sample image category label.

[0017] In some embodiments, the step of traversing a preset image description corpus, decoding the target category label sequence, and obtaining a pseudo description text includes:

[0018] Calculating the distribution probability of the target category label sequence using a preset prediction function to obtain a distribution probability value;

[0019] The image description corpus is traversed, and the target category label sequence is decoded according to the distribution probability value to obtain the pseudo description text.

[0020] In some embodiments, the step of training a preset neural network model according to the sample features to obtain an image description generation model includes:

[0021] Inputting the sample feature pair into the neural network model, wherein the neural network model includes a generating network and a discriminating network;

[0022] The sample feature pairs are encoded by the generative network to obtain target feature data; and the target feature data are loss calculated by the discriminative network to obtain a feature loss value;

[0023] The loss function of the neural network model is optimized according to the feature loss value to update the neural network model and obtain the image description generation model.

[0024] In some embodiments, the sample feature pair includes a first feature pair and a second feature pair, the target feature data includes a first text feature and a first image feature, and the steps of encoding the sample feature pair through the generative network to obtain the target feature data; and performing loss calculation on the target feature data through the discriminative network to obtain a feature loss value include:

[0025] Performing encoding processing on the sample image in the first feature pair to obtain a first image feature, and performing encoding processing on the pseudo description text in the second feature pair to obtain a first text feature;

[0026] The first image feature is described and generated by the generation network to obtain a first description text; and the first text feature is described and generated by the generation network to obtain a second description text;

[0027] Performing text parsing on the first description text through the discriminant network to obtain a second image feature, and performing similarity calculation on the second image feature and the first image feature to obtain a first loss value;

[0028] Calculating the similarity between the second description text and the pseudo description text through the discriminant network to obtain a second loss value;

[0029] The first loss value and the second loss value are weightedly calculated according to a preset weight parameter to obtain the feature loss value.

[0030] To achieve the above objective, a second aspect of an embodiment of the present application provides a method for generating an image description, the method comprising:

[0031] Obtaining a target image to be processed;

[0032] The target image is input into an image description generation model for image description generation processing to obtain a target image description text, wherein the image description generation model is trained according to the training method as described in any one of the first aspects.

[0033] To achieve the above-mentioned purpose, a third aspect of an embodiment of the present application proposes a model training device, the device comprising:

[0034] A sample image acquisition module, used to acquire a sample image;

[0035] A detection module, used to perform target detection on the sample image to obtain a sample image category label;

[0036] An encoding module, used to encode the sample image category label through a preset reference state factor to obtain a target category label sequence;

[0037] A decoding module, used to traverse a preset image description corpus, decode the target category label sequence, and obtain a pseudo description text;

[0038] A sample feature pair construction module, used to construct a sample feature pair according to the sample image and the pseudo description text;

[0039] The training module is used to train a preset neural network model according to the sample features to obtain an image description generation model, and the image description generation model is used to generate a target image description text.

[0040] To achieve the above-mentioned purpose, a fourth aspect of an embodiment of the present application provides an image description generating device, the device comprising:

[0041] A target image acquisition module, used to acquire a target image to be processed;

[0042] An image description generation module is used to input the target image into an image description generation model for image description generation processing to obtain a target image description text, wherein the image description generation model is trained according to the training method as described in any one of the first aspects.

[0043] To achieve the above-mentioned purpose, the fifth aspect of an embodiment of the present application proposes an electronic device, which includes a memory, a processor, a program stored on the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the method described in the first aspect or the method described in the second aspect is realized.

[0044] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium used for computer-readable storage, and the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method described in the first aspect or the method described in the second aspect above.

[0045] The model training method, image description generation method and apparatus, device and medium proposed in the present application obtain sample images; perform target detection on the sample images to obtain sample image category labels, which can effectively eliminate image content with low relevance in the sample images, so as to more accurately detect image categories with high relevance to the sample images. The sample image category labels are encoded by a preset reference state factor to obtain a target category label sequence; a preset image description corpus is traversed to decode the target category label sequence to obtain a pseudo description text, so that the sample images and the corresponding pseudo description texts can be paired to obtain multiple sample feature pairs. Finally, the preset neural network model is trained according to the sample feature pairs to obtain an image description generation model, which is used to generate target image description text. The embodiment of the present application can detect the image content of the sample image more specifically through target detection, encoding and decoding, and generate corresponding description text, so that the generated pseudo-initial description text is more in line with the image content. At the same time, multiple sample feature pairs can be constructed based on the sample image and its pseudo-description text, which can achieve spatial alignment of image features and text features, so that the neural network model can be more focused on learning the semantic content of the image during the training process, thereby improving the training effect of the model. The embodiment of the present application can improve the accuracy of the generated target image description text by inputting the target image to be processed into the image description generation model for image description generation processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a flow chart of the training method of the model provided in the embodiment of the present application;

[0047] Figure 2 yes Figure 1 Flow chart of step S102 in FIG.

[0048] Figure 3 yes Figure 1 Flow chart of step S104 in FIG.

[0049] Figure 4 yes Figure 1 Flow chart of step S106 in FIG.

[0050] Figure 5 yes Figure 4 Flow chart of step S402 in FIG.

[0051] Figure 6 is a flowchart of an image description generating method provided in an embodiment of the present application;

[0052] Figure 7 It is a structural schematic diagram of a training device for a model provided in an embodiment of the present application;

[0053] Figure 8 is a structural schematic diagram of an image description generating device provided in an embodiment of the present application;

[0054] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0058] First, some nouns involved in this application are analyzed:

[0059] Artificial Intelligence (AI): It is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science. AI attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0060] Natural Language Processing (NLP): NLP uses computers to process, understand, and apply human languages ​​(such as Chinese, English, etc.). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text understanding. Natural language processing is commonly used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis, and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0061] Information Extraction: A text processing technology that extracts specified types of entity, relationship, event and other factual information from natural language text and forms structured data output. Information extraction is a technology that extracts specific information from text data. Text data is composed of some specific units, such as sentences, paragraphs, and chapters. Text information is composed of some small specific units, such as characters, words, phrases, sentences, paragraphs, or a combination of these specific units. Extracting noun phrases, names, place names, etc. from text data are all text information extraction. Of course, the information extracted by text information extraction technology can be various types of information.

[0062] Object Detection: The task of object detection is to find all the objects of interest in the image and determine their categories and locations. It is one of the core problems in the field of computer vision. The core problems of object detection include four categories, namely (1) classification problem: that is, to which category the image in the picture (or a certain area) belongs. (2) Positioning problem: the object may appear at any position in the image. (3) Size problem: the object has various sizes. (4) Shape problem: the object may have various shapes. Object detection is divided into two major series: RCNN [Region-based Convolutional Neural Networks] series and YOLO series. The RCNN series is a representative algorithm based on region detection, and YOLO is a representative algorithm based on region extraction.

[0063] Softmax classifier: It is a generalization of the logistic regression classifier facing multiple classifications, and the output is the probability value belonging to different categories.

[0064] Encoder: It converts the input sequence into a vector of fixed length.

[0065] Decoder: It converts the previously generated fixed vector into an output sequence; the input sequence can be text, voice, image, or video; the output sequence can be text or image.

[0066] Generative Adversarial Networks (GAN): is a deep learning model and one of the most promising methods for unsupervised learning on complex distributions in recent years. The model produces fairly good output through the mutual game learning of (at least) two modules in the framework: Generative Model (GM) and Discriminative Model (DM).

[0067] Image caption generates a natural language description for an image and uses the generated description to help applications understand the semantics expressed in the image visual scene. For example, image captioning can convert image retrieval into text retrieval, which is used to classify images and improve image retrieval results. People usually only need to take a quick glance to describe the details of the image visual scene, while automatically adding descriptions to images is a comprehensive and arduous computer vision task that requires converting the complex information contained in the image into a natural language description. Compared with ordinary computer vision tasks, image captioning not only requires identifying objects from images, but also requires associating the identified objects with natural semantics and describing them in natural language. Therefore, image description requires people to extract deep features of images, associate them with semantic features, and convert them for generating descriptions.

[0068] At present, most image descriptions are often generated using supervised learning methods, that is, the basic features of the image are extracted through a supervised learning model, and corresponding description sentences are generated based on the basic features to obtain image description text. The training of supervised learning models often requires a large amount of training corpus, including training images and corresponding description sentences. However, it is difficult to obtain a large amount of high-quality training corpus, which affects the training effect of the model. Therefore, how to improve the training effect of the model has become a technical problem that needs to be solved urgently.

[0069] Based on this, the embodiments of the present application provide a model training method, an image description generation method and apparatus, a device, and a medium, aiming to improve the training effect of the model.

[0070] The model training method, image description generation method and device, equipment, and medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the model training method in the embodiments of the present application is described.

[0071] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, AI is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0072] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0073] The training method of the model provided in the embodiment of the present application relates to the field of artificial intelligence technology. The image description generation method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN [Content Delivery Network, content distribution network] and big data and artificial intelligence platforms; the software can be an application that implements the image description generation method, etc., but is not limited to the above forms.

[0074] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0075] Figure 1 is an optional flowchart of the training method of the model provided in the embodiment of the present application, Figure 1 The method may include but is not limited to steps S101 to S106.

[0076] Step S101, obtaining a sample image;

[0077] Step S102, performing target detection on the sample image to obtain a sample image category label;

[0078] Step S103, encoding the sample image category label by using a preset reference state factor to obtain a target category label sequence;

[0079] Step S104, traversing a preset image description corpus, decoding the target category label sequence to obtain a pseudo description text;

[0080] Step S105, constructing a sample feature pair according to the sample image and the pseudo description text;

[0081] Step S106, training a preset neural network model according to the sample features to obtain an image description generation model, and the image description generation model is used to generate a target image description text.

[0082] In the steps S101 to S106 shown in the embodiment of the present application, by performing target detection on the sample image, the sample image category label is obtained, which can effectively eliminate the image content with low correlation in the sample image, so that the image category with high correlation with the sample image can be detected more accurately. The sample image category label is encoded by the preset reference state factor to obtain the target category label sequence; the preset image description corpus is traversed, and the target category label sequence is decoded to obtain the pseudo description text, which can pair the sample image and the corresponding pseudo description text to obtain multiple sample feature pairs. Finally, the preset neural network model is trained according to the sample feature pairs to obtain the image description generation model, which is used to generate the target image description text. Through target detection, encoding and decoding, the image content of the sample image can be detected more specifically, and the corresponding description text can be generated, so that the generated pseudo description text is more in line with the image content. At the same time, multiple sample feature pairs can be constructed according to the sample image and its pseudo description text, which can realize the spatial alignment of image features and text features, so that the neural network model can be more focused on learning the semantic content of the image during the training process, and the training effect of the model is improved.

[0083] In step S101 of some embodiments, the sample image may be a three-dimensional image; in some embodiments, the three-dimensional image may be obtained through computed tomography (CT); in another embodiment, the three-dimensional image may be obtained through magnetic resonance imaging (MRI).

[0084] In some medical application scenarios, the sample images may be medical images, and the type of objects included in the sample images is lesions, i.e., the part of the body where the disease occurs. Medical images refer to images of internal tissues, such as the stomach, abdomen, heart, knees, and brain, obtained in a non-invasive manner for medical treatment or medical research, such as electronic CT, MRI, ultrasound (US), X-ray images, electroencephalograms, and optical photographic lights generated by medical instruments.

[0085] In step S101 of some embodiments, the sample image may also be obtained from a preset image database, or obtained by intercepting a video clip. The sample image may be represented as I, where I∈R 3×W×H , where W is the width of the sample image, H is the height of the sample image, and 3 is the number of channels of the sample image.

[0086] See also Figure 2 In some embodiments, step S102 may include but is not limited to steps S201 to S205:

[0087] Step S201, inputting the sample image into a preset target detection model, wherein the target detection model includes a convolutional layer, an RPN layer, a RoI pooling layer and a fully connected layer;

[0088] Step S202, extracting features from the sample image through a convolutional layer to obtain a sample feature map;

[0089] Step S203, performing target detection on the sample feature map through the RPN layer and preset coordinate information to obtain an initial detection frame;

[0090] Step S204, performing feature fusion processing on the sample feature map and the initial detection frame through the preset scaling parameters of the RoI pooling layer to obtain a sample fusion feature map;

[0091] Step S205, calculating the classification probability of the sample fusion feature map through the prediction function of the fully connected layer and the preset image category label to obtain the classification probability value, and screening the preset image category label according to the classification probability value to obtain the sample image category label.

[0092] In step S201 of some embodiments, the sample image is input into a preset target detection model, which may be a Faster R-CNN model, including a convolutional layer, an RPN layer (Region Proposal Network), a RoI pooling layer, and a fully connected layer.

[0093] In step S202 of some embodiments, the sample image is convolved by a convolution layer to further capture the image features of the sample image and generate a sample feature map corresponding to the sample image. The sample feature map can be recorded as x fr .

[0094] In step S203 of some embodiments, in order to better cover the image spatial features and improve the accuracy of the generated image description, spatial features are introduced in the target detection process, that is, preset coordinate information, for example, four coordinate parameters are preset, where the coordinate parameters can be expressed as These four coordinate parameters can represent the location information of each preset image anchor frame in the sample feature map, where i is the i-th target to be detected, x ix ,y iy Respectively represent the horizontal and vertical coordinates of the center point of the preset image anchor box, w iw Represents the width of the preset image anchor box, h ih Represents the height of the preset image anchor box, and the center point of the preset image anchor box is used as the image anchor point. The image anchor point of the target feature map is classified by the classification function of the RPN layer to obtain the label image feature, where the classification function can be a softmax function, etc., without limitation. Then, the label image feature is predicted by bounding box regression to obtain the offset of the image anchor point, and the target is located according to the offset and the label image, so as to obtain the initial detection box corresponding to the target feature map.

[0095] In step S204 of some embodiments, the features corresponding to each initial detection frame are selected on the target feature map through the RoI Pooling layer, and the feature dimension is set to a fixed value. The four coordinate parameters are learned through linear regression so that the offset is constantly close to the real frame, thereby obtaining an accurate target detection frame, that is, an image target detection frame. Further, the sample feature map and the image target detection frame are subjected to feature fusion processing according to the preset scaling parameters to obtain a sample fusion feature map x fr -prop .

[0096] In step S205 of some embodiments, the prediction function is a softmax function, and the specific category of each feature point on the sample fusion feature map is calculated by the softmax function, that is, a probability distribution is created on each preset image category label to obtain a probability vector, which is a classification probability value, indicating the probability of each feature point belonging to each preset image category label; further, the preset image category labels are screened according to the classification probability values, and the preset image category labels with classification probability values ​​in the top ten are selected as sample image category labels, wherein the sample image category labels can be expressed as Sample image category labels can be used to generate pseudo description text.

[0097] In step S103 of some embodiments, since the pseudo description text is mainly generated based on the independent sample image category labels in the embodiments of the present application, the generated text sentences may lack language coherence. Therefore, the sample image category labels are encoded by the preset reference state factors to obtain the target category label sequence, that is, the finite state automaton A is used. i To represent the target category label sequence in, Σ represents the vocabulary of the pre-acquired image description corpus, S i is an automatic state collection, represents the initial state, δ i is the state transfer function, is the final state of the target category label sequence. Through this encoding process, a label sequence for image description generation can be obtained.

[0098] See also Figure 3 In some embodiments, step S104 may include but is not limited to steps S301 to S302:

[0099] Step S301, calculating the distribution probability of the target category label sequence using a preset prediction function to obtain a distribution probability value;

[0100] Step S302, traverse the image description corpus, decode the target category label sequence according to the distribution probability value, and obtain a pseudo description text.

[0101] The word segment in the collection table Σ. Through this prediction function, the distribution probability of the target category label sequence can be calculated to obtain the distribution probability value p.

[0102] In step S302 of some embodiments, during the word generation process at each moment, the current state of the image description corpus and the target category label sequence is traversed, the output vocabulary sequence is updated through the state transition function, and the b word segments with the highest distribution probability values ​​obtained from each decoding are added to the generated vocabulary sequence. In the above example, through multiple loop iterations, the pseudo description text Y corresponding to the sample image is output.

[0103] In step S105 of some embodiments, the sample image and the pseudo description text generated based on the sample image are used to construct various types of sample feature pairs. For example, the sample feature pairs may include image-text pairs or text-image pairs, etc.

[0104] See also Figure 4 In some embodiments, step S106 may include but is not limited to steps S401 to S402:

[0105] Step S401, inputting the sample feature pair into a neural network model, the neural network model includes a generator network and a discriminator network;

[0106] Step S402, encoding the sample feature pairs through the generation network to obtain target feature data; and performing loss calculation on the target feature data through the discriminant network to obtain a feature loss value;

[0107] Step S403, optimizing the loss function of the neural network model according to the feature loss value to update the neural network model and obtain an image description generation model.

[0108] In step S401 of some embodiments, the sample feature pair is input into a neural network model. The neural network model can be constructed based on a generative adversarial network, and the neural network includes a generative network and a discriminative network.

[0109] In step S402 of some embodiments, the sample feature pairs are encoded by a generative network to generate image description information corresponding to the sample image, and the generated image description information is used as the target feature data, wherein the generative network can be constructed based on the LSTM algorithm. The target feature data is subjected to loss calculation by a discriminant network to obtain a feature loss value, wherein the feature loss value is used to determine whether the generated image description information can be matched and distinguished with the sentences in the pre-acquired corpus.

[0110] Furthermore, the input sample feature pair x t It can be expressed as

[0111] x t =W e S t ,t∈{0,...,T-1};

[0112] The process of encoding sample feature pairs through the generative network and generating target feature data can be expressed as:

[0113]

[0114] S t ~p t ,t∈{1,...,T};

[0115]

[0116] Among them, ~ represents the sampling operation, T is the length of the generated sentence, and W e Learn parameters for word embedding, x t is the input of the generative network, s t is the one-hot encoding of the generated vocabulary of the generator network, To generate the hidden layer state of the network, p t is the probability of generating a word at time t. T They are the two identifiers for the beginning and end of the sentence, and the initial hidden state is 0. To determine the hidden layer state of the network, q t The probability that the generated description is judged as true by the discriminant network.

[0117] Furthermore, in order to improve the training effect of the model, a reward mechanism is introduced in the training process, that is, when the generated image description information is difficult to distinguish from the sentences in the pre-acquired corpus through the feature loss value, Perform gradient descent to update the parameters of the discriminant network. For adversarial rewards, the loss function can be expressed as shown in formula (1):

[0118]

[0119] In step S403 of some embodiments, back propagation is performed based on the feature loss value to update the loss parameter of the loss function to optimize the loss function, and then back propagation is performed based on the loss function to update the neural network model by optimizing the loss function to obtain the image description generation model.

[0120] See also Figure 5 In some embodiments, the sample feature pair includes a first feature pair and a second feature pair, the target feature data includes a first text feature and a first image feature, and step S402 may also include but is not limited to steps S501 to S505:

[0121] Step S501, encoding the sample image in the first feature pair to obtain a first image feature, and encoding the pseudo description text in the second feature pair to obtain a first text feature;

[0122] Step S502, describing and generating the first image feature through a generative network to obtain a first description text; and describing and generating the first text feature through a generative network to obtain a second description text;

[0123] Step S503, performing text parsing on the first description text through a discriminant network to obtain a second image feature, and performing similarity calculation on the second image feature and the first image feature to obtain a first loss value;

[0124] Step S504, calculating the similarity between the second description text and the pseudo description text through the discriminant network to obtain a second loss value;

[0125] Step S505, performing weighted calculation on the first loss value and the second loss value according to a preset weight parameter to obtain a feature loss value.

[0126] In step S501 of some embodiments, a bidirectional reconstruction method is used to encode the sample feature pair and calculate the loss. Specifically, the bidirectional reconstruction process can be divided into a first reconstruction process (i.e., an image reconstruction process) and a second reconstruction process (i.e., a sentence reconstruction process). In the first reconstruction process, the sample image in the first feature pair is encoded, and the sample image is encoded by an encoder to extract the image feature representation to obtain the first image feature; in the second reconstruction process, the pseudo-description text core in the first feature pair is encoded to extract the sentence feature representation to obtain the first text feature.

[0127] In step S502 of some embodiments, in a first reconstruction process, a first image feature is described and generated through a generative network to obtain a first description text; in a second reconstruction process, a first text feature is described and generated through a generative network to obtain a second description text.

[0128] In step S503 of some embodiments, in the first reconstruction process, the first description text is parsed by the discriminant network, the image feature is returned according to the sentence feature, the second image feature is obtained, the second image feature is compared with the input first image feature, the image similarity between the second image feature and the first image feature is calculated, and the first loss value is obtained.

[0129] In step S504 of some embodiments, in the second reconstruction process, the second description text is compared with the input pseudo description text through the discriminant network, and the text similarity between the second description text and the pseudo description text is calculated to obtain a second loss value.

[0130] In step S505 of some embodiments, the preset weight parameters can be set according to actual needs without limitation. The first loss value and the second loss value are weighted according to the preset weight parameters to obtain a feature loss value. The feature loss value can characterize the process of training the model by combining the adversarial reward, sentence reconstruction loss function and image reconstruction loss function in the embodiment of the present application.

[0131] Furthermore, the image reconstruction loss function can be expressed as shown in formula (2):

[0132]

[0133] in,

[0134] x -1 =FC(f i m); FC represents the fully connected layer;

[0135]

[0136] The image reconstruction reward is proportional to the negative reconstruction loss function, that is,

[0137] The sentence reconstruction loss function can be expressed as shown in formula (3):

[0138]

[0139] in, Sentences that are considered true The tth word of .

[0140] Through the above steps S501 to S505, bidirectional reconstruction is introduced in the training process, and the image and text features are aligned in a common space, so that the generated sentences can better express the image semantics, thereby achieving the purpose of improving the model effect.

[0141] The training method of the model of the embodiment of the present application, by acquiring a sample image; performing target detection on the sample image, obtaining a sample image category label, can effectively eliminate the image content with low correlation in the sample image, so that the image category with high correlation with the sample image can be detected more accurately. The sample image category label is encoded by a preset reference state factor to obtain a target category label sequence; the preset image description corpus is traversed, and the target category label sequence is decoded to obtain a pseudo description text, so that the sample image and the corresponding pseudo description text can be paired to obtain multiple sample feature pairs. Finally, the preset neural network model is trained according to the sample feature pair to obtain an image description generation model, and the image description generation model is used to generate the target image description text. The embodiment of the present application can detect the image content of the sample image more specifically through target detection, encoding and decoding, and generate the corresponding description text, so that the generated pseudo description text is more in line with the image content. At the same time, multiple sample feature pairs can be constructed according to the sample image and its pseudo description text, and a two-way reconstruction process is introduced, which can realize the spatial alignment of image features and text features, so that the neural network model is more focused on learning the image semantic content during the training process, and the training effect of the model is improved.

[0142] See also Figure 6 The present application also provides an image description generating method, the method comprising:

[0143] Step S601, obtaining a target image to be processed;

[0144] Step S602: input the target image into an image description generation model for image description generation processing to obtain a target image description text, wherein the image description generation model is trained according to the training method of the first aspect embodiment.

[0145] In step S601 of some embodiments, a target image to be processed is obtained; the target image may be obtained by a camera, a video camera, or a CT scan, or may be obtained by other methods, but is not limited thereto.

[0146] In step S602 of some embodiments, the target image is input into an image description generation model, and image recognition is performed on the target image through the image description generation model. Through target detection, encoding and decoding, etc., the image content of the target image can be detected more specifically, and a corresponding target image description text is generated, so that the generated target image description text is more in line with the image content, and spatial alignment of image features and text features can be achieved. When generating the target image description information, the image description generation model can pay attention to the image semantic content in the target image, and perform image description generation processing according to the image semantic content, so that the obtained target image description text can be more accurate.

[0147] The image description generation method of the embodiment of the present application can improve the accuracy of the generated target image description text by inputting the target image to be processed into the image description generation model for image description generation processing.

[0148] See also Figure 7 The present application also provides a model training device, which can implement the above-mentioned model training method, and the device includes:

[0149] The sample image acquisition module 701 is used to acquire a sample image;

[0150] The detection module 702 is used to perform target detection on the sample image and obtain a category label of the sample image;

[0151] The encoding module 703 is used to encode the sample image category label through a preset reference state factor to obtain a target category label sequence;

[0152] A decoding module 704 is used to traverse a preset image description corpus, decode the target category label sequence, and obtain a pseudo description text;

[0153] A sample feature pair construction module 705 is used to construct a sample feature pair according to the sample image and the pseudo description text;

[0154] The training module 706 is used to train a preset neural network model according to sample features to obtain an image description generation model, and the image description generation model is used to generate a target image description text.

[0155] In some embodiments, the detection module 702 includes:

[0156] A first input unit is used to input the sample image into a preset target detection model, wherein the target detection model includes a convolution layer, an RPN layer, a RoI pooling layer and a fully connected layer;

[0157] A feature extraction unit, used to extract features from the sample image through a convolution layer to obtain a sample feature map;

[0158] The target detection unit is used to perform target detection on the sample feature map through the RPN layer and preset coordinate information to obtain an initial detection frame;

[0159] A feature fusion unit is used to perform feature fusion processing on the sample feature map and the initial detection frame through the preset scaling parameters of the RoI pooling layer to obtain a sample fusion feature map;

[0160] The first calculation unit is used to calculate the classification probability of the sample fusion feature map through the prediction function of the fully connected layer and the preset image category label to obtain the classification probability value, and screen the preset image category label according to the classification probability value to obtain the sample image category label.

[0161] In some embodiments, the decoding module 704 includes:

[0162] A second calculation unit is used to calculate the distribution probability of the target category label sequence through a preset prediction function to obtain a distribution probability value;

[0163] The decoding unit is used to traverse the image description corpus and decode the target category label sequence according to the distribution probability value to obtain a pseudo description text.

[0164] In some embodiments, the training module 706 includes:

[0165] A second input unit is used to input the sample feature pair into the neural network model, and the neural network model includes a generating network and a discriminating network;

[0166] A processing unit is used to encode the sample feature pairs through a generating network to obtain target feature data; and to calculate the loss of the target feature data through a discriminant network to obtain a feature loss value;

[0167] The optimization unit is used to optimize the loss function of the neural network model according to the feature loss value to update the neural network model and obtain the image description generation model.

[0168] In some embodiments, the sample feature pair includes a first feature pair and a second feature pair, the target feature data includes a first text feature and a first image feature, and the processing unit includes:

[0169] An encoding unit, used for encoding the sample image in the first feature pair to obtain a first image feature, and encoding the pseudo description text in the second feature pair to obtain a first text feature;

[0170] A description generation unit, configured to generate a description of the first image feature through a generation network to obtain a first description text; and to generate a description of the first text feature through a generation network to obtain a second description text;

[0171] A third calculation unit is used to perform text parsing on the first description text through a discriminant network to obtain a second image feature, and perform similarity calculation on the second image feature and the first image feature to obtain a first loss value;

[0172] a fourth calculation unit, configured to calculate a similarity between the second description text and the pseudo description text through a discriminant network to obtain a second loss value;

[0173] The weighted calculation unit is used to perform weighted calculation on the first loss value and the second loss value according to a preset weight parameter to obtain a characteristic loss value.

[0174] The specific implementation of the training device for this model is basically the same as the specific implementation of the training method for the above-mentioned model, and will not be repeated here.

[0175] See also Figure 8 The present application also provides an image description generating device, which can implement the above-mentioned image description generating method, and the device includes:

[0176] The target image acquisition module 801 is used to acquire the target image to be processed;

[0177] The image description generation module 802 is used to input the target image into the image description generation model for image description generation processing to obtain the target image description text, wherein the image description generation model is trained according to the training method of the first aspect embodiment.

[0178] The specific implementation of the image description generating device is substantially the same as the specific implementation of the above-mentioned image description generating method, and will not be described in detail here.

[0179] The embodiment of the present application also provides an electronic device, the electronic device comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, wherein when the program is executed by the processor, the training method of the above-mentioned model or the image description generation method is realized. The electronic device may be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0180] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0181] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0182] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the model training method or image description generation method of the embodiment of the present application.

[0183] Input / output interface 903, used to implement information input and output;

[0184] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0185] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0186] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0187] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned model training method or image description generation method.

[0188] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0189] The model training method, model training device, image description generation method, image description generation device, electronic device and storage medium provided in the embodiments of the present application obtain sample images; perform target detection on the sample images to obtain sample image category labels, which can effectively eliminate image content with low relevance in the sample images, so as to more accurately detect image categories with high relevance to the sample images. The sample image category labels are encoded by a preset reference state factor to obtain a target category label sequence; a preset image description corpus is traversed to decode the target category label sequence to obtain a pseudo description text, so that the sample image and the corresponding pseudo description text can be paired to obtain multiple sample feature pairs. Finally, the preset neural network model is trained according to the sample feature pairs to obtain an image description generation model, which is used to generate target image description text. The embodiment of the present application can detect the image content of the sample image more specifically through target detection, encoding and decoding, and generate corresponding description text, so that the generated pseudo-initial description text is more in line with the image content. At the same time, multiple sample feature pairs can be constructed based on the sample image and its pseudo-description text, which can achieve spatial alignment of image features and text features, so that the neural network model can be more focused on learning the semantic content of the image during the training process, thereby improving the training effect of the model. The embodiment of the present application can improve the accuracy of the generated target image description text by inputting the target image to be processed into the image description generation model for image description generation processing.

[0190] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0191] It can be understood by those skilled in the art that Figure 1-5 , Figure 6The technical solutions shown in the figure do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figure, or a combination of certain steps, or different steps.

[0192] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0193] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0194] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0195] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0196] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0197] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0198] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0199] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0200] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A model training method, characterized in that: The method comprises: Get a sample image; Performing target detection on the sample image based on a preset target detection model to obtain a sample image category label, wherein the target detection model includes a convolutional layer, an RPN layer, a RoI pooling layer, and a fully connected layer; The sample image category label is encoded by a preset reference state factor to obtain a target category label sequence; a preset image description corpus is traversed to decode the target category label sequence to obtain a pseudo description text; Constructing a sample feature pair according to the sample image and the pseudo description text; training a preset neural network model according to the sample feature pair to obtain an image description generation model, wherein the image description generation model is used to generate a target image description text; The performing target detection on the sample image based on a preset target detection model to obtain a sample image category label includes: Extracting features of the sample image through the convolution layer to obtain a sample feature map; The target is detected on the sample feature map through the RPN layer and the preset coordinate information, the center point of each preset image anchor frame in the sample feature map is used as the image anchor point, the image anchor point is classified through the classification function of the RPN layer to obtain the label image feature, the frame regression prediction is performed on the label image feature to obtain the offset of the image anchor point, the target is located according to the offset and the label image feature, and the initial detection frame is obtained; wherein the coordinate information is used to indicate the height and width of each preset image anchor frame in the sample feature map, and the horizontal and vertical coordinates of the center point of the preset image anchor frame; Determine the features corresponding to the initial detection frame on the sample feature map through the RoIpooling layer, set the feature dimension to a fixed value, learn the coordinate parameters in the coordinate information through linear regression, obtain the image target detection frame, and perform feature fusion processing on the sample feature map and the image target detection frame according to the preset scaling parameters of the RoIpooling layer to obtain a sample fusion feature map; The classification probability of each feature point on the sample fusion feature map is calculated by using the prediction function of the fully connected layer and the preset image category label to obtain the classification probability value of each feature point, and the classification probability value of each preset image category label is counted according to the classification probability value of each feature point, and the preset image category labels with the classification probability values ​​in the top ten are selected as the sample image category labels; The traversing a preset image description corpus, decoding the target category label sequence, and obtaining a pseudo description text includes: The distribution probability of the target category label sequence is calculated through a preset prediction function to obtain a distribution probability value; In the word generation process at each moment, the current state of the image description corpus and the target category label sequence is traversed, the output vocabulary sequence is updated through the state transfer function, and several word segments with the highest distribution probability values ​​obtained in each decoding are added to the generated vocabulary sequence. Through multiple loop iterations, the pseudo description text corresponding to the sample image is output.

2. The training method according to claim 1, characterized in that: The step of training a preset neural network model according to the sample features to obtain an image description generation model comprises: Inputting the sample feature pair into the neural network model, wherein the neural network model includes a generating network and a discriminating network; The sample feature pairs are encoded by the generative network to obtain target feature data; and the target feature data are loss calculated by the discriminative network to obtain a feature loss value; The loss function of the neural network model is optimized according to the feature loss value to update the neural network model and obtain the image description generation model.

3. The training method according to claim 2, characterized in that: The sample feature pairs include a first feature pair and a second feature pair, the target feature data include a first text feature and a first image feature, and the sample feature pairs are encoded by the generative network to obtain the target feature data; The step of performing loss calculation on the target feature data through the discriminant network to obtain a feature loss value includes: Performing encoding processing on the sample image in the first feature pair to obtain a first image feature, and performing encoding processing on the pseudo description text in the second feature pair to obtain a first text feature; The first image feature is described and generated by the generation network to obtain a first description text; and the first text feature is described and generated by the generation network to obtain a second description text; Performing text parsing on the first description text through the discriminant network to obtain a second image feature, and performing similarity calculation on the second image feature and the first image feature to obtain a first loss value; Calculating the similarity between the second description text and the pseudo description text through the discriminant network to obtain a second loss value; The first loss value and the second loss value are weightedly calculated according to a preset weight parameter to obtain the feature loss value.

4. A method for generating an image description, characterized in that: For generating a target image description text, the method comprises: Obtaining a target image to be processed; The target image is input into an image description generation model for image description generation processing to obtain a target image description text, wherein the image description generation model is trained according to the training method according to any one of claims 1 to 3.

5. A model training device, characterized in that: The device comprises: A sample image acquisition module, used to acquire a sample image; A detection module, used to perform target detection on the sample image based on a preset target detection model to obtain a sample image category label, wherein the target detection model includes a convolutional layer, an RPN layer, a RoI pooling layer, and a fully connected layer; An encoding module, used to encode the sample image category label through a preset reference state factor to obtain a target category label sequence; A decoding module, used to traverse a preset image description corpus, decode the target category label sequence, and obtain a pseudo description text; A sample feature pair construction module, used to construct a sample feature pair according to the sample image and the pseudo description text; A training module, used for training a preset neural network model according to the sample features to obtain an image description generation model, wherein the image description generation model is used to generate a target image description text; The performing target detection on the sample image based on a preset target detection model to obtain a sample image category label includes: Extracting features of the sample image through the convolution layer to obtain a sample feature map; The target is detected on the sample feature map through the RPN layer and the preset coordinate information, the center point of each preset image anchor frame in the sample feature map is used as the image anchor point, the image anchor point is classified through the classification function of the RPN layer to obtain the label image feature, the frame regression prediction is performed on the label image feature to obtain the offset of the image anchor point, the target is located according to the offset and the label image feature, and the initial detection frame is obtained; wherein the coordinate information is used to indicate the height and width of each preset image anchor frame in the sample feature map, and the horizontal and vertical coordinates of the center point of the preset image anchor frame; Determine the features corresponding to the initial detection frame on the sample feature map through the RoIpooling layer, set the feature dimension to a fixed value, learn the coordinate parameters in the coordinate information through linear regression, obtain the image target detection frame, and perform feature fusion processing on the sample feature map and the image target detection frame according to the preset scaling parameters of the RoIpooling layer to obtain a sample fusion feature map; The classification probability of each feature point on the sample fusion feature map is calculated by using the prediction function of the fully connected layer and the preset image category label to obtain the classification probability value of each feature point, and the classification probability value of each preset image category label is counted according to the classification probability value of each feature point, and the preset image category labels with the classification probability values ​​in the top ten are selected as the sample image category labels; The traversing a preset image description corpus, decoding the target category label sequence, and obtaining a pseudo description text includes: The distribution probability of the target category label sequence is calculated through a preset prediction function to obtain a distribution probability value; In the word generation process at each moment, the current state of the image description corpus and the target category label sequence is traversed, the output vocabulary sequence is updated through the state transfer function, and several word segments with the highest distribution probability values ​​obtained in each decoding are added to the generated vocabulary sequence. Through multiple loop iterations, the pseudo description text corresponding to the sample image is output.

6. An image description generating device, characterized in that: The device comprises: A target image acquisition module, used to acquire a target image to be processed; An image description generation module is used to input the target image into an image description generation model for image description generation processing to obtain a target image description text, wherein the image description generation model is trained according to the training method according to any one of claims 1 to 3.

7. An electronic device, characterized in that: The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the training method of the model as described in any one of claims 1 to 3 or the steps of the image description generation method as described in claim 4 are realized.

8. A storage medium, the storage medium being a computer-readable storage medium, used for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the model training method as described in any one of claims 1 to 3 or the steps of the image description generation method as described in claim 4.

Images