Image description method and related device
By using backbone networks to extract feature in image description, combined with joint learning models and reinforcement learning optimization technology, the problem of insufficient anthropomorphism in image description is solved, and higher accuracy and anthropomorphism effects are achieved.
Patent Information
- Application Number
- CN202411643529.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-18
- Publication Date
- 2025-03-11
AI Technical Summary
The prior art has failed to achieve anthropomorphism in image descriptions, focusing only on accuracy without considering the naturalness and human appearance of the description.
The backbone network is used to extract the image feature, and combined with the joint learning model, including the object detection network and the image description network, to improve the model performance through complementary loss information. At the same time, the model is optimized and trained by combining forward feedback and reinforcement learning, and anthropomorphic optimization samples are introduced to optimize the loss function.
It achieves higher accuracy and anthropomorphism of image description, improving the application and efficiency performance of the model in practical application scenarios.
Smart Images
Figure CN119600401A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing, and more specifically, to an image description method and related devices. Background Art
[0002] Image description technology is a cross-modal semantic understanding fusion technology that takes images as input and uses models and calculations to enable the computer to output natural language description text corresponding to the image content. Today, when general artificial intelligence big models are widely used, this technology, as the basic ability of big models to understand input images, can enhance the big models' understanding of image context and foreground and background, and is widely used in intelligent retrieval, intelligent generation, intelligent recommendation, and intelligent question and answer.
[0003] At present, when describing an image, only the accuracy of the image description is emphasized, and no attention is paid to the anthropomorphism of the image description. Therefore, how to achieve the anthropomorphism of the image description is a technical problem that those skilled in the art need to solve urgently. Summary of the invention
[0004] In view of this, the present application provides an image description method and related devices to solve the problem of realizing anthropomorphic image description.
[0005] In order to solve the above technical problems, this application adopts the following technical solutions:
[0006] An image description method, comprising:
[0007] Get the image to be processed;
[0008] The feature map of the image to be processed is determined by using a backbone network; the backbone network includes a plurality of basic modules, and an embedding layer is configured before the first basic module; the basic modules sequentially include a point-by-point convolution layer, a first convolution layer, and a second convolution layer, and the input of the point-by-point convolution layer is also fused with the output of the second convolution layer; the output of the basic module after passing through the third convolution layer is used as the input of the next basic module, and the output of the basic module is also fused with the output of the next basic module; a global average pooling layer, a fourth convolution layer, and a fully connected layer are sequentially arranged after the last basic module;
[0009] The feature map is input into the joint learning model to obtain the image description information of the image to be processed; the joint learning model includes a target detection network and an image description network; the loss information of the target detection network and the image description network are complementary; the joint learning model is a model obtained by optimizing the training based on the training samples and then using the optimized samples by combining forward feedback and reinforcement learning; the optimized samples are personified optimized samples.
[0010] Optionally, the training process of the joint learning model includes:
[0011] Acquire a training sample; the training sample includes a first sample image and a first image description sentence corresponding to the first sample image;
[0012] Converting the first image description sentence into a first tag sequence;
[0013] generating a first sample feature of the first sample image;
[0014] Using the first sample image, the first sample feature corresponding to the first sample image, and the first label sequence, a basic model is trained to obtain an initial model;
[0015] The initial model is optimized and trained by using the optimized samples in a combination of positive feedback and reinforcement learning to obtain a joint learning model.
[0016] Optionally, converting the first image description sentence into a first tag sequence includes:
[0017] Converting the first image description sentence into first target data in a preset format;
[0018] Filter out sub-data of a specified type in the first target data to obtain second target data;
[0019] The second target data is converted into a first tag sequence; the first tag sequence includes a triple consisting of an object, object location information, and object attributes.
[0020] Optionally, the initial model is optimized and trained by combining positive feedback and reinforcement learning using the optimized samples to obtain a joint learning model, including:
[0021] Acquire an optimized sample; the optimized sample includes a second sample image, a second image description statement corresponding to the second sample image, a second sample feature, and a second tag sequence;
[0022] Predicting description prediction information of the second sample image using the initial model;
[0023] Determining a score for the descriptive prediction information according to a positive feedback evaluation method;
[0024] Calculating a reinforcement learning loss function according to the score, the description prediction information of the second sample image, and the optimized sample;
[0025] The initial model is optimized and trained using the reinforcement learning loss function to obtain a joint learning model.
[0026] Optionally, obtain an optimized sample, including:
[0027] Acquire a second sample image and a second image description sentence corresponding to the second sample image;
[0028] Extracting sample features of the second sample image using a feature extraction model as second sample features;
[0029] Determine a second marker sequence corresponding to the second sample feature.
[0030] Optionally, the reinforcement learning loss function is:
[0031]
[0032] Where LOSS is the reinforcement learning loss function, N represents the average number of words included in the second image description sentence, M represents the size of the vocabulary used in model optimization, and y ij represents the hot encoding vector of the jth word in the i-th second image description sentence, z ij represents the predicted true value probability of the jth word in the i-th second image description sentence, and FB is the score of the second image description sentence.
[0033] Optionally, the loss function of the joint learning model is:
[0034] L=(1-μ)L O (θ,β)+μL C (θ,α)
[0035] Wherein, L is the loss function of the joint learning model; μ is the loss weight of the image description network, and L O (θ, β) is the loss function of the target detection network; L C (θ, α) is the loss function of the image description network;
[0036] L O (θ,β)=L0+L smooth
[0037] Among them, L0 is the paradigm loss function, L smooth is a smooth loss function; θ and β are the prior parameters of the backbone network and the target detection network respectively;
[0038]
[0039] Among them, θ and α are the prior parameters of the backbone network and the image description network; I is the input image, y t ′ is the true value sequence of the input image, t represents the tth input image, and T is the total number of input images.
[0040] An image description device, comprising:
[0041] An image acquisition module, used for acquiring an image to be processed;
[0042] A feature extraction module, used to determine the feature map of the image to be processed by using a backbone network; the backbone network includes multiple basic modules, and an embedding layer is configured before the first basic module; the basic modules sequentially include a point-by-point convolution layer, a first convolution layer, and a second convolution layer, and the input of the point-by-point convolution layer is also fused with the output of the second convolution layer; the output of the basic module after passing through the third convolution layer is used as the input of the next basic module, and the output of the basic module is also fused with the output of the next basic module; a global average pooling layer, a fourth convolution layer, and a fully connected layer are sequentially arranged after the last basic module;
[0043] A description information determination module is used to input the feature map into a joint learning model to obtain image description information of the image to be processed; the joint learning model includes a target detection network and an image description network; the loss information of the target detection network and the image description network are complementary; the joint learning model is a model obtained by optimizing the training based on training samples and using optimized samples in a combination of positive feedback and reinforcement learning; the optimized samples are personified optimized samples.
[0044] An electronic device comprises at least one processor and a memory connected to the processor, wherein:
[0045] The memory is used to store computer programs;
[0046] The processor is used to execute the computer program so that the electronic device can implement the above-mentioned image description method.
[0047] A computer storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the above-mentioned image description method.
[0048] The present application provides an image description method and related devices. In the present application, a backbone network is used to extract features from the processed image. In the backbone network, the input of the point-by-point convolution layer is also fused with the output of the second convolution layer, and the output of the basic module is also fused with the output of the next basic module, which can strengthen the fusion of feature information between channels, so that the accuracy of the extracted features is higher, thereby improving the accuracy and anthropomorphism of subsequent image descriptions. In addition, in the present invention, when describing an image, a joint learning model is used for image description. The joint learning model includes a target detection network and an image description network. The loss information of the target detection network and the image description network is complementary, and the purpose of improving the model performance without losing accuracy is achieved by complementary loss information. When generating the joint learning model, after training based on the training sample, the anthropomorphic optimization sample is used to optimize the training by combining positive feedback and reinforcement learning. The present invention is based on the reinforcement learning method of positive feedback, and by introducing the positive feedback index of the anthropomorphic optimization sample, the loss function is optimized to achieve a more anthropomorphic image description sentence output. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work.
[0050] Figure 1 A flowchart of an image description method provided by an embodiment of the present invention;
[0051] Figure 2 A backbone network structure diagram provided for an embodiment of the present invention;
[0052] Figure 3 A schematic diagram of an image description method provided by an embodiment of the present invention;
[0053] Figure 4 A model training flow chart provided for an embodiment of the present invention;
[0054] Figure 5 A model optimization flow chart provided for an embodiment of the present invention;
[0055] Figure 6 A schematic diagram of the structure of an image description device provided by an embodiment of the present invention;
[0056] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0057] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0058] In order to enable those skilled in the art to more clearly understand the present invention, the professional terms in the present invention are now explained.
[0059] Image caption: It is a comprehensive problem that integrates computer vision, natural language processing, and machine learning. It is similar to translating a picture into a descriptive text. This task is very easy for humans, but very challenging for machines. It not only requires the use of models to understand the content of the picture, but also requires the use of natural language to express the relationship between them.
[0060] Transformer: It is a sequence generation neural network based on the sequence-to-sequence seq2seq structure. The basic unit of Transformer is the attention neuron, which replaces RNN (Recurrent Neural Network) and CNN (Convolutional Neural Network). Compared with RNN, the advantage of attention is that the training process is parallel calculation, unlike RNN which can only be calculated serially, so it is more suitable for training in large-scale distributed clusters; the feature extraction ability of attention is stronger than that of RNN. Compared with CNN, the advantage of attention is that it can see the global data.
[0061] Reinforcement Learning (RL), also known as reinforcement learning, evaluation learning or enhanced learning, is one of the paradigms and methodologies of machine learning. It is used to describe and solve the problem of how an agent can maximize rewards or achieve specific goals through learning strategies during its interaction with the environment.
[0062] GPT-2, the second generation pre-trained generative language model.
[0063] Joint learning (multi-task learning), given m learning tasks, all or part of which are related but not exactly the same, the goal of multi-task learning is to help improve the performance of each task by using the knowledge contained in these m tasks.
[0064] Image description technology is a cross-modal semantic understanding fusion technology that takes images as input and uses models and calculations to enable the computer to output natural language description text corresponding to the image content. Today, when general artificial intelligence big models are widely used, this technology, as the basic ability of big models to understand input images, can enhance the big models' understanding of image context and foreground and background, and is widely used in intelligent retrieval, intelligent generation, intelligent recommendation, and intelligent question and answer.
[0065] In a wide range of application scenarios, the image description generated by the model is not only required to be more accurate and closer to the expression of the image content, but also the algorithm model is required to have higher efficiency and performance, and the algorithm output results are closer to human scenarios, so as to provide capability support for actual application scenarios. However, the existing image description algorithm models usually take one of the indicators of accuracy or diversity as the purpose of their research, and do not pay enough attention to efficiency performance and applicability. For example, in the intelligent question-answering model, the performance and applicability of image understanding and image description generation can directly affect the user experience. Single accuracy or diversity can no longer meet the user's immediate response needs for image description generation in actual application scenarios. Users hope that the generated image description can be closer to the human description of the image, and can be more anthropomorphic, so as to be directly used in content understanding, content production and other fields. The existing codec-based and transformer-based image description models use more complex feature extraction networks and more complete attention mechanisms to improve the accuracy and diversity of image descriptions, but ignore the actual anthropomorphic use of the model at the application level.
[0066] To this end, an embodiment of the present invention provides an image description method based on joint learning, which improves the efficiency and performance of the model and the degree of anthropomorphism of the generated description sentences, so as to enhance the applicability of the image description model in actual scenarios.
[0067] Specifically, in order to solve the problem of insufficient generation performance and inability to generate anthropomorphic image descriptions in existing image description generation methods. The embodiment of the present invention takes the transformer structure as the basic structure, designs an overall structure of an image description method model based on joint learning and positive feedback reinforcement learning, designs a backbone network suitable for the joint learning method, jointly learns the image description generation task and the image target detection task and complements the loss information to improve the efficiency and performance of image description generation, designs a positive feedback reinforcement learning method to optimize the loss function to obtain a more anthropomorphic image description, and makes the image description model more applicable and applicable to more diversified application scenarios.
[0068] Reference Figure 1 , an embodiment of the present invention provides an image description method, which may include:
[0069] S11, obtaining an image to be processed.
[0070] In the embodiment of the present invention, the image to be processed is an image that needs to be described, and can be an image of any field or any scene. In practical applications, in order to improve the processing efficiency of subsequent models, the image to be processed can be standardized to a fixed size.
[0071] S12. Determine a feature map of the image to be processed using a backbone network.
[0072] In the present invention, if the image is directly used as the model input, the amount of data is large. Therefore, in an embodiment of the present invention, feature extraction is performed on the image, and the extracted feature map is used as the input of the model.
[0073] When performing feature extraction, the feature extraction network structure adopts a backbone network, that is, the backbone network is used for feature extraction.
[0074] Backbone network structure reference Figure 2 , the backbone network includes multiple basic modules, such as Figure 2 As shown, the backbone network is mainly composed of four identical basic modules, each of which is for a different number of acquisition channels. In the embodiment of the present invention, the backbone network is for 4, 8, 16, and 32 channels.
[0075] The first basic module is preceded by an embedding layer, the embedding layer having a step length of 4.
[0076] The basic module includes a point-by-point convolution layer, a first convolution layer, and a second convolution layer in sequence. Figure 2 As shown, each basic module includes a point-by-point convolution layer with a stride of 3 and two convolution layers of different sizes, the size of the first convolution layer is 3×3, and the size of the second convolution layer is 1×1. In order to better extract features, the input of the point-by-point convolution layer is also fused with the output of the second convolution layer.
[0077] The output of the basic module after passing through the third convolutional layer is used as the input of the next basic module, such as Figure 2 As shown in Figure 1, the output of the basic module passes through the third convolutional layer with a stride of 2 for spatial downsampling and channel expansion, and is merged into the next basic module.
[0078] The output of the basic module is also fused with the output of the next basic module, such as Figure 2 As shown in FIG, the output of the previous basic module is directly jump-connected to the next basic module and is aggregated with the output of the next basic module to strengthen the fusion of feature information between channels.
[0079] After the last basic module, a global average pooling layer, a fourth convolutional layer and a fully connected layer are sequentially arranged. The global average pooling layer is used to pool the output, and the size of the fourth convolutional layer is 1×1. After the output of the fourth convolutional layer passes through the fully connected layer, a feature map is obtained.
[0080] The backbone network can generate feature maps of the following four dimensions for a single input image:
[0081]
[0082] Among them, b represents the batch size of the input image, S represents the image feature collection size after the backbone network structure module, H represents the image height, and W represents the image width. This feature map is used as the input for the object detection and image description tasks.
[0083] S13: Input the feature map into the joint learning model to obtain image description information of the image to be processed.
[0084] Specifically, in order to achieve better image description, a joint learning model is used in the embodiment of the present invention to perform image description. The joint learning model includes a target detection network and an image description network. Figure 3 As shown, the target detection network can be a center point detection network, and the image description network is a transformer. The transformer is a sequence generation neural network based on the seq2seq structure, and the basic unit of the transformer is the attention neuron. The transformer can be specifically a GPT-2 transformer.
[0085] The loss information of the object detection network and the image description network are complementary. Specifically, object detection and description generation combine complementary loss information.
[0086] First, for the image description network used to predict the image description generation output given an input, the loss function L of the standard language modeling image description network is used C (θ, α), which can be specifically the cross entropy loss function:
[0087]
[0088] Wherein, θ and α are the prior parameters of the backbone network and the image description network (such as GPT-2 transformer); I is the input image, y t ′ is the true value sequence of the input image, t represents the tth input image, and T is the total number of input images.
[0089] In addition, the loss function of the target detection network is designed. Considering the center point offset and target classification offset in the target detection task, the paradigm loss function L0 and the smooth loss function L smooth The combination of is used as the loss function of the target detection network, namely:
[0090] L O (θ,β)=L0+L smooth
[0091] Among them, L O (θ, β) is the loss function of the target detection network, L0 is the paradigm loss function, L smooth is a smooth loss function; θ and β are the prior parameters of the backbone network and the target detection network, respectively.
[0092] Finally, the overall loss function, that is, the loss function of the joint learning model, is constructed to guide the model training process:
[0093] L=(1-μ)L O (θ,β)+μL C (θ,α)
[0094] Among them, L is the loss function of the joint learning model, μ is the loss weight of the image description network, and experiments have shown that a value of μ between 0.15 and 0.3 has a better effect.
[0095] In this embodiment, the loss function of the joint learning model is composed of a target detection loss function and an image description loss function. Since a joint learning method is used and a backbone network is shared, the loss function will guide the backbone network to train the two tasks and achieve joint results. The separate evaluations of the two tasks will eventually act on the overall loss function, thereby affecting the structure and value of the parameters of the backbone network, so as to obtain performance without sacrificing accuracy while reducing the order of magnitude of parameters, thereby improving the reasoning performance.
[0096] In another implementation of the present application, the joint learning model is a model obtained by optimizing the training based on the training samples and then using the optimized samples to adopt a combination of positive feedback and reinforcement learning; the optimized samples are humanized optimized samples.
[0097] After training the joint learning model, Figure 3 As shown, the image to be processed is used as the image input, and the feature is extracted through the backbone network (specifically, it can be a 34-layer structure of a deep aggregation network) to obtain a feature map. The feature map is input into the joint learning model, and the image description sentence can be obtained and output based on the image description network. In addition, the target detection network, such as the center point detection network, can output the target detection result and use it according to the actual situation.
[0098] In this embodiment, a backbone network is used to extract features from the processed image. In the backbone network, the input of the point-by-point convolution layer is also fused with the output of the second convolution layer, and the output of the basic module is also fused with the output of the next basic module, which can strengthen the fusion of feature information between channels, so that the accuracy of the extracted features is higher, thereby improving the accuracy and anthropomorphism of subsequent image descriptions. In addition, in the present invention, when describing an image, a joint learning model is used for image description. The joint learning model includes a target detection network and an image description network. The loss information of the target detection network and the image description network is complementary, and the purpose of improving the model performance without losing accuracy is achieved by complementary loss information. When generating the joint learning model, after training based on the training sample, the anthropomorphic optimization sample is used to optimize the training by combining positive feedback and reinforcement learning. The present invention is based on the reinforcement learning method of positive feedback, and optimizes the loss function by introducing the positive feedback index of the anthropomorphic optimization sample to achieve a more anthropomorphic image description sentence output.
[0099] In addition, an overall framework of an image description model based on joint learning is constructed. The core components are the backbone network, the object detection network and the image description network. The image description network improves the performance of image description by combining a more complex object detection network and sharing the complementary information between the two tasks. The center detection-based object detection network is used to process the object detection task, and the transformer-based structure is used as a language decoder to process the image description task. The two networks share a feature extraction backbone network, and the two networks are run in parallel. A joint loss function is designed to guide the training process, so that the two can influence and learn from each other and improve the overall performance of the image description model.
[0100] The above embodiments mention that the joint learning model needs to be trained. The embodiments of the present invention introduce the training process of the joint learning model. Figure 2 , the training process of the joint learning model includes:
[0101] S21. Obtain training samples.
[0102] The training sample includes a first sample image and a first image description sentence corresponding to the first sample image.
[0103] Specifically, a dictionary may be compiled according to actual needs, and the dictionary is the training sample in the embodiment of the present invention.
[0104] Get the image description sentences from the dataset file and compile them into a structured dictionary. Each image corresponds to a description sentence type. The key of the dictionary is the image name and the value is the image description sentence. During the training process, the model is allowed to read images from the dictionary at will.
[0105] In order to distinguish from subsequent optimized samples, each image in the dictionary in this embodiment is referred to as a first sample image, and the image description sentence corresponding to the first sample image is referred to as a first image description sentence. That is, the dictionary includes the image name of the first sample image, the first sample image, and the first image description sentence of the first sample image.
[0106] S22: Convert the first image description sentence into a first tag sequence.
[0107] The first tag sequence is a triplet, which includes key information of the image. Specifically, the conversion process is:
[0108] First, the first image description sentence is converted into first target data in a preset format. The preset format is English that can be processed by the processor. In this embodiment, the image or description missing samples in the dictionary are cleaned up, and for the remaining data without missing data, all image description sentence texts are converted into lowercase English.
[0109] Secondly, the sub-data of a specified type in the first target data is filtered out to obtain the second target data. Specifically, the specified type is numbers, that is, the numbers in the data converted into lowercase English are filtered out to obtain the second target data.
[0110] Finally, the second target data is converted into a first tag sequence, wherein the first tag sequence includes a triple consisting of an object, object location information, and object attributes.
[0111] Specifically, a marking tool is used to obtain the second target data and convert it into a first marking sequence, where the marking sequence is a triple containing an object, object location information, and object attributes, such as the triple of <child, (123.1, 123.4), (smile, red shirt...)> for training purposes.
[0112] S23: Generate a first sample feature of the first sample image.
[0113] When extracting features from the first sample image, the above-mentioned backbone network may be used to extract features to obtain first sample features.
[0114] S24: Train a basic model using the first sample image, a first sample feature corresponding to the first sample image, and a first label sequence to obtain an initial model.
[0115] In this embodiment, the first sample image, the first sample feature corresponding to the first sample image, and the first label sequence are used as inputs of the basic model to train the basic model. During the training process, the loss function of the joint learning model is calculated once for each iteration. When the loss function of the joint learning model changes slightly or the number of iterations reaches a preset number, the training is stopped to obtain the initial model.
[0116] It should be noted that the basic model has the same structure as the joint learning model, including the target detection network and the image description network. During the training process, the parameters of the basic model are continuously optimized to obtain the initial model.
[0117] S25. Utilize the optimized samples and adopt a combination of positive feedback and reinforcement learning to optimize and train the initial model to obtain a joint learning model.
[0118] In this embodiment, after the above training, the joint learning model can have a higher image description ability. In order to further improve the anthropomorphic ability of image description, some anthropomorphic samples are used to optimize the initial model. The optimization process adopts a combination of positive feedback and reinforcement learning.
[0119] Specifically, refer to Figure 5 , step S25 may include:
[0120] S31. Obtain optimized samples.
[0121] The optimized sample includes a second sample image, a second image description sentence corresponding to the second sample image, a second sample feature and a second label sequence.
[0122] The second sample image, the second image description sentence corresponding to the second sample image, the second sample feature and the second tag sequence are similar to the first sample image, the first image description sentence corresponding to the first sample image, the first sample feature and the first tag sequence.
[0123] In practical applications, when using positive feedback reinforcement learning for optimization training, it is necessary to construct an image description vocabulary, which includes multiple words that conform to human usage habits, that is, these words are anthropomorphic words. Different words in the image description vocabulary are allowed to freely form various image descriptions.
[0124] For a batch of second sample images, the words in the image description vocabulary can be used to describe the images, and the second image description sentences corresponding to the second sample images can be obtained. Since the words in the image description vocabulary conform to human language habits, the second image description sentences corresponding to the second sample images are sentences that conform to human language habits, and the optimization operation of the model can be performed using the second image description sentences to enable the model to output anthropomorphic image description sentences.
[0125] After obtaining the second sample image, it is necessary to extract features from it to obtain sample features, just like the first sample image. In the embodiment of the present invention, in order to better optimize the model, a feature extraction model different from the above-mentioned backbone network is used to extract sample features of the second sample image, and the extracted features are used as second sample features. Among them, the feature extraction model is trained by transfer learning, and the specific model type can be configured according to actual conditions.
[0126] It should be noted that, in the embodiment of the present invention, in order to improve the accuracy of feature extraction, BLEU (Bilingual Evaluation Understudy) can be used to score the second image description sentence. If the score is low, it means that the image description accuracy is low. At this time, a small amount of features of the image can be extracted. If the score is high, it means that the image description accuracy is high. At this time, a large amount of features of the image can be extracted.
[0127] After obtaining the second sample feature, a second tag sequence corresponding to the second sample feature is determined. Specifically, the second sample feature is converted into an image attribute to obtain a second tag sequence, and the second sample image, the second image description sentence corresponding to the second sample image, the second sample feature and the second tag sequence are stored together in a vocabulary table for use in optimization training.
[0128] S32: Use the initial model to predict description prediction information of the second sample image.
[0129] In this embodiment, the second sample image is extracted through the above-mentioned backbone network to obtain a feature map, the feature map is input into the initial model, and the initial model outputs description prediction information of the second sample image.
[0130] S33. Determine the score of the description prediction information according to a positive feedback evaluation method.
[0131] In this embodiment, a positive feedback evaluation method is used to score the image description predicted by the model, and the scoring range is designed to be -100 to +100. The closer the image description is to that of a human, the higher the score.
[0132] S34. Calculate a reinforcement learning loss function according to the score, the description prediction information of the second sample image, and the optimized sample.
[0133] Among them, the reinforcement learning paradigm is introduced into the loss function, and positive feedback evaluation is encapsulated. The reinforcement learning loss function is:
[0134]
[0135] Where LOSS is the reinforcement learning loss function, N represents the average number of words included in the second image description sentence, M represents the size of the vocabulary used in model optimization, and y ij represents the hot encoding vector of the jth word in the i-th second image description sentence, z ij represents the predicted true value probability of the jth word in the i-th second image description sentence, and FB is the score of the second image description sentence.
[0136] Based on the above-mentioned score, the description prediction information of the second sample image, and the optimized sample, the parameter values of each parameter in the loss function are obtained, and then the reinforcement learning loss function is calculated using the parameter values. The preference of human language can be obtained, and the model can be further optimized through the preference.
[0137] S35. Utilize the reinforcement learning loss function to optimize the initial model and obtain a joint learning model.
[0138] In this embodiment, reinforcement learning is performed in an iterative manner using optimized samples. Prior parameters obtained from positive feedback are introduced into the model parameter iteration process. The reinforcement learning algorithm uses the image description output by the initial model prediction to compare with the image description in the optimized sample and the score of the image description, and calculates the custom loss of reinforcement learning, that is, the reinforcement learning loss function mentioned above, which is used to correct the deviation of the prediction model in the degree of anthropomorphism. Parameters are refined during iteration to ensure the optimization of model parameters in the joint learning process and improve the degree of anthropomorphism of the generated sentences.
[0139] In this embodiment, in response to the problem that the prior art pays little attention to the degree of anthropomorphism of description sentences, the overall structural model of image description proposed in the present invention designs a reinforcement learning method based on positive feedback. By preprocessing the anthropomorphic optimization samples (including feature extraction and generation of label sequences) and introducing positive feedback indicators, the loss function is optimized to achieve a more anthropomorphic image description sentence output, which can be used in application scenarios where anthropomorphic image descriptions are actually required.
[0140] In addition, the existing technologies mostly focus on content evaluation indicators for image description sentence generation, but do not pay much attention to improving efficiency and performance, or use model compression methods such as knowledge distillation, which requires sacrificing model generation quality to a certain extent while improving model performance. The present invention designs an overall structure of an image description model based on joint learning, and designs a backbone network for joint learning feature extraction, which uses a method of joint learning of complex tasks and image description tasks to complement loss information so as to improve model efficiency and performance without losing accuracy.
[0141] Based on the above-mentioned embodiment of the image description method, another embodiment of the present invention provides an image description device, referring to Figure 6 , which may include:
[0142] An image acquisition module 11 is used to acquire an image to be processed;
[0143] A feature extraction module 12 is used to determine the feature map of the image to be processed by using a backbone network; the backbone network includes a plurality of basic modules, and an embedding layer is configured before the first basic module; the basic modules sequentially include a point-by-point convolution layer, a first convolution layer, and a second convolution layer, and the input of the point-by-point convolution layer is also fused with the output of the second convolution layer; the output of the basic module after passing through the third convolution layer is used as the input of the next basic module, and the output of the basic module is also fused with the output of the next basic module; a global average pooling layer, a fourth convolution layer, and a fully connected layer are sequentially arranged after the last basic module;
[0144] The description information determination module 13 is used to input the feature map into the joint learning model to obtain the image description information of the image to be processed; the joint learning model includes a target detection network and an image description network; the loss information of the target detection network and the image description network are complementary; the joint learning model is a model obtained by optimizing the training based on the training samples and using the optimized samples by combining positive feedback and reinforcement learning; the optimized samples are humanized optimized samples.
[0145] In one implementation, a model training module is also included, and the model training module includes:
[0146] A sample acquisition submodule, used to acquire a training sample; the training sample includes a first sample image and a first image description sentence corresponding to the first sample image;
[0147] A conversion submodule, configured to convert the first image description sentence into a first tag sequence;
[0148] A feature generation submodule, used to generate a first sample feature of the first sample image;
[0149] A training submodule, configured to train a basic model using the first sample image, a first sample feature corresponding to the first sample image, and a first label sequence to obtain an initial model;
[0150] The optimization submodule is used to optimize the initial model by using the optimization samples and adopting a combination of positive feedback and reinforcement learning to obtain a joint learning model.
[0151] In one implementation, the conversion submodule includes:
[0152] A first conversion unit, configured to convert the first image description sentence into first target data in a preset format;
[0153] a filtering unit, configured to filter out sub-data of a specified type in the first target data to obtain second target data;
[0154] The second conversion unit is used to convert the second target data into a first tag sequence; the first tag sequence includes a triple consisting of an object, object location information, and object attributes.
[0155] In one implementation, the optimization submodule includes:
[0156] A sample acquisition unit, configured to acquire an optimized sample; the optimized sample comprises a second sample image, a second image description statement corresponding to the second sample image, a second sample feature, and a second tag sequence;
[0157] A prediction unit, configured to predict description prediction information of the second sample image using the initial model;
[0158] A scoring unit, used to determine a score of the description prediction information according to a positive feedback evaluation method;
[0159] a loss calculation unit, configured to calculate a reinforcement learning loss function according to the score, the description prediction information of the second sample image, and the optimized sample;
[0160] The optimization unit is used to optimize the initial model by using the reinforcement learning loss function to obtain a joint learning model.
[0161] In one implementation, the sample acquisition unit includes:
[0162] A data acquisition subunit, used for acquiring a second sample image and a second image description sentence corresponding to the second sample image;
[0163] A feature extraction subunit, configured to extract a sample feature of the second sample image using a feature extraction model as a second sample feature;
[0164] The sequence determination subunit is used to determine a second marker sequence corresponding to the second sample feature.
[0165] In one implementation, the reinforcement learning loss function is:
[0166]
[0167] Where LOSS is the reinforcement learning loss function, N represents the average number of words included in the second image description sentence, M represents the size of the vocabulary used in model optimization, and y ij represents the hot encoding vector of the jth word in the i-th second image description sentence, z ij represents the predicted true value probability of the jth word in the i-th second image description sentence, and FB is the score of the second image description sentence.
[0168] In one implementation, the loss function of the joint learning model is:
[0169] L=(1-μ)L O (θ,β)+μL C (θ,α)
[0170] Wherein, L is the loss function of the joint learning model; μ is the loss weight of the image description network, and L O (θ, β) is the loss function of the target detection network; L C (θ, α) is the loss function of the image description network;
[0171] L O (θ,β)=L0+L smooth
[0172] Among them, L0 is the paradigm loss function, L smooth is a smooth loss function; θ and β are the prior parameters of the backbone network and the target detection network respectively;
[0173]
[0174] Among them, θ and α are the prior parameters of the backbone network and the image description network; I is the input image, y t ′ is the true value sequence of the input image, t represents the tth input image, and T is the total number of input images.
[0175] In this embodiment, a backbone network is used to extract features from the processed image. In the backbone network, the input of the point-by-point convolution layer is also fused with the output of the second convolution layer, and the output of the basic module is also fused with the output of the next basic module, which can strengthen the fusion of feature information between channels, so that the accuracy of the extracted features is higher, thereby improving the accuracy and anthropomorphism of subsequent image descriptions. In addition, in the present invention, when describing an image, a joint learning model is used for image description. The joint learning model includes a target detection network and an image description network. The loss information of the target detection network and the image description network is complementary, and the purpose of improving the model performance without losing accuracy is achieved by complementary loss information. When generating the joint learning model, after training based on the training sample, the anthropomorphic optimization sample is used to optimize the training by combining positive feedback and reinforcement learning. The present invention is based on the reinforcement learning method of positive feedback, and optimizes the loss function by introducing the positive feedback index of the anthropomorphic optimization sample to achieve a more anthropomorphic image description sentence output.
[0176] It should be noted that, for the working process of each module, sub-module, unit and sub-unit in the embodiment of the present invention, please refer to the corresponding description in the above embodiment, which will not be repeated here.
[0177] An embodiment of the present application also provides an electronic device, including at least one processor and a memory connected to the processor, wherein:
[0178] The memory is used to store computer programs;
[0179] The processor is used to execute the computer program so that the electronic device can implement the above-mentioned image description method.
[0180] refer to Figure 7 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application. The electronic device in the embodiments of the present application may include but is not limited to fixed terminals such as mobile phones, notebook computers, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 7 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0181] like Figure 7As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0182] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.
[0183] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any image description method provided in the embodiment of the present application.
[0184] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any image description method provided in the embodiment of the present application.
[0185] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. An image description method, characterized in that: include: Get the image to be processed; The feature map of the image to be processed is determined by using a backbone network; the backbone network includes a plurality of basic modules, and an embedding layer is configured before the first basic module; the basic modules sequentially include a point-by-point convolution layer, a first convolution layer, and a second convolution layer, and the input of the point-by-point convolution layer is also fused with the output of the second convolution layer; the output of the basic module after passing through the third convolution layer is used as the input of the next basic module, and the output of the basic module is also fused with the output of the next basic module; a global average pooling layer, a fourth convolution layer, and a fully connected layer are sequentially arranged after the last basic module; The feature map is input into the joint learning model to obtain the image description information of the image to be processed; the joint learning model includes a target detection network and an image description network; the loss information of the target detection network and the image description network are complementary; the joint learning model is a model obtained by optimizing the training based on the training samples and then using the optimized samples by combining forward feedback and reinforcement learning; the optimized samples are personified optimized samples.
2. The image description method according to claim 1, characterized in that: The training process of the joint learning model includes: Acquire a training sample; the training sample includes a first sample image and a first image description sentence corresponding to the first sample image; Converting the first image description sentence into a first tag sequence; generating a first sample feature of the first sample image; Using the first sample image, the first sample feature corresponding to the first sample image, and the first label sequence, a basic model is trained to obtain an initial model; The initial model is optimized and trained by using the optimized samples in a combination of positive feedback and reinforcement learning to obtain a joint learning model.
3. The image description method according to claim 2, characterized in that: Converting the first image description sentence into a first tag sequence includes: Converting the first image description sentence into first target data in a preset format; Filter out sub-data of a specified type in the first target data to obtain second target data; The second target data is converted into a first tag sequence; the first tag sequence includes a triple consisting of an object, object location information, and object attributes.
4. The image description method according to claim 2, characterized in that: The initial model is optimized and trained by combining positive feedback and reinforcement learning using the optimized samples to obtain a joint learning model, including: Acquire an optimized sample; the optimized sample includes a second sample image, a second image description statement corresponding to the second sample image, a second sample feature, and a second tag sequence; Predicting description prediction information of the second sample image using the initial model; Determining a score for the descriptive prediction information according to a positive feedback evaluation method; Calculating a reinforcement learning loss function according to the score, the description prediction information of the second sample image, and the optimized sample; The initial model is optimized and trained using the reinforcement learning loss function to obtain a joint learning model.
5. The image description method according to claim 4, characterized in that: Get optimized samples, including: Acquire a second sample image and a second image description sentence corresponding to the second sample image; Extracting sample features of the second sample image using a feature extraction model as second sample features; Determine a second marker sequence corresponding to the second sample feature.
6. The image description method according to claim 4, characterized in that: The reinforcement learning loss function is: Where LOSS is the reinforcement learning loss function, N represents the average number of words included in the second image description sentence, M represents the size of the vocabulary used in model optimization, and y ij represents the hot encoding vector of the jth word in the i-th second image description sentence, z ij represents the predicted true value probability of the jth word in the i-th second image description sentence, and FB is the score of the second image description sentence.
7. The image description method according to claim 1, characterized in that: The loss function of the joint learning model is: L=(1-μ)L O (θ,β)+μL C (will) Wherein, L is the loss function of the joint learning model; μ is the loss weight of the image description network, and L O (θ, β) is the loss function of the target detection network; L C (θ, α) is the loss function of the image description network; L O (θ,β)=L0+L smooth Among them, L0 is the paradigm loss function, L smooth is a smooth loss function; θ and β are the prior parameters of the backbone network and the target detection network respectively; Among them, θ and α are the prior parameters of the backbone network and the image description network; I is the input image, y t ′ is the true value sequence of the input image, t represents the tth input image, and T is the total number of input images.
8. An image description device, characterized in that: include: An image acquisition module, used for acquiring an image to be processed; A feature extraction module, used to determine the feature map of the image to be processed by using a backbone network; the backbone network includes multiple basic modules, and an embedding layer is configured before the first basic module; the basic modules sequentially include a point-by-point convolution layer, a first convolution layer, and a second convolution layer, and the input of the point-by-point convolution layer is also fused with the output of the second convolution layer; the output of the basic module after passing through the third convolution layer is used as the input of the next basic module, and the output of the basic module is also fused with the output of the next basic module; a global average pooling layer, a fourth convolution layer, and a fully connected layer are sequentially arranged after the last basic module; A description information determination module is used to input the feature map into a joint learning model to obtain image description information of the image to be processed; the joint learning model includes a target detection network and an image description network; the loss information of the target detection network and the image description network are complementary; the joint learning model is a model obtained by optimizing the training based on training samples and using optimized samples in a combination of positive feedback and reinforcement learning; the optimized samples are personified optimized samples.
9. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the image description method as described in any one of claims 1 to 7.
10. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the image description method as described in any one of claims 1 to 7.