Image description generation method and device, electronic equipment and storage medium

Abstract

The invention discloses an image description generation method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a to-be-processed image, calling a target image encoder model of a local end to carry out the image feature extraction of the to-be-processed image, obtaining the image feature information, and carrying out the image description generation when the network state of the local end is a first network state. The image feature information is uploaded to a cloud end connected with the local end, so that a first text decoder model in the cloud end decodes the image feature information, and first image description information sent by the cloud end is received. By adopting the embodiment of the invention, the image feature information extracted by the local end can be decoded by adopting the large model which is deployed in the cloud end and has relatively large model parameters, so that high-quality image description information can be obtained, and the quality of the image description information output by the local end is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an image description generation method, device, electronic device and storage medium. Background Art

[0002] Image caption is a very challenging task that combines computer vision and natural language processing. Simply put, image captioning is the process of using computers to process the analysis and description of each target background in an image by people using the visual system. Image captioning is relatively convenient for humans, but it is quite difficult for computers because computers not only need to find the target and background in the image, but also need to understand the relationship between them.

[0003] However, if you want to obtain high-quality image description information, you usually need a large image feature extraction network model and text decoder model. However, these large models are difficult to deploy on mobile terminals, resulting in poor quality of image description information generated by mobile terminals, which cannot meet user needs.

[0004] Therefore, how to improve the quality of image description information output by mobile terminals is an urgent problem to be solved. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide an image description generation method, device, electronic device and storage medium to solve the technical problem of poor quality of image description information output by a mobile terminal.

[0006] In a first aspect, an embodiment of the present invention provides an image description generation method, which is applied to a local terminal, and the method includes:

[0007] Get the image to be processed;

[0008] Calling the local target image encoder model to extract image features from the image to be processed to obtain image feature information;

[0009] When the network state of the local end is the first network state, uploading the image feature information to a cloud connected to the local end, so that a first text decoder model in the cloud can decode the image feature information;

[0010] Receive first image description information sent by the cloud.

[0011] In a second aspect, an embodiment of the present invention provides an image description generation method, which is applied to the cloud, and the method includes:

[0012] Receiving image feature information sent by a local terminal connected to the cloud;

[0013] Calling a first text decoder model deployed in the cloud to decode the image feature information to obtain first image description information;

[0014] The first image description information is sent to the local end.

[0015] In a third aspect, an embodiment of the present invention provides an image description generating device, which is applied to a local end, and the device includes:

[0016] A first acquisition module, used for acquiring an image to be processed;

[0017] An image feature extraction module is used to call the target image encoder model of the local end to extract image features of the image to be processed to obtain image feature information;

[0018] The uploading module is used to upload the image feature information to a cloud connected to the local end when the network state of the local end is a first network state, so that the first text decoder model in the cloud can decode the image feature information.

[0019] In a fourth aspect, an embodiment of the present invention provides an image description generating device, which is applied in the cloud, and the device includes:

[0020] A second receiving module, used to receive image feature information sent by a local terminal connected to the cloud;

[0021] A first decoding module, configured to call a first text decoder model deployed in the cloud to decode the image feature information to obtain first image description information;

[0022] A sending module is used to send the first image description information to the local end.

[0023] In a fifth aspect, an embodiment of the present invention provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the above-mentioned image description generation methods when executing the computer program.

[0024] In a sixth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any one of the above-mentioned image description generation methods are implemented.

[0025] The embodiments of the present invention provide an image description generation method, device, electronic device and storage medium. The method uploads image feature information extracted by a local end to a cloud connected to the local end, and decodes the image feature information through a large model with large model parameters deployed in the cloud, so as to obtain high-quality image description information, thereby effectively improving the quality of the image description information output by the local end. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a flow chart of an image description generating method provided by an embodiment of the present invention;

[0027] Figure 2 is another flowchart of the method for generating image description provided by an embodiment of the present invention;

[0028] Figure 3 is a schematic diagram of a training method provided by an embodiment of the present invention;

[0029] Figure 4 It is a flow chart of a method for obtaining a training data set provided by an embodiment of the present invention;

[0030] Figure 5 It is a flow chart of a method for performing knowledge distillation provided by an embodiment of the present invention;

[0031] Figure 6 is a third flow chart of the image description generating method provided by an embodiment of the present invention;

[0032] Figure 7 is a structural schematic diagram of an image description generating device provided by an embodiment of the present invention;

[0033] Figure 8 is another structural schematic diagram of the image description generating device provided by an embodiment of the present invention;

[0034] Fig. 9 is a third structural schematic diagram of the image description generating device provided by an embodiment of the present invention;

[0035] Fig.10 is a structural schematic diagram of an electronic device provided by an embodiment of the present invention;

[0036] Fig.11 This is another structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0039] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0040] In the related art, methods for generating image description information include the following three categories:

[0041] 1. Based on pre-trained image classifiers and long short-term memory (LSTM) networks, this solution uses a deep residual network (ResNet) or InceptionNet network to extract image features, and then converts the feature vector into LSTM input through a switching module, which then decodes and outputs image description information. The model performance of this solution is poor, the generalization ability is weak, and it has not been trained with large-scale data, so it is impossible to obtain high-quality image description information.

[0042] 2. Based on large-scale pre-trained data and Transformer models, the data comes from large-scale image and text datasets crawled from the web and small-scale fine-tuning datasets with manual annotations. This solution uses Transformer as a decoder to enhance network performance. Since the data comes from web crawling, there is a lot of noise, which is not friendly to small models on the end side. Small models are difficult to fit such complex data. Secondly, since the Transformer and image feature extraction network models are large, they are difficult to deploy on the end side.

[0043] 3. Based on the pre-trained neural network model (Contrastive Language-Image Pre-Training, CLIP) and pre-trained large language model for matching images and texts, this solution uses the image encoder of CLIP trained on 5 billion image-text pairs to extract image features, and uses the pre-trained large language model to output image description information. During training, the image encoder and large language model of CLIP are fixed, and only the converter from image feature vector to text input vector is trained. This solution has huge model parameters, and since part of the network needs to be frozen, the network cannot be compressed, making it difficult to deploy on the end side.

[0044] It can be seen that the image description information generation methods provided in the related art cannot be well applied on the mobile terminal side, resulting in the mobile terminal being unable to obtain high-quality image description information and unable to meet user needs.

[0045] In order to solve the technical problems existing in the related art, the embodiment of the present invention provides an image description generation method applied to a local terminal, see Figure 1 , Figure 1 is a flow chart of a method for generating an image description provided by an embodiment of the present invention. Figure 1 As shown, the image description generation method provided in this embodiment includes steps 101 to 104;

[0046] Step 101: Obtain an image to be processed.

[0047] In this embodiment, the image to be processed provided in this embodiment is an image that needs to be described and input by a user to the local terminal. The image description is intended to understand the image content and output a descriptive sentence of the image content. The description sentence should include relevant information such as the subject, action, location scene, etc. without omission.

[0048] Among them, in order to standardize the input of the model provided by this embodiment, that is, to standardize the input of the target image encoder model in the subsequent local end, the steps for obtaining the image to be processed provided by this embodiment can be: obtaining the original image input by the user to the local end; performing size optimization processing on the original image to adjust the size of the original image to the target size, and obtaining the image to be processed.

[0049] Optionally, the target size provided in this embodiment corresponds to the training image used in the training process of the target image encoder model. Therefore, the target size provided in this embodiment is the size of the training image used in the training process of the target image encoder model, for example, a size of 224*224.

[0050] Step 102, calling the local target image encoder model to extract image features from the image to be processed to obtain image feature information.

[0051] In this embodiment, the target image encoder model provided in this embodiment is a small model with smaller model parameters that is pre-deployed on the local end. The small model is mainly used to extract image features of the image to be processed to extract image feature information representing the image content in the image to be processed.

[0052] As an optional embodiment, this embodiment can use models such as ResNet or InceptionNet to extract image feature information of the image to be processed, or use the image encoder model in the CLIP model to extract image feature information of the image to be processed. As long as the model can extract the image feature information of the image to be processed, no specific limitation is made here.

[0053] Among them, this embodiment does not directly upload the image to be processed to the cloud and perform image feature extraction in the cloud, but uses a local target image encoder model to extract image features from the image to be processed. This not only avoids the risk of leakage of privacy information contained in the image, but also saves data traffic to a great extent.

[0054] Step 103, when the network state of the local end is the first network state, the image feature information is uploaded to the cloud connected to the local end, so that the first text decoder model in the cloud can decode the image feature information.

[0055] In this embodiment, after extracting the image feature information of the image to be processed, the network status of the local end can be detected to determine the network status of the local end. If the network status of the local end is detected to be the first network status, the extracted image feature information can be uploaded to the cloud, so that the image feature information extracted by the local end can be decoded using a large model with larger model parameters, thereby generating high-quality image description information.

[0056] Compared with decoding and processing image feature information through a small model with smaller model parameters deployed locally, this embodiment decodes and processes image feature information through a large model with larger model parameters deployed on the cloud, thereby obtaining higher quality image description information and effectively improving the quality of the image description information generated locally.

[0057] Among them, the large model deployed on the cloud with large model parameters, that is, the first text decoder model provided in this embodiment, can be a Transformer decoder model with large model parameters, or it can be a pre-trained large language model. As long as it is a model that can output high-quality image description information, no specific limitation is made here.

[0058] It should be noted that the first network state provided in this embodiment may be a state where there is a network. In this way, the image feature information can be uploaded to the cloud through the network, so that the image description generation process of the image to be processed can be completed by using the local end and the cloud to work together.

[0059] Step 104: Receive first image description information sent by the cloud.

[0060] In this embodiment, the first image description information received by this embodiment is obtained by decoding a large model with larger model parameters in the cloud. The large model can generate high-quality image description information. Therefore, by uploading the image feature information to the cloud and receiving the first image description information returned by the cloud, the local end provided by this embodiment can output high-quality image description information to the user, effectively improving the quality of the image description information that can be generated by the local end and improving the user experience.

[0061] It should be noted that the local end provided in this embodiment can be a mobile terminal such as a mobile phone device, a PC terminal such as a computer device, and a PAD terminal such as a tablet device. Optionally, this embodiment mainly uses a mobile terminal as the local end in this embodiment to improve the quality of the image description information generated by the mobile terminal.

[0062] In some embodiments, see Figure 2 , Figure 2 is another flow chart of the image description generation method provided by an embodiment of the present invention, such as Figure 2 As shown, the image description generation method provided in this embodiment includes steps 201 to 203;

[0063] Step 201, obtaining an image to be processed.

[0064] Step 202: Call the local target image encoder model to extract image features from the image to be processed to obtain image feature information.

[0065] Step 203, when the network state of the local end is the second network state, the image feature information is input into the second text decoder model of the local end for decoding processing to obtain the second image description information.

[0066] The network signal in the first network state provided by this embodiment is better than the network signal in the second network state. Specifically, the second network state provided by this embodiment can be a state of no network, or a state where the network signal is worse than the network signal in the first network state, that is, a state of poor network signal.

[0067] In this embodiment, in the case of no network or poor network signal, this embodiment can still use the second text decoder model deployed on the local end to decode the image feature information, thereby obtaining the second image description information corresponding to the image to be processed. Although the performance of the second text decoder model deployed on the local end is not as good as the performance of the first text decoder model deployed on the cloud, that is, the quality of the second image description information is lower than that of the first image description information, the key information of the image to be processed can be described, which effectively improves the user experience.

[0068] Therefore, in this embodiment, when the network status of the local end is the first network status, the image feature information of the image to be processed can be uploaded to the cloud, so that the image feature information can be decoded and processed using the large model deployed on the cloud, thereby obtaining high-quality first image description information. By converting the image to be processed into image feature information and then uploading it to the cloud, not only can the risk of leakage of privacy information contained in the image to be processed be avoided, but data traffic can also be saved to a great extent, and the model performance will not be lost at the same time. The large model on the cloud can obtain image description information with better quality than the local end; when the network status of the local end is the second network status, the image feature information of the image to be processed can be input into the small model deployed on the local end, so that the image feature information can be decoded and processed by the small model, thereby obtaining the second image description information whose quality is lower than the first image description information but can describe the key information in the image to be processed, so that users can use the image description generation service in extreme environments, effectively improving the user experience.

[0069] In some embodiments, in order to use the target image encoder model deployed on the local end to perform image feature extraction processing on the image to be processed, thereby obtaining image feature information of the image to be processed, before calling the target image encoder model on the local end to extract image features of the image to be processed and obtain image feature information, the image description generation method provided by this embodiment may also include: replacing the image encoder sub-model in the pre-trained first image-text similarity matching model with the initial image encoder model on the local end to obtain a replaced second image-text similarity matching model, the first image-text similarity matching model including an image encoder sub-model and a text encoder sub-model; fixing the model parameters of the text encoder sub-model in the second image-text similarity matching model, and training the second image-text similarity matching model until convergence to obtain a trained target image-text similarity matching model; using the image encoder sub-model in the target image-text similarity matching model as the target image encoder model on the local end.

[0070] The pre-trained first image similarity matching model provided in this embodiment can be a large image similarity matching model, that is, an image similarity matching model with larger model parameters. The method for training the second image similarity matching model provided in this embodiment is mainly carried out by contrastive learning. For details, please refer to Figure 3 , Figure 3 is a schematic diagram of a training method provided by an embodiment of the present invention, such as Figure 3 As shown, Figure 3 The text encoder in is a text encoder sub-model with fixed model parameters in the pre-trained first image similarity matching model provided in this embodiment, Figure 3 The image encoder in is the local initial image encoder model provided by this embodiment. By encoding the text encoder sub-model with fixed model parameters and the local initial image encoder model respectively, text encoding information: T1, T2, T3, ..., Tn, and image encoding information: I1, I2, I3, ..., In are obtained, and the corresponding text encoding information and image encoding information are compared and learned respectively to improve the feature extraction capability of the local initial image encoder model. When the model parameters of the local initial image encoder model converge, the trained target image encoder model of this embodiment is obtained.

[0071] In some embodiments, in order to be able to use the first image decoder model deployed in the cloud to decode the image feature information of the image to be processed to obtain the first image description information corresponding to the image to be processed, when the network state of the local end is the first network state, the image feature information is uploaded to the cloud connected to the local end for the first text decoder model in the cloud to decode the image feature information before the step, the image description generation method provided in this embodiment may also include: obtaining a training data set; using the training data set to train the image description generation model to be trained until convergence to obtain a trained image description generation model; deploying the text decoder sub-model in the trained image description generation model as the first text decoder model to the cloud connected to the local end.

[0072] The training data set provided in this embodiment is a high-definition image collected in different scenes, and the annotation data in the training data set is a Chinese image description. Figure 4 , Figure 4 is a flow chart of a method for obtaining a training data set provided by an embodiment of the present invention, such as Figure 4As shown in the figure, we can first use the open-source image description model from the academic community to generate image descriptions for the collected training data set to obtain the corresponding English image description sentences. Since the image text data crawled from the Internet has a lot of noise and poor correspondence between images and texts, and the open-source model from the academic community has strong performance and good results, the purpose of this step is to obtain data with better correspondence between images and texts, reduce the noise in the training data set, facilitate network training, and significantly improve model performance.

[0073] Even large models may have problems with poor correspondence between images and text. Figure 4 As shown, this embodiment can also use the image-text matching network to filter out samples with low image-text similarity. Since the large model in the academic community is trained on English, this embodiment can then translate English into Chinese, and there will be some errors in the translation, which will reduce the quality of the training data. Therefore, the translation quality evaluation network can also be used to filter out samples with poor translation to obtain high-quality Chinese image description sentences.

[0074] At this point, a Chinese training data set has been obtained, and the training data set provided by this embodiment has less noise, good correspondence between pictures and texts, smooth text, and high quality, which can facilitate the training of the model and significantly improve the model performance.

[0075] After the image description generation model to be trained is trained using the training data set provided by the above embodiment until convergence, the trained image description generation model can be obtained, and then the text decoder sub-model in the trained image description generation model is deployed as the first text decoder model to the cloud connected to the local end, so as to facilitate the subsequent decoding of the image feature information by the first text decoder model in the cloud, thereby obtaining high-quality, Chinese image description information.

[0076] In some embodiments, after completing the deployment of the first text decoder model in the cloud, the present embodiment may also use the first text decoder model to perform knowledge distillation on the initial text decoder model preset on the local side, so as to obtain a second text decoder model that has the same function as the first text decoder model and can be deployed on the local side. Specifically, when the network state on the local side is the second network state, the image feature information is input into the second text decoder model on the local side for decoding processing, and before the step of obtaining the second image description information, the image description generation method provided by the present embodiment may also include: using the first text decoder model in the cloud as a teacher network, performing knowledge distillation processing on the initial text decoder model preset on the local side, so as to migrate the model parameters of the first text decoder model to the initial text decoder model, and obtain the second text decoder model; deploying the second text decoder model on the local side.

[0077] Among them, see Figure 5 , Figure 5 is a flow chart of a method for performing knowledge distillation provided by an embodiment of the present invention, such as Figure 5 As shown, first, the training images in the training data set are input into the target image encoder model to obtain the image feature information corresponding to the training images, and then the model parameters of the first text decoder model are fixed, and the first text decoder model with fixed model parameters is used as the teacher network. The image feature information is used to perform knowledge distillation on the features of the initial text decoder model preset on the local side, including the parameters of the middle layer of the network, the output results and other features. At the same time, the initial text decoder model is trained with the output and data labels of the teacher network until convergence. In this way, a second text decoder model with the same function as the first text decoder model and which can be deployed on the local side can be obtained.

[0078] As an optional embodiment, in the process of using the first text decoder model as the teacher network to train the initial text decoder model provided in this embodiment, a preset loss function can be used to perform a constrained training process. Specifically, the preset loss function provided in this embodiment can be:

[0079]

[0080] in, feat t Represents the characteristics of the teacher network, feat s represents the characteristics of the initial text decoder model, CELoss represents the cross entropy loss, β corresponds to the loss coefficient of the teacher network as the label, γ corresponds to the loss coefficient corresponding to the original label of the data, and α corresponds to the coefficient corresponding to the feature distillation loss.

[0081] Through the knowledge distillation method provided in the above embodiment, a second text decoder model can be obtained which has the same function as the first text decoder model and can be deployed on the local end.

[0082] At this point, by adopting the above-mentioned embodiment, a large model capable of decoding processing, i.e., a first text decoder model, can be deployed on the cloud, and a target image encoder model capable of extracting image feature information and a small model capable of decoding processing, i.e., a second text decoder model, can also be deployed on the local end. In this way, the local end provided by this embodiment can complete the image description generation process of the image to be processed regardless of whether there is a network or not, and in the network state with a network, the image feature information of the image to be processed can be uploaded to the cloud, so that the image feature information can be decoded using the large model deployed on the cloud, thereby obtaining high-quality first image description information. When the network state of the local end is the second network state, the image feature information of the image to be processed can be input into the small model deployed on the local end, so that the image feature information can be decoded by the small model, thereby obtaining the second image description information that is lower in quality than the first image description information but can describe the key information in the image to be processed.

[0083] In summary, an embodiment of the present invention provides an image description generation method applied to a local end, the method comprising obtaining an image to be processed, calling a target image encoder model on the local end to extract image features from the image to be processed, obtaining image feature information, and uploading the image feature information to a cloud connected to the local end when the network state of the local end is a first network state, so that a first text decoder model in the cloud can decode the image feature information, and receiving the first image description information sent by the cloud. By adopting an embodiment of the present invention, a large model with large model parameters deployed in the cloud can be used to decode the image feature information extracted by the local end, so as to obtain high-quality image description information, which effectively improves the quality of the image description information generated by the local end.

[0084] In some embodiments, in order to solve the same technical problem, the present invention provides an image description generation method applied to the cloud, see Figure 6 , Figure 6 FIG. 4 is a third flow chart of the image description generation method provided by an embodiment of the present invention. Figure 6 As shown, the image description generation method provided in this embodiment includes steps 301 to 303;

[0085] Step 301, receiving image feature information sent by a local terminal connected to the cloud.

[0086] In this embodiment, the image feature information provided by this embodiment is the image feature information of the image to be processed uploaded by the local end. Since the cloud provided by this embodiment receives the image to be processed not uploaded by the local end, the risk of privacy information leakage in the image to be processed can be effectively avoided.

[0087] Step 302: Call a first text decoder model deployed in the cloud to decode the image feature information to obtain first image description information.

[0088] In this embodiment, a large model with large model parameters that can be used for decoding is pre-deployed in the cloud, that is, the first text decoder model provided in the above embodiment. The construction method of the first text decoder model can be found in the method provided in the above embodiment, which will not be repeated here.

[0089] In this way, by receiving the image feature information sent by the local end connected to the cloud, the image feature information of the image to be processed in the local end can be decoded by the first text decoder model in the cloud to obtain high-quality first image description information.

[0090] Step 303: Send the first image description information to the local end.

[0091] After obtaining the high-quality first image description information output by the first text decoder model, the first image description information can be sent to the local end to provide high-quality image description information for the local end user. Compared with decoding the image feature information through a small model with smaller model parameters deployed on the local end, this embodiment decodes the image feature information through a large model with larger model parameters deployed on the cloud, which can obtain higher-quality image description information, effectively improve the quality of the image description information generated by the local end, and thus improve the user experience of the local end user.

[0092] According to the method described in the above embodiment, this embodiment will be further described from the perspective of an image description generating device, which can be implemented as an independent entity or integrated into an electronic device, such as a terminal, which can include a mobile phone, a tablet computer, etc.

[0093] In order to solve the same technical problem, the embodiment of the present invention provides an image description generating device applied to a local end, see Figure 7 , Figure 7 is a schematic diagram of the structure of an image description generating device provided by an embodiment of the present invention, such as Figure 7 As shown, the image description generating device 400 provided by the embodiment of the present invention includes: a first acquisition module 401, an image feature extraction module 402, an upload module 403 and a first receiving module 404;

[0094] The first acquisition module 401 is used to acquire an image to be processed.

[0095] In this embodiment, the first acquisition module 401 is specifically used to: acquire the original image input by the user to the local end; perform size optimization processing on the original image to adjust the size of the original image to the target size, and obtain the image to be processed.

[0096] The image feature extraction module 402 is used to call the local target image encoder model to extract image features from the image to be processed to obtain image feature information.

[0097] The uploading module 403 is used to upload the image feature information to the cloud connected to the local end when the network state of the local end is the first network state, so that the first text decoder model in the cloud can decode the image feature information.

[0098] The first receiving module 404 is configured to receive first image description information sent by the cloud.

[0099] In some embodiments, see Figure 8 , Figure 8 is another structural schematic diagram of the image description generating device provided by an embodiment of the present invention, such as Figure 8 As shown, the image description generating device 400 provided by the embodiment of the present invention further includes: a first decoding module 405;

[0100] The first decoding module 405 is used to input the image feature information into the second text decoder model of the local end for decoding processing to obtain the second image description information when the network state of the local end is the second network state.

[0101] The network signal in the first network state is better than the network signal in the second network state.

[0102] In some embodiments, see Figure 8 , the image description generating device 400 provided by the embodiment of the present invention further includes: a first training module 406, a second training module 407 and a knowledge distillation module 408;

[0103] Among them, the first training module 406 is used to replace the image encoder sub-model in the pre-trained first image-text similarity matching model with the initial image encoder model at the local end to obtain the replaced second image-text similarity matching model, where the first image-text similarity matching model includes an image encoder sub-model and a text encoder sub-model; fix the model parameters of the text encoder sub-model in the second image-text similarity matching model, and train the second image-text similarity matching model until convergence to obtain the trained target image-text similarity matching model; use the image encoder sub-model in the target image-text similarity matching model as the target image encoder model at the local end.

[0104] The second training module 407 is used to obtain a training data set; use the training data set to train the image description generation model to be trained until convergence, so as to obtain a trained image description generation model, wherein the image description generation model includes an image encoder sub-model and a text decoder sub-model; and deploy the text decoder sub-model in the trained image description generation model as a first text decoder model to a cloud connected to the local end.

[0105] The knowledge distillation module 408 is used to use the first text decoder model in the cloud as a teacher network to perform knowledge distillation processing on the initial text decoder model preset on the local side, so as to migrate the model parameters of the first text decoder model to the initial text decoder model to obtain a second text decoder model; and deploy the second text decoder model on the local side.

[0106] In some other embodiments, in order to solve the same technical problem, the present invention also provides an image description generating device applied to the cloud, see Fig. 9 , Fig. 9 is a schematic diagram of a third structure of the image description generating device provided by an embodiment of the present invention, such as Fig. 9 As shown, the image description generating device 500 provided by the embodiment of the present invention includes: a second receiving module 501, a second decoding module 502 and a sending module 503;

[0107] The second receiving module 501 is used to receive image feature information sent by a local terminal connected to the cloud.

[0108] The second decoding module 502 is used to call the first text decoder model deployed in the cloud to decode the image feature information to obtain the first image description information.

[0109] The sending module 503 is used to send the first image description information to the local end.

[0110] During specific implementation, the above modules and / or units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above modules and / or units can refer to the previous method embodiments. The specific beneficial effects that can be achieved can also refer to the beneficial effects in the previous method embodiments, which will not be repeated here.

[0111] Also, see Fig.10 , Fig.10 1 is a schematic diagram of a structure of an electronic device provided by an embodiment of the present invention, and the electronic device may be a mobile terminal such as a smart phone, a tablet computer, or the like. Fig.10 As shown, the electronic device 600 includes a processor 601 and a memory 602. The processor 601 is electrically connected to the memory 602.

[0112] The processor 601 is the control center of the electronic device 600. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device 600 and processes data by running or loading applications stored in the memory 602 and calling data stored in the memory 602, thereby monitoring the electronic device 600 as a whole.

[0113] In this embodiment, the processor 601 in the electronic device 600 will load the instructions corresponding to the processes of one or more applications into the memory 602 according to the following steps, and the processor 601 will run the application stored in the memory 602, thereby implementing any step in the image description generation method provided in the above embodiment.

[0114] The electronic device 600 can implement the steps in any embodiment of the image description generation method provided in the embodiments of the present invention, and therefore can achieve the beneficial effects that can be achieved by any image description generation method provided in the embodiments of the present invention. For details, please refer to the previous embodiments and will not be repeated here.

[0115] See also Fig.11 , Fig.11 is another structural schematic diagram of an electronic device provided by an embodiment of the present invention, such as Fig.11 As shown, Fig.11 The specific structural block diagram of the electronic device provided by the embodiment of the present invention is shown, and the electronic device can be used to implement the image description generation method provided in the above embodiment. The electronic device 700 can be a mobile terminal such as a smart phone or a laptop computer.

[0116] The RF circuit 710 is used to receive and send electromagnetic waves, realize the mutual conversion between electromagnetic waves and electrical signals, and thus communicate with a communication network or other devices. The RF circuit 710 may include various existing circuit elements for performing these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, user identity module (SIM) cards, memories, etc. The RF circuit 710 can communicate with various networks such as the Internet, corporate intranets, wireless networks, or communicate with other devices through wireless networks. The above-mentioned wireless networks may include cellular telephone networks, wireless local area networks, or metropolitan area networks. The above-mentioned wireless networks can use various communication standards, protocols and technologies, including but not limited to Global System for Mobile Communication (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Wireless Fidelity (Wi-Fi) (such as Institute of Electrical and Electronics Engineers standards IEEE 802.11a, IEEE 802.11b, IEEE802.11g and / or IEEE802.11n), Voice over Internet Protocol (VoIP), Worldwide Interoperability for Microwave Access (Wi-Max), other protocols for email, instant messaging and short messages, and any other suitable communication protocols, even those that have not yet been developed.

[0117] The memory 720 may be used to store software programs and modules, such as program instructions / modules corresponding to the image description generation method in the above-mentioned embodiment. The processor 780 executes various functional applications and generates image descriptions by running the software programs and modules stored in the memory 720.

[0118] The memory 720 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 720 may further include a memory remotely arranged relative to the processor 780, and these remote memories may be connected to the electronic device 700 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0119] The input unit 730 can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control. Specifically, the input unit 730 may include a touch-sensitive surface 731 and other input devices 732. The touch-sensitive surface 731, also known as a touch display screen or touch pad, can collect user touch operations on or near it (such as operations performed by users using fingers, styluses, or any other suitable objects or accessories on or near the touch-sensitive surface 731), and drive corresponding connection devices according to a pre-set program. Optionally, the touch-sensitive surface 731 may include a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into touch point coordinates, and then sends it to the processor 780, and can receive and execute commands sent by the processor 780. In addition, the touch-sensitive surface 731 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch-sensitive surface 731, the input unit 730 may further include other input devices 732. Specifically, the other input devices 732 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

[0120] The display unit 740 can be used to display information input by the user or information provided to the user and various graphical user interfaces of the electronic device 700, which can be composed of graphics, text, icons, videos and any combination thereof. The display unit 740 may include a display panel 741, and optionally, the display panel 741 may be configured in the form of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc. Further, the touch-sensitive surface 731 may cover the display panel 741, and when the touch-sensitive surface 731 detects a touch operation on or near it, it is transmitted to the processor 780 to determine the type of the touch event, and then the processor 780 provides corresponding visual output on the display panel 741 according to the type of the touch event. Although in the figure, the touch-sensitive surface 731 and the display panel 741 are implemented as two independent components to implement input and output functions, in some embodiments, the touch-sensitive surface 731 and the display panel 741 can be integrated to implement input and output functions.

[0121] The electronic device 700 may also include at least one sensor 750, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 741 according to the brightness of the ambient light, and the proximity sensor may generate an interrupt when the flip cover is closed or closed. As a type of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary, which can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc. that can also be configured in the electronic device 700, they will not be repeated here.

[0122] The audio circuit 760, the speaker 761, and the microphone 762 can provide an audio interface between the user and the electronic device 700. The audio circuit 760 can transmit the electrical signal converted from the received audio data to the speaker 761, which is converted into a sound signal for output; on the other hand, the microphone 762 converts the collected sound signal into an electrical signal, which is received by the audio circuit 760 and converted into audio data, and then the audio data is output to the processor 780 for processing, and then sent to another terminal through the RF circuit 710, or the audio data is output to the memory 720 for further processing. The audio circuit 760 may also include an earplug jack to provide communication between an external headset and the electronic device 700.

[0123] The electronic device 700 can help the user receive requests, send information, etc. through the transmission module 770 (such as a Wi-Fi module), which provides the user with wireless broadband Internet access. Although the transmission module 770 is shown in the figure, it is understandable that it is not a necessary component of the electronic device 700 and can be omitted as needed without changing the essence of the invention.

[0124] The processor 780 is the control center of the electronic device 700. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 720, and calling data stored in the memory 720, it executes various functions of the electronic device 700 and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 780 may include one or more processing cores; in some embodiments, the processor 780 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communications. It can be understood that the above-mentioned modem processor may not be integrated into the processor 780.

[0125] The electronic device 700 also includes a power supply 790 (such as a battery) for supplying power to various components. In some embodiments, the power supply can be logically connected to the processor 780 through a power management system, so that the power management system can manage charging, discharging, and power consumption management. The power supply 790 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0126] Although not shown, the electronic device 700 also includes a camera (such as a front camera, a rear camera), a Bluetooth module, etc., which will not be described in detail here. Specifically in this embodiment, the display unit of the electronic device is a touch screen display, and the mobile terminal also includes a memory, and one or more programs, wherein one or more programs are stored in the memory, and are configured to be executed by one or more processors to implement any step in the image description generation method provided in the above embodiment.

[0127] In specific implementation, the above modules can be implemented as independent entities, or can be arbitrarily combined and implemented as the same or several entities. The specific implementation of the above modules can be found in the previous method embodiments, which will not be repeated here.

[0128] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by controlling related hardware through instructions, and the instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. To this end, an embodiment of the present invention provides a storage medium, which stores multiple instructions, and the instructions can be executed by the processor to implement any step in the image description generation method provided in the above embodiments.

[0129] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0130] Since the instructions stored in the storage medium can execute the steps in any embodiment of the image description generation method provided in the embodiments of the present invention, the beneficial effects that can be achieved by any image description generation method provided in the embodiments of the present invention can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0131] The above is a detailed introduction to an image description generation method, device, electronic device and storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and its core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application. Moreover, for those of ordinary skill in the art, without departing from the principles of the present invention, several improvements and modifications can be made, and these improvements and modifications are also regarded as the scope of protection of the present invention.

Claims

1. A method for generating an image description, characterized in that: Applied to the local end, the method includes: Get the image to be processed; Calling the local target image encoder model to extract image features from the image to be processed to obtain image feature information; When the network state of the local end is the first network state, uploading the image feature information to a cloud connected to the local end, so that a first text decoder model in the cloud can decode the image feature information; Receive first image description information sent by the cloud.

2. The method according to claim 1, characterized in that The method further comprises: When the network state of the local end is the second network state, inputting the image feature information into the second text decoder model of the local end for decoding processing to obtain second image description information; The network signal in the first network state is better than the network signal in the second network state.

3. The method according to claim 1, characterized in that Before the step of calling the local target image encoder model to extract image features from the image to be processed to obtain image feature information, the method further includes: Replacing the image encoder sub-model in the pre-trained first image-text similarity matching model with the initial image encoder model at the local end to obtain a replaced second image-text similarity matching model, wherein the first image-text similarity matching model includes an image encoder sub-model and a text encoder sub-model; Fixing the model parameters of the text encoder sub-model in the second image-text similarity matching model, and training the second image-text similarity matching model until convergence, to obtain a trained target image-text similarity matching model; The image encoder sub-model in the target image-text similarity matching model is used as the target image encoder model at the local end.

4. The method according to claim 1, characterized in that Before the step of uploading the image feature information to a cloud connected to the local end so that a first text decoder model in the cloud can decode the image feature information when the network state of the local end is the first network state, the method further includes: Get the training dataset; Using the training data set to train the image description generation model to be trained until convergence, to obtain a trained image description generation model; The text decoder sub-model in the trained image description generation model is deployed as a first text decoder model in a cloud connected to the local end.

5. The method according to claim 2, characterized in that Before the step of inputting the image feature information into the second text decoder model of the local end for decoding processing to obtain the second image description information when the network state of the local end is the second network state, the method further includes: Using the first text decoder model in the cloud as a teacher network, performing knowledge distillation processing on the initial text decoder model preset on the local end, so as to migrate the model parameters of the first text decoder model to the initial text decoder model, thereby obtaining a second text decoder model; The second text decoder model is deployed on the local end.

6. The method according to claim 1, characterized in that The step of obtaining the image to be processed comprises: Acquire the original image input by the user to the local end; The original image is subjected to size optimization processing to adjust the size of the original image to a target size, thereby obtaining an image to be processed.

7. A method for generating an image description, characterized in that: Applied to the cloud, the method includes: Receiving image feature information sent by a local terminal connected to the cloud; Calling a first text decoder model deployed in the cloud to decode the image feature information to obtain first image description information; The first image description information is sent to the local end.

8. An image description generating device, characterized in that: Applied to the local end, the device comprises: A first acquisition module, used for acquiring an image to be processed; An image feature extraction module is used to call the target image encoder model of the local end to extract image features of the image to be processed to obtain image feature information; an uploading module, configured to upload the image feature information to a cloud connected to the local end when the network state of the local end is a first network state, so that a first text decoder model in the cloud can decode the image feature information; The first receiving module is used to receive the first image description information sent by the cloud.

9. An image description generating device, characterized in that: Applied to the cloud, the device comprises: A second receiving module, used to receive image feature information sent by a local terminal connected to the cloud; A second decoding module is used to call a first text decoder model deployed in the cloud to decode the image feature information to obtain first image description information; A sending module is used to send the first image description information to the local end.

10. An electronic device, characterized in that: The electronic device comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor implements the steps in the method according to any one of claims 1 to 7 when executing the computer program.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the method according to any one of claims 1 to 7 are implemented.

Images