Image description generation method and device, electronic equipment and storage medium

Abstract

The invention discloses an image description generation method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining to-be-processed text information and a to-be-processed image, enabling the to-be-processed text information and the to-be-processed image to be related to the to-be-processed image, and carrying out the processing of the to-be-processed text information and the to-be-processed image, and inputting the text information to be processed into the trained image description generation model for image description generation processing to obtain a first image description related to the text information to be processed and a second image description related to the image to be processed, and determining a target image description of the image to be processed according to the first image description and the second image description. According to the embodiment of the invention, the to-be-processed text information related to the to-be-processed image and the to-be-processed image are utilized to generate the image description at the same time, and the to-be-processed text information and the image description corresponding to the to-be-processed image are combined, so that the quality of the generated image description can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an image description generation method, device, electronic device and storage medium. Background Art

[0002] Image caption is a very challenging task that combines computer vision and natural language processing. Simply put, image captioning is the process of using computers to process the analysis and description of each target background in an image by people using the visual system. Image captioning is relatively convenient for humans, but it is quite difficult for computers because computers not only need to find the target and background in the image, but also need to understand the relationship between them.

[0003] However, although the image description generation method in the related art can well complete the description of the image, it is unable to give a practical description of the actual scene in which the image itself is located, resulting in poor quality of the generated image description and failure to meet user needs.

[0004] Therefore, how to improve the quality of generated image descriptions is an urgent problem to be solved. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide a method, device, electronic device and storage medium for generating an image description, so as to solve the technical problem of poor quality of the image description generated in the related art.

[0006] In a first aspect, an embodiment of the present invention provides a method for generating an image description, comprising:

[0007] Acquire text information to be processed and an image to be processed, wherein the text information to be processed is text information related to the image to be processed;

[0008] Inputting the to-be-processed text information and the to-be-processed image into a trained image description generation model to perform image description generation processing, thereby obtaining a first image description related to the to-be-processed text information and a second image description related to the to-be-processed image;

[0009] A target image description of the image to be processed is determined according to the first image description and the second image description.

[0010] In a second aspect, an embodiment of the present invention provides an image description generating device, including:

[0011] A first acquisition module, used to acquire text information to be processed and an image to be processed, wherein the text information to be processed is text information related to the image to be processed;

[0012] An input module, used for inputting the text information to be processed and the image to be processed into the trained image description generation model to perform image description generation processing, so as to obtain a first image description related to the text information to be processed and a second image description related to the image to be processed;

[0013] A determination module is used to determine a target image description of the image to be processed according to the first image description and the second image description.

[0014] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the above-mentioned image description generation methods when executing the computer program.

[0015] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any one of the above-mentioned image description generation methods are implemented.

[0016] An embodiment of the present invention provides an image description generation method, device, electronic device and storage medium. The method utilizes additional text information related to an image to be processed and simultaneously performs image description generation processing on the image to be processed, thereby obtaining image descriptions corresponding to the text information to be processed and the image to be processed respectively. Then, by combining the image descriptions corresponding to the text information to be processed and the image to be processed respectively, the quality of the image description of the image to be processed can be effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic diagram of processing virtual application data in the related technology provided by an embodiment of the present invention;

[0018] Figure 2 Schematic diagram of the working principle of the image description generation model provided by an embodiment of the present invention;

[0019] Figure 3 It is a flow chart of a training method of an image description generation model provided by an embodiment of the present invention;

[0020] Figure 4 is a structural schematic diagram of an image description generating device provided by an embodiment of the present invention;

[0021] Figure 5 is another structural schematic diagram of the image description generating device provided by an embodiment of the present invention;

[0022] Figure 6is a structural schematic diagram of an electronic device provided by an embodiment of the present invention;

[0023] Figure 7 This is another structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0026] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0027] In the related art, although the image description generation method in the related art can well complete the description of the image, it is unable to give a practical description of the actual scene in which the image itself is located, resulting in poor quality of the generated image description and failure to meet user needs.

[0028] In order to solve the technical problems existing in the related art, the embodiment of the present invention provides an image description generation method, please also refer to Figure 1 and Figure 2 , Figure 1 is a flow chart of an image description generation method provided by an embodiment of the present invention. Figure 2 1 is a schematic diagram of the working principle of the image description generation model provided by an embodiment of the present invention. The image description generation method provided by this embodiment includes steps 101 to 103;

[0029] Step 101, obtaining text information to be processed and an image to be processed.

[0030] The text information to be processed is text information related to the image to be processed.

[0031] In this embodiment, the text information to be processed provided by this embodiment may be text information pre-annotated on the image to be processed, or may be attribute information of the image to be processed itself. Specifically, in the application scenario of a photo album, the text information to be processed provided by this embodiment may be time information of generating the image to be processed, location information when generating the image to be processed, or may be remark information added by the user to the image to be processed.

[0032] It should be noted that the text information to be processed provided in this embodiment may be one piece of information or multiple pieces of information, that is, the text information to be processed may be one or more of time information, address information and remarks information.

[0033] Step 102, input the text information to be processed and the image to be processed into the trained image description generation model for image description generation processing, and obtain a first image description related to the text information to be processed and a second image description related to the image to be processed.

[0034] In this embodiment, the trained image description generation model provided in this embodiment is mainly used to predict text information to obtain text information that can be used as image description of the input image simultaneously input into the model. At the same time, the model is also used to perform image description generation processing on the image to obtain the original image description of the input image.

[0035] As an optional embodiment, in order to obtain text information that can be used as an image description of the image to be processed and the original image description of the image to be processed, respectively, the image description generation model provided by this embodiment may include a first generation sub-model and a second generation sub-model. Specifically, the steps provided by this embodiment of inputting the text information to be processed and the image to be processed into the trained target generation model for image description generation processing to obtain a first image description related to the text information to be processed and a second image description related to the image to be processed may specifically be: calling the first generation sub-model to perform prediction processing on the text encoding vector of the text information to be processed to obtain a first image description of the image to be processed, the first image description being at least part of the text information in the text information to be processed; calling the second generation sub-model to perform decoding processing on the text encoding vector of the text information to be processed and the image encoding vector of the image to be processed to obtain a second image description of the image to be processed.

[0036] Among them, the first generation sub-model provided in this embodiment can be a text prediction model, and the second generation sub-model provided in this embodiment can be an autoencoder model. The second generation sub-model provided in this embodiment can generate a higher quality image description by using the text information to be processed during the decoding process by simultaneously decoding the text encoding vector of the text information to be processed and the image encoding vector of the image to be processed. In this way, the quality of the generated image description can be effectively improved by adopting the embodiment of the present invention.

[0037] In some embodiments, in order to process the text information to be processed and the image to be processed, the present embodiment needs to obtain the encoding vectors corresponding to the text information to be processed and the image to be processed, so that the image description generation model after training provided by the present embodiment can be used to generate image descriptions for the text information to be processed and the image to be processed. Therefore, the image description generation model provided by the present embodiment may also include a text encoder sub-model and an image encoder sub-model. In order to make the first generation sub-model and the second generation sub-model provided by the present embodiment more powerfully express the feature information of the text information to be processed and the image to be processed, so as to improve the quality of the subsequently generated image description, the present embodiment adopts an attention mechanism to perform attention calculation on the encoding vectors of the text information to be processed and the image to be processed. Specifically, before calling the first generation sub-model, performing predictive processing on the text encoding vector of the text information to be processed, and obtaining the first image description of the image to be processed, the image description generation method provided by the present embodiment may also include: calling the text encoder sub-model, performing text encoding processing on the text information to be processed, and obtaining a first encoding vector; calculating the attention matrix of the first encoding vector, and obtaining the text encoding vector of the text information to be processed.

[0038] Similarly, before calling the second generation sub-model to decode the text coding vector of the text information to be processed and the image coding vector of the image to be processed to obtain the second image description of the image to be processed, the image description generation method provided by this embodiment may also include: calling the image encoder sub-model to perform image coding on the image to be processed to obtain a second coding vector; calculating the attention matrix of the second coding vector to obtain the image coding vector of the image to be processed.

[0039] In this embodiment, since there is not necessarily a direct connection between the text information to be processed and the image content of the image to be processed provided by this embodiment, the calculation efficiency of the attention matrix can be effectively improved by separately calculating the attention matrix of the first coding vector and the second coding vector.

[0040] In this way, by processing the text encoding vector and the image encoding vector obtained through attention calculation through the first generation sub-model and the second generation sub-model, an image description containing more information, that is, higher quality, can be obtained.

[0041] In this embodiment, after obtaining the text coding vector of the text information to be processed and the image coding vector of the image to be processed, the step of calling the second generation sub-model provided in this embodiment to decode the text coding vector of the text information to be processed and the image coding vector of the image to be processed to obtain the second image description of the image to be processed can specifically be: linearly adding the text coding vector of the text information to be processed and the image coding vector of the image to be processed to obtain the target coding vector; calling the second generation sub-model to decode the target coding vector to obtain the second image description of the image to be processed.

[0042] Through the trained image description generation model provided in this embodiment, it is possible to obtain text information that can be used as an image description of the image to be processed, as well as the original image description of the image to be processed itself, thereby facilitating the subsequent combination of the text information and the original image description to obtain a higher quality image description.

[0043] Step 103: Determine a target image description of the image to be processed according to the first image description and the second image description.

[0044] In this embodiment, in order to further improve the quality of the finally generated image description, the step provided in this embodiment is to determine the target image description of the image to be processed based on the first image description and the second image description, which can specifically be: performing weighted summation on the first image description and the second image description to obtain the target image description of the image to be processed.

[0045] The sum of the first weight of the first image description and the second weight of the second image description provided in this embodiment is 1, and the first weight is less than the second weight. Optionally, the first weight provided in this embodiment may be 0.25, and the second weight may be 0.75. The values ​​of the first weight and the second weight are not limited to 0.25 and 0.75, as long as the first weight is less than the second weight, and no specific limitation is made here.

[0046] Since the second image description is obtained by performing image description generation processing based on the coding vector corresponding to the text information to be processed and the image to be processed, in order to further improve the accuracy of the generated image description, this embodiment also needs to perform a weighted summation of the first image description and the second image description to further improve the quality of the generated image description.

[0047] As an optional embodiment, in order to further improve the quality of generated image description, the step of determining the target image description of the image to be processed based on the first image description and the second image description provided in this embodiment can specifically be: using a replication network to process the first image description to obtain a final processed image description that is combined with the second image description, and performing weighted summation on the processed first image description and the second image description to obtain the target image description of the image to be processed.

[0048] Among them, the replication network can continue to determine the text information that can be used as the final target image description from the first image description, thereby further improving the quality of the generated image description. Specifically, the replication network provided in this embodiment mainly uses the following formula to determine the text information that can be used as the final target image description:

[0049] psi(y=t)=sigma(hM)s

[0050] Among them, psi represents the probability of generating text t at time y, sigma is the logarithm function, h represents the hidden state, that is, the output of the text encoder sub-model, the matrix M is the linear mapping, that is, the first generation sub-model, and s is the output of the first generation sub-model at the previous moment (y-1).

[0051] In some embodiments, to use the trained image description generation model normally, see Figure 3 , Figure 3 FIG. 1 is a flow chart of a training method for an image description generation model provided by an embodiment of the present invention. Figure 3 As shown, before the step of inputting the text information to be processed and the image to be processed into the trained image description generation model to perform image description generation processing to obtain the first image description related to the text information to be processed and the second image description related to the image to be processed, the image description generation method provided by the embodiment may further include steps 201 to 204;

[0052] Step 201, obtaining a training data set.

[0053] The training data set includes training images, training text information, and label information, and the label information includes first label information of the training text information and second label information of the image description of the training image.

[0054] Step 202: Train the first generation sub-model to be trained according to the training text information and the label information until convergence, and obtain the trained first generation sub-model.

[0055] In this embodiment, the training process of the first generation sub-model provided by this embodiment can be specifically as follows: inputting the text encoding vector of the training text data into the first generation sub-model for prediction processing to obtain a prediction result, and then optimizing the model parameters of the first generation sub-model according to the loss value between the prediction result and the first label information until convergence. Among them, the first label information provided by this embodiment is at least part of the text information in the training text information, and the part of the text information can be text information of the target type preset by the user, such as time information, location information, etc., so that the trained first generation sub-model can be capable of determining the text information that needs to be added to the final target image description from the training text information.

[0056] As an optional embodiment, in order to improve the quality of the target image description obtained after the final combination, this embodiment also needs to determine the predicted text information and the position to be added to the second image description when combined with the second image description. Specifically, when the first label information contains time information, the word embedding and position embedding methods can be used to encode the first label information and the training text information; when the first label information contains position information, the word embedding and type embedding methods can be used to determine the positional features of the time information in the first label information and the training text information, or the positional features of the position information, so that the trained first generation sub-model also has the ability to identify the specific position of the time information / position information in the text information.

[0057] It should be noted that this embodiment can not only predict the specific position of time information and location information in text information, but also predict the specific position of other types of information, such as weather information, environmental information, etc., in text information. It only needs to use the corresponding training data set to train the first generation sub-model, and no specific limitation is made here.

[0058] Step 203: Train the second generation sub-model to be trained according to the training images and label information until convergence, to obtain a trained second generation sub-model.

[0059] In this embodiment, the training process of the second generation sub-model provided in this embodiment can be specifically as follows: the text encoding vector of the training text data and the image encoding vector of the training image are input into the second generation sub-model for generation processing to obtain an output result, which is the image description of the training image, and then according to the loss value between the output result and the second label information, the model parameters of the second generation sub-model are optimized until convergence. Among them, the second label information provided in this embodiment is one of the label information of the training image, that is, one training image can correspond to multiple second label information, so that the second generation sub-model can have the ability to output a variety of different image descriptions.

[0060] Among them, the training data set provided by this embodiment includes a large number of training images, training text information, and label information. However, the embodiment of the present invention can effectively reduce the number of training data sets required for model training to convergence by separating the learning ability of text embedding position and the ability of image description generation in different models (first generation sub-model and second generation sub-model). Compared with the related art of training only one image description generation model, the embodiment of the present invention can greatly reduce the number of required training sets, thereby effectively reducing the time of model training, reducing the computing resources required for training the model, and thus improving the training efficiency of the model.

[0061] Step 204, combining the trained first generation sub-model and the trained second generation sub-model to obtain an image description generation model.

[0062] In this embodiment, after combining the trained first generation sub-model and the trained second generation sub-model, the text encoder sub-model, image encoder sub-model and replication network mentioned in the above embodiment can also be combined to obtain an image description generation model that can generate higher quality image descriptions.

[0063] In summary, an embodiment of the present invention provides an image description generation method, which includes obtaining text information to be processed and an image to be processed, wherein the text information to be processed is text information related to the image to be processed, inputting the text information to be processed and the image to be processed into a trained image description generation model for image description generation processing, obtaining a first image description related to the text information to be processed and a second image description related to the image to be processed, and determining a target image description of the image to be processed based on the first image description and the second image description. By adopting the embodiment of the present invention, the image description generation processing is performed simultaneously using the text information to be processed related to the image to be processed and the image to be processed, and combining the image descriptions corresponding to the text information to be processed and the image to be processed, the quality of the generated image description can be effectively improved.

[0064] According to the method described in the above embodiment, this embodiment will be further described from the perspective of an image description generating device, which can be implemented as an independent entity or integrated into an electronic device, such as a terminal, which can include a mobile phone, a tablet computer, etc.

[0065] See also Figure 4 , Figure 4 is a schematic diagram of the structure of an image description generating device provided by an embodiment of the present invention, such as Figure 4 As shown, the image description generating device 300 provided by the embodiment of the present invention includes: a first acquisition module 301, an input module 302 and a determination module 303;

[0066] The first acquisition module 301 is used to acquire text information to be processed and an image to be processed, where the text information to be processed is text information related to the image to be processed.

[0067] The input module 302 is used to input the text information to be processed and the image to be processed into the trained image description generation model for image description generation processing, so as to obtain a first image description related to the text information to be processed and a second image description related to the image to be processed.

[0068] The determination module 303 is used to determine a target image description of the image to be processed according to the first image description and the second image description.

[0069] In some embodiments, the image description generation model provided in this embodiment includes a first generation sub-model and a second generation sub-model, and the input module provided in this embodiment includes: a first calling unit and a second calling unit;

[0070] The first calling unit is used to call the first generating sub-model to perform prediction processing on the text encoding vector of the text information to be processed to obtain a first image description of the image to be processed, wherein the first image description is at least part of the text information in the text information to be processed;

[0071] The second calling unit is used to call the second generation sub-model to decode the text encoding vector of the text information to be processed and the image encoding vector of the image to be processed to obtain a second image description of the image to be processed.

[0072] In some embodiments, the image description generation model provided in this embodiment also includes a text encoder sub-model and an image encoder sub-model, see Figure 5 , Figure 5 is another structural schematic diagram of the image description generating device provided by an embodiment of the present invention, such as Figure 5As shown, the image description generating device 300 provided in this embodiment further includes: a first calling module 304, a first calculating module 305, a second calling module 306 and a second calculating module 307;

[0073] The first calling module 304 is used to call the text encoder sub-model to perform text encoding processing on the text information to be processed to obtain a first encoding vector.

[0074] The first calculation module 305 is used to calculate the attention matrix of the first encoding vector to obtain the text encoding vector of the text information to be processed.

[0075] The second calling module 306 is used to call the image encoder sub-model to perform image encoding processing on the image to be processed to obtain a second encoding vector.

[0076] The second calculation module 307 is used to calculate the attention matrix of the second coding vector to obtain the image coding vector of the image to be processed.

[0077] In some embodiments, the first calling unit is specifically used to: linearly add the text encoding vector of the text information to be processed and the image encoding vector of the image to be processed to obtain a target encoding vector; call the second generation sub-model to decode the target encoding vector to obtain a second image description of the image to be processed.

[0078] In some embodiments, the determination module 303 provided in this embodiment is specifically used to: perform weighted summation on the first image description and the second image description to obtain a target image description of the image to be processed.

[0079] The sum of the first weight of the first image description and the second weight of the second image description provided in this embodiment is 1, and the first weight is smaller than the second weight.

[0080] In some embodiments, please see Figure 5 The image description generating device 300 provided in this embodiment further includes: a second acquisition module 308, a first training module 309, a second training module 310 and a combining module 311;

[0081] The second acquisition module 308 is used to acquire a training data set.

[0082] The training data set includes training images, training text information, and label information, and the label information includes first label information of the training text information and second label information of the image description of the training image.

[0083] The first training module 309 is used to train the first generation sub-model to be trained according to the training text information and the label information until convergence, so as to obtain the trained first generation sub-model.

[0084] The second training module 310 is used to train the second generation sub-model to be trained according to the training images and label information until convergence, so as to obtain the trained second generation sub-model.

[0085] The combining module 311 is used to combine the trained first generation sub-model and the trained second generation sub-model to obtain an image description generation model.

[0086] During specific implementation, the above modules and / or units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above modules and / or units can refer to the previous method embodiments. The specific beneficial effects that can be achieved can also refer to the beneficial effects in the previous method embodiments, which will not be repeated here.

[0087] Also, see Figure 6 , Figure 6 1 is a schematic diagram of a structure of an electronic device provided by an embodiment of the present invention, and the electronic device may be a mobile terminal such as a smart phone, a tablet computer, or the like. Figure 6 As shown, the electronic device 400 includes a processor 401 and a memory 402. The processor 401 is electrically connected to the memory 402.

[0088] The processor 401 is the control center of the electronic device 400. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device 400 and processes data by running or loading applications stored in the memory 402 and calling data stored in the memory 402, thereby monitoring the electronic device 400 as a whole.

[0089] In this embodiment, the processor 401 in the electronic device 400 will load the instructions corresponding to the processes of one or more applications into the memory 402 according to the following steps, and the processor 401 will run the application stored in the memory 402, thereby implementing any step in the image description generation method provided in the above embodiment.

[0090] The electronic device 400 can implement the steps in any embodiment of the image description generation method provided in the embodiments of the present invention, and therefore can achieve the beneficial effects that can be achieved by any image description generation method provided in the embodiments of the present invention. For details, please refer to the previous embodiments and will not be repeated here.

[0091] See also Figure 7 , Figure 7 is another structural schematic diagram of an electronic device provided by an embodiment of the present invention, such as Figure 7 As shown, Figure 7The specific structural block diagram of the electronic device provided by the embodiment of the present invention is shown, and the electronic device can be used to implement the image description generation method provided in the above embodiment. The electronic device 500 can be a mobile terminal such as a smart phone or a laptop computer.

[0092] The RF circuit 510 is used to receive and send electromagnetic waves, realize the mutual conversion between electromagnetic waves and electrical signals, and thus communicate with a communication network or other devices. The RF circuit 510 may include various existing circuit elements for performing these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, user identity module (SIM) cards, memories, etc. The RF circuit 510 can communicate with various networks such as the Internet, corporate intranets, wireless networks, or communicate with other devices through wireless networks. The above-mentioned wireless networks may include cellular telephone networks, wireless local area networks, or metropolitan area networks. The above-mentioned wireless networks can use various communication standards, protocols and technologies, including but not limited to Global System for Mobile Communication (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Wireless Fidelity (Wi-Fi) (such as Institute of Electrical and Electronics Engineers standards IEEE 802.11a, IEEE 802.11b, IEEE802.11g and / or IEEE802.11n), Voice over Internet Protocol (VoIP), Worldwide Interoperability for Microwave Access (Wi-Max), other protocols for email, instant messaging and short messages, and any other suitable communication protocols, even those that have not yet been developed.

[0093] The memory 520 may be used to store software programs and modules, such as program instructions / modules corresponding to the image description generation method in the above-mentioned embodiment. The processor 580 executes various functional applications and generates image descriptions by running the software programs and modules stored in the memory 520.

[0094] The memory 520 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 520 may further include a memory remotely arranged relative to the processor 580, and these remote memories may be connected to the electronic device 500 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0095] The input unit 530 can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control. Specifically, the input unit 530 may include a touch-sensitive surface 531 and other input devices 532. The touch-sensitive surface 531, also known as a touch display screen or a touch pad, can collect user touch operations on or near it (such as operations performed by users using fingers, styluses, or any other suitable objects or accessories on or near the touch-sensitive surface 531), and drive corresponding connection devices according to a pre-set program. Optionally, the touch-sensitive surface 531 may include a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into touch point coordinates, and then sends it to the processor 580, and can receive and execute commands sent by the processor 580. In addition, the touch-sensitive surface 531 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch-sensitive surface 531, the input unit 530 may further include other input devices 532. Specifically, the other input devices 532 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

[0096] The display unit 540 can be used to display information input by the user or information provided to the user and various graphical user interfaces of the electronic device 500, which can be composed of graphics, text, icons, videos and any combination thereof. The display unit 540 may include a display panel 541, and optionally, the display panel 541 may be configured in the form of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc. Further, the touch-sensitive surface 531 may cover the display panel 541, and when the touch-sensitive surface 531 detects a touch operation on or near it, it is transmitted to the processor 580 to determine the type of the touch event, and then the processor 580 provides a corresponding visual output on the display panel 541 according to the type of the touch event. Although in the figure, the touch-sensitive surface 531 and the display panel 541 are implemented as two independent components to implement input and output functions, in some embodiments, the touch-sensitive surface 531 and the display panel 541 can be integrated to implement input and output functions.

[0097] The electronic device 500 may also include at least one sensor 550, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 541 according to the brightness of the ambient light, and the proximity sensor may generate an interrupt when the flip cover is closed or closed. As a type of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary, which can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc. that can also be configured in the electronic device 500, they will not be repeated here.

[0098] The audio circuit 560, the speaker 561, and the microphone 562 can provide an audio interface between the user and the electronic device 500. The audio circuit 560 can transmit the electrical signal converted from the received audio data to the speaker 561, which is converted into a sound signal for output; on the other hand, the microphone 562 converts the collected sound signal into an electrical signal, which is received by the audio circuit 560 and converted into audio data, and then the audio data is output to the processor 580 for processing, and then sent to another terminal through the RF circuit 510, or the audio data is output to the memory 520 for further processing. The audio circuit 560 may also include an earplug jack to provide communication between an external headset and the electronic device 500.

[0099] The electronic device 500 can help the user receive requests, send information, etc. through the transmission module 570 (such as a Wi-Fi module), which provides the user with wireless broadband Internet access. Although the transmission module 570 is shown in the figure, it is understandable that it is not a necessary component of the electronic device 500 and can be omitted as needed without changing the essence of the invention.

[0100] The processor 580 is the control center of the electronic device 500. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 520, and calling data stored in the memory 520, it executes various functions of the electronic device 500 and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 580 may include one or more processing cores; in some embodiments, the processor 580 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communications. It can be understood that the above-mentioned modem processor may not be integrated into the processor 580.

[0101] The electronic device 500 also includes a power supply 590 (such as a battery) for supplying power to various components. In some embodiments, the power supply can be logically connected to the processor 580 through a power management system, so that the power management system can manage charging, discharging, and power consumption management. The power supply 590 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0102] Although not shown, the electronic device 500 also includes a camera (such as a front camera, a rear camera), a Bluetooth module, etc., which will not be described in detail here. Specifically in this embodiment, the display unit of the electronic device is a touch screen display, and the mobile terminal also includes a memory, and one or more programs, wherein one or more programs are stored in the memory, and are configured to be executed by one or more processors to implement any step in the image description generation method provided in the above embodiment.

[0103] In specific implementation, the above modules can be implemented as independent entities, or can be arbitrarily combined and implemented as the same or several entities. The specific implementation of the above modules can be found in the previous method embodiments, which will not be repeated here.

[0104] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by controlling related hardware through instructions, and the instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. To this end, an embodiment of the present invention provides a storage medium, which stores multiple instructions, and the instructions can be executed by the processor to implement any step in the image description generation method provided in the above embodiments.

[0105] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0106] Since the instructions stored in the storage medium can execute the steps in any embodiment of the image description generation method provided in the embodiments of the present invention, the beneficial effects that can be achieved by any image description generation method provided in the embodiments of the present invention can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0107] The above is a detailed introduction to an image description generation method, device, electronic device and storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and its core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application. Moreover, for those of ordinary skill in the art, without departing from the principles of the present invention, several improvements and modifications can be made, and these improvements and modifications are also regarded as the scope of protection of the present invention.

Claims

1. A method for generating an image description, characterized in that: include: Acquire text information to be processed and an image to be processed, wherein the text information to be processed is text information related to the image to be processed; Inputting the to-be-processed text information and the to-be-processed image into a trained image description generation model to perform image description generation processing, thereby obtaining a first image description related to the to-be-processed text information and a second image description related to the to-be-processed image; A target image description of the image to be processed is determined according to the first image description and the second image description.

2. The method according to claim 1, characterized in that The image description generation model includes a first generation sub-model and a second generation sub-model; The step of inputting the to-be-processed text information and the to-be-processed image into the trained target generation model for image description generation processing to obtain a first image description related to the to-be-processed text information and a second image description related to the to-be-processed image includes: Calling the first generation sub-model to perform prediction processing on the text encoding vector of the to-be-processed text information to obtain a first image description of the to-be-processed image, wherein the first image description is at least part of the text information in the to-be-processed text information; The second generation sub-model is called to decode the text encoding vector of the text information to be processed and the image encoding vector of the image to be processed to obtain a second image description of the image to be processed.

3. The method according to claim 2, characterized in that The image description generation model also includes a text encoder sub-model and an image encoder sub-model; Before the step of calling the first generation sub-model to perform prediction processing on the text encoding vector of the to-be-processed text information to obtain the first image description of the to-be-processed image, the method further includes: Calling the text encoder sub-model to perform text encoding processing on the text information to be processed to obtain a first encoding vector; Calculating the attention matrix of the first encoding vector to obtain a text encoding vector of the text information to be processed; Before the step of calling the second generation sub-model to decode the text encoding vector of the text information to be processed and the image encoding vector of the image to be processed to obtain a second image description of the image to be processed, the method further includes: Calling the image encoder sub-model to perform image encoding processing on the image to be processed to obtain a second encoding vector; An attention matrix of the second coding vector is calculated to obtain an image coding vector of the image to be processed.

4. The method according to claim 3, characterized in that The calling of the second generation sub-model to decode the text encoding vector of the to-be-processed text information and the image encoding vector of the to-be-processed image to obtain a second image description of the to-be-processed image includes: Linearly adding the text encoding vector of the to-be-processed text information and the image encoding vector of the to-be-processed image to obtain a target encoding vector; The second generation sub-model is called to decode the target encoding vector to obtain a second image description of the image to be processed.

5. The method according to claim 1, characterized in that The step of determining a target image description of the image to be processed according to the first image description and the second image description includes: A weighted sum is performed on the first image description and the second image description to obtain a target image description of the image to be processed.

6. The method according to claim 5, characterized in that A sum of a first weight of the first image description and a second weight of the second image description is 1, and the first weight is less than the second weight.

7. The method according to claim 1, characterized in that Before the step of inputting the text information to be processed and the image to be processed into the trained image description generation model to perform image description generation processing to obtain a first image description related to the text information to be processed and a second image description related to the image to be processed, the method further includes: Acquire a training data set, the training data set comprising training images, training text information, and label information, the label information comprising first label information of the training text information and second label information of an image description of the training image; According to the training text information and the label information, the first generation sub-model to be trained is trained until convergence, so as to obtain a trained first generation sub-model; According to the training image and the label information, the second generation sub-model to be trained is trained until convergence to obtain a trained second generation sub-model; The trained first generation sub-model and the trained second generation sub-model are combined to obtain an image description generation model.

8. An image description generating device, characterized in that: include: A first acquisition module is used to acquire text information to be processed and an image to be processed, wherein the text information to be processed is text information related to the image to be processed; An input module, used for inputting the text information to be processed and the image to be processed into the trained image description generation model to perform image description generation processing, so as to obtain a first image description related to the text information to be processed and a second image description related to the image to be processed; A determination module is used to determine a target image description of the image to be processed according to the first image description and the second image description.

9. An electronic device, characterized in that: The electronic device comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor implements the steps in the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the method according to any one of claims 1 to 7 are implemented.

Images