Image description information generation method and device, computer device, and storage medium

Abstract

The application relates to an image description information generation method and device, computer equipment and a storage medium, and relates to the technical field of image processing. The method comprises the following steps: acquiring a target image; extracting a semantic feature set and a visual feature set of the target image; performing attention fusion on the semantic features of the target image and the visual features of the target image at n time steps through an attention fusion network in an image description information generation model to obtain description words corresponding to the n time steps; and generating image description information of the target image based on the description words corresponding to the n time steps. Through the above method, the advantages of visual features in generating visual words and the advantages of semantic features in generating non-visual features are complementary in the process of generating image description information, so that the accuracy of generating image description information is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to an image description information generation method and device, computer equipment and a storage medium. BACKGROUND

[0002] With the development of image recognition technology, the "picture speaking" function of a computer, i.e., image description, can be realized by an algorithm, which converts content information in an image into image description information.

[0003] In related technologies, attention is often paid to generating image description information of an image by extracting visual features of the image, i.e., after obtaining the visual features of the image by using an encoder, a recurrent neural network is used to generate the overall description of the image.

[0004] However, in the above related technology, only relying on the visual features of the image can result in poor accuracy of non-visual words, such as verbs and prepositions, in the generated image description information, thereby resulting in poor accuracy of the generated image description information. SUMMARY

[0005] Embodiments of the present application provide an image description information generation method and device, computer equipment and a storage medium, which can improve the accuracy of generated image description information. The technical solution is as follows:

[0006] In one aspect, an image description information generation method is provided, and the method comprises:

[0007] obtaining a target image;

[0008] extracting a semantic feature set of the target image;

[0009] extracting a visual feature set of the target image;

[0010] The attention fusion network in the image description information generation model is used for attention fusion of semantic features of the target image and visual features of the target image at n time steps to obtain description words corresponding to the n time steps; input of the attention fusion network at a t-th time step includes a semantic attention vector at the t-th time step, a visual attention vector at the t-th time step, and an output result of the attention fusion network at a (t-1)-th time step; the semantic attention vector at the t-th time step is obtained by performing attention mechanism processing on the semantic feature set at the t-th time step; the visual attention vector at the t-th time step is obtained by performing attention mechanism processing on the visual feature set at the t-th time step; the output result of the attention fusion network at the (t-1)-th time step is used to indicate a description word corresponding to the (t-1)-th time step; the t-th time step is any one of the n time steps; 1≤t≤n, and t and n are positive integers;

[0011] Based on the description words of the target image at the n time steps, image description information of the target image is generated.

[0012] In another aspect, an image description information generation device is provided, and the device includes:

[0013] A target image acquisition module is configured to acquire a target image.

[0014] A semantic feature extraction module is configured to extract a semantic feature set of the target image.

[0015] A visual feature extraction module is configured to extract a visual feature set of the target image.

[0016] The description word acquisition module is configured to acquire description words corresponding to the n time steps by performing attention fusion on semantic features of the target image and visual features of the target image on the n time steps through an attention fusion network in an image description information generation model; input of the attention fusion network on the tth time step includes a semantic attention vector on the tth time step, a visual attention vector on the tth time step, and an output result of the attention fusion network on a (t-1)th time step; the semantic attention vector on the tth time step is obtained by performing attention mechanism processing on the semantic feature set on the tth time step; the visual attention vector on the tth time step is obtained by performing attention mechanism processing on the visual feature set on the tth time step; the output result of the attention fusion network on the (t-1)th time step is used to indicate a description word corresponding to the (t-1)th time step; the tth time step is any one of the n time steps; 1≤t≤n, and t and n are positive integers;

[0017] The image description information generation module is configured to generate image description information of the target image based on the description words of the target image on the n time steps.

[0018] In a possible implementation, the description word acquisition module is configured to input the semantic attention vector on the tth time step, the visual attention vector on the tth time step, a hidden layer vector on the (t-1)th time step, and the output result of the attention fusion network on the (t-1)th time step to the attention fusion network on the tth time step, to obtain the output result of the attention fusion network on the tth time step and the hidden layer vector on the tth time step.

[0019] Alternatively,

[0020] In the tth time step, the semantic attention vector on the tth time step, the visual attention vector on the tth time step, and the output result of the attention fusion network on the (t-1)th time step are input to the attention fusion network, to obtain the output result of the attention fusion network on the tth time step and the hidden layer vector on the tth time step.

[0021] In a possible implementation, a hyperparameter is arranged in the attention fusion network, and the hyperparameter is used to indicate a weight of the visual attention vector and the semantic attention vector in the attention fusion network.

[0022] In a possible implementation, the apparatus further includes:

[0023] a semantic attention vector generation module, configured to generate, at the t th time step, the semantic attention vector at the t th time step based on the hidden layer vector at the (t-1) th time step and the semantic feature set.

[0024] The semantic attention generation module comprises:

[0025] a semantic feature weight obtaining sub-module, configured to obtain, based on the hidden layer vector at the (t-1) th time step and the semantic feature set, a weight of each semantic feature in the semantic feature set at the (t-1) th time step;

[0026] a semantic attention vector generation sub-module, configured to generate, based on the weight of each semantic feature in the semantic feature set at the (t-1) th time step and the semantic feature set, the semantic attention vector at the t th time step.

[0027] In a possible implementation, the apparatus further comprises:

[0028] a visual attention generation module, configured to generate, at the t th time step, the visual attention vector at the t th time step based on the hidden layer vector at the (t-1) th time step and the visual feature set.

[0029] In a possible implementation, the visual attention generation module comprises:

[0030] a semantic feature weight obtaining sub-module, configured to obtain, based on the hidden layer vector at the (t-1) th time step and the visual feature set, a weight of each visual feature in the visual feature set at the (t-1) th time step;

[0031] a visual attention vector generation sub-module, configured to generate, based on the weight of each visual feature in the visual feature set at the (t-1) th time step and the visual feature set, the visual attention vector at the t th time step.

[0032] In a possible implementation, the semantic feature extraction module comprises:

[0033] a semantic feature vector obtaining sub-module, configured to obtain a semantic feature vector of the target image;

[0034] a semantic feature extraction sub-module, configured to extract the semantic feature set of the target image based on the semantic feature vector.

[0035] In a possible implementation, the semantic feature extraction sub-module comprises:

[0036] The attribute word extraction unit is configured to extract, based on the semantic feature vector, a set of attribute words corresponding to the target image from a vocabulary, the set of attribute words being a set of candidate description words for describing the target image.

[0037] The semantic feature extraction unit is configured to obtain a set of word vectors corresponding to the set of attribute words as the set of semantic features of the target image.

[0038] In a possible implementation, the attribute word extraction unit is configured to obtain, based on the semantic feature vector, a matching probability of each vocabulary in the vocabulary, the matching probability being a probability that each vocabulary in the vocabulary matches the target image.

[0039] The attribute word extraction unit is configured to extract, as the attribute word, a vocabulary in the vocabulary that has a matching probability greater than a matching probability threshold.

[0040] In a possible implementation, the attribute word extraction unit is configured to input the semantic feature vector into a vocabulary detector to obtain the set of attribute words extracted by the vocabulary detector from the vocabulary based on the semantic feature vector.

[0041] The vocabulary detector is a vocabulary detection model trained by a weakly supervised method of multiple-instance learning.

[0042] In a possible implementation, before the visual feature extraction module extracts the set of visual features of the target image, the apparatus further includes:

[0043] The sub-region division module is configured to divide the target image into at least one sub-region.

[0044] The visual feature extraction module is configured to extract visual features of the at least one sub-region respectively to form the set of visual features.

[0045] In another aspect, a computer device is provided, which includes a processor and a memory, the memory storing at least one instruction, at least one program, a code set, or an instruction set, which is loaded and executed by the processor to implement the image description information generation method.

[0046] In another aspect, a computer readable storage medium is provided, which stores at least one computer program, which is loaded and executed by a processor to implement the image description information generation method.

[0047] In another aspect, a computer program product or computer program is provided, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the image description information generation method provided in various optional implementation manners described above.

[0048] The technical solutions provided in the present application can include the following beneficial effects:

[0049] By extracting the semantic feature set and the visual feature set of the target image respectively, and using the attention fusion network in the image description information generation model, the attention fusion of the semantic features and the visual features is realized, so that based on the comprehensive effect of the output results of the visual features and the semantic features of the target image at the previous time step, the description words of the target image at the current time step are generated, and then the image description information corresponding to the target image is generated, so that in the generation process of the image description information, the advantages of the visual features in generating visual words and the advantages of the semantic features in generating non-visual features are complementary, thereby improving the accuracy of generating the image description information. BRIEF DESCRIPTION OF DRAWINGS

[0050] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and serve to explain the principles of the present application together with the specification.

[0051] Figure 1 A schematic diagram of a system used by an image description information generation method provided by an exemplary embodiment of the present application is shown;

[0052] Figure 2 A flowchart of an image description information generation method provided by an exemplary embodiment of the present application is shown;

[0053] Figure 3 A schematic diagram of extracting word information in an image based on different attention according to an exemplary embodiment of the present application is shown;

[0054] Figure 4 A schematic diagram of selecting a corresponding target image in a video scene according to an exemplary embodiment of the present application is shown;

[0055] Figure 5 A framework diagram of a model training stage and an image description information generation stage according to an exemplary embodiment of the present application is shown;

[0056] Figure 6 A flowchart of a training method of an image description information generation model provided by an exemplary embodiment of the present application is shown;

[0057] Figure 7 A flow chart of a model training and an image description information generation method is shown according to an example embodiment of the present application.

[0058] Figure 8 A schematic diagram of an image description information generation process is shown according to an example embodiment of the present application.

[0059] Figure 9 An input and output schematic diagram of an attention fusion network is shown according to an example embodiment of the present application.

[0060] Figure 10 A framework diagram of an image description information generation apparatus is shown according to an example embodiment of the present application.

[0061] Figure 11 A structural block diagram of a computer device is shown according to an example embodiment of the present application.

[0062] Figure 12 A structural block diagram of a computer device is shown according to an example embodiment of the present application. DETAILED DESCRIPTION

[0063] The example embodiments will be described in detail herein with reference to the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following example embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0064] It should be understood that "multiple" referred to herein means two or more. The "and / or" describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. The character " / " generally represents that the associated objects before and after it are in an "or" relationship.

[0065] The embodiments of the present application provide an image description information generation method, which can improve the accuracy of image description information generation. In order to facilitate understanding, the following explains several terms related to the present application.

[0066] 1) Image description (Image Caption)

[0067] Image description refers to taking an image as input, obtaining a natural language description of the corresponding image through a model and calculation. The image description can be a description of the properties of each component in the image, or a description of the relationship between each part.

[0068] 2) CNN (Convolutional Neural Networks)

[0069] CNN is a feedforward neural network, a unidirectional multi-layered neural network. Neurons in the same layer are not interconnected, and information is transmitted between layers in only one direction. Apart from the input and output layers, all the layers in between are hidden layers, which can be one or more. CNN can directly extract features from the pixel features at the bottom layer of the image, and is the most commonly used implementation model of encoders, responsible for encoding images into vectors.

[0070] 3) RNN (Recurrent Neural Networks)

[0071] Recurrent Neural Networks (RNNs) are a type of neural network used to process sequential data. In an RNN, the current output of a sequence is related to the previous outputs. Based on a neural network with fixed weights, external inputs, and internal states, an RNN can be viewed as a dynamic behavior of the internal state with respect to the weights and external inputs as parameters. RNNs are the most commonly used implementation model for decoders, responsible for translating the image vectors generated by the encoder into textual descriptions of the images.

[0072] 4) LSTM (Long Short Term Memory) network

[0073] LSTM is a time recurrent neural network used to predict important times in a time series with relatively long intervals or delays; it is a special type of RNN.

[0074] 5) Attention Mechanism

[0075] Attention mechanisms are essentially a set of weighted coefficients learned autonomously by the network, used in a "dynamically weighted" manner to emphasize regions of interest to the user while suppressing irrelevant background regions. In the field of computer vision, attention mechanisms can be broadly classified into two categories: strong attention and soft attention.

[0076] Attention mechanisms are often used in RNNs. When processing a portion of the target image, an RNN with an attention mechanism processes the portion of the target image that was focused on in the previous state, rather than processing all the pixels of the target image. This can reduce the processing complexity of the task.

[0077] 6) Artificial Intelligence (AI)

[0078] Artificial intelligence is the theory, method, technology and application system of using digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0079] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software technologies. Artificial intelligence basic technologies generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc. The display device shown in the present application mainly relates to computer vision technology and machine learning / deep learning, etc.

[0080] 7) Machine Learning (ML)

[0081] Machine learning is a multi-disciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, etc. It is a subject that studies how computers simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge structure to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent, and its applications are widespread in various fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0082] Please refer to Figure 1 , which shows a schematic diagram of a system used by the image description information generation method provided by an exemplary embodiment of the present application, as shown in Figure 1 , the system includes a server 110 and a terminal 120.

[0083] Among them, the above-mentioned server 110 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms.

[0084] The terminal 120 can be a terminal device with network connection function and image display function and / or video playing function. Further, the terminal can be a terminal with image description information generation function. For example, the terminal 120 can be a smart phone, a tablet computer, an e-book reader, smart glasses, a smart watch, a smart television, an MP3 player, an MP4 player, a laptop computer, a desktop computer, and the like.

[0085] Optionally, the system includes one or more servers 110 and a plurality of terminals 120. The number of servers 110 and terminals 120 is not limited in the embodiments of the present application.

[0086] The terminal and the server are connected through a communication network. Optionally, the communication network is a wired network or a wireless network.

[0087] Optionally, the wireless network or the wired network uses standard communication technology and / or protocol. The network is usually the Internet, but can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a private network or a virtual private network of any combination. In some embodiments, technologies and / or formats such as hyper text mark-up language (HTML), extensible mark-up language (XML) and the like are used to represent data exchanged through the network. In addition, all or some links can be encrypted using conventional encryption technologies such as secure socket layer (SSL), transport layer security (TLS), virtual private network (VPN), Internet protocol security (IPsec) and the like. In other embodiments, custom and / or dedicated data communication technologies can be used instead of or in addition to the above data communication technologies. The present application does not limit here.

[0088] Figure 2A flowchart of an image description information generation method provided by an example embodiment of the present application is shown. The method is executed by a computer device, which can be implemented as a terminal or a server. The terminal or server can be Figure 1 The terminal or server is shown as Figure 2 The image description information generation method includes the following steps:

[0089] Step 210: Obtain a target image.

[0090] In a possible implementation, the target image can be a locally stored image, or the target image can also be an image obtained in real time based on a user specified operation. For example, the target image can be an image obtained in real time by the user based on a screen capture operation, or can be an image captured in real time on the terminal screen when the user triggers generation of image description information by long pressing a specified area of the screen, or can be an image obtained in real time based on an image capturing component of the terminal. The present application does not limit the manner of obtaining the target image.

[0091] Step 220: Extract a semantic feature set of the target image.

[0092] The semantic feature set of the target image is used to indicate a set of word vectors corresponding to candidate description words of image information describing the target image.

[0093] Step 230: Extract a visual feature set of the target image.

[0094] The visual feature set of the target image is used to indicate a set of image features obtained based on RGB distribution and other features of pixel points of the target image.

[0095] Step 240: Through an attention fusion network in the image description information generation model, perform attention fusion on the semantic features of the target image and the visual features of the target image at n time steps, to obtain description words corresponding to the n time steps.

[0096] The input of the attention fusion network at the t-th time step includes a semantic attention vector at the t-th time step, a visual attention vector at the t-th time step, and an output result of the attention fusion network at a (t-1)-th time step; the semantic attention vector at the t-th time step is obtained by performing an attention mechanism on the semantic feature set at the t-th time step; the visual attention vector at the t-th time step is obtained by performing an attention mechanism on the visual feature set at the t-th time step; the output result of the attention fusion network at the (t-1)-th time step is used to indicate a descriptive word corresponding to the (t-1)-th time step; the t-th time step is any one of n time steps; 1≤t≤n, and t and n are positive integers. The n time steps represent the number of time steps required to generate image description information of a target image.

[0097] In the embodiment of the application, when generating image description information, a next word is generated based on a generated word after generating the word, wherein the time required to generate a word is referred to as a time step. Optionally, the number n of time steps can be a non-fixed value greater than 1, and the generation of the descriptive word is ended in response to the generated descriptive word being a word or character indicating the end of the descriptive word generation process.

[0098] The image description information generation model in the embodiment of the application is generated by training a sample image and image description information corresponding to the sample image, wherein the image description information of the sample image can be text information.

[0099] The semantic attention vector can simultaneously strengthen the generation of visual words and non-visual words by using multiple attributes; the visual word refers to word information that can be directly extracted based on pixel information of an image, such as a word corresponding to a noun in the image; and the non-visual word refers to word information that is less likely to be extracted or cannot be directly extracted based on pixel information of an image, such as a verb and a preposition.

[0100] The visual attention vector can strengthen the generation of visual words and has good performance in extracting visual words in an image. Figure 3 An example of extracting word information in an image based on different attentions according to an example embodiment of the application is shown in FIG. 1. Figure 3 As shown in FIG. 1, Figure 3 Part A of FIG. 1 shows the weight changes of each word obtained by the specified image under the action of the semantic attention mechanism; Figure 3Part B in FIG. 6 shows the weight changes of each word of the same specified image under the action of the visual attention mechanism. For the three words "people", "standing" and "table", under the semantic attention mechanism, at the moment when each word is generated, the weight corresponding to each word reaches the peak, that is, the semantic attention mechanism will focus on the word with the highest relevance to the current context. Corresponding to "people", "standing" and "table", under the visual attention mechanism, when the visual words are generated, for example, "people" and "table", the visual attention will focus on the corresponding image region in the specified image. As shown in FIG. 6, when "people" is generated, the visual attention focuses on the region 310 of the specified image containing the face, and when "table" is generated, the visual attention focuses on the region 320 of the specified image containing the table; but when the non-visual word is generated based on the visual attention mechanism, such as "standing", the visual attention mechanism focuses on the irrelevant and possibly misleading image region 330. Figure 3

[0101] Therefore, in order to combine the advantages of the visual attention mechanism in generating visual words and the advantages of the semantic attention mechanism in generating non-visual words, in the embodiment of the present application, the visual attention and the semantic attention are combined, which can more accurately guide the generation of visual words and non-visual words, and reduce the interference of visual attention in generating non-visual words, so that the image description is more complete and rich.

[0102] In step 250, the image description information of the target image is generated based on the description words corresponding to the n time steps.

[0103] In a possible implementation, the description words corresponding to the n time steps are sorted in a specified order, such as sequential sorting, to generate the target image description information of the target image.

[0104] In summary, the image description information generation method provided by the embodiment of the present application extracts the semantic feature set and the visual feature set of the target image respectively, and uses the attention fusion network in the image description information generation model to realize the attention fusion of the semantic features and the visual features. Based on the comprehensive effect of the output results of the visual features and the semantic features of the target image at the previous time step, the description word of the target image at the current time step is generated, and then the image description information corresponding to the target image is generated, so that in the generation process of the image description information, the advantages of the visual features in generating visual words and the advantages of the semantic features in generating non-visual features are complementary, thereby improving the accuracy of generating the image description information.

[0105] ​Illustratively, the scheme described in the embodiments of the present application can be applied to, but not limited to, the following scenarios:

[0106] 1. A scenario in which visually impaired people obtain image information;

[0107] The visual function of visually impaired people (i.e., people with visual impairment) is impaired to some extent, because the visual acuity is reduced or the visual field is damaged, so that the normal vision cannot be achieved, and the acquisition of visual information is affected. For example, when a visually impaired person uses a mobile phone to view text or video, because the complete visual information content cannot be obtained through vision, the information in the image needs to be obtained with the aid of hearing. One possible way is that the user selects the area or range of the content to be viewed, generates image description information corresponding to the area through the image description information generation method in the embodiments of the present application, and converts the image description information from text information to audio information for playing, thereby assisting the visually impaired person to obtain complete image information.

[0108] Figure 4 A schematic diagram of corresponding target image selection in a video scenario is shown according to an example embodiment of the present application, as shown in Figure 4 The target image can be an image obtained based on the specified operation of the user in the video being played, or can be an image obtained based on the specified operation of the user from the dynamic image of the live room displayed in the live preview interface in real time. The dynamic image displayed in the live preview interface is used to assist the user to make a decision on whether to enter the live room to watch through the real-time content in the live room. In one possible implementation, the user can determine the current (when the click operation is received) image in the area by clicking (specified operation) the dynamic image to obtain the target image. In order to strengthen the display of the selection of the target image, the area can be highlighted, such as highlighted display, or enlarged display, etc. As shown in Figure 4 The range of the area 410 is highlighted.

[0109] 2. An early education scenario;

[0110] In the early education scenario, because the recognition range of objects or text of young children is limited, teaching through images will have better teaching effect. In this scenario, the image information description of the image touched by the young children can be performed through the method shown in the present application, so as to transmit information to the young children from two directions of vision and hearing, stimulate the learning interest of the young children, and improve the information transmission effect.

[0111] The scheme related in the present application includes a model training phase and an image description information generation phase. Figure 5 is a framework diagram of a model training phase and an image description information generation phase according to an example embodiment, as shown in Figure 5As shown, in the model training stage, the model training device 510 obtains a visual-semantic double attention (VSDA) model, i.e., an image description information generation model, by using pre-set training samples (including sample images and image description information corresponding to the sample images, i.e., sequentially arranged description words), the visual-semantic double attention model including a semantic attention network, a visual attention network and an attention fusion network. In the image description information generation stage, the image description information generation device 520 processes the input target image based on the visual-semantic double attention model to obtain image description information corresponding to the target image.

[0112] The model training device 510 and the image description information generation device 520 can be computer devices, for example, the computer devices can be fixed computer devices such as personal computers and servers, or the computer devices can be mobile computer devices such as tablet computers and e-book readers.

[0113] Optionally, the model training device 510 and the image description information generation device 520 can be the same device, or the model training device 510 and the image description information generation device 520 can be different devices. When the model training device 510 and the image description information generation device 520 are different devices, the model training device 510 and the image description information generation device 520 can be the same type of device, for example, the model training device 510 and the image description information generation device 520 can both be servers; or the model training device 510 and the image description information generation device 520 can be different types of devices, for example, the image description information generation device 520 can be a personal computer or a terminal, and the model training device 510 can be a server. The embodiments of the present application do not limit the specific types of the model training device 510 and the image description information generation device 520.

[0114] Figure 6 A flowchart of a method for training an image description information generation model is shown, the method can be executed by a computer device, which can be implemented as a terminal or a server, and the terminal or the server can be a terminal or a server as shown in the above embodiments of the present application. Figure 1 The terminal or the server as shown, for example, Figure 6 The method for training the image description information generation model includes the following steps:

[0115] In step 610, a set of sample images is obtained, the set of sample images including at least two image samples and image description information corresponding to the at least two image samples respectively.

[0116] At step 620, the image description information generation model is obtained based on the sample image set.

[0117] The image description information generation model can be a visual-semantic dual attention model, including a semantic attention network, a visual attention network, and an attention fusion network. The semantic attention network is used to obtain a semantic attention vector based on a semantic feature set of a target image. The visual attention network is used to obtain a visual attention vector based on a visual feature set of the target image. The attention fusion network is used to perform attention fusion on the semantic features and the visual features of the target image to obtain a description word constituting image description information corresponding to the target image.

[0118] In summary, the training method of the image description information generation model provided in the embodiments of the present application obtains the image description information generation model including the semantic attention network, the visual attention network, and the attention fusion network based on the sample image set. In the process of generating image description information, the image description information generation model is used to generate a description word of a target image at a current time step based on the comprehensive effect of the output results of the visual features and the semantic features of the target image at a previous time step, and further generate image description information corresponding to the target image. In the process of generating image description information, the advantages of the visual features in generating visual words and the advantages of the semantic features in generating non-visual features are complementary, thereby improving the accuracy of generating image description information.

[0119] In the embodiments of the present application, the model training process can be performed by a server, and the image description information generation process can be performed by the server or a terminal. When the image description information generation process is performed by the terminal, the server sends the trained visual-semantic dual attention model to the terminal, so that the terminal can process the obtained target image based on the visual-semantic dual attention model to obtain image description information of the target image. The following embodiments take the example of the model training process and the image description information generation process performed by the server. Figure 7 A flowchart of a model training and image description information generation method provided in an example embodiment of the present application is shown. The method is performed by a computing device, as shown in Figure 7 The model training and image description information generation method includes the following steps:

[0120] At step 701, a sample image set is obtained, which includes at least two image samples and image description information corresponding to the at least two image samples, respectively.

[0121] Illustratively, the image description information corresponding to each sample image can be labeled by relevant personnel.

[0122] At step 702, the image description information generation model is trained based on the sample image set to obtain the image description information generation model.

[0123] The image description information generation model is a visual-semantic dual attention model, which includes a semantic attention network, a visual attention network, and an attention fusion network. The semantic attention network is used to obtain a semantic attention vector based on a semantic feature set of a target image. The visual attention network is used to obtain a visual attention vector based on a visual feature set of the target image. The attention fusion network is used to perform attention fusion on the semantic features and the visual features of the target image to obtain a description word that constitutes image description information corresponding to the target image.

[0124] In a possible implementation, the image description information generation model further includes a semantic convolutional neural network and a visual convolutional neural network. The semantic convolutional neural network is used to process the target image to obtain a semantic feature vector of the target image, so as to obtain a description word set corresponding to the target image. The visual convolutional neural network is used to process the target image to obtain a visual feature set corresponding to the target image.

[0125] In a possible implementation, the above process is implemented as follows: each sample image in the sample image set is input into the image description information generation model to obtain predicted image description information corresponding to each sample image.

[0126] Based on the predicted image description information corresponding to each sample image and the image description information corresponding to each sample image, a loss function value is calculated.

[0127] Based on the loss function value, the parameters of the image description information generation model are updated.

[0128] Since the output result (i.e., the predicted image description information) of the image description information generation model based on the sample image needs to be similar to the image description information corresponding to the sample image, so as to ensure the accuracy of the image description information generation model in generating image description information of a target image in application, the image description information generation model needs to be trained multiple times in the training process of the image description information generation model, and each parameter in each network in the image description information generation model is updated until the image description information generation model converges.

[0129] Let θ represent all the parameters involved in the image description information generation model. Given a target sequence (Ground Truth Sequence) {w1, w2,..., wn}, i.e., a description word sequence in the image description information of the target image, the loss function is a minimum cross entropy (Cross Entropy loss) function, and the loss function value corresponding to the image description information generation model is calculated as follows: t

[0130] ​

[0131] The probability of each description word in the image description information predicted by the image description information generation model is represented by The parameters in the image description information generation model are adjusted based on the calculation result of the loss function.

[0132] Step 703, obtaining the target image.

[0133] In response to the generation process of the image description information being performed by the server, the target image can be an image obtained by the terminal and then sent to the server for image description information acquisition, and accordingly, the server receives the target image.

[0134] Step 704, obtaining the semantic feature vector of the target image.

[0135] In a possible implementation, the target image is input into the semantic convolutional neural network to obtain the semantic feature vector of the target image output by the semantic convolutional neural network.

[0136] The semantic convolutional neural network can be a fully convolutional network (FCN), or can also be a convolutional neural network (CNN). Through the processing of the target image by the semantic convolutional neural network, a rough graph representation vector of the target image, i.e., the semantic feature vector of the target image, can be obtained.

[0137] For each target image, not all attribute words correspond to the target image. If probability calculation or verification is performed on all words in the attribute word library, excessive and unnecessary data processing will be caused. Therefore, before obtaining the description word set, the semantic feature vector used to indicate the attributes of the target image obtained from the target image is used to filter all attribute words in the vocabulary library to obtain an attribute word set that can correspond to the target image, i.e., a candidate description word set.

[0138] Step 705, based on the semantic feature vector, extracting a semantic feature set of the target image.

[0139] In a possible implementation, based on the semantic feature vector, an attribute word set corresponding to the target image is extracted from the vocabulary library; the attribute word set is a candidate description word set for describing the target image.

[0140] The word vector set corresponding to the attribute word set is obtained as the semantic feature set of the target image.

[0141] The candidate description words in the attribute word set are attribute words corresponding to the context of the target image. The number of the candidate description words in the attribute word set is not limited in the present application. Different forms of a unified word, such as play, playing, and plays, can be included in the candidate description words.

[0142] In a possible implementation, the candidate description words can be filtered from the vocabulary library by obtaining the matching probability of each vocabulary, and the process is implemented as follows:

[0143] Based on the semantic feature vector, the matching probability of each vocabulary in the vocabulary library is obtained, which is the probability that each vocabulary in the vocabulary library matches the target image.

[0144] The vocabulary in the vocabulary library with a matching probability greater than a matching probability threshold is extracted as an attribute word.

[0145] In a possible implementation, the probability of each attribute word in the image can be calculated by the Noisy-OR method. In order to improve the accuracy of the obtained attribute words, the probability threshold can be obtained as 0.5. It should be noted that the probability threshold can be adjusted according to actual conditions, and the present application does not limit this.

[0146] In order to improve the accuracy of the attribute words, in a possible implementation, the vocabulary detector can be pre-trained to obtain the attribute words from the vocabulary library based on the feature vector of the target image, that is:

[0147] The feature vector is input into the vocabulary detector, so that the vocabulary detector extracts the attribute words from the vocabulary library based on the feature vector.

[0148] Optionally, the vocabulary detector is a vocabulary detection model trained by a weak supervision method of multiple instance learning (MIL).

[0149] Step 706, extracting a visual feature set of the target image.

[0150] In a possible implementation, the target image is input into a visual convolutional neural network to obtain the visual feature set of the target image.

[0151] In a possible implementation, before extracting the visual feature set of the target image, the target image can be preprocessed, that is:

[0152] The target image is divided into sub-regions to obtain at least one sub-region.

[0153] The visual feature set of the target image includes:

[0154] extract visual features of the at least one sub-region respectively to form a visual feature set.

[0155] The target image is divided at equal intervals to obtain the at least one sub-region, wherein the division interval can be set differently based on different sizes of the target image, and the application does not limit the division size of the sub-region.

[0156] In step 707, the semantic features of the target image and the visual features of the target image are fused by the attention fusion network in the image description information generation model at n time steps to obtain description words corresponding to the n time steps.

[0157] In the tth time step, the semantic attention vector in the tth time step, the visual attention vector in the tth time step, the hidden layer vector in the (t-1)th time step, and the output result of the attention fusion network in the (t-1)th time step are input into the attention fusion network to obtain the output result of the attention fusion network in the tth time step and the hidden layer vector in the tth time step.

[0158] Or,

[0159] In the tth time step, the semantic attention vector in the tth time step, the visual attention vector in the tth time step, and the output result of the attention fusion network in the (t-1)th time step are input into the attention fusion network to obtain the output result of the attention fusion network in the tth time step and the hidden layer vector in the tth time step.

[0160] That is, in one possible implementation, the output result in the current time step can be obtained by the semantic attention vector and the visual attention vector acting on the output result in the previous time step; or in another possible implementation, in order to improve the accuracy of the output result in each time step, the output result in the current time step, i.e., the word vector of the description word corresponding to the current time step, can be obtained based on the semantic attention vector, the visual attention vector, the hidden layer vector in the previous time step, and the output result in the previous time step.

[0161] In order to obtain the description words of the target image in each time step, the attention vector of the target image in each time step is first obtained, which includes the semantic attention vector and the visual attention vector. Taking the tth time step as an example, when the semantic attention vector is obtained: in the tth time step, the semantic attention vector corresponding to the tth time step is generated based on the hidden layer vector output corresponding to the (t-1)th time step and the semantic feature set of the target image.

[0162] The hidden layer vector indicates the generated intermediate content corresponding to the generated description word, and the hidden layer vector contains historical information or context information for indicating the generation of the next description word, so that the next generated description word is more consistent with the current context.

[0163] The t-th time step represents any time step in n time steps, n represents the number of time steps required to generate the image description information, 1≤t≤n, and t and n are positive integers; when generating the semantic attention vector corresponding to the current time step, the semantic attention vector corresponding to the current time step is generated based on the hidden layer vector output in the last time step and the semantic feature set of the target image.

[0164] In a possible implementation, the hidden layer vector output corresponding to the t-1-th time step and the semantic feature set of the target image are input into the semantic attention network in the image description information generation model. The semantic attention vector corresponding to the t-th time step output by the semantic attention network is obtained.

[0165] The semantic attention network is used to obtain the weight of each semantic feature in the semantic feature set at the t-1-th time step based on the hidden layer vector at the t-1-th time step and the semantic feature set;

[0166] The semantic attention vector at the t-th time step is generated based on the weight of each semantic feature in the semantic feature set at the t-1-th time step and the semantic feature set.

[0167] The corresponding semantic attention vector in each time step is the weight sum of each attribute word, and the calculation formula is:

[0168] c t =b i ·h t-1

[0169] β t =softmax(c t )

[0170]

[0171] b i ={b1,...,.b L} indicates the attribute obtained from the image, and L indicates the length of the attribute, that is, the number of attribute words. Here b i is the word vector of each attribute word, c t represents a long-term memory vector, h t-1 represents the hidden layer vector at the t-1-th time step, β t represents the weight of each attribute word at the t-th time step, and A tThe semantic attention vector of the t-th time step is represented.

[0172] Taking the t-th time step as an example, when the visual attention vector is obtained: at the t-th time step, the visual attention vector at the t-th time step is generated based on the hidden layer vector at the t-1-th time step and the visual feature set.

[0173] When generating the visual attention vector corresponding to the current time step, the visual attention vector corresponding to the current time step is generated based on the hidden layer vector output by the previous time step and the visual feature set of the target image.

[0174] In a possible implementation, the hidden layer vector output by the t-1-th time step and the visual feature set of the target image are input into the visual attention model in the image description information generation model to obtain the semantic attention vector of the t-th time step output by the visual attention model.

[0175] The visual attention model is configured to obtain the weight of each visual feature in the visual feature set at the t-1-th time step based on the hidden layer vector at the t-1-th time step and the visual feature set.

[0176] The visual attention vector at the t-th time step is generated based on the weight of each visual feature in the visual feature set at the t-1-th time step and the visual feature set.

[0177] The visual attention vector corresponding to each time step is the weight sum of the visual features of each sub-region, and the calculation formula is:

[0178] α t =softmax(a i ·h t-1 )

[0179]

[0180] a i ={a1,…,a m} represents the visual features of each sub-region, i.e., the focal region m of the image, represents the number of sub-regions, i.e., the number of extracted visual features, and a t represents the weight corresponding to each visual feature, and V t represents the visual attention vector of the t-th time step.

[0181] In the calculation of the weight corresponding to the visual feature of each sub-region, an element-wise multiplication strategy is used to obtain better performance.

[0182] Since the attention model can capture more detailed sub-regional image features, the soft attention model can adaptively focus on the corresponding region when generating the description vocabulary of different objects, and the performance is better, so the soft attention mechanism is adopted as the visual attention model in the embodiments of the present application.

[0183] The visual attention model and the semantic attention model calculate the weight of the corresponding feature vector at each time step. Since the corresponding hidden layer vector is different in different time steps, the weight of each feature vector obtained at each time step is also different, so that the image focus area that pays attention to the context corresponding to each time step is more consistent and the feature words used to generate the image description.

[0184] In a possible implementation, the attention fusion network in the image description information generation model can be implemented as a sequence network, which includes an LSTM (Long Short Term Memory) network, a Transformer network, etc.

[0185] Taking the sequence network as an LSTM network as an example, when generating image description information, the visual attention vector V and the semantic attention vector A are used as additional input parameters of the LSTM network, and the two attention features are merged into the LSTM unit node to guide the generation of image description information. By guiding the image description information generation model to simultaneously pay attention to the visual features and semantic features of the image, the two feature vectors can complement each other.

[0186] In the embodiments of the present application, BOS and EOS symbols are used to represent the beginning and end of a sentence, respectively. Based on this, the LSTM network generates a description word based on the visual attention vector and the semantic attention vector, and the formula is as follows:

[0187]

[0188] i t =σ(W ix x t +W ih h t-1 +b i )

[0189] f t =σ(W fx x t +W fh h t-1 +b f )

[0190] o t =σ(W ox x t +W oh ht-1 +b o )

[0191]

[0192] h t =o t ⊙tanh(c t )

[0193] s t =W s h t

[0194] Where σ represents the sigmoid function; φ represents the maxout nonlinear activation function with two units. (representing unit); i t Represents input gate, f t This indicates a forget gate, o t This indicates the output gate.

[0195] LSTM uses a softmax function to output the probability distribution of the next word:

[0196] w t ~softmax(s t )

[0197] In one possible implementation, the attention fusion network in the image description information generation model is configured with hyperparameters that indicate the weights of the visual attention vector and the semantic attention vector in the attention fusion network.

[0198] During the generation of image description information, visual attention features and semantic attention features influence the image description information generated by the image description information generation model in different ways. The visual attention vector V guides the model to focus on relevant regions of the image, while the semantic attention vector A strengthens the generation of attribute words with the highest relevance. Since these two attention vectors are complementary, a hyperparameter is set in the attention fusion network to determine the optimal combination between the two attention vectors. Taking the attention fusion network as an LSTM network as an example, the formula for generating descriptive words based on the updated LSTM network using the visual attention vector and the semantic attention vector is as follows:

[0199]

[0200] i t =σ(W ix x t +W ih h t-1 +bi )

[0201] f t = σ(W fx x t + W fh h t-1 + b f )

[0202] o t = σ(W ox x t + W oh h t-1 + b o )

[0203]

[0204] h t = o t ⊙ tanh(c t )

[0205] s t = W s h t

[0206] wherein z represents a hyperparameter, the value range of which is [0.1, 0.9], to represent different weights of the two attention vectors, the greater z is, the greater the weight of the visual feature in the attention guidance, and vice versa, the smaller z is, the greater the weight of the semantic feature in the attention guidance.

[0207] It should be noted that the numerical value of the hyperparameter can be set according to the performance of the model under different weight distribution, and the application does not limit the numerical value of the hyperparameter.

[0208] In step 708, the image description information of the target image is generated based on the description words corresponding to the n time steps.

[0209] In a possible implementation, the image description information generated by the image description information generation model is description information in a fixed language, such as description information in English. In order to make the image description information more suitable for the use requirements of different users, in a possible implementation, in response to the generated target image description information being non-specified language description information, the generated target image description information is changed to specified language description information, for example, the image description information generated by the image description information generation model is description information in English, and the user's required information is information in Chinese. Therefore, after the image description information generation model generates the image description information in English, the image description information in English is translated into image description information in Chinese and then outputted.

[0210] The language type of the output image description information can be set by the user according to actual needs, and the language type of the image description information is not limited in the application.

[0211] In a possible implementation, since the generated image description information is text information, in order to facilitate the user to obtain the image description information, the image description information can be converted into voice image description information based on a TTS (Text-To-Speech, speech synthesis) technology, and the voice image description information is transmitted to the user in the form of voice playing.

[0212] The above process can be implemented that the server sends the obtained voice image description information to the terminal, so that the terminal plays the image description information according to the obtained voice image description information.

[0213] In summary, the model training and the image description information generation method provided in the embodiments of the application can extract the semantic feature set and the visual feature set of the target image respectively, and use the attention fusion network in the image description information generation model to realize the attention fusion of the semantic features and the visual features, so that the description word of the target image at the current time step is generated based on the comprehensive effect of the visual features and the semantic features of the target image on the output result at the previous time step at each time step of generating the image description information, and then the image description information corresponding to the target image is generated, so that the advantages of the visual features in generating visual words and the advantages of the semantic features in generating non-visual features are complementary in the generation process of the image description information, thereby improving the accuracy of generating the image description information.

[0214] Meanwhile, before the semantic attention network obtains the weights of the attribute words, the words in the word library are screened based on the feature vector of the image to obtain the attribute words related to the image as candidate description words, and the weight calculation is performed based on the candidate description words, thereby reducing the data processing amount of the semantic attention network, reducing the data processing pressure of the image description information generation model while ensuring the processing accuracy.

[0215] Taking the input of the attention fusion network as an LSTM network, the input of the attention fusion network includes the hidden layer vector at the previous time step, the output result at the previous time step, the visual attention vector at the current time step, and the semantic attention vector at the current time step, Figure 8 FIG. 1 shows a schematic diagram of an image description information generation process according to an example embodiment of the application, as Figure 8As shown, after acquiring the target image 810, the target image 810 is input into the image description information generation model 820. Through the semantic convolutional neural network 821 in the image description information generation model 820, the semantic feature vector of the target image is obtained. Then, candidate descriptive words 823 corresponding to the target image are obtained based on the word detector 822, thereby obtaining the semantic feature set corresponding to the target image. Simultaneously, the target image 810 is input into the visual convolutional neural network 824 in the image description information generation model 820 to obtain the visual feature set 825 corresponding to the target image. The semantic feature set is then input into the semantic attention network 826, so that the semantic attention network 826 obtains the semantic attention vector A corresponding to the current time step based on the hidden layer vector output from the previous time step. t t represents the current time step. When t = 1, the hidden layer vector output by the previous time step is the preset hidden layer vector. Correspondingly, the visual feature set is input into the visual attention network 827 so that the visual attention network 827 can obtain the visual attention vector V corresponding to the current time step based on the input hidden layer vector output by the previous time step. t ; the visual attention vector V t Semantic attention vector A t The hidden layer vector output from the previous time step and the descriptor x output from the previous time step. t (i.e., yt-1), input into LSTM network 828, to obtain the descriptor y corresponding to the current time step output by LSTM network 828. t When t=1, the descriptive words output in the previous time step are obtained as preset start words or characters; repeat the above process until the descriptive words output by the LSTM network are end words or end characters, and then arrange the obtained descriptive words in order to obtain the image description information 830 of the target image.

[0216] in, Figure 9 The diagram illustrates the input and output of an attention fusion network according to an exemplary embodiment of this application, as shown below. Figure 9 As shown, at time step t, the input to the attention fusion network 910 includes the hidden layer vector h corresponding to time step t-1. t-1 Based on h t-1 The generated visual attention vector V at time step t t Based on h t-1 The generated semantic attention vector A tand the graph representation vector of the descriptor output at the t-1 time step (i.e., the output vector at the t-1 time step), the output of the attention fusion network 910 includes the output vector at the t time step and the hidden layer vector at the t time step (used to generate the next descriptor). Wherein, the visual attention vector is calculated by the visual attention network 930 based on the weighted sum of the visual features corresponding to each sub-region, and the semantic attention vector is calculated by the semantic attention network 920 based on the weighted sum of each attribute word.

[0217] Figure 10 A framework diagram of an image description information generation device provided by an example embodiment of the present application is shown as follows. Figure 10 As shown in the figure, the device includes:

[0218] The target image acquisition module 1010 is configured to acquire a target image.

[0219] The semantic feature extraction module 1020 is configured to extract a semantic feature set of the target image.

[0220] The visual feature extraction module 1030 is configured to extract a visual feature set of the target image.

[0221] The descriptor acquisition module 1040 is configured to perform attention fusion on the semantic features of the target image and the visual features of the target image at n time steps through an attention fusion network in an image description information generation model, and acquire descriptors corresponding to the n time steps. The input of the attention fusion network at the t time step includes a semantic attention vector at the t time step, a visual attention vector at the t time step, and an output result of the attention fusion network at the t-1 time step. The semantic attention vector at the t time step is obtained by performing attention mechanism processing on the semantic feature set at the t time step. The visual attention vector at the t time step is obtained by performing attention mechanism processing on the visual feature set at the t time step. The output result of the attention fusion network at the t-1 time step is used to indicate the descriptor corresponding to the t-1 time step. The t time step is any one of the n time steps, 1≤t≤n, and t and n are positive integers.

[0222] The image description information generation module 1050 is configured to generate image description information of the target image based on the descriptors of the target image at the n time steps.

[0223] In a possible implementation, the description word obtaining module 1040 is configured to input the semantic attention vector at the t-th time step, the visual attention vector at the t-th time step, the hidden layer vector at the (t-1)-th time step, and an output result of the attention fusion network at the (t-1)-th time step into the attention fusion network at the t-th time step, to obtain the output result of the attention fusion network at the t-th time step and the hidden layer vector at the t-th time step.

[0224] Or,

[0225] In a possible implementation, the description word obtaining module 1040 is configured to input the semantic attention vector at the t-th time step, the visual attention vector at the t-th time step, and an output result of the attention fusion network at the (t-1)-th time step into the attention fusion network at the t-th time step, to obtain the output result of the attention fusion network at the t-th time step and the hidden layer vector at the t-th time step.

[0226] In a possible implementation, the attention fusion network is provided with a hyperparameter, where the hyperparameter is used to indicate a weight of the visual attention vector and the semantic attention vector in the attention fusion network.

[0227] In a possible implementation, the apparatus further includes:

[0228] The semantic attention vector generation module is configured to generate the semantic attention vector at the t-th time step based on the hidden layer vector at the (t-1)-th time step and the semantic feature set.

[0229] The semantic attention generation module includes:

[0230] The semantic feature weight obtaining submodule is configured to obtain a weight of each semantic feature in the semantic feature set at the (t-1)-th time step based on the hidden layer vector at the (t-1)-th time step and the semantic feature set.

[0231] The semantic attention vector generation submodule is configured to generate the semantic attention vector at the t-th time step based on the weight of each semantic feature in the semantic feature set at the (t-1)-th time step and the semantic feature set.

[0232] In a possible implementation, the apparatus further includes:

[0233] a visual attention generation module configured to generate, at the tth time step, the visual attention vector at the tth time step based on the hidden layer vector at the (t-1)th time step and the visual feature set.

[0234] In a possible implementation, the visual attention generation module comprises:

[0235] a semantic feature weight obtaining sub-module configured to obtain, based on the hidden layer vector at the (t-1)th time step and the visual feature set, a weight of each visual feature in the visual feature set at the (t-1)th time step;

[0236] a visual attention vector generation sub-module configured to generate, based on the weight of each visual feature in the visual feature set at the (t-1)th time step and the visual feature set, the visual attention vector at the tth time step.

[0237] In a possible implementation, the semantic feature extraction module 1020 comprises:

[0238] a semantic feature vector obtaining sub-module configured to obtain a semantic feature vector of the target image;

[0239] a semantic feature extraction sub-module configured to extract the semantic feature set of the target image based on the semantic feature vector.

[0240] In a possible implementation, the semantic feature extraction sub-module comprises:

[0241] an attribute word extraction unit configured to extract, based on the semantic feature vector, an attribute word set corresponding to the target image from a vocabulary library; the attribute word set refers to a set of candidate description words for describing the target image;

[0242] a semantic feature extraction unit configured to obtain, as the semantic feature set of the target image, a word vector set corresponding to the attribute word set.

[0243] In a possible implementation, the attribute word extraction unit is configured to obtain, based on the semantic feature vector, a matching probability of each vocabulary in the vocabulary library, where the matching probability refers to a probability that each vocabulary in the vocabulary library matches the target image.

[0244] extract, as the attribute word, a vocabulary in the vocabulary library whose matching probability is greater than a matching probability threshold.

[0245] In a possible implementation, the attribute word extraction unit is configured to input the semantic feature vector into a vocabulary detector, and obtain the set of attribute words extracted by the vocabulary detector from the vocabulary library based on the semantic feature vector.

[0246] The vocabulary detector is a vocabulary detection model trained by a weakly supervised method of multiple-instance learning.

[0247] In a possible implementation, before the visual feature extraction module extracts the set of visual features of the target image, the apparatus further includes:

[0248] a sub-region division module configured to divide the target image into at least one sub-region;

[0249] The visual feature extraction module is configured to extract visual features of the at least one sub-region respectively, and compose the set of visual features.

[0250] To sum up, the image description information generation apparatus provided by the embodiments of the present application extracts the set of semantic features and the set of visual features of the target image respectively, and uses the attention fusion network in the image description information generation model to realize the attention fusion of the semantic features and the visual features, so that the description word of the target image at the current time step is generated based on the comprehensive effect of the output result of the visual features and the semantic features of the target image at the previous time step at each time step of generating the image description information, and the image description information corresponding to the target image is further generated, so that the advantages of the visual features in generating visual words and the advantages of the semantic features in generating non-visual features are complementary in the generation process of the image description information, thereby improving the accuracy of generating the image description information.

[0251] Figure 11 A structural block diagram of a computer device 1100 is shown, which is an example of the computer device according to an embodiment of the present application. The computer device can be implemented as a server in the above-mentioned scheme. The computer device 1100 includes a central processing unit (CPU) 1101, a system memory 1104 including a random access memory (RAM) 1102 and a read-only memory (ROM) 1103, and a system bus 1105 connecting the system memory 1104 and the central processing unit 1101. The computer device 1100 further includes a mass storage device 1106 for storing an operating system 1109, application programs 1110 and other program modules 1111.

[0252] The mass storage device 1106 is connected to the central processing unit 1101 through a mass storage controller (not shown) connected to the system bus 1105. The mass storage device 1106 and its associated computer readable media provide nonvolatile storage for computer device 1100. That is, the mass storage device 1106 can include a computer readable medium (not shown) such as a hard disk or a Compact Disc Read-Only Memory (CD-ROM) drive.

[0253] Without loss of generality, the computer readable media can include computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes RAM, ROM, Erasable Programmable Read Only Memory (EPROM), Electrically-Erasable Programmable Read-Only memory (EEPROM), flash memory or other solid state memory technology, CD-ROM, Digital Versatile Disc (DVD), or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices. It should be understood by those skilled in the art that computer storage media does not limit to the above-mentioned several kinds. The system memory 1104 and the mass storage device 1106 mentioned above can be collectively referred to as memory.

[0254] According to various embodiments of the present disclosure, the computer device 1100 can also operate in connection with a remote computer through a network such as the Internet. That is, the computer device 1100 can connect to a network 1108 through a network interface unit 1107 connected to the system bus 1105, or in other words, the network interface unit 1107 can be used to connect to other types of networks or remote computer systems (not shown).

[0255] The memory also includes at least one instruction, at least one program, a code set or an instruction set, which is stored in the memory, and the central processing unit 1101 implements all or part of the steps of the image description information generation method shown in each embodiment by executing the at least one instruction, at least one program, code set or instruction set.

[0256] Figure 12A structural block diagram of a computer device 1200 provided by an example embodiment of the present application is shown. The computer device 1200 can be implemented as the face quality assessment device and / or the quality assessment model training device described above, such as a smartphone, a tablet computer, a notebook computer, or a desktop computer. The computer device 1200 can also be referred to as a user device, a portable terminal, a laptop terminal, a desktop terminal, and other names.

[0257] Generally, the computer device 1200 includes a processor 1201 and a memory 1202.

[0258] The processor 1201 can include one or more processing cores, such as a 4-core processor, a 12-core processor, and the like. The processor 1201 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), a PLA (Programmable Logic Array). The processor 1201 can also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also referred to as a CPU (Central Processing Unit). The coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 1201 can be integrated with a GPU (Graphics Processing Unit) for rendering and drawing content required to be displayed by a display screen. In some embodiments, the processor 1201 can further include an AI (Artificial Intelligence) processor for processing machine learning related computing operations.

[0259] The memory 1202 can include one or more computer-readable storage media, which can be non-transitory. The memory 1202 can also include a high-speed random access memory, and a non-volatile memory such as one or more disk storage devices, flash storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1202 is used to store at least one instruction for being executed by the processor 1201 to implement the image description information generation method provided by the method embodiments of the present application.

[0260] In some embodiments, the computer device 1200 can further include a peripheral device interface 1203 and at least one peripheral device. The processor 1201, the memory 1202 and the peripheral device interface 1203 can be connected through a bus or a signal line. Each peripheral device can be connected to the peripheral device interface 1203 through a bus, a signal line or a circuit board. Specifically, the peripheral device includes at least one of a radio frequency circuit 1204, a display screen 1205, a camera assembly 1206, an audio circuit 1207, a positioning assembly 1208 and a power supply 1209.

[0261] The peripheral device interface 1203 can be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1201 and the memory 1202. In some embodiments, the processor 1201, the memory 1202 and the peripheral device interface 1203 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1201, the memory 1202 and the peripheral device interface 1203 can be implemented on a separate chip or circuit board, and the present embodiment is not limited in this regard.

[0262] In some embodiments, the computer device 1200 further includes one or more sensors 1210. The one or more sensors 1210 include, but are not limited to, an acceleration sensor 1211, a gyroscope sensor 1212, a pressure sensor 1213, a fingerprint sensor 1214, an optical sensor 1215 and a proximity sensor 1216.

[0263] Those skilled in the art can understand that the structure shown in the above Figure 12 The structure shown in the above does not constitute a limitation on the computer device 1200, and can include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0264] In an exemplary embodiment, a computer readable storage medium is also provided for storing at least one instruction, at least one program, a code set or an instruction set, which is loaded and executed by a processor to implement all or part of the steps of the above image description information generation method. For example, the computer readable storage medium can be a Read-Only Memory (ROM), a Random Access Memory (RAM), a Compact Disc Read-Only Memory (CD-ROM), a magnetic tape, a floppy disk and an optical data storage device, etc.

[0265] In an exemplary embodiment, there is also provided a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to cause the computer device to perform all or part of the method of any of the embodiments described above. Figure 2 、 Figure 6 or Figure 7 any of the embodiments described above.

[0266] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the application encompass any and all variations or modifications of the application herein described and falling within the scope of the general inventive concept. It is to be understood that the application is not to be limited to the specific examples described and that many modifications, changes, and substitutions are intended within the foregoing disclosure and in the appended claims.

[0267] It is to be understood that the application is not limited to the precise details of design or construction described above and illustrated in the drawings. The scope of the application is to be limited only by the claims appended hereto.

Claims

1. A method for generating image description information, characterized in that, The method includes: Acquire the target image; Extract the semantic feature set of the target image; Extract the set of visual features from the target image; An attention fusion network in an image description information generation model performs attention fusion on the semantic and visual features of the target image at n time steps to obtain descriptive words corresponding to the n time steps. The input of the attention fusion network at time step t includes the semantic attention vector, the visual attention vector, and the output at time step (t-1). The semantic attention vector at time step t is obtained by processing the semantic feature set with an attention mechanism at time step t. The visual attention vector at time step t is obtained by processing the visual feature set with an attention mechanism at time step t. The output of the attention fusion network at time step (t-1) indicates the descriptive word corresponding to time step (t-1). Time step t is any one of the n time steps; 1 ≤ t ≤ n, and t and n are both positive integers. Based on the descriptive words of the target image at the n time steps, image description information of the target image is generated.

2. The method according to claim 1, characterized in that, The attention fusion network in the image description information generation model performs attention fusion on the semantic features and visual features of the target image at n time steps to obtain the descriptive words corresponding to the n time steps, including: At the t-th time step, the semantic attention vector, the visual attention vector, the hidden layer vector, and the output of the attention fusion network at the (t-1)-th time step are input into the attention fusion network to obtain the output of the attention fusion network at the t-th time step and the hidden layer vector at the t-th time step. or, At the t-th time step, the semantic attention vector at the t-th time step, the visual attention vector at the t-th time step, and the output result of the attention fusion network at the (t-1)-th time step are input into the attention fusion network to obtain the output result of the attention fusion network at the t-th time step and the hidden layer vector at the t-th time step.

3. The method according to claim 2, characterized in that, The attention fusion network is configured with hyperparameters that indicate the weights of the visual attention vector and the semantic attention vector in the attention fusion network.

4. The method according to claim 2, characterized in that, The method further includes: At the t-th time step, based on the hidden layer vector at the (t-1)-th time step and the semantic feature set, the semantic attention vector at the t-th time step is generated.

5. The method according to claim 4, characterized in that, The step of generating the semantic attention vector at the t-th time step based on the hidden layer vector at the (t-1)-th time step and the semantic feature set includes: Based on the hidden layer vector at the (t-1)th time step and the semantic feature set, obtain the weight of each semantic feature in the semantic feature set at the (t-1)th time step; Based on the weights of each semantic feature in the semantic feature set at the (t-1)th time step, and the semantic feature set, the semantic attention vector at the tth time step is generated.

6. The method according to claim 2, characterized in that, The method further includes: At the t-th time step, the visual attention vector at the t-th time step is generated based on the hidden layer vector at the (t-1)-th time step and the visual feature set.

7. The method according to claim 6, characterized in that, The step of generating the visual attention vector at the t-th time step based on the hidden layer vector at the (t-1)-th time step and the visual feature set includes: Based on the hidden layer vector at the (t-1)th time step and the visual feature set, obtain the weight of each visual feature in the visual feature set at the (t-1)th time step; Based on the weights of each visual feature in the visual feature set at the (t-1)th time step, and the visual feature set, the visual attention vector at the tth time step is generated.

8. The method according to claim 1, characterized in that, The extraction of the semantic feature set of the target image includes: Obtain the semantic feature vector of the target image; Based on the semantic feature vector, the semantic feature set of the target image is extracted.

9. The method according to claim 8, characterized in that, The step of extracting the semantic feature set of the target image based on the semantic feature vector includes: Based on the semantic feature vector, an attribute word set corresponding to the target image is extracted from the vocabulary; the attribute word set refers to a set of candidate descriptive words that describe the target image. The set of word vectors corresponding to the set of attribute words is obtained as the set of semantic features of the target image.

10. The method according to claim 9, characterized in that, The step of extracting attribute words corresponding to the target image from the vocabulary based on the semantic feature vector includes: Based on the semantic feature vector, the matching probability of each word in the vocabulary is obtained, where the matching probability refers to the probability that each word in the vocabulary matches the target image. Extract words from the vocabulary that have a matching probability greater than a matching probability threshold, and use them as attribute words.

11. The method according to claim 9, characterized in that, The step of extracting attribute words corresponding to the target image from the vocabulary based on the semantic feature vector includes: The semantic feature vector is input into the vocabulary detector to obtain the set of attribute words extracted by the vocabulary detector from the vocabulary based on the semantic feature vector; The vocabulary detector is a vocabulary detection model trained using a weakly supervised method of multi-instance learning.

12. The method according to claim 1, characterized in that, Before extracting the set of visual features from the target image, the method further includes: The target image is divided into sub-regions to obtain at least one sub-region; The extraction of the visual feature set of the target image includes: Visual features of each of the at least one sub-region are extracted to form the visual feature set.

13. An image description information generation device, characterized in that, The device includes: The target image acquisition module is used to acquire the target image; The semantic feature extraction module extracts a set of semantic features from the target image. A visual feature extraction module is used to extract a set of visual features from the target image; The descriptor acquisition module is used to generate an attention fusion network in the model using image description information. It performs attention fusion on the semantic features and visual features of the target image at n time steps to obtain the descriptors corresponding to the n time steps. The input of the attention fusion network at time step t includes the semantic attention vector at time step t, the visual attention vector at time step t, and the output of the attention fusion network at time step (t-1). The semantic attention vector at time step t is obtained by processing the semantic feature set with an attention mechanism at time step t. The visual attention vector at time step t is obtained by processing the visual feature set with an attention mechanism at time step t. The output of the attention fusion network at time step (t-1) indicates the descriptor corresponding to time step (t-1). The t-th time step is any one of the n time steps; 1 ≤ t ≤ n, and t and n are both positive integers. The image description information generation module is used to generate image description information of the target image based on the descriptive words of the target image at the n time steps.

14. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one instruction, at least one program, a code set, or an instruction set, the at least one instruction, the at least one program, the code set, or the instruction set being loaded and executed by the processor to implement the image description information generation method as described in any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to implement the image description information generation method as described in any one of claims 1 to 12.

Images