Method, device, storage medium and processor for generating text information based on images

Abstract

Embodiments of the present application provide a method and device for generating text information based on an image, a storage medium and a processor, and belong to the field of image-text mutual interpretation in computer vision. The method comprises: obtaining a to-be-processed picture and a sample image dataset, and determining similar images in the sample image dataset; performing feature extraction on the to-be-processed picture to obtain regional features of the to-be-processed picture; filtering and optimizing the regional features of the to-be-processed picture to obtain subject regional features of the to-be-processed picture; performing feature extraction on the similar images in the sample image dataset to obtain a plurality of similar image features; obtaining first fusion features by using a weighted mean method based on the plurality of similar image features; fusing the first fusion features to the subject regional features of the to-be-processed picture by using a splicing method to obtain second fusion features; and inputting the second fusion features into a fine-grained multi-layer long short-term memory network to generate text information corresponding to the to-be-processed picture.

Description

Technical Field

[0001] This invention relates to the field of image-text translation in computer vision, specifically to a method for generating text information based on images, an apparatus for generating text information based on images, a machine-readable storage medium, and a processor. Background Technology

[0002] Image-to-text generation has been a widely studied task for many years. It involves understanding images through computer vision and then generating descriptive language for the images using natural language processing. This task has a wide range of applications, from navigation for the blind and children's education to intelligent understanding of surveillance video content. In recent years, with the development of deep learning, research on image-based text generation has made significant progress and has become a hot research area in computer vision.

[0003] Chinese Patent Publication No. CN110555332A discloses a method, electronic device, and storage medium for generating text information based on an image. The method includes: acquiring first image feature information of an image to be processed, and text attribute information corresponding to at least one text attribute; and determining the text information corresponding to the image to be processed based on the first image feature information and the text attribute information. This solution combines the image feature information and the text attribute information of the image to be processed, achieving the generation of text information corresponding to the image based on multi-dimensional features, effectively improving the accuracy of text information generated from images.

[0004] However, this patent does not adequately address the lack of controllability and interpretability in current image-to-text generation methods; their behavior is difficult to influence and interpret based on user preferences. Existing network architectures implicitly select which regions in the image to focus on at each time step. This architecture cannot control which regions need to be described or the importance of each region, leading to incorrect descriptions of some image regions and the generation of text that does not meet user needs. The fineness and accuracy of image-to-text generation by existing network architectures need improvement. Summary of the Invention

[0005] The purpose of this application is to provide a method, apparatus, storage medium, and processor for generating text information based on images. The method can solve the problem that existing networks incorrectly describe some image regions, thereby generating text that does not meet people's needs.

[0006] To achieve the above objectives, the first aspect of this application provides a method for generating text information based on an image, the method comprising:

[0007] Obtain the image to be processed and the sample image dataset, and determine similar images in the sample image dataset based on the image to be processed and the sample image dataset; wherein, the sample image dataset comes from an image dataset that has been labeled with text labels;

[0008] Feature extraction is performed on the image to be processed to obtain the regional features of the image;

[0009] The text control signal is used to filter and select the regional features of the image to be processed, so as to obtain the main regional features of the image to be processed.

[0010] A convolutional neural network is used to extract features from similar images in the sample image dataset to obtain multiple similar image features;

[0011] Based on the features of the multiple similar images, a first fusion feature is obtained by weighted averaging.

[0012] The first fusion feature is fused into the main region feature of the image to be processed using a splicing method to obtain the second fusion feature;

[0013] The second fusion feature is input into a fine-grained multilayer long short-term memory network to generate text information corresponding to the image to be processed.

[0014] In this embodiment of the application, feature extraction of the image to be processed to obtain regional features of the image to be processed includes: using an object detection algorithm to extract features of the image to be processed to obtain regional features of the image to be processed.

[0015] In this embodiment of the application, the step of using text control signals to filter and select the regional features of the image to be processed to obtain the main regional features of the image to be processed includes:

[0016] The mean hash algorithm is used to calculate the similarity between the image to be processed and the sample image dataset, and the similar images in the sample image dataset and the text labels corresponding to the similar images are determined based on the similarity.

[0017] Calculate the probability of words appearing in the text tags corresponding to the similar images;

[0018] Words with a frequency of at least a preset frequency appearing in the text tags corresponding to similar images are used as text control signals to filter and select the best regional features of the image to be processed, thereby obtaining the main regional features of the image to be processed.

[0019] In this embodiment of the application, determining similar images in the sample image dataset based on the image to be processed and the sample image dataset includes:

[0020] The similarity between the image to be processed and the sample image dataset is calculated using the mean hash algorithm.

[0021] Based on the similarity level, images with a similarity level of not less than a preset threshold in the sample image dataset are selected as similar images.

[0022] In this embodiment of the application, the first fusion feature obtained by weighted averaging based on the multiple similar image features is calculated according to the following formula:

[0023]

[0024] Among them, v i Represents similar image features; This represents the first fusion feature obtained using the weighted average method.

[0025] In this embodiment of the application, the second fused feature is input into a fine-grained multilayer long short-term memory network to generate text information corresponding to the image to be processed, including:

[0026] The second fused feature is input into the first layer of the long short-term memory network in the fine-grained multilayer long short-term memory network for feature weighting to obtain the context vector;

[0027] The context vector is input into the second layer of the fine-grained multilayer long short-term memory network for feature decoding, and the probability of the currently generated word is output.

[0028] The word with the highest probability is selected as the text information corresponding to the image to be processed.

[0029] In this embodiment, the first layer of long short-term memory network includes a first long short-term memory network; the second layer of long short-term memory network includes a second long short-term memory network and a third long short-term memory network; the context vector includes visual information c. t and complete target object region information b t ;

[0030] The context vector is input into the second layer of the fine-grained multilayer long short-term memory network for feature decoding, and the probability of the currently generated word is output, including:

[0031] The visual information c of the context vector t The input is processed by the second long short-term memory network for feature decoding, and the complete target object region information b from the context vector is obtained. t Input the third long short-term memory network for feature decoding and output the probability of the currently generated word.

[0032] A second aspect of this application provides an apparatus for generating text information based on an image, the apparatus comprising:

[0033] The acquisition module is used to acquire the image to be processed and the sample image dataset, and to determine similar images in the sample image dataset based on the image to be processed and the sample image dataset; wherein, the sample image dataset comes from an image dataset that has been labeled with text labels;

[0034] The first feature extraction module is used to extract features from the image to be processed to obtain the regional features of the image to be processed;

[0035] The filtering module is used to filter and select the regional features of the image to be processed using text control signals to obtain the main regional features of the image to be processed.

[0036] The second feature extraction module is used to extract features from similar images in the sample image dataset using a convolutional neural network to obtain multiple similar image features.

[0037] The weighting module is used to obtain the first fusion feature based on the multiple similar image features by using a weighted average method;

[0038] The stitching module is used to stitch the first fusion feature into the main region feature of the image to be processed to obtain the second fusion feature;

[0039] The generation module is used to input the second fused features into a fine-grained multilayer long short-term memory network to generate text information corresponding to the image to be processed.

[0040] A third aspect of this application provides a processor configured to perform the above-described method for generating text information based on an image.

[0041] A fourth aspect of this application provides a machine-readable storage medium storing instructions that, when executed by a processor, configure the processor to perform the above-described method for generating text information based on an image.

[0042] Compared with the prior art, the above-mentioned technical solution of the present invention has the following beneficial effects:

[0043] (1) The present invention uses a feature extraction method for similar images and obtains a fusion feature by weighting the multiple features. The aim is to use the text labels and image feature information of similar images at the same time to give the text generation system the subject and object region features of the image, thereby enhancing the accuracy of its output text.

[0044] (2) This invention uses a fine-grained multilayer long short-term memory network structure as a text generation system to generate more accurate and refined text. This network structure expands the dimensions based on the traditional two-layer long short-term memory network, and simultaneously receives visual information and complete target object region information when outputting words, thereby enabling the text generation system to acquire more image information and further improving the refinement and accuracy of the generated text.

[0045] Other features and advantages of the embodiments of this application will be described in detail in the following detailed description section. Attached Figure Description

[0046] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings:

[0047] Figure 1 This illustration schematically shows an application environment diagram of the method for generating text information based on an image according to an embodiment of this application;

[0048] Figure 2 The illustration shows a flowchart of a method for generating text information based on an image according to an embodiment of this application;

[0049] Figure 3 This illustration schematically shows a network diagram for generating text information based on an image according to an embodiment of this application;

[0050] Figure 4 This schematic diagram illustrates a structural block diagram of an apparatus for generating text information based on an image according to an embodiment of the present application;

[0051] Figure 5 The diagram illustrates the internal structure of a computer device according to an embodiment of this application. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of this application and are not intended to limit the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0053] It should be noted that if the embodiments of this application involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" or "second" may explicitly or implicitly include at least one of those features. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0054] The method for generating text information based on images provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices, and server 104 can be a standalone server or a server cluster consisting of multiple servers.

[0055] Figure 2 The illustration shows a flowchart of a method for generating text information based on an image according to an embodiment of this application. Figure 2 As shown, in one embodiment of this application, a method for generating text information based on an image is provided. This embodiment mainly applies this method to the above-mentioned... Figure 1 Taking terminal 102 (or server 104) as an example, the following steps are included:

[0056] Step 110: Obtain the image to be processed and the sample image dataset, and determine similar images in the sample image dataset based on the image to be processed and the sample image dataset; wherein, the sample image dataset comes from an image dataset that has been labeled with text labels.

[0057] In this embodiment, the image to be processed is the image for which a text description needs to be generated, i.e., the input image. The sample image dataset comes from the open-source COCO dataset, which contains hundreds of thousands of images, many of which are of the same type as the image to be processed.

[0058] Step 120: Extract features from the image to be processed to obtain the regional features of the image to be processed.

[0059] In this embodiment, an object detection algorithm is used to extract features from the image to be processed, obtaining regional features of the image. For example, the object detection algorithm is used to extract features from the image to be processed, obtaining a maximum of 32 image region features:

[0060] R'=(r0,r1,...,r N (1);

[0061] Where, r N R' represents the features of the Nth image region, and R' is the set of features of all regions in the image to be processed extracted by the object detection algorithm.

[0062] The target detection algorithm used in this embodiment is: Faster Regions with CNN features (Faster R-CNN).

[0063] Step 130: Use text control signals to filter and select the regional features of the image to be processed to obtain the main regional features of the image to be processed.

[0064] R'=(r0,r1,...,r N → R = (r0, r1, ..., r K (2);

[0065] Where, r N The feature r represents the Nth image region. K R represents the features of the Kth image region, and R is the main region feature of the image to be processed, where K ≤ N.

[0066] In this embodiment, the text control signal originates from the text tags corresponding to similar images. The similarity between the image to be processed and the sample image dataset is calculated using a mean hash algorithm, obtaining the similarity between the images in the sample image dataset and the input image. The K images with the highest similarity and their corresponding text tags are selected, and the probability of word occurrence in the text tags corresponding to the K images with the highest similarity is calculated. Words with a frequency not lower than a preset frequency in the text tags corresponding to similar images are used as text control signals to filter and optimize the regional features of the image to be processed, obtaining the main regional features of the image to be processed.

[0067] In this embodiment, the selection of the image region K is crucial. If K is too small, it cannot describe the entire image; if K is too large, the model will generate cumbersome image text. Through comparative experiments, this invention selects K=3 as the optimal value for the parameter.

[0068] Step 140: Use a convolutional neural network to extract features from similar images in the sample image dataset to obtain multiple similar image features.

[0069] Step 150: Based on the multiple similar image features, the first fusion feature is obtained by weighted averaging.

[0070] In this embodiment, the first fusion feature is calculated according to formula (3):

[0071]

[0072] Among them, v i Represents similar image features; This represents the first fusion feature obtained using the weighted average method.

[0073] Step 160: The first fusion feature is fused into the main region feature of the image to be processed using a stitching method to obtain the second fusion feature.

[0074] In this embodiment, according to formula (4), the first fusion feature is fused to the main region feature of the image to be processed using a splicing method to obtain the second fusion feature:

[0075]

[0076] in, Represents the first fusion feature; R * This indicates the second fusion feature.

[0077] In this embodiment, step 140 performs convolutional extraction on the features of similar images in the sample image dataset in step 110 to obtain multiple similar image features. Step 150 uses a weighted average to obtain a first fusion feature, and then uses a splicing operation to fuse the first fusion feature into the main region features of the image to be processed obtained in step 130, finally obtaining a second fusion feature.

[0078] Step 170: Input the second fused feature into a fine-grained multilayer long short-term memory network to generate text information corresponding to the image to be processed.

[0079] In this embodiment, as Figure 3 As shown, the fine-grained multilayer long short-term memory network includes two layers of long short-term memory networks: a first-layer long short-term memory network and a second-layer long short-term memory network. The first-layer long short-term memory network includes a first long short-term memory network 1; the second-layer long short-term memory network includes a second long short-term memory network 2 and a third long short-term memory network 3.

[0080] In this embodiment, the second fused feature is input into the first layer of the fine-grained multilayer long short-term memory network for feature weighting to obtain a context vector; the context vector is input into the second layer of the fine-grained multilayer long short-term memory network for feature decoding to output the probability of the currently generated word; the word with the highest probability is selected as the text information corresponding to the image to be processed.

[0081] In this embodiment, as Figure 3 As shown, the fine-grained multilayer long short-term memory network consists of two layers: the first layer is a long short-term memory network (first long short-term memory network 1) that generates visual information c. t and complete target object region information b t c t Used for the generation of visual words, b t Simultaneously, information from multiple semantically consistent spatial regions is aggregated. By combining these visual regions, a visual region completely surrounds the target object is formed. The second layer consists of two long short-term memory networks (LSM2 and LSM3) that simultaneously receive visual information and complete target object region information when outputting words. This allows the text generation system to acquire more image information, further improving the refinement and accuracy of the generated text.

[0082] For example, the second fusion feature from step 160 is used as the input to the first long short-term memory layer in the fine-grained multilayer long short-term memory network, and feature weighting is performed to obtain the context vector c. t b t :

[0083]

[0084]

[0085] in, yes and r t Compatibility function between regions It is a hidden layer in the first layer of the Long Short-Term Memory network, r t It is R * The feature vector in It's a sentinel used to generate noun blocks in the text. W ss W g It is a learnable parameter, α t These are weighting coefficients.

[0086]

[0087] b t =β t R * (8);

[0088] Among them, R * It is the second fusion feature; These are the hidden states of the first Long Short-Term Memory network 1, which generate a context vector c through an attention mechanism. t b t .

[0089] Then, the context vector c t b t As input to the second layer of the fine-grained multilayer long short-term memory network, it is used for feature decoding, and finally outputs the word probability of the currently generated word:

[0090]

[0091]

[0092] p=λp1+(1-λ)p2 (11);

[0093] in, This is the current hidden state of the second long short-term memory network 2, b p For biasing the fully connected layer; This is the current hidden state of the third Long Short-Term Memory network, b q The bias of the fully connected layer is such that the sum of the probabilities of generating words is 1. To satisfy this constraint, normalization is required. The conditional probability distribution output by the second layer of the long short-term memory network is integrated through the λ parameter (λ is set to 0.5 in this paper). Finally, the word probability p of the currently generated word is output, and the word with the highest probability is selected as the text information corresponding to the image to be processed.

[0094] Figure 2 This is a flowchart illustrating a method for generating text information based on an image in one embodiment. It should be understood that, although... Figure 2 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 2 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0095] Figure 3 This illustration shows a network diagram illustrating the generation of text information based on an image according to an embodiment of this application. For example... Figure 3As shown, this embodiment of the application presents a fine-grained image-to-text network based on text control signals, comprising an input image, an image feature extraction system, and a text generation system. The input image provides the image for which text descriptions need to be generated, and is directly connected to the image feature extraction system. The image feature extraction system includes: a feature extraction unit, a text-based control signal unit, and a similar image feature guidance unit. The feature extraction unit extracts all region features from the input image and is directly connected to the text-based control signal unit and the similar image feature guidance unit. The text-based control signal unit uses text control signals (text labels corresponding to similar images) to select features of a set of important regions to obtain features of the main body region in the input image. The similar image feature guidance unit uses guidance information from similar image features to obtain features of the object and background regions in the input image. Simultaneously, these two units are used to assign subject and object region features to the text generation system, thereby generating text that better matches the image content. The text generation system includes a fine-grained multilayer long short-term memory network, which uses the image features output by the image feature extraction system as input to generate text describing the image. The fine-grained multilayer long short-term memory network consists of two layers of long short-term memory network units. The first layer of long short-term memory network units processes image features, generating visual information and complete target object region information respectively. The second layer of long short-term memory network units consists of two long short-term memory networks, receiving the visual information and complete target object region information input from the first layer of long short-term memory network units respectively, thereby generating the final text describing the image.

[0096] In one embodiment, such as Figure 4 As shown, an apparatus for generating text information based on an image is provided, including an acquisition module 210, a first feature extraction module 220, a filtering module 230, a second feature extraction module 240, a weighting module 250, a splicing module 260, and a generation module 270, wherein:

[0097] The acquisition module 210 is used to acquire the image to be processed and the sample image dataset, and to determine similar images in the sample image dataset based on the image to be processed and the sample image dataset; wherein, the sample image dataset comes from an image dataset that has been labeled with text labels.

[0098] The first feature extraction module 220 is used to extract features from the image to be processed to obtain the regional features of the image to be processed;

[0099] The filtering module 230 is used to filter and select the regional features of the image to be processed using text control signals to obtain the main regional features of the image to be processed.

[0100] The second feature extraction module 240 is used to extract features from similar images in the sample image dataset using a convolutional neural network to obtain multiple similar image features.

[0101] The weighting module 250 is used to obtain the first fusion feature based on the multiple similar image features by using a weighted average method.

[0102] The stitching module 260 is used to stitch the first fusion feature into the main region feature of the image to be processed to obtain the second fusion feature.

[0103] The generation module 270 is used to input the second fused features into a fine-grained multilayer long short-term memory network to generate text information corresponding to the image to be processed.

[0104] The device for generating text information based on images includes a processor and a memory. The aforementioned acquisition module 210, first feature extraction module 220, filtering module 230, second feature extraction module 240, weighting module 250, splicing module 260, and generation module 270 are all stored in the memory as program units. The processor executes the aforementioned program modules stored in the memory to implement the corresponding functions.

[0105] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and methods for generating text information based on images can be implemented by adjusting kernel parameters.

[0106] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0107] This application provides a storage medium storing a program that, when executed by a processor, implements the above-described method for generating text information based on an image.

[0108] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 5As shown in the figure, the computer device includes a processor A01, a network interface A02, a display screen A04, an input device A05, and a memory (not shown) connected via a system bus. The processor A01 provides computing and control capabilities. The memory includes internal memory A03 and a non-volatile storage medium A06. The non-volatile storage medium A06 stores an operating system B01 and a computer program B02. The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 stored in the non-volatile storage medium A06. The network interface A02 is used for communication with external terminals via a network connection. When the computer program is executed by the processor A01, it implements a method for generating text information based on images. The display screen A04 can be a liquid crystal display (LCD) or an e-ink display. The input device A05 can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0109] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0110] In one embodiment, the image-based text information generation device provided in this application can be implemented as a computer program, and the computer program can be implemented as follows: Figure 5 The device runs on the computer shown. The computer device's memory can store the various program modules that make up the image-based text information generation device, for example, Figure 4 The diagram shows an acquisition module 210, a first feature extraction module 220, a filtering module 230, a second feature extraction module 240, a weighting module 250, a splicing module 260, and a generation module 270. The computer program comprised of these modules causes a processor to execute the steps of the image-based text information generation methods described in the various embodiments of this application.

[0111] Figure 5 The computer device shown can be used as follows Figure 4The image-based text information generation device shown executes step 110 via acquisition module 210. A computer device can execute step 120 via first feature extraction module 220. A computer device can execute step 130 via filtering module 230. A computer device can execute step 140 via second feature extraction module 240. A computer device can execute step 150 via weighting module 250. A computer device can execute step 160 via splicing module 260. A computer device can execute step 170 via generation module 270.

[0112] This application provides a device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs the following steps:

[0113] Step 110: Obtain the image to be processed and the sample image dataset, and determine similar images in the sample image dataset based on the image to be processed and the sample image dataset; wherein, the sample image dataset comes from an image dataset that has been labeled with text labels.

[0114] Step 120: Extract features from the image to be processed to obtain the regional features of the image to be processed.

[0115] Step 130: Use text control signals to filter and select the regional features of the image to be processed to obtain the main regional features of the image to be processed.

[0116] Step 140: Use a convolutional neural network to extract features from similar images in the sample image dataset to obtain multiple similar image features.

[0117] Step 150: Based on the multiple similar image features, the first fusion feature is obtained by weighted averaging.

[0118] Step 160: The first fusion feature is fused into the main region feature of the image to be processed using a stitching method to obtain the second fusion feature.

[0119] Step 170: Input the second fused feature into a fine-grained multilayer long short-term memory network to generate text information corresponding to the image to be processed.

[0120] In one embodiment, feature extraction of the image to be processed to obtain regional features of the image to be processed includes: using an object detection algorithm to extract features of the image to be processed to obtain regional features of the image to be processed.

[0121] In one embodiment, the step of using text control signals to filter and optimize the regional features of the image to be processed to obtain the main regional features of the image to be processed includes:

[0122] The mean hash algorithm is used to calculate the similarity between the image to be processed and the sample image dataset, and the similar images in the sample image dataset and the text labels corresponding to the similar images are determined based on the similarity.

[0123] Calculate the probability of words appearing in the text tags corresponding to the similar images;

[0124] Words with a frequency of at least a preset frequency appearing in the text tags corresponding to similar images are used as text control signals to filter and select the best regional features of the image to be processed, thereby obtaining the main regional features of the image to be processed.

[0125] In one embodiment, determining similar images in the sample image dataset based on the image to be processed and the sample image dataset includes:

[0126] The similarity between the image to be processed and the sample image dataset is calculated using the mean hash algorithm.

[0127] Based on the similarity level, images with a similarity level of not less than a preset threshold in the sample image dataset are selected as similar images.

[0128] In one embodiment, the first fusion feature, obtained by weighted averaging based on the multiple similar image features, is calculated according to the following formula:

[0129]

[0130] Among them, v i Represents similar image features; This represents the first fusion feature obtained using the weighted average method.

[0131] In one embodiment, the second fused feature is input into a fine-grained multilayer long short-term memory network to generate text information corresponding to the image to be processed, including:

[0132] The second fused feature is input into the first layer of the long short-term memory network in the fine-grained multilayer long short-term memory network for feature weighting to obtain the context vector;

[0133] The context vector is input into the second layer of the fine-grained multilayer long short-term memory network for feature decoding, and the probability of the currently generated word is output.

[0134] The word with the highest probability is selected as the text information corresponding to the image to be processed.

[0135] In one embodiment, the first layer of long short-term memory network includes a first long short-term memory network; the second layer of long short-term memory network includes a second long short-term memory network and a third long short-term memory network; the context vector includes visual information c. t and complete target object region information b t ;

[0136] The context vector is input into the second layer of the fine-grained multilayer long short-term memory network for feature decoding, and the probability of the currently generated word is output, including:

[0137] The visual information c of the context vector t The input is processed by the second long short-term memory network for feature decoding, and the complete target object region information b from the context vector is obtained. t Input the third long short-term memory network for feature decoding and output the probability of the currently generated word.

[0138] This application's embodiments utilize an image feature extraction system to address the problem of existing networks incorrectly describing certain regions, thus generating text that does not meet user needs. Simultaneously, a text generation system can generate more accurate, refined text that better matches the image content. From navigation for the blind and children's education to intelligent understanding of surveillance video content, this invention has a wide range of applications in the future.

[0139] This application uses a fine-grained multilayer long short-term memory network to generate text describing an image. The first layer of the long short-term memory network processes image features, generating visual information and complete target object region information respectively. The second layer of the long short-term memory network consists of two long short-term memory networks, which respectively receive the visual information and complete target object region information input from the first layer of the long short-term memory network, thereby generating the final text describing the image.

[0140] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0141] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0142] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0143] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0144] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0145] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0146] Computer-readable media include both permanent and non-permanent, removable and non-removable media, which can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0147] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0148] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for generating text information based on images, characterized in that, The method includes: Obtain the image to be processed and the sample image dataset, and determine similar images in the sample image dataset based on the image to be processed and the sample image dataset; wherein, the sample image dataset comes from an image dataset that has been labeled with text labels; Feature extraction is performed on the image to be processed to obtain the regional features of the image to be processed, wherein the regional features are the features of multiple image regions in the image to be processed. The text control signal is used to filter and select the regional features of the image to be processed, so as to obtain the main regional features of the image to be processed. A convolutional neural network is used to extract features from similar images in the sample image dataset to obtain multiple similar image features; Based on the features of the multiple similar images, a first fusion feature is obtained by weighted averaging. The first fusion feature is fused into the main region feature of the image to be processed using a splicing method to obtain the second fusion feature; The second fused feature is input into a fine-grained multilayer long short-term memory network to generate text information corresponding to the image to be processed, including: The second fused feature is input into the first layer of the fine-grained multilayer long short-term memory network for feature weighting to obtain a context vector, which includes visual information. and complete target object region information ; The context vector is input into the second layer of the fine-grained multilayer long short-term memory network for feature decoding, and the probability of the currently generated word is output. The word with the highest probability is selected as the text information corresponding to the image to be processed; The context vector is input into the second layer of the fine-grained multilayer long short-term memory network for feature decoding, and the probability of the currently generated word is output, including: Visual information of the context vector The input is processed by a second long short-term memory network for feature decoding, and the complete target object region information of the context vector is obtained. Input the third long short-term memory network for feature decoding, integrate the two conditional probability distributions obtained from the decoding, and output the probability of the currently generated word.

2. The method for generating text information based on an image according to claim 1, characterized in that, Feature extraction is performed on the image to be processed to obtain the regional features of the image, including: The image to be processed is used to extract features using an object detection algorithm to obtain the regional features of the image.

3. The method for generating text information based on an image according to claim 1, characterized in that, The step of using text control signals to filter and select the regional features of the image to be processed to obtain the main regional features of the image to be processed includes: The mean hash algorithm is used to calculate the similarity between the image to be processed and the sample image dataset, and the similar images in the sample image dataset and the text labels corresponding to the similar images are determined based on the similarity. Calculate the probability of words appearing in the text tags corresponding to the similar images; Words with a frequency of at least a preset frequency appearing in the text tags corresponding to similar images are used as text control signals to filter and select the best regional features of the image to be processed, thereby obtaining the main regional features of the image to be processed.

4. The method for generating text information based on an image according to claim 1, characterized in that, The step of determining similar images in the sample image dataset based on the image to be processed and the sample image dataset includes: The similarity between the image to be processed and the sample image dataset is calculated using the mean hash algorithm. Based on the similarity level, images with a similarity level of not less than a preset threshold in the sample image dataset are selected as similar images.

5. The method for generating text information based on an image according to claim 1, characterized in that, The first fusion feature, obtained based on the multiple similar image features, is calculated using a weighted average method according to the following formula: ； in, Represents similar image features; This represents the first fusion feature obtained using the weighted average method.

6. An apparatus for generating text information based on an image, characterized in that, The device includes: The acquisition module is used to acquire the image to be processed and the sample image dataset, and to determine similar images in the sample image dataset based on the image to be processed and the sample image dataset; wherein, the sample image dataset comes from an image dataset that has been labeled with text labels; The first feature extraction module is used to extract features from the image to be processed to obtain the regional features of the image to be processed, wherein the regional features are the features of multiple image regions in the image to be processed. The filtering module is used to filter and select the regional features of the image to be processed using text control signals to obtain the main regional features of the image to be processed. The second feature extraction module is used to extract features from similar images in the sample image dataset using a convolutional neural network to obtain multiple similar image features. The weighting module is used to obtain the first fusion feature based on the multiple similar image features by using a weighted average method; The stitching module is used to stitch the first fusion feature into the main region feature of the image to be processed to obtain the second fusion feature; The generation module is used to input the second fused features into a fine-grained multilayer long short-term memory network to generate text information corresponding to the image to be processed, including: The second fused feature is input into the first layer of the fine-grained multilayer long short-term memory network for feature weighting to obtain a context vector, which includes visual information. and complete target object region information ; The context vector is input into the second layer of the fine-grained multilayer long short-term memory network for feature decoding, and the probability of the currently generated word is output. The word with the highest probability is selected as the text information corresponding to the image to be processed; The context vector is input into the second layer of the fine-grained multilayer long short-term memory network for feature decoding, and the probability of the currently generated word is output, including: Visual information of the context vector The input is processed by a second long short-term memory network for feature decoding, and the complete target object region information of the context vector is obtained. Input the third long short-term memory network for feature decoding, integrate the two conditional probability distributions obtained from the decoding, and output the probability of the currently generated word.

7. A processor, characterized in that, It is configured to perform the method of generating text information based on an image according to any one of claims 1 to 5.

8. A machine-readable storage medium storing instructions thereon, characterized in that, When executed by a processor, this instruction causes the processor to be configured to perform the method of generating text information based on an image according to any one of claims 1 to 5.

Images