Methods, apparatuses, electronic devices, and storage media for generating image-descriptive text

Abstract

This disclosure provides a method for generating image descriptive text, comprising: processing the grid features of a target image to construct multiple pseudo-region features of the target image, wherein the pseudo-region features are used to characterize the local visual information of the target image; performing encoding enhancement processing on the grid features and pseudo-region features respectively to obtain grid enhancement features fused with multi-layer image encoding results and pseudo-region enhancement features fused with multi-layer local encoding results; and performing semantic parsing on the grid enhancement features and pseudo-region enhancement features to generate image descriptive text for characterizing the visual information in the target image. This disclosure also provides an image descriptive text generation apparatus, electronic device, and storage medium.

Description

Technical Field

[0001] This disclosure relates to the field of computer science and technology, and in particular to a method, apparatus, electronic device, and storage medium for generating image description text. Background Technology

[0002] With the rapid development of multimedia technology and computer networks, the multimodal nature of data has become increasingly prominent, and the volume of data has exploded. The demand for multimodal data processing from academia and industry is also becoming increasingly urgent. On the other hand, the increasing maturity of deep learning technology provides strong technical support for processing multimodal data. With the arrival of the big data era, massive amounts of image data are generated every moment, such as photos taken by mobile phones and surveillance images captured by security cameras. However, the utilization rate of image data is declining. Therefore, how to automatically construct descriptive text based on human language using this ever-increasing image data, such as generating image captions and image retrieval, has become an urgent problem to be solved.

[0003] In related technologies, image description methods typically employ region features extracted by object detectors as visual representations. Since these detectors are pre-trained on large-scale visual genome datasets, they can generate discriminative representations of salient regions in images and provide complete object information for description. However, region features, extracted from salient regions of an image, have significant drawbacks. For example, they often ignore contextual information within the background. In this case, the trained model struggles to capture relationships between objects; furthermore, region feature extraction is time-consuming. Additionally, pre-trained object detectors may produce noise, overlap, or false detections, ultimately limiting the performance ceiling of text construction models.

[0004] Grid features from image data have achieved impressive performance in constructing descriptive text; however, directly manipulating grid features in a flat manner inevitably disrupts the spatial relationships between grids. Combining region features and grid features from image data as visual input increases the computational cost of the model; the fusion process is complex; and issues such as redundant noise can arise. Summary of the Invention

[0005] To address at least one of the aforementioned technical problems, this disclosure provides a method, apparatus, electronic device, and storage medium for generating image description text.

[0006] According to one aspect of this disclosure, a method for generating image descriptive text is proposed, comprising: processing grid features of a target image to construct multiple pseudo-region features of the target image, wherein the pseudo-region features are used to characterize local visual information of the target image; performing encoding enhancement processing on the grid features and the pseudo-region features respectively to obtain grid enhancement features fused with multi-layer image encoding results and pseudo-region enhancement features fused with multi-layer local encoding results; and performing semantic parsing on the grid enhancement features and the pseudo-region enhancement features to generate image descriptive text for characterizing visual information in the target image.

[0007] In some implementations, processing the grid features of the target image to construct multiple pseudo-region features of the target image includes: pre-setting multiple feature clusters; calculating the similarity between each grid feature and the feature cluster, so as to map each grid feature to the feature cluster with the highest similarity to itself, wherein grid features in the same feature cluster represent the same visual information; and normalizing multiple grid features in the same feature cluster to obtain multiple pseudo-region features for representing each feature cluster.

[0008] In some implementations, the step of encoding and enhancing the grid features and the pseudo-region features respectively to obtain grid-enhanced features fused with multi-layer image encoding results and pseudo-region-enhanced features fused with multi-layer local encoding results includes: calling a grid encoder to encode the grid features to obtain the hierarchical features generated by each encoding layer in the grid encoder and the initial grid features output by the grid encoder; connecting the hierarchical features generated by each encoding layer in the grid encoder to construct multi-layer grid features; and fusing the initial grid features and the multi-layer grid features according to a certain ratio to construct the grid-enhanced features.

[0009] In some embodiments, the step of encoding and enhancing the grid features and the pseudo-region features respectively to obtain grid-enhanced features fused with multi-layer image encoding results and pseudo-region-enhanced features fused with multi-layer local encoding results includes: calling a pseudo-region encoder to encode the pseudo-region features to obtain the hierarchical features generated by each encoding layer in the pseudo-region encoder and the initial pseudo-region features output by the pseudo-region encoder; concatenating the hierarchical features generated by each encoding layer in the pseudo-region encoder to construct multi-layer pseudo-region features; and fusing the initial pseudo-region features and the multi-layer pseudo-region features in a proportional manner to construct the pseudo-region-enhanced features.

[0010] In some implementations, the step of semantically parsing the mesh enhancement features and the pseudo-region enhancement features to generate image description text for characterizing visual information in the target image includes: concatenating the mesh enhancement features and the pseudo-region enhancement features to generate a feature string; and calling a text decoder to parse the feature string and the image annotation of the target image to obtain the image description text corresponding to the feature string, wherein the image description text is used to characterize visual information in the target image.

[0011] In some implementations, before processing the grid features of the target image to construct multiple pseudo-region features of the target image, the method includes: adding positional encoding to the initial grid features of the target image to obtain the grid features used to describe the spatial structural relationships of the target image.

[0012] In some implementations, before processing the grid features of the target image to construct multiple pseudo-region features about the target image, the method includes: constructing a generative model for generating the image description text based on the target image.

[0013] According to another aspect of this disclosure, an image description text generation model is proposed, comprising: a spatial feature perception module for processing grid features of a target image to construct multiple pseudo-region features of the target image, wherein the pseudo-region features are used to characterize local visual information of the target image; a feature enhancement module for encoding enhancement processing of the grid features and the pseudo-region features respectively to obtain grid enhancement features fused with multi-layer image encoding results and pseudo-region enhancement features fused with multi-layer local encoding results; and a text decoder for semantic parsing of the grid enhancement features and the pseudo-region enhancement features to generate image description text characterizing visual information in the target image.

[0014] According to another aspect of this disclosure, an electronic device is provided, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, causing the processor to perform the image description text generation method described in any of the above embodiments.

[0015] According to another aspect of this disclosure, a readable storage medium is provided, wherein executable instructions are stored therein, which, when executed by a processor, are used to implement the image description text generation method described in any of the above embodiments. Attached Figure Description

[0016] The accompanying drawings illustrate exemplary embodiments of the present disclosure and, together with the description thereof, serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0017] Figure 1 This is a flowchart illustrating a method for generating image description text according to an exemplary embodiment of this disclosure.

[0018] Figure 2 This is an architectural diagram of a method for generating image description text according to an exemplary embodiment of this disclosure.

[0019] Figure 3 This is a diagram illustrating the architecture of a model for generating image description text, which is an exemplary embodiment of this disclosure.

[0020] Figure 4 This is a schematic diagram of a multi-scale position-aware encoder, which is an exemplary embodiment of the present disclosure.

[0021] Figure 5 This is a schematic diagram illustrating the execution of the spatial feature perception module in an exemplary embodiment of this disclosure.

[0022] Figure 6 This is a schematic diagram of the decoder-side structure according to an exemplary embodiment of this disclosure.

[0023] Figure 7 This is a block diagram of an apparatus for generating image description text according to an exemplary embodiment of this disclosure. Detailed Implementation

[0024] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the accompanying drawings.

[0025] It should be noted that, where there is no conflict, the embodiments and features described in this disclosure can be combined with each other. The technical solutions of this disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0026] Unless otherwise stated, the exemplary implementations / embodiments shown are to be understood as providing exemplary features of various details that provide ways in which the technical concepts of this disclosure can be implemented in practice. Therefore, unless otherwise stated, the features of various implementations / embodiments may be additionally combined, separated, interchanged and / or rearranged without departing from the technical concepts of this disclosure.

[0027] The use of crosshairs and / or shading in the accompanying drawings is generally used to clarify the boundaries between adjacent components. Thus, unless otherwise stated, the presence or absence of crosshairs or shading does not convey or indicate any preference or requirement for the specific material, material properties, dimensions, proportions, commonalities between the illustrated components, or any other characteristics, properties, etc., of the components. Furthermore, in the accompanying drawings, the dimensions and relative dimensions of components may be exaggerated for clarity and / or descriptive purposes. When exemplary embodiments can be implemented differently, a specific process sequence may be performed in a different order than that described. For example, two consecutively described processes may be performed substantially simultaneously or in the reverse order of their description. Furthermore, the same reference numerals denote the same components.

[0028] When a component is referred to as being "on" or "above" another component, "connected to," or "joined to" another component, the component may be directly on, directly connected to, or directly joined to the other component, or there may be intermediate components. However, when a component is referred to as being "directly on" another component, "directly connected to," or "directly joined to" another component, there are no intermediate components. Therefore, the term "connection" can refer to a physical connection, an electrical connection, etc., and may or may not have intermediate components.

[0029] The terminology used herein is for the purpose of describing particular embodiments and is not intended to be limiting. As used herein, unless the context clearly indicates otherwise, the singular forms “a” and “the” are intended to include the plural forms as well. Furthermore, when the terms “comprising” and / or “including” and variations thereof are used in this specification, it indicates the presence of the stated features, integrals, steps, operations, parts, components, and / or groups thereof, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, parts, components, and / or groups thereof. It should also be noted that, as used herein, the terms “substantially,” “about,” and other similar terms are used as approximate terms rather than as terms of degree, thus explaining the inherent biases in measurements, calculated values, and / or provided values ​​that would be recognized by one of ordinary skill in the art.

[0030] Figure 1 This is a flowchart illustrating a method for generating image description text according to an exemplary embodiment of this disclosure. The following is in conjunction with... Figure 1 The method for generating image description text according to this disclosure, S100, will be described in detail.

[0031] The method S100 for generating image description text includes:

[0032] Step S102: Process the grid features of the target image to construct multiple pseudo-region features of the target image, wherein the pseudo-region features are used to characterize the local visual information of the target image.

[0033] Step S104: Encode and enhance the grid features and pseudo-region features respectively to obtain grid enhanced features that fuse multi-layer image coding results and pseudo-region enhanced features that fuse multi-layer local coding results.

[0034] Step S106: Semantic parsing is performed on the grid enhancement features and pseudo-region enhancement features to generate image description text used to characterize the visual information in the target image.

[0035] The target image refers to the image in the image data that contains descriptive text, and it can be any image in the image data.

[0036] In some implementations, step S102 is specifically implemented as follows: pre-setting multiple feature clusters; calculating the similarity between each grid feature and the feature cluster, so as to map each grid feature to the feature cluster with the highest similarity to itself, wherein grid features in the same feature cluster represent the same visual information; and normalizing multiple grid features in the same feature cluster to obtain multiple pseudo-region features used to represent each feature cluster.

[0037] In some implementations, step S104 is specifically implemented as follows: calling the mesh encoder to encode the mesh features, obtaining the hierarchical features generated by each encoding layer in the mesh encoder and the initial mesh features output by the mesh encoder; connecting the hierarchical features generated by each encoding layer in the mesh encoder to construct multi-layer mesh features; and fusing the initial mesh features and multi-layer mesh features according to a ratio to construct mesh enhancement features.

[0038] In some implementations, step S104 may further include: calling a pseudo-region encoder to encode the pseudo-region features to obtain the hierarchical features generated by each encoding layer in the pseudo-region encoder and the initial pseudo-region features output by the pseudo-region encoder; connecting the hierarchical features generated by each encoding layer in the pseudo-region encoder to construct multi-layer pseudo-region features; and fusing the initial pseudo-region features and multi-layer pseudo-region features in proportion to construct pseudo-region enhancement features.

[0039] In some implementations, step S106 is specifically implemented as follows: connecting the mesh enhancement feature and the pseudo-region enhancement feature to generate a feature string; and calling a text decoder to parse the feature string and the image annotation of the target image to obtain the image description text corresponding to the feature string, wherein the image description text is used to characterize the visual information in the target image.

[0040] In some implementations, prior to step S102, the method includes: adding positional encoding to the initial grid features of the target image to obtain grid features for describing the spatial structural relationships of the target image.

[0041] In some implementations, prior to step S102, a generative model is constructed for generating image description text based on the target image.

[0042] The generative model is used at least to perform steps S102 to S106 to output image description text corresponding to the target image. The process of building the generative model includes: preparing a sample dataset, constructing a vocabulary based on training samples, constructing a pre-trained model, and training and optimizing the pre-trained model using the sample dataset and vocabulary. Finally, the pre-trained model after training and optimization is used as the generative model for image description text.

[0043] The construction of the pre-trained model includes at least adding a spatial feature perception module to generate pseudo-region features; it also adds a multi-scale position perception module to capture local details between grids and simulate the interaction between adjacent grids, so that the output pseudo-region features have the ability to combine context.

[0044] Figure 2 This is an architectural diagram of an image description text generation method according to an exemplary embodiment of this disclosure. The following is in conjunction with... Figure 2 The main architecture of the image description text generation method disclosed herein is introduced.

[0045] This invention first uses a pre-trained regional convolutional neural network Faster-RCNN to extract grid features, with the neural network ResNeXt (suggesting the next dimension)-101 as its backbone.

[0046] First, a pre-trained model is constructed. Faster R-CNN is used as the pre-trained model to divide the target image into a grid, obtaining a 7×7 grid shape with 2048 bits per grid. Then, the pre-trained model is used to extract grid features from the target image.

[0047] Furthermore, the main objective of this disclosure is to leverage the advantages of both grid features and region features through feature fusion, while avoiding the direct use of region features, and to improve the spatial information loss problem that occurs when the Transformer model processes grid features. Therefore, the proposed grid features are used by a spatial feature awareness module to generate multiple pseudo-region features based on the clustering information of the grid features, and combined with position encoding, enabling the pseudo-region features to represent spatial position information. The pseudo-region features are input into the pseudo-region encoder of a Transformer model with self-attention. The pseudo-region features are analyzed through each encoding layer (i.e., sub-encoder) of the pseudo-region encoder, where there can be three encoding layers. Further, the hierarchical features corresponding to each encoding layer and the initial pseudo-region features output by the highest-level encoding layer can be obtained. Even further, the hierarchical features and the initial pseudo-region features are fused according to weights. Similarly, the mesh features are also input into the mesh encoder, where they are analyzed through various encoding layers (i.e., sub-encoders), with up to three encoding layers. Further, the hierarchical features corresponding to each encoding layer, as well as the initial mesh features output by the highest-level encoding layer, are obtained. Even further, the hierarchical features and the initial mesh features are fused according to their weights. This entire process is performed at the encoder end of the Transformer model.

[0048] Finally, a local visual modeling module enhances the contextual information of the mesh features. Then, a multilayer perceptron fuses the mesh enhancement features and pseudo-region enhancement features processed by the encoder. This integrates the advantages of both mesh and region features and fully utilizes the multi-level information of the features. The processed feature string is then fed into the text decoder of the Transformer model for analysis in conjunction with image annotations to generate image description text. The text decoder can also be configured with three decoders. The aforementioned process is performed on the decoder side of the Transformer.

[0049] Based on the aforementioned architecture, the method for generating image description text will be explained in detail below.

[0050] First, we construct a model for generating image-descriptive text.

[0051] The first step is to prepare the sample dataset. The sample dataset can be the object detection dataset MS-COCO2014, assuming it contains 123,287 images: 82,783 training images, 40,504 validation images, and 40,775 test images. Each image has 5 annotations covering 80 different object categories. Offline testing will be performed using any segmentation method, with 5,000 images used for validation, 5,000 for testing, and the remaining 113,287 images used for training.

[0052] The second step is to extract image grid features. First, a Faster-RCNN model with ResNeXt as the backbone is constructed. The pre-trained weights can be selected from existing technologies. All images after any segmentation method are input into the pre-trained model to extract the target image with a grid shape of 7×7, where each grid has 2048 bits.

[0053] The third step is to construct the vocabulary. All training sentences are preprocessed by converting them to lowercase and removing words that appear less than a preset number of times, such as 6 times. Finally, a vocabulary containing 9488 unique words is obtained.

[0054] The fourth step is to build a pre-trained model.

[0055] Figure 3 This is a diagram illustrating the architecture of an image description text generation model for an exemplary embodiment of this disclosure. (Reference) Figure 3 This model is based on the Transformer model, which mainly consists of an encoder and a decoder. A spatial feature awareness module is added to this model. This module generates multiple pseudo-region features based on the clustering information of grid features, thus addressing the high computational cost and inflexibility of directly extracting region features. Both pseudo-region features and grid features need to be processed by the encoder. A multi-scale position awareness module capable of modeling local information is added between the self-attention layer and the feedforward layer of the grid encoder. This module can capture local details between grids and simulate the interaction between adjacent grids. The framework of the pseudo-region encoder remains unchanged. All features from each encoding layer are merged into the top feature. The merged features are then fed into a multilayer perceptron. The processed pseudo-region enhancement features and grid enhancement features are fused and fed back to the text decoder of the Transformer model to generate image description text for the target image. The text decoder consists of a self-attention layer, a cross-attention layer, and a feedforward layer.

[0056] Figure 4 This is a schematic diagram of a multi-scale position-aware encoder, representing an exemplary embodiment of this disclosure. Figure 4As shown, by adding positional encoding to the extracted grid features through a multi-scale position-aware encoder, the spatial structural relationships in the image can be better understood. The multi-scale position-aware encoder consists of a multi-head self-attention module, a multi-scale position-aware module, and a feedforward pre-connection layer, from top to bottom. They are all connected through residuals, which can better enhance the model's fitting ability. Self-attention calculation maps a set of queries and key-value pairs to the output, and the output is calculated as the weight of the value, obtained by the similarity function between the query and the corresponding key. The features processed by multi-head self-attention are input into the multi-scale position-aware module, which consists of two cascaded 2D-CNNs connected by the ReLU activation function and finally processed by the Sigmoid function. The features are then fed back to the feedforward layer, which consists of two linear transformations with another ReLU activation function in between. The encoder disclosed herein consists of three encoding layers (i.e., sub-encoders). The multi-scale position-aware encoder is prior art, and its optimization and improvement results also fall within the protection scope of this disclosure.

[0057] Figure 5 This is a schematic diagram illustrating the execution of a spatial feature perception module according to an exemplary embodiment of this disclosure. Figure 5 As shown, the spatial feature perception module can convert the original grid features into pseudo-region features of multiple regions. Specifically, this disclosure uses discrete grid features, aggregates them into several word spaces to obtain pseudo-region features, and clusters grid features with multiple centroids without explicit supervision. The purpose of these centroids is to integrate grid features with similar semantic information to represent possible discriminative regions.

[0058] First, design the N learnable clusters as C = {c1, c2, ..., c...} N}, where the number of N can be set to any value. The similarity between grid features and feature clusters is calculated using dot products. Given each grid feature g, it can be mapped to its corresponding feature cluster R in the following way: m,n ,as follows:

[0059]

[0060] Where, d n d k c represents the trainable parameters. n Let g represent the nth learnable cluster. m Let represent the grid feature of the m-th grid, and k represent the learnable cluster index. Then, each center P is obtained by weighted integration of all grid features. n The features are represented as follows:

[0061]

[0062] Where LN represents the normalization operation, P represents the parameters that can be trained and learned. n This represents the pseudo-region feature, m represents the grid number, and M represents the total number of grid features.

[0063] Then, the generated region features are encoded with location information and fed into the encoder. The encoder consists of a multi-head self-attention module and a feedforward fully connected layer.

[0064] To fully utilize the semantic information from the bottom and top layers of the encoder, the features output from each layer are fused, taking grid features as an example. Given the output features of each encoder as follows: To integrate low-level and high-level visual information, we employ a multilayer perceptron to obtain features. It can measure the contribution of each layer's features. Finally, since the output of the top encoding layer contains more important visual information, to prevent the insertion of additional noise perturbations, this disclosure will... and characteristics The features are then blended proportionally to obtain mesh-enhanced features:

[0065]

[0066] Where γ is an adjustable factor, which can be set to 0.2.

[0067] The method for obtaining pseudo-region enhancement features is the same as that for grid enhancement features, and will not be repeated here.

[0068] The processed mesh enhancement features and pseudo-region features are concatenated and then fed into the text decoder along with the image annotations.

[0069] Figure 6 This is a schematic diagram of the decoder-side structure according to an exemplary embodiment of this disclosure. (Reference) Figure 6 It's important to note that multi-head self-attention in the decoder requires look-ahead masking because future information cannot be seen during prediction, so both the current token and subsequent tokens need to be masked. In multi-head attention, this layer differs from the self-attention mechanism where Q = K = V. Here, matrix Q originates from the output of the previous decoder, while matrices K and V originate from the encoder's output, resulting in Q! = K = V. Here, Q represents the word vector, and K and V represent the image features processed by the encoder. This is to ensure that the decoder's tokens receive more attention from the encoder. The decoder of the Transformer model is existing technology, and any optimizations or improvements to it fall within the scope of this disclosure.

[0070] The fifth step is to train the pre-trained model.

[0071] Both the encoder and decoder have three layers, each using eight self-attention heads. The internal size of the feedforward fully connected layer is 2048. The multi-scale position awareness module processes features in three paths. From left to right, they are 1×1 convolution + 3×3 convolution + BatchNorm, 1×1 convolution + BatchNorm, and a BatchNorm layer. The number of pseudo-regions is 5, and γ = 0.2 during feature fusion.

[0072] The training process is as follows:

[0073] First, the model is optimized using the cross-entropy loss function. The batch size is 50, and the model is trained for 18 epochs with an initial learning rate of 0.0001. The cross-entropy loss function can be defined as follows:

[0074]

[0075] Among them, L CE The output value represents the difference between the model output value and the expected value, T represents the length of the image description text (i.e., the number of characters), t represents the character index, θ represents the model parameters, and p represents the model output value. θ This represents the probability distribution generated by the neural network, which is used to predict the next word. This represents the t-th word in the true sequence. This represents the first t-1 sequences.

[0076] Furthermore, a reinforcement learning strategy is adopted, using the CIDEr score as r(·) reward for self-criticism sequence training, thereby optimizing the CIDEr score, with an initial learning rate of 5e-6.

[0077] Furthermore, the average reward is used during training, and the gradient expression for a single sample is:

[0078]

[0079] in, This represents the gradient operation with respect to the parameter θ, i.e., the partial derivative with respect to θ; r represents the reward value of the i-th sample; represents the reward obtained by the model after generating the sequence; and k is a countable number of sampling sequences. For the i-th sampling sequence, The mean of the rewards obtained from the sampled sequence (i.e., the baseline of the rewards).

[0080] Finally, after training is complete, the weights of the pre-trained model are saved and tested. The pre-trained model that can obtain the desired descriptive text is then used as the image descriptive text generation model.

[0081] Table 1

[0082]

[0083] Table 1 compares the image description text generation method of this disclosure (i.e., Ours) with other methods for generating image description captions, such as the RFNet model, GCN-LSTM model, ORT model, AoANet model, and M... 2 Transformer and X-Transformer models, among others. This method achieved good results on the evaluation metrics B@1 (%), B@4 (%), M (%), R (%), C (%), and S (%). The five evaluation metrics are BLEU, METEOR, ROUGE-L, CIDER-D, and SPICE. The first two metrics were originally designed for machine translation, while ROUGE-L was initially used for automatic summarization. However, since machine translation and automatic text summarization are both for text sequences, this metric can also be used for image captioning. CIDER-D and SPICE are mainly used to evaluate image descriptions. Generally, the higher the model's score on these metrics, the more similar the generated captions are to human-annotated reference captions. Specific values ​​are shown in Table 1 and are not listed individually.

[0084] This disclosed method for generating image description text constructs pseudo-region features using the grid features of the target image, combining the advantages of both grid and region features. Furthermore, by setting up a multi-scale location-aware module, it receives fine-grained local information from different regions, enhancing the interaction between adjacent grids. Through a feature enhancement mechanism, top-level and bottom-level features are combined, resulting in a feature string that incorporates multi-level semantic information, preventing information loss. These steps enable the generated image description text to combine contextual information from the background, presenting a more complete and visually informative content.

[0085] Figure 7 This is a block diagram of an apparatus for generating image description text according to an exemplary embodiment of this disclosure.

[0086] like Figure 7As shown, this disclosure proposes an image description text generation model 1000, comprising: a spatial feature perception module 1002, used to process the grid features of the target image and construct multiple pseudo-region features of the target image, wherein the pseudo-region features are used to characterize the local visual information of the target image; a feature enhancement module 1004, used to perform encoding enhancement processing on the grid features and pseudo-region features respectively, to obtain grid enhancement features fused with multi-layer image encoding results and pseudo-region enhancement features fused with multi-layer local encoding results; and a text decoder 1006, used to perform semantic parsing on the grid enhancement features and pseudo-region enhancement features to generate image description text used to characterize the visual information in the target image.

[0087] The modules in the image description text generation model 1000 are set up to execute the various steps in the image description text generation method. The execution principle and steps can be referred to above and will not be repeated here.

[0088] The apparatus may include corresponding modules that perform one or more steps in the flowchart above. Therefore, each or more steps in the flowchart above may be performed by a corresponding module, and the apparatus may include one or more of these modules. A module may be one or more hardware modules specifically configured to perform a corresponding step, or implemented by a processor 1300 configured to perform a corresponding step, or stored in a computer-readable medium for implementation by the processor 1300, or implemented by some combination thereof.

[0089] This hardware architecture can be implemented using a bus architecture. The bus architecture can include any number of interconnect buses 1100 and bridges, depending on the specific application and overall design constraints of the hardware. Bus 1100 connects various other circuits 1400, including one or more processors 1300, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400 such as peripherals, voltage regulators, power management circuitry, external antennas, etc.

[0090] Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. Bus 1100 can be categorized as an address bus, data bus, control bus, etc. For ease of representation, only one connection line is used in this diagram, but this does not imply that there is only one bus or one type of bus 1100.

[0091] Any process or method description in the flowcharts or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of this disclosure pertain. Processor 1300 performs the various methods and processes described above. For example, the method embodiments of this disclosure may be implemented as software programs tangibly contained in a machine-readable medium, such as memory 1300. In some embodiments, part or all of the software program may be loaded and / or installed via memory 1300 and / or a communication interface. When the software program is loaded into memory 1300 and executed by processor 1300, one or more steps of the methods described above may be performed. Alternatively, in other embodiments, processor 1300 may be configured to perform one of the methods described above by any other suitable means (e.g., by means of firmware).

[0092] The logic and / or steps represented in the flowchart or otherwise described herein may be implemented in any readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a system including processor 1300 or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0093] For the purposes of this specification, a "readable storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM). Furthermore, a readable storage medium can even be paper or other suitable media on which a program can be printed, since a program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in memory.

[0094] It should be understood that various parts of this disclosure can be implemented in hardware, software, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0095] Those skilled in the art will understand that all or part of the steps of the methods described above can be implemented by a program instructing related hardware, and the program can be stored in a readable storage medium. When executed, the program includes one or a combination of the steps of the method implementation.

[0096] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into a single processing module, or each unit can exist physically separately, or two or more units can be integrated into a single module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a readable storage medium. The storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0097] Those skilled in the art should understand that the above embodiments are merely for illustrating the present disclosure and are not intended to limit the scope of the disclosure. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present disclosure.

Claims

1. A method for generating image-descriptive text, characterized in that, include: The grid features of the target image are processed to construct multiple pseudo-region features of the target image, which are used to characterize the local visual information of the target image; The grid features and the pseudo-region features are encoded and enhanced respectively to obtain grid enhanced features that fuse multi-layer image coding results and pseudo-region enhanced features that fuse multi-layer local coding results. as well as Semantic parsing is performed on the grid enhancement features and the pseudo-region enhancement features to generate image description text that characterizes the visual information in the target image; The process of processing the grid features of the target image to construct multiple pseudo-region features of the target image includes: pre-setting multiple feature clusters; calculating the similarity between each grid feature and the feature cluster, so as to map each grid feature to the feature cluster with the highest similarity to itself, wherein grid features in the same feature cluster represent the same visual information; and normalizing multiple grid features in the same feature cluster to obtain multiple pseudo-region features for representing each feature cluster. The step of encoding and enhancing the grid features and the pseudo-region features respectively to obtain grid-enhanced features fused with multi-layer image encoding results and pseudo-region-enhanced features fused with multi-layer local encoding results includes: calling a grid encoder to encode the grid features to obtain the hierarchical features generated by each encoding layer in the grid encoder and the initial grid features output by the grid encoder; connecting the hierarchical features generated by each encoding layer in the grid encoder to construct multi-layer grid features; and fusing the initial grid features and the multi-layer grid features according to a certain ratio to construct the grid-enhanced features. The step of encoding and enhancing the grid features and the pseudo-region features respectively to obtain grid-enhanced features fused with multi-layer image encoding results and pseudo-region-enhanced features fused with multi-layer local encoding results includes: calling a pseudo-region encoder to encode the pseudo-region features to obtain the hierarchical features generated by each encoding layer in the pseudo-region encoder and the initial pseudo-region features output by the pseudo-region encoder; connecting the hierarchical features generated by each encoding layer in the pseudo-region encoder to construct multi-layer pseudo-region features; and fusing the initial pseudo-region features and the multi-layer pseudo-region features according to a certain ratio to construct the pseudo-region-enhanced features.

2. The method for generating image descriptive text according to claim 1, characterized in that, The step of performing semantic parsing on the mesh enhancement features and the pseudo-region enhancement features to generate image description text for characterizing the visual information in the target image includes: Concatenate the mesh enhancement feature and the pseudo-region enhancement feature to generate a feature string; and A text decoder is invoked to parse the feature string and the image annotation of the target image to obtain the image description text corresponding to the feature string, wherein the image description text is used to characterize the visual information in the target image.

3. The method for generating image description text according to claim 1, characterized in that, Before processing the grid features of the target image to construct multiple pseudo-region features about the target image, the process includes: Position encoding is added to the initial grid features of the target image to obtain the grid features used to describe the spatial structural relationships of the target image.

4. The method for generating image descriptive text according to claim 1, characterized in that, Before processing the grid features of the target image to construct multiple pseudo-region features about the target image, the process includes: Construct a generative model for generating image description text based on the target image.

5. An electronic device, characterized in that, include: The memory stores execution instructions; as well as A processor that executes execution instructions stored in the memory, causing the processor to perform the method for generating image description text according to any one of claims 1 to 4.

6. A readable storage medium, characterized in that, The readable storage medium stores execution instructions, which, when executed by a processor, are used to implement the method for generating image description text according to any one of claims 1 to 4.

Images