Image description method, device, equipment, and storage medium

Abstract

The present application discloses an image description method, apparatus, device and storage medium, including: extracting an image feature map and mask information of an image to be described; reducing the dimension of the image feature map and the mask information to obtain a target image feature map and target mask information as a position code; using a preset encoder to perform image encoding on the target image feature map to obtain a corresponding image feature map encoding result; using a preset text extractor to extract text information from pre-encoded text, and inputting the text information, the target mask information and the image feature map encoding result into a preset decoder to generate text corresponding to the image to be described. The present application can extract semantic information between the foreground and the background, and use the mask information as a position code, which is conducive to the overall structural convergence, can improve the overall adaptability and robustness, reduce the amount of encoding and decoding calculations, and achieve more accurate image description.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image description method, device, equipment, and storage medium. Background Art

[0002] Currently, computers automatically establish relationships between low-level visual features such as scenes, colors, and targets in images to obtain high-level semantic information that humans can understand, that is, a text description of the image. This technology can be applied to fields such as humanistic care, smart homes, smart transportation, and medical diagnosis. For example, it can help visually impaired people understand objects around them and road conditions. In security scenarios, it can directly warn of dangerous behaviors of targets in the scene based on scene images obtained by cameras. In the field of medical diagnosis, it can directly obtain diagnostic results through medical images to improve the efficiency of doctors' diagnosis. Image description technology has important practical significance for people's daily lives.

[0003] In the existing technology, image description can be simply divided into two parts: image feature extraction and encoding and decoding structure. Currently, Faster R-CNN (target detection algorithm) is commonly used to extract image features, and then the mainstream encoding and decoding structure Transformer is used to complete the fusion of image and text information. However, this method extracts more features of foreground targets, and the image information obtained is not comprehensive, which affects the accuracy of the image description results. At the same time, the use of the existing Transformer encoding and decoding structure makes the model computationally huge, and the trained model has poor practicality.

[0004] In summary, it can be seen that how to extract richer image information, reduce the amount of encoding and decoding calculations, and achieve more accurate image description is a problem to be solved in this field. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide an image description method, device, equipment, and storage medium, which can extract richer image information, reduce the amount of encoding and decoding calculations, and achieve more accurate image description. The specific scheme is as follows:

[0006] In a first aspect, the present application discloses an image description method, comprising:

[0007] Extracting image feature map and mask information of the image to be described;

[0008] Performing dimensionality reduction on the image feature map and the mask information to obtain a target image feature map and target mask information as a position code;

[0009] Using a preset encoder to perform image encoding on the target image feature map to obtain a corresponding image feature map encoding result;

[0010] A preset text extractor is used to extract text information from the pre-encoded text, and the text information, the target mask information and the image feature map encoding result are input into a preset decoder to generate text corresponding to the image to be described.

[0011] Optionally, extracting the image feature map and mask information of the image to be described includes:

[0012] Extracting features of foreground objects and backgrounds in an image feature map to describe an image based on a panoramic segmentation algorithm to obtain correlations between the foreground objects and the background, correlations between different foreground objects, and mask information;

[0013] Correspondingly, the using a preset encoder to perform image encoding on the target image feature map to obtain a corresponding image feature map encoding result includes:

[0014] The preset encoder is used to process the correlation between the foreground object and the background and the correlation between different foreground objects to obtain a corresponding image feature map encoding result.

[0015] Optionally, the performing dimensionality reduction on the image feature map and the mask information to obtain a target image feature map and target mask information as a position code includes:

[0016] The image feature map and the mask information are input into a preset downsampling module to obtain a target image feature map after dimension reduction and target mask information as a position code.

[0017] Optionally, the preset encoder includes a first LN normalization layer, a first multi-head self-attention mechanism, a first IN normalization layer, a first FFN layer, and a second IN normalization layer connected in sequence; accordingly, the image encoding of the target image feature map using the preset encoder to obtain a corresponding image feature map encoding result includes:

[0018] Normalizing all target image features in the target image feature map based on the first LN normalization layer to obtain normalized target image features;

[0019] Using the first multi-head self-attention mechanism and the first IN normalization layer to perform feature calculation and normalization processing on the normalized target image features to obtain a relationship between the normalized target image features;

[0020] The first FFN layer and the second IN normalization layer are used to process the relationship between the normalized target image features to obtain a corresponding image feature map encoding result.

[0021] Optionally, the using a preset encoder to perform image encoding on the target image feature map to obtain a corresponding image feature map encoding result includes:

[0022] The target image feature map is encoded multiple times based on a preset number of encoding times to generate a corresponding image feature map encoding result.

[0023] Optionally, the step of inputting the text information, the target mask information and the image feature map encoding result into a preset decoder to generate text corresponding to the image to be described includes:

[0024] The target mask information and the image feature map encoding result are spliced ​​to obtain corresponding splicing information, and then the splicing information and the text information are input into a preset decoder to generate a text description corresponding to the image to be described.

[0025] Optionally, the preset decoder includes a second multi-head self-attention mechanism, a second LN normalization layer, a third multi-head self-attention mechanism, a third LN normalization layer, a second FFN layer, and a fourth LN normalization layer connected in sequence; wherein, inputting the splicing information and the text information into the preset decoder to generate a text description corresponding to the image to be described includes:

[0026] Extracting text information from the pre-encoded text using a preset text extractor, and inputting the text information into the second multi-head self-attention mechanism and the second LN normalization layer to obtain normalized text information;

[0027] Processing the splicing information and the normalized text information using the third multi-head self-attention mechanism, the third LN normalization layer, the second FFN layer, and the fourth LN normalization layer to obtain text features corresponding to different foreground objects and backgrounds in the image feature map;

[0028] The text features corresponding to the image feature map are globally extracted using a fully connected layer to obtain global text features corresponding to the image feature map, and the global text features are input into the Softmax function to obtain a text score corresponding to the image to be described, and the corresponding text is found according to the text score to generate the text corresponding to the image to be described.

[0029] In a second aspect, the present application discloses an image description device, comprising:

[0030] An information acquisition module, used to extract image feature maps and mask information of the image to be described;

[0031] An information dimension reduction module, used for reducing the dimension of the image feature map and the mask information to obtain a target image feature map and target mask information as a position code;

[0032] An image encoding module, used to perform image encoding on the target image feature map using a preset encoder to obtain a corresponding image feature map encoding result;

[0033] The image description module is used to use a preset text extractor to extract text information from pre-encoded text, and input the text information, the target mask information and the image feature map encoding result into a preset decoder to generate text corresponding to the image to be described.

[0034] In a third aspect, the present application discloses an electronic device, comprising:

[0035] Memory, used to store computer programs;

[0036] The processor is used to execute the computer program to implement the steps of the image description method disclosed above.

[0037] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the steps of the aforementioned disclosed image description method are implemented.

[0038] It can be seen that the present application discloses an image description method, including: extracting an image feature map and mask information of an image to be described; reducing the dimension of the image feature map and the mask information to obtain a target image feature map and target mask information as a position code; using a preset encoder to perform image coding on the target image feature map to obtain a corresponding image feature map coding result; using a preset text extractor to extract text information from the pre-coded text, and inputting the text information, the target mask information and the image feature map coding result into a preset decoder to generate text corresponding to the image to be described. It can be seen that the present application extracts the image feature map of the image to be described to obtain the foreground target features and the semantic information of the foreground target and the background, and uses the extracted target mask information as the position code of the preset decoder, which is conducive to the overall structural convergence, improves the overall adaptability and robustness, and improves the overall accuracy of the image description. The text information is extracted from the pre-coded text in a low-computation manner, and the preset decoder obtained by adjusting the structure in the decoder can reduce the overall complexity and solve the problem of large computational complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0040] Figure 1 A flow chart of an image description method disclosed in this application;

[0041] Figure 2 A panoramic segmentation result map disclosed in this application;

[0042] Figure 3 A mask visualization diagram disclosed in this application;

[0043] Figure 4 A downsampling module diagram disclosed in this application;

[0044] Figure 5 A flowchart of a specific image description method disclosed in this application;

[0045] Figure 6 A flowchart of another specific image description method disclosed in this application;

[0046] Figure 7 This is a general structural diagram of a model disclosed in this application;

[0047] Figure 8 A codec module structure diagram disclosed in this application;

[0048] Fig. 9 This is a schematic diagram of the structure of an image description device disclosed in this application;

[0049] Fig.10 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] Currently, image description can be simply divided into two parts: image feature extraction and encoding and decoding structure. Faster R-CNN is commonly used to extract image features, and then the mainstream encoding and decoding structure Transformer is used to complete the fusion of image and text information. However, this method extracts more features of foreground objects, and the image information obtained is not comprehensive, which affects the accuracy of the image description results. At the same time, the use of the existing Transformer encoding and decoding structure makes the model computationally huge, and the trained model has poor practicality.

[0052] To this end, the present application provides an image description solution that can extract richer image information, reduce the amount of encoding and decoding calculations, and achieve more accurate image description.

[0053] Reference Figure 1 As shown, an embodiment of the present invention discloses an image description method, which specifically includes:

[0054] Step S11: extracting the image feature map and mask information of the image to be described.

[0055] In this embodiment, Panoptic FPN (panoramic segmentation algorithm) based on ResNeXt-101 (residual network) is used to extract image features. This algorithm can effectively extract the relationship between foreground targets and backgrounds, and between foreground targets, and at the same time obtain the color, material and other attribute features of foreground targets. Compared with the commonly used target detection algorithms, more features of foreground targets are extracted. The use of Panoptic FPN effectively makes up for the missing background semantic information of the target detection algorithm. Figure 2 As shown, through Panoptic FPN, the size of the image feature map obtained is: [C, H, W]. In addition to the image features, the mask information obtained by Panoptic FPN segmentation also needs to be retained, such as Figure 3 As shown, the size of the mask is: [C, H, W].

[0056] Step S12: performing dimension reduction on the image feature map and the mask information to obtain a target image feature map and target mask information as a position code.

[0057] In this embodiment, the image feature map and the mask information are input into a preset downsampling module to obtain a target image feature map after dimensionality reduction and target mask information as position encoding. It is understandable that since the image feature map obtained above has a large dimension, directly inputting the image feature map into the preset encoder and the preset decoder will easily lead to a significant increase in the amount of calculation of the Transformer model therein. Therefore, Figure 4As shown, the image feature map and the mask information of size [C, H, W] are input into the preset downsampling module. After passing through a 3*3 convolution layer with a step size of 4, the dimension size becomes [4c, h / 4, w / 4]; after passing through a 3*3 convolution layer with a step size of 2, the dimension size becomes [8c, h / 8, w / 8]; finally, after passing through a 1*1 convolution layer, the dimension is reduced to [C, H / 8, W / 8]. After the dimension reduction by the preset downsampling module, the size of the image feature map and the mask information becomes [C, H / 8, W / 8], and finally the adjusted fixed size of the target image feature map and the target mask information is obtained, and the mask information is used as the position code of the preset decoder.

[0058] Step S13: using a preset encoder to perform image encoding on the target image feature map to obtain a corresponding image feature map encoding result.

[0059] In this embodiment, the target image feature map is input into the preset encoder so that the preset encoder performs image encoding on the target image feature map and obtains a corresponding image feature map encoding result.

[0060] Step S14: using a preset text extractor to extract text information from the pre-encoded text, and inputting the text information, the target mask information and the image feature map encoding result into a preset decoder to generate text corresponding to the image to be described.

[0061] In this embodiment, a preset text extractor is used to extract text information from a pre-encoded text. It can be understood that since the present application extracts text information in advance before the text information is passed into the decoder, the problem of large amount of calculation of the Transformer structure in the preset decoder can be avoided; and the text information, the target mask information and the image feature map encoding results are input into the preset decoder to obtain the global semantic information of the image to be described and the text information.

[0062] It can be seen that the present application discloses an image description method, including: extracting an image feature map and mask information of an image to be described; reducing the dimension of the image feature map and the mask information to obtain a target image feature map and target mask information as a position code; using a preset encoder to perform image coding on the target image feature map to obtain a corresponding image feature map coding result; using a preset text extractor to extract text information from the pre-coded text, and inputting the text information, the target mask information and the image feature map coding result into a preset decoder to generate text corresponding to the image to be described. It can be seen that the present application extracts the image feature map of the image to be described to obtain the foreground target features and the semantic information of the foreground target and the background, and uses the extracted target mask information as the position code of the preset decoder, which is conducive to the overall structural convergence, improves the overall adaptability and robustness, and improves the overall accuracy of the image description. The text information is extracted from the pre-coded text in a low-computation manner, and the preset decoder obtained by adjusting the structure in the decoder can reduce the overall complexity and solve the problem of large computational complexity.

[0063] Reference Figure 5 As shown, referring to the figure, the embodiment of the present invention discloses a specific image description method. Compared with the previous embodiment, this embodiment further illustrates and optimizes the technical solution. Specifically:

[0064] Step S21: extracting features of the foreground object and the background in the image feature map to describe the image based on a panoramic segmentation algorithm to obtain the correlation between the foreground object and the background, the correlation between different foreground objects, and mask information.

[0065] In this embodiment, a panoramic segmentation algorithm is used to obtain the relationship between the foreground target and the background, and the relationship between foreground targets in the image to be described, and at the same time, the attribute characteristics such as color and material of the foreground target are obtained; the mask information obtained by the panoramic segmentation algorithm is used as a position code, wherein the mask information may specifically include but is not limited to information such as the position and size of the foreground target relative to the background.

[0066] Step S22: performing dimension reduction on the image feature map and the mask information to obtain a target image feature map and target mask information as a position code.

[0067] For more specific processing procedures in the above step S22, reference may be made to the corresponding contents disclosed in the above embodiments, which will not be described in detail here.

[0068] Step S23: using a preset encoder to process the correlation between the foreground object and the background and the correlation between different foreground objects to obtain a corresponding image feature map encoding result.

[0069] In this embodiment, the correlation between the foreground target and the background and the correlation between different foreground targets in the image feature map obtained above are input into the preset encoder and processed by the encoder to obtain the corresponding image feature map encoding result.

[0070] Step S24: splicing the target mask information and the image feature map encoding result to obtain corresponding splicing information, extracting text information from the pre-encoded text, and then inputting the splicing information and the text information into a preset decoder to generate a text description corresponding to the image to be described.

[0071] In this embodiment, the target mask information obtained after dimensionality reduction is spliced ​​with the image feature map encoding result obtained by processing the preset encoder, and the spliced ​​information and the text information are input into a preset decoder, and then a text describing the image to be described is automatically generated.

[0072] It can be seen that in this embodiment, since the mask information contains information such as the size and position of the foreground target, using the mask information obtained by the panoramic segmentation algorithm as a position code is beneficial to extracting the semantic information between the foreground target and the background, and facilitates the understanding of the information contained in the image to be described.

[0073] Reference Figure 6 As shown, with reference to the figure, the embodiment of the present invention discloses another specific image description method. Compared with the previous embodiment, this embodiment further illustrates and optimizes the technical solution. Specifically:

[0074] Step S31: extracting the image feature map and mask information of the image to be described.

[0075] Step S32: performing dimension reduction on the image feature map and the mask information to obtain a target image feature map and target mask information as a position code.

[0076] For more specific processing procedures in the above step S31 and step S32, reference may be made to the corresponding contents disclosed in the above embodiments, which will not be described again.

[0077] Step S33: Based on the first LN normalization layer, all target image features in the target image feature map are normalized to obtain normalized target image features; the first multi-head self-attention mechanism and the first IN normalization layer are used to perform feature calculation and normalization on the normalized target image features to obtain the relationship between the normalized target image features; the relationship between the normalized target image features is processed using the first FFN layer and the second IN normalization layer to obtain the corresponding image feature map encoding result.

[0078] In this embodiment, the preset encoder includes a first LN normalization layer, a first multi-head self-attention mechanism, a first IN normalization layer, a first FFN (Feed-Forward Network) layer, and a second IN normalization layer connected in sequence; and the target image feature map is encoded multiple times based on a preset number of encodings to generate a corresponding image feature map encoding result. It can be understood that the preset encoder in this application is modified on the basis of the Transformer encoder, and the modified parts are as follows: a Layer Normalization layer is added as the first LN normalization layer; the original LN normalization layer is changed to an Instance Normalization layer as the first IN normalization layer; and the ReLU activation function in the first FFN layer is changed to a GeLU (Gaussian errorlinear units) activation function. First, all target image features in the target feature map are normalized through the first LN layer, wherein the LN normalization method normalizes each training sample to the same distribution, which can accelerate the convergence speed of the model; then, the first multi-head self-attention mechanism is entered, and the q (query), v (value), and k (key) input by the first multi-head self-attention mechanism are all the target image features themselves, so that the relationship between each pixel in the target image features can be learned, thereby more fine-grainedly reflecting the relationship between foreground targets, which is more conducive to accurately describing the image; then, the image features first input are connected using residual connections to improve the feature expression ability of the model, and then the first IN normalization layer is entered. Since the image and text themselves are two modal information, the use of the IN layer in the encoder is conducive to the conversion between different forms of information and promotes the convergence of the model; after the first FFN layer, the GeLU activation function introduces a random regularization method, which can increase the nonlinear expression ability of the network, so the ReLU activation function of the FFN in the encoder is modified to the GeLU activation function; a residual connection is used to prevent the gradient from disappearing or exploding due to excessive depth. The above operation is repeated 6 times in total to obtain the output of the preset encoder, that is, the corresponding image feature map encoding result is obtained.

[0079] Step S34: Use a preset text extractor to extract text information from the pre-encoded text, and input the text information into a second multi-head self-attention mechanism and a second LN normalization layer to obtain normalized text information.

[0080] In this embodiment, the preset decoder includes a second multi-head self-attention mechanism, a second LN normalization layer, a third multi-head self-attention mechanism, a third LN normalization layer, a second FFN layer, and a fourth LN normalization layer connected in sequence; it can be understood that the preset decoder in this application is modified on the basis of the Transformer decoder, and the modified parts are as follows: the 6-layer Transformer decoder is changed to a 1-layer preset decoder; the ReLU activation function of the second FFN layer is changed to a GeLU activation function.

[0081] Reference Figure 7 As shown, first, the 6-layer Transformer decoder is changed to a 1-layer preset decoder because the present application uses the preset text extractor to extract refined text information from the pre-encoded text, and no longer requires a 6-layer Transformer decoder structure. Since the number of units in the preset decoder is reduced, the complexity of the network can be greatly reduced; and because the GeLU activation function has excellent expressiveness in processing sequence problems, the second FFN layer in the preset decoder also uses the GeLU activation function. First, the text is processed by the preset text extractor to obtain h text information, which is used as q of the third multi-head self-attention mechanism after the second multi-head self-attention mechanism and the second LN normalization layer. It should be noted that in order to reduce the amount of calculation in the subsequent model calculation process, the GRU (Gate Recurrent Unit) network and LSTM (Long Short-Term Memory) can be used to extract the text information.

[0082] Step S35: Process the splicing information and the normalized text information using a third multi-head self-attention mechanism, a third LN normalization layer, a second FFN layer, and a fourth LN normalization layer to obtain text features corresponding to different foreground targets and backgrounds in the image feature map, wherein the splicing information is information obtained by splicing the target mask information and the encoding result of the image feature map.

[0083] In this embodiment, the splicing information obtained by splicing the image feature map encoding result output by the preset encoder with the mask information is used as k and v, and is input into the third multi-head self-attention mechanism together with q of the third multi-head self-attention mechanism after passing through the second multi-head self-attention mechanism and the second LN normalization layer, and then passes through the third LN normalization layer, the second FFN layer, and the fourth LN normalization layer to obtain the output of the preset decoder.

[0084] Step S36: Use a fully connected layer to globally extract the text features corresponding to the image feature map to obtain global text features corresponding to the image feature map, and input the global text features into the Softmax function to obtain a text score corresponding to the image to be described, and find the corresponding text according to the text score to generate the text corresponding to the image to be described.

[0085] In this embodiment, the result output by the preset decoder is passed to the fully connected layer so that the fully connected layer can extract its global information again, and then the score of the text information is obtained through Softmax, that is, each position in a sentence corresponds to the number of texts in the dictionary, and finally a sentence that accurately describes the image is obtained.

[0086] In this embodiment, refer to Figure 8 As shown, the structure of the encoder in the codec module structure diagram includes LN layers, MHA (Multi-Head Attention, multi-head self-attention mechanism), IN layers, FFN layers, and IN layers connected in sequence. It can be understood that when Image Feature is input into the encoder, the encoded image feature result is obtained after processing in each layer of the encoder, and the encoded image feature result is spliced ​​with mask to obtain spliced ​​splicing information, and the splicing information is used as k and v in the input decoder, wherein the structure of the decoder in the codec module structure diagram includes MHA, LN layers, MHA, LN layers, FFN layers, and LN layers connected in sequence. It can be understood that the extracted h is input into the decoder, and after processing, it is used as q through the processing of each layer connected in sequence in the subsequent decoder, and finally the output of the preset decoder is obtained, and the output is extracted again through Linear (fully connected layer) to obtain global text features, and the global text features are input into Softmax to obtain the score of the text, and finally a sentence accurately describing the image is obtained.

[0087] It can be seen that the present application extracts text information by utilizing a preset text extractor and modifies and adjusts the structure of the Transformer encoder and the Transformer decoder, thereby avoiding a large number of matrix multiplications in the codec to obtain global semantic information and reducing the computational complexity of the model.

[0088] Reference Fig. 9 As shown, an embodiment of the present invention discloses a schematic diagram of the structure of an image description device, which specifically includes:

[0089] An information acquisition module 11 is used to extract an image feature map and mask information of an image to be described;

[0090] An information dimension reduction module 12, used for reducing the dimension of the image feature map and the mask information to obtain a target image feature map and target mask information as a position code;

[0091] An image encoding module 13 is used to perform image encoding on the target image feature map using a preset encoder to obtain a corresponding image feature map encoding result;

[0092] The image description module 14 is used to use a preset text extractor to extract text information from the pre-encoded text, and input the text information, the target mask information and the image feature map encoding result into a preset decoder to generate text corresponding to the image to be described.

[0093] It can be seen that the present application discloses an image description method, including: extracting an image feature map and mask information of an image to be described; reducing the dimension of the image feature map and the mask information to obtain a target image feature map and target mask information as a position code; using a preset encoder to perform image coding on the target image feature map to obtain a corresponding image feature map coding result; using a preset text extractor to extract text information from the pre-coded text, and inputting the text information, the target mask information and the image feature map coding result into a preset decoder to generate text corresponding to the image to be described. It can be seen that the present application extracts the image feature map of the image to be described to obtain the foreground target features and the semantic information of the foreground target and the background, and uses the extracted target mask information as the position code of the preset decoder, which is conducive to the overall structural convergence, improves the overall adaptability and robustness, and improves the overall accuracy of the image description. The text information is extracted from the pre-coded text in a low-computation manner, and the preset decoder obtained by adjusting the structure in the decoder can reduce the overall complexity and solve the problem of large computational complexity.

[0094] Furthermore, the present application also discloses an electronic device. Fig.10This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.

[0095] Fig.10 The present invention provides a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps of the image description method disclosed in any of the above embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0096] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0097] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0098] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, and may be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program that can be used to complete the image description method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 may further include computer programs that can be used to complete other specific tasks.

[0099] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the aforementioned disclosed image description method. The specific steps of the method can refer to the corresponding contents disclosed in the aforementioned embodiments, and will not be repeated here.

[0100] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0101] Professionals may further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application. The steps of the method or algorithm described in conjunction with the embodiments disclosed herein can be directly implemented with a software module executed by a hardware or processor, or a combination of the two. The software module can be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0102] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0103] The above is a detailed introduction to an image description method, device, equipment, and storage medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for a person skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. An image description method, characterized in that: include: Extract image feature map and mask information of the image to be described based on the panoramic segmentation algorithm; Performing dimensionality reduction on the image feature map and the mask information to obtain a target image feature map and target mask information as a position code; Performing image encoding on the target image feature map using a preset encoder to obtain a corresponding image feature map encoding result; wherein the preset encoder includes a first LN normalization layer, a first multi-head self-attention mechanism, a first IN normalization layer, a first FFN layer, and a second IN normalization layer connected in sequence; Utilizing a preset text extractor, extracting text information from a pre-encoded text, and inputting the text information, the target mask information, and the image feature map encoding result into a preset decoder to generate text corresponding to the image to be described; wherein the preset decoder comprises a second multi-head self-attention mechanism, a second LN normalization layer, a third multi-head self-attention mechanism, a third LN normalization layer, a second FFN layer, and a fourth LN normalization layer connected in sequence; The step of inputting the target mask information and the image feature map encoding result into a preset decoder to generate text corresponding to the image to be described includes: The target mask information and the image feature map encoding result are spliced ​​to obtain corresponding splicing information, and then the splicing information and the text information are input into a preset decoder to generate a text description corresponding to the image to be described.

2. The image description method according to claim 1, characterized in that: The step of extracting the image feature map and mask information of the image to be described includes: Extracting features of foreground objects and backgrounds in an image feature map to describe an image based on a panoramic segmentation algorithm to obtain correlations between the foreground objects and the background, correlations between different foreground objects, and mask information; Correspondingly, the using a preset encoder to perform image encoding on the target image feature map to obtain a corresponding image feature map encoding result includes: The preset encoder is used to process the correlation between the foreground object and the background and the correlation between different foreground objects to obtain a corresponding image feature map encoding result.

3. The image description method according to claim 1, characterized in that: The step of reducing the dimension of the image feature map and the mask information to obtain a target image feature map and target mask information as a position code includes: The image feature map and the mask information are input into a preset downsampling module to obtain a target image feature map after dimension reduction and target mask information as a position code.

4. The image description method according to claim 1, characterized in that: The using a preset encoder to perform image encoding on the target image feature map to obtain a corresponding image feature map encoding result includes: Normalizing all target image features in the target image feature map based on the first LN normalization layer to obtain normalized target image features; Using the first multi-head self-attention mechanism and the first IN normalization layer to perform feature calculation and normalization processing on the normalized target image features to obtain a relationship between the normalized target image features; The first FFN layer and the second IN normalization layer are used to process the relationship between the normalized target image features to obtain a corresponding image feature map encoding result.

5. The image description method according to claim 4, characterized in that: The using a preset encoder to perform image encoding on the target image feature map to obtain a corresponding image feature map encoding result includes: The target image feature map is encoded multiple times based on a preset number of encoding times to generate a corresponding image feature map encoding result.

6. The image description method according to claim 1, characterized in that: The step of inputting the splicing information and the text information into a preset decoder to generate a text description corresponding to the image to be described includes: Extracting text information from the pre-encoded text using a preset text extractor, and inputting the text information into the second multi-head self-attention mechanism and the second LN normalization layer to obtain normalized text information; Processing the splicing information and the normalized text information using the third multi-head self-attention mechanism, the third LN normalization layer, the second FFN layer, and the fourth LN normalization layer to obtain text features corresponding to different foreground objects and backgrounds in the image feature map; The text features corresponding to the image feature map are globally extracted using a fully connected layer to obtain global text features corresponding to the image feature map, and the global text features are input into a Softmax function to obtain a text score corresponding to the image to be described, and the corresponding text is found according to the text score to generate the text corresponding to the image to be described.

7. An image description device, characterized in that: include: An information acquisition module, used for extracting image feature maps and mask information of the image to be described based on a panoramic segmentation algorithm; An information dimension reduction module, used for reducing the dimension of the image feature map and the mask information to obtain a target image feature map and target mask information as a position code; An image encoding module, used to perform image encoding on the target image feature map using a preset encoder to obtain a corresponding image feature map encoding result; wherein the preset encoder includes a first LN normalization layer, a first multi-head self-attention mechanism, a first IN normalization layer, a first FFN layer, and a second IN normalization layer connected in sequence; An image description module is used to extract text information from a pre-encoded text using a preset text extractor, and input the text information, the target mask information and the image feature map encoding result into a preset decoder to generate text corresponding to the image to be described; wherein the preset decoder includes a second multi-head self-attention mechanism, a second LN normalization layer, a third multi-head self-attention mechanism, a third LN normalization layer, a second FFN layer, and a fourth LN normalization layer connected in sequence; The image description module is specifically used to splice the target mask information and the image feature map encoding result to obtain corresponding splicing information, and then input the splicing information and the text information into a preset decoder to generate a text description corresponding to the image to be described.

8. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the image description method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: Used to store a computer program; wherein, when the computer program is executed by a processor, the steps of the image description method according to any one of claims 1 to 6 are implemented.

Images