Image description generation method and device, electronic device and storage medium
By combining the multimodal feature fusion of object detection and text detection, image description information is generated, and the problem of dispersion and low context correlation of image description information in the prior art is solved, and the accuracy and quality of image description information is improved.
Patent Information
- Application Number
- CN202210282766.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-22
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2042-03-22
AI Technical Summary
In the prior art, the generation of image description information depends on the object detection model, resulting in dispersion of description statements, low context correlation of image description text, affecting the accuracy of image description information.
By combining object detection and text detection, the target area embedding vector and text embedding vector are obtained, stitching and feature fusion are performed, and semantic analysis is performed using a pre-trained image description generation model to generate initial image description information and filter it. Processed to remove repetitive verses.
The accuracy and quality of image description information are improved, and the context relevance and accuracy of description information are enhanced through multimodal feature fusion and text screening.
Smart Images

Figure CN114648631B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image description generation method and device, an electronic device, and a storage medium. Background Art
[0002] Image description refers to taking an image as input and outputting a natural language (English, Chinese, etc.) sentence that describes the image content through models and calculations. The generated sentence is a sentence that can describe the image content.
[0003] At present, the generation of most image description information relies on target detection models. The target detection model can better capture image details, but there are also problems such as the generated description sentences are relatively scattered and the relevance of the contextual information of the image description text is not high, which affects the accuracy of the image description information. Therefore, how to improve the accuracy of generated image description information has become a technical problem that needs to be solved urgently. Summary of the invention
[0004] The main purpose of the embodiments of the present application is to propose an image description generation method and device, an electronic device and a storage medium, aiming to improve the accuracy of generating image description information.
[0005] To achieve the above objective, a first aspect of an embodiment of the present application provides a method for generating an image description, the method comprising:
[0006] Get the original image;
[0007] Performing target detection on the original image to obtain a target region embedding vector;
[0008] Performing text detection on the original image to obtain a text embedding vector;
[0009] Concatenating the text embedding vector and the target region embedding vector to obtain a target vocabulary embedding vector;
[0010] Performing feature fusion on the text embedding vector, the target region embedding vector and the target vocabulary embedding vector to obtain a target feature vector;
[0011] Performing semantic analysis on the target feature vector through a pre-trained image description generation model to obtain initial image description information;
[0012] The initial image description information is screened to obtain target image description information.
[0013] In some embodiments, the step of performing target detection on the original image to obtain a target region embedding vector includes:
[0014] Inputting the original image into a preset target detection model; wherein the target detection model includes a convolution layer, an RPN layer, a RoI pooling layer and a fully connected layer;
[0015] Performing global feature extraction on the original image through the convolution layer to obtain multiple target feature maps;
[0016] Performing target detection on the target feature map through the RPN layer, the RoI pooling layer and preset coordinate parameters to obtain an image target detection frame;
[0017] Mapping the image object detection frame to a preset first vector space through the fully connected layer to obtain an initial detection embedding vector;
[0018] The initial detection embedding vector is subjected to dimension-changing processing by a bilinear peak interpolation method to obtain the target region embedding vector.
[0019] In some embodiments, the step of performing text detection on the original image to obtain a text embedding vector includes:
[0020] Inputting the original image into a preset text detection model, wherein the text detection model includes a text recognition network, a classification network and an encoding network;
[0021] Performing text recognition processing on the original image through the text recognition network to obtain text identification data;
[0022] Performing confidence calculation on the text identification data through the classification network to obtain first confidence data;
[0023] Performing confidence calculation on the text identification data through the encoding network to obtain second confidence data;
[0024] The text embedding vector is obtained according to the first confidence data and the second confidence data.
[0025] In some embodiments, the step of concatenating the text embedding vector and the target region embedding vector to obtain a target vocabulary embedding vector includes:
[0026] Mapping the text embedding vector to a preset second vector space to obtain a text feature vector; and mapping the target region embedding vector to the second vector space to obtain a target region feature vector;
[0027] The text feature vector and the target region feature vector are concatenated to obtain the target vocabulary embedding vector.
[0028] In some embodiments, the step of performing feature fusion on the text embedding vector, the target region embedding vector, and the target vocabulary embedding vector to obtain a target feature vector includes:
[0029] Inputting the text embedding vector, the target region embedding vector and the target vocabulary embedding vector into a preset feature fusion model; wherein the feature fusion model includes a first decoding layer and a transformer layer;
[0030] Decoding the target vocabulary embedding vector through the first decoding layer to obtain a target vocabulary latent variable;
[0031] The text embedding vector, the target region embedding vector, and the target vocabulary latent variable are subjected to feature fusion through the transformer layer to obtain the target feature vector.
[0032] In some embodiments, the image description generation model includes a second decoding layer, a first pointer network, a second pointer network and a linear layer, and the step of performing semantic analysis on the target feature vector by using the pre-trained image description generation model to obtain initial image description information includes:
[0033] Decoding the target feature vector through the second decoding layer to obtain a candidate image expression segment;
[0034] Performing vocabulary probability calculation on the candidate image expression word segment through the first pointer network and the preset target detection category to obtain a first probability value;
[0035] Calculating the vocabulary probability of the candidate image expression word segment by using the second pointer network and the preset identification tag to obtain a second probability value;
[0036] Calculating the vocabulary probability of the candidate image expression word segment through the linear layer and the preset dictionary to obtain a third probability value;
[0037] A target image expression segment is determined according to the first probability value, the second probability value, and the third probability value, and the initial image description information is generated according to the target image expression segment.
[0038] In some embodiments, the step of screening the initial image description information to obtain the target image description information includes:
[0039] Performing a repeatability check on the initial image description information to obtain a definition value of each image word in the initial image description information;
[0040] The image words are screened according to the definition value to obtain the target image description information.
[0041] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides an image description generating device, the device comprising:
[0042] An image acquisition module, used for acquiring original images;
[0043] A target detection module is used to perform target detection on the original image to obtain a target region embedding vector;
[0044] A text detection module, used to perform text detection on the original image to obtain a text embedding vector;
[0045] A concatenation module, used for concatenating the text embedding vector and the target region embedding vector to obtain a target vocabulary embedding vector;
[0046] A feature fusion module, used for performing feature fusion on the text embedding vector, the target region embedding vector and the target vocabulary embedding vector to obtain a target feature vector;
[0047] An image description generation module is used to perform semantic analysis on the target feature vector through a pre-trained image description generation model to obtain initial image description information;
[0048] The screening and processing module is used to screen and process the initial image description information to obtain target image description information.
[0049] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program, when executed by the processor, implements the method described in the first aspect above.
[0050] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium used for computer-readable storage, and the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method described in the first aspect above.
[0051] The image description generation method and device, electronic device and storage medium proposed in the present application obtain the original image; perform target detection on the original image to obtain the target area embedding vector, and perform text detection on the original image to obtain the text embedding vector. It can combine the two methods of target detection and text detection, and detect the target area and important text information of the original image at the same time, and extract the regional feature information and text feature information of the original image. Then, the text embedding vector and the target area embedding vector are spliced to obtain the target vocabulary embedding vector, and then the text embedding vector, the target area embedding vector and the target vocabulary embedding vector are feature fused to obtain the target feature vector. Thus, the target feature vector is semantically analyzed by the pre-trained image description generation model to obtain the initial image description information. In this way, the multimodal features of the original image can be conveniently feature fused to improve the accuracy of the generated image description information. Finally, the present application also screens the initial image description information, further performs text screening on the text content of the initial image description information, removes repeated words, and obtains the target image description information, which can further improve the quality of the generated image description information. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] Figure 1 is a flowchart of an image description generating method provided in an embodiment of the present application;
[0053] Figure 2 yes Figure 1 Flow chart of step S102 in FIG.
[0054] Figure 3 yes Figure 1 Flow chart of step S103 in FIG.
[0055] Figure 4 yes Figure 1 Flow chart of step S104 in FIG.
[0056] Figure 5 yes Figure 1 Flow chart of step S105 in FIG.
[0057] Figure 6 yes Figure 1 Flow chart of step S106 in FIG.
[0058] Figure 7 yes Figure 1 Flow chart of step S107 in FIG.
[0059] Figure 8 is a structural schematic diagram of an image description generating device provided in an embodiment of the present application;
[0060] Fig. 9It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION
[0061] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0062] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.
[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.
[0064] First, some nouns involved in this application are analyzed:
[0065] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0066] Natural language processing (NLP): NLP uses computers to process, understand and apply human languages (such as Chinese, English, etc.). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text understanding. Natural language processing is often used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.
[0067] Information Extraction (NER): A text processing technology that extracts specified types of entity, relationship, event and other factual information from natural language text and forms structured data output. Information extraction is a technology that extracts specific information from text data. Text data is composed of some specific units, such as sentences, paragraphs, and chapters. Text information is composed of some small specific units, such as characters, words, phrases, sentences, paragraphs, or a combination of these specific units. Extracting noun phrases, names, place names, etc. from text data are all text information extraction. Of course, the information extracted by text information extraction technology can be various types of information.
[0068] Computed Tomography (CT): It uses precisely collimated X-ray beams, gamma rays, ultrasound, etc., together with extremely sensitive detectors to perform cross-sectional scans around a certain part of the human body one by one. It has the characteristics of fast scanning time and clear images, and can be used to detect a variety of diseases. According to the different rays used, it can be divided into: X-ray CT (X-CT) and gamma-ray CT (γ-CT), etc.
[0069] Object Detection: The task of object detection is to find all the objects of interest in the image and determine their categories and locations. It is one of the core problems in the field of computer vision. The core problems of object detection include four categories, namely (1) Classification problem: that is, to which category the image in the picture (or a certain area) belongs. (2) Positioning problem: the object may appear at any position in the image. (3) Size problem: the object has various sizes. (4) Shape problem: the object may have various shapes. Object detection is divided into two major series: RCNN series and YOLO series. RCNN series is a representative algorithm based on region detection, and YOLO is a representative algorithm based on region extraction.
[0070] Bilinear interpolation: Bilinear interpolation uses the pixel values of the four neighboring points and assigns different weights according to their distance from the interpolation point for linear interpolation. This method has an averaged low-pass filtering effect, and the edges are smoothed to produce a more coherent output image.
[0071] Decoder: It converts the previously generated fixed vector into an output sequence; the input sequence can be text, voice, image, video; the output sequence can be text or image.
[0072] Transformer layer: The transformer layer includes an attention layer, an add&norm layer, a feed forward layer, and an add&norm layer that are adjacent in sequence. In the attention layer, P input vectors are obtained from the previous layer of the transformer layer. With any first input vector among the P input vectors as the center, the intermediate vector corresponding to the first input vector is obtained based on the correlation between each input vector within the preset attention window range and the first input vector, so that the P intermediate vectors corresponding to the P input vectors are determined; in the pooling layer, the P intermediate vectors are merged into Q output vectors, wherein the multiple output vectors obtained by the transformer layer are used as feature representations of the current input.
[0073] Image description refers to taking an image as input and outputting a natural language (English, Chinese, etc.) sentence that describes the image content through models and calculations. The generated sentence is a sentence that can describe the image content.
[0074] At present, the generation of most image description information relies on target detection models. The target detection model can better capture image details, but there are also problems such as the generated description sentences are relatively scattered and the relevance of the contextual information of the image description text is not high, which affects the accuracy of the image description information. Therefore, how to improve the accuracy of generated image description information has become a technical problem that needs to be solved urgently.
[0075] Based on this, the embodiments of the present application provide an image description generation method and device, an electronic device and a storage medium, aiming to improve the accuracy of generating picture text.
[0076] The image description generation method and device, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the image description generation method in the embodiments of the present application is described.
[0077] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0078] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0079] The image description generation method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The image description generation method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the image description generation method, etc., but is not limited to the above forms.
[0080] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.
[0081] Figure 1 is an optional flowchart of the image description generation method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S107.
[0082] Step S101, obtaining an original image;
[0083] Step S102, performing target detection on the original image to obtain a target region embedding vector;
[0084] Step S103, performing text detection on the original image to obtain a text embedding vector;
[0085] Step S104, concatenating the text embedding vector and the target region embedding vector to obtain a target vocabulary embedding vector;
[0086] Step S105, performing feature fusion on the text embedding vector, the target region embedding vector and the target vocabulary embedding vector to obtain a target feature vector;
[0087] Step S106, performing semantic analysis on the target feature vector through a pre-trained image description generation model to obtain initial image description information;
[0088] Step S107, screening the initial image description information to obtain target image description information.
[0089] The embodiment of the present application can realize the generation of target image description information through a preset multimodal image description generation model. The preset multimodal image description generation model can include three parts. The first part is the region detection and text detection part, which can be implemented based on the target detection model and the text detection model. For example, the target detection model is used to perform target detection on the original image to obtain a target region embedding vector, and the text detection model is used to perform text detection on the original image to obtain a text embedding vector; the second part is the feature fusion part, which can be implemented based on the feature fusion model. For example, the text embedding vector and the target region embedding vector are spliced through the feature fusion model to obtain a target vocabulary embedding vector, and the feature fusion model is used to perform feature fusion on the text embedding vector, the target region embedding vector and the target vocabulary embedding vector to obtain a target feature vector; the third part is the image description generation part, which can be implemented based on the image description generation model. For example, a pre-trained image description generation model is used to perform semantic analysis on the target feature vector to obtain initial image description information, and the initial image description information is screened to obtain the target image description information.
[0090] Steps S101 to S107 shown in the embodiment of the present application, by performing target detection on the original image to obtain the target area embedding vector, and performing text detection on the original image to obtain the text embedding vector, can combine the two methods of target detection and text detection, and simultaneously detect the target area and important text information of the original image, and extract the regional feature information and text feature information of the original image. The text embedding vector and the target area embedding vector are spliced to obtain the target vocabulary embedding vector, and then the text embedding vector, the target area embedding vector and the target vocabulary embedding vector are feature fused to obtain the target feature vector, and the target feature vector is semantically analyzed by the pre-trained image description generation model to obtain the initial image description information, which can conveniently fuse the multimodal features of the original image and improve the accuracy of the generated image description information. By screening the initial image description information, the text content of the initial image description information is further screened, and repeated words are eliminated to obtain the target image description information, which can improve the quality of the generated image description information.
[0091] In step S101 of some embodiments, the original image may be a three-dimensional image; in some embodiments, the three-dimensional image may be obtained through computed tomography (CT); in another embodiment, the three-dimensional image may be obtained through magnetic resonance imaging (MRI).
[0092] In some medical application scenarios, the above-mentioned original image may be a medical image, and the type of object contained in the original image is a lesion, that is, a part of the body where a disease occurs. Medical images refer to images of internal tissues obtained in a non-invasive manner for medical treatment or medical research, such as images of the stomach, abdomen, heart, knee, and brain, such as computed tomography (CT), magnetic resonance imaging (MRI), ultrasound (US), X-ray images, electroencephalograms, and optical photographic lights generated by medical instruments.
[0093] In step S101 of some embodiments, the original image may also be obtained from a preset image database, or by intercepting a video clip. The original image may be represented as I, I∈R 3×W×H , where W is the width of the original image, H is the height of the original image, and 3 is the number of channels of the original image.
[0094] See also Figure 2In some embodiments, step S102 may include but is not limited to steps S201 to S205:
[0095] Step S201, inputting the original image into a preset target detection model; wherein the target detection model includes a convolutional layer, an RPN layer, a RoI pooling layer, and a fully connected layer;
[0096] Step S202, extracting global features from the original image through a convolutional layer to obtain multiple target feature maps;
[0097] Step S203, performing target detection on the target feature map through the RPN layer, the RoI pooling layer and the preset coordinate parameters to obtain the image target detection frame;
[0098] Step S204, mapping the image object detection box to a preset first vector space through a fully connected layer to obtain an initial detection embedding vector;
[0099] Step S205, performing dimension-changing processing on the initial detection embedding vector by bilinear peak interpolation to obtain the target region embedding vector.
[0100] In step S201 of some embodiments, the original image is input into a preset target detection model, which may be a Faster R-CNN model, including a convolutional layer, an RPN layer, a RoI pooling layer, and a fully connected layer.
[0101] In step S202 of some embodiments, the original image is convolved by a convolution layer to capture the image features of the original image, extract m target features, and generate a target feature map corresponding to each target feature. The target feature map can be recorded as
[0102] In step S203 of some embodiments, in order to better cover the image spatial features and improve the accuracy of the generated image description, spatial features are introduced in the target detection process, that is, four coordinate parameters are preset, where the coordinate parameters can be expressed as These four coordinate parameters can represent the position information of the target point and the preset image anchor frame. m ,y m Respectively represent the horizontal and vertical coordinates of the center point of the preset image anchor box, w m Represents the width of the preset image anchor box, h mRepresents the height of the preset image anchor box, and the center point of the preset image anchor box is used as the image anchor point. The image anchor point of the target feature map is classified by the classification function of the RPN layer to obtain the label image feature, where the classification function can be a softmax function, etc., without limitation. Then, the label image feature is predicted by bounding box regression to obtain the offset of the image anchor point, and the target is located according to the offset and the label image, so as to obtain the initial detection box corresponding to the target feature map.
[0103] Furthermore, the features corresponding to each initial detection frame are selected on the target feature map through the RoI Pooling layer, and the feature dimension is set to a constant value. The four coordinate parameters are learned through linear regression so that the offset continuously approaches the real frame, thereby obtaining an accurate target detection frame, that is, the image target detection frame.
[0104] In step S204 of some embodiments, the image object detection box is mapped from the semantic space to the vector space through the MLP network of the fully connected layer, and the image object detection box is mapped to a preset first vector space to obtain an initial detection embedding vector.
[0105] In step S205 of some embodiments, the feature mapping area corresponding to the M target features in the initial detection embedding vector is adjusted to d dimensions by bilinear interpolation, so as to implement dimension-changing processing of the initial detection embedding vector, so that the initial detection embedding vector meets the preset vector dimension requirement, where d can be any value, for example, d is 256 or 512, etc., without limitation. In this way, the target area embedding vector can be obtained more conveniently. The target region embedding vector can be expressed as Among them, W1, W2, W3 are learning parameters, LN represents layer normalization, x conf-o It is a preset target corresponding confidence table, and the regional feature information of the original image can be reflected through the target region embedding vector.
[0106] See also Figure 3 In some embodiments, step S103 may include but is not limited to steps S301 to S305:
[0107] Step S301, inputting the original image into a preset text detection model, wherein the text detection model includes a text recognition network, a classification network and an encoding network;
[0108] Step S302, performing text recognition processing on the original image through a text recognition network to obtain text identification data;
[0109] Step S303, performing confidence calculation on the text identification data through a classification network to obtain first confidence data;
[0110] Step S304, performing confidence calculation on the text identification data through the encoding network to obtain second confidence data;
[0111] Step S305: Obtain a text embedding vector according to the first confidence data and the second confidence data.
[0112] In step S301 of some embodiments, the original image is input into a preset text detection model, which includes a text recognition network, a classification network and an encoding network, wherein the text recognition network can be constructed based on text recognition technology (text OCR technology), the classification network can be constructed based on the Fast Text model, and the encoding network can be constructed based on the PHOC model.
[0113] In step S302 of some embodiments, text recognition is first performed on the original image using character recognition software in a text recognition module to obtain candidate texts, and then the candidate texts are cleaned to remove text data with blurred characters in the candidate texts, and sentence expansion is performed on the incomplete text data in the candidate texts, for example, by performing synonym replacement, part of speech modification, and other operations to expand the sentences, and finally obtain text identification data.
[0114] In step S303 of some embodiments, the confidence of the text identification data is calculated by the layer classification function (such as the softmax function) of the Fast Text model to obtain the first confidence data, wherein the first confidence data includes the text feature Level Features and location features
[0115] In step S304 of some embodiments, the confidence of the text identification data is calculated by the PHOC model to obtain second confidence data, wherein the second confidence data includes a confidence table reflecting the regional features and text features of the original image.
[0116] In step S305 of some embodiments, the first confidence data and the second confidence data are superimposed to obtain a text embedding vector Specifically, for text features Target feature map And level characteristics After superposition, linear processing is performed to respectively analyze the position features Confidence Table Perform linear processing, that is, text embedding vector It can be expressed as
[0117]
[0118] Among them, W4, W5, W6, W7, W8 are learning parameters, and LN stands for layer normalization.
[0119] See also Figure 4 In some embodiments, step S104 may include but is not limited to steps S401 to S402:
[0120] Step S401, mapping the text embedding vector to a preset second vector space to obtain a text feature vector; and mapping the target region embedding vector to the second vector space to obtain a target region feature vector;
[0121] Step S402, concatenating the text feature vector and the target region feature vector to obtain a target vocabulary embedding vector.
[0122] In step S401 of some embodiments, the text embedding vector and the target area embedding vector may be first input into a preset feature fusion model, and the text embedding vector and the target area embedding vector may be mapped respectively through the fully connected layer of the feature fusion model. Specifically, the feature fusion model may be constructed based on a transformer model, including a fully connected layer, a first decoding layer, and a transformer layer. The text embedding vector and the target area embedding vector may be mapped from a semantic space to a vector space through the fully connected layer of the feature fusion model, and the text embedding vector may be mapped to a preset second vector space to obtain a text feature vector; and the target area embedding vector may be mapped to the second vector space to obtain a target area feature vector.
[0123] In step S402 of some embodiments, when the text feature vector and the target region feature vector are concatenated, the text feature vector and the target region feature vector may be added together to obtain the target vocabulary embedding vector.
[0124] See also Figure 5 In some embodiments, step S105 may also include but is not limited to steps S501 to S503:
[0125] Step S501, inputting the text embedding vector, the target region embedding vector and the target vocabulary embedding vector into a preset feature fusion model; wherein the feature fusion model includes a first decoding layer and a transformer layer;
[0126] Step S502, decoding the target vocabulary embedding vector through the first decoding layer to obtain the target vocabulary latent variable;
[0127] Step S503, performing feature fusion on the text embedding vector, the target region embedding vector, and the target vocabulary latent variable through the transformer layer to obtain a target feature vector.
[0128] In step S501 of some embodiments, the text embedding vector, the target region embedding vector and the target vocabulary embedding vector are input into a preset feature fusion model, wherein the feature fusion model can be constructed based on a transformer model, including a fully connected layer, a first decoding layer and a transformer layer.
[0129] In step S502 of some embodiments, the target vocabulary embedding vector is decoded by the first decoding layer to capture the vocabulary-level features of the target vocabulary embedding vector and obtain the target vocabulary latent variable
[0130] In step S503 of some embodiments, the text embedding vector x is transformed into ocr , target detection embedding vector x obj , target vocabulary latent variables Perform feature fusion to jointly generate feature vectors This jointly generated feature vector is used as the target feature vector, where is the target vocabulary latent variable output by the first decoding layer at the previous moment, and mmt represents the feature fusion function of the feature fusion model.
[0131] See also Figure 6 In some embodiments, the image description generation model includes a second decoding layer, a first pointer network, a second pointer network and a linear layer, and step S106 further includes but is not limited to steps S601 to S605:
[0132] Step S601, decoding the target feature vector through the second decoding layer to obtain a candidate image expression segment;
[0133] Step S602, calculating the vocabulary probability of the candidate image expression segment through the first pointer network and the preset target detection category to obtain a first probability value;
[0134] Step S603, calculating the vocabulary probability of the candidate image expression word segment through the second pointer network and the preset identification tag to obtain a second probability value;
[0135] Step S604, calculating the lexical probability of the candidate image expression word segment through a linear layer and a preset dictionary to obtain a third probability value;
[0136] Step S605 , determining a target image expression segment according to the first probability value, the second probability value, and the third probability value, and generating initial image description information according to the target image expression segment.
[0137] In step S601 of some embodiments, the target feature vector is decoded by the second decoding layer to capture the vocabulary-level features of the target feature vector and obtain candidate image expression segments. In order to improve the accuracy of the generated image description information, the generated image description information expression segments can be based on preset segments. in, Preset identification labels in, Preset target detection categories in, It is determined that R represents the R language, that is, the candidate image expression segments are respectively input into the first pointer network, the second pointer network and the linear layer for vocabulary probability calculation to realize the screening of the candidate image expression segments.
[0138] In step S602 of some embodiments, the first pointer network and the preset target detection category are used to calculate the vocabulary probability of the candidate image expression segment, and the weight ratio of the candidate image expression segment in each preset target detection category is calculated to obtain the first probability value of each preset target detection category. in, PN represents the pointer function of the first pointer network.
[0139] In step S603 of some embodiments, the lexical probability of the candidate image expression word segment is calculated by using the second pointer network and the preset identification tag, and the weight ratio of the candidate image expression word segment in each preset identification tag is calculated to obtain the second probability value of each preset identification tag. in, PN represents the pointer function of the second pointer network.
[0140] In step S604 of some embodiments, the lexical probability of the candidate image expression segment is calculated through the linear layer and the preset dictionary, and the weight proportion of each word in the preset dictionary of the candidate image expression segment is calculated to obtain the third probability value of the candidate image expression segment in the preset dictionary. in, W, b represent learning parameters respectively.
[0141] In step S605 of some embodiments, since the above-mentioned image description generation process considers three sources, namely, text detection, target detection, and text generation at the previous moment, it may cause the problem of repeated words or inaccurate predictions in the generated image description information. Therefore, by fitting the vocabulary scores of the three sources to avoid competition between the three sources, that is, when determining the target vocabulary according to the first probability value, the second probability value, and the third probability value, the score of each candidate image expression word segment is calculated to obtain the expression score The calculation process of the expression score of the nth candidate image expression segment can be shown as formula (1):
[0142]
[0143] in, is the third probability value of the nth candidate image expression segment;
[0144] A second probability value of a repeated word segment of a preset identification tag and a preset target detection category;
[0145] The third probability value of the repeated word segment of the preset target detection category and the preset dictionary.
[0146] According to the above calculation formula, the total score of the candidate image expression segments determined based on the preset segments, identification tags, and target detection categories is y t ∈R V+M+N .
[0147] Furthermore, based on the relationship between the total score of the candidate image expression segments and a preset score threshold, the candidate image expression segments whose total scores are greater than or equal to the preset score threshold are selected as target image expression segments, and the target image expression segments are spliced according to a preset splicing order to obtain initial image description information.
[0148] See also Figure 7 In some embodiments, step S107 may also include but is not limited to steps S701 to S702:
[0149] Step S701, performing a repeatability check on the initial image description information to obtain a definition value of each image word in the initial image description information;
[0150] Step S702: filter the image words according to the definition values to obtain target image description information.
[0151] In step S701 of some embodiments, at a preset time t, according to the total score y of the target image expression segment in the initial image description informationt , perform repetitive detection on the initial image description information to control whether the word can be generated at the current moment. Specifically, if the image word a has appeared before, then according to the preset repetitive detection mechanism M t,i,j Define image word a as negative infinity. Image word a will no longer appear in subsequent image information generation. If image word a has not appeared before, then according to the preset repetitive detection mechanism M t,i,j Define the image word a as 0, and retain the word a in the subsequent image information generation. This method can conveniently calculate the definition value of each image word in the initial image description information to obtain the definition value M of each image word t (M t is negative infinity or zero).
[0152] In step S702 of some embodiments, the image words are screened according to the definition value of each image word, and the image words with a definition value of negative infinity can be conveniently eliminated, thereby obtaining the target image description text. The target image description text can be expressed as t = argmax(y t +M t ), where argmax is the text generation function, y t is the total score of the target image expression segment, M t Defined value for image word.
[0153] The image description generation method of the embodiment of the present application obtains the original image; performs target detection on the original image to obtain the target area embedding vector, and performs text detection on the original image to obtain the text embedding vector. It can combine the two methods of target detection and text detection, and detect the target area and important text information of the original image at the same time, and extract the regional feature information and text feature information of the original image. Then, the text embedding vector and the target area embedding vector are spliced to obtain the target vocabulary embedding vector, and then the text embedding vector, the target area embedding vector and the target vocabulary embedding vector are feature fused to obtain the target feature vector, so that the target feature vector is semantically analyzed by the pre-trained image description generation model to obtain the initial image description information. In this way, the multimodal features of the original image can be conveniently feature fused to improve the accuracy of the generated image description information. Finally, the present application also screens the initial image description information, further performs text screening on the text content of the initial image description information, removes repeated words, and obtains the target image description information, which can further improve the quality of the generated image description information.
[0154] See also Figure 8The embodiment of the present application further provides an image description generating device, which can implement the above-mentioned image description generating method, and the image description generating device includes:
[0155] The image acquisition module 801 is used to acquire the original image;
[0156] The target detection module 802 is used to perform target detection on the original image and obtain a target region embedding vector;
[0157] A text detection module 803 is used to perform text detection on the original image to obtain a text embedding vector;
[0158] A concatenation module 804 is used to concatenate the text embedding vector and the target region embedding vector to obtain a target vocabulary embedding vector;
[0159] A feature fusion module 805 is used to perform feature fusion on the text embedding vector, the target region embedding vector and the target vocabulary embedding vector to obtain a target feature vector;
[0160] An image description generation module 806 is used to perform semantic analysis on the target feature vector through a pre-trained image description generation model to obtain initial image description information;
[0161] The screening processing module 807 is used to screen the initial image description information to obtain the target image description information.
[0162] In some embodiments, the object detection module 802 includes:
[0163] A first input unit is used to input the original image into a preset target detection model; wherein the target detection model includes a convolution layer, an RPN layer, a RoI pooling layer and a fully connected layer;
[0164] A global feature extraction unit, used to extract global features of the original image through a convolution layer to obtain multiple target feature maps;
[0165] The target detection unit is used to perform target detection on the target feature map through the RPN layer, the RoI pooling layer and the preset coordinate parameters to obtain the image target detection frame;
[0166] A first mapping unit, used to map the image object detection frame to a preset first vector space through a fully connected layer to obtain an initial detection embedding vector;
[0167] The variable dimension processing unit is used to perform variable dimension processing on the initial detection embedding vector by bilinear peak interpolation to obtain the target area embedding vector.
[0168] In some embodiments, the text detection module 803 includes:
[0169] A second input unit is used to input the original image into a preset text detection model, wherein the text detection model includes a text recognition network, a classification network and an encoding network;
[0170] A text recognition unit, used to perform text recognition processing on the original image through a text recognition network to obtain text identification data;
[0171] A first confidence calculation unit, used to perform confidence calculation on the text identification data through a classification network to obtain first confidence data;
[0172] A second confidence calculation unit, used to perform confidence calculation on the text identification data through the encoding network to obtain second confidence data;
[0173] The text embedding vector generating unit is used to obtain the text embedding vector according to the first confidence data and the second confidence data.
[0174] In some embodiments, the splicing module 804 includes:
[0175] A second mapping unit is used to map the text embedding vector to a preset second vector space to obtain a text feature vector; and map the target region embedding vector to the second vector space to obtain a target region feature vector;
[0176] The concatenation unit is used to concatenate the text feature vector and the target region feature vector to obtain the target vocabulary embedding vector.
[0177] In some embodiments, the feature fusion module 805 includes:
[0178] A third input unit is used to input the text embedding vector, the target region embedding vector and the target vocabulary embedding vector into a preset feature fusion model; wherein the feature fusion model includes a first decoding layer and a transformer layer;
[0179] A first decoding unit, configured to decode the target vocabulary embedding vector through a first decoding layer to obtain a target vocabulary latent variable;
[0180] The feature fusion unit is used to fuse the text embedding vector, the target region embedding vector, and the target vocabulary latent variable through the transformer layer to obtain the target feature vector.
[0181] In some embodiments, the image description generation model includes a second decoding layer, a first pointer network, a second pointer network and a linear layer, and the image description generation module 806 includes:
[0182] A decoding unit, used for decoding the target feature vector through the second decoding layer to obtain a candidate image expression segment;
[0183] A first vocabulary probability calculation unit, used to calculate the vocabulary probability of the candidate image expression segment through a first pointer network and a preset target detection category to obtain a first probability value;
[0184] A second vocabulary probability calculation unit, used to calculate the vocabulary probability of the candidate image expression segment through a second pointer network and a preset identification tag to obtain a second probability value;
[0185] A third vocabulary probability calculation unit, used to calculate the vocabulary probability of the candidate image expression word segment through a linear layer and a preset dictionary to obtain a third probability value;
[0186] The image expression word segment determination unit is used to determine the target image expression word segment according to the first probability value, the second probability value and the third probability value, and generate initial image description information according to the target image expression word segment.
[0187] In some embodiments, the screening processing module 807 includes:
[0188] A repeatability detection unit, used to perform repeatability detection on the initial image description information to obtain a definition value of each image word in the initial image description information;
[0189] The screening unit is used to screen the image words according to the definition value to obtain the target image description information.
[0190] The specific implementation of the image description generating device is substantially the same as the specific implementation of the above-mentioned image description generating method, and will not be described in detail here.
[0191] The embodiment of the present application also provides an electronic device, the electronic device comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program is executed by the processor to realize the above-mentioned image description generation method. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.
[0192] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:
[0193] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;
[0194] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the image description generation method of the embodiment of this application;
[0195] Input / output interface 903, used to implement information input and output;
[0196] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);
[0197] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);
[0198] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .
[0199] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned image description generation method.
[0200] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0201] The image description generation method, image description generation device, electronic device and storage medium proposed in the present application obtain the original image; perform target detection on the original image to obtain the target area embedding vector, and perform text detection on the original image to obtain the text embedding vector. It can combine the two methods of target detection and text detection, and detect the target area and important text information of the original image at the same time, and extract the regional feature information and text feature information of the original image. Then, the text embedding vector and the target area embedding vector are spliced to obtain the target vocabulary embedding vector, and then the text embedding vector, the target area embedding vector and the target vocabulary embedding vector are feature fused to obtain the target feature vector. Thus, the target feature vector is semantically analyzed by the pre-trained image description generation model to obtain the initial image description information. In this way, the multimodal features of the original image can be conveniently fused to improve the accuracy of the generated image description information. Finally, the present application also performs repetitive screening on the initial image description information, further performs text screening on the text content of the initial image description information, removes repetitive words, and obtains the target image description information, which can further improve the quality of the generated image description information.
[0202] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0203] It can be understood by those skilled in the art that Figure 1-7 The technical solutions shown in the figure do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figure, or a combination of certain steps, or different steps.
[0204] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0205] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.
[0206] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0207] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0208] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0209] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0210] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0211] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.
[0212] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.
Claims
1. A method for generating an image description, characterized in that: The method comprises: Get the original image; Performing target detection on the original image to obtain a target region embedding vector; Performing text detection on the original image to obtain a text embedding vector; Concatenating the text embedding vector and the target region embedding vector to obtain a target vocabulary embedding vector; Performing feature fusion on the text embedding vector, the target region embedding vector and the target vocabulary embedding vector to obtain a target feature vector; Performing semantic analysis on the target feature vector through a pre-trained image description generation model to obtain initial image description information; Screening the initial image description information to obtain target image description information; The performing target detection on the original image to obtain the target region embedding vector includes: Inputting the original image into a preset target detection model; wherein the target detection model includes a convolution layer, an RPN layer, a RoI pooling layer and a fully connected layer; The original image is subjected to global feature extraction through the convolution layer, m target features are extracted, and a target feature map corresponding to each target feature is generated, wherein the target feature map is recorded as , m is an integer greater than 0; Performing target detection on the target feature map through the RPN layer, the RoI pooling layer and preset coordinate parameters to obtain an image target detection frame; Mapping the image object detection frame to a preset first vector space through the fully connected layer to obtain an initial detection embedding vector; The initial detection embedding vector is subjected to dimension-changing processing by bilinear peak interpolation to obtain the target region embedding vector, which is used to indicate the region feature information of the original image. The target region embedding vector is recorded as , is the learning parameter, LN stands for layer normalization, The confidence table corresponding to the preset target, is a preset coordinate parameter, which is expressed as , Respectively represent the horizontal and vertical coordinates of the center point of the preset image anchor frame, Represents the width of the preset image anchor box. Represents the height of the preset image anchor box; The target feature map is subjected to target detection through the RPN layer, the RoI pooling layer and the preset coordinate parameters to obtain an image target detection frame, including: classifying the image anchor points of the target feature map through the classification function of the RPN layer to obtain label image features; performing border regression prediction on the label image features to obtain the offset of the image anchor points, performing target positioning according to the offset and the label image features, and obtaining an initial detection frame corresponding to the target feature map; selecting features corresponding to each initial detection frame on the target feature map through the RoI Pooling layer, setting the feature dimension to a fixed value, and obtaining the image target detection frame through linear regression learning of the preset coordinate parameters; The step of performing feature fusion on the text embedding vector, the target region embedding vector and the target vocabulary embedding vector to obtain a target feature vector includes: Inputting the text embedding vector, the target region embedding vector and the target vocabulary embedding vector into a preset feature fusion model; wherein the feature fusion model includes a first decoding layer and a transformer layer; Decoding the target vocabulary embedding vector through the first decoding layer to obtain a target vocabulary latent variable; The text embedding vector, the target region embedding vector, and the target vocabulary latent variable are subjected to feature fusion through the transformer layer to obtain the target feature vector.
2. The image description generation method according to claim 1, characterized in that: The step of performing text detection on the original image to obtain a text embedding vector comprises: Inputting the original image into a preset text detection model, wherein the text detection model includes a text recognition network, a classification network and an encoding network; Performing text recognition processing on the original image through the text recognition network to obtain text identification data; Performing confidence calculation on the text identification data through the classification network to obtain first confidence data; Performing confidence calculation on the text identification data through the encoding network to obtain second confidence data; The text embedding vector is obtained according to the first confidence data and the second confidence data.
3. The image description generating method according to claim 1, characterized in that: The step of concatenating the text embedding vector and the target region embedding vector to obtain a target vocabulary embedding vector comprises: Mapping the text embedding vector to a preset second vector space to obtain a text feature vector; and mapping the target region embedding vector to the second vector space to obtain a target region feature vector; The text feature vector and the target region feature vector are concatenated to obtain the target vocabulary embedding vector.
4. The image description generation method according to claim 1, characterized in that: The image description generation model includes a second decoding layer, a first pointer network, a second pointer network and a linear layer. The step of performing semantic analysis on the target feature vector through the pre-trained image description generation model to obtain initial image description information includes: Decoding the target feature vector through the second decoding layer to obtain a candidate image expression segment; Performing vocabulary probability calculation on the candidate image expression word segment through the first pointer network and the preset target detection category to obtain a first probability value; Calculating the vocabulary probability of the candidate image expression word segment by using the second pointer network and the preset identification tag to obtain a second probability value; Calculating the vocabulary probability of the candidate image expression word segment through the linear layer and the preset dictionary to obtain a third probability value; A target image expression segment is determined according to the first probability value, the second probability value, and the third probability value, and the initial image description information is generated according to the target image expression segment.
5. The image description generation method according to any one of claims 1 to 4, characterized in that: The step of screening the initial image description information to obtain the target image description information comprises: Performing a repeatability check on the initial image description information to obtain a definition value of each image word in the initial image description information; The image words are screened according to the definition value to obtain the target image description information.
6. An image description generating device, characterized in that: The device comprises: An image acquisition module, used for acquiring original images; A target detection module is used to perform target detection on the original image to obtain a target region embedding vector; A text detection module, used to perform text detection on the original image to obtain a text embedding vector; A concatenation module, used for concatenating the text embedding vector and the target region embedding vector to obtain a target vocabulary embedding vector; A feature fusion module, used for performing feature fusion on the text embedding vector, the target region embedding vector and the target vocabulary embedding vector to obtain a target feature vector; An image description generation module is used to perform semantic analysis on the target feature vector through a pre-trained image description generation model to obtain initial image description information; A screening and processing module, used for screening and processing the initial image description information to obtain target image description information; The performing target detection on the original image to obtain the target region embedding vector includes: Inputting the original image into a preset target detection model; wherein the target detection model includes a convolution layer, an RPN layer, a RoI pooling layer and a fully connected layer; The original image is subjected to global feature extraction through the convolution layer, m target features are extracted, and a target feature map corresponding to each target feature is generated, wherein the target feature map is recorded as , m is an integer greater than 0; Performing target detection on the target feature map through the RPN layer, the RoI pooling layer and preset coordinate parameters to obtain an image target detection frame; Mapping the image object detection frame to a preset first vector space through the fully connected layer to obtain an initial detection embedding vector; The initial detection embedding vector is subjected to dimension-changing processing by bilinear peak interpolation to obtain the target region embedding vector, which is used to indicate the region feature information of the original image. The target region embedding vector is recorded as , is the learning parameter, LN stands for layer normalization, The confidence table corresponding to the preset target, is a preset coordinate parameter, which is expressed as , Respectively represent the horizontal and vertical coordinates of the center point of the preset image anchor frame, Represents the width of the preset image anchor box. Represents the height of the preset image anchor box; The target feature map is subjected to target detection through the RPN layer, the RoI pooling layer and the preset coordinate parameters to obtain an image target detection frame, including: classifying the image anchor points of the target feature map through the classification function of the RPN layer to obtain label image features; performing border regression prediction on the label image features to obtain the offset of the image anchor points, performing target positioning according to the offset and the label image features, and obtaining an initial detection frame corresponding to the target feature map; selecting features corresponding to each initial detection frame on the target feature map through the RoI Pooling layer, setting the feature dimension to a fixed value, and obtaining the image target detection frame through linear regression learning of the preset coordinate parameters; The step of performing feature fusion on the text embedding vector, the target region embedding vector and the target vocabulary embedding vector to obtain a target feature vector includes: Inputting the text embedding vector, the target region embedding vector and the target vocabulary embedding vector into a preset feature fusion model; wherein the feature fusion model includes a first decoding layer and a transformer layer; Decoding the target vocabulary embedding vector through the first decoding layer to obtain a target vocabulary latent variable; The text embedding vector, the target region embedding vector, and the target vocabulary latent variable are subjected to feature fusion through the transformer layer to obtain the target feature vector.
7. An electronic device, characterized in that: The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the steps of the image description generating method as described in any one of claims 1 to 5 are realized.
8. A storage medium, the storage medium being a computer-readable storage medium, used for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the image description generation method according to any one of claims 1 to 5.
Citation Information
Patent Citations
Image description generation method, system and device and storage medium
CN111368118A
Text similarity calculation method and device, electronic equipment and storage medium
CN113887215A
