Image description method and device, equipment, storage medium and program product

Abstract

The invention discloses an image description method and device, equipment, a storage medium and a program product. According to the specific technical scheme, the method comprises the steps of obtaining a to-be-recognized image; extracting regional features and grid features of the to-be-recognized image by using the image description generation model; utilizing an image description generation model to predict and obtain an input text based on the region features and the grid features; calculating through a self-attention mechanism to obtain self-attention features of the input text; respectively calculating a first cross attention feature of the self-attention feature and the region feature and a second cross attention feature of the self-attention feature and the grid feature; splicing the first cross attention feature and the second cross attention feature to obtain a spliced feature; and generating image description information corresponding to the to-be-recognized image through the splicing features. Therefore, the natural language description can be generated by combining the context information in the image and the detail information of the object image content, and the accuracy of the image description language is improved.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and in particular relates to a method, apparatus, device, storage medium, and program product for image description. Background Art

[0002] Image captioning tasks involve converting input image content into natural language descriptions. Currently, image captioning tasks have broad application prospects in multiple fields. For example, in the field of robotic health consultations, robots can use image captioning tasks to identify medical images and quickly identify the location of patient lesions.

[0003] Currently, image description tasks are primarily implemented using methods based on modality translation. Specifically, an encoder-decoder architecture is employed, where the encoder converts the image to be recognized into an intermediate modality, and the decoder decodes the intermediate modality to obtain the corresponding natural language description.

[0004] Current image description methods based on modal translation often suffer from the problem of inaccurate natural language descriptions generated. Summary of the Invention

[0005] The embodiments of the present application provide a method, apparatus, device, storage medium, and program product for describing an image, which can generate a natural language description by combining contextual information in an image and detailed information about the object image content, thereby improving the accuracy of the image description language.

[0006] In a first aspect, an embodiment of the present application provides a method for describing an image, including: Obtain the image to be recognized; Extracting regional features and grid features of the image to be identified using an image description generation model, wherein the grid features are used to represent context information of the image to be identified, and the regional features are used to represent object image content information contained in the image to be identified; Predicting input text based on the region features and the grid features using the image description generation model; Obtaining a self-attention feature of the input text by calculating the self-attention mechanism in the image description generation model; Calculating, by the image description generation model, a first cross-attention feature of the self-attention feature and the region feature, and a second cross-attention feature of the self-attention feature and the grid feature; splicing the first cross-attention feature and the second cross-attention feature through the image description generation model to obtain a spliced feature; Image description information corresponding to the image to be identified is generated using the image description generation model and the splicing features.

[0007] In a possible implementation, before extracting the region features and grid features of the image to be identified using the image description generation model, the method further includes: Scaling the image to be identified according to a preset size to obtain a scaled image; Performing feature extraction on the scaled image to obtain a first feature map; Performing dimensionality reduction processing on the first feature map to obtain a second feature map; Performing an expansion operation on the second feature map to obtain a third feature map; The extracting of the regional features and grid features of the image to be identified includes: The region features and the grid features of the third feature map are extracted.

[0008] In one possible implementation, calculating the self-attention feature of the input text through the self-attention mechanism in the image description generation model includes: Input the input text into the embedding layer to obtain an embedding vector corresponding to the input text; Calculating the position code of the input text according to the position of the text between the input texts; Summing the embedding vector and the positional encoding to obtain an input tensor; The input tensor is input into the self-attention layer to obtain the self-attention feature.

[0009] In a possible implementation, the generating of image description information corresponding to the image to be identified by using the image description generation model and the splicing features includes: Performing dimensionality reduction processing on the splicing features to obtain dimensionality reduction features; Performing residual operations and normalization operations on the dimensionality reduction features and the self-attention features to obtain output features; Inputting the output features into a feedforward network to obtain features to be identified output by the feedforward network; The features to be identified are input into a preset activation function to obtain the image description information.

[0010] In a possible implementation, before extracting the region features and grid features of the image to be identified using the image description generation model, the method further includes: Get the training dataset; Using the training data set to train the Transformer model to obtain a training prediction result; Calculating a focal loss function value using the label data corresponding to the training data set, the training prediction result, and a preset focal loss function; When the focus loss function value satisfies a preset iteration stopping condition, the image description generation model is obtained.

[0011] In one possible implementation, calculating the focus loss function value using the label data corresponding to the training data set, the training prediction result, and a preset focus loss function includes: The focus loss function value is calculated according to the following formula:

[0012] in, is the focal loss function value, is the first regulating factor, , is the second regulatory factor, , is the label data corresponding to the training prediction result, Predict results for the training.

[0013] In a second aspect, an embodiment of the present application provides an image description device, including: An acquisition module, used to acquire an image to be identified; an extraction module, configured to extract regional features and grid features of the image to be identified using an image description generation model, wherein the grid features are used to represent context information of the image to be identified, and the regional features are used to represent object image content information contained in the image to be identified; A prediction module, configured to predict an input text based on the region features and the grid features using the image description generation model; A calculation module, configured to calculate the self-attention features of the input text through the self-attention mechanism in the image description generation model; The calculation module is further configured to calculate, by using the image description generation model, a first cross-attention feature of the self-attention feature and the region feature, and a second cross-attention feature of the self-attention feature and the grid feature; a splicing module, configured to splice the first cross-attention feature and the second cross-attention feature using the image description generation model to obtain a spliced feature; A generation module is used to generate image description information corresponding to the image to be identified through the image description generation model and the splicing features.

[0014] In a third aspect, an embodiment of the present application provides a terminal device, the device comprising: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the method described in the image of the first aspect is implemented.

[0015] In a fourth aspect, an embodiment of the present application provides a computer storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method for describing an image according to the first aspect is implemented.

[0016] In a fifth aspect, an embodiment of the present application provides a computer program product. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the image description method as described in the first aspect.

[0017] An image description method, apparatus, device, storage medium and program product of an embodiment of the present application, after acquiring the image to be identified, respectively extracts the region features and grid features of the image to be identified, wherein the region features are used to represent the object image content information contained in the image to be identified, and the grid features are used to represent the context information in the image to be identified. In this way, after the input text is subsequently predicted, the first cross-attention features of the region features and the self-attention features of the input text, as well as the second cross-attention features of the grid features and the self-attention features are respectively calculated, wherein the first cross-attention features and the second cross-attention features can calibrate the input text from the contextual connection dimension of the image to be identified and the detail dimension of the object image content area, respectively. In this way, after the first cross-attention features and the second cross-attention features are subsequently spliced together to obtain the spliced features, the spliced features simultaneously characterize the grid features and the region features of the image to be identified, thereby ensuring the accuracy of the generated image description information. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0019] Figure 1 This is a flow chart of an image description method provided in an embodiment of the present application; Figure 2 This is a flow chart of a method for preprocessing an image to be identified provided in an embodiment of the present application; Figure 3 is an exemplary schematic diagram of a method for preprocessing an image to be identified provided in an embodiment of the present application; Figure 4 is an exemplary schematic diagram of a feature extraction method provided in an embodiment of the present application; Figure 5 is an exemplary schematic diagram of an input tensor calculation method provided in an embodiment of the present application; Figure 6This is an exemplary schematic diagram of an output feature calculation method provided in an embodiment of the present application; Figure 7 is an exemplary schematic diagram of a feedforward network structure provided in an embodiment of the present application; Figure 8 This is a flow chart of a model training method provided in an embodiment of the present application; Figure 9 is an exemplary schematic diagram of a model training method provided in an embodiment of the present application; Figure 10 This is an exemplary schematic diagram of an image description generation model structure provided in an embodiment of the present application; Figure 11 Schematic diagram of the structure of an image description device provided in an embodiment of the present application; Figure 12 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is merely to provide a better understanding of the present application by illustrating the examples of the present application.

[0021] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0022] Currently, image description methods mainly use the following two methods to generate natural language descriptions of images: The first method is to pre-build a database containing the correspondence between images and natural language descriptions, calculate the similarity between the image to be identified and the images stored in the database, obtain the target image with the highest similarity, reorganize the natural language description corresponding to the target image, and obtain the natural language description of the image to be identified.

[0023] The first method mentioned above is limited by the pre-built database. Since the images in the database cannot effectively cover different scenes, it is difficult to generate accurate natural language descriptions.

[0024] The second method is to pre-train the model using cross-entropy loss to obtain a target model. The target model is used to extract the grid features of the image to be recognized and then generate the corresponding natural language description.

[0025] In the second approach, since grid features represent the contextual information of the object image content in the image to be identified, the natural language description generated based on grid features will lose local details of the object image content, resulting in inaccurate natural language descriptions. Furthermore, image description tasks suffer from a long-tail distribution phenomenon, whereby some object image content or scene image content appears much more frequently than other image content. Therefore, the target model trained using cross-entropy loss tends to predict frequently occurring object categories, resulting in inaccurate natural language descriptions.

[0026] In order to solve the problems of the prior art, the embodiments of the present application provide a method, apparatus, device, storage medium and program product for describing an image. The following first describes the method for describing an image provided by the embodiments of the present application. Figure 1 As shown, the method is applied to an electronic device, and the method includes: S101: Obtain an image to be recognized.

[0027] S102: Utilize an image description generation model to extract regional features and grid features of the image to be identified.

[0028] Among them, the grid feature is used to represent the context information of the image to be identified, and the region feature is used to represent the object image content information contained in the image to be identified.

[0029] S103: Utilize the image description generation model to predict input text based on the region features and the grid features.

[0030] Among them, the image description generation model includes a decoder and an encoder. The encoder is used to extract region features and grid features, and the decoder is used to generate input text. The generated input text is then calibrated according to the region features and grid features to obtain image description information.

[0031] S104. Calculate the self-attention features of the input text through the self-attention mechanism in the image description generation model.

[0032] It can be understood that the self-attention mechanism can calculate the association relationship between each text in the input text. In this way, the independent word embedding corresponding to each text in the input text can be converted into context-related features, and the dependency relationship of the entire input text can be constructed to avoid grammatical errors in the generated image description information.

[0033] S105. Calculate the first cross-attention features of the self-attention features and the region features and the second cross-attention features of the self-attention features and the grid features respectively through the image description generation model.

[0034] Specifically, by calculating cross-attention features, the input text can be compared with both the region features and the grid features, preventing image alignment failures. For example, if the input text contains the word "dog," calculating cross-attention features can prevent alignment of the input text with the image content region of "cat" in the image to be recognized.

[0035] S106. Splice the first cross-attention feature and the second cross-attention feature through the image description generation model to obtain a spliced feature.

[0036] Specifically, the first cross-attention feature and the second cross-attention feature are aligned and spliced according to the channel dimension to obtain a spliced feature.

[0037] S107: Generate image description information corresponding to the image to be identified by using the image description generation model and the splicing features.

[0038] Using the above method, after obtaining the image to be identified, the regional features and grid features of the image to be identified are extracted respectively, wherein the regional features are used to represent the object image content information contained in the image to be identified, and the grid features are used to represent the context information in the image to be identified. In this way, after the input text is subsequently predicted, the first cross-attention features of the regional features and the self-attention features of the input text, as well as the second cross-attention features of the grid features and the self-attention features are calculated respectively, wherein the first cross-attention features and the second cross-attention features can calibrate the input text from the contextual connection dimension of the image to be identified and the detail dimension of the object image content area, respectively. In this way, after the first cross-attention features and the second cross-attention features are spliced together to obtain the spliced features, the spliced features simultaneously characterize the grid features and the regional features of the image to be identified, thereby ensuring the accuracy of the generated image description information.

[0039] After obtaining the image to be identified, the image to be identified is preprocessed so that the preprocessed image to be identified meets the input image requirements of the image generation model. Figure 2 As shown, before the above S102, extracting the regional features and grid features of the image to be identified by using the image description generation model, the method further includes: S201 , scaling the image to be recognized according to a preset size to obtain a scaled image.

[0040] S202 to S204 are processes for preprocessing the image to be recognized, and the preprocessing process can be processed by a Shifted Windows Transformer (Swin Transformer) model.

[0041] Specifically, the preset size is determined based on the input requirements of the Swin Transformer model. The preset size can be set according to actual needs.

[0042] S202: Perform feature extraction on the scaled image to obtain a first feature map.

[0043] S203: Perform dimensionality reduction processing on the first feature map to obtain a second feature map.

[0044] It should be noted that after the image description generation model generates the first feature map, since the first feature map contains a large number of channels, dimensionality reduction processing can be used to reduce the number of channels contained in the first feature map, thereby reducing the data processing amount of the image description generation model and improving the processing efficiency of the image description generation model.

[0045] S204: Perform an expansion operation on the second feature map to obtain a third feature map.

[0046] Accordingly, the regional features and grid features of the image to be identified are extracted, including: Extract regional features and grid features of the third feature map.

[0047] By using the method provided in the embodiment of the present application, before the image to be identified is input into the image description generation model, the image to be identified is scaled according to a preset size, and then the key features in the image to be identified can be effectively extracted through feature extraction, dimensionality reduction operations, and expansion operations, thereby reducing the number of features processed by the image description generation model and improving processing efficiency.

[0048] The following combination Figure 3 Introduce the above preprocessing process, such as Figure 3As shown, the electronic device obtains an image to be recognized, which is an image of a physical object captured by a camera. The image to be recognized is scaled to a size of 384×384 pixels. The scaled image is then input into the Swin Transformer for feature extraction and dimensionality reduction, resulting in a first feature map with a size of 12×12×1024.

[0049] Specifically, the Swin Transformer divides the scaled image into 12×12 blocks according to the pre-configured patch size, and then obtains the 1024-dimensional features corresponding to each block through linear embedding or staged downsampling.

[0050] The first feature map is then input into the linear layer to obtain a second feature map after dimensionality reduction, where the second feature map is stored as 12×12×512. In this way, the dimension contained in the feature map is further reduced and the key features are retained.

[0051] The second feature is expanded to obtain a third feature map of 144×512, that is, each spatial position corresponds to a 512-dimensional feature vector.

[0052] After the above preprocessing, the image description generation model extracts the regional features and grid features of the third feature map, as follows Figure 4 As shown in the figure, after generating the third feature map, the feature map is input into the Vision Transformer model and the Detection Transformer (DETR) model. The Vision Transformer model extracts grid features, which include features from the first to fourth images, where the first to fourth images are uniformly segmented. The DETR model then extracts region features. Assuming that the image to be identified contains both cat and dog images, the region features include features from both cat and dog images. The grid features are sized 144×512, meaning they contain 144 spatial locations, each corresponding to a 512-dimensional feature vector. The region features are sized 100×512, meaning they contain 100 spatial locations, each corresponding to a 512-dimensional feature vector.

[0053] In some embodiments of the present application, with respect to the above S104, the self-attention features of the input text are calculated by the self-attention mechanism in the image description generation model, which can be specifically implemented as follows: Step 1: Input the input text into the embedding layer to obtain the embedding vector corresponding to the input text.

[0054] Step 2: Calculate the position code of the input text according to the position of the text between the input texts.

[0055] It should be noted that the present application does not limit the method of calculating the position code. For example, the position code can be calculated by calculating the sine value.

[0056] Step 3: Sum the embedding vector and the positional encoding to obtain the input tensor.

[0057] The following combination Figure 5 Introduce the process of obtaining the input tensor by the above calculation, such as Figure 5 As shown in the figure, during the model training process, the model uses the real labeled text shifted right by one as the input text. During the model inference process, the model uses the start tag <sos>, regional features and grid features to predict the first text, and then add the predicted first text to the input sequence through iterative generation, re-input the input sequence into the model, predict the second text, and repeat the above steps until the end marker is output <eos>.

[0058] Figure 5 For the model training process, after the real annotated text is input into the model, the input text is shifted right by one position and then input into the embedding layer to obtain the embedding vector. For example, if the input text is "a dog and a cat are sitting on the ground.", the input text after right shift processing is " <sos>a dog and a cat are sitting on the ground." The embedding vector output by the embedding layer has a size of 100×512. To obtain the position information between each text in the input text, the position encoding corresponding to each text is calculated, and then the position encoding and the embedding vector are summed to obtain the input tensor, which has a size of 100×512.

[0059] Step 4: Input the input tensor into the self-attention layer to obtain the self-attention feature.

[0060] The calculation formula of the self-attention mechanism is as follows:

[0061] in, The input tensor is obtained by linear transformation through the linear layer. . for The number of channels, is the i-th element, is the jth element.

[0062] The corresponding transformation formula is as follows:

[0063] in, for The corresponding linear transformation matrix.

[0064] The corresponding transformation formula is as follows:

[0065] in, for The corresponding linear transformation matrix.

[0066] The corresponding transformation formula is as follows:

[0067] in, for The corresponding linear transformation matrix.

[0068] Using the method provided in the embodiment of the present application, when the input text is predicted, the position code of the input text can be calculated based on the positional relationship between the texts in the input text, and then the position code and the embedding vector are summed. The positional relationship between the texts in the input text can be added to the input tensor, thereby ensuring the accuracy of the natural language description generated subsequently.

[0069] It should be noted that the above-mentioned image description generation model also includes a feedforward network. The above-mentioned S107, generating image description information corresponding to the image to be identified by using the image description generation model and the splicing features, can be specifically implemented as follows: Step A: Perform dimensionality reduction processing on the splicing features to obtain dimensionality reduction features.

[0070] Step B: Perform residual operation and normalization operation on the dimensionality reduction features and self-attention features to obtain output features.

[0071] Specifically, the output features are obtained by processing the splicing features by the feature fusion module in the image description generation model. The feature fusion module obtains the output features by processing according to the following steps, such as Figure 6 As shown: After calculating the self-attention feature XAtten, the cross-attention of XAtten and G and the cross-attention of XAtten and R are calculated to obtain the first cross-attention feature and the second cross-attention feature, where G represents the grid feature and R represents the region feature. The first cross-attention feature and the second cross-attention feature are then concatenated according to the channel dimension to obtain the concatenated feature. Finally, the dimensionality reduction feature corresponding to the concatenated feature and the self-attention feature are subjected to residual operation and normalization operation to obtain the output feature.

[0072] Specifically, the formula for calculating the cross-attention feature refers to the formula for calculating the self-attention feature mentioned above. It should be noted that when calculating the cross-attention feature, Q is obtained by linear transformation of XAtten, and K and V are obtained by linear transformation of G or R. The size of Q is 100×512, the size of K and V obtained by G transformation is 144×512, and the size of K and V obtained by R transformation is 100×512. The size of the spliced feature is 100×1024, and the size of the dimensionality reduction feature is 100×512.

[0073] Through Figure 6 The feature fusion module shown in the figure can effectively combine the contextual information of the grid features and the local features of the regional features to enhance the model's understanding of the image content and improve the accuracy of the generated natural language description.

[0074] Step C: Input the output features into the feedforward network to obtain the features to be identified output by the feedforward network.

[0075] Specifically, the feedforward network consists of two linear layers and an activation function layer, such as Figure 7 As shown, in the embodiment of the present application, the activation function layer adopts the GULE activation function. After the input feature passes through the first linear layer, the first linear layer performs a dimensionality increase process on the input feature, for example, converting the channel data of the input feature from 512 to 2048. Then, the second linear layer performs a dimensionality reduction operation on the dimensionality increased feature, for example, converting the channel data from 2048 to 512, thereby obtaining the feature to be recognized.

[0076] Step D: Input the features to be identified into a preset activation function to obtain image description information.

[0077] It should be noted that after acquiring the features to be recognized, the image description generation model inputs the features to be recognized into a linear layer, obtains the linearly transformed features, and then inputs the linearly transformed features into a preset activation function as the features to be recognized, thereby obtaining image description information. The linear layer is used to convert the number of channels of the features to be recognized into the length of a vocabulary, which includes different words. The image description generation model then predicts the probabilities of different words in the vocabulary and selects the word with the highest probability as the model's prediction result.

[0078] Using the method provided in the embodiments of this application, after performing residual operations and normalization on the splicing and self-attention features, the resulting output features are fed into a feedforward network, which then generates text to obtain the features corresponding to the image to be recognized. Subsequently, the features to be recognized are fed into a preset activation function to generate the corresponding image description information.

[0079] The following combination Figure 8 Introduce the training process of the image description generation model provided in the embodiment of this application, such as Figure 8 As shown, before the above S102, in which the image description generation model is used to extract the regional features and grid features of the image to be identified, the training process of the image description generation model is as follows: S801: Obtain a training data set.

[0080] S802: Use the training data set to train the Transformer model to obtain training prediction results.

[0081] S803. Calculate the focus loss function value using the label data corresponding to the training data set, the training prediction results, and the preset focus loss function.

[0082] Among them, the calculation of the focus loss function value is as follows:

[0083] in, is the focal loss function value, n represents the number of vocabulary categories, is the first regulating factor, , is the second regulatory factor, , For the label data corresponding to the training prediction results, is the training prediction result. , Greater than 0.

[0084] It should be noted that the first adjustment factor and the second adjustment factor are used to balance the impact of different samples on the loss. Since the number of negative samples with a label value of 0 is much greater than the number of positive samples with a label value of 1 in the image description task, the distribution of the focus loss function can be adjusted by adjusting the first adjustment factor. , n is the number of negative samples, m is the number of positive samples, for the text at the head of the long-tail distribution, the corresponding output probability approaches 1, at this time The value of is close to 0. For the text at the tail of the long-tail distribution, the corresponding output probability is close to 0. The value of approaches 1. That is, for the text at the tail of the long-tail distribution, the loss function value does not change significantly, while for the text at the head of the long-tail distribution, the loss function value decreases, which is equivalent to increasing the weight of the text at the tail of the long-tail distribution in the loss function, thereby alleviating the problem of the long-tail distribution.

[0085] S804. When the focus loss function value satisfies a preset iteration stopping condition, an image description generation model is obtained.

[0086] The preset iteration stopping condition is the maximum number of iterations or the focus loss function value is less than a preset threshold. Specifically, the preset iteration stopping condition is set according to actual business needs.

[0087] Among them, the process of training the model using the focal loss function is as follows: Figure 9 As shown in the figure, after the image is input, the model predicts the output result, calculates the focus loss function value based on the label and the output result of the model, and updates the model parameters based on the focus loss function value through backpropagation. The calculation is iterated until the calculated focus loss function value meets the preset iteration stop condition, and a trained image description generation model is obtained.

[0088] According to an embodiment of the present application, the Transformer model is iteratively trained using a preset focus loss function until the calculated focus loss function value satisfies a preset iteration stop condition, indicating that the accuracy of the prediction result of the current Transformer model is high. Therefore, the trained Transformer model can be used as the above-mentioned image description generation model. In addition, the preset focus loss function includes a first adjustment factor and a second adjustment factor. The electronic device can dynamically adjust the first adjustment factor and the second adjustment factor in real time according to the training prediction result and the label data to adjust the weights of positive and negative samples, thereby avoiding the Transformer model from focusing too much on negative samples, resulting in inaccurate model prediction results, and alleviating the problem of inaccurate prediction results due to long-tail distribution.

[0089] like Figure 10 As shown, Figure 10 This is a diagram of the overall structure of the image description generation model provided in an embodiment of the present application, wherein the image description generation model includes a Swin Transformer model, a Vision Transformer model, a DETR model, an embedding layer, a self-attention layer, a feature extraction module, a feedforward network layer, an activation function softmax layer, and a focal loss calculation module.

[0090] The Swin Transformer model is used to preprocess the received image to be recognized. The specific preprocessing process is described in the above embodiment and will not be repeated here. The Vision Transformer model and DETR model are used to extract grid features and regional features respectively, and the extracted grid features and regional features are input into the feature fusion module; The embedding layer is used to generate the embedding vector of the input text obtained by the model's initial prediction; The self-attention layer is used to calculate the self-attention features of the input text; The feature fusion module is used to generate output features. The specific method of generating output features is described in the above embodiment and will not be repeated here. The feedforward network is used to further process the output features to obtain the features to be identified.

[0091] The softmax layer is used to generate natural language descriptions; The focus loss calculation module is used to calculate the focus loss function value based on the generated natural language description and label, and backpropagate and update the model parameters according to the calculated focus loss function value until the number of iterations meets the preset iteration stop condition or the focus loss function value meets the preset iteration stop condition.

[0092] The above image description generation model can be used to combine the grid features and region features of the image to be identified, calibrate the input text generated initially, and add more descriptive words. Taking the image to be identified including an airplane as an example, Figure 10 The image description generation model shown in the figure generates a natural language description of "a red, white, and blue airplane is flying in the blue sky." Compared to the natural language description "an airplane is flying in the sky" generated by the prior art, it adds a description of the airplane's color. For an image to be recognized that includes a person and a swan, the generated natural language description of "a man taking a picture of a swan next to a lake" adds contextual information compared to the natural language description "a man taking a picture of a swan" generated by the prior art. Therefore, the image description method provided by the embodiments of the present application can generate a more complete and accurate natural language description.

[0093] Based on the same concept, the embodiment of the present application provides an image description device, such as Figure 11 As shown, the device includes: An acquisition module 1101 is used to acquire an image to be recognized; An extraction module 1102 is configured to extract regional features and grid features of the image to be identified using an image description generation model, wherein the grid features are used to represent context information of the image to be identified, and the regional features are used to represent object image content information contained in the image to be identified; Prediction module 1103, configured to predict input text based on the region features and the grid features using the image description generation model; A calculation module 1104 is configured to calculate a self-attention feature of the input text through a self-attention mechanism in the image description generation model; The calculation module 1104 is further configured to calculate, by using the image description generation model, a first cross-attention feature of the self-attention feature and the region feature, and a second cross-attention feature of the self-attention feature and the grid feature; a splicing module 1105 , configured to splice the first cross-attention feature and the second cross-attention feature using the image description generation model to obtain a spliced feature; The generating module 1106 is configured to generate image description information corresponding to the image to be identified by using the image description generating model and the splicing features.

[0094] In a possible implementation, the device further includes: A scaling module, configured to scale the image to be identified according to a preset size to obtain a scaled image; A feature extraction module, configured to extract features from the zoomed image to obtain a first feature map; A dimensionality reduction module, configured to perform dimensionality reduction processing on the first feature map to obtain a second feature map; an expansion module, configured to perform an expansion operation on the second feature map to obtain a third feature map; The extraction module 1102 is specifically configured to: The region features and the grid features of the third feature map are extracted.

[0095] In one possible implementation, the calculation module 1104 is specifically configured to: Input the input text into the embedding layer to obtain an embedding vector corresponding to the input text; Calculating the position code of the input text according to the position of the text between the input texts; Summing the embedding vector and the positional encoding to obtain an input tensor; The input tensor is input into the self-attention layer to obtain the self-attention feature.

[0096] In a possible implementation, the splicing module 1105 is specifically configured to: Performing dimensionality reduction processing on the splicing features to obtain dimensionality reduction features; Performing residual operations and normalization operations on the dimensionality reduction features and the self-attention features to obtain output features; Inputting the output features into a feedforward network to obtain features to be identified output by the feedforward network; The features to be identified are input into a preset activation function to obtain the image description information.

[0097] In a possible implementation, the device further includes: The acquisition module 1101 is further used to acquire a training data set; A training module, configured to train a Transformer model using the training data set to obtain training prediction results; The calculation module 1104 is further configured to calculate a focus loss function value using the label data corresponding to the training data set, the training prediction result, and a preset focus loss function; An iteration module is used to obtain the image description generation model when the focus loss function value meets a preset iteration stopping condition.

[0098] In one possible implementation, the calculation module 1104 is specifically configured to: The focus loss function value is calculated according to the following formula:

[0099] in, is the focal loss function value, is the first regulating factor, , is the second regulatory factor, , is the label data corresponding to the training prediction result, Predict results for the training.

[0100] It should be noted that the image description device is a device corresponding to the above-mentioned image description method. All implementation methods in the above-mentioned method embodiments are applicable to the embodiments of the device and can achieve the same technical effects.

[0101] Figure 12 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown.

[0102] The electronic device may include a processor 1201 and a memory 1202 storing computer program instructions.

[0103] Specifically, the processor 1201 may include a central processing unit (CPU) or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0104] Memory 1202 may include a large-capacity memory for data or instructions. By way of example and not limitation, memory 1202 may include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 1202 may include removable or non-removable (or fixed) media. Where appropriate, memory 1202 may be internal or external to the integrated gateway disaster recovery device. In certain embodiments, memory 1202 is a non-volatile solid-state memory.

[0105] In certain embodiments, the memory 1202 may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage media device, an optical storage media device, a flash memory device, an electrical, optical, or other physical / tangible memory storage device. Thus, generally, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to an aspect of the present disclosure.

[0106] The processor 1201 reads and executes computer program instructions stored in the memory 1202 to implement any one of the image description methods in the above embodiments.

[0107] In one example, the electronic device may further include a communication interface 1203 and a bus 1204. Figure 12 As shown, the processor 1201 , the memory 1202 , and the communication interface 1203 are connected via a bus 1204 and communicate with each other.

[0108] The communication interface 1203 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0109] Bus 1204 includes hardware, software, or both that couples components of an electronic device to each other. By way of example, and not limitation, a bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Super Transmission (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 1204 may include one or more buses. Although embodiments herein describe and illustrate a particular bus, this application contemplates any suitable bus or interconnect.

[0110] In addition, in conjunction with the image description method in the above embodiments, the present application embodiment may provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the image description methods in the above embodiments is implemented.

[0111] An embodiment of the present application further provides a computer program product, including a computer program, which implements any one of the image description methods in the above embodiments when the computer program is processed and executed.

[0112] It should be understood that the present application is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted here. In the above embodiments, several specific steps are described and illustrated as examples. However, the method process of the present application is not limited to the specific steps described and illustrated. Those skilled in the art can make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present application.

[0113] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they may be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, and the like. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored in a machine-readable medium or transmitted via a data signal carried in a carrier wave over a transmission medium or communication link. "Machine-readable medium" may include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable read-only memory (EROM), floppy disks, compact disc read-only memory (CD-ROM), optical discs, hard disks, optical fiber media, radio frequency (RF) links, and the like. Code segments can be downloaded via computer networks such as the Internet and intranets.

[0114] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps. In other words, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0115] Aspects of the present disclosure have been described above with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block in the flowcharts and / or block diagrams, as well as combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine such that execution of these instructions by the processor of the computer or other programmable data processing device enables the implementation of the functions / actions specified in one or more blocks in the flowcharts and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It should also be understood that each block in the block diagrams and / or flowcharts, as well as combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware that performs the specified functions or actions, or by a combination of dedicated hardware and computer instructions.

[0116] The above is only a specific implementation method of the present application. Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the scope of protection of the present application is not limited to this. Any technician familiar with this technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the scope of protection of this application.< / sos> < / eos> < / sos>

Claims

1. A method for describing an image, characterized in that: include: Obtain the image to be recognized; Extracting regional features and grid features of the image to be identified using an image description generation model, wherein the grid features are used to represent context information of the image to be identified, and the regional features are used to represent object image content information contained in the image to be identified; Predicting input text based on the region features and the grid features using the image description generation model; Obtaining a self-attention feature of the input text by calculating the self-attention mechanism in the image description generation model; Calculating, by the image description generation model, a first cross-attention feature of the self-attention feature and the region feature, and a second cross-attention feature of the self-attention feature and the grid feature; splicing the first cross-attention feature and the second cross-attention feature through the image description generation model to obtain a spliced feature; Image description information corresponding to the image to be identified is generated using the image description generation model and the splicing features.

2. The method according to claim 1, characterized in that Before extracting the region features and grid features of the image to be identified by using the image description generation model, the method further includes: Scaling the image to be identified according to a preset size to obtain a scaled image; Performing feature extraction on the scaled image to obtain a first feature map; Performing dimensionality reduction processing on the first feature map to obtain a second feature map; Performing an expansion operation on the second feature map to obtain a third feature map; The extracting of the regional features and grid features of the image to be identified includes: The region features and the grid features of the third feature map are extracted.

3. The method according to claim 1, characterized in that The self-attention feature of the input text is calculated by the self-attention mechanism in the image description generation model, including: Input the input text into the embedding layer to obtain an embedding vector corresponding to the input text; Calculating the position code of the input text according to the position of the text between the input texts; Summing the embedding vector and the positional encoding to obtain an input tensor; The input tensor is input into the self-attention layer to obtain the self-attention feature.

4. The method according to claim 1, wherein The generating of image description information corresponding to the image to be identified by using the image description generation model and the splicing features includes: Performing dimensionality reduction processing on the splicing features to obtain dimensionality reduction features; Performing residual operations and normalization operations on the dimensionality reduction features and the self-attention features to obtain output features; Inputting the output features into a feedforward network to obtain features to be identified output by the feedforward network; The features to be identified are input into a preset activation function to obtain the image description information.

5. The method according to claim 1, wherein Before extracting the region features and grid features of the image to be identified by using the image description generation model, the method further includes: Get the training dataset; Using the training data set to train the Transformer model to obtain a training prediction result; Calculating a focal loss function value using the label data corresponding to the training data set, the training prediction result, and a preset focal loss function; When the focus loss function value satisfies a preset iteration stopping condition, the image description generation model is obtained.

6. The method according to claim 5, characterized in that The calculating of the focus loss function value using the label data corresponding to the training data set, the training prediction result and the preset focus loss function includes: The focus loss function value is calculated according to the following formula: in, is the focal loss function value, n represents the number of vocabulary categories, is the first regulating factor, , is the second regulatory factor, , is the label data corresponding to the training prediction result, Predict results for the training.

7. An image description device, characterized in that: include: An acquisition module, used for acquiring an image to be identified; an extraction module, configured to extract regional features and grid features of the image to be identified using an image description generation model, wherein the grid features are used to represent context information of the image to be identified, and the regional features are used to represent object image content information contained in the image to be identified; A prediction module, configured to predict an input text based on the region features and the grid features using the image description generation model; A calculation module, configured to calculate the self-attention features of the input text through the self-attention mechanism in the image description generation model; The calculation module is further configured to calculate, by using the image description generation model, a first cross-attention feature of the self-attention feature and the region feature, and a second cross-attention feature of the self-attention feature and the grid feature; a splicing module, configured to splice the first cross-attention feature and the second cross-attention feature using the image description generation model to obtain a spliced feature; A generation module is used to generate image description information corresponding to the image to be identified through the image description generation model and the splicing features.

8. An electronic device, characterized in that: The device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the image description method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the image description method according to any one of claims 1 to 6.

10. A computer program product, characterized in that When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device is caused to execute the image description method according to any one of claims 1 to 6.

Images