Image description generation method and device, computer equipment and storage medium

Abstract

The invention discloses an image description generation method and device, computer equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the steps of obtaining a to-be-processed image, and performing image segmentation on the to-be-processed image to obtain a plurality of image blocks; constructing an image block sequence based on the plurality of image blocks, and encoding the image block sequence to obtain an image feature vector; obtaining an image content text corresponding to the to-be-processed image, and encoding the image content text to obtain a text feature vector; constructing a feature vector combination based on the image feature vector and the text feature vector; and importing the feature vector combination into a pre-trained image description generation model, and outputting an image description result of the to-be-processed image. The invention also relates to the technical field of block chains, and the to-be-processed image is stored in a block chain network. According to the method, the requirements of large-scale data sets and computing resources are reduced by utilizing the pre-training model, the computing cost is remarkably reduced, and the accessibility of the model is improved.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to an image description generation method, device, computer equipment and storage medium. Background Art

[0002] Image description is a technology that combines computer vision and natural language processing, aiming to enable computers to "speak from images". Specifically, image description technology can generate corresponding natural language descriptions based on the visual information presented by the image. This process requires not only that the computer can recognize objects, scenes, actions and other elements in the image, but also that it needs to understand the relationship between these elements and express this information in a natural language form that humans can understand.

[0003] With the development of deep learning technology, especially the breakthroughs in image recognition and feature extraction by convolutional neural networks, as well as the development and application of Transformer models, image description generation has gradually become possible. By training models to identify objects, scenes, actions and other elements in images and converting them into coherent and accurate text descriptions, this technology can not only help the visually impaired to better understand their surroundings, but also play an important role in many fields such as artistic creation, education, advertising and entertainment.

[0004] However, the rapid development of this technology also faces huge challenges. For example, current image description schemes require end-to-end training using large-scale models and datasets, and most advanced visual language models incur high computational costs during pre-training, which not only limits the accessibility of the model but also places higher demands on resource utilization.

[0005] Therefore, how to reduce computational costs while maintaining model performance has become an important topic in current research. Summary of the invention

[0006] The purpose of the embodiments of the present application is to propose an image description generation method, apparatus, computer device and storage medium to solve the technical problems that existing image description solutions require the use of large-scale models and data sets for end-to-end training, resulting in high computing costs, limiting the accessibility of the model, and requiring high resource utilization.

[0007] In order to solve the above technical problems, the present application embodiment provides an image description generation method, which adopts the following technical solution:

[0008] A method for generating an image description, comprising:

[0009] Acquire an image to be processed, and perform image segmentation on the image to be processed to obtain a number of image blocks;

[0010] An image block sequence is constructed based on a number of image blocks, and the image block sequence is encoded to obtain an image feature vector;

[0011] Obtain the image content text corresponding to the image to be processed, and encode the image content text to obtain a text feature vector;

[0012] Based on the image feature vector and the text feature vector, construct a feature vector combination;

[0013] The feature vector combination is imported into the pre-trained image description generation model, and the image description result of the image to be processed is output.

[0014] Furthermore, the steps of obtaining the image to be processed and performing image segmentation on the image to be processed to obtain a plurality of image blocks specifically include:

[0015] Acquire an image to be processed, and perform preprocessing on the image to be processed, wherein the preprocessing includes adjusting the image size, normalizing pixel values, and standardizing colors;

[0016] The preprocessed image to be processed is input into a pre-trained image processing model, wherein the image processing model is built based on the Transformer network architecture and includes an image segmentation layer and a coding layer;

[0017] The fixed blocks in the image segmentation layer are used to segment the input image to be processed to obtain a number of image blocks.

[0018] Furthermore, the steps of constructing an image block sequence based on a plurality of image blocks and encoding the image block sequence to obtain an image feature vector specifically include:

[0019] Flatten each image block to obtain several image block flattening vectors;

[0020] Adding a position identifier to each image block based on the position of each image block in the image to be processed;

[0021] Combining the flattened vectors of each image block according to the position identifiers of each image block to obtain an image block sequence;

[0022] The encoding layer is used to encode the image block sequence to obtain the image feature vector.

[0023] Furthermore, the steps of obtaining the image content text corresponding to the image to be processed and encoding the image content text to obtain the text feature vector specifically include:

[0024] Perform character recognition on the image to be processed to obtain the image content text;

[0025] Perform text segmentation processing on the image content text to obtain content text segmentation;

[0026] The encoding layer is used to encode the content text segmentation to obtain the text feature vector.

[0027] Furthermore, the image processing model also includes a decoding layer and a fully connected layer, and the steps of constructing a feature vector combination based on the image feature vector and the text feature vector specifically include:

[0028] Mapping the image feature vector and the text feature vector to a preset feature space respectively;

[0029] Obtain feature vector mapping results from feature space;

[0030] Use the decoding layer to decode the feature vector mapping result to obtain the decoded features;

[0031] The decoded features are input into the fully connected layer, and the image feature combination is predicted through the fully connected layer;

[0032] The image feature combination is vectorized to obtain a feature vector combination.

[0033] Furthermore, before the step of mapping the image feature vector and the text feature vector to the preset feature space respectively, the method further includes:

[0034] Calculate the feature weights of the image feature vector and the text feature vector based on a preset feature weight algorithm;

[0035] Use feature weights to perform feature weighting on image feature vectors and text feature vectors respectively;

[0036] The weighted image feature vector and text feature vector are mapped into the feature space.

[0037] Furthermore, before the step of constructing an image block sequence based on a plurality of image blocks and encoding the image block sequence to obtain an image feature vector, the method further includes:

[0038] Obtaining pre-collected historical images and obtaining historical content text corresponding to the historical images;

[0039] Build a training dataset based on historical images and historical content texts, and input the training dataset into the initial model built based on the Transformer network architecture;

[0040] Perform image segmentation on the historical image to obtain a number of historical image blocks;

[0041] A historical image block sequence is constructed based on a number of historical image blocks, and the historical image block sequence is encoded to obtain a historical image feature vector;

[0042] Encode the historical content text to obtain the historical text feature vector;

[0043] Based on the historical image feature vector and the historical text feature vector, a historical feature vector combination is constructed;

[0044] The feature association relationship in the historical feature vector combination is learned through the initial model to obtain a pre-trained image processing model.

[0045] In order to solve the above technical problems, the embodiment of the present application further provides an image description generating device, which adopts the following technical solution:

[0046] An image description generating device, comprising:

[0047] An image segmentation module is used to obtain an image to be processed and perform image segmentation on the image to be processed to obtain a number of image blocks;

[0048] An image encoding module, used for constructing an image block sequence based on a number of image blocks, and encoding the image block sequence to obtain an image feature vector;

[0049] A text encoding module is used to obtain the image content text corresponding to the image to be processed, and encode the image content text to obtain a text feature vector;

[0050] A feature combination module is used to construct a feature vector combination based on an image feature vector and a text feature vector;

[0051] The description generation module is used to import the feature vector combination into the pre-trained image description generation model and output the image description result of the image to be processed.

[0052] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0053] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of any one of the above-mentioned methods for generating image descriptions when executing the computer-readable instructions.

[0054] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0055] A computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the image description generation method described in any one of the above are implemented.

[0056] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0057] The present application discloses an image description generation method, device, computer equipment and storage medium, and belongs to the field of artificial intelligence technology. The present application obtains a to-be-processed image, and performs image segmentation on the to-be-processed image to obtain a number of image blocks; constructs an image block sequence based on the several image blocks, and encodes the image block sequence to obtain an image feature vector; obtains the image content text corresponding to the to-be-processed image, and encodes the image content text to obtain a text feature vector; constructs a feature vector combination based on the image feature vector and the text feature vector; imports the feature vector combination into a pre-trained image description generation model, and outputs the image description result of the to-be-processed image. The present application realizes image description generation by utilizing a pre-trained model in combination with image semantic features and text semantic features, thereby reducing the demand for large-scale data sets and computing resources, so that the image description generation task can be efficiently processed in a resource-limited environment, significantly reducing the computing cost and improving the accessibility of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0059] Figure 1 An exemplary system architecture diagram is shown in which the present application can be applied;

[0060] Figure 2 A flowchart of an embodiment of a method for generating image description according to the present application is shown;

[0061] Figure 3 A schematic structural diagram of an embodiment of an image description generating device according to the present application is shown;

[0062] Figure 4 A schematic structural diagram of an embodiment of a computer device according to the present application is shown. DETAILED DESCRIPTION

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0064] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0065] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0066] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0067] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0068] The terminal device 101 can be various electronic devices with a display screen and supporting web browsing. In addition to the laptop computer 1011, the tablet computer 1012 or the mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer and a desktop computer, etc.

[0069] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0070] It should be noted that the image description generation method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the image description generation device is generally arranged in the server / terminal device.

[0071] It should be understood that Figure 1 The numbers of terminal devices, networks and servers in the embodiment are merely illustrative. The above system may have any number of terminal devices, networks and servers according to implementation requirements.

[0072] Continue to refer Figure 2 , shows a flow chart of an embodiment of a method for generating image description according to the present application. The image description generating method comprises the following steps:

[0073] S201, obtaining an image to be processed, and performing image segmentation on the image to be processed to obtain a plurality of image blocks.

[0074] Specifically, first, the system acquires the image to be processed, which usually comes from various real-life scenes or data sets and contains visual information to be analyzed and described. The purpose of image segmentation on the image to be processed is to divide the image into multiple small areas or image blocks to facilitate a more detailed analysis of the image. Image segmentation can use a variety of techniques, such as segmentation methods based on convolutional neural networks (CNN), image slicing methods, or sliding window techniques. Each image block represents a local area of ​​the image. Such segmentation helps capture the details and local features of the image. The segmented image blocks can be fixed-size blocks or adaptive-size areas.

[0075] In a specific embodiment of the present application, the image processing model is a Vi TL / 14 model pre-trained based on the Transformer network architecture, and the Vi TL / 14 model is used as an image encoder to implement image segmentation and image block processing. "ViT" stands for Visual Transformer, indicating that the model applies the Transformer model to computer vision tasks; the "L" in "L / 14" represents that the size of the model is Large, and "14" means that the resolution of each image block (patch) is 14x14 pixels.

[0076] The V i T model divides the image into multiple small patches, each of which is considered a "token", similar to words in natural language processing. These patches are converted into feature vectors through a linear embedding layer, and position encoding is added to preserve spatial information. These feature vectors are then fed into a standard Transformer encoder for processing to learn global and local features in the image.

[0077] S202, constructing an image block sequence based on a plurality of image blocks, and encoding the image block sequence to obtain an image feature vector.

[0078] Specifically, constructing an image block sequence based on several image blocks obtained by image segmentation is to arrange the image blocks into a sequence in a certain order, which can be from left to right, from top to bottom, or according to a specific grid structure. The purpose of constructing an image block sequence is to maintain the spatial structure information of the image so that the subsequent encoding step can effectively capture the relationship between the image blocks.

[0079] Next, these image block sequences are encoded, which is usually done through deep learning models, such as convolutional neural networks (CNNs) or visual transformers. The encoding process converts each image block into a high-dimensional feature vector, which integrates the local feature information of the image block and can capture the overall information of the image through model training. Finally, the feature vectors of all image blocks are combined to generate a comprehensive image feature vector, which represents the high-dimensional features of the entire image and provides valuable information input for the image description generation model.

[0080] S203, obtaining the image content text corresponding to the image to be processed, and encoding the image content text to obtain a text feature vector.

[0081] Specifically, obtaining the image content text corresponding to the image to be processed refers to obtaining the text or label on the image. These texts usually provide semantic information of the image and can be implemented based on optical character recognition (OCR) technology. Encoding the image content text is to convert these text data into a machine-processable form, usually using a natural language processing (NLP) model such as BERT, GPT or Transformer model. The encoding step segments the text into words, converts it into word vectors, and uses a pre-trained language model to perform deep semantic analysis on it to generate a text feature vector. The text feature vector is a high-dimensional vector that can effectively represent the semantic information and contextual relationships in the text.

[0082] S204: construct a feature vector combination based on the image feature vector and the text feature vector.

[0083] Specifically, the process of constructing a feature vector combination based on the image feature vector and the text feature vector involves fusing these two feature vectors. The image feature vector represents the visual information of the image, while the text feature vector represents the semantic information of the image. The goal of the feature vector combination is to effectively combine these two types of information to form a comprehensive feature representation. Fusion methods can include simple concatenation, weighted averaging, or more complex fusion techniques, such as through attention mechanisms or bidirectional encoders. The purpose of this combination process is to generate a feature vector that can fully describe the image content by integrating the information of the image and text, providing rich input data for the image description generation model.

[0084] S205, importing the feature vector combination into a pre-trained image description generation model, and outputting an image description result of the image to be processed.

[0085] Specifically, the last step is to import the feature vector combination into a pre-trained image description generation model, which aims to convert the comprehensive feature vector into a natural language image description. The pre-trained image description generation model is usually a deep learning model, such as a pre-trained LLM model (i.e. Large Language Model l), which can convert the input feature vector combination into a fluent and semantically rich text description and output a natural language description of the image to be processed. The generated image description should be able to accurately reflect the content, scene and main objects of the image, so as to meet the needs of image understanding and automatic description.

[0086] LLM is a natural language processing (NLP) model based on deep learning, which can learn and understand the grammar and semantics of natural language to generate human-readable text. This model usually has billions to hundreds of billions of parameters and is trained through large-scale corpora to achieve deep understanding and generation capabilities of natural language.

[0087] In the above embodiments, the present application realizes image description generation by utilizing a pre-trained model in combination with image semantic features and text semantic features, thereby reducing the demand for large-scale data sets and computing resources, so that the image description generation task can be efficiently processed in a resource-limited environment, significantly reducing computing costs and improving the accessibility of the model.

[0088] Furthermore, the steps of obtaining the image to be processed and performing image segmentation on the image to be processed to obtain a plurality of image blocks specifically include:

[0089] Acquire an image to be processed, and perform preprocessing on the image to be processed, wherein the preprocessing includes adjusting the image size, normalizing pixel values, and standardizing colors;

[0090] The preprocessed image to be processed is input into a pre-trained image processing model, wherein the image processing model is built based on the Transformer network architecture, and the image processing model includes an image segmentation layer and a coding layer;

[0091] The fixed blocks in the image segmentation layer are used to segment the input image to be processed to obtain a number of image blocks.

[0092] In this embodiment, first, the system preprocesses the image to be processed, including resizing, pixel value normalization, and color standardization, to ensure that the image meets the requirements of the model input. Then, the preprocessed image is input into a pre-trained image processing model based on the Transformer network architecture. The advantage of the image processing model is that the self-attention mechanism of the Transformer can effectively capture the global and local information of the image, thereby improving the ability of feature representation. The image processing model includes an image segmentation layer and an encoding layer, wherein the image segmentation layer is responsible for dividing the image into several fixed-size image blocks, which can be processed independently, thereby decomposing the complexity of the image and improving processing efficiency. Each image block is then input into the encoding layer for feature extraction, and the encoding layer uses the ability of the Transformer to perform deep feature representation on these image blocks.

[0093] Through the above steps, the image segmentation layer reduces the computational complexity by processing local information, while the Transformer architecture enhances the model's ability to capture global and local information of the image, improving the accuracy and efficiency of image feature extraction.

[0094] It should be noted that the V i T model requires that the size of the input image is fixed, usually 224x224 pixels (or other preset sizes). If the size of the input image does not meet the requirements, it must be adjusted to the specified size. The V i T model usually requires the input pixel values ​​to be normalized. Common normalization is to scale the pixel values ​​of the image from the range [0,255] to [0,1] or normalize with a specific mean and standard deviation, which can help the model better adapt to different types of image inputs. In addition, the V i T model requires that the input image color space is a specific standard, such as converting the RGB color space to BGR (a common format in OpenCV).

[0095] Furthermore, the steps of constructing an image block sequence based on a plurality of image blocks and encoding the image block sequence to obtain an image feature vector specifically include:

[0096] Flatten each image block to obtain several image block flattening vectors;

[0097] Adding a position identifier to each image block based on the position of each image block in the image to be processed;

[0098] Combining the flattened vectors of each image block according to the position identifiers of each image block to obtain an image block sequence;

[0099] The encoding layer is used to encode the image block sequence to obtain the image feature vector.

[0100] Flattening of image patches refers to the process of converting the two-dimensional structure of each image patch (usually a three-dimensional structure of height, width, and channels) into a one-dimensional vector. Simply put, it is to straighten the pixel values ​​in the image block and arrange them in a row. For example, the size of an image block is 16x16, each pixel has 3 channels (i.e., RGB channels), and the shape of this image block is 16x16x3, i.e., 16 pixels in height, 16 pixels in width, and 3 channel pixels. In other words, each image block contains 16*16*3=768 pixel values. The flattening operation is to convert this 16x16x3 image block into a one-dimensional vector with a length of 768. The flattened vector will be used as the input of the Transformer encoding layer.

[0101] In this embodiment, first, an image flattening operation is performed on each image block, and each image block is converted from a two-dimensional space into a one-dimensional vector, namely, an image block flattened vector. After obtaining a number of image block flattened vectors, it is necessary to identify the specific position of each image block in the image to be processed, which can be achieved by assigning a position code to each image block. The position code is used to characterize the relative or absolute position of the image block in the overall image. The position identifier can help the model understand the spatial relationship between the various parts of the image. Next, the image block flattened vectors are combined according to the position identifiers of each image block to generate an image block sequence. This sequence maintains the spatial structure information of the image block and provides structured input data for the encoding layer. Finally, the encoding layer is used to encode this image block sequence. The encoding layer usually uses the self-attention mechanism in the Transformer architecture to capture the complex relationship and global context between image blocks. After being processed by the encoding layer, the image feature vector obtained integrates the information of all image blocks to form a high-level feature representation of the entire image.

[0102] Through the above steps, the complete expression of local information and spatial relations of the image is ensured by flattening the image blocks, identifying the positions and combining the sequences. The self-attention mechanism of the Transformer encoding layer is used to deeply encode the image block sequence and generate high-quality image feature vectors.

[0103] Furthermore, the steps of obtaining the image content text corresponding to the image to be processed and encoding the image content text to obtain the text feature vector specifically include:

[0104] Perform character recognition on the image to be processed to obtain the image content text;

[0105] Perform text segmentation processing on the image content text to obtain content text segmentation;

[0106] The encoding layer is used to encode the content text segmentation to obtain the text feature vector.

[0107] In this embodiment, first, the text information in the image is extracted by performing character recognition on the image to be processed, and the text in the image is converted into machine-readable text data using optical character recognition (OCR) technology. Next, the acquired image content text is subjected to text segmentation processing, and the continuous text string is split into vocabulary units or tags, with the purpose of converting the original text into processable words or vocabulary unit forms. Commonly used word segmentation methods include space-based word segmentation, character-level word segmentation, or the use of pre-trained word segmenters (such as BERT's word segmenter). Then, the content text segmentation is encoded using the encoding layer, and each vocabulary unit is converted into a high-dimensional feature vector. The encoding layer vectorizes the text through the embedding layer and the self-attention mechanism to capture the semantics and contextual relationships in the text.

[0108] Through the above steps, the text is vectorized using the encoding layer, which provides a deep semantic understanding of the text content and significantly improves the accuracy of text information processing.

[0109] Furthermore, the image processing model also includes a decoding layer and a fully connected layer, and the steps of constructing a feature vector combination based on the image feature vector and the text feature vector specifically include:

[0110] Mapping the image feature vector and the text feature vector to a preset feature space respectively;

[0111] Obtain feature vector mapping results from feature space;

[0112] Use the decoding layer to decode the feature vector mapping result to obtain the decoded features;

[0113] The decoded features are input into the fully connected layer, and the image feature combination is predicted through the fully connected layer;

[0114] The image feature combination is vectorized to obtain a feature vector combination.

[0115] Feature vector mapping converts a feature vector from the original space to a preset feature space through some transformation (such as linear transformation, nonlinear transformation, etc.), with the purpose of converting feature representations from different sources to the same or compatible feature space for further similarity comparison, fusion, matching, etc. After mapping to the same feature space, features from different modalities or sources can be combined for joint modeling.

[0116] In this embodiment, first, the image feature vector and the text feature vector are mapped to a preset feature space respectively. The feature space is a high-dimensional space used to map features of different modalities (image and text) to a unified representation space. This mapping process is implemented by a specific linear transformation or nonlinear mapping function to ensure that the image features and text features can be effectively compared and fused in the same feature space. Then, the feature vector mapping results are obtained from the feature space. These results represent the representation of the image features and text features in the preset feature space. Then, the decoding layer is used to decode these feature vector mapping results to generate decoding features. The decoding layer usually uses the reverse mapping technology to convert the high-dimensional feature vector into the original or interpretable feature form. These decoding features can reconstruct the key information of the image and text. The decoding features are input to the fully connected layer, and further processed and combined by the fully connected layer to predict the image feature combination. The fully connected layer is responsible for nonlinearly transforming the decoding features and integrating the associated information of each feature. Finally, the image feature combination is vectorized to obtain a feature vector combination. These feature vector combinations integrate the information of the image and text.

[0117] Through the above steps, by mapping the image feature vector and the text feature vector into a unified feature space, the effective integration of different modal information is achieved. The features are processed and combined using the decoding layer and the fully connected layer, ensuring the efficient fusion and expression of the features.

[0118] Furthermore, before the step of mapping the image feature vector and the text feature vector to the preset feature space respectively, the method further includes:

[0119] Calculate the feature weights of the image feature vector and the text feature vector based on a preset feature weight algorithm;

[0120] Use feature weights to perform feature weighting on image feature vectors and text feature vectors respectively;

[0121] The weighted image feature vector and text feature vector are mapped into the feature space.

[0122] In this embodiment, first, the feature weights of the image feature vector and the text feature vector are calculated based on a preset feature weight algorithm. The feature weight algorithm is used to determine the importance or influence of the image features and the text features. These weights can be calculated according to specific criteria, such as the confidence, importance score or relevance of the features. The feature weights are usually set by model parameters obtained through training, domain knowledge or statistical methods. Next, these feature weights are used to perform feature weighting processing on the image feature vector and the text feature vector respectively. Feature weighting involves applying weights to each feature to adjust its contribution in the final feature combination. This process helps to highlight important features and reduce the impact of irrelevant features, thereby improving the quality and accuracy of feature expression. The weighted image feature vector and text feature vector will be mapped into the feature space. This mapping process converts the feature vector into a unified high-dimensional space, so that features of different modalities can be compared and fused in the same space.

[0123] Through the above steps, by calculating and applying feature weights, fine adjustment and optimization of image and text features are achieved. Feature weighting improves the contribution of important features while reducing the impact of noise. Mapping the weighted features to a unified feature space helps to accurately fuse image and text information and improve the performance and accuracy of multimodal learning tasks.

[0124] Furthermore, before the step of constructing an image block sequence based on a plurality of image blocks and encoding the image block sequence to obtain an image feature vector, the method further includes:

[0125] Obtaining pre-collected historical images and obtaining historical content text corresponding to the historical images;

[0126] Build a training dataset based on historical images and historical content texts, and input the training dataset into the initial model built based on the Transformer network architecture;

[0127] Perform image segmentation on the historical image to obtain a number of historical image blocks;

[0128] A historical image block sequence is constructed based on a number of historical image blocks, and the historical image block sequence is encoded to obtain a historical image feature vector;

[0129] Encode the historical content text to obtain the historical text feature vector;

[0130] Based on the historical image feature vector and the historical text feature vector, a historical feature vector combination is constructed;

[0131] The feature association relationship in the historical feature vector combination is learned through the initial model to obtain a pre-trained image processing model.

[0132] In this embodiment, first, the pre-collected historical images and their corresponding historical content texts are obtained. These data will be used as training data sets. The historical images and their corresponding text descriptions provide rich multimodal information for training image processing models. Next, a training data set is constructed based on these historical images and texts, and is input into an initial model built based on the Transformer network architecture. The model uses Transformer to process image and text data, and can effectively capture local and global features in the image, as well as contextual relationships in the text. The historical image is segmented, and the image is divided into several blocks. Each image block represents a local area in the image. An image block sequence is constructed based on these image blocks, and the sequence is encoded to obtain historical image feature vectors. These feature vectors express the details and structural information of the image; at the same time, the historical content text is encoded to obtain historical text feature vectors, which express the semantic information of the text. The historical image feature vectors and the historical text feature vectors are combined to form a historical feature vector combination. By inputting these feature vector combinations into the initial model for learning, the model can capture the feature association relationship between the image and the text, thereby training a pre-trained image processing model that can effectively process multimodal data.

[0133] Through the above steps, by systematically processing historical image and text data and using the Transformer network to learn the complex relationship between images and text, the model's ability to understand and process images and text is improved.

[0134] In the above embodiment, the present application discloses an image description generation method, which belongs to the field of artificial intelligence technology. The present application obtains a to-be-processed image, and performs image segmentation on the to-be-processed image to obtain a number of image blocks; constructs an image block sequence based on the several image blocks, and encodes the image block sequence to obtain an image feature vector; obtains the image content text corresponding to the to-be-processed image, and encodes the image content text to obtain a text feature vector; constructs a feature vector combination based on the image feature vector and the text feature vector; imports the feature vector combination into a pre-trained image description generation model, and outputs the image description result of the to-be-processed image. The present application realizes image description generation by utilizing a pre-trained model in combination with image semantic features and text semantic features, thereby reducing the demand for large-scale data sets and computing resources, so that the image description generation task can be efficiently processed in a resource-limited environment, significantly reducing the computing cost and improving the accessibility of the model.

[0135] In this embodiment, the electronic device (eg, Figure 1 The server shown in the figure) can receive instructions or obtain data through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G connection, Wi-Fi connection, Bluetooth connection, Wi MAX connection, Zigbee connection, UWB (ultrawi deband) connection, and other wireless connection methods currently known or to be developed in the future.

[0136] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned image information to be processed, the above-mentioned image information to be processed can also be stored in a node of a blockchain.

[0137] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0138] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0139] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0140] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0141] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0142] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of an image description generating device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0143] like Figure 3 As shown, the image description generating device 300 described in this embodiment includes:

[0144] The image segmentation module 301 is used to obtain an image to be processed and perform image segmentation on the image to be processed to obtain a plurality of image blocks;

[0145] An image encoding module 302 is used to construct an image block sequence based on a plurality of image blocks, and encode the image block sequence to obtain an image feature vector;

[0146] The text encoding module 303 is used to obtain the image content text corresponding to the image to be processed, and encode the image content text to obtain a text feature vector;

[0147] A feature combination module 304 is used to construct a feature vector combination based on the image feature vector and the text feature vector;

[0148] The description generation module 305 is used to import the feature vector combination into the pre-trained image description generation model and output the image description result of the image to be processed.

[0149] Furthermore, the image segmentation module 301 is specifically used for:

[0150] Obtain an image to be processed and preprocess the image to be processed, wherein the preprocessing includes adjusting the image size, normalizing pixel values, and standardizing colors; input the preprocessed image to be processed into a pre-trained image processing model, wherein the image processing model is built based on a Transformer network architecture, and the image processing model includes an image segmentation layer and a coding layer; use the fixed blocks in the image segmentation layer to segment the input image to be processed to obtain a plurality of image blocks.

[0151] Furthermore, the image encoding module 302 is specifically used for:

[0152] Each image block is flattened to obtain several image block flattened vectors; a position identifier is added to each image block based on the position of each image block in the image to be processed; the flattened vectors of each image block are combined according to the position identifier of each image block to obtain an image block sequence; the image block sequence is encoded using a coding layer to obtain an image feature vector.

[0153] Furthermore, the text encoding module 303 is specifically used for:

[0154] Perform character recognition on the image to be processed to obtain the image content text; perform text segmentation on the image content text to obtain content text segmentation; use the encoding layer to encode the content text segmentation to obtain a text feature vector.

[0155] Furthermore, the image processing model further includes a decoding layer and a fully connected layer, and the feature combination module 404 is specifically used for:

[0156] The image feature vector and the text feature vector are respectively mapped to a preset feature space; the feature vector mapping result is obtained from the feature space; the feature vector mapping result is decoded using a decoding layer to obtain a decoded feature; the decoded feature is input into a fully connected layer, and the image feature combination is predicted through the fully connected layer; the image feature combination is vectorized to obtain a feature vector combination.

[0157] Furthermore, the image description generating device 300 further includes a feature weighting module, which is specifically used for:

[0158] The feature weights of the image feature vector and the text feature vector are calculated based on a preset feature weight algorithm; the feature weights are used to perform feature weighting on the image feature vector and the text feature vector respectively; and the weighted image feature vector and text feature vector are mapped to the feature space.

[0159] Furthermore, the image description generating device 300 further includes a model training module, which is specifically used for:

[0160] Acquire pre-collected historical images and the historical content texts corresponding to the historical images; construct a training data set based on the historical images and the historical content texts, and input the training data set into an initial model built based on a Transformer network architecture; perform image segmentation on the historical images to obtain a number of historical image blocks; construct a historical image block sequence based on the several historical image blocks, and encode the historical image block sequence to obtain a historical image feature vector; encode the historical content text to obtain a historical text feature vector; construct a historical feature vector combination based on the historical image feature vector and the historical text feature vector; learn the feature association relationship in the historical feature vector combination through the initial model to obtain a pre-trained image processing model.

[0161] In the above embodiment, the present application discloses an image description generation device, which belongs to the field of artificial intelligence technology. The present application obtains a to-be-processed image, and performs image segmentation on the to-be-processed image to obtain a number of image blocks; constructs an image block sequence based on the several image blocks, and encodes the image block sequence to obtain an image feature vector; obtains the image content text corresponding to the to-be-processed image, and encodes the image content text to obtain a text feature vector; constructs a feature vector combination based on the image feature vector and the text feature vector; imports the feature vector combination into a pre-trained image description generation model, and outputs the image description result of the to-be-processed image. The present application realizes image description generation by utilizing a pre-trained model in combination with image semantic features and text semantic features, thereby reducing the demand for large-scale data sets and computing resources, so that the image description generation task can be efficiently processed in a resource-limited environment, significantly reducing the computing cost and improving the accessibility of the model.

[0162] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0163] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 4 having a memory 41, a processor 42, and a network interface 43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital processor (DSP), an embedded device, etc.

[0164] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0165] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk equipped on the computer device 4, a smart memory card (Smart Med ia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of the image description generation method, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0166] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run computer-readable instructions or process data stored in the memory 41, such as computer-readable instructions for running the image description generation method.

[0167] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0168] In the above embodiment, the present application discloses a computer device, which belongs to the field of artificial intelligence technology. The present application obtains a to-be-processed image, and performs image segmentation on the to-be-processed image to obtain a number of image blocks; constructs an image block sequence based on the several image blocks, and encodes the image block sequence to obtain an image feature vector; obtains the image content text corresponding to the to-be-processed image, and encodes the image content text to obtain a text feature vector; constructs a feature vector combination based on the image feature vector and the text feature vector; imports the feature vector combination into a pre-trained image description generation model, and outputs the image description result of the to-be-processed image. The present application realizes image description generation by utilizing a pre-trained model in combination with image semantic features and text semantic features, thereby reducing the demand for large-scale data sets and computing resources, so that the image description generation task can be efficiently processed in a resource-limited environment, significantly reducing the computing cost and improving the accessibility of the model.

[0169] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the image description generation method as described above.

[0170] In the above embodiment, the present application discloses a computer-readable storage medium, which belongs to the field of artificial intelligence technology. The present application obtains a to-be-processed image, and performs image segmentation on the to-be-processed image to obtain a number of image blocks; constructs an image block sequence based on the several image blocks, and encodes the image block sequence to obtain an image feature vector; obtains the image content text corresponding to the to-be-processed image, and encodes the image content text to obtain a text feature vector; constructs a feature vector combination based on the image feature vector and the text feature vector; imports the feature vector combination into a pre-trained image description generation model, and outputs the image description result of the to-be-processed image. The present application realizes image description generation by utilizing a pre-trained model in combination with image semantic features and text semantic features, thereby reducing the demand for large-scale data sets and computing resources, so that the image description generation task can be efficiently processed in a resource-limited environment, significantly reducing the computing cost and improving the accessibility of the model.

[0171] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0172] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0173] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A method for generating an image description, characterized in that: include: Acquire an image to be processed, and perform image segmentation on the image to be processed to obtain a plurality of image blocks; constructing an image block sequence based on a plurality of the image blocks, and encoding the image block sequence to obtain an image feature vector; Obtaining image content text corresponding to the image to be processed, and encoding the image content text to obtain a text feature vector; Constructing a feature vector combination based on the image feature vector and the text feature vector; The feature vector combination is imported into a pre-trained image description generation model, and the image description result of the image to be processed is output.

2. The image description generating method according to claim 1, characterized in that: The step of obtaining the image to be processed and performing image segmentation on the image to be processed to obtain a plurality of image blocks specifically includes: Acquire an image to be processed, and preprocess the image to be processed, wherein the preprocessing includes adjusting image size, normalizing pixel values, and standardizing colors; Inputting the preprocessed image to be processed into a pre-trained image processing model, wherein the image processing model is built based on a Transformer network architecture, and the image processing model includes an image segmentation layer and a coding layer; The fixed blocks in the image segmentation layer are used to perform image segmentation on the input image to be processed to obtain a plurality of image blocks.

3. The image description generating method according to claim 2, characterized in that: The step of constructing an image block sequence based on a plurality of the image blocks, and encoding the image block sequence to obtain an image feature vector specifically includes: Performing image flattening on each of the image blocks to obtain a plurality of image block flattening vectors; Adding a position identifier to each image block based on the position of each image block in the image to be processed; Combining the flattened vectors of each of the image blocks according to the position identifiers of each of the image blocks to obtain the image block sequence; The image block sequence is encoded using the encoding layer to obtain the image feature vector.

4. The image description generating method according to claim 2, characterized in that: The step of obtaining the image content text corresponding to the image to be processed, encoding the image content text, and obtaining a text feature vector specifically includes: Performing character recognition on the image to be processed to obtain the image content text; Performing text segmentation processing on the image content text to obtain content text segmentation; The content text segmentation is encoded using the encoding layer to obtain the text feature vector.

5. The image description generating method according to claim 2, characterized in that: The image processing model further includes a decoding layer and a fully connected layer. The step of constructing a feature vector combination based on the image feature vector and the text feature vector specifically includes: Mapping the image feature vector and the text feature vector to a preset feature space respectively; Acquire a feature vector mapping result from the feature space; Decoding the feature vector mapping result using the decoding layer to obtain a decoding feature; Inputting the decoded features into the fully connected layer, and predicting the image feature combination through the fully connected layer; The image feature combination is vectorized to obtain the feature vector combination.

6. The image description generating method according to claim 5, characterized in that: Before the step of mapping the image feature vector and the text feature vector to a preset feature space respectively, the method further includes: Calculate the feature weights of the image feature vector and the text feature vector based on a preset feature weight algorithm; Using the feature weights to perform feature weighting on the image feature vector and the text feature vector respectively; The weighted image feature vector and the text feature vector are mapped into the feature space.

7. The image description generating method according to claim 5, characterized in that: Before the step of constructing an image block sequence based on a plurality of the image blocks and encoding the image block sequence to obtain an image feature vector, the step further includes: Acquire pre-collected historical images and acquire historical content text corresponding to the historical images; Constructing a training data set based on the historical images and the historical content text, and inputting the training data set into an initial model built based on the Transformer network architecture; Performing image segmentation on the historical image to obtain a plurality of historical image blocks; Constructing a historical image block sequence based on a plurality of the historical image blocks, and encoding the historical image block sequence to obtain a historical image feature vector; Encoding the historical content text to obtain a historical text feature vector; Constructing a historical feature vector combination based on the historical image feature vector and the historical text feature vector; The feature association relationship in the historical feature vector combination is learned through the initial model to obtain a pre-trained image processing model.

8. An image description generating device, characterized in that: include: An image segmentation module is used to obtain an image to be processed and perform image segmentation on the image to be processed to obtain a plurality of image blocks; An image encoding module, used for constructing an image block sequence based on a plurality of the image blocks, and encoding the image block sequence to obtain an image feature vector; A text encoding module, used for obtaining the image content text corresponding to the image to be processed, and encoding the image content text to obtain a text feature vector; A feature combination module, used for constructing a feature vector combination based on the image feature vector and the text feature vector; The description generation module is used to import the feature vector combination into a pre-trained image description generation model and output the image description result of the image to be processed.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the image description generating method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the image description generating method according to any one of claims 1 to 7 are implemented.

Images