Image description method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an image description method and device, electronic equipment and a storage medium, and belongs to the technical field of artificial intelligence and finance. The method comprises the following steps: acquiring original image data and original image indication information; carrying out expansion processing on the original image indication information to obtain initial image indication information; performing image feature extraction on the original image data to obtain image embedding features; performing text feature extraction on the initial image indication information to obtain text embedding features; performing feature fusion on the image embedding feature and the text embedding feature to obtain a target information embedding feature; and performing image description on the original image data according to the target information embedding features to obtain target image description information. According to the embodiment of the invention, the complementarity of the image data and the image indication information can be fully utilized, and the accuracy of image description is improved.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence and financial technology, and in particular to an image description method and device, an electronic device and a storage medium. Background Art

[0002] Image information description is a technology that converts image content into text description. It can be applied to a variety of scenarios. For example, in the financial field, it is used to describe the image information of massive financial images and establish an image information database, which can improve the efficiency and accuracy of image retrieval.

[0003] At present, machine learning models are mainly used to directly generate image information from images. However, in actual applications, due to the wide variety of image types, machine learning models cannot accurately identify the core content of each image and accurately describe the image information, which affects the accuracy of image description.

[0004] Therefore, how to improve the accuracy of image description has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The main purpose of the embodiments of the present application is to provide an image description method and device, an electronic device and a storage medium, aiming to improve the accuracy of image description.

[0006] To achieve the above objective, a first aspect of an embodiment of the present application provides an image description method, the method comprising:

[0007] Acquiring original image data and original image indication information;

[0008] Performing expansion processing on the original image indication information to obtain initial image indication information;

[0009] Performing image feature extraction on the original image data to obtain image embedding features;

[0010] Extracting text features from the initial image indication information to obtain text embedding features;

[0011] Fusing the image embedding feature with the text embedding feature to obtain a target information embedding feature;

[0012] The original image data is described according to the target information embedding feature to obtain target image description information.

[0013] In some embodiments, the step of fusing the image embedding feature and the text embedding feature to obtain the target information embedding feature includes:

[0014] Performing feature splicing on the image embedding feature and the text embedding feature to obtain an image-text splicing feature;

[0015] Performing matrix transformation on the image-text splicing features to obtain an image-text transformation matrix;

[0016] Perform attention calculation on the image-text transformation matrix to obtain the target information embedding feature.

[0017] In some embodiments, performing attention calculation on the image-text transformation matrix to obtain the target information embedding feature includes:

[0018] Performing mapping transformation on the image-text transformation matrix to obtain a target image mapping matrix; wherein the target image mapping matrix includes a query mapping matrix, a key mapping matrix and a value mapping matrix;

[0019] Performing aggregation calculation on the query mapping matrix and the key mapping matrix to obtain an attention score;

[0020] Normalizing the attention score to obtain an attention weight;

[0021] The attention weight and the value mapping matrix are aggregated and calculated to obtain the target information embedding feature.

[0022] In some embodiments, extracting text features from the initial image indication information to obtain text embedding features includes:

[0023] Performing text cleaning on the initial image indication information to obtain target image indication information;

[0024] Performing word segmentation processing on the target image indication information to obtain an indication information word segmentation sequence;

[0025] Text encoding is performed on the indication information word segmentation sequence to obtain the text embedding feature.

[0026] In some embodiments, extracting image features from the original image data to obtain image embedding features includes:

[0027] Performing image preprocessing on the original image data to obtain target image data; wherein the target image data includes a plurality of target sub-image data;

[0028] For each of the target sub-image data, performing image coding processing on the target sub-image data to obtain a sub-image coding feature;

[0029] Acquire image position information of each of the target sub-image data; wherein the image position information is used to represent the position of the target sub-image data in the target image data;

[0030] Based on the image position information, feature fusion is performed on a plurality of the sub-image encoding features to obtain the image embedding feature.

[0031] In some embodiments, performing image preprocessing on the original image data to obtain target image data includes:

[0032] Performing denoising processing on the original image data to obtain denoised image data;

[0033] Performing sampling processing on the denoised image data to obtain initial image data;

[0034] Performing image enhancement on the initial image data to obtain enhanced image data;

[0035] Perform image segmentation on the enhanced image data to obtain the target image data.

[0036] In some embodiments, the step of expanding the original image indication information to obtain the initial image indication information includes:

[0037] Normalizing the original image indication information to obtain standardized image indication information;

[0038] Performing text expansion on the standard image indication information through a preset text expansion model to obtain original image indication information;

[0039] The original image indication information is verified based on the standard image indication information to obtain the initial image indication information.

[0040] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides an image description device, the device comprising:

[0041] An image data acquisition module, used to acquire original image data and original image indication information;

[0042] An indication information expansion module, used for expanding the original image indication information to obtain initial image indication information;

[0043] An image feature extraction module is used to extract image features from the original image data to obtain image embedding features;

[0044] A text feature extraction module, used to extract text features from the initial image indication information to obtain text embedding features;

[0045] A feature fusion module, used for fusing the image embedding feature and the text embedding feature to obtain a target information embedding feature;

[0046] The description information generating module is used to perform image description on the original image data according to the target information embedding feature to obtain target image description information.

[0047] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0048] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0049] The image description method and device, electronic device and storage medium proposed in the present application obtain initial image indication information by expanding the original image indication information, thereby enriching the semantic details of the description text; then, image feature extraction is performed on the original image data to obtain image embedding features, and text feature extraction is performed on the initial image indication information to obtain text embedding features, thereby capturing the visual content in the image and the language characteristics in the text; the image embedding features and the text embedding features are feature fused to obtain the target information embedding features; finally, the original image data is described according to the target information embedding features, thereby generating target image description information that is both accurate and rich in details, making full use of the complementarity between the image data and the image indication information, not only improving the accuracy of the image description, but also enhancing the richness and readability of the image description, so that the generated image description is closer to human understanding and expression habits. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a flowchart of an image description method provided in an embodiment of the present application;

[0051] Figure 2 yes Figure 1 Flow chart of step S102 in FIG.

[0052] Figure 3 yes Figure 1 Flow chart of step S103 in FIG.

[0053] Figure 4 yes Figure 3 Flow chart of step S301 in FIG.

[0054] Figure 5 yes Figure 1 Flow chart of step S104 in FIG.

[0055] Figure 6 yes Figure 1Flow chart of step S105 in FIG.

[0056] Figure 7 yes Figure 6 Flow chart of step S603 in FIG.

[0057] Figure 8 is a schematic diagram of the structure of an image description device provided in an embodiment of the present application;

[0058] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0062] First, some nouns involved in this application are analyzed:

[0063] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0064] Image Caption: Image caption is a technique for generating natural language descriptions for images. The generated descriptions can help applications understand the semantics expressed in the visual scene of the image. For example, image caption can convert image retrieval into text retrieval, which is used to classify images and improve image retrieval results. People usually only need to take a quick glance to describe the details of the visual scene of an image, while automatically adding descriptions to images is a comprehensive and arduous computer vision task that requires converting the complex information contained in the image into natural language descriptions. Compared with ordinary computer vision tasks, image captioning requires not only identifying objects from images, but also associating the identified objects with natural semantics and describing them in natural language. Therefore, image captioning requires people to extract deep features of images, associate them with semantic features, and convert them for generating descriptions.

[0065] In addition, image description can be applied in financial scenarios to describe image information of massive financial images and establish an image information database, which can improve the efficiency and accuracy of image retrieval.

[0066] At present, machine learning models are mainly used to directly generate image information from images. However, in actual applications, due to the wide variety of image types, machine learning models cannot accurately identify the core content of each image and accurately describe the image information, which affects the accuracy of image description.

[0067] Based on this, the embodiments of the present application provide an image description method and device, an electronic device and a storage medium, aiming to improve the accuracy of image description.

[0068] The image description method and device, electronic device, and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the image description method in the embodiments of the present application is described.

[0069] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0070] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0071] The image description method provided in the embodiment of the present application relates to the field of artificial intelligence and financial technology. The image description method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the image description method, etc., but is not limited to the above forms.

[0072] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0073] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0074] Figure 1 is an optional flowchart of the image description method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S106.

[0075] Step S101, obtaining original image data and original image indication information;

[0076] Step S102, expanding the original image indication information to obtain initial image indication information;

[0077] Step S103, extracting image features from the original image data to obtain image embedding features;

[0078] Step S104, extracting text features from the initial image indication information to obtain text embedding features;

[0079] Step S105, fusing the image embedding features and the text embedding features to obtain the target information embedding features;

[0080] Step S106, performing image description on the original image data according to the target information embedding feature to obtain target image description information.

[0081] In steps S101 to S106 illustrated in the embodiment of the present application, first, the original image indication information is expanded to obtain the initial image indication information, thereby enriching the semantic details of the description text; then, image feature extraction is performed on the original image data to obtain image embedding features, and text feature extraction is performed on the initial image indication information to obtain text embedding features, thereby capturing the visual content in the image and the language characteristics in the text; the image embedding features and the text embedding features are fused to obtain the target information embedding features; finally, the original image data is described according to the target information embedding features, thereby generating target image description information that is both accurate and rich in details, making full use of the complementarity between the image data and the image indication information, not only improving the accuracy of the image description, but also enhancing the richness and readability of the image description, so that the generated image description is closer to human understanding and expression habits.

[0082] In step S101 of some embodiments, in a financial field scenario, the original image data is a financial image, wherein the financial image includes but is not limited to: financial statements and account book images, transaction vouchers and transaction record images, financial charts and report images, identity authentication documents, qualification certification documents, etc.

[0083] By describing the image information of massive financial images and establishing a complete image information database, the efficiency and accuracy of image retrieval can be improved, which is conducive to business development.

[0084] It should be noted that the original image indication information is used to guide the content direction of the image description, helping the model to quickly locate the key information of the image, thereby improving the accuracy of the image description.

[0085] In some embodiments, the original image indication information may be annotations, comments, etc. recorded by technicians or business personnel when obtaining the original image data, or may be extracted through a pre-trained image recognition model, but is not limited thereto.

[0086] For example:

[0087] If the original image data is a financial report, the original image indication information may be "xx industry financial report, with table";

[0088] If the original image data is a qualification certification document, the original image indication information may be "business license".

[0089] See also Figure 2 In some embodiments, step S102 may include but is not limited to steps S201 to S203:

[0090] Step S201, normalizing the original image indication information to obtain normalized image indication information;

[0091] Step S202, performing text expansion on the standard image indication information through a preset text expansion model to obtain original image indication information;

[0092] Step S203: verifying the original image indication information based on the standard image indication information to obtain initial image indication information.

[0093] In the steps S201 to S203 shown in the embodiment of the present application, the original image indication information is normalized to ensure that the data input into the text expansion model is accurate; then, the standard image indication information is expanded through the preset text expansion model to obtain richer and more detailed original image indication information. Finally, the original image indication information is verified based on the standard image indication information to further ensure the accuracy and relevance of the expanded content, improve the efficiency and accuracy of text expansion, ensure that the final generated initial image indication information not only meets the language specifications, but also can fully express the core information of the original image data, which helps to improve the accuracy of subsequent image descriptions.

[0094] In step S201 of some embodiments, the original image indication information is normalized, wherein the normalization includes but is not limited to removing irrelevant characters, correcting spelling errors, converting to a uniform format (such as all lowercase), performing word form restoration, etc., which can be achieved by calling a natural language processing tool, but is not limited to this.

[0095] In step S202 of some embodiments, the preset text expansion model may be a large language model, such as a GPT model, a text generation variant in the BERT series, etc., but is not limited thereto, and the content input to the text expansion model may be "using 'standard image indication information' to expand sentences and retain 'standard image indication information' in the original sentence."

[0096] In some embodiments, step S203 may include but is not limited to the following steps:

[0097] Querying the original image indication information based on the standard image indication information to obtain an information query result;

[0098] If the information query result indicates that the original image indication information contains the standard image indication information, the original image indication information is used as the initial image indication information;

[0099] If the information query result indicates that the original image indication information does not include the standard image indication information, the standard image indication information is re-textualized using the text expansion model, that is, the process returns to step S202 and step S203.

[0100] For example: if the original image data is a qualification certificate document, the original image indication information is "Business License", the standard image indication information after normalization is "Business License", and the original image indication information obtained after text expansion through the text expansion model is "When applying for a company account, you need to provide a business license as proof of identity."

[0101] In some embodiments, the standard image indication information is text-expanded through a preset text expansion model to obtain the original image indication information. The number of the original image indication information may be one or more. When the number of the original image indication information is multiple, one may be selected based on specific rules. Specifically, the specific rules may be selecting the original image indication information with the greatest matching degree, or the original image indication information with the largest number of words, but is not limited thereto.

[0102] In some embodiments, after step S202 and before step S203, the image description method may further include: performing grammar checking, fluency evaluation, content adjustment, etc. on the original image indication information.

[0103] See also Figure 3 In some embodiments, step S103 may include but is not limited to steps S301 to S304:

[0104] Step S301, performing image preprocessing on the original image data to obtain target image data; wherein the target image data includes a plurality of target sub-image data;

[0105] Step S302, for each target sub-image data, performing image coding processing on the target sub-image data to obtain a sub-image coding feature;

[0106] Step S303, obtaining image position information of each target sub-image data; wherein the image position information is used to represent the position of the target sub-image data in the target image data;

[0107] Step S304: performing feature fusion on multiple sub-image coding features based on the image position information to obtain image embedding features.

[0108] In steps S301 to S304 shown in the embodiment of the present application, target image data is obtained by performing image preprocessing on the original image data; wherein the target image data includes multiple target sub-image data, and then for each target sub-image data, image encoding processing is performed on the target sub-image data to obtain sub-image encoding features, and image position information of each target sub-image data is obtained; wherein the image position information is used to characterize the position of the target sub-image data in the target image data; based on the image position information, feature fusion is performed on multiple sub-image encoding features to obtain image embedding features, which can make full use of image spatial structure information and local detail features, enhance the comprehensiveness and accuracy of image feature expression, and thus improve the accuracy of subsequent image description.

[0109] See also Figure 4 In some embodiments, step S301 may include but is not limited to steps S401 to S404:

[0110] Step S401, performing denoising processing on the original image data to obtain denoised image data;

[0111] Step S402, sampling the denoised image data to obtain initial image data;

[0112] Step S403, performing image enhancement on the initial image data to obtain enhanced image data;

[0113] Step S404: performing image segmentation on the enhanced image data to obtain target image data.

[0114] In steps S401 to S404 shown in the embodiment of the present application, the original image data is denoised to reduce noise interference in the image and improve image quality; the denoised image data is then sampled to retain key information of the image and reduce the amount of data, thereby improving image processing efficiency; the initial image data is then enhanced to highlight features in the image; and finally, the enhanced image data is segmented to divide the image into multiple target sub-image data of the same size, thereby providing more detailed image data for subsequent processing and effectively improving the accuracy and efficiency of image description.

[0115] In step S401 of some embodiments, the original image data can be denoised by smoothing filtering (such as mean filtering, Gaussian filtering), median filtering and other techniques to obtain denoised image data, which can reduce noise interference in the image and improve image quality.

[0116] In step S402 of some embodiments, the denoised image data may be upsampled or downsampled to convert the size of the denoised image data to a target size, for example, the size of the denoised image data may be adjusted to a 256*256 image.

[0117] Specifically, it is necessary to select whether to perform upsampling or downsampling on the denoised image data according to the size relationship between the denoised image data and the target size;

[0118] When the size of the denoised image data is larger than the target size, downsampling is performed on the denoised image data; when the size of the denoised image data is smaller than the target size, upsampling is performed on the denoised image data; when the size of the denoised image data is equal to the target size, no sampling is performed on the denoised image data.

[0119] In step S403 of some embodiments, the initial image data is subjected to contrast enhancement, sharpening, and grayscale adjustment to improve the visual effect of the image and enhance the features of the region of interest in the image.

[0120] Specifically,

[0121] Contrast enhancement: Makes the image clearer by stretching the contrast range of the image.

[0122] Sharpen: Emphasizes the edges and details of an image, making it appear sharper.

[0123] Histogram equalization: Adjust the grayscale distribution of the image to make the contrast of the image more uniform.

[0124] In step S404 of some embodiments, the enhanced image data is segmented based on a preset image segmentation size to obtain target image data, wherein the target image data includes a plurality of target sub-image data.

[0125] For example, the size of the de-enhanced image data is adjusted to 256*256, the image segmentation size is 32*32, and the number of target sub-image data is 64.

[0126] In step S302 of some embodiments, image encoding processing may be performed on each target sub-image data through a picture encoder of a ViT (Vision Transformer) model, a DeiT (Data-efficient image Transformer) model, etc. to obtain multiple sub-image encoding features.

[0127] It should be noted that the target image data is segmented and includes multiple target sub-image data, each of which has its own position, i.e., image position information; for example, the position of the third target sub-image data is the first row and the third column;

[0128] In step S304 of some embodiments, multiple sub-image coding features are spliced ​​in a left-to-right and top-to-bottom order based on the image position information to obtain an image embedding feature;

[0129] For example, if the size of the target image data is 256*256, the size of the target sub-image data is 32*32, and the number of target sub-image data is 64, the image embedding feature can be expressed as:

[0130] A=[a1,a2,…,a64];

[0131] It should be noted that the sizes of the initial image data, the enhanced image data and the target image data are all preset target sizes, for example, a size of 256*256.

[0132] See also Figure 5 In some embodiments, step S104 may also include but is not limited to steps S501 to S503:

[0133] Step S501, performing text cleaning on the initial image indication information to obtain the target image indication information;

[0134] Step S502, performing word segmentation processing on the target image indication information to obtain an indication information word segmentation sequence;

[0135] Step S503: Perform text encoding processing on the indicative information word segmentation sequence to obtain text embedding features.

[0136] In steps S501 to S503 shown in the embodiment of the present application, irrelevant or erroneous text content is effectively removed by text cleaning of the initial image indication information, thereby ensuring the accuracy and purity of the information; word segmentation is performed on the target image indication information to divide continuous text into vocabulary units; finally, text encoding is performed on the indication information word segmentation sequence to obtain text embedding features, thereby improving the processing efficiency of text information, facilitating the rapid location of key information of the image, and thus improving the accuracy of the image description.

[0137] In step S501 of some embodiments, the text cleaning of the initial image indication information may be implemented by using regular expressions or NLP libraries (such as NLTK or spaCy), including but not limited to removing special symbols, punctuation marks, stop words, etc.

[0138] In step S502 of some embodiments, the target image indication information may be segmented by a segmentation tool, such as a Han lp tool, a Jieba tool, etc., to segment the continuous target image indication information into an indication information segmentation sequence consisting of multiple words or units.

[0139] In some embodiments, the indicator information word sequence includes a plurality of indicator information words.

[0140] In step S503 of some embodiments, each indicative information word in the indicative information word segmentation sequence is subjected to text encoding processing through a pre-trained word embedding model to obtain multiple text sub-embedding features as numerical representations of the indicative information word; and then the multiple text sub-embedding features are feature concatenated to obtain a text embedding feature.

[0141] In some embodiments, the pre-trained word embedding model can be a BERT model, a Word2Vec model, a GloVe model, or an embedding model based on deep learning, but is not limited thereto.

[0142] See also Figure 6 In some embodiments, step S105 includes but is not limited to steps S601 to S603:

[0143] Step S601, performing feature splicing on the image embedding feature and the text embedding feature to obtain an image-text splicing feature;

[0144] Step S602, performing matrix transformation on the image-text splicing features to obtain an image-text transformation matrix;

[0145] Step S603, performing attention calculation on the image-text transformation matrix to obtain target information embedding features.

[0146] Steps S601 to S603 shown in the embodiment of the present application effectively integrate information of two different modalities, image and text, by performing feature splicing of image embedding features and text embedding features, to form a more comprehensive image-text splicing feature. Subsequently, the spliced ​​image-text splicing features are matrix transformed, and the spatial structure and relationship of the features are adjusted through linear or nonlinear transformation to obtain an image-text transformation matrix. Finally, the attention mechanism is used to perform attention calculation on the transformed image-text transformation matrix to obtain the target information embedding feature. It can improve the richness and accuracy of feature expression, enhance the model's ability to capture important information, and thus improve the accuracy of image description.

[0147] In step S601 of some embodiments, since both the image embedding feature and the text embedding feature are in vector form, the feature concatenation is to sequentially concatenate the image embedding feature with the text embedding feature through a special separation tag [seq] into a feature vector, that is:

[0148] Image-text splicing features = [image embedding features, [seq], text embedding features].

[0149] In step S602 of some embodiments, the image-text splicing feature is transformed into a target matrix through linear or nonlinear transformation to obtain an image-text transformation matrix, which can better adapt to the image description task;

[0150] It should be noted that the dimension of the target matrix needs to be determined according to the dimension of the model output features.

[0151] In step S603 of some embodiments, the image-text transformation matrix may be subjected to attention calculation by a Transformer encoder having a single-head attention mechanism to obtain target information embedding features.

[0152] See also Figure 7 In some embodiments, step S603 may include but is not limited to steps S701 to S704:

[0153] Step S701, mapping and transforming the image-text transformation matrix to obtain a target image mapping matrix; wherein the target image mapping matrix includes a query mapping matrix, a key mapping matrix and a value mapping matrix;

[0154] Step S702, performing aggregation calculation on the query mapping matrix and the key mapping matrix to obtain an attention score;

[0155] Step S703, normalize the attention score to obtain the attention weight;

[0156] Step S704, perform aggregation calculation on the attention weight and the value mapping matrix to obtain the target information embedding feature.

[0157] In the steps S701 to S704 shown in the embodiment of the present application, the image-text transformation matrix is ​​mapped and transformed to decompose it into a query mapping matrix, a key mapping matrix and a value mapping matrix. By performing aggregation calculations on the query mapping matrix and the key mapping matrix, the dependency between different elements in the image-text transformation matrix can be measured to obtain an attention score. Subsequently, the attention score is normalized, different attention weights are assigned to different elements, and the attention weights and the value mapping matrix are aggregated to obtain the target information embedding feature, which improves the learning ability of the image-text transformation matrix and helps to improve the accuracy of image description.

[0158] In some embodiments, step S701 may include but is not limited to the following steps:

[0159] Performing mapping transformation on the image-text transformation matrix to obtain an initial image mapping matrix; wherein the initial image mapping matrix includes a query matrix, a key matrix and a value matrix;

[0160] Perform matrix calculation on the query matrix based on the preset learning factor to obtain the query mapping matrix;

[0161] Perform matrix calculation on the key matrix based on the preset learning factor to obtain a key mapping matrix;

[0162] Perform matrix calculation on the value matrix based on the preset learning factor to obtain a value mapping matrix;

[0163] The query mapping matrix, the key mapping matrix and the value mapping matrix are combined to obtain a target image mapping matrix.

[0164] The preset learning factor is a randomly generated matrix, and the data dimension of the learning factor is the same as the data dimension of the image-text transformation matrix, which is used to adjust the query matrix, key matrix and value matrix for attention calculation, so as to better learn the information in the image-text transformation matrix. Specifically, the matrix calculation process can use vector multiplication or vector addition, but is not limited thereto.

[0165] Specifically, by calculating the dot product of the query mapping matrix and the key mapping matrix, and applying the softmax function, Z-score normalization, maximum normalization and other methods to perform normalization calculations according to the column dimension (usually the sequence length dimension), the attention weight is obtained, which reflects the correlation strength between different elements in the image-text transformation matrix. Finally, the attention weight and the value mapping matrix are weightedly calculated to obtain the target information embedding features.

[0166] In step S106 of some embodiments, the target information embedded features are decoded by a preset image description generation model to generate target image description information corresponding to the original image data.

[0167] Specifically, the preset image description generation model can be an autoregressive model, which acts as a decoder. The image description generation model is used to decode the target information embedded features, generate multiple words, and concatenate the multiple words to obtain complete description information, that is, the target image description information.

[0168] See also Figure 8 The embodiment of the present application further provides an image description device, which can implement the above-mentioned image description method, and the device includes:

[0169] The image data acquisition module 801 is used to acquire original image data and original image indication information;

[0170] The indication information expansion module 802 is used to expand the original image indication information to obtain the initial image indication information;

[0171] The image feature extraction module 803 is used to extract image features from the original image data to obtain image embedding features;

[0172] A text feature extraction module 804 is used to extract text features from the initial image indication information to obtain text embedding features;

[0173] A feature fusion module 805 is used to fuse the image embedding feature and the text embedding feature to obtain the target information embedding feature;

[0174] The description information generating module 806 is used to perform image description on the original image data according to the target information embedding feature to obtain target image description information.

[0175] The specific implementation of the image description device is substantially the same as the specific implementation of the above-mentioned image description method, and will not be described in detail herein.

[0176] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned image description method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0177] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0178] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0179] The memory 902 may be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 may store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the image description method of the embodiment of this application;

[0180] Input / output interface 903, used to implement information input and output;

[0181] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WI FI, Bluetooth, etc.);

[0182] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0183] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0184] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned image description method is implemented.

[0185] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0186] The image description method and device, electronic device and storage medium provided in the embodiments of the present application obtain initial image indication information by expanding the original image indication information, thereby enriching the semantic details of the description text; then, image feature extraction is performed on the original image data to obtain image embedding features, and text feature extraction is performed on the initial image indication information to obtain text embedding features, thereby capturing the visual content in the image and the language characteristics in the text; the image embedding features and the text embedding features are feature fused to obtain target information embedding features; finally, image description is performed on the original image data according to the target information embedding features, thereby generating target image description information that is both accurate and rich in details, making full use of the complementarity between the image data and the image indication information, not only improving the accuracy of the image description, but also enhancing the richness and readability of the image description, so that the generated image description is closer to human understanding and expression habits.

[0187] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0188] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0189] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0190] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0191] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0192] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0193] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0194] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0195] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0196] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0197] The preferred embodiments of the present application are described above with reference to the accompanying drawings, but the scope of the rights of the present application is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present application should be within the scope of the rights of the present application.

Claims

1. An image description method, characterized in that: The method comprises: Acquiring original image data and original image indication information; Performing expansion processing on the original image indication information to obtain initial image indication information; Performing image feature extraction on the original image data to obtain image embedding features; Extracting text features from the initial image indication information to obtain text embedding features; Fusing the image embedding feature with the text embedding feature to obtain a target information embedding feature; The original image data is described according to the target information embedding feature to obtain target image description information.

2. The method according to claim 1, characterized in that: The step of fusing the image embedding feature and the text embedding feature to obtain the target information embedding feature includes: Performing feature splicing on the image embedding feature and the text embedding feature to obtain an image-text splicing feature; Performing matrix transformation on the image-text splicing features to obtain an image-text transformation matrix; Perform attention calculation on the image-text transformation matrix to obtain the target information embedding feature.

3. The method according to claim 2, characterized in that The performing attention calculation on the image-text transformation matrix to obtain the target information embedding feature includes: Performing mapping transformation on the image-text transformation matrix to obtain a target image mapping matrix; wherein the target image mapping matrix includes a query mapping matrix, a key mapping matrix and a value mapping matrix; Performing aggregation calculation on the query mapping matrix and the key mapping matrix to obtain an attention score; Normalizing the attention score to obtain an attention weight; The attention weight and the value mapping matrix are aggregated and calculated to obtain the target information embedding feature.

4. The method according to any one of claims 1 to 3, characterized in that: The extracting text features from the initial image indication information to obtain text embedding features includes: Performing text cleaning on the initial image indication information to obtain target image indication information; Performing word segmentation processing on the target image indication information to obtain an indication information word segmentation sequence; Text encoding is performed on the indication information word segmentation sequence to obtain the text embedding feature.

5. The method according to any one of claims 1 to 3, characterized in that: The extracting image features from the original image data to obtain image embedding features includes: Performing image preprocessing on the original image data to obtain target image data; wherein the target image data includes a plurality of target sub-image data; For each of the target sub-image data, performing image coding processing on the target sub-image data to obtain a sub-image coding feature; Acquire image position information of each of the target sub-image data; wherein the image position information is used to represent the position of the target sub-image data in the target image data; Based on the image position information, feature fusion is performed on a plurality of the sub-image encoding features to obtain the image embedding feature.

6. The method according to claim 5, characterized in that The performing image preprocessing on the original image data to obtain target image data includes: Performing denoising processing on the original image data to obtain denoised image data; Performing sampling processing on the denoised image data to obtain initial image data; Performing image enhancement on the initial image data to obtain enhanced image data; Perform image segmentation on the enhanced image data to obtain the target image data.

7. The method according to any one of claims 1 to 3, characterized in that: The step of performing expansion processing on the original image indication information to obtain the initial image indication information includes: Normalizing the original image indication information to obtain standardized image indication information; Performing text expansion on the standard image indication information through a preset text expansion model to obtain original image indication information; The original image indication information is verified based on the standard image indication information to obtain the initial image indication information.

8. An image description device, characterized in that: The device comprises: An image data acquisition module, used to acquire original image data and original image indication information; An indication information expansion module, used for expanding the original image indication information to obtain initial image indication information; An image feature extraction module is used to extract image features from the original image data to obtain image embedding features; A text feature extraction module, used to extract text features from the initial image indication information to obtain text embedding features; A feature fusion module, used for fusing the image embedding feature and the text embedding feature to obtain a target information embedding feature; The description information generating module is used to perform image description on the original image data according to the target information embedding feature to obtain target image description information.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images