Multimodal picture understanding method and device based on cross-modal label fusion

Abstract

The embodiment of the disclosure discloses a multi-modal picture understanding method and device based on cross-modal mark fusion, and relates to the technical field of artificial intelligence. The method comprises the following steps: obtaining an image and text included in a multi-modal picture; inputting the image into an image encoder to obtain at least one image feature, determining a first visual token based on the at least one image feature; and obtaining a first text token included in the text by using a text segmenter; obtaining a second visual token based on the first visual token and the first text token; and inputting the second visual token and the first text token into a preset large language model after merging to understand the multi-modal picture. The embodiment of the disclosure uses a weighted combination of a text token set to represent a visual token, and adopts cross-attention weight to fuse the semantic expressions of multiple modal marks, thereby effectively improving the understanding of the multi-modal picture.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, and in particular, the present disclosure relates to a multi-modal picture understanding method and device based on cross-modal label fusion. BACKGROUND

[0002] The technical field of multi-modal picture understanding is one of the hot research directions in the field of artificial intelligence in recent years. It integrates data of multiple modalities (such as images and texts) to comprehensively understand and process, simulate human multi-sensory perception, and realize deep understanding and semantic analysis of image content.

[0003] In the prior art, data of different modalities (such as images and texts) have different feature dimensions and representation methods. For example, an image is a high-dimensional pixel value, while a text is a discrete symbol sequence. This difference makes the fusion between data of different modalities complicated, thereby affecting the understanding of multi-modal pictures. SUMMARY

[0004] The embodiments of the present disclosure provide a multi-modal picture understanding method and device based on cross-modal label fusion, which can solve the technical problem that the feature dimensions and representation methods of data of different modalities are different and cannot be deeply fused in the prior art. The technical solutions provided by the present disclosure are as follows:

[0005] According to a first aspect of the embodiments of the present disclosure, a multi-modal picture understanding method based on cross-modal label fusion is provided, which comprises:

[0006] obtaining an image and a text included in a multi-modal picture;

[0007] inputting the image into an image encoder to obtain at least one image feature, determining a first visual token based on the at least one image feature, and obtaining a first text token included in the text using a text segmenter;

[0008] obtaining a second visual token based on the first visual token and the first text token; wherein the second visual token comprises a weighted combination visual token component and an attention visual token component; the weighted combination visual token component is determined by weighted summation of second text tokens in a text token set, and the attention visual token component is determined by cross-attention calculation of the first visual token and the first text token;

[0009] The second visual token and the first text token are input into a preset large language model after being combined, and the multi-modal picture is understood.

[0010] As an optional implementation, the first visual token is determined based on the at least one image feature, including:

[0011] The at least one image feature is combined to obtain a third visual token.

[0012] The dimension of the text token in the text token set is determined, and the dimension of the third visual token is linearly mapped to the dimension of the text token in the text token set based on a first preset matrix to obtain the first visual token, the dimension of the first visual token being the same as the dimension of the text token.

[0013] As an optional implementation, the weighted combination visual token component is determined by weighted summation of the second text token in the text token set, including:

[0014] A text token set is determined, the text token set including at least one text token, each text token in the text token set being obtained by performing word segmentation on target text;

[0015] The first visual token is linearly mapped to determine the second text token and the weight of the second text token; the second text token being a text token in the text token set;

[0016] The second text token is weighted summed based on the second text token and the weight of the second text token to determine the weighted combination visual token component.

[0017] As an optional implementation, the first visual token is linearly mapped to determine the weight of the second text token, including:

[0018] The component of the first visual token on the second text token is determined based on the first visual token and a second preset matrix;

[0019] The component of the second text token is normalized to determine the weight of the second text token.

[0020] As an optional implementation, the attention visual token component is determined by cross-attention calculation of the first visual token and the first text token, and includes:

[0021] Based on the first visual token and the first text token, a first cross-attention weight matrix of the first visual token and the first text token is determined.

[0022] Based on the first cross-attention weight matrix and the set of text tokens, the attention visual token component is determined.

[0023] As an optional implementation, each text token in the set of text tokens includes a corresponding text token identifier.

[0024] The determination of the attention visual token component based on the first cross-attention weight matrix and the set of text tokens includes:

[0025] Based on the dimension of the text token in the set of text tokens, the dimension of each first visual token in the cross-attention weight matrix is expanded to the same dimension as the dimension of the text token in the set of text tokens, to obtain a second cross-attention weight matrix.

[0026] Based on the size relationship of the text token identifier in the set of text tokens, the order of the text token associated with each first visual token in the cross-attention weight matrix is adjusted to obtain a third cross-attention weight matrix.

[0027] Based on the third cross-attention weight matrix and the set of text tokens, the attention visual component is determined.

[0028] As an optional implementation, the determination of the attention visual component based on the third cross-attention weight matrix and the set of text tokens includes:

[0029] The third cross-attention weight matrix is multiplied by the set of text tokens to obtain the attention visual component.

[0030] According to a second aspect of the embodiments of the present disclosure, a multi-modal picture understanding device based on cross-modal label fusion is provided, and the device includes:

[0031] A first processing module is configured to obtain an image and text included in a multi-modal picture.

[0032] The second processing module is configured to input the image into an image encoder to obtain at least one image feature, determine a first visual token based on the at least one image feature, and obtain a first text token included in the text by using a text segmenter;

[0033] The third processing module is configured to obtain a second visual token based on the first visual token and the first text token; the second visual token includes a weighted combination visual token component and an attention visual token component; the weighted combination visual token component is determined by weighted summation of second text tokens in a text token set, and the attention visual token component is determined by cross-attention calculation of the first visual token and the first text token.

[0034] The fourth processing module is configured to input the second visual token and the first text token into a preset large language model after merging to understand the multi-modal picture.

[0035] According to a third aspect of the embodiments of the present disclosure, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory, and the processor executes the computer program to implement the steps of the method according to any one of the first aspect.

[0036] According to a fourth aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the steps of the method according to any one of the first aspect.

[0037] The technical scheme provided by the embodiments of the present disclosure has the following beneficial effects:

[0038] The embodiments of the present disclosure provide a multi-modal picture understanding method and device based on cross-modal mark fusion. The embodiments of the present disclosure use weighted combination of a text token set to represent a visual token, to ensure that the data distribution of the visual token is consistent with the data distribution of the text token. Meanwhile, the cross-attention weight of the text token and the visual token is also used to fuse the multi-modal mark, which can effectively fuse the semantic expression of various modal marks and effectively improve the understanding of the multi-modal picture. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical scheme in the embodiments of the present disclosure, the drawings needed in the description of the embodiments of the present disclosure will be briefly introduced.

[0040] Figure 1A structural schematic diagram of a multi-modal picture understanding method based on cross-modal label fusion provided by an embodiment of the present disclosure is provided.

[0041] Figure 2 A flowchart of a multi-modal picture understanding method based on cross-modal label fusion provided by an embodiment of the present disclosure is provided.

[0042] Figure 3 A structural schematic diagram of a multi-modal picture understanding device based on cross-modal label fusion provided by an embodiment of the present disclosure is provided.

[0043] Figure 4 A structural schematic diagram of an electronic device provided by an embodiment of the present disclosure is provided. DETAILED DESCRIPTION

[0044] Embodiments of the present disclosure will be described below with reference to the accompanying drawings. It should be understood that the embodiments described below in conjunction with the drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present disclosure, and do not constitute a limitation on the technical solutions of the embodiments of the present disclosure.

[0045] Those skilled in the art can understand that, unless specifically stated otherwise, the singular forms "a", "an" and "the" used herein can also include the plural forms. It should be further understood that the terms "comprise" and "include" used in the embodiments of the present disclosure mean that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements and / or components, but do not exclude other features, information, data, steps, operations, elements, components and / or combinations thereof supported by the present technology. It should be understood that when we say that an element is "connected" or "coupled" to another element, the element can be directly connected or coupled to the other element, or it can mean that the element and the other element establish a connection relationship through an intermediate element. In addition, "connected" or "coupled" used herein can include wireless connection or wireless coupling. The term "and / or" used herein indicates that at least one of the items defined by the term, for example, "A and / or B" or "A, B" indicates implementation as "A", or implementation as "B", or implementation as "A and B".

[0046] In order to make the purpose, technical solutions and advantages of the present disclosure clearer, the embodiments of the present disclosure will be described in further detail below with reference to the accompanying drawings.

[0047] First, the technical terms related to the present disclosure are introduced and explained:

[0048] An image encoder is a tool that converts raw image data into abstract feature representations. The core function is to extract key information in images (such as color, shape, texture, object structure, etc.) and compress it into a more compact and more suitable feature form for subsequent processing.

[0049] Both visual tokens and text tokens are basic units used to represent different modal information in multimodal models. The former is used to represent visual information, and the latter is used to represent text information. The following is a specific introduction:

[0050] Visual tokens are a kind of discrete representation of visual content (such as images, videos). Mainstream models convert visual inputs such as images into visual tokens through visual encoders, and then input them into large language models to realize the understanding of visual information.

[0051] Text tokens are a kind of discrete representation of text information. Text tokens usually have a fixed vocabulary, and each token corresponds to an element in the vocabulary. Multimodal models capture the semantic information of text by learning the relationship between these text tokens.

[0052] The text token set is a collection of all non-repeated minimal semantic units (i.e. text tokens) obtained by splitting a piece of text in natural language processing. It is the product of converting text from its original literal form into a basic unit that can be processed by a model.

[0053] The field of multimodal (Multimodal Large Language Model, MLLM) picture understanding technology is one of the hot directions of artificial intelligence research in recent years. It integrates data from multiple modalities (such as images, text, etc.), comprehensively understands and processes, and simulates the ability of human multi-sensory perception, to realize the deep understanding and semantic analysis of image content.

[0054] In the prior art, data of different modalities (such as images and text) have different feature dimensions and representation methods. For example, images are high-dimensional pixel values, while text is a discrete symbol sequence. This difference makes the fusion between different modal data complex, thereby affecting the understanding of multimodal pictures.

[0055] The multimodal picture understanding method and device based on cross-modal token fusion provided by the present disclosure aim to solve at least one of the above technical problems in the prior art.

[0056] The technical solutions of the embodiments of the present disclosure and the technical effects produced by the technical solutions of the present disclosure will be described below through the description of several exemplary embodiments. It should be noted that the following embodiments can be mutually referenced, borrowed or combined. For the same terms, similar features and similar implementation steps in different embodiments, they will not be described repeatedly.

[0057] The embodiment of the disclosure provides a multi-modal picture understanding model based on cross-modal mark fusion, which is used for realizing a multi-modal picture understanding method based on cross-modal mark fusion provided by the embodiment of the disclosure. Figure 1 As shown in the figure, the multi-modal picture understanding model based on cross-modal mark fusion comprises an image encoder, a visual token fusioner, a text tokenizer and a large language model.

[0058] Specifically, in the embodiment of the disclosure, the image in the multi-modal picture is input into the image encoder, and at least one image feature is output. The at least one image feature is merged to obtain a visual token.

[0059] Specifically, in the embodiment of the disclosure, the visual token fusioner comprises twice linear mapping, a normalized exponential function and a cross attention module; wherein the first linear mapping is used to align the dimension of the visual token with the dimension of the text token; the second linear mapping is used to convert the visual token to a component on each text token in the text token set; the normalized exponential function is used to normalize the component on each text token, to obtain the weight of each text token in the text token set, and to obtain a weighted combined visual token component by weighted summation according to each text token in the text token set; the cross attention module is used to calculate a cross attention component of the visual token and the text token; and based on the weighted combined visual token component and the cross attention component, a visual token output by the visual token fusioner is obtained.

[0060] Specifically, in the embodiment of the disclosure, the text tokenizer is used to perform tokenization processing on the text in the multi-modal picture to obtain a first text token.

[0061] Specifically, in the embodiment of the disclosure, the visual token output by the visual token fusioner and the text token output by the text tokenizer are merged and input into the large language model. The large language model is used to perform associated understanding on the image and the text in the multi-modal picture, and to output text description information of the multi-modal picture.

[0062] Figure 2 A flowchart of a multi-modal picture understanding method based on cross-modal mark fusion provided by the embodiment of the disclosure is shown in the figure. Figure 2 As shown in the figure, the method comprises:

[0063] S201, obtaining an image and a text included in a multi-modal picture.

[0064] Specifically, in the embodiment of the disclosure, the image is the core carrier of the multi-modal picture, and the purpose of obtaining the image is to extract original visual data to provide a basis for subsequent processing.

[0065] Specifically, in the embodiments of the present disclosure, for the text directly existing in the multi-modal picture, an OCR (Optical Character Recognition) technology can be used to recognize the text in the image, and the OCR technology can convert the text pixels in the multi-modal picture into editable text strings.

[0066] S202, input the image into an image encoder to obtain at least one image feature, determine a first visual token based on the at least one image feature, and obtain first text tokens included in the text using a text segmenter.

[0067] Specifically, in the embodiments of the present disclosure, the image encoder is a model specially processing images, which extracts and compresses features of the image and finally outputs at least one image feature; wherein the image feature is an abstract representation of the image, which contains key information (such as color, shape, texture or object contour, etc.) of the image, and is no longer in the form of original pixels.

[0068] Specifically, in the embodiments of the present disclosure, the first visual token is obtained based on the image feature output by the encoder. The first visual token can be understood as the smallest semantic unit of the image, or the image can be divided into multiple small regions, and the features of each region are taken as a visual token.

[0069] Specifically, in the embodiments of the present disclosure, the text segmenter is a tool specially used for splitting text, which splits continuous text into discrete units according to certain rules, and the units obtained after splitting are the first text tokens, which are the smallest semantic units of the text.

[0070] S203, obtain a second visual token based on the first visual token and the first text token; wherein the second visual token includes a weighted combination visual token component and an attention visual token component; the weighted combination visual token component is determined by weighted summation of the second text token in the text token set, and the attention visual token component is determined by cross-attention calculation of the first visual token and the first text token.

[0071] Specifically, in the embodiments of the present disclosure, the first visual token is the smallest semantic unit extracted from the image, and the first text token is the smallest semantic unit split from the text. Through the processing of these two types of tokens, the generated second visual token is no longer a pure visual feature, but an enhanced visual representation that fuses the text semantics, which can better associate the visual content with the text semantics.

[0072] Specifically, in the embodiments of the present disclosure, the second visual token includes two components, which are a weighted combination visual token component and an attention visual token component.

[0073] The weighted combination visual token component is obtained by weighted summation of second text tokens in the text token set, and the second text token refers to a text token that is most closely associated with the first visual token and is filtered from the text token set; each second text token is assigned a weight (the weight reflects the association strength of the second text token with the first visual token), and the second text tokens are added according to the weights to obtain the weighted combination visual token component; that is, the embodiments of the present disclosure use the weighted combination of the text token set to represent the visual token, so as to ensure that the data distribution of the visual token is consistent with the data distribution of the text token. The attention visual token component is calculated by the cross-attention mechanism, and the cross-attention weight between the text token and the visual token is calculated to enhance the influence of the text token on the visual token, thereby effectively fusing the semantic expressions of various modal data to improve the effect of multi-modal picture understanding.

[0074] S204, input the second visual token and the first text token into a preset large language model after merging, and understand the multi-modal picture.

[0075] Specifically, in the embodiments of the present disclosure, the merging of the second visual token and the first text token is not a simple splicing, but the integration of two different types of tokens into a unified input format that can be recognized by the large language model. By merging the two types of tokens, the core features observed in the image and the semantic information mentioned in the text are integrated together, so that the large language model can obtain both "visual clues" and "language clues".

[0076] Specifically, in the embodiments of the present disclosure, the large language model is a model that can understand and generate natural language. Based on the merged visual token and text token, the large language model finds the internal relationship between the visual features and the text semantics; at the same time, the large language model utilizes its language generation capability to convert this association into coherent natural language, thereby realizing the understanding of the multi-modal picture.

[0077] In the embodiments of the present disclosure, the weighted combination of the text token set is used to represent the visual token, so as to ensure that the data distribution of the visual token is consistent with the data distribution of the text token; meanwhile, by calculating the cross-attention weight of the text token and the visual token, the influence of the text token on the visual token is enhanced, the semantic expression of various modal data is effectively fused, and the understanding of the multi-modal picture is improved.

[0078] On the basis of the above-mentioned embodiments, as an optional embodiment, the first visual token is determined based on at least one image feature, including:

[0079] The at least one image feature is merged to obtain a third visual token;

[0080] The dimension of the text token in the text token set is determined; the dimension of the third visual token is linearly mapped to the dimension of the text token in the text token set based on a first preset matrix to obtain the first visual token, and the dimension of the first visual token is the same as the dimension of the text token.

[0081] Specifically, in the embodiments of the present disclosure, the multiple image features output by the image encoder can be features of different regions and different levels of the image. These dispersed image features are integrated into a unified feature representation, and the integrated feature representation is the third visual token.

[0082] Specifically, in the embodiments of the present disclosure, the text token in the text token set has its inherent dimension feature, and the dimension is usually determined by the representation method of the text. For example, each text token is represented by a 512-dimensional vector.

[0083] Specifically, in the embodiments of the present disclosure, the third visual token is linearly mapped by the first preset matrix to finally obtain the first visual token; wherein the first preset matrix is a pre-set (or obtained by training and learning) weight matrix, which is used to convert the dimension of the third visual token to the same dimension as the text token set. Linear mapping is a dimension conversion realized by matrix operation. For example: the third visual token is 1024-dimensional, and the text token in the text token set is 512-dimensional. The matrix will compress or convert the 1024-dimensional visual feature to 512-dimensional. The dimension of the first visual token obtained after linear mapping is completely consistent with the dimension of the text token in the text token set, which provides a basic condition for the subsequent association between the visual token and the text token.

[0084] Specifically, in the embodiments of the present disclosure, the calculation formula of the first visual token is shown in formula (1):

[0085]

[0086] wherein, is the first visual token, N is the length of the first visual token, and D represents the dimension of the text token. is a weight matrix, which is responsible for aligning the dimension of the first visual token and the dimension of the text token set. is the third visual token, N is the length of the first visual token, and d represents the dimension of the first visual token. As can be seen from formula (1), the dimension of the first visual token is the same as the dimension of the text token set.

[0087] The embodiments of the present disclosure linearly map the dimension of the third visual token to the dimension by using the first preset matrix, so that the dimension of the first visual token is completely consistent with the dimension of the text token, thereby breaking the "dimension barrier" of the visual feature and the text feature in the original representation, and providing a basic condition for the subsequent association of the visual token and the text token.

[0088] On the basis of the above embodiments, as an optional embodiment, the weighted combination visual token component is determined by weighted summation of the second text token in the text token set, comprising:

[0089] determining a text token set, the text token set comprising at least one text token, each text token in the text token set being obtained by performing a word segmentation process on a target text;

[0090] linearly mapping the first visual token to determine the second text token and the weight of the second text token; the second text token being a text token in the text token set;

[0091] based on the second text token and the weight of the second text token, performing weighted summation on the second text token to determine the weighted combination visual token component.

[0092] Specifically, in the embodiments of the present disclosure, the text token set is obtained by performing a word segmentation process on a target text (obtained after the word segmentation process, wherein the word segmentation process splits the continuous target text into a plurality of minimum semantic units (i.e. text tokens), and all the split text tokens collectively constitute the "text token set" after deduplication.

[0093] Specifically, in the embodiments of the present disclosure, the linear mapping is to convert the features of the first visual token to the semantic space of the text token through a preset weight matrix, to determine the component of the first visual token on each text token in the text token set; then, the component on each text token is converted into a probability value as the weight of the second text token through a normalization exponential function Softmax; wherein the higher the weight, the higher the matching degree of the text token with the first visual token. Finally, based on the screened second text token and its weight, the weighted combination visual token component is obtained through weighted summation.

[0094] It should be noted that the second text token is a text token on the text token set, which can be understood as taking the text token in the text token set as the second text token.

[0095] Specifically, in the embodiments of the present disclosure, the dimensions of the first visual token and the text token are aligned through linear mapping, thereby eliminating the differences in feature dimensions and representation forms between the visual token and the text token of two modalities, providing conditions for subsequent cross-modal fusion and interaction.

[0096] On the basis of the above embodiments, as an optional embodiment, the linear mapping is performed on the first visual token to determine the weight of the second text token, comprising:

[0097] Based on the first visual token and the second preset matrix, the component of the first visual token on the second text token is determined;

[0098] The component of the second text token is normalized to determine the weight of the second text token.

[0099] Specifically, in the embodiments of the present disclosure, based on the first visual token and the second preset matrix, the component of the first visual token corresponding to each text token in the text token set is determined, and the component is a specific value obtained through linear mapping, which directly reflects the original association degree of the first visual token with each text token; wherein the greater the value of the association degree, the closer the association between the visual token and the corresponding text token. Based on the component of the second text token, the component of the second text token is converted into a weight through normalization; wherein the weight of each second text token is between 0 and 1, and the sum of the weights of all second text tokens is 1.

[0100] Specifically, in the embodiments of the present disclosure, the calculation formula of the weighted combination of the visual token components is shown in formula (2):

[0101]

[0102] wherein, is a second preset matrix, which is a weight matrix for converting the visual token into the text token set; N is the length of the first visual token, and V is the size of the text token set (i.e., the number of text tokens in the text token set).

[0103] Specifically, in the embodiments of the present disclosure, the calculation formula of the weight of the second text token is shown in formula (3):

[0104]

[0105] wherein, is the weight of the second text token, is a normalized exponential function, which is used to normalize the component of the second text token and output a corresponding probability value as the weight of the second text token.

[0106] Specifically, in the embodiments of the present disclosure, after obtaining the weight of the second text token, the calculation formula of the weighted combination of the visual token components is shown in formula (4):

[0107]

[0108] wherein, is the text token set, V is the size of the text token set, and D is the dimension of the text token set.

[0109] In the embodiments of the present disclosure, through the linear mapping of the first visual token and the second preset matrix, the component of each text token corresponding to the visual feature is calculated, the abstract visual feature is converted into a quantitative associated value with the text token, and it is clear that "which text semantics correspond to the visual content" has a numerical basis.

[0110] On the basis of the above embodiments, as an optional embodiment, the attention visual token component is determined by the first visual token and the first text token through cross-attention calculation, comprising:

[0111] determining a first cross-attention weight matrix of the first visual token and the first text token based on the first visual token and the first text token;

[0112] determine the attention visual token component based on the first cross-attention weight matrix and the text token set.

[0113] Specifically, in the embodiments of the present disclosure, each numerical value in the first cross-attention weight matrix represents the association strength between the first visual token and the first text token. The higher the numerical value, the stronger the correlation between the first visual token and the first text token.

[0114] Specifically, in the embodiments of the present disclosure, the calculation formula of the first cross-attention weight matrix is as shown in formula (5):

[0115]

[0116] wherein, is the first cross-attention weight matrix; is the first visual token; is the first text token.

[0117] Specifically, in the embodiments of the present disclosure, the first cross-attention weight matrix has quantified the association strength between the first visual token and the first text token; and the text token set contains all basic semantic units (i.e. text tokens) in the text. Through the combination of the two, the first visual feature is focused on the most relevant text token in the text token set; that is, the text token with high association strength will occupy a more important position in the visual component, and the text token with weak association will be weakened.

[0118] In the embodiments of the present disclosure, the attention visual token component not only retains the core visual features of the image, but also deeply integrates the most relevant semantics in the text and the visual, thereby realizing the targeted association of visual and text information.

[0119] On the basis of the above embodiments, as an optional embodiment, each text token in the text token set comprises a corresponding text token identifier.

[0120] determining the attention visual token component based on the first cross-attention weight matrix and the text token set, comprising:

[0121] based on the dimension of the text token in the text token set, expanding the dimension of each first visual token in the cross-attention weight matrix to the same dimension as the text token in the text token set, to obtain a second cross-attention weight matrix;

[0122] Based on the size relationship of the text token set, the order of the text token associated with each first visual token in the cross-attention weight matrix is adjusted to obtain a third cross-attention weight matrix.

[0123] Based on the third cross-attention weight matrix and the text token set, an attention visual component is determined.

[0124] Specifically, in the embodiments of the present disclosure, based on the dimension of the text token in the text token set, the first cross-attention weight matrix is dimensionally expanded so that the dimension of each first visual token in the matrix is consistent with the dimension of the text token, to obtain a second cross-attention weight matrix. For example, the first cross-attention matrix is , that is, N rows and M columns, N represents the length of the first visual token, and M represents the dimension of the first visual token; the dimension of the first visual token is widened to by filling with 0 values, to obtain , that is, N rows and V columns, N represents the length of the first visual token, and V represents the dimension of the first visual token.

[0125] Specifically, in the embodiments of the present disclosure, according to the size relationship (such as the numerical order of the identifier) of each text token identifier in the text token set, the order of the text token associated with each first visual token in the second cross-attention weight matrix is adjusted to obtain a third cross-attention weight matrix; wherein the size relationship of the text token identifier usually corresponds to the original syntax order or logical order of the text (for example, identifier 1 corresponds to the first text token in the text token set, and identifier 2 corresponds to the second text token in the text token set); after adjusting the order, the attention weight is consistent with the arrangement order of each text token in the text token set.

[0126] Specifically, in the embodiments of the present disclosure, the calculation formula of the attention visual component is as shown in formula (6):

[0127]

[0128] wherein, is the third cross-attention weight matrix; is the text token set.

[0129] In the embodiments of the present disclosure, based on the text token set and the third cross-attention weight matrix, the obtained attention visual token component not only retains the visual features, but also deeply fuses the semantic information of the text token in the text token set.

[0130] On the basis of each of the above embodiments, as an optional embodiment, based on the third cross-attention weight matrix and the text token set, the attention visual component is determined, including:

[0131] The third cross-attention weight matrix is multiplied with the text token set to obtain the attention visual component.

[0132] Specifically, in the embodiments of the present disclosure, each value in the third cross-attention weight matrix represents the association strength of the first visual token and the corresponding text token in the text token set, and the matrix has completed dimension adjustment and order optimization, and can be accurately matched with the text token set; the text token set contains the feature representation of all text tokens.

[0133] Specifically, in the embodiments of the present disclosure, the third cross-attention weight matrix is multiplied with the text token set, which essentially weights each token feature in the text token set with attention weight, that is, the text token with high association strength (large weight value) has a higher proportion in the result; the text token with low association strength (small weight value) has a lower proportion.

[0134] In the embodiments of the present disclosure, by multiplying the third cross-attention weight matrix with the text token set, the core semantic information of the text token set is retained, and the association logic with the visual feature is reflected through the attention weight, realizing the targeted fusion of visual and text information.

[0135] Figure 3 A structural schematic diagram of a multi-modal picture understanding device based on cross-modal label fusion provided by the embodiments of the present disclosure is shown in FIG. 3, which includes a first processing module 3001, a second processing module 3002, a third processing module 3003, and a fourth processing module 3004. Wherein: Figure 3

[0136] The first processing module 3001 is configured to obtain an image and text included in a multi-modal picture.

[0137] The second processing module 3002 is configured to input the image into an image encoder to obtain at least one image feature, determine a first visual token based on the at least one image feature, and use a text segmenter to obtain a first text token included in the text.

[0138] ​The third processing module 3003 is configured to obtain a second visual token based on the first visual token and the first text token; the second visual token comprises a weighted combination visual token component and an attention visual token component; the weighted combination visual token component is determined by weighted summation of the second text token in the text token set; and the attention visual token component is determined by cross-attention calculation of the first visual token and the first text token.

[0139] The fourth processing module 3004 is configured to input the second visual token and the first text token into a preset large language model after merging, and perform understanding on the multi-modal picture.

[0140] The multi-modal picture understanding device based on cross-modal label fusion provided by the embodiments of the present disclosure can perform the multi-modal picture understanding method based on cross-modal label fusion provided by the embodiments of the present disclosure, and the implementation principles are similar. The actions performed by each module in the multi-modal picture understanding device based on cross-modal label fusion provided by the embodiments of the present disclosure correspond to the steps in the multi-modal picture understanding method based on cross-modal label fusion provided by the embodiments of the present disclosure. For the detailed function description of each module of the multi-modal picture understanding device based on cross-modal label fusion provided by the embodiments of the present disclosure, refer to the description in the corresponding method shown in the foregoing, which will not be repeated here.

[0141] The embodiments of the present disclosure use weighted combination of the text token set to represent the visual token, so as to ensure that the data distribution of the visual token is consistent with the data distribution of the text token. Meanwhile, the cross-attention weight of the text token and the visual token is also used to fuse the multi-modal label, which can effectively fuse the semantic expression of various modal labels, and effectively improve the understanding of the multi-modal picture.

[0142] Figure 4 A structural schematic diagram of an electronic device provided by the embodiments of the present disclosure is shown in Figure 4 The electronic device 4000 comprises a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, through a bus 4002. Optionally, the electronic device 4000 can further comprise a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual application, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present disclosure.

[0143] The processor 4001 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic device, transistor logic device, hardware component, or any combination thereof. It can implement or execute the various exemplary logical blocks, modules and circuits described in connection with the present disclosure. The processor 4001 can also be a combination of implementing computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0144] The bus 4002 can include a path that transmits information between the above-mentioned components. The bus 4002 can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, or the like. The bus 4002 can be divided into an address bus, a data bus, a control bus, and the like. For convenience of representation, Figure 4 In the figure, only one thick line is used, but it does not mean that there is only one bus or only one type of bus.

[0145] The memory 4003 can be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device that can store information and instructions, an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, an optical disk storage (including a compact disk, a laser disk, an optical disk, a digital versatile disk, a Blu-ray disk, and the like), a magnetic disk storage medium, other magnetic storage device, or any other medium capable of carrying or storing computer programs and capable of being read by a computer, without limitation.

[0146] The memory 4003 is configured to store a computer program for implementing the embodiments of the present disclosure, and the processor 4001 is configured to control the execution of the computer program stored in the memory 4003. The processor 4001 is configured to execute the computer program stored in the memory 4003 to implement the steps shown in the foregoing method embodiments.

[0147] The electronic device can include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (Personal Digital Assistant), a PAD (Tablet Personal Computer), a PMP (Portable Multimedia Player), a car terminal (for example, a car navigation terminal), and the like, and a stationary terminal such as a digital TV, a desktop computer, and the like. Figure 4 The electronic device shown is merely an example and should not impose any limitation on the functions and use range of the embodiments of the present disclosure.

[0148] The embodiments of the present disclosure provide a computer readable storage medium having stored thereon a computer program, which, when executed by a processor, can implement the steps and corresponding contents of the foregoing method embodiments.

[0149] It should be noted that the computer readable medium of the present disclosure described above can be a computer readable signal medium or a computer readable medium or any combination of the two. The computer readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more conductive wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or component. In the present disclosure, the computer readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer readable program code. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium other than the computer readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or component. The program code contained in the computer readable medium can be transmitted by any suitable medium, including but not limited to an electrical wire, an optical cable, an RF (radio frequency) or the like, or any suitable combination of the above.

[0150] The embodiments of the present disclosure further provide a computer program product comprising a computer program, which, when executed by a processor, can implement the steps and corresponding contents of the foregoing method embodiments. Compared with the prior art, the following can be achieved:

[0151] The terms "first", "second", "third", "fourth", "1", "2", and the like (if any) in the description, claims, and drawings of the present disclosure, and the above-described drawings are used to distinguish similar objects, and do not have to be used to describe a particular order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described.

[0152] It should be understood that, although the flowcharts of the embodiments of the present disclosure indicate various operation steps by arrows, the implementation order of these steps is not limited to the order indicated by the arrows. Unless otherwise specified herein, in some implementation scenarios of the embodiments of the present disclosure, the implementation steps in each flowchart can be executed in other orders as required. In addition, part or all of the steps in each flowchart can include multiple sub-steps or multiple stages based on the actual implementation scenario. Part or all of these sub-steps or stages can be executed at the same time, and each of these sub-steps or stages can also be executed at different times. In the scenario where the execution times are different, the execution order of these sub-steps or stages can be flexibly configured according to requirements, and the embodiments of the present disclosure do not limit this.

[0153] The above is only an optional implementation of some implementation scenarios of the present disclosure, and it should be pointed out that, for ordinary skilled persons in the technical field, other similar implementation means based on the technical concept of the present disclosure can also be adopted without departing from the technical concept of the present disclosure, and such implementation also falls within the protection scope of the embodiments of the present disclosure.

Claims

1. A multi-modal picture understanding method based on cross-modal label fusion, characterized in that, The method comprises: obtaining an image and text included in a multi-modal picture; inputting the image into an image encoder to obtain at least one image feature, determining a first visual token based on the at least one image feature, and obtaining a first text token included in the text using a text segmenter; obtaining a second visual token based on the first visual token and the first text token; wherein the second visual token comprises a weighted combination visual token component and an attention visual token component; the weighted combination visual token component is determined by weighted summation of second text tokens in a text token set, the second text token being a text token most closely associated with the first visual token selected from the text token set; and the attention visual token component is determined by cross-attention calculation of the first visual token and the first text token; merging the second visual token and the first text token and inputting them into a preset large language model to understand the multi-modal picture; the calculation formula of the weighted combination visual token component is as follows: wherein, is a weight of the second text token; is a set of text tokens; is a normalization exponential function for normalizing the component of the second text token, outputting a corresponding probability value as the weight of the second text token; is a second preset matrix for converting the visual token into a weight matrix of the set of text tokens; is a weight matrix responsible for aligning the dimension of the first visual token and the dimension of the text token in the set of text tokens; is a third visual token; V is a size of the set of text tokens; and D is a dimension of the set of text tokens.

2. The multi-modal picture understanding method based on cross-modal label fusion according to claim 1, characterized in that, determining the first visual token based on the at least one image feature comprises: merging the at least one image feature to obtain a third visual token; determining the dimension of a text token in the text token set; linearly mapping the dimension of the third visual token to the dimension of the text token in the text token set based on a first preset matrix to obtain the first visual token, the dimension of the first visual token being the same as the dimension of the text token.

3. The multi-modal picture understanding method based on cross-modal label fusion according to claim 1 or 2, characterized in that, the weighted combination visual token component being determined by weighted summation of second text tokens in a text token set comprises: determining a text token set, the text token set comprising at least one text token, each text token in the text token set being obtained by segmenting a target text; linearly mapping the first visual token to determine the second text token and the weight of the second text token; based on the second text token and the weight of the second text token, performing weighted summation on the second text token to determine the weighted combination visual token component.

4. The multi-modal picture understanding method based on cross-modal label fusion according to claim 3, characterized in that, the linear mapping of the first visual token to determine the weight of the second text token comprises: based on the first visual token and a second preset matrix, determining the component of the first visual token on the second text token; normalizing the component of the second text token to determine the weight of the second text token.

5. The multi-modal picture understanding method based on cross-modal label fusion according to claim 1 or 2, characterized in that, The attention visual token component is determined by cross-attention calculation based on the first visual token and the first text token, and includes: Based on the first visual token and the first text token, a first cross-attention weight matrix of the first visual token and the first text token is determined; Based on the first cross-attention weight matrix and the set of text tokens, the attention visual token component is determined.

6. The multi-modal picture understanding method based on cross-modal label fusion according to claim 5, characterized in that, Each text token in the set of text tokens includes a corresponding text token identifier; The determination of the attention visual token component based on the first cross-attention weight matrix and the set of text tokens includes: Based on the dimension of the text token in the set of text tokens, the dimension of each first visual token in the cross-attention weight matrix is expanded to the same dimension as the text token in the set of text tokens, to obtain a second cross-attention weight matrix; Based on the size relationship of the text token identifier in the set of text tokens, the order of the text token associated with each first visual token in the cross-attention weight matrix is adjusted to obtain a third cross-attention weight matrix; Based on the third cross-attention weight matrix and the set of text tokens, the attention visual component is determined.

7. The multi-modal picture understanding method based on cross-modal label fusion according to claim 6, characterized in that, The determination of the attention visual component based on the third cross-attention weight matrix and the set of text tokens includes: The third cross-attention weight matrix is multiplied by the set of text tokens to obtain the attention visual component.

8. A multi-modal picture understanding apparatus based on cross-modal label fusion, characterized in that, The device includes: A first processing module for obtaining an image and text included in a multi-modal picture; A second processing module for inputting the image into an image encoder to obtain at least one image feature, determining a first visual token based on the at least one image feature, and using a text tokenizer to obtain a first text token included in the text; A third processing module for obtaining a second visual token based on the first visual token and the first text token; wherein the second visual token includes a weighted combination visual token component and an attention visual token component; the weighted combination visual token component is determined by weighted summation of second text tokens in a set of text tokens, the second text tokens being the text tokens most closely associated with the first visual token selected from the set of text tokens; the attention visual token component is determined by cross-attention calculation based on the first visual token and the first text token; A fourth processing module for inputting the second visual token and the first text token into a pre-set large language model after merging to understand the multi-modal picture; The calculation formula of the weighted combination visual token component is as follows: wherein, is a weight of the second text token; is a set of text tokens; is a normalization exponential function for normalizing the component of the second text token, outputting a corresponding probability value as the weight of the second text token; is a second preset matrix for converting the visual token into a weight matrix of the set of text tokens; is a weight matrix responsible for aligning the dimension of the first visual token and the dimension of the text token in the set of text tokens; is a third visual token; V is a size of the set of text tokens; and D is a dimension of the set of text tokens.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory, wherein the computer program, when executed by the processor, is arranged to perform the method of any one of claims 1 to 8. The processor executes the computer program to implement the method of any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the method of any one of claims 1 to 7.

Images