Multi-modal picture understanding method and device based on discrete cosine transform

Abstract

The embodiment of the invention discloses a multi-modal picture understanding method and device based on discrete cosine transform, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining an image and a text included in a multi-modal picture; performing discrete cosine transform on the image to obtain a frequency domain feature vector, and converting the frequency domain feature vector into a first visual token; obtaining a text token included in the text by using a text word segmentation device; inputting the text token and the first visual tokens into a Q-former module, and compressing the number of the first visual tokens to obtain a second visual token; and merging the second visual token and the text token, and inputting the merged second visual token and text token into a preset large language model to obtain text description information of the multi-modal picture. According to the embodiment of the invention, the fine-grained sensing capability of the high-resolution picture is improved, the number of visual tokens is reduced, and the computing resources of a large language model are saved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and more specifically, to a method and apparatus for multimodal image understanding based on discrete cosine transform. Background Technology

[0002] Multimodal Large Language Model (MLLM) image understanding technology has become one of the hot research areas in artificial intelligence in recent years. It integrates data from multiple modalities (such as images and text) to comprehensively understand and process data, simulate the human ability to perceive multiple senses, and achieve deep understanding and semantic parsing of image content.

[0003] Existing multimodal image understanding technologies employ visual encoders to transform raw pixel information into feature vectors that can be understood by cross-modal models, thereby supporting key capabilities such as object recognition, scene parsing, and relation reasoning. However, visual encoders lack the ability to perceive the fine granularity of high-resolution multimodal images; furthermore, visual encoders require segmentation processing of multimodal images, leading to semantic discontinuities. Summary of the Invention

[0004] This disclosure provides a method and apparatus for multimodal image understanding based on discrete cosine transform, which can solve the technical problems in the prior art of insufficient fine-grained perception of high-resolution multimodal images by visual encoders, and semantic discontinuity after multimodal image segmentation. The technical solution provided by this disclosure is as follows: According to a first aspect of the present disclosure, a multimodal image understanding method based on discrete cosine transform is provided, the method comprising: Obtain multimodal images, including both images and text; The image is subjected to discrete cosine transform to obtain a frequency domain feature vector, and the frequency domain feature vector is converted into a first visual token; and a text token is obtained by using a text segmenter. The text token and the first visual token are input into the Q-former module, and the number of the first visual tokens is compressed to obtain the second visual token; The second visual token and the text token are merged and input into a preset large language model to obtain the text description information of the multimodal image.

[0005] As an optional implementation, performing a discrete cosine transform on the image to obtain a frequency domain feature vector includes: The RGB components of each pixel in the image are linearly transformed to obtain an image in YCbCr format; the YCbCr format image includes Y channel pixels, Cb channel pixels, and Cr channel pixels; The pixels of each channel are divided according to the preset pixel matrix to obtain the pixel blocks of each channel; Perform a discrete cosine transform on the pixel blocks of each channel to obtain the discrete cosine transform coefficient matrix corresponding to the pixel blocks in each channel; The frequency domain feature vector is obtained based on the discrete cosine transform coefficient matrix corresponding to the pixel block in each channel.

[0006] As an optional implementation, the step of dividing the pixels of each channel based on a preset pixel matrix to obtain pixel blocks for each channel includes: Based on an 8×8 resolution, the pixels of the Y channel are divided to obtain pixel blocks of the Y channel; Based on a resolution of 16×16, the Cb channel pixels and the Cr channel pixels are divided into Cb channel pixel blocks and Cr channel pixel blocks respectively.

[0007] As an optional implementation, obtaining the frequency domain feature vector based on the discrete cosine transform coefficient matrix corresponding to the pixel block in each channel includes: Based on the size of the discrete cosine transform coefficient matrix corresponding to the Y channel pixel block, the discrete cosine transform coefficient matrix corresponding to the Cb channel pixel block and the discrete cosine transform coefficient matrix corresponding to the Cr channel pixel block are upsampled. By merging the discrete cosine transform coefficient matrix corresponding to the pixel blocks in the Y channel, the discrete cosine transform coefficient matrix corresponding to the pixel blocks in the upsampled Cb channel, and the discrete cosine transform coefficient matrix corresponding to the pixel blocks in the upsampled Cr channel, the frequency domain feature vector corresponding to the multimodal image is obtained.

[0008] As an optional implementation, converting the frequency domain feature vector into a first visual token includes: Based on the dimension of the text token, a one-dimensional convolution operation is performed on the frequency domain feature vector to obtain the first visual token; The first visual token has the same dimension as the text token.

[0009] As an optional implementation, the step of inputting the text token and the first visual token into the Q-former module, compressing the number of the first visual tokens to obtain the second visual token includes: Determine the self-attention weight between the preset query token and the text token; Based on the self-attention weight and the preset query token, the preset query token is updated to obtain the updated query token; Calculate the cross-attention weight between the updated query token and the first visual token; Based on the cross-attention weights, cross-attention calculation is performed on the first visual token to obtain the third visual token; The third visual token is input into the feedforward neural network to obtain the second visual token.

[0010] As an optional implementation, the feedforward neural network includes two layers of linear transformation and a ReLU activation function; the calculation formula for the second visual token is as follows:

[0011] in, Z1 represents the output of the feedforward neural network when the input is Z1, i.e., the second visual token; Z1 is the input of the feedforward neural network, i.e., the third visual token; W2 is the weight matrix of the second linear transformation; max(0, ˙) is the ReLU activation function; W1 is the weight matrix of the first linear transformation; a is the bias vector of the first linear transformation; b is the bias vector of the second linear transformation.

[0012] According to a third aspect of the present disclosure, a multimodal image understanding device based on discrete cosine transform is provided, the device comprising: The first processing module is used to obtain the images and text included in the multimodal image; The second processing module is used to perform discrete cosine transform on the image to obtain a frequency domain feature vector, and convert the frequency domain feature vector into a first visual token; and to use a text segmenter to obtain the text token included in the text. The third processing module is used to input the text token and the first visual token into the Q-former module, compress the number of the first visual token, and obtain the second visual token. The fourth processing module is used to fuse the second visual token and the text token and input them into a preset large language model to obtain the text description information of the multimodal image.

[0013] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in any of the first aspects.

[0014] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps of the method as described in any of the first aspects.

[0015] The beneficial effects of the technical solutions provided in this disclosure are: This disclosure provides a multimodal image understanding method and apparatus based on discrete cosine transform. This disclosure employs discrete cosine transform to convert image processing from the spatial domain to the frequency domain, avoiding the limitations of image allocation rate and the technical problem of semantic discontinuity caused by image segmentation in visual encoders.

[0016] Furthermore, this embodiment of the present disclosure uses Q-Former to compress the visual tokens, which reduces the number of visual tokens, saves computing resources, and improves the processing speed of large language models. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments of this disclosure will be briefly introduced below.

[0018] Figure 1 A schematic diagram of the structure of a multimodal image understanding model based on discrete cosine transform provided in this disclosure embodiment; Figure 2 A flowchart illustrating a multimodal image understanding method based on discrete cosine transform provided in this disclosure embodiment; Figure 3 A schematic diagram of the processing flow of a discrete cosine transform module provided in an embodiment of this disclosure; Figure 4 A schematic diagram of the processing flow of a frequency domain adapter provided in an embodiment of this disclosure; Figure 5 A schematic diagram of the processing flow of a Q-former module provided in an embodiment of this disclosure; Figure 6 A schematic diagram of the structure of a multimodal image understanding device based on discrete cosine transform provided in this disclosure embodiment; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0019] The embodiments of this disclosure are described below with reference to the accompanying drawings. It should be understood that the embodiments described below with reference to the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of this disclosure, and do not constitute a limitation on the technical solutions of the embodiments of this disclosure.

[0020] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the terms “comprising” and “including” as used in embodiments of this disclosure mean that the corresponding feature can be implemented as the presented feature, information, data, step, operation, element, and / or component, but do not exclude implementation as other features, information, data, step, operation, element, component, and / or combinations thereof supported by the art. It should be understood that when we say that an element is “connected” or “coupled” to another element, the one element can be directly connected or coupled to the other element, or it can mean that the one element and the other element are connected through an intermediate element. Furthermore, “connected” or “coupled” as used herein can include wireless connection or wireless coupling. The term “and / or” as used herein indicates at least one of the items defined by the term, for example, “A and / or B” or “A, B” indicates implementation as “A,” or implementation as “B,” or implementation as “A and B.”

[0021] To make the objectives, technical solutions, and advantages of this disclosure clearer, the embodiments of this disclosure will be described in further detail below with reference to the accompanying drawings.

[0022] First, the technical terms used in this disclosure will be introduced and explained: YCbCr is a color space designed for video and image transmission and storage, widely used in digital imaging, video encoding, and display systems. YCbCr consists of three channels, each playing a different visual role: The Y channel (Luminance) represents the brightness information (i.e., grayscale value) of an image, reflecting the brightness of the picture. Only the Y component is retained to display a black and white image.

[0023] The Cb channel (Chrominance Blue) represents the difference between the blue component and the brightness, used to describe the relative intensity of blue in an image.

[0024] The Cr channel (Chrominance Red) represents the difference between the red component and the brightness, and is used to describe the relative intensity of red in an image.

[0025] Visual tokens and text tokens are both basic units used in multimodal models to represent different modal information; the former represents visual information, and the latter represents text information. The following is a detailed introduction: A visual token is a discrete representation of visual content (such as images and videos). Mainstream models typically use a visual encoder to convert visual inputs such as images into visual tokens, and then embed them into the context of a large language model to achieve visual information understanding.

[0026] A text token is a discrete representation of text information. Text tokens usually have a fixed vocabulary, and each token corresponds to an element in the vocabulary. Multimodal models capture the semantic and syntactic information of the text by learning the relationships between these text tokens.

[0027] The multimodal image understanding method and apparatus based on discrete cosine transform disclosed herein are intended to solve at least one of the above-mentioned technical problems in the prior art.

[0028] The following description of several exemplary embodiments illustrates the technical solutions of this disclosure and the technical effects produced by these solutions. It should be noted that the following embodiments can be referenced, learned from, or combined with each other. Identical terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0029] This disclosure provides a multimodal image understanding model based on discrete cosine transform (DCT) to implement the multimodal image understanding method based on DCT provided in this disclosure. Figure 1 As shown, the multimodal image understanding model based on discrete cosine transform includes: a discrete cosine transform module, a frequency domain adapter, a Q-former module, a text segmenter, and a large language model.

[0030] Specifically, in this embodiment of the present disclosure, the discrete cosine transform module converts the image from the spatial domain to the frequency domain through discrete cosine transform to obtain a frequency domain feature vector.

[0031] Specifically, in this embodiment of the disclosure, the frequency domain adapter is used to convert frequency domain feature vectors into visual tokens.

[0032] Specifically, in this embodiment of the disclosure, a text segmenter is used to segment text to obtain text tokens.

[0033] Specifically, in this embodiment of the disclosure, the Q-former module is used to compress visual tokens. The output visual tokens have sufficient visual features, but the number is greatly reduced, thus saving the computing resources of large language models.

[0034] Specifically, in this embodiment of the disclosure, the large language model understands the images and text in the multimodal image based on the visual token output by the Q-former module and the text token output by the text segmenter, and outputs the text description information of the multimodal image.

[0035] Figure 2 This is a flowchart illustrating a multimodal image understanding method based on discrete cosine transform, provided in an embodiment of this disclosure; as shown below. Figure 2 As shown, the method includes: S201. Obtain the images and text included in the multimodal image.

[0036] Specifically, in this embodiment of the disclosure, the image is the core carrier of the multimodal image. The goal of acquiring the image is to extract the original visual data (such as pixels, colors, and structures) to provide a basis for subsequent processing. For example, an image loading tool can be used to read the image file and obtain its original pixel matrix, which contains pixel values ​​in RGB or grayscale mode, as well as data such as the image size and resolution.

[0037] Specifically, in this embodiment of the disclosure, for text that is directly present in a multimodal image, OCR (Optical Character Recognition) technology can be used to recognize the text in the image. OCR technology will convert the text pixels in the multimodal image into an editable text string.

[0038] S202, Perform discrete cosine transform on the image to obtain the frequency domain feature vector, and convert the frequency domain feature vector into the first visual token; and use a text segmenter to obtain the text token included in the text.

[0039] Specifically, in this embodiment, the matrix composed of pixels belongs to the "spatial domain" information, which directly reflects the visual content through the brightness and color of each pixel, but the data volume is large and contains redundant information. Discrete cosine transform converts the image from the "spatial domain" to the "frequency domain", decomposing the pixel information of the image into "signals" of different frequencies. For example, the low-frequency part determines the overall outline, and the high-frequency part determines the texture details.

[0040] Specifically, in this embodiment of the disclosure, "token" can be understood as the smallest semantic symbol. Converting the frequency domain feature vector into the first visual token is essentially a further transformation of the abstract digital vector into a "visual symbol" that is more suitable for model processing.

[0041] Specifically, in this embodiment of the disclosure, natural language (such as Chinese and English) consists of continuous characters or words, but directly processing the entire text is very difficult. The function of the text segmenter is to break down continuous text into the smallest meaningful units according to language rules, and each smallest unit is a text token.

[0042] S203. Input the text token and the first visual token into the Q-former module, compress the number of the first visual token, and obtain the second visual token.

[0043] Specifically, in this embodiment of the disclosure, the Q-former module is a module specifically designed for handling "cross-modal information interaction". The Q-former module compresses the number of first visual tokens. This compression is not simply a reduction in quantity, but rather a selection of the most critical visual features based on the text tokens, effectively reducing the number of visual tokens output to the large language model while retaining the most crucial visual features.

[0044] S204. Merge the second visual token and the text token and input them into the preset large language model to obtain the text description information of the multimodal image.

[0045] Specifically, in this embodiment of the disclosure, the merging of the second visual token and the text token is not a simple concatenation, but rather the integration of two different types of tokens into a unified input format that can be recognized by a large language model. By merging these two types of tokens, the core features observed in the image and the semantic information mentioned in the text are integrated together, allowing the large language model to simultaneously acquire both "visual cues" and "linguistic cues".

[0046] Specifically, in this embodiment of the disclosure, the large language model is a model that can understand and generate natural language. Based on the merged visual token and text token, the large language model finds the intrinsic connection between visual features and text semantics. At the same time, the large language model uses its own language generation capabilities to transform this connection into coherent natural language. The final generated text description information is a comprehensive result combining image visual features and text information.

[0047] In this embodiment, the image is transformed from the spatial domain to the frequency domain using discrete cosine transform, which is not limited by the image allocation rate and avoids the technical problem of semantic discontinuity caused by image segmentation in visual encoders. Simultaneously, the Q-Former module compresses visual tokens, reducing the number of visual tokens and saving computational resources for large language models.

[0048] Based on the above embodiments, as an optional embodiment, a discrete cosine transform is performed on the image to obtain a frequency domain feature vector, including: The RGB components of each pixel in the image are linearly transformed to obtain an image in YCbCr format; the YCbCr format image includes Y channel pixels, Cb channel pixels, and Cr channel pixels; The pixels of each channel are divided according to the preset pixel matrix to obtain the pixel blocks of each channel; Perform a discrete cosine transform on the pixel blocks of each channel to obtain the discrete cosine transform coefficient matrix corresponding to the pixel blocks in each channel; The frequency domain feature vector is obtained based on the discrete cosine transform coefficient matrix corresponding to the pixel block in each channel.

[0049] Specifically, in this embodiment, each pixel of the image is represented using "RGB format." However, the three components of RGB are highly correlated, making it difficult to extract brightness or color features individually. YCbCr format is a color format more suitable for image processing. It splits the RGB color components into three independent channels, which are less correlated with each other, facilitating separate processing. In this embodiment, the R, G, and B values ​​of each pixel are converted into Y, Cb, and Cr values ​​using a preset linear formula, ultimately obtaining an image in YCbCr format. At this point, the pixels in the image are split into three independent channel pixel matrices: the Y channel pixel matrix, the Cb channel pixel matrix, and the Cr channel pixel matrix.

[0050] Specifically, in this embodiment of the disclosure, the pixel matrices of the three channels Y, Cb, and Cr are segmented respectively, and each channel ultimately yields a series of independent pixel blocks. This is the core transformation step from the "spatial domain" to the "frequency domain," with the aim of extracting the frequency domain features of the pixel blocks.

[0051] Specifically, in this embodiment of the disclosure, the pixel block of each channel still belongs to the "spatial domain" information. By performing a discrete cosine transform on this pixel block, it is converted into a discrete cosine transform coefficient matrix in the "frequency domain". In this way, the visual features of the pixel block are transformed from specific pixel values ​​into signal intensities of different frequencies.

[0052] Specifically, in this embodiment of the disclosure, the discrete cosine transform coefficient matrix of each channel pixel block is organized according to a preset rule to finally form a "frequency domain feature vector". For example, the discrete cosine transform coefficients of all pixel blocks in the Y channel are collected first, and then the discrete cosine transform coefficients of the Cb and Cr channels are concatenated in sequence to form a frequency domain feature vector. This frequency domain feature vector condenses the frequency domain features of the image in brightness (Y) and color (Cb, Cr).

[0053] In this embodiment, after converting the image to YCbCr format, the luminance information (Y channel) and chrominance information (Cb and Cr channels) in the image are separated. After extracting frequency domain features from the Y, Cb, and Cr channels respectively, the feature vectors of the three channels can independently reflect the luminance-related frequency domain information (Y channel) and the color-related frequency domain information (Cb and Cr channels). This separation allows the frequency domain feature vectors to retain both the luminance-dominated structural information and the color-related detailed information. Subsequent tasks can flexibly utilize the features of different channels according to requirements, improving the targeting of frequency domain feature usage.

[0054] Based on the above embodiments, as an optional embodiment, the pixels of each channel are divided according to a preset pixel matrix to obtain pixel blocks for each channel, including: Based on an 8×8 resolution, the pixels of the Y channel are divided to obtain pixel blocks of the Y channel; Based on a resolution of 16×16, the pixels of the Cb channel and the Cr channel are divided into pixel blocks for the Cb channel and pixel blocks for the Cr channel.

[0055] like Figure 3 As shown, assume the image resolution is (H, W), where H is the number of pixels in height and W is the number of pixels in width. For the Y channel, divided into 8×8 blocks, the height H can be divided into H / 8 blocks (e.g., if H=256, then 256 / 8=32 blocks), and the width W can be divided into W / 8 blocks. Therefore, in terms of the "number of blocks," a structure of (H / 8, W / 8) is formed. Simultaneously, each 8×8 discrete cosine transform coefficient matrix has 8×8=64 coefficients, so the "coefficient dimension" C=64, meaning each pixel block corresponds to 64 frequency coefficients. This can be understood as having H / 8 blocks in the height direction and W / 8 blocks in the width direction, with each pixel block containing 64 frequency coefficients, each frequency coefficient corresponding to a different frequency of brightness feature. For the Cb and Cr channels, divided into 16×16 blocks, the original height H can be divided into H / 16 blocks, and the width W can be divided into W / 16 blocks, thus forming a block count structure of (H / 16, W / 16). Each pixel block has a discrete cosine transform coefficient matrix with 16×16=256 coefficients, but the 64 most critical coefficients are selected and uniformly set to C=64. This can be understood as follows: for the Cb and Cr channels, there are H / 16 blocks in the height direction and W / 16 blocks in the width direction. Each block contains 64 frequency coefficients, and each frequency coefficient corresponds to a different frequency of color feature.

[0056] Specifically, in this embodiment, the Y channel carries brightness information and has the greatest impact on visual effects, requiring the preservation of more details; therefore, using smaller pixel blocks can capture local changes in brightness more precisely. Cb and Cr are color channels, and the human eye is not sensitive to their details, such as subtle color changes that are not easily perceived; therefore, larger pixel blocks can both preserve the main color features and reduce the amount of data, improving subsequent processing efficiency.

[0057] In this embodiment of the disclosure, after dividing the pixel blocks according to a preset matrix, the discrete cosine transform can focus on the feature extraction of local areas. This division avoids information confusion during global processing, allowing the local features such as texture and brightness changes of each pixel block to be captured independently, and the subsequent feature vector can more accurately reflect the local visual pattern of the image.

[0058] Based on the above embodiments, as an optional embodiment, a frequency domain feature vector is obtained based on the discrete cosine transform coefficient matrix corresponding to the pixel block in each channel, including: Based on the size of the discrete cosine transform coefficient matrix corresponding to the Y channel pixel block, the discrete cosine transform coefficient matrix corresponding to the Cb channel pixel block and the discrete cosine transform coefficient matrix corresponding to the Cr channel pixel block are upsampled. By merging the discrete cosine transform coefficient matrix corresponding to the pixel blocks in the Y channel, the discrete cosine transform coefficient matrix corresponding to the pixel blocks in the upsampled Cb channel, and the discrete cosine transform coefficient matrix corresponding to the pixel blocks in the upsampled Cr channel, the frequency domain feature vector corresponding to the multimodal image is obtained.

[0059] Specifically, in this embodiment of the disclosure, the Y channel undergoes discrete cosine transform on 8×8 pixel blocks, resulting in a relatively large coefficient matrix. The Cb and Cr channels undergo discrete cosine transform on 16×16 pixel blocks, resulting in relatively small coefficient matrices. This size difference means that the coefficient matrices of the three channels cannot be directly concatenated.

[0060] Specifically, in this embodiment of the disclosure, the function of upsampling is to adjust the coefficient matrices of the Cb and Cr channels to the same size as the coefficient matrix of the Y channel, ensuring that the features of the three channels are aligned in spatial distribution. For example, the discrete cosine transform coefficient matrices of the Cb and Cr channels can be "enlarged" using preset interpolation or mapping rules, making them completely consistent with the Y channel in terms of the number of blocks and matrix dimensions.

[0061] Specifically, in this embodiment, when the discrete cosine transform coefficient matrices of the Y channel, the upsampled Cb channel, and the Cr channel are of completely identical size, a "stitching" operation can be performed. For example, first, all discrete cosine transform coefficients of the Y channel are taken, and then all discrete cosine transform coefficients of the upsampled Cb channel and Cr channel are stitched together sequentially; this can be understood as stitching the discrete cosine transform coefficients of the three matrices together to obtain a frequency domain feature vector. The frequency domain feature vector simultaneously contains the brightness frequency domain feature of the Y channel, the blue difference frequency domain feature of the Cb channel, and the red difference frequency domain feature of the Cr channel, and the frequency domain feature vectors of the three channels are completely aligned in spatial dimensions.

[0062] In this embodiment of the disclosure, upsampling can stretch the discrete cosine coefficient matrices of the Cb and Cr channels to the same size as the discrete cosine coefficient matrix of the Y channel; it eliminates the differences in the spatial dimensions of the features of different channels, avoids feature misalignment caused by size mismatch during subsequent merging, and ensures that the frequency domain features of luminance and chrominance are integrated in the same dimension.

[0063] Based on the above embodiments, as an optional embodiment, converting the frequency domain feature vector into a first visual token includes: Based on the dimension of the text token, a one-dimensional convolution operation is performed on the frequency domain feature vector to obtain the first visual token; The dimensions of the first visual token are the same as those of the text token.

[0064] like Figure 4 As shown, assuming the image resolution is (H, W), the discrete cosine transform (DCT) coefficient matrix of the Y channel is (H / 8, W / 8, 64); and the DCT coefficient matrix of the Cb channel is (H / 16, W / 16, 64). Upsampling expands the DCT coefficient matrices of the Cb and Cr channels to the same size as the DCT coefficient matrix of the Y channel. For the frequency domain feature vector, H / 8 and W / 8 represent the dimensions of the frequency domain feature vector in the "height" and "width" directions, respectively. These two values ​​correspond to the number of spatial arrangements of the image after the discrete cosine transform and subsequent upsampling. 3C represents the dimension of the frequency domain feature vector; where the "3" in 3C corresponds to the three channels of the image (Y, Cb, Cr), meaning that the total dimension after merging the feature dimensions of the three channels is 3C.

[0065] Specifically, in this embodiment, the frequency domain feature vector is a string of numbers extracted from the image, containing frequency domain information of luminance (Y) and color (Cb, Cr), and its "dimensionality" is determined by image features (such as the 3C mentioned above). The text token, on the other hand, is a semantic symbol extracted from text, and each token is converted into a vector of fixed dimensions. Therefore, the dimensions of the frequency domain feature vector and the text token may differ, and this inconsistency in dimensions will prevent them from being associated and processed by the Q-former. Therefore, it is necessary to convert the frequency domain feature vector into a visual token with the same dimensions as the text token.

[0066] Specifically, in this embodiment of the disclosure, a weighted calculation is performed on adjacent numbers in the frequency domain feature vector using a preset convolution kernel. This not only preserves the core features of the original vector but also converts the vector dimension to the text token dimension by adjusting the number and size of the convolution kernel. For example, if the dimension of the frequency domain feature vector is 192 and the dimension of the text token is 768, the convolution kernel parameters can be designed to change the length of the output vector after the convolution operation from 192 to 768. During this process, the frequency domain feature vector is converted into a new vector, namely, the first visual token, and the dimension of the first visual token is exactly the same as the dimension of the text token.

[0067] In this embodiment, the dimensions of the text token and the visual token may not be the same. By performing a one-dimensional convolution operation, the visual token is mapped to the same dimension as the text token, ensuring that the two can be directly associated in the Q-former module, thus laying the foundation for understanding subsequent multimodal images.

[0068] Based on the above embodiments, as an optional embodiment, the text token and the first visual token are input into the Q-former module, and the number of the first visual tokens is compressed to obtain the second visual token, including: Determine the predefined self-attention weights between the query token and the text token; Based on the self-attention weight and the preset query token, the preset query token is updated to obtain the updated query token; Calculate the cross-attention weight between the updated query token and the first-view token; Based on the cross-attention weights, cross-attention calculation is performed on the first visual token to obtain the third visual token; The third visual token is input into the feedforward neural network to obtain the second visual token.

[0069] like Figure 5As shown, the Q-former module takes a visual token, a query token, and a text token as inputs. The visual token, derived from image features (such as the previously mentioned "first visual token"), carries frequency domain and semantic information and serves as the input to the visual modality. The query token is a set of learnable vectors used to extract related content from the visual and text tokens. The text token carries textual semantics and serves as the input to the text modality. The Q-former module includes a self-attention module, a cross-attention module, and a feedforward neural network. The self-attention module performs self-attention calculations on the query and text tokens to obtain textual context information. The cross-attention module, building upon self-attention, allows the query token to more accurately extract key frequency domain features from the visual token, compressing the number of video tokens. The visual token output from the cross-attention module and the textual context information output from the self-attention module are input into the feedforward neural network to output the final visual token, which is then used by the subsequent large language module.

[0070] Specifically, in this embodiment of the disclosure, the preset query token first learns the semantic information in the text token through self-attention calculation with the text token. For example, if the text token is "describes the red flower in the image", the self-attention weight will cause the query token to focus on the semantic features related to "red" and "flower". By learning the semantic information in the text token, the query token no longer simply filters features based on visuals, but captures relevant visual information in a targeted manner according to the text requirements.

[0071] Specifically, in this embodiment of the disclosure, the query token and text token self-attention weights are... The calculation formula is shown in formula (1):

[0072] in, For text tokens, , The length of the text token. For the text token dimension; To query the token, : To query the token length and , Text token dimension.

[0073] In the above formula (1), This means concatenating the text token and the preset query token into a feature matrix (for example, the text token is 100 vectors, the query token is 16 vectors, and the concatenation forms a 116×D matrix, where D is the feature dimension). This is the transpose of the matrix. Multiplying the two matrices yields an attention score matrix, where each element represents the association strength between a text token and a query token; this is then processed by an activation function. The attention score matrix is ​​transformed into a probability distribution to obtain the self-attention weights of the query token and the text token. The higher the weight, the stronger the association between the corresponding text token and the query token.

[0074] It should be noted that, in the embodiments disclosed herein, It is an activation function widely used in machine learning and deep learning to convert a set of arbitrary real numbers into numerical values ​​representing a probability distribution.

[0075] Specifically, in this embodiment of the disclosure, the preset query token is updated based on the self-attention weight and the preset query token to obtain the updated query token. The calculation formula for the updated query token is shown in formula (2):

[0076] in, For the updated text token, ; For the updated query token, .

[0077] Specifically, in this embodiment of the disclosure, when calculating the cross-attention between the updated query token and the first visual token, the cross-attention weight will be tilted towards "visual features related to text semantics". For example, if the first visual token contains information such as "flower outline", "red chromaticity frequency domain features", and "background green leaves", the cross-attention will prioritize focusing on "flower outline" and "red features" (related to "red flowers" in the text token), while weakening the weight of "background green leaves". Finally, the third visual token obtained through cross-attention calculation filters out redundant visual information unrelated to the text, thereby achieving visual feature filtering guided by text semantics.

[0078] Specifically, in this embodiment of the disclosure, the formula for calculating the cross-attention weight between the updated query token and the first visual token is as shown in formula (3):

[0079] in, The cross-attention weight between the updated query token and the first-view token. ; The updated query token; For first-view token, , The length of the first visual token. This is the first visual token dimension.

[0080] Specifically, in this embodiment of the disclosure, after obtaining the cross-attention weight between the updated query token and the first visual token, the first visual token is cross-attention calculated based on the cross-attention weight to obtain the third visual token; wherein, the calculation formula of the third visual token is as shown in formula (4):

[0081] in, For third-person vision token, .

[0082] Specifically, in this embodiment, the third visual token is a text-related visual feature selected from the first visual token through cross-attention. However, these features may still be scattered or insufficiently represented. A feedforward network (FFN) aggregates these scattered related features, strengthening their correlation. Simultaneously, the output dimension of the feedforward network remains consistent with the input dimension, but the distribution pattern of the features is closer to the needs of a large language model.

[0083] In this embodiment of the disclosure, the number of visual tokens is compressed through the Q-former module. This compression is not a simple reduction in quantity, but rather a selection of the most critical visual features based on text semantics, which effectively reduces the number of visual tokens input to the large language model and saves the computational resources of the large language model.

[0084] Based on the above embodiments, as an optional embodiment, the feedforward neural network includes two linear transformation layers and a ReLU activation function; the calculation formula for the second visual token is as follows:

[0085] Where FFN(Z1) represents the output of the feedforward neural network when the input is Z1, i.e., the second visual token; Z1 is the input of the feedforward neural network, i.e., the third visual token; W2 is the weight matrix of the second linear transformation; max(0,˙) is the ReLU activation function; W1 is the weight matrix of the first linear transformation; a is the bias vector of the first linear transformation; b is the bias vector of the second linear transformation.

[0086] Specifically, in this embodiment of the disclosure, the operation of the feedforward neural network consists of three key steps, including: The first layer of linear transformation: The input Z1 is linearly mapped using the weight matrix W1 and the bias vector a, i.e., "W1˙Z1+a" is calculated. This step adjusts the dimensionality distribution of the features and adjusts the feature baseline using the bias vector a, preparing for subsequent nonlinear processing.

[0087] The ReLU activation function calculates the output of the first-layer linear transformation, specifically max(0, W1˙Z1+a), retaining only positive features and setting negative features to 0. By introducing a non-linear transformation, it captures the complex relationships between features, avoiding the limitation of feature representation to simple linear relationships.

[0088] The second layer of linear transformation involves linearly mapping the output of the ReLU activation function again using the weight matrix W2 and the bias vector b, i.e., calculating "W2˙max(0, W1˙Z1+a)+b". By readjusting the features from the non-linear processing to the target dimension (consistent with the dimension of the second visual token) and further optimizing the feature distribution using the bias vector b, the output is ensured to meet the requirements of the downstream task.

[0089] In this embodiment of the disclosure, the feedforward neural network formula refines the third visual token into a second visual token that is more expressive and better suited to downstream tasks through two layers of linear transformation and ReLU activation function, providing a high-quality visual carrier for subsequent cross-modal semantic alignment.

[0090] Figure 6 A schematic diagram of a multimodal image understanding device based on discrete cosine transform provided in this disclosure embodiment is shown below. Figure 6 As shown, the device includes: a first processing module 6001, a second processing module 6002, a third processing module 6003, and a fourth processing module 6004. Wherein: The first processing module 6001 is used to obtain the images and text included in the multimodal image; The second processing module 6002 is used to perform discrete cosine transform on the image to obtain a frequency domain feature vector, and convert the frequency domain feature vector into a first visual token; and to use a text segmenter to obtain the text token included in the text. The third processing module 6003 is used to input the text token and the first visual token into the Q-former module, compress the number of the first visual token, and obtain the second visual token; The fourth processing module 6004 is used to fuse the second visual token and the text token and input them into a preset large language model to obtain the text description information of the multimodal image.

[0091] The multimodal image understanding device based on discrete cosine transform provided in this disclosure can execute the multimodal image understanding method based on discrete cosine transform provided in this disclosure. The implementation principles are similar. The actions performed by each module in the multimodal image understanding device based on discrete cosine transform provided in each embodiment of this disclosure correspond to the steps in the multimodal image understanding method based on discrete cosine transform provided in each embodiment of this disclosure. For detailed functional descriptions of each module in the multimodal image understanding device based on discrete cosine transform provided in this disclosure, please refer to the descriptions in the corresponding methods shown above, which will not be repeated here.

[0092] This embodiment employs Discrete Cosine Transform (DCT) to transform image processing from the spatial domain to the frequency domain, thus avoiding the limitations of image allocation rate and the technical problem of semantic discontinuity caused by image segmentation in visual encoders. Simultaneously, Q-Former is used to compress visual tokens, reducing the number of visual tokens and saving computational resources for large language models.

[0093] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure, such as... Figure 7 As shown, the electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of this disclosure.

[0094] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with this disclosure. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0095] Bus 4002 may include a pathway for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0096] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium capable of carrying or storing computer programs and capable of being read by a computer, without limitation herein.

[0097] The memory 4003 is used to store computer programs that execute embodiments of the present disclosure, and is controlled by the processor 4001 to execute them. The processor 4001 is used to execute the computer programs stored in the memory 4003 to implement the steps shown in the foregoing method embodiments.

[0098] The electronic device package may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (such as in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0099] This disclosure provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps and corresponding content of the aforementioned method embodiments.

[0100] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium, a computer-readable medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0101] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, can implement the steps and corresponding content of the aforementioned method embodiments. Compared with the prior art, it can achieve: The terms “first,” “second,” “third,” “fourth,” “1,” “2,” etc. (if present) in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in a sequence other than that shown in the figures or text.

[0102] It should be understood that although arrows indicate various operation steps in the flowcharts of the embodiments of this disclosure, the order in which these steps are implemented is not limited to the order indicated by the arrows. Unless explicitly stated herein, in some implementation scenarios of the embodiments of this disclosure, the implementation steps in each flowchart can be executed in other orders as required. Furthermore, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on the actual implementation scenario. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage can also be executed at different times. In scenarios where execution times differ, the execution order of these sub-steps or stages can be flexibly configured as required, and the embodiments of this disclosure do not limit this.

[0103] The above are merely optional implementation methods for some implementation scenarios of this disclosure. It should be noted that for those skilled in the art, other similar implementation methods based on the technical concept of this disclosure, without departing from the technical concept of this disclosure, also fall within the protection scope of the embodiments of this disclosure.

Claims

1. A multimodal image understanding method based on discrete cosine transform, characterized in that, The method includes: Obtain multimodal images, including both images and text; The image is subjected to discrete cosine transform to obtain a frequency domain feature vector, and the frequency domain feature vector is converted into a first visual token; and a text token is obtained by using a text segmenter. The text token and the first visual token are input into the Q-former module, and the number of the first visual tokens is compressed to obtain the second visual token; The second visual token and the text token are merged and input into a preset large language model to obtain the text description information of the multimodal image.

2. The multimodal image understanding method based on discrete cosine transform according to claim 1, characterized in that, The step of performing a discrete cosine transform on the image to obtain a frequency domain feature vector includes: The RGB components of each pixel in the image are linearly transformed to obtain an image in YCbCr format; the YCbCr format image includes Y channel pixels, Cb channel pixels, and Cr channel pixels; The pixels of each channel are divided according to the preset pixel matrix to obtain the pixel blocks of each channel; Perform a discrete cosine transform on the pixel blocks of each channel to obtain the discrete cosine transform coefficient matrix corresponding to the pixel blocks in each channel; The frequency domain feature vector is obtained based on the discrete cosine transform coefficient matrix corresponding to the pixel block in each channel.

3. The multimodal image understanding method based on discrete cosine transform according to claim 2, characterized in that, The process of dividing the pixels of each channel based on a preset pixel matrix to obtain pixel blocks for each channel includes: Based on an 8×8 resolution, the pixels of the Y channel are divided to obtain pixel blocks of the Y channel; Based on a resolution of 16×16, the Cb channel pixels and the Cr channel pixels are divided into Cb channel pixel blocks and Cr channel pixel blocks respectively.

4. The multimodal image understanding method based on discrete cosine transform according to claim 2 or 3, characterized in that, The process of obtaining the frequency domain feature vector based on the discrete cosine transform coefficient matrix corresponding to the pixel block in each channel includes: Based on the size of the discrete cosine transform coefficient matrix corresponding to the Y channel pixel block, the discrete cosine transform coefficient matrix corresponding to the Cb channel pixel block and the discrete cosine transform coefficient matrix corresponding to the Cr channel pixel block are upsampled. By merging the discrete cosine transform coefficient matrix corresponding to the pixel blocks in the Y channel, the discrete cosine transform coefficient matrix corresponding to the pixel blocks in the upsampled Cb channel, and the discrete cosine transform coefficient matrix corresponding to the pixel blocks in the upsampled Cr channel, the frequency domain feature vector corresponding to the multimodal image is obtained.

5. The multimodal image understanding method based on discrete cosine transform according to claim 4, characterized in that, The step of converting the frequency domain feature vector into a first visual token includes: Based on the dimension of the text token, a one-dimensional convolution operation is performed on the frequency domain feature vector to obtain the first visual token; The first visual token has the same dimension as the text token.

6. The multimodal image understanding method based on discrete cosine transform according to claim 5, characterized in that, The step of inputting the text token and the first visual token into the Q-former module, compressing the number of the first visual tokens to obtain the second visual token, includes: Determine the self-attention weight between the preset query token and the text token; Based on the self-attention weight and the preset query token, the preset query token is updated to obtain the updated query token; Calculate the cross-attention weight between the updated query token and the first visual token; Based on the cross-attention weights, cross-attention calculation is performed on the first visual token to obtain the third visual token; The third visual token is input into the feedforward neural network to obtain the second visual token.

7. The multimodal image understanding method based on discrete cosine transform according to claim 6, characterized in that, The feedforward neural network includes two linear transformation layers and a ReLU activation function; the calculation formula for the second visual token is as follows: in, Z1 represents the output of the feedforward neural network when the input is Z1, i.e., the second visual token; Z1 is the input of the feedforward neural network, i.e., the third visual token; W2 is the weight matrix of the second linear transformation; max(0, ˙) is the ReLU activation function; W1 is the weight matrix of the first linear transformation; a is the bias vector of the first linear transformation; b is the bias vector of the second linear transformation.

8. A multimodal image understanding device based on discrete cosine transform, characterized in that, The device includes: The first processing module is used to obtain the images and text included in the multimodal image; The second processing module is used to perform discrete cosine transform on the image to obtain a frequency domain feature vector, and convert the frequency domain feature vector into a first visual token; and to use a text segmenter to obtain the text token included in the text. The third processing module is used to input the text token and the first visual token into the Q-former module, compress the number of the first visual token, and obtain the second visual token. The fourth processing module is used to fuse the second visual token and the text token and input them into a preset large language model to obtain the text description information of the multimodal image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the method of any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images