Multi-modal data processing method and electronic equipment

Abstract

The invention provides a multi-modal data processing method and electronic equipment. The multi-modal data processing method comprises the steps of obtaining a to-be-processed image and task indication information; the task indication information represents that a target task is executed at least based on the to-be-processed image; segmenting the to-be-processed image into a plurality of image blocks, and determining a plurality of visual lexical elements corresponding to each image block and probability distribution corresponding to the plurality of visual lexical elements; the probability corresponding to the visual lexical element represents the probability that each image block belongs to the visual lexical element; generating a visual embedding feature corresponding to each image block based on the feature corresponding to each visual lexical element in the plurality of visual lexical elements and the probability distribution, and obtaining a plurality of visual embedding features of the plurality of image blocks; and processing the plurality of visual embedded features and the textual features of the task indication information by using a multi-modal model to generate an execution result of the target task.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of data processing, and particularly relates to a multi-modal data processing method and an electronic device. BACKGROUND

[0002] A multi-modal model can process data of multiple modalities including text, image, and audio, and generate natural language output. In some examples, the multi-modal model improves inference performance through discrete representation. In actual inference deployment, only the maximum value index of the probability distribution is used for the discrete representation, which limits the deep understanding and expression ability of the model for complex visual scenes, so that the processing performance of the multi-modal model is poor and the corresponding task execution result cannot be accurately generated. SUMMARY

[0003] The present disclosure provides a multi-modal data processing method and an electronic device.

[0004] According to an aspect of the present disclosure, a multi-modal data processing method is provided, including: obtaining an image to be processed and task indication information; the task indication information representing that a target task is to be executed based on at least the image to be processed; segmenting the image to be processed into a plurality of image blocks, determining a plurality of visual tokens corresponding to each image block, and a probability distribution corresponding to the plurality of visual tokens; the probability corresponding to the visual token representing a probability that each image block belongs to the visual token; generating a visual embedding feature corresponding to each image block based on a feature corresponding to each visual token in the plurality of visual tokens and the probability distribution, to obtain a plurality of visual embedding features of the plurality of image blocks; and processing the plurality of visual embedding features and a text feature of the task indication information by using a multi-modal model to generate an execution result of the target task.

[0005] According to an embodiment of the present disclosure, determining the plurality of visual tokens corresponding to each image block and the probability distribution corresponding to the plurality of visual tokens includes: performing feature encoding on each image block to obtain a feature vector corresponding to each image block; determining a similarity between each feature vector and each visual token in a target visual codebook, and determining the plurality of visual tokens corresponding to each image block based on the similarity; the similarity between each image block and any visual token in the plurality of visual tokens is greater than a target threshold; and determining the probability distribution corresponding to the plurality of visual tokens according to the similarity between each image block and the plurality of visual tokens.

[0006] According to an embodiment of the present disclosure, generating the visual embedding feature corresponding to each image block based on the feature corresponding to each visual token in the plurality of visual tokens and the probability distribution includes: performing weighted calculation based on the feature corresponding to each visual token in the plurality of visual tokens and the probability distribution to obtain the visual embedding feature corresponding to each image block.

[0007] According to embodiments of this disclosure, a multimodal model is used to process multiple visual embedding features and text features of task instruction information to generate the execution result of a target task. This includes: processing the text features of the task instruction information to obtain a text embedding vector; generating multiple visual key-value pairs corresponding to the visual embedding vectors of the multiple visual embedding features, and storing the multiple visual key-value pairs in a target cache block; fusing the multiple visual embedding vectors and the text embedding vectors to obtain an initial multimodal vector sequence; using the multiple visual key-value pairs in the target cache block, performing self-attention calculation based on the initial multimodal vector sequence to obtain a target multimodal vector sequence, and generating the execution result of the target task based on the target multimodal vector sequence.

[0008] According to embodiments of this disclosure, storing multiple visual key-value pairs into a target cache block includes: calculating the storage capacity of the target cache block based on the number of multiple image blocks and the dimension of each visual embedding vector in the multiple visual embedding vectors; and allocating contiguous storage space for the multiple visual key-value pairs based on the storage capacity, so as to store the multiple visual key-value pairs into the target cache block.

[0009] According to embodiments of this disclosure, the text embedding vector includes multiple placeholders, the number of which is the same as the number of multiple visual embedding vectors; the multiple visual embedding vectors and the text embedding vector are fused to obtain an initial multimodal vector sequence, including: replacing multiple placeholders in the text embedding vector with multiple visual embedding vectors to obtain the initial multimodal vector sequence.

[0010] According to embodiments of this disclosure, a target multimodal vector sequence is obtained by performing self-attention calculation based on an initial multimodal vector sequence using multiple visual key-value pairs in a target cache block, including: generating text key-value pairs for text embedding vectors; concatenating the text key-value pairs with multiple visual key-value pairs in the target cache block to obtain an initial key-value sequence; and performing self-attention calculation based on the initial multimodal vector sequence and the initial key-value sequence to obtain the target multimodal vector sequence.

[0011] According to embodiments of this disclosure, the method further includes: in response to obtaining new task indication information, processing the text features of the new task indication information to obtain a new text embedding vector; the new task indication information represents the execution of a new target task based at least on the image to be processed; determining a new initial multimodal vector sequence based on the target multimodal vector sequence and the new text embedding vector; and performing self-attention calculation based on the new initial multimodal vector sequence using multiple visual key-value pairs in the target cache block to generate the execution result of the new target task.

[0012] According to embodiments of this disclosure, self-attention computation is performed based on a new initial multimodal vector sequence using multiple visual key-value pairs in the target cache block to generate a new execution result for the target task. This includes: calculating new text key-value pairs corresponding to new text embedding vectors, and concatenating the new text key-value pairs with multiple visual key-value pairs to obtain a new initial key-value sequence; based on the new initial multimodal vector sequence and the new initial key-value sequence, calling the cached content of the target cache block with a reuse marker, and performing self-attention computation to generate a new execution result for the target task; wherein the target cache block is set with a reuse marker and has a corresponding reference counter; in response to the cached content of the target cache block with the reuse marker being called, the corresponding reference counter is incremented; in response to the cached content of the target cache block with the reuse marker not being called within a target duration, the corresponding reference counter is decremented; in response to the reference counter's count satisfying the target condition, the cached content in the target cache block is deleted.

[0013] Another aspect of this disclosure provides an electronic device, an acquisition module for acquiring an image to be processed and task instruction information; the task instruction information represents the execution of a target task based at least on the image to be processed; a processor for segmenting the image to be processed into multiple image blocks, determining multiple visual terms corresponding to each image block, and a probability distribution corresponding to the multiple visual terms; the probability corresponding to the visual terms represents the probability that each image block belongs to that visual term; generating visual embedding features corresponding to each image block based on the features and probability distribution of each visual term in the multiple visual terms, thereby obtaining multiple visual embedding features of multiple image blocks; processing the multiple visual embedding features and the text features of the task instruction information using a multimodal model to generate the execution result of the target task; and an output module for outputting the execution result of the target task.

[0014] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0015] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0016] Figure 1 This is a flowchart of a multimodal data processing method according to an embodiment of the present disclosure;

[0017] Figure 2 This is a flowchart of a multimodal data processing method according to another embodiment of the present disclosure;

[0018] Figure 3 This is a schematic diagram of a multimodal data processing method according to an embodiment of the present disclosure;

[0019] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure; and

[0020] Figure 5 This is a schematic block diagram of an example electronic device used to implement embodiments of the present disclosure. Detailed Implementation

[0021] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0022] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of data (including but not limited to user personal information) comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals.

[0023] Figure 1 This is a flowchart of a multimodal data processing method according to an embodiment of the present disclosure.

[0024] like Figure 1 As shown, the multimodal data processing method of this embodiment includes operations S110-S140.

[0025] In operation S110, the image to be processed and task instruction information are acquired; the task instruction information indicates that the target task is to be performed based at least on the image to be processed.

[0026] In embodiments of this disclosure, the image to be processed refers to the input image data that needs to be processed. For example, the image to be processed may include a photograph of a natural scene or a design drawing. In implementation, the image can be an RGB image or a grayscale image of any size.

[0027] In embodiments of this disclosure, task instruction information may include instructions or questions expressed in text form, used to characterize a target task to be performed at least based on the image to be processed. For example, task instruction information may include "describe the contents of this image" or "answer related questions based on the image," which is essentially a text sequence.

[0028] In embodiments of this disclosure, task instruction information may include instructions or questions expressed in audio form, used to characterize a target task to be performed at least based on the image to be processed. For example, task instruction information may include a user's voice instruction to "describe the contents of this image" or "answer related questions based on the image," which is essentially audio information.

[0029] In embodiments of this disclosure, the target task refers to a specific operation that needs to be performed by the multimodal model, such as visual question answering, image description generation, or visual reasoning tasks. The target task can be defined by task instruction information and depends on the image content.

[0030] In embodiments of this disclosure, the image to be processed and task indication information are acquired. The steps of this embodiment can acquire the image to be processed and task indication information through an input interface. For example, a user uploads an image and enters a text question; the system receives this data as the starting point for multimodal processing. The methods for acquiring the image to be processed and task indication information may include file upload, real-time acquisition, or database querying.

[0031] In operation S120, the image to be processed is segmented into multiple image blocks, and multiple visual terms corresponding to each image block are determined, as well as the probability distribution of the multiple visual terms; the probability corresponding to the visual term represents the probability that each image block belongs to that visual term.

[0032] In embodiments of this disclosure, an image block refers to a small region unit obtained after segmenting an image to be processed. Each image block represents a part of the image; for example, an image can be uniformly divided into a 16x16 grid, and each grid unit is an image block.

[0033] In embodiments of this disclosure, visual lexical units refer to discrete visual basic units used to represent the semantic content of image patches. Each visual lexical unit corresponds to a unique index or identifier. For example, a visual lexical unit can represent visual concepts such as "edge," "texture," or "object part."

[0034] In the embodiments of this disclosure, the probability distribution refers to the set of probabilities that each image patch is associated with multiple visual words, representing similarity in the form of probability values. The probability values ​​characterize the likelihood that an image patch belongs to a specific visual word. The probability distribution can reflect the strength of the association between image patches and visual words. The probability distribution reflects semantic uncertainty; an image patch may belong to multiple words with a high probability. Determining the probability distribution corresponding to multiple visual words can avoid information loss in binarization decisions. For example, an image patch may belong to word A with a probability of 0.6, to word B with a probability of 0.3, and to word C with a probability of 0.1.

[0035] In embodiments of this disclosure, the image to be processed is segmented into multiple image blocks. The steps of this embodiment can divide the image to be processed into multiple image blocks using an image segmentation algorithm. The segmentation process can select the optimal segmentation strategy based on image size and coverage thresholds to ensure that the size and number of image blocks meet the model input requirements. After segmentation, each image block enters the visual processing flow as an independent unit.

[0036] In embodiments of this disclosure, multiple visual terms corresponding to each image patch and a probability distribution corresponding to the multiple visual terms are determined. This action can be performed by a visual word segmenter. Converting the image from pixel space to discrete probability space, segmenting the image patches ensures the granularity of processing, and determining the visual terms and probability distributions achieves a discretized representation of semantics.

[0037] In operation S130, based on the features and probability distribution corresponding to each visual word in multiple visual words, visual embedding features corresponding to each image patch are generated, resulting in multiple visual embedding features for multiple image patches.

[0038] In the embodiments of this disclosure, the features corresponding to visual lexical units refer to the abstract representation of each visual lexical unit in the model. Features capture the semantic information of the lexical units for subsequent computation. For example, the features of visual lexical unit A may encode the semantics of "edge," and the features of visual lexical unit B may encode the semantics of "color." Visual embedding features refer to the final semantic representation of each image patch. As input to the multimodal model, visual embedding features combine the clarity of discrete semantics with the richness of continuous information.

[0039] In the embodiments of this disclosure, visual embedding features corresponding to each image patch are generated based on the features and probability distribution of each visual word in a plurality of visual words, resulting in multiple visual embedding features for multiple image patches. For each image patch, the system generates visual embedding features for the image patch based on the features and corresponding probability distribution of its associated multiple visual words. Generating visual embedding features based on probability distribution can ensure the complete preservation of semantic information and avoid the information loss caused by using only the word with the highest probability in related technologies.

[0040] In operation S140, a multimodal model is used to process multiple visual embedded features and textual features of task instruction information to generate the execution result of the target task.

[0041] In the embodiments of this disclosure, a multimodal model refers to a model capable of processing multimodal data simultaneously. The multimodal model is used to fuse multimodal inputs and generate outputs. The multimodal model can be a model based on vLLM, which provides efficient inference support for the multimodal model, including features such as memory management, dynamic batching, and attention optimization, thereby enabling efficient deployment of discrete visual word segmentation models.

[0042] In embodiments of this disclosure, textual features refer to the feature representation of task instruction information after processing, such as converting a text sequence into a symbolic form that the model can understand. Textual features carry the semantics of the task instructions and are used to guide the multimodal reasoning process.

[0043] In the embodiments of this disclosure, the execution result refers to the target task output generated by the multimodal model, and the execution result can be presented in the form of a text sequence. For example, the execution result may include a description of an image, an answer to a question, or a reasoning conclusion. The execution result is the final product of multimodal processing and directly serves user needs.

[0044] In the embodiments of this disclosure, visual embedding features and textual features are input into a multimodal model. The multimodal model aligns and integrates these two types of features within a unified space through an internal fusion mechanism. During the fusion process, the model can utilize positional encoding and type identification to achieve precise matching between features, avoiding semantic conflicts. The multimodal model can then perform autoregressive generation based on the fused feature sequence, progressively inferring the execution result in text form.

[0045] For example, given the task instruction "Describe the main object in the image", the multimodal model generates a text response such as "There is a car in the center of the image" based on visual embedding features.

[0046] Through the embodiments of this disclosure, by determining multiple visual terms and their probability distributions corresponding to each image patch, and based on the probability distributions and visual term features, the generated visual embedding features can retain richer semantic information. The method of this embodiment, by introducing probability distributions to generate visual embedding features, effectively solves the information loss problem caused by using only the maximum value index of the probability distribution in related technologies. This enhances the multimodal model's deep understanding and expressive ability of complex visual scenes, thereby improving the accuracy of the obtained task execution results and the overall processing performance.

[0047] Figure 2 This is a flowchart of a multimodal data processing method according to another embodiment of the present disclosure.

[0048] like Figure 2 As shown, in this embodiment, determining multiple visual terms corresponding to each image block and the probability distribution corresponding to the multiple visual terms includes operations S210-S230.

[0049] In operation S210, feature encoding is performed on each image block to obtain the feature vector corresponding to each image block.

[0050] In embodiments of this disclosure, feature encoding can refer to the process of converting image patches into numerical representations using a neural network model. For example, a convolutional neural network using a visual word segmenter can extract abstract features of image patches. A feature vector is a fixed-dimensional numerical sequence obtained after feature encoding, used to characterize the semantic content of an image patch. For example, an image patch may be encoded into a 512-dimensional vector, where each dimension captures different visual attributes (e.g., texture or shape).

[0051] In operation S220, the similarity between each feature vector and each visual word in the target visual codebook is determined, and multiple visual words corresponding to each image block are determined based on the similarity; the similarity between each image block and any visual word among the multiple visual words is greater than the target threshold.

[0052] In embodiments of this disclosure, the target visual codebook refers to a predefined visual vocabulary containing a set of discrete visual lexical units, each representing a basic visual concept (such as "edge" or "circle"). The target visual codebook has a fixed size; for example, it contains 1000 visual lexical units.

[0053] In the embodiments of this disclosure, similarity refers to the strength of the association between a feature vector and a visual word, which can be calculated using a distance metric (such as cosine similarity). A higher similarity value indicates a semantic similarity between the image patch and the visual word. The target threshold is a preset similarity threshold value used to filter visual words with strong associations. For example, a threshold of 0.7 would retain only visual words with a similarity greater than 0.7.

[0054] In embodiments of this disclosure, the similarity between each feature vector and each visual word in the target visual codebook is determined, and multiple visual words corresponding to each image patch are determined based on the similarity. The similarity between each feature vector and all visual words in the target visual codebook is calculated, and multiple visual words with similarity greater than the target threshold are selected based on a target threshold.

[0055] For example, if the feature vector of an image patch has similarities of 0.8, 0.75, and 0.6 (threshold 0.7) with word A, word B, and word C in the codebook, then word A and word B are determined to be the visual words corresponding to the image patch.

[0056] In operation S230, the probability distribution corresponding to the multiple visual words is determined based on the similarity between each image patch and the multiple visual words.

[0057] In the embodiments of this disclosure, the probability distribution refers to the set of probabilities that each image patch is associated with multiple visual words. The probability values ​​are obtained by similarity normalization and represent the likelihood that the image patch belongs to each word. For example, the probability distribution can be represented as [word A: 0.53, word B: 0.47], with a sum of 1.

[0058] In embodiments of this disclosure, a probability distribution corresponding to multiple visual words is determined based on the similarity between each image patch and multiple visual words. The similarity values ​​of the selected visual words can be linearly normalized and converted into a probability distribution.

[0059] For example, for similarity values ​​of 0.8 and 0.75, the normalized probability distribution is approximately [lexicon A: 0.52, lexicon B: 0.48]. This distribution preserves semantic uncertainty.

[0060] The embodiments of this disclosure, through feature encoding and similarity-driven probability distribution determination, effectively utilize the complete association information of visual lexical units, avoiding information loss caused by using only the maximum value index in related technologies. The method of this embodiment improves the understanding accuracy of multimodal models for complex visual scenes, enhances the robustness of the probability distribution, and thus improves the accuracy of task execution results.

[0061] In some embodiments of this disclosure, visual embedding features corresponding to each image patch are generated based on the features and probability distribution corresponding to each visual word in a plurality of visual words, including: performing weighted calculation based on the features and probability distribution corresponding to each visual word in a plurality of visual words to obtain visual embedding features corresponding to each image patch.

[0062] In embodiments of this disclosure, weighted computation refers to a mathematical fusion operation in which the features of each visual word are scaled and summed according to their probability weights.

[0063] In the embodiments of this disclosure, a weighted calculation is performed based on the features and probability distribution corresponding to each visual word in a plurality of visual words to obtain the visual embedding features corresponding to each image patch. The system obtains the features and probability distribution of multiple visual words corresponding to each image patch. Then, a weighted calculation is performed, multiplying the features of each visual word by its corresponding probability weight, and summing all the weighted features to obtain the visual embedding features.

[0064] Specifically, for each image patch, the calculation process follows a distribution-weighted formula, where the visual embedding feature is equal to the sum of the products of each visual word feature and its probability. This mechanism ensures that the information in the probability distribution is fully preserved, and the generated visual embedding features can more accurately reflect the semantic content of the image patch, while supporting the model's deep understanding of complex visual scenes.

[0065] For example, an image patch corresponds to two visual terms: term A (representing "edge") and term B (representing "color"). The features of term A capture the geometric information of the edge, while the features of term B capture the spectral information of the color. The probability distribution shows that the probability of the image patch belonging to term A is 0.7, and the probability of it belonging to term B is 0.3. In the weighted calculation, the features of term A are multiplied by 0.7, and the features of term B are multiplied by 0.3. The results are then summed to obtain the visual embedding feature of the image patch. This feature simultaneously incorporates the semantics of both edges and color, but is more biased towards edge information, thereby improving the accuracy of subsequent multimodal processing.

[0066] Through the embodiments of this disclosure, a weighted calculation mechanism is used to deeply fuse probability distributions with visual lexical features, avoiding the information loss problem of traditional methods that only use the maximum value index. The method of this embodiment improves the expressive power and robustness of visual embedding features, enhances the understanding accuracy of multimodal models for complex visual scenes, and thus significantly improves the accuracy and reliability of task execution results.

[0067] In some embodiments of this disclosure, a multimodal model is used to process multiple visual embedding features and text features of task instruction information to generate the execution result of the target task. This includes: processing the text features of the task instruction information to obtain a text embedding vector; generating multiple visual key-value pairs of visual embedding vectors corresponding to multiple visual embedding features, and storing the multiple visual key-value pairs in a target cache block; fusing the multiple visual embedding vectors and the text embedding vectors to obtain an initial multimodal vector sequence; using the multiple visual key-value pairs in the target cache block, performing self-attention calculation based on the initial multimodal vector sequence to obtain a target multimodal vector sequence, and generating the execution result of the target task based on the target multimodal vector sequence.

[0068] In the embodiments of this disclosure, text features refer to the symbolic representation of task instruction information after processing by a text segmenter. Text embedding vectors refer to the continuous numerical representation of text features after transformation through an embedding layer, used to capture the semantic information of the text. For example, the task instruction "describe this image" can be converted into a series of numerical vectors.

[0069] In embodiments of this disclosure, the text features of the task instruction information are processed to obtain a text embedding vector. The task instruction information is input into a text segmenter, which decomposes it into discrete sequences of text words. These sequences of text words are then mapped to continuous text embedding vectors through an embedding layer.

[0070] In the embodiments of this disclosure, visual key-value pairs refer to key-value pairs in the self-attention mechanism, used to store contextual information of visual embedding vectors. Each visual embedding vector corresponds to a key-value pair, where the key is used to calculate attention weights and the value is used to generate attention output. The target cache block refers to a dedicated storage area allocated for visual key-value pairs, employing a contiguous block allocation strategy to ensure efficient storage and locality of access.

[0071] In the embodiments of this disclosure, multiple visual key-value pairs are generated for visual embedding vectors corresponding to multiple visual embedding features, and these multiple visual key-value pairs are stored in a target cache block. This step is implemented through a key-value block reuse mechanism. A corresponding key and value are calculated for each visual embedding vector to form a visual key-value pair. Contiguous storage space is allocated for the target cache block in the cache, and all visual key-value pairs are stored in the target cache block.

[0072] In the embodiments of this disclosure, the initial multimodal vector sequence refers to the unified sequence formed by fusing visual and text embedding vectors, which serves as the input for self-attention computation. This sequence maintains the relative positional relationship between visual and textual information.

[0073] In embodiments of this disclosure, multiple visual embedding vectors and text embedding vectors are fused to obtain an initial multimodal vector sequence. This step is achieved through a multimodal fusion mechanism. For example, specific placeholders in the text embedding vectors can be replaced with corresponding visual embedding vectors to achieve the concatenation of the two modalities. Then, position encoding and type identification are used to ensure the precise alignment of each element in the sequence, forming the initial multimodal vector sequence.

[0074] In the embodiments of this disclosure, multiple visual key-value pairs in the target cache block are used to perform self-attention computation based on an initial multimodal vector sequence to obtain a target multimodal vector sequence, and the execution result of the target task is generated based on the target multimodal vector sequence. This step is implemented through a self-attention mechanism and cache reuse. A query vector is generated based on the initial multimodal vector sequence. Then, pre-stored visual key-value pairs are read from the target cache block, concatenated with text key-value pairs, and used for attention computation. Through autoregressive generation of a language model, the target multimodal vector sequence is converted into an execution result in text form.

[0075] Through the embodiments of this disclosure, the caching and reuse mechanism of visual key-value pairs avoids the repeated calculation of visual information during inference, significantly improving the inference efficiency of multimodal models. The method of this embodiment reduces computational redundancy and memory usage, supports long-sequence multimodal inference, and ensures the accuracy and real-time performance of task execution results.

[0076] In some embodiments of this disclosure, storing multiple visual key-value pairs into a target cache block includes: calculating the storage capacity of the target cache block based on the number of multiple image patches and the dimension of each visual embedding vector in the multiple visual embedding vectors; and allocating contiguous storage space for the multiple visual key-value pairs based on the storage capacity to store the multiple visual key-value pairs into the target cache block.

[0077] In embodiments of this disclosure, the number of image blocks refers to the total number of image blocks obtained after segmenting the image to be processed. For example, if an image is uniformly divided into 16 image blocks, the number is 16. This number reflects the scale of visual information and directly affects the number of key-value pairs.

[0078] In embodiments of this disclosure, the dimension of a visual embedding vector refers to the length or size of each visual embedding vector. For example, a 512-dimensional vector means that the vector contains 512 numerical elements. The dimension determines the storage size of a single key-value pair. The storage capacity of the target cache block refers to the amount of memory space required by the target cache block, calculated based on the number of image patches and the vector dimension, and is used to ensure that the cache block can accommodate all visual key-value pairs.

[0079] In the embodiments of this disclosure, contiguous storage space refers to a region of memory with contiguous addresses allocated for storing visual key-value pairs. Contiguous allocation avoids memory fragmentation and improves the locality and efficiency of data access. The target cache block refers to a cache region specifically allocated for visual key-value pairs, employing a contiguous storage strategy to support efficient key-value cache reuse.

[0080] In embodiments of this disclosure, the storage capacity of the target cache block is calculated based on the number of multiple image patches and the dimension of each visual embedding vector among multiple visual embedding vectors. The system obtains the number of image patches and the dimension of each visual embedding vector. The storage capacity is calculated according to a predefined formula, for example, storage capacity = number of image patches × key-value pair size × dimension factor. The key-value pair size is determined by the dimensions of both the key and the value, which are typically related to the dimension of the visual embedding vector. The calculation process must ensure that the storage capacity covers the total size of all visual key-value pairs, while considering memory alignment requirements to avoid access conflicts.

[0081] In embodiments of this disclosure, contiguous storage space is allocated for multiple visual key-value pairs based on storage capacity to store the multiple visual key-value pairs into a target cache block. Based on the calculated storage capacity, the system allocates a contiguous storage region in the key-value cache as the target cache block. The allocation strategy considers the requirements of block size alignment and parallel processing of multiple images; for example, space is allocated sequentially according to image block order, ensuring that each visual key-value pair occupies a fixed offset. The visual key-value pairs are stored sequentially into this space; for example, the key-value pairs of the first image block are stored at the starting address, and subsequent key-value pairs are stored contiguously.

[0082] Through embodiments of this disclosure, the cache management of visual key-value pairs is optimized by accurately calculating storage capacity and allocating contiguous space, reducing memory fragmentation and access latency. The method of this embodiment improves the inference efficiency and stability of multimodal models, supports large-scale image processing, and effectively reduces video memory overhead.

[0083] In some embodiments of this disclosure, the text embedding vector includes multiple placeholders, the number of which is the same as the number of visual embedding vectors; the multiple visual embedding vectors and the text embedding vector are fused to obtain an initial multimodal vector sequence, including: replacing multiple placeholders in the text embedding vector with multiple visual embedding vectors to obtain the initial multimodal vector sequence.

[0084] In the embodiments of this disclosure, placeholders are special marker positions reserved in the text embedding vector to indicate the insertion point of visual information. Each placeholder corresponds to the embedding position of an image patch, and the number of placeholders is exactly the same as that of the visual embedding vector.

[0085] In embodiments of this disclosure, multiple placeholders in the text embedding vector are replaced with multiple visual embedding vectors to obtain an initial multimodal vector sequence. Dynamic replacement is achieved through a multimodal fusion mechanism. The system identifies the positions of all placeholders in the text embedding vector; these placeholders are pre-set in the text sequence to mark the insertion points of visual information. Each placeholder has a specific positional code to ensure precise alignment with its corresponding visual embedding vector. According to a preset mapping relationship, each placeholder is replaced with its corresponding visual embedding vector. The replacement process maintains the sequence's order and positional consistency, ensuring that visual information is inserted into the correct contextual position. After the replacement is complete, the original text embedding vector is transformed into a unified sequence containing both text and visual information.

[0086] Understandably, the advantage of the replacement mechanism in this embodiment lies in maintaining the structural integrity of the sequence while achieving the natural fusion of information from different modalities. Through position alignment and type identification, the system can accurately distinguish between text and visual elements, avoiding semantic confusion.

[0087] For example, the task instruction is "Describe the content of this image". The text embedding vector contains 5 placeholders, represented by special markers [IMG1], [IMG2], [IMG3], [IMG4], and [IMG5]. Simultaneously, the image to be processed is segmented into 5 image patches, generating 5 visual embedding vectors. During fusion processing, the system first identifies the positions of the 5 placeholders in the text embedding vectors. Then, [IMG1] is replaced with the first visual embedding vector in sequence, [IMG2] with the second, and so on. The final initial multimodal vector sequence structure is: [visual embedding vector 1, visual embedding vector 2, visual embedding vector 3, visual embedding vector 4, visual embedding vector 5, "describe", "this", "image of", "content"].

[0088] Through the embodiments of this disclosure, a placeholder replacement mechanism is used to achieve precise fusion of visual and textual information, ensuring the structural integrity of the multimodal sequence. The method of this embodiment improves the model's ability to process cross-modal information and enhances the accuracy and stability of the inference process.

[0089] In some embodiments of this disclosure, a target multimodal vector sequence is obtained by performing self-attention computation based on an initial multimodal vector sequence using multiple visual key-value pairs in the target cache block, including: generating text key-value pairs of text embedding vectors; concatenating the text key-value pairs with multiple visual key-value pairs in the target cache block to obtain an initial key-value sequence; and performing self-attention computation based on the initial multimodal vector sequence and the initial key-value sequence to obtain the target multimodal vector sequence.

[0090] In the embodiments of this disclosure, text key-value pairs refer to key-value pairs generated from text embedding vectors through a self-attention mechanism, used to store the contextual representation of text information. The keys are used to calculate attention weights, and the values ​​are used to generate attention outputs.

[0091] In embodiments of this disclosure, text key-value pairs of text embedding vectors are generated. The system generates query vectors, key vectors, and value vectors based on the text embedding vectors. Specifically, the text embedding vectors can be mapped to key vectors and value vectors respectively through linear transformation layers (such as fully connected layers) to form text key-value pairs.

[0092] In embodiments of this disclosure, text key-value pairs are concatenated with multiple visual key-value pairs in the target cache block to obtain an initial key-value sequence. The initial key-value sequence is the result of sequentially concatenating visual and text key-value pairs, maintaining the relative positions between modalities. The system reads pre-stored visual key-value pairs from the target cache block and concatenates them sequentially with the generated text key-value pairs. During concatenation, visual key-value pairs can be placed at the beginning of the sequence, and text key-value pairs can be placed at the end to ensure alignment. For example, if there are N visual key-value pairs and M text key-value pairs, the length of the initial key-value sequence is N+M.

[0093] In the embodiments of this disclosure, self-attention computation is performed based on an initial multimodal vector sequence and an initial key-value sequence to obtain a target multimodal vector sequence. This step is implemented through a self-attention mechanism. The system uses the initial multimodal vector sequence as the query vector and the initial key-value sequence as the key and value inputs, and performs standard self-attention computation. Specifically, the similarity between the query and the key is calculated, attention weights are generated, and the values ​​are weighted and summed based on the weights to obtain the attention output. Finally, the target multimodal vector sequence is output, which integrates cross-modal semantic information.

[0094] Through the embodiments of this disclosure, deep interaction between visual and textual information is achieved by concatenating key-value sequences and performing self-attention computation, avoiding information loss caused by modal isolation. The method of this embodiment improves the inference accuracy and efficiency of multimodal models, supports real-time processing of complex scenarios, reduces computational redundancy, and enhances the reliability of task execution results.

[0095] In some embodiments of this disclosure, the method further includes: in response to obtaining new task indication information, processing the text features of the new task indication information to obtain a new text embedding vector; the new task indication information represents the execution of a new target task based at least on the image to be processed; determining a new initial multimodal vector sequence based on the target multimodal vector sequence and the new text embedding vector; and performing self-attention calculation based on the new initial multimodal vector sequence using multiple visual key-value pairs in the target cache block to generate the execution result of the new target task.

[0096] In embodiments of this disclosure, new task instruction information refers to subsequent task instructions proposed based on the same image to be processed, forming a sequential processing relationship with the initial target task. For example, after completing the first task of "describing image content," the user may propose a new task instruction: "counting the number of objects in the image."

[0097] In the embodiments of this disclosure, the new text embedding vector refers to the numerical representation of the new task indication information obtained after processing by a text segmenter and an embedding layer. The new text embedding vector carries the semantic connotation of the new task while maintaining the same dimensional structure as the original text embedding vector.

[0098] In embodiments of this disclosure, the new initial multimodal vector sequence refers to the input sequence formed by re-fusing the new text embedding vector with existing visual information. The new initial multimodal vector sequence contains both the visual features of the original image and the text instructions of the new task.

[0099] In the embodiments of this disclosure, in response to acquiring new task instruction information, the text features of the new task instruction information are processed to obtain a new text embedding vector. The system continuously monitors the input interface, and when new task instruction information is detected, the text processing flow is immediately initiated. The new task instruction information is processed by a text segmenter to obtain a new text embedding vector. The new text embedding vector uses the same mapping rules and dimensional space as the original text embedding vector to ensure compatibility with subsequent fusion operations.

[0100] In embodiments of this disclosure, a new initial multimodal vector sequence is determined based on the target multimodal vector sequence and the new text embedding vector. This step is implemented through a sequence reconstruction mechanism. The system fuses the processed target multimodal vector sequence with the new text embedding vector. During the fusion process, the system ensures the structural integrity of the new sequence through positional encoding remapping. The positional information of the visual part remains unchanged, while the text part is re-encoded according to the new sequence length.

[0101] In the embodiments of this disclosure, multiple visual key-value pairs in the target cache block are used to perform self-attention computation based on a new initial multimodal vector sequence to generate the execution result of a new target task. The system directly calls the visual key-value pairs already stored in the target cache block, avoiding repeated visual feature extraction and key-value pair computation for the same image. During the self-attention computation, only corresponding key-value pairs need to be generated for the new text portion. The self-attention computation adopts an incremental processing strategy, using the new initial multimodal vector sequence as the query input, and interacting with the reused visual key-value pairs and the newly generated text key-value pairs for attention.

[0102] For example, a user initially presents a street view image with the objective task of "describing the vehicles in the image." After processing this, the user provides a new task instruction: "counting the number of pedestrians." The system first converts this new instruction into a new text embedding vector. Then, it fuses the newly generated target multimodal vector sequence (containing visual features of the street view image and text features from the initial task) with the new text embedding vector to form a new initial multimodal vector sequence. The system directly reuses the visual key-value pairs of the street view image stored in the target cache block, only needing to generate corresponding key-value pairs for the new text instruction. Through self-attention computation, the model can understand the new task based on the existing visual context, ultimately outputting a result such as "there are 3 pedestrians in the image." This entire process avoids redundant computation of image features, improving response speed.

[0103] By establishing a cross-task cache reuse mechanism, the efficiency of multimodal systems in processing serialized tasks is significantly improved through the embodiments of this disclosure. The method of this embodiment reduces redundant computation overhead, supports real-time multi-round interactions, and ensures context consistency between tasks, thereby improving the efficiency of the multimodal data processing method of this embodiment.

[0104] In some embodiments of this disclosure, self-attention computation is performed based on a new initial multimodal vector sequence using multiple visual key-value pairs in the target cache block to generate a new execution result for the target task. This includes: calculating new text key-value pairs corresponding to new text embedding vectors, and concatenating the new text key-value pairs with multiple visual key-value pairs to obtain a new initial key-value sequence; based on the new initial multimodal vector sequence and the new initial key-value sequence, calling the cached content of the target cache block with a reuse marker, and performing self-attention computation to generate a new execution result for the target task; wherein the target cache block is set with a reuse marker and has a corresponding reference counter; in response to the cached content of the target cache block with the reuse marker being called, the corresponding reference counter is incremented; in response to the cached content of the target cache block with the reuse marker not being called within the target duration, the corresponding reference counter is decremented; in response to the reference counter's count satisfying the target condition, the cached content in the target cache block is deleted.

[0105] In the embodiments of this disclosure, the new text key-value pair refers to the key-value pair generated by the new text embedding vector through a self-attention mechanism, used to store the contextual representation of the new task instruction information. For example, for the new task instruction information "count the number of objects in an image," its text embedding vector will generate corresponding keys and values, forming a new text key-value pair. The new initial key-value sequence refers to the unified sequence formed by concatenating the new text key-value pairs with the existing visual key-value pairs in the target cache block. This sequence serves as the input for self-attention computation, containing the complete context of visual information and the newly added text instruction.

[0106] In embodiments of this disclosure, a reuse flag is an identifier set for a target cache block to indicate whether the cache block can be reused. For example, the reuse flag may include a Boolean value (such as True / False), where True indicates that the cached content is available for subsequent tasks to access. A reference counter is a counting device associated with the target cache block to track the number of times the target cache block is accessed. The reference counter's count changes dynamically with each access event, for example, incrementing with each access and decrementing when no access is made.

[0107] In the embodiments of this disclosure, the target condition refers to the threshold condition that triggers the deletion of cached content. The target condition can be set based on the count value of a reference counter. For example, when the count value drops to 0, the target condition is met, and the system automatically deletes the cached content to free up memory. The target duration refers to a preset time window used to determine whether the cache block has not been accessed. For example, the target duration can be set to 10 seconds; if the cached content is not accessed during this period, the reference counter is decremented.

[0108] In the embodiments of this disclosure, new text key-value pairs corresponding to the new text embedding vector are calculated, and these new text key-value pairs are concatenated with multiple visual key-value pairs to obtain a new initial key-value sequence. The system generates corresponding keys and values ​​based on the new text embedding vector to form new text key-value pairs. The system reads stored visual key-value pairs from the target cache block and concatenates them sequentially with the new text key-value pairs. During concatenation, the visual key-value pairs are placed at the beginning of the sequence, and the new text key-value pairs are placed at the end, forming a new initial key-value sequence. This step avoids redundant calculation of visual key-value pairs.

[0109] In the embodiments of this disclosure, based on a new initial multimodal vector sequence and a new initial key-value sequence, the cached content of the target cache block with a reuse flag is invoked to perform self-attention computation to generate the execution result of the new target task. This step is implemented through cache invocation and self-attention computation. The system first checks the reuse flag of the target cache block. If it is marked as reusable, the visual key-value pairs within it are directly invoked. Simultaneously, a reference counter is incremented to record this invocation. Using the new initial multimodal vector sequence as the query input and the new initial key-value sequence as the key and value, self-attention computation is performed.

[0110] In the embodiments of this disclosure, the system continuously tracks the usage status of the target cache block: when cached content is accessed, the reference counter increments. If it is not accessed within a target duration, the counter decrements. When the counter meets a target condition (e.g., drops to 0), the system automatically deletes the cached content and releases memory. This design ensures the reasonable reclamation of cache resources and avoids invalid occupation.

[0111] For example, a user first assigns an image processing task, "Describe a vehicle." The system generates visual key-value pairs and stores them in the target cache block, setting the reuse flag to True and the reference counter to 1 initially. Subsequently, the user proposes a new task, "Count the number of pedestrians." The system calculates the text key-value pairs for the new task, calls the visual key-value pairs in the target cache block (incrementing the reference counter to 2), concatenates them to form a new initial key-value sequence, and generates the execution result "There are 3 pedestrians in the image" through self-attention calculation. If the cache block is not called again within the target duration (e.g., 1 minute), the reference counter decreases to 1. If it remains unused, the final count drops to 0, triggering cache deletion.

[0112] The embodiments of this disclosure achieve intelligent management of cached resources through reuse tags and reference counters, significantly improving the efficiency of multimodal systems in processing continuous tasks. The method of this embodiment reduces memory overhead, avoids redundant computations, and ensures timely release of resources, thereby enhancing system stability and scalability.

[0113] Figure 3This is a schematic diagram of a multimodal data processing method according to an embodiment of the present disclosure.

[0114] In embodiments of this disclosure, such as Figure 3 As shown, the system receives input image and text data. The image data undergoes standardization processing via an image preprocessor, including resizing and normalization operations, and is then segmented into multiple regular image patches. These image patches are input to a visual word segmenter, which extracts the depth features of each image patch using a pre-trained visual backbone network and calculates its similarity to all words in the visual codebook, generating a probability distribution matrix. This matrix not only contains the most likely visual word indexes but also retains complete probability distribution information, providing a rich data foundation for subsequent weighted distribution calculations.

[0115] In the embodiments of this disclosure, the system generates visual embedding vectors through an embedding lookup table and a distributed weighting mechanism. The embedding lookup table maps discrete visual terms to continuous feature representations, while the distributed weighting mechanism uses a probability distribution matrix to weight and fuse the features of multiple visual terms, generating visual embedding vectors that retain more complete information. Simultaneously, the input text is processed by a text segmenter and converted into a sequence of text embedding vectors. The visual and text embedding vectors are aligned and concatenated in a unified representation space to form an initial multimodal sequence.

[0116] In the embodiments of this disclosure, the system achieves computational optimization through a key-value cache reuse module. This module generates key-value pairs for visual embedding vectors and stores them in cache blocks marked with reuse tags. During the autoregressive generation process, the system manages the cache lifecycle through a reference counting mechanism, enabling cross-timestep reuse of visual key-value pairs. Each time a new lexical is generated, the system only needs to calculate the key-value pairs for the newly added text portion, directly reusing the cached visual key-value pairs, significantly reducing computational redundancy.

[0117] In embodiments of this disclosure, multimodal sequences and key-value pairs are input to a self-attention computation module. This module calculates the association weights between elements of different modalities through a query-key-value attention mechanism, achieving cross-modal information fusion. The attention output is converted into a probability distribution on the vocabulary by a decoder, and the text sequence is gradually output through an autoregressive generation method.

[0118] Understandably, this embodiment significantly improves processing efficiency while ensuring inference quality through distributed weighted embedding and cache reuse, providing a reliable solution for complex multimodal applications.

[0119] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure.

[0120] In this embodiment of the disclosure, the electronic device 400 includes: an acquisition module 410, configured to acquire an image to be processed and task indication information; the task indication information represents the execution of a target task based at least on the image to be processed; a processor 420, configured to segment the image to be processed into multiple image blocks, determine multiple visual words corresponding to each image block, and the probability distribution corresponding to the multiple visual words; the probability corresponding to the visual words represents the probability that each image block belongs to the visual word; based on the features and probability distribution corresponding to each visual word in the multiple visual words, generate visual embedding features corresponding to each image block, and obtain multiple visual embedding features of multiple image blocks; process the multiple visual embedding features and the text features of the task indication information using a multimodal model to generate the execution result of the target task; and an output module 430, configured to output the execution result of the target task.

[0121] In this embodiment, the electronic device 400 may include a cloud server cluster, an edge computing device, a portable smart device, etc. The electronic device 400 includes a complete multimodal data processing system.

[0122] In this embodiment, the acquisition module 410 is a multimodal data input interface. The processor 420 is the core computing unit of the system. The output module 430 may include a display output unit, a speech synthesis system, etc.

[0123] In this embodiment of the disclosure, processor 420 can execute the multimodal data processing method described above.

[0124] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0125] Figure 5 This is a schematic block diagram of an example electronic device used to implement embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0126] like Figure 5As shown, device 500 includes a computing unit 501, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 502 or a computer program loaded from storage unit 508 into random access memory (RAM) 503. RAM 503 may also store various programs and data required for the operation of device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0127] Multiple components in device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0128] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as multimodal data processing methods. For example, in some embodiments, the multimodal data processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the multimodal data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform multimodal data processing methods by any other suitable means (e.g., by means of firmware).

[0129] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0130] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0131] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0132] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: electronic devices for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0133] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0134] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is established by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0135] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0136] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A multimodal data processing method, comprising: Acquire the image to be processed and task instruction information; The task instruction information indicates that the target task is to be performed at least based on the image to be processed; The image to be processed is divided into multiple image blocks, and multiple visual words corresponding to each image block are determined, as well as the probability distribution of the multiple visual words; the probability corresponding to the visual word represents the probability that each image block belongs to that visual word. Based on the features corresponding to each visual word in the plurality of visual words and the probability distribution, a visual embedding feature corresponding to each image patch is generated, thereby obtaining a plurality of visual embedding features for the plurality of image patches; The multiple visual embedding features and the textual features of the task instruction information are processed using a multimodal model to generate the execution result of the target task.

2. The method according to claim 1, wherein determining the plurality of visual terms corresponding to each image patch and the probability distribution corresponding to the plurality of visual terms comprises: Each image block is feature-encoded to obtain a feature vector corresponding to each image block; The similarity between each feature vector and each visual word in the target visual codebook is determined, and multiple visual words corresponding to each image block are determined based on the similarity; the similarity between each image block and any visual word among the multiple visual words is greater than a target threshold. Based on the similarity between each image patch and the plurality of visual terms, the probability distribution corresponding to the plurality of visual terms is determined.

3. The method according to claim 1, wherein generating visual embedding features corresponding to each image patch based on the features corresponding to each visual word in the plurality of visual words and the probability distribution comprises: Based on the features corresponding to each visual word in the plurality of visual words and the probability distribution, a weighted calculation is performed to obtain the visual embedding features corresponding to each image block.

4. The method according to claim 1, wherein processing the plurality of visual embedding features and the textual features of the task instruction information using a multimodal model to generate the execution result of the target task includes: The text features of the task instruction information are processed to obtain a text embedding vector; Generate multiple visual key-value pairs for visual embedding vectors corresponding to multiple visual embedding features, and store the multiple visual key-value pairs into the target cache block; The multiple visual embedding vectors and the text embedding vectors are fused to obtain an initial multimodal vector sequence; Using the multiple visual key-value pairs in the target cache block, self-attention calculation is performed based on the initial multimodal vector sequence to obtain the target multimodal vector sequence, and the execution result of the target task is generated based on the target multimodal vector sequence.

5. The method according to claim 4, wherein storing the plurality of visual key-value pairs into the target cache block comprises: The storage capacity of the target cache block is calculated based on the number of the plurality of image patches and the dimension of each visual embedding vector in the plurality of visual embedding vectors. Based on the storage capacity, allocate contiguous storage space for the plurality of visual key-value pairs to store the plurality of visual key-value pairs into the target cache block.

6. The method according to claim 4, wherein the text embedding vector includes multiple placeholders, the number of which is the same as the number of the multiple visual embedding vectors; the step of fusing the multiple visual embedding vectors and the text embedding vector to obtain an initial multimodal vector sequence includes: The placeholders in the text embedding vector are replaced with the visual embedding vectors to obtain the initial multimodal vector sequence.

7. The method according to claim 4, wherein the step of using the plurality of visual key-value pairs in the target cache block to perform self-attention calculation based on the initial multimodal vector sequence to obtain the target multimodal vector sequence comprises: Generate the text key-value pairs of the text embedding vector; The text key-value pairs are concatenated with the plurality of visual key-value pairs in the target cache block to obtain an initial key-value sequence; Self-attention calculation is performed based on the initial multimodal vector sequence and the initial key value sequence to obtain the target multimodal vector sequence.

8. The method according to claim 4, further comprising: In response to acquiring new task instruction information, the text features of the new task instruction information are processed to obtain a new text embedding vector; The new task instruction information indicates that a new target task should be performed based at least on the image to be processed; Based on the target multimodal vector sequence and the new text embedding vector, a new initial multimodal vector sequence is determined; Using the multiple visual key-value pairs in the target cache block, self-attention computation is performed based on the new initial multimodal vector sequence to generate the execution result of the new target task.

9. The method according to claim 8, wherein the step of using the plurality of visual key-value pairs in the target cache block to perform self-attention computation based on the new initial multimodal vector sequence to generate the execution result of the new target task includes: Calculate the new text key-value pairs corresponding to the new text embedding vector, and concatenate the new text key-value pairs with the multiple visual key-value pairs to obtain a new initial key-value sequence; Based on the new initial multimodal vector sequence and the new initial key-value sequence, the cached content of the target cache block with reuse tag is called to perform self-attention calculation in order to generate the execution result of the new target task; The target cache block is marked with a reuse flag and has a corresponding reference counter. When the cached content of the target cache block with the reuse flag is accessed, the reference counter is incremented. When the cached content of the target cache block with the reuse flag is not accessed within a target duration, the reference counter is decremented. When the reference counter's count meets the target condition, the cached content in the target cache block is deleted.

10. An electronic device, comprising: The acquisition module is used to acquire the image to be processed and task indication information; The task instruction information indicates that the target task is to be performed at least based on the image to be processed; The processor is configured to segment the image to be processed into multiple image blocks, determine multiple visual terms corresponding to each image block, and the probability distribution corresponding to the multiple visual terms; the probability corresponding to the visual term represents the probability that each image block belongs to that visual term. Based on the features corresponding to each visual word in the plurality of visual words and the probability distribution, a visual embedding feature corresponding to each image patch is generated, thereby obtaining a plurality of visual embedding features for the plurality of image patches; The multiple visual embedding features and the textual features of the task instruction information are processed using a multimodal model to generate the execution result of the target task; The output module is used to output the execution result of the target task.

Images