Multi-modal reasoning method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a multi-modal reasoning method and device, electronic equipment, a storage medium and a program product. The method comprises the steps of obtaining input data; the input data comprises a text and an image; respectively processing the text and the image by using a coding module of a multi-modal reasoning model to obtain text features and image features; performing vector fusion on the text features and the image features to obtain text fusion features and image fusion features; and performing coarse classification on the text and the image based on the text fusion features and the image fusion features by using a hierarchical sensing module of the multi-modal reasoning model to obtain a candidate set, and performing fine classification on the candidate set to obtain a matching result. According to the technical scheme, the reasoning speed can be increased on a CPU platform of a C86 architecture.

Description

Technical Field

[0001] This application belongs to the field of multimodal learning technology, and specifically relates to a multimodal reasoning method, device, electronic device, storage medium and program product. Background Technology

[0002] In cross-modal tasks involving images and text, existing techniques typically employ deep learning models (such as the VisionTransformer model and the standard BERT model) to extract image and text features. While these models offer high accuracy, they suffer from high computational complexity and lengthy inference phases, making them unsuitable for the real-time requirements of low-computing-power platforms. Furthermore, the numerous unnecessary computational paths and repetitive feature calculations during inference further limit their application in low-computing-power devices. Summary of the Invention

[0003] The purpose of this application is to provide a multimodal inference method, apparatus, electronic device, storage medium and program product, which aims to solve the problem of slow inference speed of existing CLIP models on low-performance C86 architecture CPU platforms.

[0004] According to a first aspect of this application, a multimodal reasoning method is provided, comprising: Obtain input data; the input data includes text and images; The text and image are processed separately using the encoding module of the multimodal reasoning model to obtain text features and image features; The text features and the image features are fused by vectorization to obtain text fusion features and image fusion features; Using the hierarchical perception module of the multimodal reasoning model, a candidate set is obtained from the text and image based on the text fusion features and the image fusion features, and then a matching result is obtained from the candidate set through fine classification.

[0005] In an optional implementation, the method further includes: The image is then normalized to obtain a normalized image; Adjust the normalized image to the first resolution image with the set resolution; After mapping the text to a character identifier sequence, it is divided into a set of preprocessed texts of fixed length.

[0006] In an optional implementation, the encoding module includes an image encoding module and a text encoding module; the encoding module utilizes a multimodal inference model to process the text and image respectively, obtaining text features and image features, including: The normalized image is encoded using the image encoding module to obtain second-resolution image features, and / or the first-resolution image is encoded using the image encoding module to obtain first-resolution image features; The preprocessed text is encoded using the text encoding module to obtain the text features.

[0007] In an optional implementation, the text and image are processed separately using the encoding module of a multimodal inference model to obtain text features and image features, including: The image is divided into smaller image blocks; Extract local features from each of the small image patches; Global features of the image are extracted based on the small image patches; The image features are generated based on the local features and the global features.

[0008] In an optional implementation, the text and image are processed separately using the encoding module of a multimodal inference model to obtain text features and image features, including: Map the text to an embedding vector; Extract the contextual features at each position of the embedding vector; Based on the contextual features, extract the text features corresponding to the text.

[0009] In an optional implementation, the hierarchical perception module includes a coarse classification module and a fine classification module; the image features include second-resolution image features and first-resolution image features; the second-resolution image features are obtained by encoding the image using the image encoding module; the first-resolution image features are obtained by encoding the first-resolution image using the image encoding module, and the first-resolution image is obtained by adjusting the resolution of the image; the resolution of the first-resolution image is lower than the resolution of the image. Using the hierarchical perception module of the multimodal inference model, a candidate set is obtained from the text and image based on the text fusion features and the image fusion features, and then a matching result is obtained from the candidate set through fine classification, including: Using the coarse classification module, the candidate set is obtained from the text and image based on the text features and the first resolution image features; Using the fine classification module, the matching result is obtained from the candidate set by fine classification based on the text features and the second resolution image features.

[0010] In an optional implementation, the multimodal inference model employs a sparse activation mechanism during training.

[0011] In an optional implementation, the multimodal inference model is an ONNX format model and is loaded onto an inference engine that integrates the ZenDNN library.

[0012] According to a second aspect of this application, a multimodal reasoning apparatus is provided, comprising: The acquisition module is configured to acquire input data, which includes text and images. The extraction module is configured to process the text and image respectively using the encoding module of the multimodal reasoning model to obtain text features and image features; The fusion module is configured to perform vector fusion of the text features and the image features to obtain text fusion features and image fusion features; The inference module is configured to utilize the hierarchical perception module of the multimodal inference model to obtain a candidate set from the text and image based on the text fusion features and the image fusion features, and then obtain a matching result from the candidate set through fine classification.

[0013] The function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above function.

[0014] In one possible design, the above-described device includes a memory and a processor. The memory stores one or more computer instructions that support the device in performing the corresponding methods described above, and the processor is configured to execute the computer instructions stored in the memory. The device may also include a communication interface for communicating with other devices or communication networks.

[0015] A third aspect of this application provides an electronic device including a processor and a memory, the memory storing a plurality of instructions, and the processor being configured to read the instructions and execute the method of the first aspect described above.

[0016] A fourth aspect of this application provides a computer-readable storage medium storing a plurality of instructions that can be read by a processor and used to execute the method described in the first aspect.

[0017] The fifth aspect of this application provides a computer program product comprising computer instructions which, when executed by a processor, are used to implement the methods described in any of the preceding aspects.

[0018] Compared with related technologies, the technical solution of this application has at least the following advantages: This application introduces a hierarchical perception module that mimics the biological nervous system, dividing model inference into two stages: coarse classification and fine classification. In the coarse classification stage, rapid inference using a first-resolution image filters out possible candidate sets. In the fine classification stage, a second-resolution refinement classification is performed on the candidate sets, thereby improving classification accuracy. Through this approach, this application achieves a 2-3 times faster inference speed on the C86 architecture CPU platform compared to the CLIP model, while significantly reducing resource consumption. While maintaining model recognition accuracy, this application can meet the requirements of tasks such as image retrieval and content moderation, and is suitable for deploying multimodal models in embedded devices, edge computing platforms, and low-power terminal devices, providing a solution for efficiently running Chinese multimodal tasks in resource-constrained environments.

[0019] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures and processes shown in the description and the accompanying drawings. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating a multimodal reasoning method according to an exemplary embodiment of this application.

[0022] Figure 2 This is a schematic diagram of the processing flow related to the layer perception module according to an exemplary embodiment of this application.

[0023] Figure 3 This is a structural block diagram of a multimodal inference device according to an exemplary embodiment of this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] The method provided in this application can be implemented in a terminal environment that may include one or more of the following components: a processor, a memory, and a display screen. The memory stores at least one instruction, which is loaded and executed by the processor to implement the method described in the following embodiments.

[0026] A processor may include one or more processing cores. The processor uses various interfaces and lines to connect various parts of the terminal, and performs various functions and processes data by running or executing instructions, programs, code sets or instruction sets stored in memory, and by calling data stored in memory.

[0027] Memory can include random access memory (RAM) or read-only memory (ROM). Memory can be used to store instructions, programs, code, code sets, or instructions.

[0028] The display screen is used to show the user interface of each application.

[0029] In addition, those skilled in the art will understand that the structure of the terminal described above does not constitute a limitation on the terminal. The terminal may include more or fewer components, or combine certain components, or have different component arrangements. For example, the terminal may also include radio frequency circuits, input units, sensors, audio circuits, power supplies, and other components, which will not be described in detail here.

[0030] The CLIP (Contrastive Language-Image Pre-Training) model extracts image features using the VisionTransformer (ViT) model and text features using the standard Transformer model. It then achieves semantic alignment between the image and text through contrastive learning in a shared vector space. The model is trained by optimizing similarity matching using contrastive loss, and during inference, it performs matching by calculating the embedding similarity between the image and text.

[0031] The inventors of this application have discovered the following shortcomings in using the CLIP model for inference on domestically produced C86 architecture CPUs: 1. The CLIP model uses the Vision Transformer model and the standard Transformer model, which has high computational complexity and is not friendly to low-computing-power devices.

[0032] 2. The CLIP model requires a large amount of embedding similarity calculation during the inference phase, which takes a long time and is difficult to meet real-time requirements.

[0033] Based on the above analysis, this application proposes an exemplary multimodal reasoning method. See also... Figure 1 The multimodal reasoning method includes: Step S101: Obtain input data; the input data includes text and images; Step S102: The text and image are processed separately using the encoding module of the multimodal reasoning model to obtain text features and image features; Step S103: Perform vector fusion of the text features and the image features to obtain text fusion features and image fusion features; Step S104: Using the hierarchical perception module of the multimodal reasoning model, a candidate set is obtained from the text and image coarse classification based on the text fusion features and the image fusion features, and then a matching result is obtained from the candidate set fine classification.

[0034] In this embodiment, the multimodal inference model is a pre-trained model. This model can perform similarity matching on text and images in the input data. That is, it can match images similar to text from multiple input images, and it can also match text similar to images from multiple input texts. It is understood that the input data can contain one or more texts, and the input data can contain one or more images. When the input data includes multiple texts and multiple images, and the goal of the inference is to match images similar to multiple texts, the matching result output by the multimodal inference model can be a set of images that match each text. Each image set can include one or more images, and these images are sorted according to their similarity. Similarly, when the input data includes multiple images and multiple texts, and the goal of the inference is to match text similar to multiple images, the matching result output by the multimodal inference model can include a set of texts that match each image. Each text set can include one or more texts, and these texts are sorted according to their similarity.

[0035] The multimodal reasoning model comprises an encoding module and a hierarchical perception module. The encoding module processes the input images and text separately, obtaining corresponding image features and text features respectively. The hierarchical perception module mimics the hierarchical perception mechanism of biological neural systems, dividing the reasoning process into two stages: coarse classification and fine classification. The coarse classification stage performs rapid reasoning to filter out possible candidate sets; the fine classification stage refines the classification of these candidate sets to obtain the final matching results, thereby improving classification accuracy. It should be noted that the candidate set is either a set of images that are potentially similar to each text, or a set of texts that are potentially similar to each image. The final matching results are either a set of images whose similarity to each text is greater than a set value and sorted by similarity, or a set of texts whose similarity to each image is greater than a set value and sorted by similarity.

[0036] In some embodiments, vector fusion is used to integrate high-dimensional feature vectors of text and images into a unified vector space to achieve cross-modal semantic alignment and similarity calculation. As described above, the image features and text features generated after the image and text are encoded by separate modules are high-dimensional feature vectors, and they are usually located in different semantic spaces, resulting in poor performance when directly compared and matched. To address this issue, this application introduces vector fusion technology. Vector fusion projects image features and text features into a common vector space through a shared linear transformation.

[0037] In linear projection, assuming the image features are... Text features are After being projected into a shared vector space, the image fusion features and text fusion features are as follows: .

[0038] In the formula, , . and They are independent linear projection matrices. It is the vector dimension of image features. It is the vector dimension of the text features. This is the dimension of the shared vector space. After text features and image features are unified into the same vector space through linear projection, the image fusion features and text fusion features can be represented as follows: as well as .

[0039] In some embodiments, the hierarchical perception module leverages the hierarchical perception characteristics of biological neural networks, simplifying the model structure through two stages: coarse classification and fine classification. In the coarse classification stage, potential candidate matching results, i.e., the candidate set, can be quickly filtered using image fusion features. Then, further fine classification is performed on the candidate set. In the fine classification stage, a high-precision model can be used for further refined inference.

[0040] The multimodal inference method proposed in this application can be applied to the C86 architecture CPU platform to address the slow inference speed of the current CLIP model on domestic C86 architecture CPU platforms. The text encoding module of this application can use TinyBert to replace the transformer model used in the CLIP model, significantly reducing computational complexity while maintaining accurate capture of Chinese semantics. The image encoding module can use MobileViT to replace the Vision Transformer used in the CLIP model, achieving high-dimensional feature extraction from images through efficient architecture design, thereby further reducing computational requirements. In addition, this application introduces a hierarchical perception module that mimics biological neural systems, dividing model inference into two stages: coarse classification and fine classification. In the coarse classification stage, rapid inference using a first-resolution image is used to filter out possible candidate sets; in the fine classification stage, a second-resolution refinement classification is performed on the candidate sets to improve classification accuracy; the first resolution is lower than the second resolution.

[0041] Through the above solution, this application achieves a significant improvement in inference speed on the C86 architecture CPU platform, increasing speed by 2-3 times compared to the CLIP model, while substantially reducing resource consumption. While maintaining model recognition accuracy, this application can meet the requirements of tasks such as image retrieval and content moderation, and is suitable for deploying multimodal models in embedded devices, edge computing platforms, and low-power terminal devices, providing a solution for efficiently running Chinese multimodal tasks in resource-constrained environments.

[0042] In some alternative implementations, the method further includes: The image is then normalized to obtain a normalized image; Adjust the normalized image to the first resolution image with the set resolution; After mapping the text to a character identifier sequence, it is divided into a set of preprocessed texts of fixed length.

[0043] In this optional implementation, since the size and length of the images and text in the input data may differ, preprocessing can be performed on both the text and images. During preprocessing, image normalization can be applied; the purpose of normalization is to scale the pixel values ​​of the image to the range [0,1], ensuring a uniform data distribution. The normalization process can be represented by the following formula: .

[0044] in, represents the normalized pixel value of the image, and x represents the original pixel value of the image; This represents the average pixel value of all images in the input data. This represents the standard deviation of all images in the input data.

[0045] After normalization, the image can be adjusted to a preset resolution acceptable to the model. This preset resolution can be pre-defined and should be lower than the resolution of all images in the input data. During the adjustment process, the image can first be scaled proportionally according to its aspect ratio. If the image still exceeds the preset resolution after scaling, the excess parts at the top, bottom, left, and right can be cropped based on the image center to obtain the first resolution image at the preset resolution. The purpose of adjusting the image resolution is to preserve as much information as possible from the image while reducing its size.

[0046] After the above preprocessing, the resulting first-resolution image is represented in the following standard tensor format: .

[0047] in, For the number of channels, This represents the height and width of the image.

[0048] The text preprocessing is as follows: Character-level segmentation is used to divide the text sentence into segments. Then, the segmented characters are mapped to token IDs. Simultaneously, the token sequence, composed of all the tokens corresponding to the characters in the text, is divided into segments of a fixed length, for example, 128. Each resulting token sequence contains 128 tokens. During the segmentation process, token sequences longer than 128 are truncated from the beginning or end, typically retaining the semantically complete latter half (e.g., retaining the end in a dialogue scenario). Token sequences shorter than 128 are padded with 0 or other fixed character tokens, ensuring that all resulting token sequences are 128 in length. It can be understood that markers can be added to the beginning and end of each token sequence to indicate its start and end. Typically, each marker occupies one token length, so each token sequence contains 126 meaningful characters. After the above mapping, the text is represented as the following text tensor: .

[0049] in, This represents a fixed length, such as 128.

[0050] The following example illustrates the process of mapping characters in text to character identifiers: The mapping of characters to character identifiers can be understood as the process of mapping each smallest granularity unit in the text (such as a single Chinese character) to a unique integer number. For example, assuming that in the vocabulary {"我": 1, "是": 2, "中": 3, "国": 4}, the character "我" corresponds to Token ID = 1, and "中国" corresponds to [3, 4]. The mapping process includes: constructing a vocabulary, establishing a dictionary, and a conversion process.

[0051] During the process of constructing the vocabulary, first count all the characters that appear in the input data, sort them by frequency, add special tokens (such as [BOS] as the start token of the sequence, placed at the beginning of the sequence; [EOS] as the end token, placed at the end of the valid content of the sequence, before the padding token; [PAD] as the padding token, used to pad sequences that are shorter than the fixed length to the fixed length (its value can be 0); [UNK] as the unknown character, to handle characters not registered), and generate the vocabulary.

[0052] When establishing the dictionary, use dict = {"character": ID} to store the mapping relationship (such as dict["我"] = 1).

[0053] During the conversion process, for each character in the input text, obtain the corresponding character identifier, that is, the Token ID, by looking up the established dictionary. Unknown characters can be replaced with the Token ID of [UNK].

[0054] In some optional implementation manners, the encoding module includes an image encoding module and a text encoding module; the encoding module of the multimodal inference model is used to process the text and the image respectively to obtain text features and image features, including: Use the image encoding module to perform encoding processing on the normalized image to obtain second-resolution image features, and / or use the image encoding module to perform encoding processing on the first-resolution image to obtain first-resolution image features; Use the text encoding module to perform encoding processing on the preprocessed text to obtain the text features.

[0055] In this optional implementation manner, the encoding module includes an image encoding module and a text encoding module. The image encoding module is used to encode the input image to obtain corresponding image features, and the text encoding module is used to encode the input text to obtain corresponding text features.

[0056] In order to improve the inference speed of the model on a low-performance C86 architecture CPU platform, the image encoding module can use a lightweight encoding model, such as a MobileViT encoder, and use this image encoding module to encode the input image to obtain image features.

[0057] As described above, this application uses two stages, coarse classification and fine classification, to match images and text in the hierarchical coding module. To further improve the inference speed of the model, the image features corresponding to the first resolution image can be used to filter the candidate set in the coarse classification stage. For this purpose, after normalizing the images in the input data, this application also adjusts their resolution to a first resolution image with a set resolution. After being encoded by the image coding module, the first resolution image features are obtained. These first resolution image features can be used to filter the candidate set in the coarse classification stage of the hierarchical coding module. At the same time, after the original image of the input data is preprocessed by image normalization, it can be encoded by the image coding module to obtain second resolution image features. These second resolution image features can be used to further filter the final matching results from the candidate set in the fine classification stage of the hierarchical coding module.

[0058] In some embodiments, to accelerate the model's inference speed, a lightweight model, such as TinyBERT, is also used in the text encoding module. TinyBERT extracts semantics from the text using a lightweight Transformer architecture to obtain corresponding text features. It should be noted that the text features used in both the coarse and fine classification stages of the hierarchical perception module are features obtained by inputting preprocessed text into the text encoding module.

[0059] In some optional implementations, the text and image are processed separately using the encoding module of a multimodal inference model to obtain text features and image features, including: The image is divided into smaller image blocks; Extract local features from each of the small image patches; Global features of the image are extracted based on the small image patches; The image features are generated based on the local features and the global features.

[0060] In this optional implementation, the image feature extraction process is illustrated below using an image encoding module comprising lightweight convolutional layers, a Transformer module, and either pooling or Flattening layers as an example: The image to be encoded (the normalized original image or the first-resolution image after normalization and resolution adjustment) is divided into small image patches. The specific partitioning method can be set according to actual needs and is not specifically limited here. The divided small image patches are processed by the lightweight convolutional layers of the image encoding module to extract local features for each patch. These local features mainly reflect details such as image edges and textures. The local features extracted from each patch are then fed into the Transformer module of the image encoding module. This Transformer module uses a multi-head self-attention mechanism to model the relationships between the patch images, obtaining the global features of the image to be encoded. These global features reflect the global semantic information of the image. Finally, through pooling or Flattening operations, the local and global features corresponding to all patch images are integrated into a high-dimensional embedding vector. This high-dimensional embedding vector is the image feature corresponding to that image. It should be noted that the encoded image mentioned above can be the pre-processed second-resolution image mentioned above, which is the image used in the fine classification stage of the hierarchical perception module, or it can be the pre-processed first-resolution image that has been adjusted to a set resolution, which is the image used in the coarse classification stage of the hierarchical perception module.

[0061] In some optional implementations, the text and image are processed separately using the encoding module of a multimodal inference model to obtain text features and image features, including: Map the text to an embedding vector; Extract the contextual features at each position of the embedding vector; Based on the contextual features, extract the text features corresponding to the text.

[0062] In this optional implementation, the text encoding module can employ a lightweight Transformer model, such as TinyBERT. In some embodiments, the text encoding module includes a word embedding module, a multi-layer Transformer module, and a prediction layer. The multi-layer Transformer module has fewer than 5 layers, and the prediction layer can be a fully connected layer. After the input text data is mapped into a fixed-length sequence of character identifiers in the word embedding module, each character identifier (Token ID) in the character identifier sequence is converted into a continuous vector representation through an embedding matrix, thus forming the input sequence of the text encoding module. This process can be referred to the text preprocessing process mentioned above, and the input sequence can be a fixed-length preprocessed text. The embedding vector is represented as follows: .

[0063] Where V is the vocabulary size and d is the dimension of the embedding vector.

[0064] The input sequence is fed into the multi-layer Transformer module of the text encoding module for processing. The multi-layer Transformer module uses a self-attention mechanism to capture the contextual dependencies between characters in the text and generate contextual features at each position in the input sequence. Then, the prediction layer extracts global semantic vectors from these contextual features to obtain text features. In this process, the semantics of the entire sentence can be represented by setting a marker such as [CLS] to mark the position of the global semantic vector. This marker is set in the input sequence at the position corresponding to the beginning of the sentence in the text.

[0065] In some optional implementations, the hierarchical perception module includes a coarse classification module and a fine classification module; the image features include second-resolution image features corresponding to the image and first-resolution image features corresponding to the first-resolution image; the first-resolution image is obtained by adjusting the resolution of the image; the resolution of the first-resolution image is lower than the resolution of the image. Using the hierarchical perception module of the multimodal inference model, a candidate set is obtained from the text and image based on the text fusion features and the image fusion features, and then a matching result is obtained from the candidate set through fine classification, including: Using the coarse classification module, the candidate set is obtained from the text and image coarse classification based on the text fusion features and the first resolution image features; Using the fine classification module, the matching result is obtained from the candidate set by fine classification based on the text fusion features and the second resolution image features.

[0066] In this optional implementation, the hierarchical perception module includes a coarse classification module for the coarse classification stage and a fine classification module for the fine classification stage. The image features include second-resolution image features and first-resolution image features. As described above, the second-resolution image features can be obtained by normalizing the original image of the input data, inputting it into the image encoding module, and then performing vector fusion with the text features. The first-resolution module obtains the image features by normalizing the original image, adjusting it to a set resolution, inputting the first-resolution image into the image encoding module, and then performing vector fusion with the text features. The set resolution is smaller than the resolution of all images in the input data.

[0067] The coarse classification module uses a lightweight convolutional neural network model, such as MobileNet or ResNet50, to quickly generate low-precision embedding vectors corresponding to the first-resolution image features and text fusion features. It also uses cosine similarity calculation to measure the semantic relevance between the image and the text, which can quickly filter out the top N candidate texts or candidate images to add to the candidate set.

[0068] The fine-grained classification module performs a refined match between the candidate set output by the coarse-grained classification module and the original text or image. Therefore, the input to the fine-grained classification module consists of second-resolution image features and text fusion features. Furthermore, the fine-grained classification module can use high-precision models such as MobileViT to correct errors in the candidate set, thereby obtaining the final matching result. For the processing flow related to the hierarchical perception module, please refer to [link to relevant documentation]. Figure 2 As shown.

[0069] It should be noted that the coarse classification module and the fine classification module are trained through contrastive learning. Contrastive learning is an optimization process following vector fusion, used to maximize the similarity of positive sample pairs (image-text pairs) in the shared space, while minimizing the similarity of negative sample pairs (mismatched image-text pairs). The loss function E for the contrastive learning process of the coarse and fine classification modules can be expressed as follows: .

[0070] in, This represents the number of samples in a batch during training, that is, the number of image-text pairs. and Both represent the cosine similarity between the image and the text. These are the image fusion features resulting from vector fusion. These are text fusion features resulting from vector fusion. This is a sensitivity parameter used to control the distribution of relative similarity, called the temperature parameter. Cosine similarity measures the directional similarity between two vectors, and the formula is as follows: .

[0071] During training, the weights of the multimodal inference model can be adjusted using the loss function described above, so that matched image-text pairs are closer in the shared space, while unmatched pairs are farther apart.

[0072] In some alternative implementations, the multimodal inference model employs a sparse activation mechanism during training.

[0073] In this optional implementation, the training of the multimodal inference model draws on the sparse activation characteristics of biological neurons to avoid activating all computational paths, thereby reducing computational cost and resource waste. Therefore, this sparse activation mechanism improves the sparsity of the entire model by pruning redundant weights. The pruning function is expressed as follows: .

[0074] Where λ is the regularization coefficient, controlling the sparsity. T is the optimization objective function used during the training of the multimodal inference model, also known as the pruning function; The objective function is the loss function for all model weights used in training the multimodal inference model; W is the weight matrix of the multimodal inference model, which includes the model weights of the image encoding module, text encoding module, and hierarchical perception module. This objective function can penalize non-zero weights to promote sparsity in the model weights. During training, regularization can be used to improve model sparsity, allowing some weights to be safely pruned (i.e., zeroed) during inference, reducing computational cost (such as invalid calculations in matrix multiplication), thereby ensuring that the low-precision embedding vectors in the coarse classification stage contain key semantics, providing a more effective candidate set for fine classification. The above pruning formula achieves model lightweighting, optimizes task performance, and promotes model sparsity, reducing computational cost while maintaining accuracy.

[0075] The pruning function operates on all trainable model weights in a multimodal inference model. It induces sparsity in the model weights, removes redundant parameters, reduces computational complexity, and improves inference efficiency. During the sparsity process, non-zero model weights are penalized with λ||W||1, forcing some model weights to become sparser, i.e., close to zero, thus reducing computational complexity. Weights with absolute values ​​less than a threshold are removed, reducing the number of model parameters.

[0076] In some alternative implementations, the multimodal inference model is an ONNX format model and is loaded onto an inference engine with the ZenDNN library deployed.

[0077] In this optional implementation, to better adapt to CPU inference, this application introduces the ONNX Runtime inference engine to load the multimodal inference model. Simultaneously, the model content corresponding to the multimodal inference model (such as weight parameters, model structure data, etc.) is converted to ONNX format and deployed on the ONNX Runtime inference engine. Since the C86 architecture CPU is designed based on the AMD Zen architecture, to balance inference efficiency and accuracy during the inference phase, the ZenDNN library can also be integrated into the ONNX Runtime inference engine. This allows the CPU's SIMD instruction set to better optimize computationally intensive tasks such as matrix multiplication and convolution. The performance improvement of the ZenDNN library during inference can be expressed by the following formula: .

[0078] in, This is the total inference time (in seconds) after using the ZenDNN library. This represents the total number of computational operations for the baseline model that does not use the ZenDNN library. The formula represents the computational throughput per unit time (in Ops / s) after using the ZenDNN library, reflecting the optimized efficiency of the ZenDNN library for computationally intensive tasks during the inference process.

[0079] In addition, during the model quantization stage, which involves converting floating-point model weights (such as FP32) to low-precision integers (such as INT8), the model weights can be compressed from FP32 to INT8 for inference, thereby reducing the computational complexity and memory usage of the model. The change in complexity can be expressed by the formula: .

[0080] in, The complexity of inference after the model weights are compressed to INT8. This represents the complexity of inference without model weight compression. Simultaneously, accuracy calibration ensures that the loss of accuracy remains within an acceptable range; the calibration formula is: .

[0081] in, These are model weights. It is the bit width of the quantization.

[0082] Accordingly, see Figure 3 As shown, this application also provides a multimodal reasoning device, including: The acquisition module 301 is configured to acquire input data; the input data includes text and images. Extraction module 302 is configured to process the text and image respectively using the encoding module of the multimodal reasoning model to obtain text features and image features; The fusion module 303 is configured to perform vector fusion of the text features and the image features to obtain text fusion features and image fusion features; The reasoning module 304 is configured to use the hierarchical perception module of the multimodal reasoning model to obtain a candidate set from the text and image based on the text fusion features and the image fusion features, and then obtain a matching result from the candidate set for fine classification.

[0083] In some alternative embodiments, the apparatus further includes: The normalization module is configured to normalize the image to obtain a normalized image; The adjustment module is configured to adjust the normalized image to a first resolution image with a set resolution. The segmentation module is configured to map the text into a sequence of character identifiers and then segment it into a set of preprocessed texts of fixed length.

[0084] In some optional embodiments, the encoding module includes an image encoding module and a text encoding module; the extraction module includes: The image feature acquisition submodule is configured to encode the normalized image using the image encoding module to obtain second resolution image features, and / or to encode the first resolution image using the image encoding module to obtain first resolution image features. The text feature acquisition submodule is configured to encode the text using the text encoding module to obtain the text features.

[0085] In some optional embodiments, the extraction module includes: A slicing module is configured to slice the image into smaller image blocks; The local extraction submodule is configured to extract local features of the small image patches respectively; The global extraction submodule is configured to extract global features of the image based on the small image patch; The feature generation submodule is configured to generate the image features based on the local features and the global features.

[0086] In some optional embodiments, the extraction module includes: The mapping submodule is configured to map text to embedding vectors; The context extraction submodule is configured to extract context features at each position of the embedding vector; The text feature extraction submodule is configured to extract text features corresponding to the preprocessed text based on the context features.

[0087] In some optional embodiments, the hierarchical perception module includes a coarse classification module and a fine classification module; the image features include second-resolution image features and first-resolution image features; The reasoning module includes: The first filtering submodule is configured to use the coarse classification module to obtain the candidate set from the text and image coarse classification based on the text fusion features and the first resolution image features; The second filtering submodule is configured to use the fine classification module to obtain the matching result from the candidate set by fine classification based on the text fusion features and the second resolution image features.

[0088] In some optional embodiments, the multimodal inference model employs a sparse activation mechanism during training.

[0089] In some optional embodiments, the multimodal inference model is an ONNX format model and is loaded onto an inference engine that integrates the ZenDNN library.

[0090] The aforementioned multimodal reasoning device corresponds to the multimodal reasoning method described above. For specific details, please refer to the description of the multimodal reasoning device above, which will not be repeated here.

[0091] It is understood that the circuit structures, names, and parameters described in the above embodiments are merely examples. Those skilled in the art can also make readily conceived combinations and adjustments to the structural features of the above embodiments according to their needs, and the concept of this application should not be limited to the specific details of the above examples.

[0092] This application also provides an electronic device, including a processor and a memory, wherein the memory stores multiple instructions, and the processor is used to read the instructions and execute any of the methods described in the first aspect above. The processor and memory can be connected via a bus or other means, taking a bus connection as an example. The processor can be a Central Processing Unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above-mentioned types of chips.

[0093] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the methods in the embodiments of this application. The processor executes various functional applications and data processing by running the non-transitory software programs, instructions, and modules stored in the memory, thereby implementing the methods in the above-described method embodiments.

[0094] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor, etc. Furthermore, the memory may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, which can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0095] In another aspect, this disclosure also provides a computer program product comprising computer instructions that can be loaded and executed by the processor to enable the processor to perform the methods described above.

[0096] Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A multimodal reasoning method, characterized in that, include: Obtain input data; the input data includes text and images; The text and image are processed separately using the encoding module of the multimodal reasoning model to obtain text features and image features; The text features and the image features are fused by vectorization to obtain text fusion features and image fusion features; Using the hierarchical perception module of the multimodal reasoning model, a candidate set is obtained from the text and image based on the text fusion features and the image fusion features, and then a matching result is obtained from the candidate set through fine classification.

2. The multimodal reasoning method according to claim 1, characterized in that, The method further includes: The image is then normalized to obtain a normalized image; Adjust the normalized image to the first resolution image with the set resolution; After mapping the text to a character identifier sequence, it is divided into a set of preprocessed texts of fixed length.

3. The multimodal reasoning method according to claim 2, characterized in that, The encoding module includes an image encoding module and a text encoding module; the encoding module uses a multimodal inference model to process the text and image respectively, obtaining text features and image features, including: The image encoding module is used to encode the first resolution image to obtain the first resolution image features, and / or the image encoding module is used to encode the normalized image to obtain the second resolution image features; The preprocessed text is encoded using the text encoding module to obtain the text features.

4. The multimodal reasoning method according to claim 1, characterized in that, The text and image are processed separately using the encoding module of a multimodal inference model to obtain text features and image features, including: The image is divided into smaller image blocks; Extract local features from each of the small image patches; Global features of the image are extracted based on the small image patches; The image features are generated based on the local features and the global features.

5. The multimodal reasoning method according to claim 1, characterized in that, The text and image are processed separately using the encoding module of a multimodal inference model to obtain text features and image features, including: Map the text to an embedding vector; Extract the contextual features at each position of the embedding vector; Based on the contextual features, extract the text features corresponding to the text.

6. The multimodal reasoning method according to claim 3, characterized in that, The hierarchical perception module includes a coarse classification module and a fine classification module; the image features include second-resolution image features and first-resolution image features. Using the hierarchical perception module of the multimodal inference model, a candidate set is obtained from the text and image based on the text fusion features and the image fusion features, and then a matching result is obtained from the candidate set through fine classification, including: Using the coarse classification module, the candidate set is obtained from the text and image coarse classification based on the text fusion features and the first resolution image features; Using the fine classification module, the matching result is obtained from the candidate set by fine classification based on the text fusion features and the second resolution image features.

7. The multimodal reasoning method according to any one of claims 1-3, characterized in that, The multimodal inference model employs a sparse activation mechanism during training.

8. The multimodal reasoning method according to any one of claims 1-3, characterized in that, The multimodal inference model is an ONNX format model and is loaded onto an inference engine that integrates the ZenDNN library.

9. A multimodal reasoning device, characterized in that, include: The acquisition module is configured to acquire input data, which includes text and images. The extraction module is configured to process the text and image respectively using the encoding module of the multimodal reasoning model to obtain text features and image features; The fusion module is configured to perform vector fusion of the text features and the image features to obtain text fusion features and image fusion features; The inference module is configured to utilize the hierarchical perception module of the multimodal inference model to obtain a candidate set from the text and image based on the text fusion features and the image fusion features, and then obtain a matching result from the candidate set through fine classification.

10. An electronic device, characterized in that, It includes a processor and a memory, the memory storing multiple instructions, and the processor being used to read the instructions and execute the multimodal reasoning method as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions, which can be read by a processor and executed as described in any one of claims 1-8.

12. A computer program product, characterized in that, It includes computer instructions, wherein the computer instructions, when executed by a processor, implement the multimodal reasoning method as described in any one of claims 1-8.

Images