Multi-modal representation construction method and electronic equipment

Abstract

The invention provides a multi-modal representation construction method and electronic equipment, and relates to the technical field of computers and Internet. The method comprises the following steps: acquiring an image-text representation teacher model, wherein the image-text representation teacher model comprises a teacher image tower and a teacher text tower; obtaining an image-text representation student model, wherein the image-text representation student model comprises a student image tower and a student text tower; obtaining a first data set containing a plurality of training samples; based on the first data set, determining image-text comparison learning loss corresponding to the image-text representation student model, image-text comparison learning loss, and distillation loss between the image-text representation student model and the image-text representation teacher model; and freezing a student text tower, and training a student image tower of the image-text representation student model based on image-text contrast learning loss, graph contrast learning loss and distillation loss. According to the embodiment of the invention, high-performance and high-universality multi-modal semantic understanding can be realized under extremely low parameter quantity and calculation complexity.

Description

Technical Field

[0001] This disclosure relates to the field of computer and Internet technology, and in particular to a method for constructing multimodal representations and an electronic device. Background Technology

[0002] This section is intended to provide background or context for the embodiments of this disclosure as set forth in the claims. The description herein is not intended to be a prior art simply because it is included in this section.

[0003] In the field of multimodal representation, existing technologies generally suffer from a large number of model parameters and high computational resource consumption, making it difficult to adapt to the deployment requirements of mobile terminals such as smartphones.

[0004] Therefore, there is an urgent need to develop an efficient characterization method that can meet the requirements of lightweight edge deployment. Summary of the Invention

[0005] The purpose of this disclosure is to provide a method, apparatus, electronic device, computer-readable storage medium, and computer program product for constructing multimodal representations, which can achieve high-performance and versatile multimodal (text-image) semantic understanding with extremely low parameter count and computational complexity.

[0006] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0007] This disclosure provides a method for constructing multimodal representations, comprising: obtaining a teacher model for image and text representations, the teacher model including a teacher image tower and a teacher text tower, the teacher model being used to map image information and text information to the same vector space to achieve representation alignment; obtaining a student model for image and text representations, the student model including a student image tower and a student text tower, the student text tower inheriting the structure and parameters of the teacher text tower; the student image tower being obtained by model compression of the teacher image tower; obtaining a first dataset containing multiple training samples, wherein the training samples include original images, enhanced images of the original images, and text information describing the original images; based on the first dataset, determining the image-text contrast learning loss, the image-image contrast learning loss, and the distillation loss between the student model and the teacher model for image and text representations corresponding to the student model for image and text representations; freezing the student text tower, and training the student image tower of the student model for image and text representations based on the image-text contrast learning loss, the image-image contrast learning loss, and the distillation loss.

[0008] In some embodiments, the method further includes: acquiring a second dataset; determining the image-text contrast loss corresponding to the image-text representation teacher model based on the second dataset; freezing the teacher image tower and training the teacher text tower based on the image-text contrast loss corresponding to the image-text representation teacher model, so that the image-text representation teacher model can align the representations of image information and text information in the same vector space.

[0009] In some embodiments, the teacher image tower is a visual encoder that supports image embedding representation, and the teacher text tower is determined according to a text encoder that supports multilingual text embedding representation, wherein the teacher image tower and teacher text tower before training cannot represent information in the same vector space.

[0010] In some embodiments, the method further includes adding a projection layer after the text encoding model to obtain the teacher text tower, the projection layer being used for data dimensionality reduction processing.

[0011] In some embodiments, the method further includes: performing text enhancement processing on the original image and the original text information corresponding to the original image using a large language model to obtain enhanced text information describing the original image; and determining the text information describing the original image based on the enhanced text.

[0012] In some embodiments, determining the image-text contrastive learning loss corresponding to the image-text representation student model based on the first dataset includes: embedding representations of the original images in the first dataset using the student image tower to obtain multiple first image features; embedding representations of text information in the first dataset using the student text tower to obtain multiple first text features; determining the pairwise similarity between all first image features and all first text features to generate an image-text similarity matrix, wherein the diagonal elements in the image-text similarity matrix are the similarity of positive sample pairs, and the off-diagonal elements are the similarity of negative sample pairs; determining the contrastive loss from each image to text and the contrastive loss from each text to image based on the image-text similarity matrix; and determining the image-text contrastive learning loss based on the contrastive losses from each image to text and the contrastive losses from each text to image.

[0013] In some embodiments, determining the image-to-image contrast learning loss corresponding to the image-text representation student model based on the first dataset includes: embedding representations of the original images in the first dataset using the student image tower to obtain multiple second image features; embedding representations of the augmented images in the first dataset using the student image tower to obtain multiple third image features; determining the pairwise similarity between all second image features and all third image features to generate a first image-to-image similarity matrix, wherein the diagonal elements of the first image-to-image similarity matrix are the similarity of positive sample pairs, and the off-diagonal elements are the similarity of negative sample pairs; determining the contrast loss from each original image to the augmented image and the contrast loss from each augmented image to the original image based on the first image-to-image similarity matrix; and determining the image-to-image contrast learning loss based on the contrast loss from each original image to the augmented image and the contrast loss from each augmented image to the original image.

[0014] In some embodiments, determining the distillation loss between the image-text representation student model and the image-text representation teacher model based on the first dataset includes: performing feature representation on the original images in the first dataset using the student image tower in the image-text representation student model to obtain a fourth image feature; performing feature representation on the original images in the first dataset using the teacher image tower in the image-text representation teacher model to obtain a fifth image feature; determining the similarity distance loss between the fourth image feature and the fifth image feature in the same training sample; and determining the distillation loss based on the similarity distance loss corresponding to each training sample.

[0015] In some embodiments, the method further includes: reducing the network depth and / or feature width of the teacher image tower; and replacing the computationally intensive self-attention computation units in the teacher image tower with lightweight computation units to obtain the student image tower.

[0016] This disclosure provides a multimodal representation construction apparatus, including: a teacher model acquisition module, a first dataset acquisition module, a loss determination module, and a training module.

[0017] The teacher model acquisition module is used to acquire a text-image representation teacher model, which includes a teacher image tower and a teacher text tower. This model maps image and text information to the same vector space to achieve representation alignment. The teacher model acquisition module can also acquire a text-image representation student model, which includes a student image tower and a student text tower. The student text tower inherits the structure and parameters of the teacher text tower. The student image tower is obtained by model compression of the teacher image tower. The first dataset acquisition module can acquire a first dataset containing multiple training samples, including original images, enhanced images of the original images, and text information describing the original images. The loss determination module can determine, based on the first dataset, the text-image contrast learning loss, the image-image contrast learning loss, and the distillation loss between the text-image representation student model and the text-image representation teacher model. The training module can freeze the student text tower and train the student image tower of the text-image representation student model based on the text-image contrast learning loss, the image-image contrast learning loss, and the distillation loss.

[0018] This disclosure provides an electronic device comprising: a memory and a processor; the memory for storing computer program instructions; and the processor for calling the computer program instructions stored in the memory to implement the multimodal representation construction method described above.

[0019] This disclosure provides a computer-readable storage medium storing computer program instructions to implement the method for constructing multimodal representations as described in any of the preceding embodiments.

[0020] This disclosure provides a computer program product or computer program that includes computer program instructions stored in a computer-readable storage medium. The computer program instructions are read from the computer-readable storage medium, and a processor executes the computer program instructions to implement the aforementioned method for constructing multimodal representations.

[0021] The multimodal representation construction method, apparatus, electronic device, computer-readable storage medium, and computer program product provided in this disclosure introduce a multi-task joint optimization and knowledge distillation mechanism. Under the premise of freezing the student text tower, the student image tower is trained in a coordinated manner using image-text comparison loss, image-image comparison loss, and distillation loss. Ultimately, while significantly compressing the number of model parameters, a general lightweight multimodal representation model with strong cross-modal semantic alignment capability and high-quality image understanding capability is constructed.

[0022] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this disclosure. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0024] Figure 1 A schematic diagram of a scenario is shown that can be applied to a method or apparatus for constructing a multimodal representation in accordance with embodiments of this disclosure.

[0025] Figure 2 This is a flowchart illustrating a method for constructing a multimodal representation according to an exemplary embodiment.

[0026] Figure 3 This is a schematic diagram illustrating a training dataset according to an exemplary embodiment.

[0027] Figure 4 This is an architecture diagram illustrating a method for constructing a multimodal representation according to an exemplary embodiment.

[0028] Figure 5 This is a flowchart illustrating a method for constructing a multimodal representation according to an exemplary embodiment.

[0029] Figure 6 This is a flowchart illustrating a method for determining the learning loss of image-text contrastive learning according to an exemplary embodiment.

[0030] Figure 7 This is a flowchart illustrating a graph-to-graph contrastive learning loss determination method according to an exemplary embodiment.

[0031] Figure 8 This is a flowchart illustrating a method for determining distillation loss according to an exemplary embodiment.

[0032] Figure 9A This is a model evaluation method illustrated according to an exemplary embodiment.

[0033] Figure 9B This is a model evaluation result illustrated according to an exemplary embodiment.

[0034] Figure 10 This is a block diagram illustrating a multimodal characterization construction apparatus according to an exemplary embodiment.

[0035] Figure 11A schematic diagram of the structure of an electronic device suitable for implementing embodiments of the present disclosure is shown. Detailed Implementation

[0036] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0037] Those skilled in the art will recognize that embodiments of this disclosure can be a system, apparatus, device, method, or computer program product. Therefore, this disclosure can be implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0038] The features, structures, or characteristics described in this disclosure can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more specific details omitted, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0039] In this disclosure, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0040] The accompanying drawings are merely illustrative of this disclosure, and the same reference numerals in the drawings denote the same or similar parts, thus omitting repeated descriptions of them. Some block diagrams shown in the drawings do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0041] The flowchart shown in the accompanying drawings is merely illustrative and does not necessarily include all content and steps, nor does it require execution in the described order. For example, some steps may be broken down, while others may be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0042] In the description of this disclosure, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. "And / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, "at least one" means one or more, and "multiple" means two or more. The terms "first," "second," etc., do not limit the quantity or order of execution, and "first," "second," etc., do not necessarily imply differences; the terms "contains," "includes," and "has" are used to indicate an open-ended meaning of inclusion and refer to the existence of additional elements / components / etc. besides those listed.

[0043] This disclosure embodiment can be implemented by a terminal and / or a server. The terminal can obtain data from a computer device and display that data. The computer device can interact with the terminal, and can be a server hosting the application, or it can belong to the terminal (i.e., the terminal's backend), etc., without limitation.

[0044] The terminal can be a mobile phone, a laptop computer, or a playback device in a vehicle, etc., without limitation. The terminal can be considered a playback device in a vehicle, and it can display the target application. The terminal is only one example of the devices listed; the terminal in this disclosure is not limited to the listed devices. The target application in this disclosure can be any application capable of displaying multimedia information.

[0045] It is understood that the terminal mentioned in the embodiments of this disclosure can be a computer device, including but not limited to a terminal or a server. In other words, the computer device can be a server or a terminal, or a system composed of a server and a terminal. The terminal mentioned above can be an electronic device, including but not limited to mobile phones, tablets, desktop computers, laptops, handheld computers, in-vehicle devices, augmented reality / virtual reality (AR / VR) devices, head-mounted displays, smart TVs, wearable devices, smart speakers, digital cameras, webcams, and other mobile internet devices (MIDs) with network access capabilities, or terminals in scenarios such as trains, ships, and flights.

[0046] The servers mentioned above can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, vehicle-road cooperation, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0047] Optionally, the data involved in the embodiments of this disclosure may be stored in a computer device or may be stored based on cloud storage technology, without limitation.

[0048] To better understand the above-mentioned objectives, features and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0049] The exemplary embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0050] Figure 1 A schematic diagram of a scenario is shown that can be applied to a method or apparatus for constructing a multimodal representation in accordance with embodiments of this disclosure.

[0051] Please refer to Figure 1 The diagram illustrates an implementation environment provided by an exemplary embodiment of this disclosure.

[0052] like Figure 1As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0053] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, desktop computers, wearable devices, virtual reality devices, smart home devices, etc.

[0054] Server 105 can be a server that provides various services, such as a backend management server that supports the devices operated by users using terminal devices 101, 102, and 103. The backend management server can analyze and process received requests and other data, and feed the processing results back to the terminal devices.

[0055] A server can be a standalone physical server, a server cluster or a distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. This disclosure does not impose any restrictions on this.

[0056] Server 105 may, for example, acquire a teacher model for image and text representation, which includes a teacher image tower and a teacher text tower. The teacher model is used to map image information and text information to the same vector space to achieve representation alignment. Server 105 may, for example, acquire a student model for image and text representation, which includes a student image tower and a student text tower. The student text tower inherits the structure and parameters of the teacher text tower. The student image tower is obtained by model compression of the teacher image tower. Server 105 may, for example, acquire a first dataset containing multiple training samples, where the training samples include the original image, the enhanced image of the original image, and the text information describing the original image. Server 105 may, for example, determine the image-text contrast learning loss, the image-image contrast learning loss, and the distillation loss between the student model and the teacher model for image and text representation based on the first dataset. Server 105 may, for example, freeze the student text tower and train the student image tower of the student model for image and text representation based on the image-text contrast learning loss, the image-image contrast learning loss, and the distillation loss.

[0057] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Server 105 can be a single physical server or a combination of multiple servers. Depending on actual needs, it can have any number of terminal devices, networks, and servers.

[0058] Under the above system architecture, this disclosure provides a method for constructing multimodal representations, which can be executed by any electronic device with computing capabilities.

[0059] Figure 2 This is a flowchart illustrating a method for constructing a multimodal representation according to an exemplary embodiment. The method provided in this disclosure can be executed by any electronic device with computing power, for example, the method can be implemented by the above-described... Figure 1 The execution can be performed by a server or terminal device in the embodiments, or it can be performed by both a server and a terminal device. In the following embodiments, the server is used as the execution subject for illustration, but this disclosure is not limited to this.

[0060] Reference Figure 2 The method for constructing multimodal representations provided in this disclosure may include the following steps.

[0061] Step S202: Obtain the image-text representation teacher model, which includes a teacher image tower and a teacher text tower. The image-text representation teacher model is used to map image information and text information to the same vector space to achieve representation alignment.

[0062] The image-text representation teacher model can be a pre-trained, parameter-fixed dual-tower neural network model, serving as the knowledge provider in knowledge distillation. It consists of a "teacher image tower" specifically for processing images and a "teacher text tower" specifically for processing text, operating in parallel. Its core capability is to map inputs from different modalities (images and text) into a unified, semantically consistent high-dimensional vector space, thereby providing high-quality learning objectives and supervision signals for training a more lightweight "student model."

[0063] In some embodiments, semantically related images and texts, after being processed by the image and text representation teacher model, have representation vectors that are close in distance in the vector space.

[0064] In some embodiments, the teacher image tower can be a deep learning model trained on large-scale data with strong representational capabilities, whose function is to convert the input image into a fixed-dimensional numerical vector (also known as an "embedding" or "feature") rich in semantic information.

[0065] The teacher image tower can be a dedicated subnetwork in the teacher image representation model responsible for processing visual information. It is a deep neural network (usually a convolutional neural network or a visual Transformer) whose function is to encode the input image pixel data into a low-dimensional, dense floating-point vector (i.e., image representation) through multiple layers of nonlinear transformations. This vector can effectively capture the high-level semantic content of the image.

[0066] The teacher text pyramid can be a dedicated subnetwork in a graph-text representation teacher model responsible for processing textual information. It is a deep neural network (usually a Transformer-based language model) that encodes the input sequence of text symbols (such as English words or Chinese characters) into a low-dimensional, dense floating-point vector (i.e., text representation) with the same dimension as the image representation vector through embedding layers and a series of attention mechanisms. This vector can effectively capture the semantic information of the text.

[0067] In some embodiments, the teacher text tower can support multiple languages, such as Chinese and English.

[0068] In summary, the image-text representation teacher model utilizes its internal teacher image tower and teacher text tower to map image information and text information into the same vector space, respectively. Through optimization, it makes semantically related image and text vectors close to each other, thereby achieving representation alignment.

[0069] In some embodiments, the MobileCLIP-S0 (contrastive language-image pre-trained) image representation model MCI0 model can be used as the image representation model structure (such as the teacher image tower), and the jina-embeddings-v2-base-zh (Chinese optimized version of the second-generation basic-scale text embedding model) can be used as the pre-trained text representation model structure (such as the teacher text tower or the student text tower).

[0070] MobileCLIP-S0 is a deeply optimized, lightweight, multimodal AI model. Its core function is to simultaneously understand images and text, mapping them to the same semantic space for comparison. However, MobileCLIP-S0 is primarily optimized for understanding and optimizing English text. Unfortunately, MobileCLIP-S0's native understanding capabilities are very limited and unreliable.

[0071] MCi0 is an image encoder that supports image representation in Mobile CLIP-S0.

[0072] In some embodiments, jina-embeddings-v2-base-zh is a bilingual Chinese-English text embedding model that supports sequence lengths up to 8192. It is based on the BERT (Bidirectional Encoder Representation from Transformer) architecture (JinaBERT), which supports a symmetric bidirectional variant of ALiBi (a novel positional encoding method for Transformer models) to allow for longer sequence lengths. The model is specifically trained to support mixed Chinese and English inputs without bias. It has 161M parameters, supports both Chinese and English, and has an original output feature dimension of 768.

[0073] In some embodiments, a projection layer can be added after the text encoding model to obtain the teacher text pyramid. The projection layer is used for data dimensionality reduction. The text encoding layer can be the aforementioned jina-embeddings-v2-base-zh model.

[0074] Specifically, an additional layer of dimension mapping can be added to this jina-embeddings-v2-base-zh to generate the aforementioned teacher text pyramid. This dimension mapping can be used for dimensionality reduction.

[0075] Step S204: Obtain the student model of the graphic and text representation. The student model of the graphic and text representation includes the student image tower and the student text tower. The student text tower inherits the structure and parameters of the teacher text tower. The student image tower is obtained by model compression of the teacher image tower.

[0076] In some embodiments, the student image tower may be obtained by structurally simplifying the teacher image tower.

[0077] In some embodiments, the teacher image tower can be a visual encoder that supports image embedding representations, and the teacher text tower can be determined based on a text encoder that supports multilingual text embedding representations, wherein the feature vectors output by the teacher image tower and the teacher text tower before training do not have cross-modal semantic alignment capabilities.

[0078] In some embodiments, obtaining the student image tower may include the following steps: adjusting the network structure of the teacher image tower to reduce its computational complexity and / or number of parameters to obtain the student image tower; wherein the adjustment includes at least one of the following: reducing network depth, reducing feature dimensions, and replacing computationally intensive modules with lightweight modules.

[0079] In some embodiments, the teacher image tower can be compressed to obtain the student image tower by: reducing the network depth and / or feature width of the teacher image tower; and replacing the computationally intensive self-attention computation units in the teacher image tower with lightweight computation units to obtain the student image tower.

[0080] Specifically, student image towers can be obtained in the following ways.

[0081] In some embodiments, the image tower in the teacher model can be the image representation MCI0 of MobileCLIP-S0 with 11M parameters, and the image tower in the student model can be an improved version of MCI-tiny with 2M parameters.

[0082] MCI0 can be configured as follows: layers = [2, 6, 10, 2] (a total of 20 blocks); embed_dims = [64, 128, 256, 512]; token_mixers = ("repmixer", "repmixer", "repmixer", "attention") (the last one is Attention); mlp_ratios = [3, 3, 3, 3]; act_layer = nn.GELU.

[0083] MCi-Tiny can be configured as follows: layers = [2, 4, 6, 2] (reducing the depth, a total of 14 blocks, from 20 to 14); embed_dims = [32, 64, 128, 256] (width halved); token_mixers = ("repmixer", "repmixer", "repmixer", "repmixer"); mlp_ratios = [3, 3, 3, 3]; act_layer = nn.ReLU (global replacement activation function); The following section will explain the above configurations one by one.

[0084] # MCi0 (Teacher Model) Configuration.

[0085] # This is a high-performance but computationally intensive benchmark model, serving as a "teacher" in knowledge distillation.

[0086] # layers: Defines the number of blocks in each of the four stages of a neural network.

[0087] # [2, 6, 10, 2] indicates that there are a total of 20 blocks, the model is deep and has strong representation ability.

[0088] layers = [2, 6, 10, 2] # (Total 20 blocks).

[0089] # embed_dims: Defines the number of channels in the output feature maps of the four stages, i.e., the “width” of the model.

[0090] # [64, 128, 256, 512] indicates that as the network deepens, the number of channels in the feature map doubles in stages, enabling the capture of richer visual features.

[0091] embed_dims = [64, 128, 256, 512].

[0092] # token_mixers: Defines the type of core information fusion operator for each of the four phases.

[0093] The first three stages use the lightweight "RepMixer", while the final stage uses the computationally expensive "Attention".

[0094] This combination strikes a balance between efficiency and the ability to integrate global information.

[0095] token_mixers = ("repmixer", "repmixer", "repmixer", "attention") #(The last one is Attention).

[0096] # mlp_ratios: Defines the expansion ratio of the intermediate layers of the multilayer perceptron (MLP) in each block.

[0097] # [3, 3, 3, 3] indicates that the dimension of the intermediate layer of the MLP is 3 times the dimension of the input layer, and this is consistent across the entire model.

[0098] mlp_ratios = [3, 3, 3, 3].

[0099] # act_layer: Defines the type of activation function used in the network.

[0100] # nn.GELU is a high-performance but computationally complex activation function.

[0101] act_layer = nn.GELU.

[0102] # MCi-Tiny (Student Model) Configuration.

[0103] # This is a lightweight model with extreme compression of MCi0, designed for deployment on the edge.

[0104] # layers: Significantly reduces the number of layers in the intermediate stages, making the model "shallow".

[0105] # [2, 4, 6, 2] indicates that the total number of layers is reduced from 20 to 14, which significantly reduces the computational load and inference latency.

[0106] layers = [2, 4, 6, 2] # (Reduce depth, total 14 blocks, from 20 to 14).

[0107] # embed_dims: Halve the number of channels in all stages to make the model "lean".

[0108] # [32, 64, 128, 256] This is one of the most effective ways to reduce the number of parameters, but it will sacrifice some of the model's expressive power.

[0109] embed_dims = [32, 64, 128, 256] # (width halved).

[0110] # token_mixers: Replaces the expensive Attention computation in the final stage with a unified, lightweight RepMixer.

[0111] Using "RepMixer" throughout sacrifices some global modeling capabilities, but in return, it greatly improves inference speed, which is crucial for edge deployment.

[0112] token_mixers = ("repmixer", "repmixer", "repmixer", "repmixer").

[0113] # mlp_ratios: The expansion ratios that maintain consistency with the teacher model.

[0114] mlp_ratios = [3, 3, 3, 3].

[0115] # act_layer: Globally replaces GELU with ReLU, which is easier to compute.

[0116] # nn.ReLU is one of the fastest activation functions, and this replacement is for extreme inference speed optimization.

[0117] act_layer = nn.ReLU # (Global replacement activation function).

[0118] Step S206: Obtain a first dataset containing multiple training samples, wherein the training samples include the original image, the enhanced image of the original image, and text information describing the original image.

[0119] In some embodiments, initial training samples may be obtained from some open-source data or on-site data of an application. These initial training samples may include the original image and / or the original text describing the original image.

[0120] The original text mentioned above can be in Chinese or English. This original text can be relatively concise.

[0121] Based on this, a large language model can be used to perform text enhancement processing on the original image and its corresponding original text information to obtain enhanced text information describing the original image; based on the enhanced text, the text information describing the original image can be determined. For example, text enhancement processing can be performed on the original text information to obtain a bilingual description in Chinese and English, or to obtain more Chinese or English descriptions, etc., and this application does not impose any limitations on this.

[0122] Specifically, if the description text for an image is a short original text, it can be expanded into both Chinese and English using a large language model.

[0123] In some embodiments, the text information describing the original image may include Chinese text information and / or English text information. Through the above method, the teacher's text tower or the student's text tower can acquire bilingual capabilities (or have enhanced bilingual capabilities).

[0124] In some embodiments, image enhancement processing (such as randomly overlaying stickers, cropping, etc.) can be performed on the original image to obtain an enhanced image of the original image.

[0125] Figure 3 This is a schematic diagram illustrating a training dataset according to an exemplary embodiment.

[0126] refer to Figure 3 The training data collection mentioned above may include the following: open-source datasets from academia and UGC (User-Generated Content) data from within the site, in a 1:1:1 ratio.

[0127] In some embodiments, data processing can be performed using the following methods: 1. Clean up dirty data with low text-image relevance using CLIP (Contrastive Language-Image Pre-training) model; 2) Enhance text diversity by using Qwen2.5-VL (a large language model) multimodal large model annotation; 3) Enhance image diversity by randomly covering stickers, cropping, etc., to obtain enhanced images.

[0128] Step S208: Based on the first dataset, determine the graph-text contrast learning loss, graph-graph contrast learning loss, and distillation loss between the graph-text representation student model and the graph-text representation teacher model.

[0129] Image-text contrastive learning loss is a loss function that achieves cross-modal semantic alignment by bringing matching image-text pairs closer together and distancing unmatched pairs.

[0130] Image-to-image contrastive learning loss is a loss function that improves the model's understanding of the image content itself and the robustness of its representation by narrowing the distance between different image transformations (such as the original image and its corresponding enhanced image) of the same content source in the feature space.

[0131] Distillation loss is a loss function that minimizes the difference between the output representations of the student model and the teacher model in order to transfer knowledge from a large model to a small model.

[0132] Step S210: Freeze the student text tower and train the student image tower of the student model based on the image-text contrast learning loss, the image-image contrast learning loss, and the distillation loss.

[0133] In some embodiments, the student text tower (jina-embeddings-v2-base-zh-ft) can be frozen, and the student model can be trained until convergence based on the following: image-text contrast loss (randomly selecting one of the multiple texts corresponding to an image to form an image-text pair, calculating InfoNCE (Information Noise-Contrastive Estimation) contrast loss to improve the model's image-text retrieval ability), image-image contrast loss (the original image and the enhanced image form an image-image pair, calculating InfoNCE contrast loss to improve the model's image-image retrieval ability), and distillation loss (further improving the student model's ability to understand images by using the cosine similarity distance loss of the output features of the teacher image tower and the student image tower).

[0134] In the above method, the student text tower, which inherits all features of the teacher's text tower, can be fixed first. The parameters of the student image tower are then optimized solely through training, causing its generated feature representation to continuously approach the semantic space corresponding to the teacher's text tower. This approach allows the training process to be more focused, guiding the student image tower to quickly converge to a representation state with higher semantic consistency, effectively improving the overall alignment efficiency and convergence speed of the model.

[0135] In the above methods, image-text contrastive loss ensures the model's core ability for cross-modal semantic alignment, image-image contrastive loss enhances the model's understanding and generalization of image content itself (rather than surface features), and distillation loss, through knowledge transfer, enables the lightweight student model to inherit the powerful representational capabilities of the teacher model. This collaborative optimization strategy avoids training bias that may be caused by a single loss function, achieving higher performance and more robust edge multimodal representations while significantly compressing model parameters.

[0136] This technical solution constructs a lightweight student model guided by a high-performance teacher model and adopts a multi-task joint training strategy. Under the premise of freezing the student text tower, it comprehensively utilizes image-text contrast loss, image-image contrast loss and distillation loss to jointly optimize the student image tower. In the end, while significantly reducing the number of model parameters and computational complexity, it successfully constructs a general lightweight multimodal representation model with strong cross-modal semantic alignment capability (Chinese and English bilingual), high-quality image content understanding capability and excellent representation robustness, effectively solving the problem of balancing model performance, efficiency and universality.

[0137] Figure 4 This is an architecture diagram illustrating a method for constructing a multimodal representation according to an exemplary embodiment.

[0138] refer to Figure 4 This application provides a method for constructing a multimodal standard. Specifically, it may include the following steps.

[0139] refer to Figure 4 Assuming the teacher model is already trained, the student text tower of the student model integrates the structure and parameters of the teacher text tower, and the student image tower is a "slimmed-down" model of the teacher image tower.

[0140] In some embodiments, the graph-to-graph contrast loss, graph-to-text contrast loss, and distillation loss between the student model and the teacher model can be determined using training samples.

[0141] like Figure 4 As shown, the parameters of the student text tower of the student model can be frozen, and then the parameters of the student image tower can be trained using the above-mentioned graph-to-graph contrast loss, graph-to-text contrast loss, and distillation loss.

[0142] The student model trained using the above method can achieve high performance and strong versatility in multimodal semantic understanding with extremely low parameter count and computational complexity.

[0143] In some embodiments, the teacher text tower can be based on a pre-trained multilingual text encoder and obtained through fine-tuning.

[0144] In some embodiments, a projection layer may be added after the multilingual text encoder to obtain the teacher text tower before fine-tuning, and the projection layer is used for data dimensionality reduction.

[0145] In some embodiments, the aforementioned teacher text tower may be a text encoder that has been improved (e.g., by adding a projection layer for dimensionality reduction) and then fine-tuned, based on a text encoder that supports multilingual text embedding representations.

[0146] Below, this application will describe in detail how to fine-tune the teacher text tower.

[0147] Figure 5 This is a flowchart illustrating a method for constructing a multimodal representation according to an exemplary embodiment.

[0148] refer to Figure 5 The above-mentioned method for constructing multimodal representations may include the following steps.

[0149] Step S502: Obtain the second dataset.

[0150] In some embodiments, the second dataset also includes multiple training samples, each of which may include the original image, the enhanced image, and text describing the original image.

[0151] The aforementioned text may include Chinese text and / or English text.

[0152] Step S504: Based on the second dataset, determine the image-text contrast loss corresponding to the image-text representation teacher model.

[0153] Step S506: Freeze the teacher image tower and train the teacher text tower based on the image-text contrast loss corresponding to the image-text representation teacher model, so that the image-text representation teacher model can align the representations of image information and text information in the same vector space.

[0154] In some embodiments, the teacher image tower itself may be a pre-trained image encoder capable of image representation, and the teacher text tower may be a text encoder that is improved (e.g., by adding a projection layer for dimensionality reduction) from a text encoder that supports multilingual text embedding representation (such as a multilingual text encoder).

[0155] In some embodiments, a projection layer may be added after the multilingual text encoder to obtain the teacher text tower before fine-tuning, and the projection layer is used for data dimensionality reduction.

[0156] In some embodiments, freezing the teacher image tower and training the teacher text tower based on the image-text contrast loss corresponding to the image-text representation teacher model can make the text representation of the teacher text tower closer to the image representation of the teacher image tower, thereby enabling the image-text representation teacher model to align the representations of image information and text information in the same vector space.

[0157] In some embodiments, the teacher text tower and the teacher text tower can be trained simultaneously based on the image-text contrast loss corresponding to the image-text representation teacher model, so that the image-text representation teacher model can align the representations of image information and text information in the same vector space.

[0158] This technical solution successfully aligns image and text representations into a unified semantic vector space by freezing the pre-trained teacher image tower and specifically optimizing the teacher text tower using image-text contrast loss (or jointly fine-tuning the two towers). This process effectively constructs a teacher model with strong cross-modal (Chinese and English bilingual + image) semantic understanding capabilities, providing high-quality and highly consistent supervision signals for the subsequent distillation training of the student model, fundamentally ensuring the performance foundation of the lightweight multimodal representation model.

[0159] Figure 6 This is a flowchart illustrating a method for determining the learning loss of image-text contrastive learning according to an exemplary embodiment.

[0160] refer to Figure 6 The above-mentioned determination of the image-text contrast learning loss corresponding to the image-text representation student model based on the first dataset may include the following steps.

[0161] Step S602: Embedded representation of the original images in the first dataset is performed using the student image tower to obtain multiple first image features.

[0162] Step S604: Embed and represent the text information in the first dataset using the student text tower to obtain multiple first text features.

[0163] Step S606: Determine the pairwise similarity between all first image features and all first text features to generate an image-text similarity matrix, wherein the diagonal elements in the image-text similarity matrix are the similarity of positive sample pairs, and the off-diagonal elements are the similarity of negative sample pairs.

[0164] Step S608: Determine the contrast loss from each image to text and the contrast loss from each text to image based on the image-text similarity matrix.

[0165] In the image-text similarity matrix, S[i][j] represents the similarity score between the i-th image feature and the j-th text feature; i is an integer greater than or equal to 1, and j is an integer greater than or equal to 1.

[0166] In the image-text similarity matrix, the diagonal element S[i][i] represents that the i-th image and the i-th text are matched (from the same training sample), and they are positive sample pairs. In the matrix, the off-diagonal element S[i][j] (where i≠j) represents that the i-th image and the j-th text are not matched, and they are negative samples of each other.

[0167] The goal of calculating the contrast loss for "image-to-text" is to correctly identify the matching text from all texts in the entire batch for each image.

[0168] The method for calculating the contrast loss in "image-to-text" classification is as follows: Each row of the similarity matrix S is treated as an independent classification task; for the i-th row (corresponding to the i-th image), its positive sample score is S[i][i], and its negative sample score is S[i][j] for all j ≠ i in that row; a cross-entropy loss is calculated for the i-th row, which is equivalent to using a Softmax function to combine all elements (S[i][j]) of the i-th row into a single cross-entropy loss. The model is then transformed into a probability distribution, and then the model is made to maximize the probability of positive sample S[i][i].

[0169] The calculation method for "image-to-text" contrast loss can be summarized as follows: construct a matrix of the similarity between all images and texts in the batch, and take each row in the matrix (representing the similarity between a single image and all texts) as an independent classification task. By calculating the cross-entropy loss, the similarity between image-text pairs that match at the diagonal position is much higher than that of non-matching pairs in the same row, thereby driving the model to learn the ability to accurately retrieve the corresponding text from the image.

[0170] The calculation method for "text-to-image" contrast loss can refer to the calculation method for "image-to-text" contrast loss, and will not be repeated here.

[0171] Step S610: Determine the image-text contrast learning loss based on the contrast loss of each image to text and the contrast loss of each text to image.

[0172] This technical solution constructs an image-text similarity matrix and calculates the contrast loss in both the "image-to-text" and "text-to-image" directions, effectively driving the student model to learn cross-modal semantic alignment. This enables the model to achieve high-precision bidirectional retrieval capabilities for images and text in a unified vector space, significantly improving the model's cross-modal understanding and matching performance.

[0173] Figure 7 This is a flowchart illustrating a graph-to-graph contrastive learning loss determination method according to an exemplary embodiment.

[0174] refer to Figure 7 The graph-to-graph contrast learning loss corresponding to the graph-to-text representation student model is determined based on the first dataset, which may include the following steps.

[0175] Step S702: Embed the original images in the first dataset using the student image tower to obtain multiple second image features.

[0176] Step S704: Embedded representation of the enhanced images in the first dataset is performed using the student image tower to obtain multiple third image features.

[0177] Step S706: Determine the pairwise similarity between all second image features and all third image features to generate a first image-to-image similarity matrix, wherein the diagonal elements in the first image-to-image similarity matrix are the similarity of positive sample pairs, and the off-diagonal elements are the similarity of negative sample pairs.

[0178] Step S708: Determine the contrast loss from each original image to the enhanced image and the contrast loss from each enhanced image to the original image based on the first image similarity matrix.

[0179] Step S710: Determine the graph-to-graph contrast learning loss based on the contrast loss from each original image to the enhanced image and the contrast loss from each enhanced image to the original image.

[0180] In some embodiments, the graph-to-graph contrast learning loss can be determined by referring to the specific implementation method of the graph-to-text contrast learning loss, which will not be described in detail in this embodiment.

[0181] This technical solution involves inputting the original and enhanced images from the same batch into a student image tower, extracting features, constructing an image-to-image similarity matrix, and calculating the contrast loss in both directions—from the original image to the enhanced image and from the enhanced image to the original image—using each row and column of the matrix as units. Finally, these two losses are combined (e.g., by taking the average) to obtain the final image-to-image contrast learning loss.

[0182] This technical solution constructs a feature similarity matrix for image enhancement pairs and calculates bidirectional contrast loss, enabling the model to learn stable and consistent visual semantic representations from different image transformations. This significantly improves the robustness of image representation and content understanding capabilities, providing a more robust feature foundation for downstream visual tasks.

[0183] Figure 8 This is a flowchart illustrating a method for determining distillation loss according to an exemplary embodiment.

[0184] refer to Figure 8 The above method for determining distillation loss may include the following steps.

[0185] Step S802: The original images in the first dataset are represented by the student image tower in the image and text representation student model to obtain the fourth image features.

[0186] Step S804: The original images in the first dataset are represented by the teacher image tower in the image and text representation teacher model to obtain the fifth image features.

[0187] Step S806: Determine the similarity distance loss between the fourth image feature and the fifth image feature in the same training sample.

[0188] Step S808: Determine the distillation loss based on the similarity distance loss corresponding to each training sample.

[0189] This distillation loss determination scheme inputs the same original image into the image towers of both the student and teacher models, extracts their features, and calculates the similarity distance loss between them. This enables the efficient transfer of the teacher model's powerful image semantic understanding capabilities to the lightweight student model. In this way, while significantly compressing the number of model parameters, it ensures the high quality and robustness of the student model's image representation, laying a solid visual foundation for the core multimodal alignment task.

[0190] This application illustrates a method for constructing a multimodal representation according to an exemplary embodiment, which may specifically include the following steps.

[0191] 1. Data Acquisition: Obtain raw images and corresponding text information from open-source datasets in academia and UGC data within the site, in a 1:1 ratio. For details, please refer to [link / reference]. Figure 3 .

[0192] 2. Data Processing: 1) Use the CLIP model to clean up dirty data with low image-text relevance; 2) Enhance text diversity through multimodal large model annotation; 3) Enhance image diversity by randomly covering stickers, cropping, etc., to obtain enhanced images. Training samples are determined based on the original images, text information, and enhanced images.

[0193] 3. Model Structure: Reference Figure 4 We can determine that MobileCLIP-S0's image representation MCI0 is the image representation model, and jina-embeddings-v2-base-zh is the pre-trained text representation model.

[0194] a) `jina-embeddings-v2-base-zh` is a bilingual Chinese-English text embedding model that supports sequences up to 8192 bytes long. It is based on the BERT architecture (JinaBERT), which supports a symmetric bidirectional variant of ALiBi to allow for longer sequence lengths. This model is specifically trained to support mixed Chinese and English input without bias. It has 161M parameters, supports both Chinese and English, and has an original output feature dimension of 768. This application adds an extra layer of dimension mapping within `jina-embeddings-v2-base-zh` to generate the structures of the teacher's and student's text towers.

[0195] b) In terms of image representation, the teacher model is the image representation MCI0 of MobileCLIP-S0 with 11M parameters, and the student model (that is, the model that is finally deployed on the edge) is the MCI-tiny with improved structure and 2M parameters.

[0196] The MCI0 configuration is as follows: layers = [2, 6, 10, 2] (a total of 20 blocks); embed_dims = [64, 128, 256, 512]; token_mixers = ("repmixer", "repmixer", "repmixer", "attention") (the last one is Attention); mlp_ratios = [3, 3, 3, 3]; act_layer = nn.GELU.

[0197] MCi-Tiny Configuration: layers = [2, 4, 6, 2] (reduce depth, total 14 blocks, from 20 to 14); embed_dims = [32, 64, 128, 256] (width halved); token_mixers = ("repmixer", "repmixer", "repmixer", "repmixer"); mlp_ratios = [3, 3, 3, 3]; act_layer = nn.ReLU (Global Replacement Activation Function).

[0198] 4. Model Training – Teacher Model: Reference Figure 4 The image tower MCI0 can be frozen. The image-text loss of the teacher model can be determined through the training samples determined in steps 1 and 2. Based on the image-text contrast loss, the text tower of the teacher model can be trained to obtain jina-embeddings-v2-base-zh-ft.

[0199] 5. Model Training – Student Model: Reference Figure 4 It can freeze the text tower jina-embeddings-v2-base-zh-ft, and train the student model until convergence based on image-text contrast loss (randomly select one of the multiple texts corresponding to an image to form an image-text pair, calculate InfoNCE contrast loss, and improve the model's image-text retrieval ability), image-image contrast loss (the original image and the enhanced image form an image-image pair, calculate InfoNCE contrast loss, and improve the model's image-image retrieval ability), and distillation loss (through the cosine similarity distance loss of the output features of the teacher image tower and the student image tower, further improve the student model's ability to understand images).

[0200] 6. Model Evaluation: Evaluation will be conducted from two aspects: general knowledge (based on open-source / internal data, without specific business scenarios) and business capabilities (planned implementation scenarios).

[0201] Figure 9A and Figure 9B This is a model evaluation result illustrated according to an exemplary embodiment.

[0202] like Figure 9A and Figure 9B As shown, the technical solutions provided in the embodiments of this application perform well in data across different fields.

[0203] The above embodiments are improved based on at least the following technologies: (1) The image representation MCI0 of MobileCLIP-S0 is used as the base image representation, and the number of parameters of MCI0 is reduced to 2M through compression and distillation training to obtain MCI-tiny; (2) Jina-embeddings-v2-base-zh is selected as the pre-trained text representation (the advantage is that it supports Chinese and English text understanding, but it only supports text unimodality) and combined with MCI-tiny. Finally, a CLIP-style model that supports Chinese and English can be realized, and the image tower has the ability to be deployed on the edge.

[0204] It should be particularly noted that the steps in each embodiment of the above-described method for constructing multimodal representations can be overlapped, substituted, added, or deleted. Therefore, these reasonable permutations and combinations of the method for constructing multimodal representations should also fall within the scope of protection of this disclosure, and the scope of protection of this disclosure should not be limited to the described embodiments.

[0205] It should be noted that the scope of protection of this application should include, but is not limited to, the specific implementation methods described in the embodiments. Any alternative solution that uses a different name but substantially performs the same function and achieves the same technical effect falls within the scope of protection defined by the claims of this application.

[0206] Based on the same inventive concept, this disclosure also provides a multimodal characterization construction apparatus, as described in the following embodiments. Since the principle by which this apparatus solves the problem is similar to that of the method embodiments described above, the implementation of this apparatus embodiment can refer to the implementation of the method embodiments described above, and repeated details will not be elaborated further.

[0207] Figure 10 This is a block diagram illustrating a multimodal characterization construction apparatus according to an exemplary embodiment. (Refer to...) Figure 10 The multimodal representation construction apparatus 1000 provided in this embodiment may include: a teacher model acquisition module 1001, a teacher model acquisition module 1002, a first dataset acquisition module 1003, a loss determination module 1004, and a training module 1005.

[0208] The teacher model acquisition module 1001 can be used to acquire a graphical representation teacher model, which includes a teacher image tower and a teacher text tower. This model maps image and text information to the same vector space to achieve representation alignment. The teacher model acquisition module 1002 can be used to acquire a graphical representation student model, which includes a student image tower and a student text tower. The student text tower inherits the structure and parameters of the teacher text tower. The student image tower is obtained by model compression of the teacher image tower. The first dataset acquisition module 1003 can be used to acquire datasets containing multiple training datasets. The first dataset of training samples includes original images, augmented images of the original images, and text information describing the original images. The loss determination module 1004 can be used to determine, based on the first dataset, the image-text contrast learning loss, the image-to-image contrast learning loss, and the distillation loss between the image-text representation student model and the image-text representation teacher model. The training module 1005 can be used to freeze the student text tower and train the student image tower of the image-text representation student model based on the image-text contrast learning loss, the image-to-image contrast learning loss, and the distillation loss.

[0209] It should be noted that the teacher model acquisition module 1001, teacher model acquisition module 1002, first dataset acquisition module 1003, loss determination module 1004, and training module 1005 mentioned above correspond to S202 to S210 in the method embodiment. The examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in the above method embodiment. It should be noted that the above modules, as part of the apparatus, can be executed in a computer system such as a set of computer-executable instructions.

[0210] In some embodiments, the multimodal representation construction apparatus 1000 may further include: a second dataset acquisition module, a second image-text comparison loss determination module, and a teacher model training module.

[0211] The second dataset acquisition module can be used to acquire a second dataset; the second image-text contrast loss determination module can be used to determine the image-text contrast loss corresponding to the image-text representation teacher model based on the second dataset; the teacher model training module can be used to freeze the teacher image tower and train the teacher text tower based on the image-text contrast loss corresponding to the image-text representation teacher model, so that the image-text representation teacher model can align the representations of image information and text information in the same vector space.

[0212] In some embodiments, the teacher image tower is a visual encoder that supports image embedding representations, and the teacher text tower is determined based on a text encoder that supports multilingual text embedding representations, wherein the feature vectors output by the teacher image tower and teacher text tower before training do not have cross-modal semantic alignment capabilities. That is, or, independent teacher image towers and teacher text towers that are not jointly trained, their output feature vectors do not have semantic alignment capabilities and cannot be directly used for cross-modal similarity calculation.

[0213] In some embodiments, the multimodal representation construction apparatus 1000 may further include a teacher text tower determination module.

[0214] The teacher text tower determination module can be used to add a projection layer after the text encoding model to obtain the teacher text tower, and the projection layer is used for data dimensionality reduction processing.

[0215] In some embodiments, the multimodal representation construction apparatus 1000 may further include an enhanced text information acquisition module and a text information determination module.

[0216] The enhanced text information acquisition module can be used to perform text enhancement processing on the original image and the original text information corresponding to the original image through a large language model to obtain enhanced text information describing the original image; the text information determination module can be used to determine the text information describing the original image based on the enhanced text.

[0217] In some embodiments, determining the image-text contrastive learning loss corresponding to the image-text representation student model based on the first dataset includes: embedding representations of the original images in the first dataset using the student image tower to obtain multiple first image features; embedding representations of text information in the first dataset using the student text tower to obtain multiple first text features; determining the pairwise similarity between all first image features and all first text features to generate an image-text similarity matrix, wherein the diagonal elements in the image-text similarity matrix are the similarity of positive sample pairs, and the off-diagonal elements are the similarity of negative sample pairs; determining the contrastive loss from each image to text and the contrastive loss from each text to image based on the image-text similarity matrix; and determining the image-text contrastive learning loss based on the contrastive losses from each image to text and the contrastive losses from each text to image.

[0218] In some embodiments, determining the image-to-image contrast learning loss corresponding to the image-text representation student model based on the first dataset includes: embedding representations of the original images in the first dataset using the student image tower to obtain multiple second image features; embedding representations of the augmented images in the first dataset using the student image tower to obtain multiple third image features; determining the pairwise similarity between all second image features and all third image features to generate a first image-to-image similarity matrix, wherein the diagonal elements of the first image-to-image similarity matrix are the similarity of positive sample pairs, and the off-diagonal elements are the similarity of negative sample pairs; determining the contrast loss from each original image to the augmented image and the contrast loss from each augmented image to the original image based on the first image-to-image similarity matrix; and determining the image-to-image contrast learning loss based on the contrast loss from each original image to the augmented image and the contrast loss from each augmented image to the original image.

[0219] In some embodiments, determining the distillation loss between the image-text representation student model and the image-text representation teacher model based on the first dataset includes: performing feature representation on the original images in the first dataset using the student image tower in the image-text representation student model to obtain a fourth image feature; performing feature representation on the original images in the first dataset using the teacher image tower in the image-text representation teacher model to obtain a fifth image feature; determining the similarity distance loss between the fourth image feature and the fifth image feature in the same training sample; and determining the distillation loss based on the similarity distance loss corresponding to each training sample.

[0220] In some embodiments, the multimodal characterization construction apparatus 1000 may further include a depth reduction module and a computational unit replacement module.

[0221] The depth reduction module can be used to reduce the network depth and / or feature width of the teacher image tower; the computation unit replacement module can be used to replace the computationally intensive self-attention computation units in the teacher image tower with lightweight computation units to obtain the student image tower.

[0222] Since the functions of the device 1000 have been described in detail in their corresponding method embodiments, they will not be repeated here.

[0223] The modules described in the embodiments of this disclosure can be implemented in software or hardware. The described modules can also be located in a processor. The names of these modules do not, in some cases, constitute a limitation on the module itself.

[0224] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a portion of a module or program segment containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer program instructions.

[0225] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0226] Figure 11 A schematic diagram of an electronic device suitable for implementing embodiments of the present disclosure is shown. It should be noted that... Figure 11 The illustrated electronic device 1100 is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0227] like Figure 11 As shown, the electronic device 1100 includes a central processing unit (CPU) 1101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1102 or a program loaded from a storage section 1108 into a random access memory (RAM) 1103. The RAM 1103 also stores various programs and data required for the operation of the electronic device 1100. The CPU 1101, ROM 1102, and RAM 1103 are interconnected via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0228] The following components are connected to I / O interface 1105: an input section 1106 including a keyboard, mouse, etc.; an output section 1107 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1108 including a hard disk, etc.; and a communication section 1109 including a network interface card such as a LAN card, modem, etc. The communication section 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to I / O interface 1105 as needed. Removable media 1111, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1110 as needed so that computer programs read from them can be installed into storage section 1108 as needed.

[0229] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing computer program instructions for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1109, and / or installed from removable medium 1111. When the computer program is executed by central processing unit (CPU) 1101, it performs the functions defined above in the system of this disclosure.

[0230] It should be noted that the computer-readable storage medium disclosed herein may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable computer program instructions. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable storage medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. Computer program instructions contained on a computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0231] In another aspect, this disclosure also provides a computer-readable storage medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The aforementioned computer-readable storage medium carries one or more programs that, when executed by a device, enable the device to perform the following functions: acquiring a teacher model of image-text representation, the teacher model including a teacher image tower and a teacher text tower, the teacher model being used to map image information and text information to the same vector space to achieve representation alignment; acquiring a student model of image-text representation, the student model including a student image tower and a student text tower, the student text tower inheriting the structure and parameters of the teacher text tower; the student image tower being obtained by model compression of the teacher image tower; acquiring a first dataset containing multiple training samples, wherein the training samples include original images, enhanced images of the original images, and text information describing the original images; based on the first dataset, determining the image-text contrast learning loss, the image-image contrast learning loss, and the distillation loss between the student model and the teacher model of image-text representation; freezing the student text tower, and training the student image tower of the student model of image-text representation based on the image-text contrast learning loss, the image-image contrast learning loss, and the distillation loss.

[0232] According to one aspect of this disclosure, a computer program product or computer program is provided, comprising computer program instructions stored in a computer-readable storage medium. The computer program instructions are read from the computer-readable storage medium, and a processor executes the computer program instructions to implement the methods provided in various optional implementations of the above embodiments.

[0233] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions of the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, or portable hard drive) and includes several computer program instructions to cause an electronic device (such as a server or terminal device) to execute the method according to the embodiments of this disclosure.

[0234] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0235] It should be understood that this disclosure is not limited to the detailed structures, drawing arrangements or implementations shown herein; rather, this disclosure is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended claims.

Claims

1. A method for constructing multimodal representations, characterized in that, include: A teacher model for image and text representation is obtained, which includes a teacher image tower and a teacher text tower. The teacher model for image and text representation is used to map image information and text information to the same vector space to achieve representation alignment. A student model representing a graph and text is obtained, comprising a student image tower and a student text tower. The student text tower inherits the structure and parameters of the teacher text tower. The student image tower is obtained by simplifying the structure of the teacher image tower. Obtain a first dataset containing multiple training samples, wherein the training samples include original images, augmented images of the original images, and text information describing the original images; Based on the first dataset, determine the text-image contrast learning loss and the image-image contrast learning loss corresponding to the text-image representation student model, as well as the distillation loss between the text-image representation student model and the text-image representation teacher model; The student text tower is frozen, and the student image tower of the image-text representation student model is trained based on the image-text contrast learning loss, the image-image contrast learning loss, and the distillation loss. The trained image-text standard student model is used to map image information and text information to the same vector space to achieve representation alignment.

2. The method according to claim 1, characterized in that, The teacher text tower mentioned above is based on a pre-trained multilingual text encoder and obtained through fine-tuning. The method further includes: Obtain the second dataset; Based on the second dataset, determine the image-text contrast loss corresponding to the image-text representation teacher model; The teacher image tower is frozen, and the teacher text tower is fine-tuned based on the image-text contrast loss corresponding to the image-text representation teacher model, so that the image-text representation teacher model can align the representations of image information and text information in the same vector space.

3. The method according to claim 2, characterized in that, The teacher image tower is a visual encoder that supports image embedding representation, and the teacher text tower is determined based on a text encoder that supports multilingual text embedding representation, wherein the feature vectors output by the teacher image tower and the teacher text tower before training do not have cross-modal semantic alignment capabilities.

4. The method according to claim 2, characterized in that, The method further includes: A projection layer is added after the multilingual text encoder to obtain the teacher text tower before fine-tuning; the projection layer is used for data dimensionality reduction.

5. The method according to claim 1, characterized in that, The method further includes: The original image and its corresponding original text information are subjected to text enhancement processing using a multimodal large language model to obtain enhanced text information that describes the original image. Based on the enhanced text, the text information describing the original image is determined.

6. The method according to claim 1, characterized in that, Based on the first dataset, determine the image-text contrast learning loss corresponding to the image-text representation student model, including: The original images in the first dataset are embedded and represented using the student image tower to obtain multiple first image features; The text information in the first dataset is embedded and represented using the student text tower to obtain multiple first text features; Determine the pairwise similarity between all first image features and all first text features to generate an image-text similarity matrix, wherein the diagonal elements of the image-text similarity matrix are the similarity of positive sample pairs, and the off-diagonal elements are the similarity of negative sample pairs. Based on the image-text similarity matrix, determine the contrast loss from each image to text and the contrast loss from each text to image; The image-to-text contrast learning loss is determined based on the contrast loss of each image to text and the contrast loss of each text to image.

7. The method according to claim 1, characterized in that, Based on the first dataset, the graph-to-graph contrast learning loss corresponding to the graph-to-text representation student model is determined, including: The original images in the first dataset are embedded and represented using the student image tower to obtain multiple second image features; The enhanced images in the first dataset are embedded and represented using the student image tower to obtain multiple third image features; Determine the pairwise similarity between all second image features and all third image features to generate a first image-to-image similarity matrix, wherein the diagonal elements of the first image-to-image similarity matrix are the similarity of positive sample pairs, and the off-diagonal elements are the similarity of negative sample pairs; Based on the first image similarity matrix, determine the contrast loss from each original image to the enhanced image and the contrast loss from each enhanced image to the original image; The graph-to-graph contrast learning loss is determined based on the contrast loss from each original image to the enhanced image and the contrast loss from each enhanced image to the original image.

8. The method according to claim 1, characterized in that, Determining the distillation loss between the text-image representation student model and the text-image representation teacher model based on the first dataset includes: The original images in the first dataset are represented by the student image tower in the image and text representation student model to obtain the fourth image features; The teacher image tower in the image and text representation teacher model is used to perform feature representation on the original images in the first dataset to obtain the fifth image feature; Determine the similarity distance loss between the fourth and fifth image features in the same training sample; The distillation loss is determined based on the similarity distance loss corresponding to each training sample.

9. The method according to claim 1, characterized in that, Obtaining the student image tower includes: The network structure of the teacher image tower is adjusted to reduce its computational complexity and / or number of parameters in order to obtain the student image tower; The adjustments include at least one of the following: reducing network depth, reducing feature dimensions, and replacing computationally intensive modules with lightweight modules.

10. An electronic device, characterized in that, include: Memory and processor; The memory is used to store computer program instructions; the processor calls the computer program instructions stored in the memory to implement the method for constructing multimodal representations as described in any one of claims 1-9.

Images