Training method and device of image encoder, electronic equipment, computer readable storage medium and computer program product

Abstract

This application provides a training method, apparatus, electronic device, computer-readable storage medium, and computer program product for an image encoder. The method includes: extracting low-dimensional image features of a first image sample; performing semantic extraction based on the low-dimensional image features using a first image encoder to be trained, obtaining first semantic features of the first image sample; extracting high-dimensional image features of the first image sample, and extracting first text features of a first text sample describing the first image sample, wherein the dimension of the high-dimensional image features is greater than the dimension of the low-dimensional image features; determining a first distillation loss based on the first semantic features and the high-dimensional image features, and determining a first contrast loss based on the first semantic features and the first text features; updating the parameters of the first image encoder based on the first distillation loss and the first contrast loss, to obtain a trained first image encoder. This application improves the accuracy of feature extraction from images by the image encoder.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a training method, apparatus, electronic device, computer-readable storage medium, and computer program product for an image encoder. Background Technology

[0002] In the field of multimodal learning, joint representation learning of images and text is a key task, aiming to establish semantic associations across modalities. Related techniques train encoders using low-dimensional features of images and text features used to describe images to achieve image-text understanding and image generation. However, low-dimensional features lead to information loss, limit the expressive power of the encoder, and fail to fully capture the complex relationship between images and text. Summary of the Invention

[0003] This application provides a training method, apparatus, electronic device, computer-readable storage medium, and computer program product for an image encoder, which can improve the accuracy of feature extraction from images by the image encoder.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] This application provides a method for training an image encoder, the method comprising:

[0006] Extract low-dimensional image features from the first image sample;

[0007] Based on the low-dimensional image features, the first image encoder to be trained is invoked to perform semantic extraction, thereby obtaining the first semantic features of the first image sample.

[0008] Extract high-dimensional image features from the first image sample and extract first text features from the first text sample used to describe the first image sample, wherein the dimension of the high-dimensional image features is greater than the dimension of the low-dimensional image features;

[0009] A first distillation loss is determined based on the first semantic feature and the high-dimensional image feature, and a first contrast loss is determined based on the first semantic feature and the first text feature;

[0010] The parameters of the first image encoder are updated based on the first distillation loss and the first contrast loss to obtain the trained first image encoder.

[0011] This application provides a training apparatus for an image encoder, the apparatus comprising:

[0012] The first extraction module is used to extract low-dimensional image features from the first image sample;

[0013] The semantic extraction module is used to call the first image encoder to be trained to perform semantic extraction based on the low-dimensional image features, so as to obtain the first semantic features of the first image sample;

[0014] The second extraction module is used to extract high-dimensional image features of the first image sample and extract first text features of the first text sample describing the first image sample, wherein the dimension of the high-dimensional image features is greater than the dimension of the low-dimensional image features.

[0015] The loss determination module is used to determine a first distillation loss based on the first semantic feature and the high-dimensional image feature, and to determine a first contrast loss based on the first semantic feature and the first text feature;

[0016] The parameter update module is used to update the parameters of the first image encoder based on the first distillation loss and the first contrast loss, so as to obtain the trained first image encoder.

[0017] This application provides an electronic device, the electronic device comprising:

[0018] Memory is used to store executable instructions or computer programs.

[0019] The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the image encoder training method provided in the embodiments of this application.

[0020] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the image encoder training method provided in this application when executed by a processor.

[0021] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the image encoder training method provided in this application.

[0022] The embodiments of this application have the following beneficial effects:

[0023] On the one hand, by extracting the first semantic feature from the low-dimensional image features of the image sample and determining the distillation loss between it and the extracted high-dimensional image features, the first image encoder is trained. This allows the first image encoder to retain the original information of the image sample through the low-dimensional image features, understand the semantic information of the image sample through the semantic features, and retain the detailed information of the image sample through the high-dimensional image features, thereby reducing information loss. The joint training of features in multiple dimensions improves the semantic consistency of the features.

[0024] On the other hand, by determining the contrast loss between the semantic features of the image sample and the text features of the text sample used to describe the image sample, cross-modal feature alignment is achieved, enhancing the image-text understanding ability of the first image encoder. This enables the first image encoder to incorporate the text features of the text corresponding to the image when extracting image features, thereby improving the accuracy of the first image encoder in extracting features from the image. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the architecture of the image encoder training system 100 provided in an embodiment of this application;

[0026] Figure 2 This is a schematic diagram of the structure of the server 200 provided in the embodiments of this application;

[0027] Figure 3A This is a first flowchart illustrating the training method for an image encoder provided in an embodiment of this application;

[0028] Figure 3B This is a second flowchart illustrating the training method for the image encoder provided in this application embodiment;

[0029] Figure 3C This is a schematic diagram of the third process of the training method for the image encoder provided in the embodiments of this application;

[0030] Figure 3D This is a schematic diagram of the fourth process of the training method for the image encoder provided in the embodiments of this application;

[0031] Figure 3E This is a schematic diagram of the fifth process of the training method for the image encoder provided in the embodiments of this application;

[0032] Figure 3F This is a schematic diagram of the sixth process of the training method for the image encoder provided in the embodiments of this application;

[0033] Figure 3G This is a schematic diagram of the seventh process of the training method for the image encoder provided in the embodiments of this application;

[0034] Figure 3H This is a schematic diagram of the eighth process of the training method for the image encoder provided in the embodiments of this application;

[0035] Figure 3I This is a ninth flowchart illustrating the training method for the image encoder provided in this application embodiment;

[0036] Figure 3J This is a schematic diagram of the tenth process of the training method for the image encoder provided in the embodiments of this application;

[0037] Figure 3KThis is a schematic diagram of the eleventh step of the training method for the image encoder provided in the embodiments of this application;

[0038] Figure 3L This is a schematic diagram of the twelfth step of the training method for the image encoder provided in this application embodiment;

[0039] Figure 3M This is a schematic diagram of the thirteenth step of the training method for the image encoder provided in this application embodiment;

[0040] Figure 3N This is a schematic diagram of the fourteenth step of the training method for the image encoder provided in the embodiments of this application;

[0041] Figure 3O This is the fifteenth flowchart of the image encoder training method provided in the embodiments of this application;

[0042] Figure 3P This is the sixteenth flowchart of the training method for the image encoder provided in the embodiments of this application;

[0043] Figure 4 This is a flowchart illustrating the process of adjusting an image based on text, provided in an embodiment of this application.

[0044] Figure 5 This is an application illustration of adjusting an image based on text, provided in an embodiment of this application;

[0045] Figure 6 This is a schematic diagram illustrating the completion of partially missing images provided in an embodiment of this application;

[0046] Figure 7 This is a schematic diagram illustrating the completion of partially missing images provided in an embodiment of this application;

[0047] Figure 8 This is a schematic diagram of the structure of the visual encoder provided in the embodiments of this application;

[0048] Figure 9 This is a schematic diagram of the structure of the visual encoder combined with the multimodal unified understanding and generation model provided in the embodiments of this application.

[0049] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0051] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0052] In the following description, the terms "first / second / third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0053] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0054] Unless otherwise specified, "at least one" as used below refers to one or more cases, and "multiple" can refer to two or more cases.

[0055] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0056] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0057] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0058] 1) Responding to: used to indicate the conditions or states on which the operation is performed depends. When the conditions or states on which it depends are met, one or more operations can be performed in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations are performed.

[0059] 2) Human-computer interaction interface, which is an interface used to provide human-computer interaction functions / display images.

[0060] For example, graphical user interfaces (GUIs) include augmented reality (AR) interfaces, virtual reality (VR) interfaces, voice user interfaces (VUIs), interactive projection interfaces (using projection technology to display information on a flat surface), eye-tracking interfaces (interfaces controlled by detecting the user's gaze), holographic interfaces (three-dimensional holograms formed by projecting images using holographic projection technology, allowing users to see stereoscopic images without wearing special glasses), multimodal interfaces (interfaces that combine multiple interaction methods, such as tactile, visual, and auditory interaction), and brain-machine interfaces (BMIs).

[0061] 3) Low-dimensional image features refer to feature representations extracted from the first image sample using dimensionality reduction techniques. These features have fewer dimensions but retain the main information of the image. They are typically used to capture information such as the global structure, color distribution, and texture of the first image sample, and are commonly used in tasks such as image classification, retrieval, and clustering. The dimensionality of low-dimensional features is much lower than the pixel dimension of the first image sample; for example, the first image sample may have millions of pixels, while the low-dimensional features only have tens to hundreds of dimensions.

[0062] 4) High-dimensional image features refer to feature representations with higher dimensionality extracted from the first image sample. They are typically used to capture local details, edges, textures, and other information in an image. These features are usually output by intermediate layers of deep neural networks, such as feature maps in convolutional neural networks. The dimensionality of high-dimensional features is usually related to the depth and complexity of the network, typically ranging from thousands to millions of dimensions. The dimensionality of high-dimensional image features is greater than that of low-dimensional image features.

[0063] 5) Distillation loss refers to the loss function used to measure the difference between low-dimensional and high-dimensional image features during knowledge distillation. Knowledge distillation is a compression technique that transfers high-dimensional image features to low-dimensional image features, allowing the low-dimensional features to maintain high performance while having lower computational complexity and storage requirements. Distillation loss typically uses soft-target loss to calculate the difference between low-dimensional and high-dimensional image features.

[0064] 6) Contrastive loss, in contrastive learning, is a loss function used to measure the distance between similar samples and the distance between dissimilar samples. Contrastive learning is an unsupervised or semi-supervised learning method that learns to bring similar samples closer together and push dissimilar samples further apart, thereby learning discriminative feature representations. Contrastive loss typically uses a contrast loss function, optimizing the first image encoder by maximizing the similarity between similar samples and minimizing the similarity between dissimilar samples.

[0065] 7) Dynamic weights refer to the different weights assigned to different loss functions during training, based on at least one dimension of the training parameters of the first image encoder. These weights are not fixed but are dynamically adjusted as training progresses to balance the impact of different loss functions on the parameter updates of the first image encoder. The at least one dimension of training parameters includes, but is not limited to, at least one of the following: update epoch, gradient, learning rate, and loss value.

[0066] 8) Progress refers to the proportion or degree of completion of the current update round relative to the preset total number of update rounds during training. For example, if the preset total number of update rounds is 100 rounds and the current update round is 20 rounds, then the progress is 20%. Progress is often used to adjust training parameters such as learning rate and loss weights to adapt to different stages of training.

[0067] 9) Gradient norm refers to the norm (usually L2 norm) of the gradient of the loss function with respect to the parameters of the first image encoder during training. The gradient norm reflects the magnitude of parameter updates; a larger gradient norm means that the parameters will be updated to a greater extent, while a smaller gradient norm means that the parameters will be updated less. The gradient norm can be used to detect gradient explosion or vanishing problems during training, and it is also one of the bases for adjusting dynamic weights.

[0068] 10) A codebook is a set of predefined vectors, often referred to as "codewords" or "atoms." The purpose of a codebook is to map input data (such as the first vector in a feature map) to one or more codewords within the codebook, thereby achieving data compression, quantization, or representation learning. In deep learning, particularly in models such as autoencoders and generative adversarial networks, codebooks are frequently used to convert high-dimensional feature representations into low-dimensional discrete representations; this process is called "quantization."

[0069] Related technologies train encoders using low-dimensional features of images and text features used to describe images to achieve image-text understanding and image generation. However, low-dimensional features lead to information loss, limit the expressive power of the encoder, and fail to fully capture the complex relationship between images and text.

[0070] Based on the above analysis, the applicant found that the training methods of image encoders in related technologies cannot combine low-dimensional and high-dimensional features of images for multimodal learning. In response to the above problems, this application provides a training method for an image encoder that can improve the accuracy of feature extraction from images by the image encoder.

[0071] The following describes exemplary applications of the electronic devices provided in the embodiments of this application. These electronic devices can be implemented as various types of terminals such as laptops, tablets, desktop computers, set-top boxes, smartphones, smart speakers, smartwatches, smart TVs, and in-vehicle terminals, or as servers. The following will describe exemplary applications when the electronic device is implemented as a server.

[0072] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of the image encoder training system 100 provided in the embodiments of this application. In order to support the training application of an image encoder, the terminal (terminal 400-1 and terminal 400-2 are shown as examples) connects to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0073] Terminal 400 is used to acquire a first image sample and send it to server 200. Server 200 is used to train a first image encoder based on the low-dimensional and high-dimensional image features of the first image sample, as well as the first text features of the first text sample describing the first image sample. The trained first image encoder can be used in scenarios such as image completion and image generation. Server 200 sends the completed or generated image to terminal 400 for display on human-computer interaction interface 410.

[0074] The image encoder training method provided in this application can be applied to various scenarios that require image processing using a trained first image encoder, such as online media and e-commerce. Examples are given below.

[0075] Taking online media content generation as an example, the terminal is used to publish image and text content, while the server is used to upload images and text and call the trained first image encoder and pre-trained decoder to generate new image content, such as augmented reality filter effects and image style transfer.

[0076] Taking e-commerce scenarios as an example, the terminal is used to collect descriptions of product images, the server is used to convert the descriptions into semantic features through a trained first image encoder, and the semantic features are converted into product images corresponding to the descriptions through a pre-trained decoder, and then returned to the terminal.

[0077] Taking a game scenario as an example, the terminal is used to collect descriptions of the shape of virtual objects, the server is used to convert the descriptions into semantic features through a trained first image encoder, and the semantic features are converted into the corresponding virtual objects through a pre-trained decoder and returned to the terminal.

[0078] Taking image completion as an example, the terminal is used to collect partially missing images, the server is used to convert the partially missing images into semantic features corresponding to the complete images through the trained first image encoder, the semantic features are converted into complete images through the pre-trained decoder, and then returned to the terminal.

[0079] Taking the intelligent chat scenario as an example, the terminal is used to collect the image input by the user, and the server is used to extract the semantic features of the image through the trained first image encoder, convert the semantic features into an understanding of the image (text reply), and return it to the terminal.

[0080] In some embodiments, server 200 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminals and servers can be connected directly or indirectly via wired or wireless communication, which is not limited in this embodiment.

[0081] See Figure 2 , Figure 2 This is a schematic diagram of the structure of the server 200 provided in the embodiments of this application. Figure 2The server 200 shown includes at least one processor 210, memory 230, and at least one network interface 220. The various components of server 200 are coupled together via a bus system 240. It is understood that the bus system 240 is used to implement communication between these components. In addition to a data bus, the bus system 240 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 240.

[0082] The processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0083] The memory 230 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 230 may optionally include one or more storage devices physically located away from the processor 210.

[0084] The memory 230 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 230 described in this application embodiment is intended to include any suitable type of memory.

[0085] In some embodiments, memory 230 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0086] Operating system 231 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0087] The network communication module 232 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 220, exemplary network interfaces 220 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0088] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2 A training device 233 for an image encoder stored in memory 230 is shown. This device can be software in the form of programs and plugins, and includes the following software modules: a first extraction module 2331, a semantic extraction module 2332, a second extraction module 2333, a loss determination module 2334, and a parameter update module 2335. These modules are logically connected and can therefore be arbitrarily combined or further split according to their implemented functions. The functions of each module will be described below.

[0089] In some embodiments, the terminal or server can implement the image encoder training method provided in this application by running various computer-executable instructions or computer programs. For example, computer-executable instructions can be microprogram-level commands, machine instructions, or software instructions. Computer programs can be native programs or software modules in an operating system; they can be native applications (APPs), i.e., programs that need to be installed in the operating system to run; or they can be applets that can be embedded in any APP, i.e., programs that only need to be downloaded to a browser environment to run. In summary, the aforementioned computer-executable instructions can be any form of instruction, and the aforementioned computer programs can be any form of application, module, or plugin.

[0090] The training method of the image encoder provided in this application will be described in conjunction with exemplary applications and implementations of the server provided in the embodiments of this application.

[0091] See Figure 3A , Figure 3A This is a first flowchart illustrating the training method of the image encoder provided in this application embodiment, with the server as the main body, and combining... Figure 3A The steps shown are explained.

[0092] In step 101, low-dimensional image features of the first image sample are extracted.

[0093] Here, low-dimensional image features of the first image sample can be extracted using the encoder in a Vector Quantized Generative Adversarial Network (VQGAN). Low-dimensional image features refer to features that can concisely and effectively represent the image information of the first image sample, including at least one of the following: the contour, edge, color, texture, and shape features of the first image sample.

[0094] In some embodiments, see Figure 3B , Figure 3BThis is a schematic diagram of the second process of the training method for the image encoder provided in the embodiments of this application. Figure 3A Step 101, "Extracting low-dimensional image features from the first image sample," can be achieved through... Figure 3B Steps 1011 to 1013 are implemented, and the details are explained below.

[0095] In step 1011, feature extraction is performed on the first image sample to obtain a feature map.

[0096] In some embodiments, features can be extracted from the first image sample using convolutional and pooling layers of a convolutional neural network (CNN) to obtain a feature map, wherein the feature map includes low-level and high-level features of the first image sample.

[0097] For example, convolutional neural networks include any of the following: Visual Geometry Group (VGG), Residual Network (ResNet), and Deep Learning Architecture (Inception). Taking the ResNet50 convolutional neural network as an example, firstly, the preprocessed first image sample is input into the input layer of the ResNet50 network. The size of the first image sample is typically 224x224 pixels, with 3 channels (RGB). A 7x7 convolutional layer processes the first image sample with a stride of 2 and padding of 3. The purpose of the convolutional layer is to extract low-level features of the image, such as edges and colors. The size of the feature map after convolution becomes 112x112, and the number of channels increases to 64. Next, a 3x3 max-pooling layer pools the first image sample with a stride of 2, further reducing the size of the feature map while retaining important feature information. The size of the pooled feature map becomes 56x56, and the number of channels remains 64. ResNet50 contains multiple residual blocks, which consist of several convolutional layers and shortcut connections. The residual blocks are composed of convolutional layers (connections). Their main function is to address the vanishing gradient problem in deep networks and extract higher-level features. The output feature map size of each residual block remains constant, but the number of channels gradually increases. Residual block 1 includes three convolutional layers with 64, 64, and 256 channels respectively, producing a 56x56 output feature map with 256 channels. Residual block 2 includes four convolutional layers with 128, 128, and 512 channels respectively, producing a 28x28 output feature map with 256 channels. The number of channels is 512. Residual block 3 includes 6 convolutional layers with 256, 256, and 1024 channels respectively, and the output feature map size is 14x14 with 1024 channels. Residual block 4 includes 3 convolutional layers with 512, 512, and 2048 channels respectively, and the output feature map size is 7x7 with 2048 channels. After the last residual block, a global average pooling layer is used to compress the feature map size to 1x1 while maintaining the number of channels at 2048, resulting in the final feature map.

[0098] In step 1012, for each first vector in the feature map, the distance between the first vector and each second vector in the preset codebook is determined, and the index of the second vector corresponding to the smallest distance is determined.

[0099] In some embodiments, the distance between the first vector and each second vector in the preset codebook can be determined by any of the following algorithms: Euclidean distance, Manhattan distance, cosine similarity, etc. The index of the second vector can be regarded as the position or identifier of the second vector in the codebook.

[0100] For example, taking Euclidean distance as an example, suppose each first vector in the feature map is an n-dimensional vector, and each second vector in the codebook is also an n-dimensional vector. If each first vector in the feature map is represented as f = [f1, f2, ..., f...] n A second vector in the codebook is represented as c = [c1, c2, ..., c]. n For each first vector f in the feature map, compute its relationship with each second vector c in the codebook. i The Euclidean distance between them can be found in Formula 1 below:

[0101]

[0102] Where d(f,i) represents the distance between the first vector and any second vector in the preset codebook, and c i,j Let c be the j-th element of the i-th second vector in the codebook, and ∑() represents the summation operation. Select the second vector c that has the smallest distance to the first vector from all the second vectors in the codebook. k Where k is the second vector c corresponding to the minimum distance. k The index.

[0103] In step 1013, the indices corresponding to each first vector are combined into low-dimensional image features.

[0104] In some embodiments, after determining the index corresponding to each first vector, these indices represent a discretized representation of the original image features. Each index corresponds to a specific vector in the codebook that captures certain semantic or visual properties of the original image features. Based on the structure of the feature map, these indices are arranged in order of their position in the feature map to form an index sequence or matrix as low-dimensional image features.

[0105] For example, indices can be stored as integers, with the range of each index depending on the size of the codebook. For instance, if the codebook contains 1024 vectors, then the value of each index will be between 0 and 1023. If the feature map is a two-dimensional structure, then the index matrix will maintain the same two-dimensional layout as the feature map, with the index at each position corresponding to the feature vector at that position.

[0106] This application's embodiments compress feature maps into low-dimensional image features, reducing computational and storage costs. Simultaneously, since low-dimensional image features are discretized, tasks such as feature matching and image generation are easier to perform. Furthermore, the vector quantization process can improve the encoder's generalization ability and robustness, enabling the encoder to better capture the semantic information of different images.

[0107] See also Figure 3AIn step 102, the first image encoder to be trained is invoked based on the low-dimensional image features to perform semantic extraction, thereby obtaining the first semantic features of the first image sample.

[0108] Here, the first image encoder can be a causal encoder, an autoencoder, a variational autoencoder (VAE), or a transformer.

[0109] In some embodiments, see Figure 3C , Figure 3C This is a schematic diagram of the third process of the training method for the image encoder provided in the embodiments of this application. Figure 3A Step 102, "Based on low-dimensional image features, call the first image encoder to be trained to perform semantic extraction, and obtain the first semantic features of the first image sample," can be achieved through... Figure 3C Steps 1021 to 1022 are implemented, and the details are explained below.

[0110] In step 1021, the first feature dimension of the low-dimensional image features is converted into the second feature dimension to obtain candidate image features, wherein the second feature dimension is the feature dimension adapted to the first image encoder.

[0111] In some embodiments, feature dimension transformation is essentially a two-way mapping between semantic abstraction levels and structural representations. Feature dimension transformation is typically achieved through linear transformations (such as fully connected layers) or nonlinear transformations (such as multilayer perceptrons). These transformations can adjust the feature dimensions to fit the input requirements of the first image encoder.

[0112] For example, suppose the first feature dimension of a low-dimensional image is 512, while the input feature dimension required by the first image encoder is 1024. A fully connected layer can be used to transform the 512-dimensional features into 1024-dimensional features.

[0113] In some embodiments, see Figure 3D , Figure 3D This is a schematic diagram of the fourth process of the training method for the image encoder provided in the embodiments of this application. Figure 3C Step 1021, "Converting the first feature dimension of the low-dimensional image features into the second feature dimension to obtain candidate image features," can be achieved through... Figure 3D Steps 10211 to 10212 are implemented, and the details are explained below.

[0114] In step 10211, the low-dimensional image features of the first feature dimension are transformed based on the preset weight matrix and bias vector in the pre-trained first-dimensional transformation model.

[0115] A linear transformation is performed to obtain the linear image features of the second feature dimension.

[0116] In some embodiments, the pre-trained first-dimensional transformation model can be a multilayer perceptron (MLP), CNN, recurrent neural network (RNN), generative adversarial network (GAN), or Transformer. The linear transformation is achieved through matrix multiplication and addition operations. Specifically, the product of a pre-defined weight matrix and low-dimensional image features in the first-dimensional transformation model is determined, and the sum of the product and the bias vector is determined. This sum is then used as the linear image feature of the second feature dimension.

[0117] For example, given a low-dimensional image feature x, and the weight matrix W and bias vector b in a pre-trained first-dimensional transformation model, the linear transformation can be expressed as Equation 2 below:

[0118] y = W*x + b (2)

[0119] Here, y represents the linear image feature of the second feature dimension. Assume the low-dimensional image feature x has a dimension of 512, the weight matrix W has a dimension of 1024×512, and the bias vector b has a dimension of 1024×1. Through linear transformation, the resulting linear image feature y has a dimension of 1024.

[0120] In step 10212, the linear image features are nonlinearly transformed based on the activation function in the pre-trained first-dimensional transformation model to obtain candidate image features.

[0121] In some embodiments, nonlinear transformations introduce nonlinearity through activation functions, enabling the learning of more complex feature representations. Common activation functions include the Rectified Linear Unit (ReLU), the sigmoid function, and the hyperbolic tangent function (tanh).

[0122] For example, a nonlinear transformation can be represented as: Where z represents the candidate image feature. This represents the activation function in the pre-trained first-dimensional transformation model.

[0123] Here, multiple linear and nonlinear transformations can be performed on the low-dimensional image features. For example, the low-dimensional image features are input into a pre-trained first-dimensional transformation model, where the first feature dimension of the low-dimensional image features is adapted to the input layer of the pre-trained first-dimensional transformation model. The low-dimensional image features are then linearly transformed using the weight matrix and bias vector of the pre-trained first-dimensional transformation model to obtain the first candidate vector. The first candidate vector is then nonlinearly transformed using an activation function to obtain the second candidate vector. The second candidate vector is then linearly transformed using another weight matrix and bias vector, converting its dimension to the dimension of the output layer, i.e., the second feature dimension. Finally, the activation function of the output layer is applied to the transformed second candidate vector to obtain the candidate image features.

[0124] The linear transformation in this application ensures that the feature dimension matches the input requirements of the subsequent model, facilitating feature fusion and multi-task learning. The non-linearity introduced by the activation function enables the learning of more complex feature representations, improving the model's generalization and expressive capabilities. Using the pre-trained first dimension to transform the weight matrix and bias vector in the model leverages existing knowledge, accelerating the model's convergence speed and improving its performance.

[0125] See also Figure 3C In step 1022, semantic extraction is performed on the candidate image features based on the first image encoder to be trained to obtain the first semantic features of the first image sample.

[0126] Here, semantic extraction involves learning a high-level semantic representation of features through an encoder network. The encoder can be any type of network, such as a CNN, autoencoder, or transformer. The encoder extracts semantic information from features progressively through a multi-layered neural network structure. A pre-trained CNN encoder, such as ResNet50, is used to process the transformed 1024-dimensional candidate image features. The encoder extracts higher-level semantic features through its convolutional and pooling layers, ultimately outputting the first semantic feature of the first image sample.

[0127] This application's embodiments ensure compatibility with different feature sources through feature dimension transformation, facilitating feature fusion and multi-task learning. The encoder, through learning, extracts more semantic information, improving the expressive power of features and the generalization ability of the model. The transformation and semantic extraction of low-dimensional features are generally more efficient than directly processing high-dimensional features, reducing computational resource consumption.

[0128] In some embodiments, see Figure 3E , Figure 3E This is a schematic diagram of the fifth process of the training method for the image encoder provided in the embodiments of this application. Figure 3CStep 1022, "Semantic extraction of candidate image features based on the first image encoder to be trained, to obtain the first semantic features of the first image sample," can be achieved through... Figure 3E Steps 10221 to 10225 are implemented, and the details are explained below.

[0129] In step 10221, the candidate image features are converted into a feature sequence, wherein the feature sequence includes multiple elements, each element representing at least a portion of the features of the first image sample.

[0130] In some embodiments, candidate image features are represented as two-dimensional or three-dimensional feature maps. If the feature map is a three-dimensional tensor, two dimensions correspond to spatial location (height and width), and the third dimension corresponds to feature channels (depth). Converting candidate image features into a feature sequence typically involves unfolding the two-dimensional or three-dimensional feature map of the image into a one-dimensional sequence. This can be achieved by sequentially arranging the feature vectors for each spatial location of the feature map. Each element in the feature sequence represents at least a portion of the features of the corresponding spatial location in the original image. These features include at least one of the following: color, texture, edges, shape, etc. The structure of the feature sequence reflects the spatial structure of the original image. For example, if the feature map was extracted from the image through a convolutional layer, the order of elements in the feature sequence may reflect the order of pixels in the image.

[0131] For example, suppose the candidate image features are a feature map of size H×W×D, where H and W are the height and width, and D is the feature dimension. After converting it into a feature sequence, the sequence length is H×W, and each element is a D-dimensional feature vector.

[0132] In step 10222, the feature sequence is positionally encoded to obtain a first feature representation that includes the positional information of each element.

[0133] In some embodiments, positional encoding adds information about the element's position within the feature sequence to each element. Positional encoding includes fixed-positional encoding and learnable-positional encoding. For fixed-positional encoding, a predefined function is used to generate position vectors that do not change during training. For example, sinusoidal positional encoding is a common fixed-positional encoding method that generates a vector containing sine and cosine functions for each position. For learnable-positional encoding, the position vectors are used as parameters of the model for training, and a positional embedding layer is added to the model so that the model can learn the positional information best suited for the task.

[0134] For example, for each position i, a position vector p is generated. i Its dimension is the same as the dimension of the feature vector. Each dimension p of the position vectori,2j and p i,2j+1 Calculated using the following formulas 3 and 4 respectively:

[0135]

[0136] Where D is the dimension of the feature vector, j is the index of the dimension, sin is the sine function, and cos is the cosine function. The generated position vector p... i Corresponding element f in the feature sequence i Adding or concatenating them yields the first feature representation f containing location information. i For example, f i ′=p i +f i , or, f i ′=[p i ·f i ].

[0137] In step 10223, self-attention processing is performed on each element in the first feature representation to obtain the attention weight of each element.

[0138] In some embodiments, for each element in the first feature representation, a query vector, a key vector, and a value vector are generated through three different linear transformations; for each query vector, the similarity between the query vector and all key vectors is determined, and the similarity is scaled; the scaled similarity metric is converted into attention weights through a loss function.

[0139] Following the example of step 10222 above, for the first feature representation f i Each element f in ' i ", the query vector Q is generated through three different linear transformations. i Key vector K i Sum vector V i These linear transformations are typically implemented through fully connected layers, with weight matrices W... Q W K and W V , where Q i =W Q ·f″ i K = W K ·f″ i V i =W V ·f″ i For each query vector Q i Calculate the query vector Q i With all key vectors K j Similarity S i,j =Q i ·K jTo scale the similarity metric, it is usually divided by . Where, d k If the dimension is the key vector, then the scaled similarity measure... The scaled similarity metric is converted into attention weights α using a softmax function. i,j =softmax(S′) i,j ).

[0140] In step 10224, multiple elements are weighted and summed based on attention weights to obtain the second feature representation.

[0141] Following the example in step 10223 above, the calculated attention weight α is used. i,j For all value vectors V i By performing a weighted summation, we obtain the weighted eigenvector, which is also known as the second feature representation.

[0142] In step 10225, a nonlinear transformation is performed on the second feature representation to obtain the first semantic feature of the first image sample.

[0143] In some embodiments, the second feature representation is nonlinearly transformed by a fully connected layer and an activation function to obtain the first semantic feature of the first image sample.

[0144] Following the example in step 10224 above, the ReLU activation function is used to activate the second feature representation a. i A nonlinear transformation is performed to obtain the first semantic feature s = ReLU(W·h+b), where W and b are the learnable weight matrix and bias vector, respectively.

[0145] This application's embodiments, through positional encoding and self-attention mechanisms, can better capture spatial relationships and long-range dependencies in images. The nonlinear transformation step enhances the semantic information of the feature representation, improving the encoder's performance across various tasks. This method can be applied to various image processing tasks, such as classification, detection, and segmentation, demonstrating broad applicability.

[0146] See also Figure 3A In step 103, high-dimensional image features of the first image sample are extracted, and first text features of the first text sample used to describe the first image sample are extracted, wherein the dimension of the high-dimensional image features is greater than the dimension of the low-dimensional image features.

[0147] In some embodiments, high-dimensional features refer to multi-dimensional representations formed by a series of transformations or mappings of the original image data. High-dimensional image features and first text features can be extracted using any of the following models: Contrastive Language-Image Pre-training (CLIP), Bidirectional Encoder Representations from Transformers (BERT), or Detachment Transformer (DETR).

[0148] For example, CLIP's image encoder is a Vision Transformer (ViT) based model. ViT segments the first image sample into several local patches, such as 16x16 pixel patches. Then, each patch is transformed into a vector through a linear projection. These vectors are fed into a series of Transformer blocks. Each Transformer block includes a multi-head self-attention mechanism and a feedforward neural network. Through these Transformer blocks, the image encoder learns a high-dimensional feature representation of the first image sample, which serves as the high-dimensional image feature of the first image sample. CLIP's text encoder is a Transformer-based language model. It first transforms the first text sample into a series of word embeddings, and then processes these embeddings through a series of Transformer blocks. Each Transformer block also includes a multi-head self-attention mechanism and a feedforward neural network. Through these Transformer blocks, the text encoder learns a high-dimensional feature representation of the text, which serves as the first text feature of the first text sample.

[0149] In step 104, a first distillation loss is determined based on the first semantic features and the high-dimensional image features, and a first contrast loss is determined based on the first semantic features and the first text features.

[0150] Here, distillation loss is commonly used in the knowledge distillation process, a technique for transferring knowledge from a large model (teacher model) to a smaller model (student model). The goal of distillation loss is to make the output of the student model as close as possible to the output of the teacher model, thereby inheriting the teacher model's knowledge. The definition of distillation loss typically includes the following parts:

[0151] Soft targets: The output probability distributions of the teacher model on the training data are often referred to as soft targets. Soft targets provide relative confidence about each class, rather than just the single class label in hard targets.

[0152] Temperature parameter: To make soft targets smoother, a temperature parameter is usually used to adjust the probability distribution of the teacher model's output. The temperature parameter is usually a number greater than 1, which makes the probability distribution more uniform and reduces uncertainty.

[0153] Loss function: Distillation loss typically uses Kullback-Leibler divergence to measure the difference between the student model's output and the teacher model's soft objective. Kullback-Leibler divergence measures the difference between two probability distributions, and the goal of distillation loss is to minimize this difference.

[0154] In some embodiments, see Figure 3F , Figure 3F This is a schematic diagram of the sixth process of the training method for the image encoder provided in the embodiments of this application. Figure 3A Step 104, "Determining the first distillation loss based on the first semantic features and high-dimensional image features," can be achieved through... Figure 3F Steps 1041 to 1042 are implemented, and the details are explained below.

[0155] In step 1041, the first semantic feature is mapped to the feature space of the high-dimensional image feature to obtain the high-dimensional semantic feature.

[0156] In some embodiments, the dimension of the high-dimensional semantic features is greater than the dimension of the low-dimensional semantic features, which are obtained by mapping the first semantic features to a low-dimensional image feature space. The high-dimensional semantic features are obtained by mapping the first semantic features to the feature space of the high-dimensional image features through a linear or non-linear mapping function, such as a fully connected layer or a multilayer perceptron (MLP). The goal of the mapping is to make the first semantic features and the high-dimensional image features comparable in the same feature space.

[0157] For example, suppose the high-dimensional image features have a dimension of 2048, while the semantic features have a dimension of 768 (such as the output dimension of BERT). The 768-dimensional semantic features can be mapped to the 2048-dimensional feature space through a fully connected layer.

[0158] In step 1042, based on a preset loss function, the difference between high-dimensional semantic features and high-dimensional image features is mapped to a first distillation loss.

[0159] In some embodiments, the difference between the mapped high-dimensional semantic features and high-dimensional image features is calculated using a loss function and mapped as a first distillation loss. The goal of the distillation loss is to minimize the difference between the high-dimensional semantic features and the high-dimensional image features, thereby achieving knowledge distillation.

[0160] For example, the loss function can be any of the following: Kullback-Leibler Divergence, Mean Squared Error (MSE), or Cross-Entropy Loss. First, the high-dimensional semantic features and high-dimensional image features are normalized, typically by converting the feature vectors into a probability distribution using a softmax function. Then, the difference between the two is calculated using the chosen loss function. For example, when using Kullback-Leibler Divergence, the loss function can be expressed as: L1 = KL(S∥I), where L1 represents the first distillation loss, KL represents the Kullback-Leibler loss function, S represents the high-dimensional semantic features, and I represents the high-dimensional image features.

[0161] This application combines semantic features and image features to achieve multimodal data fusion. Through distillation loss, knowledge from image features is transferred to semantic features, improving the expressive power of the semantic features. The mapping and distillation processes can be adjusted according to different task requirements, exhibiting good adaptability.

[0162] In some embodiments, see Figure 3G , Figure 3G This is a schematic diagram of the seventh process of the training method for the image encoder provided in the embodiments of this application. Figure 3A Step 104, "Determine the first contrastive loss based on the first semantic feature and the first text feature," can be achieved through... Figure 3G Steps 1043 to 1045 are implemented, and the details are explained below.

[0163] In step 1043, the first image sample and the first text sample are combined into a positive sample, wherein the first similarity between the first semantic feature of the first image sample and the first text feature of the first text sample is greater than the similarity threshold.

[0164] In some embodiments, positive samples are pairs of related images and text that describe the same or similar content. The purpose of positive samples is to enable the model to learn the correlation between images and text.

[0165] For example, the first image sample could be an image of a dog, and the first text sample could be the text "A golden dog is playing in the grass." The first similarity between the first semantic feature and the first text feature is determined using any of the following algorithms: cosine similarity, Euclidean distance, Manhattan distance, and Pearson correlation coefficient. Taking cosine similarity as an example, the vector product of the first semantic feature and the first text feature is determined; the length product of the length of the first semantic feature and the length product of the first text feature is determined; and the ratio of the vector product to the length product is determined as the similarity. For example, if the first semantic feature is m and the first text feature is n, then the vector product is m*n; if the length of the first semantic feature is |m| and the length of the first text feature is |n|, then the length product is |m|*|n|; and the ratio of the vector product to the length product is (m*n) / (|m|*|n|), which is used as the first similarity.

[0166] In step 1044, the third image sample and the third text sample are combined into a negative sample, wherein the second similarity between the image sample features of the third image sample and the third text sample features of the third text sample is less than the similarity threshold.

[0167] In some embodiments, negative samples are pairs of unrelated images and text that describe different content. The purpose of negative samples is to enable the model to learn how to distinguish between unrelated images and text.

[0168] For example, the third image sample could be an image of a dog, and the third text sample could be the text "a red sports car is driving on the highway." The first similarity between the first semantic feature and the first text feature is determined using any of the following algorithms: cosine similarity, Euclidean distance, Manhattan distance, and Pearson correlation coefficient. Taking Euclidean distance as an example, if the image sample feature is (x1, y1) and the third text sample feature is (x2, y2), the distance between the image sample feature and the third text sample feature is determined as follows: Used as the second similarity.

[0169] In step 1045, the difference between positive and negative samples is converted into a first contrast loss using a preset contrast loss function.

[0170] In some embodiments, the contrastive loss function is used to measure the difference between positive and negative samples. Common contrastive loss functions include contrastive loss and triplet loss.

[0171] For example, the formula for the comparison function is shown in Formula 5 below:

[0172]

[0173] Where L2 represents the first contrastive loss, N is the total number of sample pairs, i.e., the total number of positive and negative sample pairs, i is an index variable used to iterate through all sample pairs, from 1 to N, and y i Here, is the label (1 represents a positive sample, 0 represents a negative sample), d is the distance metric (such as Euclidean distance) used to calculate the distance between two feature vectors, and m is a predefined margin used to control the distance difference between positive and negative sample pairs. It is a feature vector in the i-th positive sample pair, usually represented as image features or text features, z i It is another feature vector in the i-th sample pair, and Pairing, in positive samples, z i Usually indicates and The characteristic of another associated modality (e.g., if) If it's an image feature, then z i (These are textual features) It is a feature vector in the i-th negative sample pair, usually represented as z i Unrelated feature vectors.

[0174] This application's embodiments clarify the relationship between images and text through positive samples, helping the model learn correct alignment. Negative samples help the model learn how to distinguish between unrelated images and text, improving the model's discriminative ability. The combination of positive and negative samples clarifies the encoder's learning objective: learning the relationship and discriminative power between images and text. The contrastive loss function improves the alignment of image and text features by optimizing the difference between positive and negative samples.

[0175] See also Figure 3A In step 105, the parameters of the first image encoder are updated based on the first distillation loss and the first contrast loss to obtain the trained first image encoder.

[0176] In some embodiments, see Figure 3H , Figure 3H This is the eighth flowchart of the training method for the image encoder provided in the embodiments of this application. Figure 3A Step 105, "Updating the parameters of the first image encoder based on the first distillation loss and the first contrast loss to obtain the trained first image encoder," can be achieved through... Figure 3H Steps 1051 to 1052 are implemented, and the details are explained below.

[0177] In step 1051, the first distillation loss and the first comparison loss are updated and merged to obtain the total loss.

[0178] In some embodiments, the fusion can be achieved by adding the first distillation loss and the first contrast loss to obtain the total loss.

[0179] For example, if the first distillation loss is L1 and the first contrast loss is L2, then the total loss can be L1 + L2.

[0180] In some embodiments, see Figure 3I , Figure 3I This is a ninth flowchart illustrating the training method for the image encoder provided in this application embodiment. Figure 3H Step 1051, "Fusing the updates of the first distillation loss and the first comparative loss to obtain the total loss," can be achieved through... Figure 3I Steps 10511 to 10512 are implemented, and the details are explained below.

[0181] In step 10511, a first dynamic weight for the first distillation loss is determined, and a second dynamic weight for the first contrast loss is determined, wherein the first dynamic weight and the second dynamic weight are dynamically updated based on training parameters of at least one dimension for each update.

[0182] Here, the training parameters for at least one dimension updated each time may include at least one of the following: the epoch of update, gradient, loss value, and learning rate. The first dynamic weight and the second dynamic weight are used to more effectively balance the importance of different first distillation losses and first contrastive losses during training.

[0183] In some embodiments, see Figure 3J , Figure 3J This is a schematic diagram of the tenth process of the training method for the image encoder provided in the embodiments of this application. Figure 3I Step 10511, "Determining the first dynamic weight of the first distillation loss and determining the second dynamic weight of the first contrast loss," can be achieved through... Figure 3J Steps 201A to 203A are implemented, and the details are explained below.

[0184] In step 201A, the loss magnitudes of the first distillation loss and the first comparison loss are determined, wherein the loss magnitude is the difference between the maximum loss and the minimum loss within a preset loss range.

[0185] In some embodiments, the maximum loss refers to the maximum achievable loss value preset during each update round of the first distillation loss and the first comparison loss, and the minimum loss refers to the minimum achievable loss value preset during each update round of the first distillation loss and the first comparison loss. The loss magnitude does not change with each update round.

[0186] For example, the preset loss range is <0.05, 0.1>, the maximum loss is 0.1, the minimum loss is 0.05, and the loss magnitude is the difference between the maximum loss and the minimum loss, which is 0.05.

[0187] Here, for each round of updates, steps 202A to 203A are performed.

[0188] In step 202A, the progress of the current update round in the preset total update rounds is determined.

[0189] In some embodiments, the ratio of the current update round to the preset total update rounds is determined as the progress of the current update round within the preset total update rounds.

[0190] For example, if the preset total number of update rounds is 10 and the current update round is 5, then the progress of the current update round in the preset total number of update rounds is 5 / 10, which is 50%.

[0191] In step 203A, based on the magnitude of the loss, losses that are negatively correlated with the schedule are determined within the loss range as a first dynamic weight, and losses that are positively correlated with the schedule are determined within the loss range as a second dynamic weight.

[0192] In some embodiments, the product of the loss magnitude and the progress is determined; the first difference between the maximum loss and the product is used as a first dynamic weight; and the second sum of the minimum loss and the product is used as a second dynamic weight. That is, as the number of update rounds increases, the first dynamic weight gradually decreases, and the second dynamic weight gradually increases.

[0193] For example, the loss magnitude is g, the progress is h, the product of the loss magnitude and the progress is g*h, and the maximum loss is L. max The minimum loss is L min The first difference between the maximum loss and the product is L. max -g*h is used as the first dynamic weight; the second sum of the minimum loss and the product is L. min +g*h is used as the second dynamic weight.

[0194] In some embodiments, the first dynamic weight and the second dynamic weight may be normalized. Specifically, the sum of the first dynamic weight and the second dynamic weight is determined; the ratio of the first dynamic weight to the sum is determined as the weight of the first distillation loss; and the ratio of the second dynamic weight to the sum is determined as the weight of the first contrast loss.

[0195] For example, the first dynamic weight is d. iThe second dynamic weight is d2, and the sum is d1+d2. Then the weight of the first distillation loss is d1 / (d1+d2), and the weight of the first comparison loss is d2 / (d1+d2).

[0196] This application embodiment enables the first image encoder to adaptively adjust the importance of different losses through dynamic loss weights, improving training flexibility. By adjusting weights in relation to training progress, the first image encoder can optimize different tasks at different stages of training, improving overall performance. Dynamic weight adjustment helps the first image encoder converge to a better solution more quickly, improving training efficiency.

[0197] In some embodiments, see Figure 3K , Figure 3K This is a schematic diagram of the eleventh step of the training method for the image encoder provided in the embodiments of this application. Figure 3I Step 10511, "Determine the first dynamic weight of the first distillation loss and determine the second dynamic weight of the first contrast loss," can also be updated for each round by executing... Figure 3K Steps 201B to 203B are implemented, and the details are explained below.

[0198] In step 201B, the first gradient of the first distillation loss and the second gradient of the first contrast loss are determined.

[0199] In some embodiments, the partial derivative of the first distillation loss with respect to the parameters of the first image encoder is determined as a first gradient of the first distillation loss; the partial derivative of the first contrast loss with respect to the parameters of the first image encoder is determined as a second gradient of the first contrast loss.

[0200] For example, if the parameters of the first image encoder are θ, the first distillation loss is L1, and the first contrast loss is L2, then the first gradient of the first distillation loss is... The second gradient of the first contrastive loss is

[0201] In step 202B, the first gradient norm of the first gradient and the second gradient norm of the second gradient are determined.

[0202] In some embodiments, the gradient norm is the square root of the sum of squares of the elements in the gradient vector, providing a scalar value representing the overall magnitude or strength of the gradient. For each gradient, the elements in the gradient vector are squared to obtain the squared gradient elements; the squared gradient elements are summed to obtain the sum of squares; and the square root of the sum of squares is taken to obtain the gradient norm.

[0203] For example, regarding the first gradient, if the first gradient vector is [g1, g2, g3], the squared gradient elements are respectively The sum of squares of the gradient elements is The first gradient norm is For the second gradient, if the second gradient vector is [m1, m2, m3], the squared gradient elements are respectively The sum of squares of the gradient elements is The second gradient norm is

[0204] In step 203B, a first dynamic weight negatively correlated with the first gradient norm is determined, and a second dynamic weight negatively correlated with the second gradient norm is determined.

[0205] In some embodiments, the first reciprocal of the first gradient norm is used as the first dynamic weight; and the second reciprocal of the second gradient norm is used as the second dynamic weight.

[0206] Following the example in step 202B above, the first reciprocal of the first gradient norm is: As the first dynamic weight; the second reciprocal of the second gradient norm is... As the second dynamic weight.

[0207] In some embodiments, the first dynamic weight and the second dynamic weight may be normalized. Specifically, the sum of the first dynamic weight and the second dynamic weight is determined; the ratio of the first dynamic weight to the sum is determined as the weight of the first distillation loss; and the ratio of the second dynamic weight to the sum is determined as the weight of the first contrast loss.

[0208] For example, if the first dynamic weight is d1 and the second dynamic weight is d2, and the sum is d1+d2, then the weight of the first distillation loss is d1 / (d1+d2) and the weight of the first contrast loss is d2 / (d1+d2).

[0209] The embodiments of this application determine dynamic weights based on the gradient norm, which can sensitively reflect the impact of loss on parameters. Adjusting the dynamic weights helps the first image encoder find a balance between different losses, improving the overall optimization effect. Through gradient-based dynamic weight adjustment, the first image encoder is more robust to gradient changes and can better cope with complex training environments.

[0210] See also Figure 3I In step 10512, the first distillation loss and the first comparison loss are weighted and summed based on the first dynamic weight and the second dynamic weight to obtain the total loss.

[0211] For example, if the first distillation loss is L1, the first contrast loss is L2, the first dynamic weight of the first distillation loss is T1, and the second dynamic weight of the first contrast loss is T2, then the total loss is T1·L1+T2·L2.

[0212] See also Figure 3H In step 1052, the parameters of the first image encoder are updated based on the total loss to obtain the trained first image encoder.

[0213] In some embodiments, optimization algorithms, such as Adaptive Moment Estimation (Adam) or Stochastic Gradient Descent (SGD), are used to update the parameters of the first image encoder based on the total loss. The optimization objective is to minimize the total loss, thereby improving the model's performance. The total loss is backpropagated to update the parameters of the first image encoder. This process of calculating the total loss and updating the parameters is iterated multiple times until the global loss converges, at which point the iteration process stops, resulting in the trained first image encoder.

[0214] For example, taking the Adam optimizer as an example, assuming the learning rate is η, the parameter update steps can be expressed as the following formula 6:

[0215]

[0216] Where θ is a parameter of the first image encoder, θ new These are the parameters for the new round of training, θ old These are the parameters from the previous training round, and L3 is the total loss. This is the gradient of the total loss with respect to the parameters. Backpropagation is implemented using the backpropagation algorithm, which calculates the gradient of each neuron from the output layer to the input layer, and updates the weights and biases of the neurons based on the gradients. Gradient descent is used to continuously update the parameters, thereby reducing the loss value. Various gradient descent algorithms can be used, such as batch gradient descent, stochastic gradient descent, adaptive gradient descent, and momentum gradient descent.

[0217] In this embodiment, the performance of the trained first image encoder is improved in tasks such as image-text matching. The first image encoder also maintains good performance on unseen data and has better generalization ability.

[0218] In some embodiments, see Figure 3L , Figure 3L This is a schematic diagram of the twelfth step of the training method for the image encoder provided in this application embodiment. In step 105, the following steps are executed: Figure 3LSteps 301 to 302 are explained in detail below.

[0219] In step 301, the parameters of the pre-trained second image encoder and the pre-trained first multimodal coding model are frozen. The pre-trained second image encoder is used to extract low-dimensional image features of the first image sample, and the pre-trained first multimodal coding model is used to extract high-dimensional image features and first text features.

[0220] Here, the second image encoder can be any of the following: VQGAN, CNN encoder, Transformer, or autoencoder. The pre-trained first multimodal coding model can be any of the following: CLIP, BERT, multimodal autoencoder, deep multimodal fusion model, or generative multimodal model. When updating the parameters of the first image encoder based on the first distillation loss and the first contrastive loss, the parameters of the pre-trained second image encoder and the pre-trained first multimodal coding model remain unchanged.

[0221] In step 302, the parameters of the first dimension transformation model are updated based on the first distillation loss and the first contrast loss to obtain the trained first dimension transformation model, wherein the first dimension transformation model is used to transform the first feature dimension of the low-dimensional image features into a second feature dimension adapted to the first image encoder.

[0222] In some embodiments, the total loss of the first distillation loss and the first contrast loss is determined, and the total loss is backpropagated to update the parameters of the first dimension transformation model. The process of calculating the total loss and updating the parameters is repeated multiple times until the global loss converges, at which point the iteration process stops, and the trained first dimension transformation model is formed.

[0223] For example, the first distillation loss and the first contrastive loss can be directly added together to obtain the total loss, or the first distillation loss and the first contrastive loss can be dynamically weighted to obtain the total loss. Backpropagation is implemented using the backpropagation algorithm, calculating the gradient of each neuron from the output layer to the input layer, and updating the neuron's weights and biases based on the gradients. Gradient descent is used to continuously update the parameters, reducing the loss value. Various gradient descent algorithms can be used, such as batch gradient descent, stochastic gradient descent, adaptive gradient descent, and momentum gradient descent.

[0224] This embodiment of the application reduces the total number of parameters and lowers the risk of overfitting by freezing the parameters of the pre-trained second image encoder and the pre-trained first multimodal coding model. It utilizes the feature representations learned by the pre-trained model while allowing other parts to be adaptively adjusted. The first dimension transformation model enables the first image encoder to flexibly handle feature inputs of different dimensions, improving its versatility.

[0225] Here, in step 105, the following processing can also be performed: based on the first distillation loss and the first contrast loss, update the parameters of the pre-trained second image encoder, the pre-trained first multimodal coding model and the first dimension transformation model respectively, to obtain the fourth image encoder, the third multimodal coding model and the trained first dimension transformation model.

[0226] This application embodiment achieves end-to-end optimization by simultaneously updating the parameters of all models, enabling the entire system to work collaboratively and improving overall performance. Fine-tuning the parameters of the pre-trained model for a specific task allows the model to better adapt to the data distribution and requirements of the current task, improving its adaptability. Updating the parameters of the second image encoder and the first multimodal encoding model improves the quality of feature representation, making the extracted features more suitable for the current task. Updating the parameters of the first dimension transformation model enables it to more effectively convert low-dimensional image features into high-dimensional features adapted to the first image encoder, improving the accuracy of feature transformation.

[0227] In some embodiments, see Figure 3M , Figure 3M This is a schematic diagram of the thirteenth step of the training method for the image encoder provided in this application embodiment. After step 302, the following steps are executed: Figure 3M Steps 303 to 307 are explained in detail below.

[0228] In step 303, if the updates of the first image encoder and the first dimension transformation model meet the preset conditions, the second image encoder to be trained is called based on the second image sample to obtain the low-dimensional sample features of the second image sample. The preset conditions include at least one of the following: the number of updated rounds is greater than or equal to the round threshold, and at least one of the first distillation loss and the first contrast loss is less than the loss threshold after the number of updated rounds.

[0229] In some embodiments, if the updates of the first image encoder and the first dimension transformation model meet the preset conditions, feature extraction is performed on the second image sample to obtain a candidate feature map; for each third vector in the candidate feature map, the distance between the third vector and each fourth vector in the preset codebook is determined, and the target index of the fourth vector corresponding to the smallest distance is determined; the target index corresponding to each third vector is combined into a low-dimensional sample feature.

[0230] For example, if the round threshold is 10 and the number of rounds updated is 11, then the second image encoder to be trained is called based on the second image sample to obtain the low-dimensional sample features of the second image sample; or, if the loss threshold is 0.08 and the first distillation loss is 0.07 after the number of rounds updated, then the second image encoder to be trained is called based on the second image sample to obtain the low-dimensional sample features of the second image sample.

[0231] In step 304, the trained first image encoder is invoked based on the low-dimensional sample features to perform semantic extraction, thereby obtaining the sample semantic features of the second image sample.

[0232] In some embodiments, the first feature dimension of the low-dimensional sample features is converted into a second feature dimension to obtain the target image features, wherein the second feature dimension is a feature dimension adapted to the first image encoder; based on the trained first image encoder, semantic extraction is performed on the target image features to obtain the sample semantic features of the second image sample.

[0233] For example, based on the preset weight matrix and bias vector in the first-dimensional transformation model after training, a linear transformation is performed on the low-dimensional sample features of the first feature dimension to obtain the low-dimensional sample features of the second feature dimension; based on the activation function in the first-dimensional transformation model after training, a non-linear transformation is performed on the low-dimensional sample features of the second feature dimension to obtain the target image features; semantic extraction is the high-level semantic representation of features learned by the encoder network. The encoder can be any of the following types: CNN, autoencoder, transformer, etc. The encoder gradually extracts the semantic information of the target image features through a multi-layer neural network structure to serve as the sample semantic features of the second image samples.

[0234] In step 305, high-dimensional sample features of the second image sample are extracted based on the first multimodal coding model to be trained, and second text sample features of the second text sample are extracted. The dimension of the high-dimensional sample features is greater than the dimension of the low-dimensional sample features. The second text sample is used to describe the second image sample.

[0235] In some embodiments, high-dimensional features refer to multi-dimensional representations formed by a series of transformations or mappings of the original image data. High-dimensional sample features and second text features can be extracted using any of the following models: CLIP, BERT, or DETR.

[0236] For example, ViT segments the second image sample into several local patches. Each patch is transformed into a vector through a linear projection. These vectors are fed into a series of Transformer blocks, each including a multi-head self-attention mechanism and a feedforward neural network. Through these Transformer blocks, the image encoder learns a high-dimensional feature representation of the second image sample, which serves as the high-dimensional sample feature of the second image sample. Similarly, the second text sample is transformed into a series of word embeddings, which are then processed through a series of Transformer blocks. Each Transformer block also includes a multi-head self-attention mechanism and a feedforward neural network. Through these Transformer blocks, the text encoder learns a high-dimensional feature representation of the text, which serves as the second text feature of the second text sample.

[0237] In step 306, a second distillation loss is determined based on the sample semantic features and high-dimensional sample features, and a second contrast loss is determined based on the sample semantic features and the second text sample features.

[0238] In some embodiments, the semantic features of the samples are mapped to the feature space of high-dimensional sample features to obtain high-dimensional sample semantic features; based on a preset loss function, the difference between the high-dimensional sample semantic features and the high-dimensional sample features is mapped to a second distillation loss. The second image sample and the second text sample are combined into a positive sample, wherein the similarity between the semantic features of the second image sample and the second text features of the second text sample is greater than a similarity threshold; the fourth image sample and the fourth text sample are combined into a negative sample, wherein the similarity between the image features of the fourth image sample and the fourth text features of the fourth text sample is less than a similarity threshold; the difference between the positive and negative samples is converted into a second contrast loss using a preset contrast loss function.

[0239] For example, the loss function used to determine the second distillation loss can be any of the following: KL divergence, MSE, or cross-entropy loss. Contrastive loss functions are used to measure the difference between positive and negative samples; common contrastive loss functions include contrastive loss and triplet loss.

[0240] In step 307, the parameters of the trained first image encoder, the trained first dimension transformation model, the second image encoder to be trained, and the first multimodal coding model to be trained are updated based on the second distillation loss and the second contrast loss, respectively, to obtain the third image encoder, the second dimension transformation model, the trained second image encoder, and the trained multimodal coding model.

[0241] In some embodiments, the total loss of the second distillation loss and the second contrast loss is determined, and the total loss is backpropagated to update the parameters of the trained first image encoder, the trained first dimension transformation model, the second image encoder to be trained, and the first multimodal coding model to be trained. The process of calculating the total loss and updating the parameters is repeated multiple times until the global loss converges, and the iteration process is stopped to obtain the third image encoder, the second dimension transformation model, the trained second image encoder, and the trained multimodal coding model.

[0242] For example, the second distillation loss and the second contrastive loss can be directly added together to obtain the total loss, or the second distillation loss and the second contrastive loss can be dynamically weighted to obtain the total loss. Backpropagation is implemented using the backpropagation algorithm, which calculates the gradient of each neuron from the output layer to the input layer, and updates the weights and biases of the neurons based on the gradients. Gradient descent is used to continuously update the parameters, thereby reducing the loss value. Various gradient descent algorithms can be used, such as batch gradient descent, stochastic gradient descent, adaptive gradient descent, and momentum gradient descent.

[0243] This application's embodiments improve the cross-modal understanding capability of a first image encoder by combining image and text information. More effective feature representations are learned through distillation and contrastive loss. The overall performance of the encoder is improved through end-to-end training and optimization. The encoder's adaptability is improved by fine-tuning the pre-trained model on specific tasks. The encoder's generalization capability is improved by fine-tuning across multiple models.

[0244] In some embodiments, see Figure 3N , Figure 3N This is a schematic diagram of the fourteenth step of the training method for the image encoder provided in this application embodiment. After step 105, the following steps are executed: Figure 3N Steps 106 to 108 are explained in detail below.

[0245] In step 106, the trained first image encoder is invoked based on the first image to obtain the second semantic features of the first image.

[0246] Here, the first image is preprocessed, including scaling and normalization, to adapt to the input requirements of the trained first image encoder. The processing order of pixels in the preprocessed first image is determined, usually from left to right and from top to bottom, and the pixels of the first image are input into the trained first image encoder in sequence. The first image encoder processes the first image pixel by pixel, and the encoding of each pixel depends on the previously encoded pixels. During the encoding process, the first image encoder extracts the semantic features of each pixel. The semantic features of multiple pixels are aggregated to obtain the second semantic features of the first image.

[0247] In step 107, second text features are extracted from the first text, and a pre-trained second multimodal coding model is called based on the second text features and the second semantic features to generate an image, thereby obtaining a second image. The first text includes modified parameters for the first image.

[0248] In some embodiments, see Figure 4 , Figure 4 This is a schematic diagram of the process of adjusting an image based on text, provided in an embodiment of this application. Second text features can be extracted from the first text using any one of CLIP, BERT, or DETR; the second text features and the second semantic features are fused through the fusion layer of a pre-trained second multimodal coding model to obtain fused features; the fused features are then input into the decoder part of the pre-trained second multimodal coding model to generate the second image.

[0249] For example, CLIP's text encoder is a Transformer-based language model. It first transforms the first text into a series of word embeddings, then processes these embeddings through a series of Transformer blocks. Each Transformer block also includes a multi-head self-attention mechanism and a feedforward neural network. Through these Transformer blocks, the text encoder learns a high-dimensional feature representation of the text, which serves as the second text feature for the first text sample. The generated second image should reflect the modified parameters specified in the first text; for example, if the first text indicates increased brightness, then the second image should be a brighter version of the first image. See also... Figure 5 , Figure 5 This is an application illustration of adjusting an image based on text provided in an embodiment of this application. The first image can be an image of a puppy, and the first text can be "modify the first image into a second image of 'a puppy playing with a kitten'". Based on the second text features of the first text and the second semantic features of the first image, a pre-trained second multimodal coding model is invoked to generate the second image, which is the image of a puppy playing with a kitten.

[0250] In step 108, third text features are extracted from the second text, and a pre-trained second multimodal coding model is called based on the third text features and the second semantic features to obtain the matching degree between the first image and the second text, wherein the second text is used to describe the first image.

[0251] In some embodiments, natural language processing techniques are used to convert the second text into a numerical vector representation, i.e., the third text features; the third text features and the second semantic features are input into the fusion layer of a pre-trained second multimodal coding model to obtain fused features; the fused features are input into the multilayer perceptron of the pre-trained second multimodal coding model, and after passing through several fully connected layers and activation functions, a matching score is finally output as the degree of matching between the first image and the second text.

[0252] For example, the third text feature is a 768-dimensional vector extracted by the BERT model, representing the semantic information of the second text; the second semantic feature is a 2048-dimensional vector extracted by the causal encoder, representing the semantic information of the first image; the third text feature and the second semantic feature are input into the fusion layer of the pre-trained second multimodal coding model to obtain a 9728-dimensional fused feature; the fused feature is input into a multilayer perceptron, and after several fully connected layers and activation functions, a matching score, such as 0.92, is finally output as the degree of matching between the first image and the second text.

[0253] This application combines image and text information to improve the cross-modal understanding capability of the first image encoder. Precise modification of images via text instructions enables controllability and creativity in image generation. The degree of matching between the image and text is evaluated to measure the understanding and generation capabilities of the first image encoder.

[0254] In some embodiments, see Figure 3O , Figure 3O This is a schematic diagram of the fifteenth step of the training method for the image encoder provided in this application embodiment. After step 105, the following steps are executed: Figure 3O Steps 109 to 113 are explained in detail below.

[0255] In step 109, the initialized image feature vector and the third text used to describe the third image to be generated are obtained.

[0256] In some embodiments, the initialized image feature vector can be a randomly generated vector or a vector sampled from a predefined distribution, used as the initial representation of the generated image. The feature dimensions of the initialized image feature vector are adapted to the first image encoder after training.

[0257] For example, the initial image feature vector can be obtained by sampling from a Gaussian distribution, a uniform distribution, or other statistical distributions.

[0258] In step 110, feature extraction is performed on the third text to obtain the fourth text features of the third text.

[0259] In some embodiments, fourth text features can be extracted from the third text using any one of CLIP, BERT, or DETR.

[0260] For example, CLIP's text encoder is a Transformer-based language model that first transforms the third text into a series of word embeddings, and then processes these embeddings through a series of Transformer blocks. Each Transformer block also includes a multi-head self-attention mechanism and a feedforward neural network. Through these Transformer blocks, the text encoder learns a high-dimensional feature representation of the text as a fourth text feature of the third text.

[0261] In step 111, the first image encoder after training is invoked based on the fourth text feature and the initialized image feature vector to obtain the target feature vector.

[0262] In some embodiments, the fourth text feature and the initialized image feature vector are fused to obtain the first fused feature; the first fused feature is input into the trained first image encoder to obtain the target feature vector.

[0263] For example, the fourth text feature and the initialized image feature vector can be directly concatenated. For instance, if the fourth feature vector is 'a' and the initialized image feature vector is 'b', then the first fused feature is 'a+b'. The first fused feature is then input into the trained first image encoder, and the semantic features of each pixel are extracted sequentially according to the storage order of each pixel in the first fused feature. The semantic features of multiple pixels are then aggregated into the target feature vector in order.

[0264] In step 112, the pre-trained second multimodal coding model is invoked based on the fourth text feature and the target feature vector to obtain the predicted feature vector.

[0265] In some embodiments, the fourth text feature and the target feature vector are fused to obtain the second fused feature; the second fused feature is input into the pre-trained second multimodal coding model to obtain the predicted feature vector.

[0266] For example, the fourth text feature and the target feature vector can be directly concatenated. For instance, if the fourth feature vector is 'a' and the target feature vector is 'c', then the second fused feature is 'a+c'. The second fused feature is further encoded through the encoding layer of a pre-trained second multimodal coding model. This encoding layer includes multiple fully connected layers, transformer layers, etc., used to extract high-level feature representations. The features processed by the encoding layer are converted into predicted feature vectors, which are used to guide image generation.

[0267] In step 113, the predicted feature vector is decoded to obtain the third image.

[0268] In some embodiments, the predicted feature vector is transformed and operated on by the decoder to generate a high-dimensional image representation; the high-dimensional image representation is then converted into an image in pixel space to serve as a third image.

[0269] For example, the decoder can include any of the following: a generator from a generative adversarial network, a decoder from a variational autoencoder, a transformer model, etc. A high-dimensional image representation can be converted to a pixel-space image using deintegration layers or upsampling layers.

[0270] This application's embodiments combine text and image features for image generation, enabling the generation of semantically consistent images based on text descriptions. Text-guided image generation improves the controllability and specificity of the generated images. Diverse images can be generated based on different text descriptions, enhancing flexibility and adaptability. Utilizing a trained encoder and multimodal model, high-quality images with good visual effects and semantic consistency can be generated.

[0271] In some embodiments, see Figure 3P , Figure 3P This is the sixteenth flowchart of the image encoder training method provided in this application embodiment. After step 105, the following steps are executed: Figure 3P Steps 114 and 115 are explained in detail below.

[0272] In step 114, the trained first image encoder is invoked to make predictions based on the partially missing fourth image, thereby obtaining the third semantic feature of the fourth image, wherein the third semantic feature represents the semantics of the completed fourth image.

[0273] In some embodiments, a partially missing fourth image is obtained, which may be due to image incompleteness caused by damage, occlusion, or other reasons; a mask is created to indicate the missing part in the partially missing fourth image. The mask is usually a binary image of the same size as the fourth image, with the missing part being 1 and the complete part being 0; the partially missing fourth image and the mask are input into a trained first image encoder, which ignores the missing part and extracts the third semantic features of the partially missing fourth image.

[0274] For example, see Figure 6 , Figure 6 This is a schematic diagram illustrating the completion of a partially missing image according to an embodiment of this application. The partially missing fourth image can be an incomplete image of a puppy. Based on the partially missing puppy image, the trained first image encoder is invoked to perform prediction, thereby obtaining the third semantic feature of the puppy image.

[0275] In step 115, the pre-trained decoder is invoked based on the third semantic feature to decode and obtain the completed fourth image.

[0276] In some embodiments, a high-dimensional image representation is generated by transforming and operating on the third semantic features through a pre-trained decoder; the high-dimensional image representation is then converted into an image in pixel space to serve as a fourth image for completion.

[0277] For example, pre-trained decoders can include any of the following: generators from generative adversarial networks, decoders from variational autoencoders, transformer models, etc. High-dimensional image representations can be converted to pixel-space images using degenerate layers or upsampling layers. See also Figure 6 The pre-trained decoder is called based on the third semantic feature to decode and obtain the completed puppy image.

[0278] This application's embodiments extract third semantic features, ensuring that the completed image maintains semantic consistency with the original image, guaranteeing that the completed part conforms to the overall content and style of the image. The pre-trained decoder model can generate high-quality image completion results with good visual effects. It can be applied to various types of image completion tasks, exhibiting high flexibility and applicability.

[0279] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0280] Related technologies often employ a separate discretized encoder during encoder training, encoding images into discrete tokens, which are then fed into a single Transformer structure to process text and image modal data, completing image-text understanding and image generation tasks in an autoregressive manner. Alternatively, images are first encoded into features in a continuous space, and then the resulting features are discretized into visual tokens. During training, the loss functions for image reconstruction and semantic alignment are jointly optimized to obtain an encoder that can be combined with models used for image-text understanding and generation.

[0281] Visual understanding tasks typically rely on high-level semantic features, while visual generation tasks typically rely on low-level discrete features. Discretizing an image preserves its low-level features, making it suitable for image generation tasks, while extracting high-level features makes it suitable for image understanding tasks. See also Figure 7 , Figure 7 This is a comparative schematic diagram of the related technologies provided in the embodiments of this application and a visual encoder. Figure 7In part (a), the encoder of the related technology only captures low-level image features (such as contours and textures) and lacks high-level semantic representation, resulting in limited visual understanding capabilities. The related technology unifies the understanding and generation of targets by discretizing continuous features, but the quantization operation leads to the loss of semantic information, and it is difficult to balance the training of dual targets. At the same time, it requires massive training data to achieve usable understanding performance, resulting in high computational resource consumption.

[0282] exist Figure 7 In part (b), this embodiment constructs a discrete-to-continuous visual encoder (TokLIP) (i.e., the first image encoder). By feeding the visual encoding output by the VQGAN encoder into the causal encoder, it achieves the transformation from discrete features to continuous space, thus enabling its application in multimodal unified understanding and generation models. This visual encoder employs a dual-channel encoding architecture, including a low-level discrete channel and a high-level continuous channel. In the low-level discrete channel, the VQGAN encoder preserves the original details of the image, ensuring the fidelity of the generation. In the high-level continuous channel, the causal encoder combines the semantics extracted by CLIP (i.e., the first multimodal encoding model), and MLP is used to achieve feature space adaptation from the low-level discrete channel to the high-level continuous channel. During the training of the visual encoder, a decoupling training strategy is applied, freezing the VQGAN parameters to avoid the conflict between the goals of generation and understanding. Contrastive learning and knowledge distillation are combined for dual-loss synergy to achieve global semantic aggregation. Simultaneously, by incorporating causal attention mechanisms and autoregressive prediction, the prediction accuracy of the visual encoder is improved.

[0283] This application's embodiments, through the designed model architecture, achieve learning from discretized labels to high-level image semantics, overcoming the limitation of visual understanding capabilities. It solves the problem of semantic defects in discrete encoders through model distillation and contrastive learning, resolves the problem of target conflict by freezing VQGAN weights, and improves training performance by training the visual encoder with a small amount of training data.

[0284] For an input H×W×3 resolution RGB image (i.e., the first image sample), a VQGAN (i.e., the second image encoder) is used to generate low-dimensional discrete labels (i.e., low-dimensional image features). The low-dimensional discrete labels are then transformed using a multilayer perceptron (MLP) (i.e., a dimension transformation model) with two hidden layers. The dimension-transformed labels are then input into a causal encoder to generate a high-dimensional semantic label sequence (i.e., semantic features). The initialized causal encoder uses pre-trained weights from CLIP (i.e., the first multimodal coding model). The high-dimensional semantic label sequence is then input into a large language model (LLM) (i.e., a pre-trained second multimodal coding model) for multi-task processing.

[0285] See Figure 8 , Figure 8 This is a schematic diagram of the visual encoder provided in this embodiment, which integrates a VQGAN encoder with a CLIP visual encoder. It incorporates high-level semantic knowledge into the discretized image labels, preserving both generation and understanding capabilities within a single encoder. Specifically, given an input image, it is first quantized using a VQGAN encoder to obtain code embeddings for the discrete labels. Next, the code embeddings are mapped to the feature dimensions of the CLIP input through a multilayer perceptron (MLP) layer. Then, the mapped embeddings are fed into a causal encoder in a Vision Transformer (ViT), which employs the same architecture as the encoder in CLIP but uses a causal attention mechanism. Finally, the visual encoder constructed in this embodiment encodes the input image into semantic features, i.e., a transformation from discrete to continuous. By training the causal encoder and MLP layers with large-scale image and text data, semantic representations are effectively learned from the discrete labeled input.

[0286] In the original CLIP architecture, the visual Transformer encoder employs a bidirectional attention mechanism to capture rich semantic relationships between image patches, thereby facilitating a holistic understanding of the input image. This bidirectional attention mechanism is applied to understanding tasks because it allows each token to interact with other tokens in the sequence, thus providing a more comprehensive understanding of the input. However, in the context of multimodal autoregressive learning, the bidirectional attention mechanism affects the end-to-end training paradigm for next-to-next-to token prediction, in which each token can only be predicted by paying attention to its preceding tokens. Therefore, during autoregressive generation, each token can only pay attention to previously generated tokens. This causal structure is crucial for autoregressive models because the information flow must follow the temporal order of token generation.

[0287] To address this mismatch, the visual encoder employs a causal attention mechanism, enabling end-to-end multimodal image generation tasks when further combined with a large language model. In this way, image and text signals, as well as understanding and generation tasks, can be fused within a single causal Transformer.

[0288] Because the learning process struggles to adapt to constrained attention mechanisms, directly training the causal encoder results in a performance degradation compared to the original bidirectional Transformer encoder. First, a pre-training initialization strategy is employed, initializing the causal encoder as a pre-trained bidirectional CLIP visual encoder, instead of training it from scratch. Next, contrastive loss and distillation loss are combined as the training objective, enabling the causal encoder to be trained effectively and efficiently. During training, only the MLP layer and the causal encoder are trainable. By freezing the VQGAN encoder, reconstruction capabilities are decoupled from the training process, avoiding potential target conflicts and simplifying the overall optimization process.

[0289] See Figure 9 , Figure 9 This is a schematic diagram illustrating the structure of the visual encoder combined with the multimodal unified understanding generative model provided in this application embodiment. The multimodal unified understanding generative model is an autoregressive structure; the input can be images and text, and the output can also be images and text. The processes of image and text understanding and generation can both be implemented within a single model, for example... Figure 9 The left side can be input with an image and text to understand the content in the image, and the right side can be input with text to generate an image corresponding to the text through the VQGAN decoder.

[0290] The visual encoder provided in this application uses less than 20% of the amount of pre-training data used in related technologies, and outperforms related technologies in classification tasks on zero-shot datasets (ImageNet) and retrieval tasks on large image datasets (Microsoft Common Objects in Context, MSCOCO), as shown in Table 1.

[0291] Table 1 Performance comparison for different tasks

[0292]

[0293]

[0294] Here, IN TOP1 represents the Top-1 accuracy (%) on the ImageNet dataset, COCO TR@1 represents the Top-1 recall (%) in the text retrieval task on the MSCOCO dataset, and COCO IR@1 represents the Top-1 recall (%) in the image retrieval task on the MSCOCO dataset.

[0295] In Table 1, TokLIP achieved an IN TOP1 of 76.4%, TR@1 of 64.06%, and IR@1 of 48.46% at a resolution of 256. TR@1 and IR@1 were higher than those of models MetaCLIP-L / 14 and QLIP, indicating that TokLIP performed better in cross-modal retrieval tasks. At a resolution of 384, TokLIP achieved an IN TOP1 of 79.2%, TR@1 of 67.76%, and IR@1 of 52.24%. TR@1 and IR@1 were higher than those of QLIP, especially in IR@1, where TokLIP reached 52.24%, while QLIP only achieved 43.00%, demonstrating TokLIP's stronger cross-modal retrieval capabilities at high resolutions. TokLIP used the “Mix 125M” and “Mix 50M” datasets, while other models such as MetaCLIP-L / 14 and QLIP used larger datasets (such as Metaclip400M and DataComp1B). Although TokLIP used a smaller dataset, its performance was better than or close to that of models using larger datasets, indicating its advantage in data utilization efficiency. At a resolution of 256 (dataset Mix 125M), TokLIP achieved an IN TOP1 of 76.4, a COCO TR@1 of 64.06, and an IR@1 of 48.46. At a resolution of 384 (dataset Mix 50M), the IN TOP1 improved to 79.2, the COCO TR@1 to 67.76, and the IR@1 jumped to 52.24. This demonstrates that increasing the resolution significantly improved the model's performance in ImageNet classification and COCO retrieval tasks, despite the reduced dataset size (125M → 50M). TokLIP optimized its feature extraction capabilities through higher resolution input. TokLIP performs efficient feature extraction at high resolutions (excellent in both classification and retrieval), is highly adaptable to small-scale, high-quality data, and has stronger semantic alignment capabilities than similar models. Furthermore, TokLIP retains the original reconstruction capabilities of low-level VQGANs, eliminating the need for dedicated optimization for image generation.

[0296] The following description continues to illustrate the exemplary structure of the image encoder training device 233 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2As shown, the software modules in the image encoder training device 233 stored in the memory 230 may include:

[0297] The first extraction module 2331 is used to extract low-dimensional image features of the first image sample.

[0298] The semantic extraction module 2332 is used to call the first image encoder to be trained based on low-dimensional image features to perform semantic extraction and obtain the first semantic features of the first image sample.

[0299] The second extraction module 2333 is used to extract high-dimensional image features of the first image sample and extract first text features of the first text sample used to describe the first image sample, wherein the dimension of the high-dimensional image features is greater than the dimension of the low-dimensional image features.

[0300] The loss determination module 2334 is used to determine a first distillation loss based on the first semantic features and high-dimensional image features, and to determine a first contrast loss based on the first semantic features and the first text features.

[0301] The parameter update module 2335 is used to update the parameters of the first image encoder based on the first distillation loss and the first contrast loss, so as to obtain the trained first image encoder.

[0302] In some embodiments, the first extraction module 2331 is further configured to extract features from the first image sample to obtain a feature map; for each first vector in the feature map, determine the distance between the first vector and each second vector in the preset codebook, and determine the index of the second vector corresponding to the smallest distance; and combine the indices corresponding to each first vector into low-dimensional image features.

[0303] In some embodiments, the semantic extraction module 2332 is further configured to convert the first feature dimension of the low-dimensional image features into the second feature dimension to obtain candidate image features, wherein the second feature dimension is the feature dimension adapted to the first image encoder; and to perform semantic extraction on the candidate image features based on the first image encoder to be trained to obtain the first semantic features of the first image sample.

[0304] In some embodiments, the semantic extraction module 2332 is further configured to perform a linear transformation on the low-dimensional image features of the first feature dimension based on the preset weight matrix and bias vector in the pre-trained first-dimensional transformation model to obtain linear image features of the second feature dimension; and to perform a non-linear transformation on the linear image features based on the activation function in the pre-trained first-dimensional transformation model to obtain candidate image features.

[0305] In some embodiments, the semantic extraction module 2332 is further configured to convert candidate image features into a feature sequence, wherein the feature sequence includes multiple elements, each element representing at least a portion of the features of the first image sample; perform position encoding on the feature sequence to obtain a first feature representation including position information of each element; perform self-attention processing on each element in the first feature representation to obtain an attention weight for each element; perform weighted summation on multiple elements based on the attention weights to obtain a second feature representation; and perform a nonlinear transformation on the second feature representation to obtain a first semantic feature of the first image sample.

[0306] In some embodiments, the loss determination module 2334 is further configured to map the first semantic feature to the feature space of the high-dimensional image feature to obtain the high-dimensional semantic feature; and based on a preset loss function, map the difference between the high-dimensional semantic feature and the high-dimensional image feature as the first distillation loss.

[0307] In some embodiments, the loss determination module 2334 is further configured to combine the first image sample and the first text sample into a positive sample, wherein the first similarity between the first semantic feature of the first image sample and the first text feature of the first text sample is greater than a similarity threshold; combine the third image sample and the third text sample into a negative sample, wherein the second similarity between the image sample feature of the third image sample and the third text sample feature of the third text sample is less than a similarity threshold; and convert the difference between the positive sample and the negative sample into a first contrast loss using a preset contrast loss function.

[0308] In some embodiments, the parameter update module 2335 is further configured to fuse the updates of the first distillation loss and the first contrast loss to obtain the total loss; and update the parameters of the first image encoder based on the total loss to obtain the trained first image encoder.

[0309] In some embodiments, the parameter update module 2335 is further configured to determine a first dynamic weight of the first distillation loss and a second dynamic weight of the first contrast loss, wherein the first dynamic weight and the second dynamic weight are dynamically updated based on the training parameters of at least one dimension for each update; and the first distillation loss and the first contrast loss are weighted and summed based on the first dynamic weight and the second dynamic weight to obtain the total loss.

[0310] In some embodiments, the parameter update module 2335 is further configured to determine the loss magnitude of the first distillation loss and the first comparison loss, wherein the loss magnitude is the difference between the maximum loss and the minimum loss in a preset loss range; for each round of update, the following processing is performed: determining the progress corresponding to the current update round in a preset total number of update rounds; based on the loss magnitude, determining a loss negatively correlated with the progress in the loss range as a first dynamic weight, and determining a loss positively correlated with the progress in the loss range as a second dynamic weight.

[0311] In some embodiments, the parameter update module 2335 is further configured to perform the following processing for each round of update: determine a first gradient of the first distillation loss and a second gradient of the first contrast loss; determine a first gradient norm of the first gradient and a second gradient norm of the second gradient; determine a first dynamic weight negatively correlated with the first gradient norm and a second dynamic weight negatively correlated with the second gradient norm.

[0312] In some embodiments, the parameter update module 2335 is further configured to freeze the parameters of the pre-trained second image encoder and the pre-trained first multimodal coding model, wherein the pre-trained second image encoder is used to extract low-dimensional image features of the first image sample, and the pre-trained first multimodal coding model is used to extract high-dimensional image features and first text features; and update the parameters of the first dimension transformation model based on the first distillation loss and the first contrast loss to obtain the trained first dimension transformation model, wherein the first dimension transformation model is used to convert the first feature dimension of the low-dimensional image features into a second feature dimension adapted to the first image encoder.

[0313] In some embodiments, the parameter update module 2335 is further configured to, if the updates of the first image encoder and the first dimension transformation model meet preset conditions, call the second image encoder to be trained based on the second image sample to obtain low-dimensional sample features of the second image sample, wherein the preset conditions include at least one of the following: the number of updated rounds is greater than or equal to a round threshold, and at least one of the first distillation loss and the first contrast loss is less than a loss threshold after the number of updated rounds; call the trained first image encoder based on the low-dimensional sample features to perform semantic extraction to obtain sample semantic features of the second image sample; and extract high-dimensional sample features of the second image sample based on the first multimodal coding model to be trained. The second text sample features are extracted, where the dimension of the high-dimensional sample features is greater than that of the low-dimensional sample features. The second text sample is used to describe the second image sample. The second distillation loss is determined based on the sample semantic features and the high-dimensional sample features, and the second contrast loss is determined based on the sample semantic features and the second text sample features. The parameters of the trained first image encoder, the trained first dimension transformation model, the second image encoder to be trained, and the first multimodal coding model to be trained are updated based on the second distillation loss and the second contrast loss, respectively, to obtain the third image encoder, the second dimension transformation model, the trained second image encoder, and the trained multimodal coding model.

[0314] In some embodiments, the parameter update module 2335 is further configured to: call a trained first image encoder based on the first image to obtain a second semantic feature of the first image; extract a second text feature from the first text; and call a pre-trained second multimodal coding model based on the second text feature and the second semantic feature to generate an image, thereby obtaining a second image, wherein the first text includes modified parameters for the first image; extract a third text feature from the second text; and call a pre-trained second multimodal coding model based on the third text feature and the second semantic feature to obtain the matching degree between the first image and the second text, wherein the second text is used to describe the first image.

[0315] In some embodiments, the parameter update module 2335 is further configured to obtain an initialized image feature vector and third text for describing the third image to be generated; extract features from the third text to obtain fourth text features of the third text; call the trained first image encoder based on the fourth text features and the initialized image feature vector to obtain a target feature vector; call the pre-trained second multimodal coding model based on the fourth text features and the target feature vector to obtain a predicted feature vector; and decode the predicted feature vector to obtain the third image.

[0316] In some embodiments, the parameter update module 2335 is further configured to call the trained first image encoder to make predictions based on the partially missing fourth image, thereby obtaining the third semantic feature of the fourth image, wherein the third semantic feature represents the semantics of the completed fourth image; and call the pre-trained decoder to decode based on the third semantic feature to obtain the completed fourth image.

[0317] This application provides a computer program product comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the image encoder training method described in this application embodiment.

[0318] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the image encoder training method provided in this application. For example, ... Figure 3A The training method for the image encoder is shown.

[0319] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0320] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0321] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0322] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0323] In summary, through the embodiments of this application, on the one hand, by extracting first semantic features from the low-dimensional image features of image samples and determining a distillation loss between them and the extracted high-dimensional image features, a first image encoder is trained. This allows the first image encoder to retain the original information of the image samples through low-dimensional image features, understand the semantic information of the image samples through semantic features, and retain detailed information of the image samples through high-dimensional image features, reducing information loss. Joint training of features across multiple dimensions improves the semantic consistency of the features. On the other hand, by determining the contrast loss between the semantic features of the image samples and the text features of the text samples used to describe the image samples, cross-modal feature alignment is achieved, enhancing the image-text understanding ability of the first image encoder. This allows the first image encoder to incorporate text features corresponding to the image when extracting image features, improving the accuracy of feature extraction for images.

[0324] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A training method for an image encoder, characterized in that, The method includes: Extract low-dimensional image features from the first image sample; Based on the low-dimensional image features, the first image encoder to be trained is invoked to perform semantic extraction, thereby obtaining the first semantic features of the first image sample. Extract high-dimensional image features from the first image sample and extract first text features from the first text sample used to describe the first image sample, wherein the dimension of the high-dimensional image features is greater than the dimension of the low-dimensional image features; A first distillation loss is determined based on the first semantic feature and the high-dimensional image feature, and a first contrast loss is determined based on the first semantic feature and the first text feature; The parameters of the first image encoder are updated based on the first distillation loss and the first contrast loss to obtain the trained first image encoder.

2. The method according to claim 1, characterized in that, The step of updating the parameters of the first image encoder based on the first distillation loss and the first contrast loss to obtain the trained first image encoder includes: The first distillation loss and the first comparison loss are merged to obtain the total loss; The parameters of the first image encoder are updated based on the total loss to obtain the trained first image encoder.

3. The method according to claim 2, characterized in that, The process of fusing the updates to the first distillation loss and the first comparison loss to obtain the total loss includes: A first dynamic weight is determined for the first distillation loss, and a second dynamic weight is determined for the first contrast loss, wherein the first dynamic weight and the second dynamic weight are dynamically updated based on training parameters of at least one dimension for each update; The first distillation loss and the first comparison loss are weighted and summed based on the first dynamic weight and the second dynamic weight to obtain the total loss.

4. The method according to claim 3, characterized in that, Determining the first dynamic weight of the first distillation loss and the second dynamic weight of the first contrast loss includes: Determine the loss magnitude of the first distillation loss and the first comparison loss, wherein the loss magnitude is the difference between the maximum loss and the minimum loss within a preset loss range; For each round of updates, the following processing is performed: Determine the progress of the current update round within the preset total number of update rounds; Based on the loss magnitude, losses negatively correlated with the progress are determined within the loss range as a first dynamic weight, and losses positively correlated with the progress are determined within the loss range as a second dynamic weight.

5. The method according to claim 3, characterized in that, Determining the first dynamic weight of the first distillation loss and the second dynamic weight of the first contrast loss includes: For each round of updates, the following processing is performed: Determine the first gradient of the first distillation loss and the second gradient of the first contrast loss; Determine the first gradient norm of the first gradient and the second gradient norm of the second gradient; A first dynamic weight negatively correlated with the first gradient norm is determined, and a second dynamic weight negatively correlated with the second gradient norm is determined.

6. The method according to any one of claims 1 to 5, characterized in that, The step of calling the first image encoder to be trained based on the low-dimensional image features to perform semantic extraction, and obtaining the first semantic features of the first image sample, includes: The first feature dimension of the low-dimensional image features is converted into the second feature dimension to obtain candidate image features, wherein the second feature dimension is the feature dimension adapted to the first image encoder. Based on the first image encoder to be trained, semantic extraction is performed on the candidate image features to obtain the first semantic features of the first image sample.

7. The method according to claim 6, characterized in that, The step of converting the first feature dimension of the low-dimensional image features into a second feature dimension to obtain candidate image features includes: Based on the pre-trained first-dimensional transformation model, the low-dimensional image features of the first feature dimension are linearly transformed to obtain the linear image features of the second feature dimension. The linear image features are nonlinearly transformed based on the activation function in the pre-trained first-dimensional transformation model to obtain candidate image features.

8. The method according to claim 6, characterized in that, The semantic extraction of the candidate image features based on the first image encoder to be trained, to obtain the first semantic features of the first image sample, includes: The candidate image features are converted into a feature sequence, wherein the feature sequence includes multiple elements, each element representing at least a portion of the features of the first image sample; The feature sequence is positionally encoded to obtain a first feature representation including the position information of each element; Perform self-attention processing on each element in the first feature representation to obtain the attention weight of each element; The second feature representation is obtained by weighted summation of multiple elements based on the attention weights. The second feature representation is subjected to a nonlinear transformation to obtain the first semantic feature of the first image sample.

9. The method according to any one of claims 1 to 8, characterized in that, When updating the parameters of the first image encoder based on the first distillation loss and the first contrast loss, the method further includes: Freeze the parameters of the pre-trained second image encoder and the pre-trained first multimodal coding model, wherein the pre-trained second image encoder is used to extract the low-dimensional image features of the first image sample, and the pre-trained first multimodal coding model is used to extract the high-dimensional image features and the first text features; The parameters of the first dimension transformation model are updated based on the first distillation loss and the first contrast loss to obtain the trained first dimension transformation model, wherein the first dimension transformation model is used to transform the first feature dimension of the low-dimensional image features into a second feature dimension adapted to the first image encoder.

10. The method according to claim 9, characterized in that, After updating the parameters of the first dimension transformation model based on the first distillation loss and the first contrast loss to obtain the trained first dimension transformation model, the method further includes: If the updates of the first image encoder and the first dimension transformation model reach a preset condition, the second image encoder to be trained is called based on the second image sample to obtain the low-dimensional sample features of the second image sample. The preset condition includes at least one of the following: the number of updated rounds is greater than or equal to the round threshold, and at least one of the first distillation loss and the first contrast loss is less than the loss threshold after the number of updated rounds. Based on the low-dimensional sample features, the trained first image encoder is invoked to perform semantic extraction, thereby obtaining the sample semantic features of the second image sample; Based on the first multimodal coding model to be trained, high-dimensional sample features of the second image sample are extracted, and second text sample features of the second text sample are extracted, wherein the dimension of the high-dimensional sample features is greater than the dimension of the low-dimensional sample features, and the second text sample is used to describe the second image sample. The second distillation loss is determined based on the sample semantic features and the high-dimensional sample features, and the second contrast loss is determined based on the sample semantic features and the second text sample features; Based on the second distillation loss and the second contrast loss, the parameters of the trained first image encoder, the trained first dimension transformation model, the second image encoder to be trained, and the first multimodal coding model to be trained are updated respectively to obtain the third image encoder, the second dimension transformation model, the trained second image encoder, and the trained multimodal coding model.

11. The method according to any one of claims 1 to 8, characterized in that, The step of determining the first distillation loss based on the first semantic feature and the high-dimensional image feature includes: The first semantic feature is mapped to the feature space of the high-dimensional image feature to obtain the high-dimensional semantic feature; Based on a preset loss function, the difference between the high-dimensional semantic features and the high-dimensional image features is mapped to a first distillation loss.

12. The method according to any one of claims 1 to 8, characterized in that, The determination of the first contrast loss based on the first semantic feature and the first text feature includes: The first image sample and the first text sample are combined into a positive sample, wherein the first similarity between the first semantic feature of the first image sample and the first text feature of the first text sample is greater than a similarity threshold. The third image sample and the third text sample are combined into a negative sample, wherein the second similarity between the image sample features of the third image sample and the third text sample features of the third text sample is less than the similarity threshold; The difference between the positive and negative samples is converted into a first contrast loss using a preset contrast loss function.

13. The method according to any one of claims 1 to 8, characterized in that, The extraction of low-dimensional image features from the first image sample includes: Feature extraction is performed on the first image sample to obtain a feature map; For each first vector in the feature map, determine the distance between the first vector and each second vector in the preset codebook, and determine the index of the second vector corresponding to the smallest distance; The indices corresponding to each of the first vectors are combined into low-dimensional image features.

14. The method according to any one of claims 1 to 8, characterized in that, After updating the parameters of the first image encoder based on the first distillation loss and the first contrast loss to obtain the trained first image encoder, the method further includes: The second semantic features of the first image are obtained by calling the trained first image encoder based on the first image. Extract second text features from the first text, and use the second text features and the second semantic features to call a pre-trained second multimodal coding model to generate an image, thereby obtaining a second image, wherein the first text includes modified parameters for the first image; Extract third text features from the second text, and call the pre-trained second multimodal coding model based on the third text features and the second semantic features to obtain the matching degree between the first image and the second text, wherein the second text is used to describe the first image.

15. The method according to any one of claims 1 to 8, characterized in that, After updating the parameters of the first image encoder based on the first distillation loss and the first contrast loss to obtain the trained first image encoder, the method further includes: Obtain the initialized image feature vector and the third text used to describe the third image to be generated; Feature extraction is performed on the third text to obtain the fourth text features of the third text; Based on the fourth text feature and the initialized image feature vector, the trained first image encoder is invoked to obtain the target feature vector; Based on the fourth text feature and the target feature vector, the pre-trained second multimodal coding model is invoked to obtain the predicted feature vector; The predicted feature vector is decoded to obtain the third image.

16. The method according to any one of claims 1 to 8, characterized in that, After updating the parameters of the first image encoder based on the first distillation loss and the first contrast loss to obtain the trained first image encoder, the method further includes: Based on the partially missing fourth image, the trained first image encoder is invoked to make predictions to obtain the third semantic feature of the fourth image, wherein the third semantic feature represents the semantics of the completed fourth image; Based on the third semantic feature, a pre-trained decoder is invoked to decode, resulting in the completed fourth image.

17. A training device for an image encoder, characterized in that, The device includes: The first extraction module is used to extract low-dimensional image features from the first image sample; The semantic extraction module is used to call the first image encoder to be trained to perform semantic extraction based on the low-dimensional image features, so as to obtain the first semantic features of the first image sample. The second extraction module is used to extract high-dimensional image features of the first image sample and extract first text features of the first text sample describing the first image sample, wherein the dimension of the high-dimensional image features is greater than the dimension of the low-dimensional image features. The loss determination module is used to determine a first distillation loss based on the first semantic feature and the high-dimensional image feature, and to determine a first contrast loss based on the first semantic feature and the first text feature; The parameter update module is used to update the parameters of the first image encoder based on the first distillation loss and the first contrast loss, so as to obtain the trained first image encoder.

18. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the training method of the image encoder according to any one of claims 1 to 16.

19. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the training method of the image encoder according to any one of claims 1 to 16 is implemented.

20. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the training method of the image encoder according to any one of claims 1 to 16 is implemented.

Images