Model training method and device for images

Abstract

The embodiment of the invention discloses a model training method and device for an image. Comprising the following steps: acquiring first training data comprising a plurality of first image samples; the method comprises the following steps: pre-training an image encoder by using first training data, performing first coding on a first image sample by using the image encoder to obtain a first hidden space representation of the first image sample, segmenting the first hidden space representation to obtain a plurality of blocks, performing mask processing on the hidden space representations corresponding to part of the blocks to obtain mask blocks, and performing second coding on the mask blocks to obtain a second image sample; performing second coding on the unmasked blocks to obtain feature representations corresponding to the unmasked blocks; inputting the feature representation corresponding to the unmasked block and the mask block into a first decoder for decoding to obtain a second hidden space representation of the first image sample; the pre-training target includes minimizing a difference between the first implicit space representation and the second implicit space representation. After the image encoder obtained through pre-training is used for constructing an image processing model, the convergence speed of the model can be greatly increased.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method and apparatus for training models for images. Background Technology

[0002] With the rapid development of artificial intelligence technology, more and more image processing is being implemented using machine learning. Among them, image understanding and image generation are two important research directions in image processing using machine learning. Traditional methods usually rely on large-scale training data to directly train the initialized image processing model (such as image classification model, image generation model, etc.), which has a very slow convergence speed and causes long-term occupation of training resources. Therefore, training efficiency urgently needs to be improved. Summary of the Invention

[0003] In view of this, this application provides a model training method and apparatus for images to improve training efficiency.

[0004] This application provides the following solution:

[0005] Firstly, a method for training a model for images is provided, the method comprising:

[0006] Acquire first training data, which includes multiple first image samples;

[0007] The image encoder is pre-trained using the first training data. The pre-training includes: performing a first encoding on the first image sample using the image encoder to obtain a first latent space representation of the first image sample; segmenting the first latent space representation into multiple blocks; masking the latent space representations corresponding to some blocks to obtain masked blocks; performing a second encoding on the unmasked blocks to obtain feature representations corresponding to the unmasked blocks; inputting the feature representations corresponding to the unmasked blocks and the masked blocks into a first decoder for decoding to obtain a second latent space representation of the first image sample. The objective of the pre-training includes minimizing the difference between the first latent space representation and the second latent space representation.

[0008] The pre-trained image encoder is used to construct an image processing model.

[0009] Secondly, a model training apparatus for images is provided, the apparatus comprising:

[0010] The sample acquisition unit is configured to acquire first training data, the first training data including a plurality of first image samples;

[0011] A pre-training unit is configured to pre-train an image encoder using the first training data. The pre-training includes: performing a first encoding on the first image sample using the image encoder to obtain a first latent space representation of the first image sample; segmenting the first latent space representation into multiple blocks; masking the latent space representations corresponding to some blocks to obtain mask blocks; performing a second encoding on the unmasked blocks to obtain feature representations corresponding to the unmasked blocks; inputting the feature representations corresponding to the unmasked blocks and the mask blocks into a first decoder for decoding to obtain a second latent space representation of the first image sample. The objective of the pre-training includes minimizing the difference between the first latent space representation and the second latent space representation. The pre-trained image encoder is used to construct an image processing model.

[0012] Thirdly, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the steps of the method described in any one of the first aspects.

[0013] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0014] This application does not directly train the image processing model, but pre-trains the image encoder through latent space mask modeling. This enables the image encoder to quickly learn the contextual dependencies and high-level semantics of the image even in an unsupervised environment. On the one hand, it reduces the dependence on labeled data; on the other hand, it eliminates the need for a teacher model to occupy additional training resources, greatly reducing resource consumption. Furthermore, the image encoder obtained through this pre-training method has excellent semantic understanding and generalization capabilities. Therefore, after building an image processing model using the pre-trained image encoder, the convergence speed of the image processing model can be significantly accelerated, improving training efficiency. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart of a model training method for images provided in an embodiment of this application;

[0017] Figure 2 A schematic diagram illustrating the principle of a pre-trained image encoder provided in an embodiment of this application;

[0018] Figure 3A schematic diagram illustrating the principle of the fine-tuned image classification model provided in the embodiments of this application;

[0019] Figure 4 A schematic diagram illustrating the principle of the fine-tuning image generation model provided in the embodiments of this application;

[0020] Figure 5 A schematic block diagram of an image-based model training apparatus provided in an embodiment of this application;

[0021] Figure 6 A schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0022] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0023] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0024] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0025] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0026] Traditional methods typically train image processing models directly on large-scale training data, resulting in slow convergence and prolonged occupation of training resources. While some techniques exist to accelerate training by aligning the image processing model with a pre-trained visual base model (a model pre-trained on large-scale image data with general visual feature extraction capabilities), using a pre-trained visual base model as the teacher model significantly increases memory consumption, and training speed still needs improvement.

[0027] In view of this, this application provides a new approach. Figure 1 This is a flowchart illustrating a model training method for images provided in an embodiment of this application. This method can be executed by an image training device, which can be a server-side application, or a plugin or SDK (Software Development Kit) functional unit within the application. Alternatively, it can be located on a computer terminal with strong computing power; this embodiment of the invention does not impose any particular limitation on this. Figure 1 As shown, the method may include the following steps:

[0028] Step 101: Obtain the first training data, which includes multiple first image samples.

[0029] Step 103: Pre-train the image encoder using the first training data. The pre-training includes: performing a first encoding on the first image sample using the image encoder to obtain a first latent space representation of the first image sample; dividing the first latent space representation into multiple blocks; masking the latent space representations corresponding to some blocks to obtain mask blocks; performing a second encoding on the unmasked blocks to obtain feature representations corresponding to the unmasked blocks; inputting the feature representations corresponding to the unmasked blocks and the mask blocks into the first decoder for decoding to obtain a second latent space representation of the first image sample. The goal of the pre-training includes minimizing the difference between the first latent space representation and the second latent space representation. The pre-trained image encoder is used to construct an image processing model.

[0030] As can be seen from the above process, this application does not directly train the image processing model, but pre-trains the image encoder through latent space mask modeling. This enables the image encoder to quickly learn the contextual dependencies and high-level semantics of the image even in an unsupervised environment. On the one hand, it reduces the dependence on labeled data; on the other hand, it eliminates the need for a teacher model to occupy additional training resources, greatly reducing resource consumption. Furthermore, the image encoder obtained through this pre-training method has excellent semantic understanding and generalization capabilities. Therefore, after building an image processing model using the pre-trained image encoder, the convergence speed of the image processing model can be significantly accelerated, improving training efficiency.

[0031] The above process and its potential effects will be described in detail below with reference to embodiments. It should be noted that the terms "first," "second," etc., used in this disclosure do not have limitations on size, order, or quantity; they are merely used to distinguish between two sets of training data. For example, "first training data" and "second training data" are used to distinguish between two sets of training data; similarly, "first encoding" and "second encoding" are used only to distinguish between two encoding processes.

[0032] The following describes step 103, namely "pre-training the image encoder using the first training data", in detail with reference to the embodiments.

[0033] The first training data used in this application embodiment includes multiple image samples, referred to as the first image samples. The first image samples may be unlabeled image samples, or they may be selected from an image dataset with labeled data.

[0034] Figure 2 This is a structural diagram of one possible composition of an image encoder provided in an embodiment of this application, such as... Figure 2 As shown, the image encoder may include a first encoder, a block convolutional network, and a second encoder.

[0035] During training, the first image sample ( Figure 2 (The first image sample in the image is for illustrative purposes only) is input to the image encoder. The first encoder is used to encode the first image sample into the latent space to obtain the first latent space representation of the first image sample.

[0036] Block-based convolutional networks are used to divide the first latent space representation into multiple blocks and then perform convolution processing.

[0037] The second encoder is used to encode the unmasked blocks to obtain the feature representations corresponding to the unmasked blocks.

[0038] The first encoder is responsible for encoding the input first image sample into the latent space, obtaining the representation in the latent space, which is the first latent space representation mentioned above. As one possible implementation, the first encoder can be a VAE encoder, which converts the high-dimensional input data into parameters of the probability distribution in the latent space.

[0039] A Variational Autoencoder (VAE) is typically constructed from a Convolutional Neural Network (CNN), which maps the input image (the first image sample during pre-training) to a latent space with probability distribution parameters: mean μ and variance σ. These two parameters define a normal distribution centered at μ and with standard deviation σ. Each dimension of the latent space is no longer a deterministic value but a distribution containing uncertainty. To sample from the probability distribution and preserve gradient differentiability, the VAE encoder introduces reparameterization, which involves randomly sampling a noise vector ∈ from the standard normal distribution and then generating the latent variable z, i.e., the first latent space representation, using the following formula:

[0040]

[0041] The block-based convolutional network divides and convolves the first latent space representation to obtain multiple blocks, each of which is a low-dimensional latent space representation. In this embodiment, during training, the obtained blocks are masked. The masking can be random or based on certain rules.

[0042] Traditional image reconstruction processes also involve masking, but this usually involves partially masking the input data. In this embodiment, however, masking is performed in the latent space. The original image typically contains a large amount of low-level redundant information (such as slight noise, color fluctuations, etc.), while the latent space contains high-level semantic features extracted by the first encoder (such as object shape, structure, component relationships, etc.). Masking and reconstruction in the latent space allows the image encoder to ignore irrelevant details and be forced to learn and understand the core semantics of the image. The learned features are more discriminative and more useful for downstream tasks. Furthermore, masking and reconstruction in the latent space significantly reduces computational cost and speeds up training.

[0043] After segmentation and convolution, each block is either unmasked or unmasked. The unmasked blocks, when grouped into a sequence, are input into a second encoder for attention-based encoding. For example, a self-attention mechanism is used to capture the global dependencies between blocks, yielding feature representations for the unmasked blocks. This second encoder can be a Transformer-based encoder, such as the ViT (VisionTransformer) encoder.

[0044] Taking the ViT encoder as an example, the unmasked blocks form a one-dimensional sequence. Each block in this sequence can be positionally encoded to preserve spatial information. The positional encodings and corresponding blocks are then concatenated and input into the ViT encoder. The ViT encoder mainly consists of a self-attention network, a normalization layer, and a feedforward neural network. The self-attention network processes the input sequence using a self-attention mechanism, calculating the global dependencies between blocks. The normalization layer normalizes the output of the self-attention network before inputting it into the feedforward neural network. The feedforward neural network performs nonlinear transformations on each block. Since the structure of the ViT encoder is already known, it will not be described in detail here.

[0045] In the embodiments of this application, the use of ViT, a more general base model, during the pre-training process can decouple it from downstream tasks, better learn global features, have stronger generalization ability, and be more conducive to transfer to a variety of downstream tasks.

[0046] The feature representations corresponding to the unmasked blocks obtained by the second encoder, together with the masked blocks, constitute the sequence input to the first decoder. The position of the masked block in the sequence corresponds to its position in the first latent space representation. The first decoder reconstructs the masked blocks, ultimately obtaining the second latent space representation of the first image sample. The first decoder can be a Transformer-based decoder, such as the ViT decoder.

[0047] The objectives employed during pre-training include minimizing the difference between the first and second latent space representations. For example, the first and second latent space representations can be used to determine the value of the MSE (Mean Squared Error) loss function, and the MSE loss function value can be used to update the model parameters of the image encoder until a preset training termination condition is met. This termination condition may include, for example, the loss function value being less than or equal to a preset loss function threshold, or the number of iterations reaching a preset threshold.

[0048] In the embodiments of this application, the first encoder, such as the VAE encoder, usually has the ability to capture high-level semantics. It is usually an existing encoder that has already been trained. Therefore, in order to avoid destroying its learned high-level semantic understanding ability in the new task, the model parameters of the first encoder can be kept unchanged (i.e., the model parameters of the first encoder are frozen), and the model parameters of the block convolutional network, the second encoder and the first decoder can be updated.

[0049] After completing the pre-training process, an image processing model can be built using the obtained image encoder, depending on the specific downstream task. This model can then be fine-tuned. Specifically, during the image processing model construction, the parameters of the pre-trained image encoder can be used to initialize the model, thus transferring the image encoder to the downstream image processing task. The image processing model can be an image understanding model, such as an image classification model or an image segmentation model, or it can be an image generation model.

[0050] For image understanding models, the parameters of the image encoder can be directly inherited, meaning they can be initialized using the parameters of the pre-trained image encoder without additional adjustments. Taking image classification models as an example... Figure 3 As shown, a pre-trained image encoder can be used to connect a classification network to build an image classification model.

[0051] The process of fine-tuning an image classification model can include: acquiring second training data, which is labeled data, including multiple second image samples and their corresponding class labels; inputting the second image samples into an image encoder, which performs a first encoding on the second image samples to obtain a first latent space representation of the second image samples, and then segments the first latent space representation into multiple blocks; performing a second encoding on each block (e.g., encoding based on a self-attention mechanism) to obtain the feature representation corresponding to the second image samples. The classification network then predicts the class of the second image samples based on the feature representations corresponding to the second image samples. The goal of this fine-tuning includes minimizing the difference between the predicted class and the class label. For example, the obtained class and class label can be used to determine the value of the cross-entropy loss function, and the model parameters of the image classification model can be updated using the value of the cross-entropy loss function until a preset training termination condition is met. The training termination condition can include, for example, the value of the loss function being less than or equal to a preset loss function threshold, or the number of iterations reaching a preset threshold.

[0052] During the aforementioned fine-tuning process, the image encoder does not perform masking on the first latent space representation. Furthermore, when the classification network predicts the category of a second image sample based on its corresponding feature representation, this representation is essentially a sequence of feature representations for each token (a term used in image processing, typically referring to an image block after segmentation). Each token, besides corresponding to an image block, also contains a Class Token at the beginning of the sequence. The classification network typically only uses the feature representation of this Class Token to predict the category of the second image sample.

[0053] For image generation models, such as Figure 4As shown, a diffusion model can be constructed using a pre-trained image encoder, and then an image generation model can be built using the diffusion model. The diffusion model essentially performs denoising, that is, it performs noise prediction and then removes the noise.

[0054] The fine-tuning of the image generation model includes: acquiring third training data, which comprises multiple third image samples; encoding the third image samples into the latent space using the first encoder in the image encoder to obtain the first latent space representation of the third image samples; adding noise to the first latent space representation of the third image samples, for example, adding random noise for T time steps, where T is a positive integer, to obtain the latent space representation of the noisy image; using a block-based convolutional network in the image encoder to segment the latent space representation of the noisy image into multiple blocks; and using a second encoder in the image encoder to encode each block based on an attention mechanism to obtain the feature representation corresponding to the noisy image, and then predicting the noise based on the feature representation corresponding to the noisy image. The goal of fine-tuning is to minimize the difference between the added noise and the predicted noise.

[0055] In the above process, the image encoder adds noise for T time steps and predicts noise for T time steps. Using the difference between the predicted noise and the added noise at each time step, it determines the value of the loss function (e.g., MSE loss). The model parameters of the image generation model are then updated using the value of the cross-entropy loss function until a preset training termination condition is met. This training termination condition may include, for example, the value of the loss function being less than or equal to a preset loss function threshold, or the number of iterations reaching a preset threshold.

[0056] During the aforementioned fine-tuning process, the image encoder does not perform masking on the first latent space representation. Furthermore, when the second encoder predicts noise based on the feature representation corresponding to the noisy image, this feature representation is essentially a sequence composed of the feature representations of each token. Each token, in addition to corresponding to a block, has a Class Token at the beginning of this sequence. Since the generation task does not require the Class Token, it can be removed before being input into the second decoder.

[0057] In addition, the image encoders mentioned above can be used directly in the image generation model, or they can be adapted. For example, if a SiT (Scalable Interpolant Transformer) encoder or a DiT (Diffusion Transformer) encoder is introduced into the image generation model (i.e., the second encoder of the image encoder in the image generation model is a SiT encoder or a DiT encoder), but since the structure of the DiT encoder and SiT encoder is slightly different from that of the ViT encoder, the trained image encoder can be adapted and then used to initialize the image generation model.

[0058] DiT is a generative model that combines Transformer with a diffusion model for high-quality image generation. It replaces the U-Net architecture in traditional diffusion models with Transformer, using an attention mechanism to better capture long-range dependencies in images. During the diffusion process, Transformer Blocks are used to denoise noisy images, gradually restoring a clear image.

[0059] SiT is an improvement on DiT, based on DiT and introducing a random interpolation framework, which allows for a more flexible connection between data distributions and Gaussian noise distributions.

[0060] To adapt the trained image encoder to the pre-trained ViT encoder, scaling and bias factors can be introduced into the normalization layer of the image encoder during the construction of the diffusion model to align it with the pre-trained ViT encoder. To adapt to the image generation task, in the construction of the diffusion model, before Self-Attention, the input x of each Transformer Block in the second encoder is adaptively normalized by the normalization layer. For example, the normalization layer can be an AdaLN (Adaptive Normalization Layer), which allows dynamic adjustment of the normalization parameters during fine-tuning, enabling the model to adapt to new tasks more quickly. The introduced scaling factor γ and bias factor β are:

[0061]

[0062] γ and β are updated and learned during the fine-tuning process. Before fine-tuning, γ and β are initialized to 1 and 0, respectively. x is the input to the normalization layer.

[0063] Furthermore, since the resolutions of the first image samples used in pre-training and the third image samples used in fine-tuning differ in some cases (e.g., 224×224 during pre-training, while the image generation model uses 256×256), in this embodiment, extended positional encoding can be performed on each block before the second encoding, i.e., before the second encoder. For example, Bicubic Interpolation can be used for extended positional encoding, and then the second encoder performs the second encoding on each block based on the results of the extended positional encoding. This method can adjust the size of the positional encoding to adapt to input images of different resolutions, thereby solving the adaptation problem when the input image size changes.

[0064] Bicubic Interpolation is a high-precision image interpolation method. When applied to position extension coding, it can convert position codes into a two-dimensional grid. Assuming the original position code is a one-dimensional sequence, it needs to be reshaped into a two-dimensional position code map to match the spatial structure of the image. Then, Bicubic interpolation is used to scale the two-dimensional position code map so that its size is consistent with the target size, mainly reflecting the number of blocks. Finally, the scaled two-dimensional position code map is restored to the sequence form.

[0065] The result of the extended positional encoding can be concatenated with each block and then input into the second encoder, which will then process it using the attention mechanism.

[0066] After the aforementioned fine-tuning is completed, the image generation model further includes a second decoder based on the diffusion model. During the actual inference process (i.e., the actual image generation process), the second decoder performs decoding processing (i.e., mapping to pixel space) based on the feature representation after denoising by the diffusion model to obtain the target image. The second decoder can be a VAE decoder or other encoders.

[0067] After the above fine-tuning, an image processing model is obtained, such as an image understanding model or an image generation model. This fine-tuned image processing model can then be used to perform corresponding image processing tasks, such as image classification, image segmentation, and image generation.

[0068] The following uses an image generation model as an example to compare the training method provided in this application embodiment with the traditional training method. The traditional training method refers to directly training the image generation model (based on a diffusion model) using training data. The training method provided in this application embodiment first pre-trains the image encoder, then uses the pre-trained image encoder to build the image generation model, and finally fine-tunes the image generation model. Experiments have verified that the training method provided in this application embodiment can significantly reduce the number of iterations required for training, thereby shortening the training time and reducing the computational resources consumed. For example, in the same experiment using the same training data, the traditional training method required 7 million training steps to converge and achieve better results. However, the training method provided in this application embodiment only requires 600,000 fine-tuning steps when applied to DiT and only 150,000 fine-tuning steps when applied to SiT to achieve convergence and better results, without increasing additional training costs and resource (e.g., GPU memory) overhead.

[0069] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0070] According to another embodiment, a model training apparatus for images is provided. For example... Figure 5 As shown, the device 500 may include a sample acquisition unit 501 and a pre-training unit 502, and may further include a model fine-tuning unit 503. The main functions of each component are as follows:

[0071] The sample acquisition unit 501 is configured to acquire first training data, which includes multiple first image samples.

[0072] The pre-training unit 502 is configured to pre-train the image encoder using the first training data. The pre-training includes: performing a first encoding on the first image sample using the image encoder to obtain a first latent space representation of the first image sample; segmenting the first latent space representation into multiple blocks; masking the latent space representations corresponding to some blocks to obtain mask blocks; performing a second encoding on the unmasked blocks to obtain feature representations corresponding to the unmasked blocks; inputting the feature representations corresponding to the unmasked blocks and the mask blocks into the first decoder for decoding to obtain a second latent space representation of the first image sample. The pre-training objective includes minimizing the difference between the first latent space representation and the second latent space representation. The pre-trained image encoder is used to construct an image processing model.

[0073] As one possible implementation, an image encoder includes a first encoder, a block convolutional network, and a second encoder.

[0074] The first encoder is used to encode the first image sample into the latent space to obtain the first latent space representation of the first image sample.

[0075] Block-based convolutional networks are used to divide the first latent space representation into multiple blocks and then perform convolution processing.

[0076] The second encoder is used to encode the unmasked blocks based on an attention mechanism to obtain the feature representations corresponding to the unmasked blocks.

[0077] During pre-training, the pre-training unit 502 is configured to keep the model parameters of the first encoder unchanged and update the model parameters of the block convolutional network, the second encoder, and the first decoder.

[0078] Furthermore, the model fine-tuning unit 503 is configured to construct an image processing model using a pre-trained image encoder and to fine-tune the image processing model.

[0079] As one possible approach, image understanding models include image classification models. When building an image processing model using a pre-trained image encoder, the model fine-tuning unit 503 can be specifically configured to: build an image classification model using the pre-trained image encoder and classification network.

[0080] When fine-tuning the image processing model, the model fine-tuning unit 503 can be specifically configured to: acquire second training data, which includes multiple second image samples and their corresponding category labels; input the second image samples into an image encoder, which performs a first encoding on the second image samples to obtain a first latent space representation of the second image samples, and divides the first latent space representation into multiple blocks; perform a second encoding on each block to obtain a feature representation corresponding to the second image samples; and the classification network predicts the category of the second image samples based on the feature representation corresponding to the second image samples. The fine-tuning objective includes minimizing the difference between the predicted category and the category label.

[0081] As another possible approach, the image understanding model includes an image generation model. When constructing an image processing model using a pre-trained image encoder, the model fine-tuning unit 503 can be specifically configured to: construct a diffusion model using the pre-trained image encoder and second decoder, and then construct an image generation model using the diffusion model.

[0082] When fine-tuning the image processing model, the model fine-tuning unit 503 can be specifically configured to: acquire third training data, which includes multiple third image samples; encode the third image samples into the latent space using the first encoder in the image encoder to obtain the first latent space representation of the third image samples; add noise to the first latent space representation of the third image samples to obtain the latent space representation of the noisy image; the block convolutional network in the image encoder divides the latent space representation of the noisy image into multiple blocks; the second encoder in the image encoder encodes each block based on an attention mechanism to obtain the feature representation corresponding to the noisy image; and predict noise based on the feature representation corresponding to the noisy image. The fine-tuning objective includes minimizing the difference between the added noise and the predicted noise.

[0083] Furthermore, the model fine-tuning unit 503 can also be configured to introduce scaling factors and bias factors into the normalization layer in the image encoder when constructing the diffusion model, and the scaling factors and bias factors are updated during the fine-tuning process.

[0084] Furthermore, the model fine-tuning unit 503 can also be configured to: perform extended positional encoding on each block using bicubic interpolation before encoding each block based on the attention mechanism; and perform attention-based encoding on each block based on the result of the extended positional encoding.

[0085] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0086] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0087] In addition, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0088] And an electronic device, comprising:

[0089] One or more processors; and

[0090] A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any of the foregoing method embodiments.

[0091] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0092] in, Figure 6 An exemplary architecture of an electronic device is shown, which may include a processor 610, a video display adapter 611, a disk drive 612, an input / output interface 613, a network interface 614, and a memory 620. The processor 610, video display adapter 611, disk drive 612, input / output interface 613, network interface 614, and memory 620 can communicate with each other via a communication bus 630.

[0093] The processor 610 can be implemented using a general-purpose CPU, microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits to execute relevant programs and implement the technical solution provided in this application.

[0094] The memory 620 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 620 can store the operating system 621 for controlling the operation of the electronic device 600, and the basic input / output system (BIOS) 622 for controlling the low-level operations of the electronic device 600. Additionally, it can store a web browser 623, a data storage management system 624, and an image-based model training device 500, etc. The aforementioned image-based model training device 500 can be the application program that specifically implements the aforementioned steps in this embodiment. In summary, when implementing the technical solution provided in this application through software or firmware, the relevant program code is stored in the memory 620 and executed by the processor 610.

[0095] Input / output interface 613 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0096] Network interface 614 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0097] Bus 630 includes a pathway for transmitting information between various components of the device, such as processor 610, video display adapter 611, disk drive 612, input / output interface 613, network interface 614, and memory 620.

[0098] It should be noted that although the above-described device only shows the processor 610, video display adapter 611, disk drive 612, input / output interface 613, network interface 614, memory 620, bus 630, etc., in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the solution of this application, and does not necessarily include all the components shown in the figures.

[0099] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer program product. This computer program product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0100] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A model training method for images, characterized in that, The method includes: Acquire first training data, which includes multiple first image samples; The image encoder is pre-trained using the first training data. The pre-training includes: performing a first encoding on the first image sample using the image encoder to obtain a first latent space representation of the first image sample; segmenting the first latent space representation into multiple blocks; masking the latent space representations corresponding to some blocks to obtain masked blocks; performing a second encoding on the unmasked blocks to obtain feature representations corresponding to the unmasked blocks; inputting the feature representations corresponding to the unmasked blocks and the masked blocks into a first decoder for decoding to obtain a second latent space representation of the first image sample. The objective of the pre-training includes minimizing the difference between the first latent space representation and the second latent space representation. The pre-trained image encoder is used to construct an image processing model.

2. The method according to claim 1, characterized in that, The image encoder includes a first encoder, a block convolutional network, and a second encoder. The first encoder is used to encode the first image sample into the latent space to obtain the first latent space representation of the first image sample; The block-based convolutional network is used to divide the first latent space representation into multiple blocks and perform convolution processing; The second encoder is used to encode the unmasked blocks based on an attention mechanism to obtain the feature representations corresponding to the unmasked blocks; During the pre-training process, the model parameters of the first encoder are kept unchanged, while the model parameters of the block convolutional network, the second encoder, and the first decoder are updated.

3. The method according to claim 1 or 2, characterized in that, The method further includes: An image processing model is constructed using the pre-trained image encoder, and the image processing model is fine-tuned. The image processing model includes an image understanding model or an image generation model.

4. The method according to claim 3, characterized in that, The image understanding model includes an image classification model; Constructing an image processing model using the pre-trained image encoder includes: constructing an image classification model using the pre-trained image encoder and classification network; Fine-tuning the image processing model includes: acquiring second training data, which includes multiple second image samples and their corresponding category labels; inputting the second image samples into the image encoder, which performs a first encoding on the second image samples to obtain a first latent space representation of the second image samples, and segments the first latent space representation to obtain multiple blocks; performing a second encoding on each block to obtain a feature representation corresponding to the second image samples; and the classification network predicts the category of the second image samples based on the feature representation corresponding to the second image samples. The fine-tuning objective includes minimizing the difference between the predicted category and the category label.

5. The method according to claim 3, characterized in that, Constructing an image processing model using the pre-trained image encoder includes: Using the pre-trained image encoder, a diffusion model is constructed, and using the diffusion model, an image generation model is constructed. Fine-tuning the image processing model includes: acquiring third training data, which includes multiple third image samples; encoding the third image samples into a latent space using a first encoder in the image encoder to obtain a first latent space representation of the third image samples; adding noise to the first latent space representation of the third image samples to obtain a latent space representation of a noisy image; segmenting the latent space representation of the noisy image into multiple blocks using a block-based convolutional network in the image encoder; encoding each block using an attention-based mechanism in the image encoder to obtain a feature representation corresponding to the noisy image; and predicting noise based on the feature representation corresponding to the noisy image. The goal of the fine-tuning includes minimizing the difference between the added noise and the predicted noise.

6. The method according to claim 5, characterized in that, The method further includes: When constructing the diffusion model, a scaling factor and a bias factor are introduced into the normalization layer of the image encoder, and the scaling factor and the bias factor are updated during the fine-tuning process.

7. The method according to claim 5, characterized in that, Before encoding each block based on an attention mechanism, the method further includes: performing extended position encoding on each block using a bicubic interpolation method; Based on the results of the extended positional encoding, the attention-based encoding of each block is performed.

8. A model training device for images, characterized in that, The device includes: The sample acquisition unit is configured to acquire first training data, the first training data including a plurality of first image samples; A pre-training unit is configured to pre-train an image encoder using the first training data. The pre-training includes: performing a first encoding on the first image sample using the image encoder to obtain a first latent space representation of the first image sample; segmenting the first latent space representation into multiple blocks; masking the latent space representations corresponding to some blocks to obtain mask blocks; performing a second encoding on the unmasked blocks to obtain feature representations corresponding to the unmasked blocks; inputting the feature representations corresponding to the unmasked blocks and the mask blocks into a first decoder for decoding to obtain a second latent space representation of the first image sample. The objective of the pre-training includes minimizing the difference between the first latent space representation and the second latent space representation. The pre-trained image encoder is used to construct an image processing model.

9. The apparatus according to claim 8, characterized in that, The device further includes: The model fine-tuning unit is configured to construct an image processing model using the pre-trained image encoder and to fine-tune the image processing model, which includes an image understanding model or an image generation model.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 8.

Images