Training methods, devices, electronic equipment, and storage media for image processing models

Abstract

This application discloses a training method, apparatus, electronic device, and storage medium for an image processing model, belonging to the field of computer processing technology. The method includes: labeling at least a portion of a labeled image in a generated stitched image to obtain a target region, and acquiring label noise in the target region; adding noise to the target region to obtain a noisy image; acquiring noise image features of the noisy image and text features extracted from the instruction text; inputting the text features and noise image features into an image processing model, outputting a predicted image and acquiring the image noise corresponding to the target region in the predicted image; calculating the noise difference between the image noise and the label noise; iteratively training the image processing model based on the noise difference to obtain a trained image processing model. This application can improve the versatility of the trained model, thereby improving the accuracy of the task processing results output when using this model for practical task processing.

Description

Technical Field

[0001] This application relates to the field of computer processing technology, and specifically to a training method, apparatus, electronic device, and storage medium for an image processing model. Background Technology

[0002] In recent years, large-scale pre-trained generative models have made significant progress in the field of image synthesis. Among them, text-to-image diffusion models have become a research hotspot due to their excellent generation quality and controllability. Related techniques involve setting different task-specific branch networks within the diffusion model and performing targeted training to obtain pre-trained models capable of processing corresponding tasks.

[0003] However, due to the rich variety of downstream tasks in image processing, such as image restoration and semantic editing, the task branch networks set in the diffusion model are limited. This makes it difficult for the pre-trained model obtained by training the diffusion model to cover all downstream tasks in all task scenarios, resulting in poor versatility of the pre-trained model. Consequently, the task processing results output when using this pre-trained model for actual task processing are inaccurate. Summary of the Invention

[0004] This application provides a training method, apparatus, electronic device, and storage medium for an image processing model, which can improve the versatility of the trained model and thus improve the accuracy of the task processing results output when using this model for actual task processing.

[0005] To achieve the above objectives, embodiments of this application provide, in one aspect, a method for training an image processing model, comprising:

[0006] For each different image processing task, obtain the source image, the label image corresponding to the source image, and the instruction text corresponding to the source image for each image processing task.

[0007] The label image and the source image are stitched together to obtain a stitched image. At least a portion of the label image in the stitched image is marked to obtain the target region, and the label noise corresponding to the target region is obtained.

[0008] Add noise to the target region to obtain a noisy image corresponding to the stitched image;

[0009] Obtain the noise image features of the noisy image, and the text features extracted from the instruction text;

[0010] Text features and noisy image features are input into the image processing model, which outputs a predicted image and obtains the image noise corresponding to the target region in the predicted image. The noise difference between the image noise and the label noise is calculated. The image processing model is iteratively trained based on the noise difference to obtain the trained image processing model. The trained image processing model is used to process the target source image according to the target instruction text to obtain the target generated image.

[0011] In some embodiments, marking at least a portion of the label image in the stitched image to obtain the target region includes:

[0012] If the instruction text representation requires image processing of the entire region of the source image, then the entire region of the label image in the stitched image is marked to obtain the target region;

[0013] Otherwise, the initial region is obtained by coarsely filtering the region to be processed in the label image of the stitched image according to the instruction text;

[0014] Based on the instruction text, determine the expected semantic type label of the region to be processed, and perform image semantic analysis on the source image to obtain the reference semantic type label corresponding to the source image;

[0015] The initial region is further refined based on the expected semantic type label and the reference semantic type label to obtain the updated region. The updated region in the stitched image is then marked to obtain the target region.

[0016] In some embodiments, noise is added to the target region to obtain a noisy image corresponding to the stitched image, including:

[0017] The stitched image is downsampled to obtain a downsampled stitched image;

[0018] Obtain a preset time step and determine the noise weight matrix based on the preset time step;

[0019] Noise is added to the target region in the downsampled stitched image based on the noise weight matrix to obtain the noise image corresponding to the stitched image.

[0020] In some embodiments, obtaining a preset time step and determining a noise weight matrix based on the preset time step includes:

[0021] When the preset time step is less than the preset time step threshold, the noise weight matrix is ​​determined according to the preset time step, wherein each noise weight value in the noise weight matrix is ​​the same and greater than the preset critical noise weight value.

[0022] When the preset time step is equal to or greater than the preset time step threshold, the noise weight matrix is ​​determined according to the preset time step. The noise weight values ​​in the noise weight matrix are radially decaying, and each noise weight value does not exceed the critical noise weight value.

[0023] In some embodiments, the image processing model includes a pre-constructed initial low-rank matrix;

[0024] The image processing model is iteratively trained based on noise differences to obtain the trained image processing model, including:

[0025] The initial low-rank matrix is ​​updated based on the noise difference to obtain the updated low-rank matrix;

[0026] The updated low-rank matrix is ​​used as the new initial low-rank matrix in the image processing model, and the steps of obtaining the source image, the label image and the instruction text corresponding to each image processing task are returned until the noise difference between the calculated image noise and the label noise is less than the preset noise threshold, thus obtaining the trained image processing model.

[0027] In some embodiments, after obtaining the trained image processing model, the method further includes:

[0028] Obtain the initial source image and the corresponding target instruction text;

[0029] The image to be processed is initialized based on the initial source image, and the area to be processed in the image is marked based on the initial source image and the target instruction text.

[0030] Obtain the reference image features of the initial source image, the target text features extracted from the target instruction text, and the features to be processed of the image to be processed;

[0031] The target text features, the features to be processed, and the reference image features are input into the trained image processing model. Based on the output results, the target predicted image is determined after predicting the region to be processed in the image to be processed.

[0032] In some embodiments, target text features, features to be processed, and reference image features are input into a trained image processing model, and the target predicted image after predicting the region to be processed in the image to be processed is determined based on the output results, including:

[0033] The target text features, the features to be processed, and the reference image features are input into the trained image processing model, and the output is the initial predicted image obtained after predicting the region to be processed in the image to be processed.

[0034] If the region to be processed is not the entire region of the image to be processed, determine the foreground prediction image corresponding to the region to be processed from the initial prediction image;

[0035] Obtain the target weight matrix corresponding to the foreground prediction image, and update the foreground prediction image according to the target weight matrix to obtain the updated foreground prediction image. The weight values ​​of each pixel in the target weight matrix have a radial decay distribution.

[0036] The target prediction image is obtained by updating the initial prediction image based on the updated foreground prediction image.

[0037] To achieve the above objectives, one embodiment of this application provides a training apparatus for an image processing model, comprising:

[0038] The acquisition module is used to acquire the source image, the label image corresponding to the source image, and the instruction text corresponding to the source image for each different image processing task.

[0039] The labeling module is used to stitch together the label image and the source image to obtain a stitched image, to label at least a portion of the label image in the stitched image to obtain the target region, and to obtain the label noise corresponding to the target region;

[0040] The noise-adding module is used to add noise to the target area to obtain a noisy image corresponding to the stitched image;

[0041] The feature extraction module is used to obtain the noise image features of the noisy image and the text features extracted from the instruction text.

[0042] The prediction module is used to input text features and noisy image features into the image processing model, output the predicted image and obtain the image noise corresponding to the target region in the predicted image, calculate the noise difference between the image noise and the label noise, and iteratively train the image processing model based on the noise difference to obtain the trained image processing model. The trained image processing model is used to process the target source image according to the target instruction text to obtain the target generated image.

[0043] To achieve the above objectives, one aspect of this application provides a computer-readable storage medium storing multiple instructions adapted for loading by a processor to execute steps in the training method of the image processing model provided in this application.

[0044] To achieve the above objectives, one aspect of this application provides a computer device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the computer program, it implements the steps in the training method of the image processing model provided in this application.

[0045] To achieve the above objectives, one aspect of this application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps in the training method of the image processing model provided in this application.

[0046] The image processing model training method, apparatus, electronic device, and storage medium proposed in this application acquire, for each different image processing task, a source image, a label image corresponding to the source image, and instruction text corresponding to the source image; concatenate the label image and the source image to obtain a concatenated image; label at least a portion of the label image in the concatenated image to obtain a target region, and acquire the label noise corresponding to the target region; add noise to the target region to obtain a noise image corresponding to the concatenated image; acquire the noise image features of the noise image and the text features extracted from the instruction text; input the text features and noise image features into the image processing model, output a predicted image, acquire the image noise corresponding to the target region in the predicted image, calculate the noise difference between the image noise and the label noise, and iteratively train the image processing model based on the noise difference to obtain a trained image processing model. The trained image processing model is used to process the target source image according to the target instruction text to obtain the target generated image.

[0047] In this embodiment, when training the image processing model, the source image and label image of any downstream task are stitched together into a composite stitched image. Only the region in the stitched image where the predicted content is to be generated is marked as the target region. This transforms diverse tasks such as image restoration and semantic editing into a unified target region prediction problem. During training, the model only adds noise to the target region and calculates the noise difference, forcing the model to learn how to output the desired image processing task result given a source image and corresponding instruction text. Thus, this embodiment allows the pre-trained image processing model to be adapted to various new downstream tasks without any modification to the model structure, greatly improving the model's versatility and scalability, and consequently increasing the accuracy of the task processing results output when using this model for actual task processing.

[0048] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a schematic diagram of the system framework corresponding to the training method of the image processing model provided in the embodiments of this application;

[0051] Figure 2 This is a flowchart illustrating the training method of the image processing model provided in the embodiments of this application;

[0052] Figure 3 This is a schematic diagram of training data for the training method of the image processing model provided in the embodiments of this application;

[0053] Figure 4 This is a data processing diagram illustrating the training method of the image processing model provided in this application embodiment;

[0054] Figure 5 This is a schematic diagram of the module structure of the training device for the image processing model provided in this application embodiment;

[0055] Figure 6 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0056] To enable those skilled in the art to better understand the solutions of this application, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0057] It should be noted that in each specific embodiment of this application, when it is necessary to obtain the source image corresponding to each image processing task, permission or consent from the relevant personnel managing the source images will be obtained first. Furthermore, the collection, use, and processing of this data will comply with relevant laws, regulations, and standards. In addition, when this application embodiment needs to obtain sensitive personal information of relevant personnel, separate permission or consent from the relevant personnel will be obtained through pop-up windows or redirection to a confirmation page. Only after obtaining the separate permission or consent of the relevant personnel will the source image corresponding to each image processing task necessary for the normal operation of this application embodiment be obtained. Other data obtained in this application embodiment are all authorized and legal data, and will not be described in detail here.

[0058] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, programmable consumer computer devices, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0059] First, describe the technical problems existing in the relevant technologies:

[0060] In recent years, large-scale pre-trained generative models have made significant progress in the field of image synthesis. Among them, text-to-image diffusion models have become a research hotspot due to their excellent generation quality and controllability. Related techniques involve setting different task-specific branch networks within the diffusion model and performing targeted training to obtain pre-trained models capable of processing corresponding tasks.

[0061] However, due to the rich variety of downstream tasks in image processing, such as image restoration and semantic editing, the task branch networks set in the diffusion model are limited. This makes it difficult for the pre-trained model obtained by training the diffusion model to cover all downstream tasks in all task scenarios, resulting in poor versatility of the pre-trained model. Consequently, the task processing results output when using this pre-trained model for actual task processing are inaccurate.

[0062] For example, in digital content creation platforms, designers often need to perform diverse editing operations on user-uploaded images, such as removing watermarks, repairing scratches, replacing backgrounds, or making local semantic modifications. Traditional methods rely on a text-to-image diffusion model pre-trained on large-scale data, and support different types of image processing needs by embedding specific task branches, such as image restoration branches and semantic editing branches, into its architecture. However, due to the wide variety and continuous evolution of downstream tasks in real-world applications, and the limitation of the number of task branches that can be integrated during the pre-training stage, the pre-trained model cannot fully cover all possible editing types. When faced with tasks not explicitly modeled in training, such as object removal under complex occlusion or cross-domain style transfer, the model lacks the corresponding prior knowledge and adaptation capabilities, resulting in content that may suffer from structural distortion, semantic inconsistency, or missing details, severely affecting the usability and visual accuracy of the editing results.

[0063] The image processing model training method, apparatus, electronic device, and storage medium proposed in this application acquire, for each different image processing task, a source image, a label image corresponding to the source image, and instruction text corresponding to the source image; concatenate the label image and the source image to obtain a concatenated image; label at least a portion of the label image in the concatenated image to obtain a target region, and acquire the label noise corresponding to the target region; add noise to the target region to obtain a noise image corresponding to the concatenated image; acquire the noise image features of the noise image and the text features extracted from the instruction text; input the text features and noise image features into the image processing model, output a predicted image, acquire the image noise corresponding to the target region in the predicted image, calculate the noise difference between the image noise and the label noise, and iteratively train the image processing model based on the noise difference to obtain a trained image processing model. The trained image processing model is used to process the target source image according to the target instruction text to obtain the target generated image.

[0064] In this embodiment, when training the image processing model, the source image and label image of any downstream task are stitched together into a composite stitched image. Only the region in the stitched image where the predicted content is to be generated is marked as the target region. This transforms diverse tasks such as image restoration and semantic editing into a unified target region prediction problem. During training, the model only adds noise to the target region and calculates the noise difference, forcing the model to learn how to output the desired image processing task result given a source image and corresponding instruction text. Thus, this embodiment allows the pre-trained image processing model to be adapted to various new downstream tasks without any modification to the model structure, greatly improving the model's versatility and scalability, and consequently increasing the accuracy of the task processing results output when using this model for actual task processing.

[0065] The specific details regarding the training methods, apparatus, electronic devices, and storage media of the image processing models provided in the embodiments of this application will be described in detail below.

[0066] Please see Figure 1 , Figure 1 This is a schematic diagram of the system framework corresponding to the image processing model training method provided in this application embodiment. The image processing model training method provided in this application embodiment can be applied to this system framework.

[0067] It includes terminal 140, Internet 130, gateway 120, server 110, etc.

[0068] Terminal 140 or server 110 may be a device that performs a training method for an image processing model.

[0069] Terminal 140 includes, but is not limited to, mobile phones, tablets, computers, and intelligent computing centers. Terminal 140 can be a single device or a collection of multiple devices. For example, multiple computers can be interconnected via a local area network, sharing a single monitor to work collaboratively, thus forming a terminal 140. Terminal 140 can communicate with the Internet 130 via wired or wireless means to exchange data.

[0070] Server 110 refers to a computer system that can provide certain services to terminal 140. Compared to ordinary terminal 140, server 110 has higher requirements in terms of stability, security, and performance. Server 110 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0071] Gateway 120, also known as an internetwork connector or protocol converter, is a computer system or device that acts as a translator, enabling network interconnection at the transport layer. It bridges the gap between two systems using different communication protocols, data formats, languages, or even completely different architectures. Gateways can also provide filtering and security functions. Messages sent from terminal 140 to server 110 are forwarded to the corresponding server 110 via gateway 120. Messages sent from server 110 to terminal 140 are also forwarded to the corresponding terminal 140 via gateway 120.

[0072] The embodiments of this application can be applied to various scenarios, such as image restoration, virtual try-on, semantic segmentation, line drawing coloring, subject generation, and other image processing tasks. The image processing model trained by the image processing model training method can be used in other scenarios; this is only an example and does not imply that the embodiments of this application limit the scenarios in which the image processing model training method can be applied.

[0073] Next, we will describe it from the perspective of the training device for the image processing model, such as... Figure 2 As shown, Figure 2 This is a schematic flowchart of the image processing model training method provided in this application embodiment. The image processing model training method is applied to an image processing model training device. Figure 2 The method may include, but is not limited to, the following steps 210 to 250. When the image processing model training device executes the image processing model training method, the specific process is as follows. It should be noted first that this embodiment... Figure 2 The order of steps 210 to 250 is not specifically limited. The order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0074] Step 210: For each different image processing task, obtain the source image, the label image corresponding to the source image, and the instruction text corresponding to the source image for each image processing task.

[0075] Step 220: stitch the label image and the source image to obtain a stitched image, mark at least a portion of the label image in the stitched image to obtain the target region, and obtain the label noise corresponding to the target region;

[0076] Step 230: Add noise to the target region to obtain a noisy image corresponding to the stitched image;

[0077] Step 240: Obtain the noise image features of the noise image and the text features extracted from the instruction text;

[0078] Step 250: Input the text features and noise image features into the image processing model, output the predicted image and obtain the image noise corresponding to the target region in the predicted image, calculate the noise difference between the image noise and the label noise, and iteratively train the image processing model according to the noise difference to obtain the trained image processing model. The trained image processing model is used to process the target source image according to the target instruction text to obtain the target generated image.

[0079] Steps 210 to 250 are described in detail below.

[0080] In step 210, for each different image processing task, the source image, the label image corresponding to the source image, and the instruction text corresponding to the source image are obtained.

[0081] Image processing tasks refer to targeted operations that perform specific operations or transformations on input images to obtain desired output results. Image processing tasks include, but are not limited to, image inpainting, image colorization, and object removal or addition. For each different image processing task, its corresponding source image, label image, and instruction text can be obtained, and the image processing model can be trained by executing steps 220 to 250.

[0082] Here, the source image refers to the original image used to train the image processing model, which needs to use the source image as a reference for image prediction. For example... Figure 3 As shown, Figure 3 This is a schematic diagram of the training data for the image processing model training method provided in this application embodiment. Different source images are obtained for different image processing tasks. Figure 3 The example only illustrates the case of obtaining a single source image. In actual training, at least one source image is required, meaning multiple images are possible. For instance, in a virtual dressing task, only the clothing image can be input as the source image. In this case, the image processing model generates a model try-on image based on the source image and the corresponding instruction text, with no special requirements for the model's appearance. Alternatively, the source images obtained in the virtual dressing task may be both the clothing image and a specified model image. In this case, the image processing model generates a model try-on image based on the clothing image, the specified model image, and the corresponding instruction text, with the model's appearance specified as the image in the input source image. Furthermore, in a subject generation task, multiple source images can be input. It should be noted that the embodiments in this application... Figure 3 The faces of the users appearing in the image have been blurred to prevent the leakage of user privacy. In practical applications, with the consent of the relevant users, unblurred images can be obtained and used for subsequent image processing.

[0083] The tag image corresponding to the source image refers to the image paired with the source image and reflecting the expected output of the source image under the instruction text. Tag images can be generated through manual annotation, professional software editing, or high-quality synthesis, and this application embodiment does not impose any limitations on this. Figure 3As shown, in the virtual dressing task, the label image presents the model wearing the clothes shown in the source image. That is, the source image can be an edge map used to guide the model's generation, a subject image used as a reference, or the original image to be edited; furthermore, the label image corresponding to the source image can be a generated image that conforms to edge features, a new scene image containing a reference subject, or an edited image, etc., and the label image can accurately correspond to the image content described by the input instruction text.

[0084] In this context, the instruction text corresponding to the source image refers to the semantic instructions that describe the required image processing operations in natural language. The instruction text clearly indicates the specific task to be performed on the source image. For example, the instruction text for a virtual dressing task could be "Put the model in the clothes shown in the input image." Similarly, in an image restoration task where the source image shows a cracked road surface, the corresponding instruction text could be "Repair the cracks in the road surface in the input image." As a semantic control signal, the instruction text helps the image processing model understand the user's intent and generate semantically appropriate content in the target area accordingly.

[0085] In step 220, the label image and the source image are stitched together to obtain a stitched image. At least a portion of the label image in the stitched image is marked to obtain the target region, and the label noise corresponding to the target region is obtained.

[0086] In some embodiments, the acquired data is first preprocessed to construct a unified training data input format. Specifically, the label image and the source image are spatially stitched together to obtain a stitched image. Subsequently, at least a portion of the regions in the stitched image originating from the label image are labeled to define the target region, which is the core part that the image processing model needs to learn to generate or modify. "At least a portion" means that when processing the stitched image, it is not necessary to label the entire label image; instead, a sub-region (partial region) of the label image can be selected for labeling. The actual labeled region is determined based on the specific content of the input source image and the corresponding instruction text. Stitching the label image and the source image refers to stitching them together on a two-dimensional plane. For example, the left region of the stitched image can be designated as the source image, and the right region as the label image corresponding to the source image.

[0087] Furthermore, in order to train better, it is also necessary to pre-generate noise that will be applied to the target area. This noise is called label noise. Label noise will be used as a learning target. During training, the image processing model needs to learn to predict a noise distribution that is as close as possible to this label noise. Label noise is the basis for the image model to learn the direction of denoising.

[0088] In some embodiments, marking at least a portion of the label image in the stitched image to obtain the target region includes:

[0089] (1.1) If the instruction text representation requires image processing of the entire region of the source image, then the entire region of the label image in the stitched image is marked to obtain the target region;

[0090] (1.2) Otherwise, according to the instruction text, the region to be processed in the label image of the stitched image is coarsely screened to obtain the initial region;

[0091] (1.3) Determine the expected semantic type label of the region to be processed according to the instruction text, and perform image semantic analysis on the source image to obtain the reference semantic type label corresponding to the source image;

[0092] (1.4) The initial region is finely filtered according to the expected semantic type label and the reference semantic type label to obtain the updated region, and the updated region in the stitched image is marked to obtain the target region.

[0093] In some embodiments, to mark the label images in the stitched image, the user's intent is first preliminarily determined based on the instruction text. If the instruction text indicates that image processing is required on the entire region of the source image, for example, if the instruction text is "perform semantic segmentation processing on the source image", it can be determined that the corresponding image task processing type is semantic segmentation. Since semantic segmentation tasks usually require semantic segmentation of the entire source image without specifying a region, in this case, the entire region of the label images in the stitched image is directly marked to obtain the target region.

[0094] Furthermore, if the instruction text only requires image processing on a portion of the source image, such as "Repair road surface cracks in the input image", then it can be determined that the corresponding image task processing type is the repair of a specified area of ​​the image. Since the label image shows a road surface that has already been repaired, the target area cannot be directly determined based solely on the label image and the corresponding instruction text. Therefore, it is necessary to first perform a coarse screening of the area to be processed in the label image of the stitched image according to the instruction text to obtain the initial area. In this example, the initial area is the road surface area in the label image.

[0095] Furthermore, to accurately identify the user's true intended processing object from the initially coarsely filtered region, the device performs in-depth semantic information extraction. Specifically, it determines the expected semantic type label of the region to be processed based on the instruction text, and simultaneously performs image semantic analysis on the source image to obtain the corresponding reference semantic type label. Here, the expected semantic type label is the operation object explicitly specified in the instruction text. For example, in "repair the road surface crack in the input image," the expected semantic type label of the region to be processed is "crack." Image semantic analysis is performed on the corresponding source image to obtain its corresponding reference semantic type label, including but not limited to "road surface," "crack," and "blue sky." The common semantic label between the expected semantic type label and the reference semantic type label is "crack." Therefore, the location of the road surface crack is located in the source image and synchronized to the label image to achieve fine filtering of the initial region to obtain the updated region. The updated region in the stitched image is then marked to obtain the target region.

[0096] In another example, a labeled image presents a model wearing a top and pants, and the corresponding instruction text is "Perform the model clothing try-on task based on the input image". In this example, the area where the model is located is first roughly selected in the labeled image as the initial area based on the instruction text. Since the expected semantic type label is "clothing" (which could be a top, pants, or a hat), if the source image only has one pants image, the corresponding reference semantic type label is "pants". At this time, the common semantic label between the expected semantic type label and the reference semantic type label is "pants". Based on this, the initial area is finely selected to obtain the updated area, and the updated area in the stitched image is marked to obtain the target area.

[0097] In other embodiments, the stitched image is first downsampled to the latent space, and a binary mask matrix with the same size as the latent feature of the stitched image is initialized. Each element in the binary mask matrix corresponds to a pixel of the latent feature of the stitched image, ensuring that the mask matrix can accurately match the spatial structure of the latent feature of the stitched image. Then, the mask marking range is optimized according to the characteristics of different downstream tasks to obtain a fully marked mask. For example, in the visual condition control task, only the "image area to be generated" is marked as 1, and the condition reference image and other areas are marked as 0. In the virtual try-on task, the "model clothing area" is marked as 1, and the reference clothing image and model pose image are marked as 0. The mask marking is made more in line with the core requirements of the task through task adaptability adjustment.

[0098] Furthermore, this labeling method achieves unified task transformation. The visual condition control task is transformed into "predicting the conditional image of region 1 based on the condition map of region 0", the image editing task is transformed into "predicting the edited content of region 1 based on the original background of region 0", and the subject-driven generation task is transformed into "predicting the new scene image of the subject in region 1 by referring to the subject in region 0". Finally, all downstream tasks are transformed into "mask region prediction" tasks, and after noise addition processing in step 230, they are input into the image processing model in a unified form without the need to design additional task-specific processing interfaces for the image processing model.

[0099] In machine learning and deep generative models, latent space refers to an abstract vector space used to represent high-dimensional raw data (such as images and text) in a low-dimensional, continuous, and structured manner. The raw data is mapped to a point in the latent space (i.e., a latent vector), which captures the key semantic features and inherent structure of the data; conversely, the original data can be reconstructed from the latent vector using a decoder. In diffusion models, latent space reduces computational complexity.

[0100] It is understood that, by combining the instruction text and the source image, the embodiments of this application accurately locate the desired processing area of ​​the label image in the stitched image, ensuring that the marked target area matches the user's true intention. Figure 1 This allows for the application of noise supervision only to the target area, avoiding redundant or erroneous learning of irrelevant areas such as the background and the model's face. This not only reduces computational overhead but also prevents the image processing model from blurring key details or deviating from the intended instruction due to uniform supervision across the entire image during subsequent image processing. As a result, more accurate image prediction results can be generated in various image processing tasks, especially local editing tasks.

[0101] In step 230, noise is added to the target region to obtain a noise image corresponding to the stitched image.

[0102] In some embodiments, noise is added to the target region in the stitched image to obtain a noisy image corresponding to the stitched image. Meanwhile, other regions in the stitched image retain their original pixel information. These other regions may be the regions of the source image, or regions of the source image and the label image other than the target region.

[0103] In some embodiments, noise is added to the target region to obtain a noisy image corresponding to the stitched image, including:

[0104] (2.1) Perform downsampling processing on the stitched image to obtain a downsampled stitched image;

[0105] (2.2) Obtain the preset time step and determine the noise weight matrix based on the preset time step;

[0106] (2.3) Add noise to the target region in the downsampled stitched image according to the noise weight matrix to obtain the noise image corresponding to the stitched image.

[0107] In some embodiments, to improve processing efficiency and focus on the deep semantic features of the image, the stitched image is first downsampled to obtain a downsampled stitched image. The purpose is to compress and transform the high-dimensional pixel space data into a lower-dimensional but more information-dense latent space. By performing subsequent operations in the latent space rather than the pixel space, the computational load and memory usage can be greatly reduced, enabling the image processing model to be trained and inferred more efficiently.

[0108] Furthermore, to simulate different noise levels during the diffusion process, a preset time step is selected from the preset time step range. The preset time step range can be from 1 to 1000, and can be set according to the actual situation. The preset time step represents the degree of image destruction, and the larger the value, the higher the noise intensity usually is. Then, the corresponding noise weight matrix is ​​determined according to the preset time step.

[0109] Furthermore, noise is added only to the marked target regions in the downsampled stitched image based on the noise weight matrix, thus obtaining the final noisy image corresponding to the stitched image. The target region portion of the noisy image is the result of mixing the original latent features with Gaussian noise according to weights, while the context regions outside the target region retain the original, undisturbed latent features. In this way, the device constructs a clear learning task for the image processing model: given a lossless visual context, learn how to recover the noisy target region.

[0110] It is understood that the embodiments of this application, by downsampling the stitched image, transform complex pixel-level operations into efficient operations in a low-dimensional latent space, greatly reducing the training cost of the model. In addition, the embodiments of this application also achieve precise quantitative control of noise injection intensity through a noise weight matrix. The strategy of adding noise only to the target region allows the model training process to be fully focused on the specified task region, avoiding invalid calculations on context regions that do not need to be changed, thereby improving the training efficiency of the model and strengthening the context association and the ability to perform precise local editing in subsequent model training.

[0111] In some embodiments, obtaining a preset time step and determining a noise weight matrix based on the preset time step includes:

[0112] (2.2.1) When the preset time step is less than the preset time step threshold, the noise weight matrix is ​​determined according to the preset time step, wherein each noise weight value in the noise weight matrix is ​​the same and greater than the preset critical noise weight value;

[0113] (2.2.2) When the preset time step is equal to or greater than the preset time step threshold, the noise weight matrix is ​​determined according to the preset time step. The noise weight values ​​in the noise weight matrix are radially decaying, and each noise weight value does not exceed the critical noise weight value.

[0114] In some embodiments, the device also employs a phased, refined noise scheduling strategy. Specifically, when the acquired preset time step is less than a preset time step threshold, a noise weight matrix with identical noise weight values ​​is determined based on the preset time step. This means that uniform noise intensity is applied throughout the target region, and each noise weight value is greater than a preset critical noise weight value. This uniform and high-intensity noise application method can completely destroy the original structure of the target region in the early stages of training, forcing the model to predict the content of the target region entirely based on the reference image and instruction text, without relying on any residual information within the target region.

[0115] Furthermore, when the preset time step is equal to or greater than a preset time step threshold, the noise weight matrix is ​​also determined based on that preset time step. However, the difference is that the noise weight values ​​in this case exhibit a special radial decay distribution. Radial decay means that the noise weight values ​​in the noise weight matrix gradually decrease from the center to the edge, forming a distribution pattern similar to water ripples or a gradient. Moreover, each noise weight value in the noise weight matrix is ​​restricted to not exceeding a preset critical noise weight value. Thus, when the image processing model has acquired a certain image processing capability in the later stages of training, by preserving more of the original structural information of the target region's edges, the image processing model is guided to better learn how to predict and generate image results for the more critical target region center.

[0116] It is understood that, in the process of adding noise from strong to weak, the embodiments of this application further divide it into two stages: in the early stage when the noise is strong, uniform high-intensity noise is used to completely erase the original information of the target area, so as to force the image processing model to rely only on the instruction text and reference image for content reconstruction, thereby enhancing its semantic driving ability and generalization; in the later stage when the noise is weak, the structural clues of the target area edge are preserved to guide the image processing model to focus on the fine generation of the central key area, while using edge context information to improve the overall visual coherence.

[0117] In step 240, the noise image features of the noise image and the text features extracted from the instruction text are obtained.

[0118] In some embodiments, the noisy image and the original instruction text are converted into feature vectors that can be directly processed by the image processing model. For example... Figure 4 As shown, Figure 4This is a data processing diagram illustrating the training method of the image processing model provided in this application embodiment. A noisy image is input into a pre-trained image autoencoder, which outputs the noise image features obtained after adding noise to the target region. Instruction text is input into a pre-trained text encoder to obtain text features. The pre-trained image autoencoder and the pre-trained text encoder are both trained based on sample noisy images.

[0119] In step 250, text features and noise image features are input into the image processing model, a predicted image is output, and the image noise corresponding to the target region in the predicted image is obtained. The noise difference between the image noise and the label noise is calculated. The image processing model is iteratively trained based on the noise difference to obtain the trained image processing model. The trained image processing model is used to process the target source image according to the target instruction text to obtain the target generated image.

[0120] In some embodiments, such as Figure 4 As shown, text features and noisy image features are input into the image processing model. The image processing model performs calculations based on the input information and outputs a predicted image. It determines the image noise based on the image features of the target region in the original noisy image features and the image features of the target region in the predicted image. Next, the noise difference between the image noise predicted by the image processing model and the label noise pre-generated in step 220 is calculated. The noise difference is usually quantified by a loss function. Finally, the image processing model is iteratively trained based on the noise difference. After sufficient training, the trained image processing model is obtained, which has the ability to process a brand-new target source image based on the target instruction text given by the user and generate a target generated image that conforms to the text description and the context of the source image.

[0121] Furthermore, the features of the noisy image Preset time steps corresponding to noisy image features Text features The noise is fed into the image processing model to obtain the image noise corresponding to the target region in the predicted image. The noise difference between image noise and label noise is calculated using the following formula, so that the image processing model can be iteratively trained based on the noise difference:

[0122] .in, Indicates noise difference; For expectations; Tag noise, To complete the masking of the target region; Represents element-wise multiplication; For image noise, A learnable image processing model; For noisy image features, For the current preset time step, These are the text features corresponding to the instruction text.

[0123] It is understood that this application addresses the problem of poor versatility and inability to cover all downstream tasks in traditional methods due to reliance on limited task branches in pre-trained models. By stitching the source image and the corresponding label image into a stitched image, the task is uniformly transformed into the problem of predicting the labeled target region. More specifically, this application normalizes various heterogeneous downstream tasks such as image inpainting and semantic editing into a single mask completion paradigm, thereby eliminating the dependence on pre-set task branch networks in traditional methods. This allows a single image processing model to flexibly adapt to any new task that can be described by this paradigm. Since the training process focuses only on noise prediction of the target region while retaining the source image as a clear context, the image processing model can learn the mapping relationship of specific tasks more efficiently. Without changing the main structure of the model, the versatility and scalability of the model are improved, ensuring that the trained model can output more accurate task processing results in various real-world task scenarios.

[0124] In addition, for ordinary users with limited professional knowledge, there is no need to focus on changes to the model structure of the image processing model as in traditional methods. They only need to focus on collecting training data and inputting it into the image processing model to complete the model training, which greatly reduces the threshold for model training for ordinary users.

[0125] In some embodiments, the image processing model is iteratively trained based on noise differences to obtain a trained image processing model, including:

[0126] (3.1) Update the initial low-rank matrix according to the noise difference to obtain the updated low-rank matrix;

[0127] (3.2) The updated low-rank matrix is ​​used as the new initial low-rank matrix in the image processing model, and the steps of obtaining the source image, the label image and the instruction text corresponding to each image processing task are returned until the noise difference between the calculated image noise and the label noise is less than the preset noise threshold, and the trained image processing model is obtained.

[0128] In some embodiments, the image processing model includes a pre-constructed initial low-rank matrix. The image processing model can be a diffusion model, and the initial low-rank matrix refers to a learnable parameter matrix with a lower rank pre-constructed within the image processing model. During training, instead of adjusting the overall network parameters of the image processing model, a low-rank adaptation technique is used to fine-tune the attention layer within the image processing model to achieve efficient downstream task adaptation. Specifically, the weight matrices of the query, key, and value in the attention mechanism of the image processing model are selected as the fine-tuning objects. By inserting learnable initial low-rank matrices before and after these weight matrices, the traditional method of adjusting all parameters of the entire model is transformed into parameter learning of the low-rank matrix; that is, parameter adjustment is achieved by multiplying the original parameter matrix and the low-rank matrix.

[0129] Furthermore, during training, the original parameters of the pre-trained diffusion model are frozen, including all network parameters except for the low-rank matrix of the attention layer. Only the parameters of the inserted low-rank matrix are updated. The total number of these learnable parameters accounts for a very small proportion of the total parameters of the model, for example, it can be controlled to within one percent. This can significantly reduce the computational resource consumption of training, maintain the lightweight nature of the model, and enable the model to quickly capture the feature associations of downstream tasks through targeted adjustments to the attention layer, such as the semantic mapping between text and target regions, and the visual matching between the source image context and the target region. This accelerates the model convergence process, achieves effective adaptation to downstream tasks in a short time, and avoids the problem of forgetting pre-trained knowledge caused by large-scale parameter adjustments.

[0130] Furthermore, after completing one adjustment of the initial low-rank matrix, the process returns to step 210, which involves obtaining the source image, the label image, and the instruction text corresponding to each image processing task. This process continues to train the image processing model based on the new training data until the noise difference between the calculated image noise and the label noise is less than a preset noise threshold, thus obtaining the trained image processing model.

[0131] Furthermore, the preset noise threshold can be set according to the actual situation, and this application embodiment does not impose any restrictions on it. When the noise difference between the calculated image noise and the label noise is less than the preset noise threshold, it indicates that the current image processing model has achieved the expected ability to generate a predicted image by denoising the noisy image, and has a stable and reliable image processing capability. At this time, the iteration can be terminated and the trained image processing model can be obtained.

[0132] In some embodiments, after obtaining the trained image processing model, the method further includes:

[0133] (4.1) Obtain the initial source image and the target instruction text corresponding to the initial source image;

[0134] (4.2) Initialize the image to be processed based on the initial source image, and mark the processing area of ​​the image to be processed based on the initial source image and the target instruction text;

[0135] (4.3) Obtain the reference image features of the initial source image, the target text features extracted from the target instruction text, and the features to be processed of the image to be processed;

[0136] (4.4) Input the target text features, the features to be processed and the reference image features into the trained image processing model, and determine the target predicted image after predicting the region to be processed in the image to be processed based on the output results.

[0137] In some embodiments, after training the image processing model, the image processing model can be applied to actual image processing. The initial source image refers to the original image to be processed, while the target instruction text is a specific operation to be performed on the initial source image, described in natural language. The definition of the initial source image is similar to that of the source image, and the definition of the target instruction text is similar to that of the instruction text, so it will not be repeated here.

[0138] Furthermore, an image to be processed is initialized based on the initial source image. The image to be processed can be a pure noise image, in which case the marked region to be processed is the entire region of the image to be processed; alternatively, a portion of the region to be processed can be marked, with the remaining region being the same as the corresponding region of the initial source image, requiring no change. The definition of the region to be processed is similar to the definition of the target region. This marking process can be done manually or through an artificial intelligence model using a method similar to steps (1.1) to (1.4).

[0139] Furthermore, the initial source image is input into an image autoencoder to obtain reference image features, the image to be processed is input into the image autoencoder to obtain features to be processed, and the target instruction text is input into a text encoder to obtain target text features. The target text features, features to be processed, and reference image features are then input into a trained image processing model. The trained image processing model will only predict and generate content for the region to be processed in the image to be processed, while the rest remains unchanged. Finally, the model outputs a complete target prediction image. The target prediction image can include prediction images corresponding to both the initial source image and the image to be processed. In practice, since users usually only need the prediction image corresponding to the image to be processed, in some embodiments, the image processing model can segment the prediction image corresponding to the image to be processed as the target prediction image before outputting the final result.

[0140] Understandably, regardless of the type of downstream task, accurate prediction of the content of a specified region image can be achieved by obtaining its corresponding initial source image and target instruction text. This improves the versatility of the trained image processing model and also enhances the accuracy of the task processing results output when using the trained image processing model for actual task processing in practical applications.

[0141] In some embodiments, target text features, features to be processed, and reference image features are input into a trained image processing model, and the target predicted image after predicting the region to be processed in the image to be processed is determined based on the output results, including:

[0142] (4.4.1) Input the target text features, the features to be processed and the reference image features into the trained image processing model, and output the initial predicted image obtained after predicting the region to be processed in the image to be processed;

[0143] (4.4.2) If the region to be processed is not the entire region of the image to be processed, determine the foreground prediction image corresponding to the region to be processed from the initial prediction image;

[0144] (4.4.3) Obtain the target weight matrix corresponding to the foreground prediction image, and update the foreground prediction image according to the target weight matrix to obtain the updated foreground prediction image. The weight values ​​of each pixel in the target weight matrix are radially decaying.

[0145] (4.4.4) Update the initial prediction image based on the updated foreground prediction image to obtain the target prediction image.

[0146] In some embodiments, the target text features, the image to be processed, and the reference image features are jointly input into the trained image processing model, and the output is an initial predicted image obtained by predicting the region to be processed in the image to be processed. If the region to be processed is the entire region of the image to be processed, the initial predicted image is directly used as the target predicted image.

[0147] Furthermore, if the region to be processed is not the entire region of the image to be processed, it means that the image content of some regions in the initial prediction image is the same as the corresponding region in the initial source image. In this case, obvious intensity or texture abrupt changes are likely to occur at the boundary between the two, forming a visually hard boundary, which may destroy the overall naturalness and continuity of the image. In this case, firstly, the foreground prediction image corresponding to the region to be processed is determined from the initial prediction image; then, the foreground prediction image is updated according to the obtained target weight matrix to obtain the updated foreground prediction image. Since the target weight matrix has a radial decay distribution, the edges of the foreground prediction image can be further softened; then, the updated foreground prediction image with softened edges is fused back into the corresponding position of the initial prediction image to replace the prediction content that may have had hard boundaries, thereby completing the final target prediction image output.

[0148] It can be understood that although a radially decaying noise weight matrix is ​​used during the training phase to guide the model to focus more on the center of the target region and utilize edge context information during reconstruction, this strategy only affects the model's ability to predict noise and cannot completely eliminate the discontinuity between the generated content and the original image at the boundary. In the inference generation phase, a radially decaying weight matrix is ​​used again to weight and fuse the foreground prediction image in order to explicitly achieve a smooth edge transition, gradually reduce the weight of the generated content at the edge of the target region, enhance the preservation of the original background, effectively alleviate the problem of harsh edges in the generated image corresponding to the region to be processed, and thus improve the overall visual effect while maintaining semantic accuracy.

[0149] like Figure 5 As shown, Figure 5 This is a schematic diagram of the module structure of the training device for the image processing model provided in this application embodiment. The training device 300 based on the image processing model may include the following modules 310 to 350:

[0150] The acquisition module 310 is used to acquire, for each different image processing task, the source image, the label image corresponding to the source image, and the instruction text corresponding to the source image;

[0151] The labeling module 320 is used to stitch together a label image and a source image to obtain a stitched image, to label at least a portion of the label image in the stitched image to obtain a target region, and to obtain the label noise corresponding to the target region;

[0152] The noise-adding module 330 is used to add noise to the target area to obtain a noise image corresponding to the stitched image;

[0153] The feature extraction module 340 is used to obtain the noise image features of the noise image and the text features extracted from the instruction text;

[0154] The prediction module 350 is used to input text features and noisy image features into the image processing model, output the predicted image and obtain the image noise corresponding to the target region in the predicted image, calculate the noise difference between the image noise and the label noise, and iteratively train the image processing model based on the noise difference to obtain the trained image processing model. The trained image processing model is used to process the target source image according to the target instruction text to obtain the target generated image.

[0155] In some embodiments, the marking module 320 is used for:

[0156] If the instruction text representation requires image processing of the entire region of the source image, then the entire region of the label image in the stitched image is marked to obtain the target region;

[0157] Otherwise, the initial region is obtained by coarsely filtering the region to be processed in the label image of the stitched image according to the instruction text;

[0158] Based on the instruction text, determine the expected semantic type label of the region to be processed, and perform image semantic analysis on the source image to obtain the reference semantic type label corresponding to the source image;

[0159] The initial region is further refined based on the expected semantic type label and the reference semantic type label to obtain the updated region. The updated region in the stitched image is then marked to obtain the target region.

[0160] In some embodiments, the noise-adding module 330 is used for:

[0161] The stitched image is downsampled to obtain a downsampled stitched image;

[0162] Obtain a preset time step and determine the noise weight matrix based on the preset time step;

[0163] Noise is added to the target region in the downsampled stitched image based on the noise weight matrix to obtain the noise image corresponding to the stitched image.

[0164] In some embodiments, the noise-adding module 330 is further configured to:

[0165] When the preset time step is less than the preset time step threshold, the noise weight matrix is ​​determined according to the preset time step, wherein each noise weight value in the noise weight matrix is ​​the same and greater than the preset critical noise weight value.

[0166] When the preset time step is equal to or greater than the preset time step threshold, the noise weight matrix is ​​determined according to the preset time step. The noise weight values ​​in the noise weight matrix are radially decaying, and each noise weight value does not exceed the critical noise weight value.

[0167] In some embodiments, the image processing model includes a pre-constructed initial low-rank matrix, and the prediction module 350 is used for:

[0168] The initial low-rank matrix is ​​updated based on the noise difference to obtain the updated low-rank matrix;

[0169] The updated low-rank matrix is ​​used as the new initial low-rank matrix in the image processing model, and the steps of obtaining the source image, the label image and the instruction text corresponding to each image processing task are returned until the noise difference between the calculated image noise and the label noise is less than the preset noise threshold, thus obtaining the trained image processing model.

[0170] In some embodiments, the prediction module 350 is further configured to:

[0171] Obtain the initial source image and the corresponding target instruction text;

[0172] The image to be processed is initialized based on the initial source image, and the area to be processed in the image is marked based on the initial source image and the target instruction text.

[0173] Obtain the reference image features of the initial source image, the target text features extracted from the target instruction text, and the features to be processed of the image to be processed;

[0174] The target text features, the features to be processed, and the reference image features are input into the trained image processing model. Based on the output results, the target predicted image is determined after predicting the region to be processed in the image to be processed.

[0175] In some embodiments, the prediction module 350 is further configured to:

[0176] The target text features, the features to be processed, and the reference image features are input into the trained image processing model, and the output is the initial predicted image obtained after predicting the region to be processed in the image to be processed.

[0177] If the region to be processed is not the entire region of the image to be processed, determine the foreground prediction image corresponding to the region to be processed from the initial prediction image;

[0178] Obtain the target weight matrix corresponding to the foreground prediction image, and update the foreground prediction image according to the target weight matrix to obtain the updated foreground prediction image. The weight values ​​of each pixel in the target weight matrix have a radial decay distribution.

[0179] The target prediction image is obtained by updating the initial prediction image based on the updated foreground prediction image.

[0180] The image processing model training method, apparatus, electronic device, and storage medium proposed in this application acquire, for each different image processing task, a source image, a label image corresponding to the source image, and instruction text corresponding to the source image; concatenate the label image and the source image to obtain a concatenated image; label at least a portion of the label image in the concatenated image to obtain a target region, and acquire the label noise corresponding to the target region; add noise to the target region to obtain a noise image corresponding to the concatenated image; acquire the noise image features of the noise image and the text features extracted from the instruction text; input the text features and noise image features into the image processing model, output a predicted image, acquire the image noise corresponding to the target region in the predicted image, calculate the noise difference between the image noise and the label noise, and iteratively train the image processing model based on the noise difference to obtain a trained image processing model. The trained image processing model is used to process the target source image according to the target instruction text to obtain the target generated image.

[0181] In this embodiment, when training the image processing model, the source image and label image of any downstream task are stitched together into a composite stitched image. Only the region in the stitched image where the predicted content is to be generated is marked as the target region. This transforms diverse tasks such as image restoration and semantic editing into a unified target region prediction problem. During training, the model only adds noise to the target region and calculates the noise difference, forcing the model to learn how to output the desired image processing task result given a source image and corresponding instruction text. Thus, this embodiment allows the pre-trained image processing model to be adapted to various new downstream tasks without any modification to the model structure, greatly improving the model's versatility and scalability, and consequently increasing the accuracy of the task processing results output when using this model for actual task processing.

[0182] like Figure 6 As shown, Figure 6 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device includes:

[0183] The processor 401 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0184] The memory 402 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 402 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 402 and is called by the processor 401 to execute the training method of the image processing model of the embodiments of this application.

[0185] Input / output interface 403 is used to implement information input and output;

[0186] The communication interface 404 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0187] Bus 405 transmits information between various components of the device (e.g., processor 401, memory 402, input / output interface 403, and communication interface 404);

[0188] The processor 401, memory 402, input / output interface 403 and communication interface 404 are connected to each other within the device via bus 405.

[0189] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described image processing model training method.

[0190] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0191] The image processing model training method, apparatus, electronic device, and storage medium proposed in this application acquire, for each different image processing task, a source image, a label image corresponding to the source image, and instruction text corresponding to the source image; concatenate the label image and the source image to obtain a concatenated image; label at least a portion of the label image in the concatenated image to obtain a target region, and acquire the label noise corresponding to the target region; add noise to the target region to obtain a noise image corresponding to the concatenated image; acquire the noise image features of the noise image and the text features extracted from the instruction text; input the text features and noise image features into the image processing model, output a predicted image, acquire the image noise corresponding to the target region in the predicted image, calculate the noise difference between the image noise and the label noise, and iteratively train the image processing model based on the noise difference to obtain a trained image processing model. The trained image processing model is used to process the target source image according to the target instruction text to obtain the target generated image.

[0192] In this embodiment, when training the image processing model, the source image and label image of any downstream task are stitched together into a composite stitched image. Only the region in the stitched image where the predicted content is to be generated is marked as the target region. This transforms diverse tasks such as image restoration and semantic editing into a unified target region prediction problem. During training, the model only adds noise to the target region and calculates the noise difference, forcing the model to learn how to output the desired image processing task result given a source image and corresponding instruction text. Thus, this embodiment allows the pre-trained image processing model to be adapted to various new downstream tasks without any modification to the model structure, greatly improving the model's versatility and scalability, and consequently increasing the accuracy of the task processing results output when using this model for actual task processing.

[0193] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0194] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0195] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0196] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0197] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0198] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0199] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0200] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0201] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0202] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0203] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A training method for an image processing model, characterized in that, include: For each different image processing task, obtain the source image, the label image, and the instruction text corresponding to the source image for each image processing task; The label image and the source image are stitched together to obtain a stitched image. At least a portion of the label image in the stitched image is marked to obtain a target region, and the label noise corresponding to the target region is obtained. Noise is added to the target region to obtain a noise image corresponding to the stitched image; Obtain the noise image features of the noise image and the text features extracted from the instruction text; The text features and the noise image features are input into the image processing model, a predicted image is output, and the image noise corresponding to the target region in the predicted image is obtained. The noise difference between the image noise and the label noise is calculated. The image processing model is iteratively trained based on the noise difference to obtain the trained image processing model. The trained image processing model is used to process the target source image according to the target instruction text to obtain the target generated image.

2. The training method for the image processing model according to claim 1, characterized in that, The step of marking at least a portion of the label image in the stitched image to obtain the target region includes: If the instruction text representation requires image processing of the entire region of the source image, then the entire region of the label image in the stitched image is marked to obtain the target region; Otherwise, the initial region is obtained by coarsely filtering the region to be processed in the label image of the stitched image according to the instruction text; Based on the instruction text, determine the expected semantic type label of the region to be processed, and perform image semantic analysis on the source image to obtain the reference semantic type label corresponding to the source image; The initial region is further refined based on the expected semantic type label and the reference semantic type label to obtain an updated region, and the updated region in the stitched image is marked to obtain the target region.

3. The training method for the image processing model according to claim 1, characterized in that, The step of adding noise to the target region to obtain a noisy image corresponding to the stitched image includes: The stitched image is downsampled to obtain a downsampled stitched image; Obtain a preset time step, and determine a noise weight matrix based on the preset time step; The target region in the downsampled stitched image is denoised according to the noise weight matrix to obtain a noise image corresponding to the stitched image.

4. The training method for the image processing model according to claim 3, characterized in that, The step of obtaining a preset time step and determining a noise weight matrix based on the preset time step includes: When the preset time step is less than the preset time step threshold, a noise weight matrix is ​​determined according to the preset time step, wherein each noise weight value in the noise weight matrix is ​​the same and greater than the preset critical noise weight value. When the preset time step is equal to or greater than the preset time step threshold, a noise weight matrix is ​​determined based on the preset time step, wherein each noise weight value in the noise weight matrix exhibits a radial decay distribution, and each noise weight value does not exceed the critical noise weight value.

5. The training method for the image processing model according to claim 1, characterized in that, The image processing model includes a pre-constructed initial low-rank matrix; The step of iteratively training the image processing model based on the noise difference to obtain the trained image processing model includes: The initial low-rank matrix is ​​updated based on the noise difference to obtain the updated low-rank matrix; The updated low-rank matrix is ​​used as the new initial low-rank matrix in the image processing model, and the steps of obtaining the source image, the label image and the instruction text corresponding to each image processing task are returned until the noise difference between the calculated image noise and the label noise is less than a preset noise threshold, thus obtaining the trained image processing model.

6. The training method for the image processing model according to claim 1, characterized in that, After obtaining the trained image processing model, the following is also included: Obtain the initial source image and the target instruction text corresponding to the initial source image; The image to be processed is initialized based on the initial source image, and the processing area of ​​the image to be processed is marked based on the initial source image and the target instruction text; Obtain the reference image features of the initial source image, the target text features extracted from the target instruction text, and the features to be processed of the image to be processed; The target text features, the features to be processed, and the reference image features are input into the trained image processing model, and the target predicted image is determined based on the output results after predicting the region to be processed in the image to be processed.

7. The training method for the image processing model according to claim 6, characterized in that, The step of inputting the target text features, the features to be processed, and the reference image features into the trained image processing model, and determining the target predicted image after predicting the region to be processed in the image to be processed based on the output results, includes: The target text features, the features to be processed, and the reference image features are input into the trained image processing model, and the output is an initial predicted image obtained by predicting the region to be processed in the image to be processed. If the region to be processed is not the entire region of the image to be processed, determine the foreground prediction image corresponding to the region to be processed from the initial prediction image; Obtain the target weight matrix corresponding to the foreground prediction image, and update the foreground prediction image according to the target weight matrix to obtain the updated foreground prediction image, wherein the weight values ​​of each pixel in the target weight matrix have a radial decay distribution; The target prediction image is obtained by updating the initial prediction image based on the updated foreground prediction image.

8. A training device for an image processing model, characterized in that, include: The acquisition module is used to acquire, for each different image processing task, the source image corresponding to the source image, the label image corresponding to the source image, and the instruction text corresponding to the source image; A labeling module is used to stitch the label image and the source image to obtain a stitched image, to label at least a portion of the label image in the stitched image to obtain a target region, and to obtain the label noise corresponding to the target region; A noise-adding module is used to add noise to the target region to obtain a noise image corresponding to the stitched image; The feature extraction module is used to obtain the noise image features of the noise image and the text features extracted from the instruction text; The prediction module is used to input the text features and the noise image features into the image processing model, output the predicted image and obtain the image noise corresponding to the target region in the predicted image, calculate the noise difference between the image noise and the label noise, and iteratively train the image processing model according to the noise difference to obtain the trained image processing model. The trained image processing model is used to process the target source image according to the target instruction text to obtain the target generated image.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the training method of the image processing model according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the training method of the image processing model according to any one of claims 1 to 7.

Images