Data processing method, device, equipment, computer readable storage medium and product

Abstract

The present disclosure provides a data processing method, device, equipment, computer readable storage medium and product, relates to the field of artificial intelligence, and particularly relates to the field of deep learning. The specific implementation scheme is: obtaining a training data set, the training data set including multiple groups of training data, the training data including a first image subjected to pixel processing in advance, a second image including random noise, and preset text description information, an edge part of a content subject in the first image being configured as a preset first pixel value, and an area other than the content subject in the first image being configured as a preset second pixel value; performing a training operation on a preset to-be-trained diffusion model by using the training data set until the to-be-trained diffusion model meets a preset convergence condition, and obtaining a trained target diffusion model. Thus, the accuracy of a target image output by the diffusion model is improved, and the diffusion model is prevented from performing random external expansion on the edge part of an input image when generating the target image.

Description

Technical Field

[0001] This disclosure relates to deep learning in artificial intelligence, and more particularly to a data processing method, apparatus, device, computer-readable storage medium, and product. Background Technology

[0002] Shooting product promotional images is a complex and time-consuming task. To quickly generate multiple product promotional images, existing technologies generally use diffusion models. However, during image generation, although the generated background can be controlled through text descriptions, there are often instances where the edges of the product image are arbitrarily expanded and supplemented, such as extending the product or adding an extra hand to a person, thus altering the main subject of the product and making it impossible to effectively generate promotional images. Summary of the Invention

[0003] This disclosure provides a data processing method, apparatus, device, computer-readable storage medium, and product for solving the technical problem of arbitrary outward expansion and supplementation of the edge portion of the input image when a diffusion model generates a target image.

[0004] According to a first aspect of this disclosure, a data processing method is provided, comprising:

[0005] Obtain a training dataset, wherein the training dataset includes multiple sets of training data, the training data includes a first image that has undergone pixel processing, a second image including random noise, and preset text description information, wherein the edge part of the main content in the first image is configured as a preset first pixel value, and the area outside the main content in the first image is configured as a preset second pixel value, wherein the first pixel value is greater than the second pixel value.

[0006] The training dataset is used to train the preset diffusion model until the diffusion model meets the preset convergence condition, thereby obtaining the trained target expansion model.

[0007] According to a second aspect of this disclosure, a data processing method is provided, comprising:

[0008] Obtain an image generation request, wherein the image generation request includes an associated image of the target item and preset background description information;

[0009] According to the image generation request, the pixel values ​​of the edge portion of the target item in the associated image are adjusted to a preset first pixel value, and the pixel values ​​of the non-target item area in the associated image are adjusted to a preset second pixel value to obtain a control image;

[0010] The control image and the background description information are input into a preset target extension model to obtain at least one extended image associated with the target item output by the target extension model;

[0011] The target extended model is obtained by training based on the data processing method described in the first aspect.

[0012] According to a third aspect of this disclosure, a data processing apparatus is provided, comprising:

[0013] The acquisition module is used to acquire a training dataset, wherein the training dataset includes multiple sets of training data, the training data includes a first image that has undergone pixel processing, a second image including random noise, and preset text description information, wherein the edge part of the main content in the first image is configured as a preset first pixel value, and the area outside the main content in the first image is configured as a preset second pixel value, wherein the first pixel value is greater than the second pixel value.

[0014] The training module is used to train a preset diffusion model to be trained using the training dataset until the diffusion model to be trained meets the preset convergence condition, thereby obtaining the trained target expansion model.

[0015] According to a fourth aspect of this disclosure, a data processing apparatus is provided, comprising:

[0016] The request acquisition module is used to acquire an image generation request, wherein the image generation request includes an associated image corresponding to the target item and preset background description information;

[0017] The pixel adjustment module is used to adjust the pixel values ​​of the edge portion of the target item in the associated image to a preset first pixel value and adjust the pixel values ​​of the non-target item area in the associated image to a preset second pixel value according to the image generation request, so as to obtain a control image;

[0018] An input module is used to input the control image and the background description information into a preset target extension model to obtain at least one extended image associated with the target item output by the target extension model;

[0019] The target extended model is obtained by training based on the data processing device described in the third aspect.

[0020] According to a fifth aspect of this disclosure, an electronic device is provided, comprising:

[0021] At least one processor; and

[0022] A memory communicatively connected to the at least one processor; wherein,

[0023] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect or the second aspect.

[0024] According to a sixth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the method described in either the first or second aspect.

[0025] According to a seventh aspect of this disclosure, a computer program product is provided, the computer program product comprising: a computer program stored in a readable storage medium, at least one processor of an electronic device being able to read the computer program from the readable storage medium, the at least one processor executing the computer program causing the electronic device to perform the method described in the first aspect or the second aspect.

[0026] The technology disclosed herein improves the accuracy of the target image output by the diffusion model and avoids the diffusion model arbitrarily expanding and supplementing the edge parts of the input image when generating the target image.

[0027] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0028] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0029] Figure 1 This is a system architecture diagram on which this disclosure is based;

[0030] Figure 2 A schematic flowchart illustrating the data processing method provided in this embodiment of the disclosure;

[0031] Figure 3 A schematic flowchart of a data processing method provided in yet another embodiment of this disclosure;

[0032] Figure 4 This is a schematic diagram of a model training scenario provided in an embodiment of the present disclosure;

[0033] Figure 5 A schematic flowchart of a data processing method provided in yet another embodiment of this disclosure;

[0034] Figure 6 A schematic flowchart of a data processing method provided in yet another embodiment of this disclosure;

[0035] Figure 7 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this disclosure;

[0036] Figure 8 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this disclosure;

[0037] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0038] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0039] This disclosure provides a data processing method, apparatus, device, computer-readable storage medium, and product, which are applied to deep learning in the field of artificial intelligence to achieve the technical effect of improving the accuracy of the target image output by the diffusion model.

[0040] The data processing methods, apparatus, devices, computer-readable storage media, and products disclosed herein can be applied to any application scenario that generates multiple images based on a diffusion model.

[0041] In the process of generating product promotional images using diffusion models, although the generated background can be controlled through text descriptions, there is a frequent issue of arbitrary expansion and addition to the edges of the product image itself, such as extending the product or adding an extra hand to a person. This alters the main product and hinders the effective generation of promotional images. To address these technical issues, related technologies utilize control maps generated based on semantic segmentation models for edge control. Semantic segmentation models segment the image, and different solid-color semantic maps are used to replace different segmented parts. However, predefined solid-color semantic maps are insufficient to cover all products, limiting the generation effect. Furthermore, the semantic color values ​​of the semantically segmented background also correspond to objects, significantly limiting the richness of the generated background and resulting in a monotonous background that negatively impacts the overall quality.

[0042] In solving the aforementioned technical problems, the inventors discovered that, to prevent the diffusion model from arbitrarily expanding and supplementing the edges of the target item when generating more images based on the target item's image, when acquiring the target item's image, the edges of the target item in the image can be set as the first pixel value, and the area outside the target item in the image can be set as the second pixel value. The diffusion model is then constrained based on the first and second pixel values ​​during model training. The first pixel value can be the maximum pixel value, and the second pixel value can be negative. Therefore, based on these first and second pixel values, the integrity and effectiveness of the product edges can be maintained during model training, effectively improving the accuracy of the generated images.

[0043] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0044] To help readers gain a deeper understanding of the implementation principles of this disclosure, the following will be discussed in conjunction with... Figures 1-9 The embodiments disclosed herein are further refined.

[0045] Figure 1 The system architecture diagram on which this disclosure is based is as follows: Figure 1 As shown, the system architecture upon which this disclosure is based includes at least a data server 11 and a server 12, wherein the data server 11 and the server 12 are communicatively connected. The server 12 is equipped with a data processing device, which can be written in languages ​​such as C / C++, Java, Shell, or Python; the data server 11 can be a cloud server or a server cluster, which stores a large amount of data.

[0046] Based on the above system architecture, server 12 can obtain training dataset from data server 11 and train the preset diffusion model to be trained based on the training dataset until the diffusion model to be trained meets the preset convergence condition, thereby obtaining the trained target expansion model.

[0047] Figure 2 This is a flowchart illustrating the data processing method provided in the embodiments of this disclosure, as shown below. Figure 2 As shown, the method includes:

[0048] Step 201: Obtain a training dataset, wherein the training dataset includes multiple sets of training data, the training data includes a first image that has undergone pixel processing, a second image including random noise, and preset text description information, wherein the edge part of the main content in the first image is configured as a preset first pixel value, and the area outside the main content in the first image is configured as a preset second pixel value, wherein the first pixel value is greater than the second pixel value.

[0049] In this embodiment, the execution entity is a data processing device, which can be coupled to a server. The server can communicate with the data server, thereby obtaining the training dataset from the data server and performing training operations on the diffusion model to be trained.

[0050] In this embodiment, to train the diffusion model to be trained, a training dataset is first required. This training dataset includes multiple sets of training data. Each set of training data consists of a first image that has undergone pre-processing of pixels, a second image including random noise, and pre-defined textual description information.

[0051] Optionally, the first image can be obtained by performing pixel processing on a preset image containing the main content. During pixel processing, the display position of the main content in the image to be processed can be determined, the edge of the main content is set as the first pixel value, and the non-main content area in the image to be processed is set as the second pixel value. The first pixel is greater than the second pixel. Therefore, the first image, based on these different first and second pixels, can effectively control edges during model training, enabling the trained target diffusion model to control edges while fully preserving the main content, thus avoiding arbitrary edge additions by the trained target diffusion model.

[0052] As one possible approach, the first pixel value can be the maximum pixel value, and the second pixel value can be a preset negative pixel value. For example, in a practical application, the first pixel value could be 512, and the second pixel value could be -255. By using the maximum pixel value to fill the edges of the main content area and using negative pixels to fill the non-main areas, the difference between the edges of the main content area and the non-main areas can be effectively improved, thereby enabling effective edge control during model training.

[0053] Furthermore, during model training, in order to improve the data processing accuracy of the trained target extended model, the loss value of the model to be trained can be determined in each training iteration, and the model to be trained can be iterated multiple times based on the loss value of each training iteration. In order to calculate the loss value, the training data can also include a second image with added random noise.

[0054] Optionally, to further restrict non-subject regions, the training dataset may also include textual descriptions that describe the background portion of the image to be processed.

[0055] Step 202: Use the training dataset to train the preset diffusion model to be trained until the diffusion model to be trained meets the preset convergence condition, and obtain the trained target expansion model.

[0056] In this embodiment, after obtaining the training dataset, the preset diffusion model to be trained can be trained based on the training dataset.

[0057] Optionally, training data from the training dataset can be input into the diffusion model to be trained to obtain the loss value output by the diffusion model. Based on this loss value, the diffusion model to be trained is iteratively trained until it meets a preset convergence condition, thereby obtaining the trained target expansion model.

[0058] The preset convergence conditions include, but are not limited to, the number of training iterations reaching a preset threshold, the training duration reaching a preset duration threshold, the loss value output by the diffusion model being trained being less than a preset loss value threshold, and the difference between the loss values ​​output by the diffusion model being trained in N training iterations being less than a preset difference threshold. In practical applications, the convergence conditions can be adjusted according to actual needs, and this disclosure does not impose any restrictions on this.

[0059] The data processing method provided in this embodiment, in order to train the diffusion model to be trained, can acquire a training dataset. This training dataset includes a first image that has undergone pre-processing of pixels, a second image with added random noise, and textual description information. In the first image, the edge portion of the main content is configured with a preset first pixel value, and the area outside the main content is configured with a preset second pixel value, where the first pixel value is greater than the second pixel value. Therefore, during the training operation of the diffusion model to be trained based on the training dataset, the edges can be controlled according to different pixel values ​​in the second image. When the diffusion model to be trained meets the preset convergence condition, the trained target expansion model is obtained. This target diffusion model improves the accuracy of the target image output by the diffusion model and avoids the diffusion model arbitrarily expanding and supplementing the edge portion of the input image when generating the target image.

[0060] Figure 3 This is a flowchart illustrating a data processing method provided in yet another embodiment of the present disclosure. Based on any of the above embodiments, such as... Figure 3 As shown, step 201 includes:

[0061] Step 301: Obtain a preset image dataset, wherein the image dataset includes multiple images to be processed and text description information corresponding to each image to be processed, and the images to be processed include a main content area.

[0062] Step 302: For each image to be processed, identify the display position of the main content in the image to be processed.

[0063] Step 303: Based on the display position of the main content, perform pixel adjustment operations on the edge portion of the main content and the area outside the main content in the image to be processed to obtain the first image.

[0064] Step 304: Add random noise to the image to be processed to obtain the second image.

[0065] Step 305: Determine the first image, the second image, and the text description information corresponding to the image to be processed as the training data corresponding to the image to be processed.

[0066] Step 306: Construct the training dataset based on the training data corresponding to each image to be processed.

[0067] In this embodiment, to obtain the training dataset, a preset image dataset is first required. This image dataset may include multiple images to be processed and corresponding text descriptions for each image. Each image to be processed includes a main content region. For example, the main content can be a target item, such as clothing, food, or any other type of content. The image to be trained includes both the main content region and a background region. To better constrain the edges of the image to be trained, the image dataset may also include text descriptions, which can be used to describe the background region.

[0068] Furthermore, after acquiring the image dataset, for each image to be processed in the dataset, the display position of the main content in that image can be determined. Based on the display position of the main content, the edge portion of the main content and the non-subject areas can be accurately determined. Then, based on these areas, the edge portion can be filled with a first pixel value, and the non-subject areas can be filled with a second pixel value to obtain the first image.

[0069] Furthermore, to improve the data processing accuracy of the trained target expansion model, the loss value of the model to be trained can be determined in each training process, and the model to be trained can be iterated multiple times based on the loss value of each training. In order to calculate the loss value, random noise can be added to the image to be processed to obtain a second image.

[0070] The first image, the second image, and the text description information corresponding to an image to be processed are identified as a set of training data. This allows multiple sets of training data to be obtained based on the image dataset, and these multiple sets of training data are then used to construct a training dataset.

[0071] Figure 4 This is a schematic diagram of a model training scenario provided in an embodiment of this disclosure, such as... Figure 4 As shown, a preset image dataset 41 can be obtained, which includes multiple images 42 to be processed and text description information 43 corresponding to each image. Each image includes a main content region 44. Based on the display position of the main content region 44, pixel adjustments are performed on the edge portion 45 of the main content region and the region outside the main content region 44 in the image 42 to obtain a first image 46. Random noise is added to the image 42 to obtain a second image 47. Training data is constructed based on the first image 46 and the second image 47. A training dataset 48 is constructed based on multiple sets of training data. Thus, a preset diffusion model 49 to be trained can be trained using the training dataset 48 until the diffusion model 49 meets the preset convergence condition, resulting in a target diffusion model 410.

[0072] The data processing method provided in this embodiment obtains a first image by performing pixel adjustments on the edge portion of the content subject and the area outside the content subject in the image to be processed based on the display position of the content subject. Random noise is added to the image to be processed to obtain a second image. Thus, the training dataset composed of the first and second images can perform edge control during model training, avoiding arbitrary edge addition by the trained target diffusion model and improving the accuracy of the images generated by the target diffusion model.

[0073] Furthermore, based on any of the above embodiments, step 303 includes:

[0074] The non-subject areas in the image to be processed, excluding the main content, are determined based on the display position of the main content.

[0075] Set the pixels in the non-subject area to a preset second pixel value.

[0076] The boundary between the display area of ​​the main content and the non-main content area is defined as the edge portion of the main content.

[0077] Set the pixels of the edge portion to a preset first pixel value.

[0078] In this embodiment, after determining the display position of the main content element in the image to be processed, the non-subject areas in the image to be processed, excluding the main content element, can be accurately determined based on this display position. After determining the non-subject areas, a preset second pixel value can be used to fill these non-subject areas.

[0079] After filling the non-subject area, the pixel values ​​of the non-subject area differ from those of the main content area. Therefore, the boundary between the display area of ​​the main content and the non-subject area can be defined as the edge of the main content. The pixels of the edge are set to a preset first pixel value. This first pixel value can be 512, and the second pixel value can be -255. By using the maximum pixel value to fill the edge of the main content and negative pixel values ​​to fill the non-subject area, the difference between the edge of the main content and the non-subject area can be effectively improved.

[0080] The data processing method provided in this embodiment sets different pixel values ​​for the edges of the main content and the non-main content areas, thereby controlling the edges while fully preserving the main content and avoiding the arbitrary generation of edges by the trained target diffusion model.

[0081] Furthermore, based on any of the above embodiments, setting the pixels of the non-subject area to a preset second pixel value includes:

[0082] The image to be processed is inverted to obtain a first inverted image.

[0083] Set the pixels of the non-subject region in the first inverted image to a preset second pixel value.

[0084] In this embodiment, to facilitate pixel processing of non-subject areas, an image inversion operation can be performed on the image to be processed to obtain a first inverted image. By inverting the image to be processed, the pixel intensity of the image can be reversed, making the foreground become the background and the background become the foreground. This allows the pixels of the non-subject areas in the first inverted image to be accurately set to a preset second pixel value.

[0085] The data processing method provided in this embodiment improves the accuracy and efficiency of pixel adjustment by performing an image inversion operation on the image to be processed to obtain a first inverted image and then performing pixel adjustment operations based on the first inverted image.

[0086] Furthermore, based on any of the above embodiments, the image to be processed includes preset annotation information, which is used to annotate the display area of ​​the main content.

[0087] The step of identifying the display position of the main content element in the image to be processed includes:

[0088] The display position of the main content in the image to be processed is determined based on the annotation information corresponding to the image to be processed.

[0089] Alternatively, identifying the display position of the main content element in the image to be processed includes:

[0090] The display position of the main content in the image to be processed is identified according to a preset subject recognition model.

[0091] In this embodiment, the image to be processed may contain preset annotation information, which is used to mark the display area of ​​the main content. Therefore, after obtaining the image dataset, the display position of the main content in the processed image can be accurately determined based on the annotation information in each image to be processed.

[0092] As one feasible approach, in order to determine the display position of the main content subject, the image to be processed can be input into a preset subject recognition model, and the display position of the main content subject in the image to be processed can be identified according to the preset subject recognition model.

[0093] Optionally, different subject recognition models can be used to identify the display position for different content subjects. For example, if the content subject is a human body, a preset human body recognition model can be used for recognition; if the content subject is a specific item, the recognition model corresponding to that specific item can be used for recognition.

[0094] The data processing method provided in this embodiment determines the display position of the main content based on the preset annotation information of the image to be processed, or identifies the display position of the main content in the image to be processed according to the preset main content recognition model. This can accurately determine the display position of the main content, improve the accuracy of subsequent pixel adjustments, and thus improve the processing accuracy of the target diffusion model trained on the training dataset.

[0095] Figure 5 This is a flowchart illustrating a data processing method provided in yet another embodiment of the present disclosure. Based on any of the above embodiments, step 202 includes:

[0096] Step 501: Input the training data in the training dataset into the diffusion model to be trained, and obtain the predicted noise information output by the diffusion model to be trained based on the first image and the text description information.

[0097] Step 502: Determine the current loss value of the diffusion model to be trained based on the predicted noise information and the random noise in the second image.

[0098] Step 503: Detect whether the diffusion model to be trained meets the preset convergence condition.

[0099] Step 504: If not, adjust the parameters of the diffusion model to be trained according to the loss value, and return to step 501.

[0100] Step 505: If yes, then determine that the diffusion model to be trained has been trained and obtain the trained target expansion model.

[0101] In this embodiment, after obtaining the training dataset, the preset diffusion model to be trained can be trained based on the training dataset.

[0102] Optionally, training data from the training dataset can be input into the diffusion model to be trained to obtain the predicted noise information output by the diffusion model based on the first image and text description information. The current loss value of the diffusion model to be trained is determined based on this predicted noise and the random noise added to the second image. It is then checked whether the diffusion model to be trained meets the preset convergence conditions.

[0103] The preset convergence conditions include, but are not limited to, the number of training iterations reaching a preset threshold, the training duration reaching a preset duration threshold, the loss value output by the diffusion model being trained being less than a preset loss value threshold, and the difference between the loss values ​​output by the diffusion model being trained in N training iterations being less than a preset difference threshold. In practical applications, the convergence conditions can be adjusted according to actual needs, and this disclosure does not impose any restrictions on this.

[0104] Optionally, if it is determined that the diffusion model to be trained currently meets the preset convergence condition, then the training can be considered complete, and the current diffusion model to be trained can be identified as the target diffusion model. Otherwise, the training can be considered incomplete. The parameters of the diffusion model to be trained can be adjusted based on this loss value. The adjusted diffusion model to be trained is then retrained based on the training dataset until the diffusion model to be trained currently meets the preset convergence condition, thus obtaining the target diffusion model.

[0105] The data processing method provided in this embodiment, after determining the current loss value of the diffusion model to be trained, checks whether the diffusion model to be trained meets a preset convergence condition. Based on the detection result, iteratively trains the diffusion model to be trained until it meets the preset convergence condition, thus obtaining the trained target expansion model. This enables the target expansion model to control edges and generate an effective output image while fully preserving the main body of the product.

[0106] Figure 6 A flowchart illustrating a data processing method provided in yet another embodiment of this disclosure is shown below. Figure 6As shown, the method includes:

[0107] Step 601: Obtain an image generation request, wherein the image generation request includes the associated image of the target item and preset background description information.

[0108] Step 602: According to the image generation request, adjust the pixel value of the edge part of the target item in the associated image to a preset first pixel value, and adjust the pixel value of the non-target item area in the associated image to a preset second pixel value to obtain a control image.

[0109] Step 603: Input the control image and the background description information into a preset target extension model to obtain at least one extended image associated with the target item output by the target extension model.

[0110] The target extended model is obtained by training based on the data processing method described in any of the above embodiments.

[0111] The execution entity in this embodiment is a data processing device, which can be coupled to a server. The server can communicate with the terminal device to obtain the image generation request sent by the terminal device, and call the target diffusion model to generate at least one extended image based on the image generation request.

[0112] In this embodiment, after completing the training operation of the diffusion model to be trained and obtaining the target diffusion model, the user can use the target diffusion model to generate expanded images based on actual needs. The target diffusion model is obtained by training using any of the above embodiments.

[0113] Optionally, an image generation request is obtained, wherein the image generation request includes an associated image corresponding to the target item and preset background description information. The associated image may include the target item, so that at least one extended image including the target item can be generated subsequently based on a target diffusion model. To make the generated extended image more tailored to the user's personalized needs, the image generation request also includes background description information, which describes the background of the extended image the user wants to generate.

[0114] Furthermore, to make the generated extended image more accurate and avoid arbitrary expansion of the target object's edges in the extended image, the pixel values ​​of the target object's edge portion in the associated image can be adjusted to a preset first pixel value, and the pixel values ​​of non-target object regions in the associated image can be adjusted to a preset second pixel value, based on the image generation request, to obtain a control image. The control image and background description information are then input into a preset target extension model to obtain at least one extended image associated with the target object, output by the target extension model.

[0115] The data processing method provided in this embodiment, after obtaining an image generation request, adjusts the pixel values ​​of the edge portion of the target item in the associated image to a preset first pixel value, and adjusts the pixel values ​​of the non-target item region in the associated image to a preset second pixel value, thereby obtaining a control image. A target expansion model trained based on the data processing method of any of the above embodiments is then used to process this control image to obtain at least one expanded image. This allows for the control of edges while fully preserving the target item, preventing arbitrary edge expansion, and generating a valid item image.

[0116] Furthermore, based on any of the above embodiments, step 602 includes:

[0117] Identify the display area of ​​the target item in the associated image.

[0118] The associated image is inverted to obtain a second inverted image.

[0119] Set the non-target item area in the second flipped image to the second pixel value.

[0120] The control image is obtained by setting the boundary between the display area of ​​the target item in the second flipped image and the area of ​​the non-target item as the first pixel value.

[0121] In this embodiment, to prevent the edges of the target object in the extended image from being arbitrarily expanded, the display area of ​​the target object in the associated image can be identified after the associated image is acquired. Any method can be used to identify the display area of ​​the target object, and this disclosure does not impose any limitations on this.

[0122] Invert the associated image to obtain a second inverted image. Identify the non-target item region in the second inverted image based on this display area. Set the non-target item region to the second pixel value.

[0123] After adjusting the pixels of the non-target item area, there is a difference in pixel values ​​between the non-target item area and the target item area. Therefore, the boundary between the display area of ​​the target item and the non-target item area in the second flipped image is set as the first pixel value to obtain the control image.

[0124] In this design, the first pixel value can be the maximum pixel value, and the second pixel value can be a preset negative pixel value. For example, in a practical application, the first pixel value could be 512, and the second pixel value could be -255. By using the maximum pixel value to fill the edge of the target object and using negative pixels to fill the non-target object area, the difference between the edge of the target object and the non-target object area can be effectively improved, thus enabling effective edge control during image generation.

[0125] The data processing method provided in this embodiment improves the processing accuracy of the target diffusion model and generates an accurate and effective target item image by adjusting the pixels of the edge parts and non-target item areas of the associated image based on the display area of ​​the target item in the associated image before image processing based on the target diffusion model.

[0126] Figure 7 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this disclosure, such as... Figure 7 As shown, the device includes an acquisition module 71 and a training module 72. The acquisition module 71 is used to acquire a training dataset, which includes multiple sets of training data. The training data includes a first image that has undergone pre-processing for pixel dimensions, a second image including random noise, and preset text description information. The edge portion of the main content in the first image is configured with a preset first pixel value, and the area outside the main content in the first image is configured with a preset second pixel value. The first pixel value is greater than the second pixel value. The training module 72 is used to train a preset diffusion model to be trained using the training dataset until the diffusion model to be trained meets a preset convergence condition, thereby obtaining the trained target expansion model.

[0127] Further, based on any of the above embodiments, the acquisition module includes: an acquisition unit, configured to acquire a preset image dataset, wherein the image dataset includes multiple images to be processed and text description information corresponding to each image to be processed, and the images to be processed include a content subject region; an identification unit, configured to identify the display position of the content subject in each image to be processed; an adjustment unit, configured to perform pixel adjustment operations on the edge portion of the content subject and the area outside the content subject in the image to be processed based on the display position of the content subject, to obtain the first image; an adding unit, configured to add random noise to the image to be processed, to obtain the second image; a determining unit, configured to determine the first image, the second image, and the text description information corresponding to the image to be processed as training data corresponding to the image to be processed; and a construction unit, configured to construct the training dataset based on the training data corresponding to each image to be processed.

[0128] Further, based on any of the above embodiments, the adjustment unit includes: a determining subunit, configured to determine a non-subject area in the image to be processed, excluding the content subject, based on the display position of the content subject; a setting subunit, configured to set the pixels of the non-subject area to a preset second pixel value; an identifying subunit, configured to determine the boundary between the display area of ​​the content subject and the non-subject area as the edge portion of the content subject; and a processing subunit, configured to set the pixels of the edge portion to a preset first pixel value.

[0129] Furthermore, based on any of the above embodiments, the setting subunit is configured to: perform an image inversion operation on the image to be processed to obtain a first inverted image; and set the pixels of the non-subject area in the first inverted image to a preset second pixel value.

[0130] Furthermore, based on any of the above embodiments, the image to be processed includes preset annotation information, which is used to annotate the display area of ​​the main content. The recognition unit includes: a recognition subunit, used to determine the display position of the main content in the image to be processed based on the annotation information corresponding to the image to be processed. Alternatively, the recognition unit includes: a recognition subunit, used to recognize the display position of the main content in the image to be processed according to a preset subject recognition model.

[0131] Furthermore, based on any of the above embodiments, the first pixel value is the maximum pixel value, and the second pixel value is a preset negative pixel value.

[0132] Further, based on any of the above embodiments, the training module includes: an input unit, configured to input training data from the training dataset into the diffusion model to be trained, and obtain prediction noise information output by the diffusion model to be trained based on the first image and the text description information; a loss value determination unit, configured to determine the current loss value of the diffusion model to be trained based on the prediction noise information and random noise in the second image; a detection unit, configured to detect whether the diffusion model to be trained meets a preset convergence condition; a first processing unit, configured to, if not, adjust the parameters of the diffusion model to be trained according to the loss value, and return to the step of inputting the training data from the training dataset into the diffusion model to be trained; and a second processing unit, configured to, if yes, determine that the diffusion model to be trained has been trained and obtain the trained target expansion model.

[0133] Figure 8 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this disclosure, such as... Figure 8As shown, the device includes: a request acquisition module 81, a pixel adjustment module 82, and an input module 83. The request acquisition module 81 is used to acquire an image generation request, wherein the image generation request includes an associated image corresponding to the target item and preset background description information. The pixel adjustment module 82 is used to adjust the pixel values ​​of the edge portion of the target item in the associated image to a preset first pixel value, and adjust the pixel values ​​of the non-target item area in the associated image to a preset second pixel value, according to the image generation request, to obtain a control image. The input module 83 is used to input the control image and the background description information into a preset target extension model to obtain at least one extended image associated with the target item output by the target extension model. The target extension model is obtained by training based on the data processing device described in any of the above embodiments.

[0134] Further, based on any of the above embodiments, the pixel adjustment module includes: a region recognition unit, used to recognize the display area of ​​the target item in the associated image; an adjustment unit, used to perform an image inversion operation on the associated image to obtain a second inverted image; a first setting unit, used to set the non-target item area in the second inverted image to a second pixel value; and a second setting unit, used to set the boundary between the display area of ​​the target item and the non-target item area in the second inverted image to the first pixel value, thereby obtaining the control image.

[0135] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0136] According to embodiments of this disclosure, this disclosure also provides an electronic device, including:

[0137] At least one processor; and

[0138] A memory communicatively connected to the at least one processor; wherein,

[0139] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method as described in any of the above embodiments.

[0140] According to embodiments of this disclosure, this disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to perform the methods described in any of the above embodiments.

[0141] According to embodiments of this disclosure, this disclosure also provides a computer program product comprising: a computer program stored in a readable storage medium, at least one processor of an electronic device being able to read the computer program from the readable storage medium, and the at least one processor executing the computer program causing the electronic device to perform the scheme provided in any of the above embodiments.

[0142] Figure 9 This is a schematic diagram of the structure of an electronic device provided for embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0143] like Figure 9 As shown, device 900 includes a computing unit 901, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 902 or a computer program loaded into random access memory (RAM) 903 from storage unit 908. RAM 903 may also store various programs and data required for the operation of device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.

[0144] Multiple components in device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of monitors, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0145] The computing unit 901 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as data processing methods. For example, in some embodiments, the data processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform data processing methods by any other suitable means (e.g., by means of firmware).

[0146] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0147] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0148] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0149] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0150] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0151] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0152] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0153] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A data processing method, comprising: Obtain an image generation request, wherein the image generation request includes an associated image of the target item and preset background description information; According to the image generation request, the pixel values ​​of the edge portion of the target item in the associated image are adjusted to a preset first pixel value, and the pixel values ​​of the non-target item area in the associated image are adjusted to a preset second pixel value to obtain a control image; The control image and the background description information are input into a preset target extension model to obtain at least one extended image associated with the target item output by the target extension model; The step of adjusting the pixel values ​​of the edge portion of the target item in the associated image to a preset first pixel value, and adjusting the pixel values ​​of the non-target item area in the associated image to a preset second pixel value, includes: Identify the display area of ​​the target item in the associated image; Perform an image inversion operation on the associated image to obtain a second inverted image; Set the non-target item area in the second inverted image to the second pixel value; The control image is obtained by setting the boundary between the display area of ​​the target item in the second inverted image and the area of ​​the non-target item as the first pixel value.

2. The method according to claim 1, further comprising: Obtain a training dataset, wherein the training dataset includes multiple sets of training data, the training data includes a first image that has undergone pixel processing, a second image including random noise, and preset text description information, wherein the edge part of the main content in the first image is configured as a preset first pixel value, and the area outside the main content in the first image is configured as a preset second pixel value, wherein the first pixel value is greater than the second pixel value. The training dataset is used to train the preset diffusion model until the diffusion model meets the preset convergence condition, thereby obtaining the trained target expansion model.

3. The method according to claim 2, wherein obtaining the training dataset includes: Obtain a preset image dataset, wherein the image dataset includes multiple images to be processed and text description information corresponding to each image to be processed, and the images to be processed include a main content area; For each image to be processed, identify the display position of the main content in the image to be processed; Based on the display position of the main content, pixel adjustment operations are performed on the edge portion of the main content and the area outside the main content in the image to be processed to obtain the first image; Random noise is added to the image to be processed to obtain the second image; The first image, the second image, and the text description information corresponding to the image to be processed are determined as the training data corresponding to the image to be processed. The training dataset is constructed based on the training data corresponding to each image to be processed.

4. The method according to claim 3, wherein performing pixel adjustment operations on the edge portion of the content subject and the area outside the content subject in the image to be processed based on the display position of the content subject to obtain the first image includes: Based on the display position of the main content, determine the non-subject areas in the image to be processed, excluding the main content area; Set the pixels in the non-subject area to a preset second pixel value; The boundary between the display area of ​​the main content and the non-main content area is defined as the edge portion of the main content. Set the pixels of the edge portion to a preset first pixel value.

5. The method according to claim 4, wherein setting the pixels of the non-subject region to a preset second pixel value comprises: Perform an image inversion operation on the image to be processed to obtain a first inverted image; Set the pixels of the non-subject region in the first inverted image to a preset second pixel value.

6. The method according to claim 3, wherein the image to be processed includes preset annotation information, the annotation information being used to annotate the display area of ​​the main content; The step of identifying the display position of the main content element in the image to be processed includes: The display position of the main content in the image to be processed is determined based on the annotation information corresponding to the image to be processed; Alternatively, identifying the display position of the main content element in the image to be processed includes: The display position of the main content in the image to be processed is identified according to a preset subject recognition model.

7. The method according to any one of claims 2-6, wherein the first pixel value is the maximum pixel value and the second pixel value is a preset negative pixel value.

8. The method according to any one of claims 2-6, wherein the step of training a preset diffusion model to be trained using the training dataset until the diffusion model to be trained satisfies a preset convergence condition to obtain the trained target expansion model includes: The training data in the training dataset is input into the diffusion model to be trained to obtain the predicted noise information output by the diffusion model to be trained based on the first image and the text description information; The current loss value of the diffusion model to be trained is determined based on the predicted noise information and the random noise in the second image. Detect whether the diffusion model to be trained meets the preset convergence condition; If not, adjust the parameters of the diffusion model to be trained according to the loss value, and return to the step of inputting the training data in the training dataset into the diffusion model to be trained; If so, the training of the diffusion model to be trained is completed, and the trained target expansion model is obtained.

9. A data processing apparatus, comprising: The request acquisition module is used to acquire an image generation request, wherein the image generation request includes an associated image corresponding to the target item and preset background description information; The pixel adjustment module is used to adjust the pixel values ​​of the edge portion of the target item in the associated image to a preset first pixel value and adjust the pixel values ​​of the non-target item area in the associated image to a preset second pixel value according to the image generation request, so as to obtain a control image; An input module is used to input the control image and the background description information into a preset target extension model to obtain at least one extended image associated with the target item output by the target extension model; The pixel adjustment module includes: A region recognition unit is used to identify the display area of ​​the target item in the associated image; The adjustment unit is used to perform an image inversion operation on the associated image to obtain a second inverted image; The first setting unit is used to set the non-target item area in the second inverted image to a second pixel value; The second setting unit is used to set the boundary between the display area of ​​the target item in the second inverted image and the non-target item area as the first pixel value to obtain the control image.

10. The apparatus according to claim 9, further comprising: The acquisition module is used to acquire a training dataset, wherein the training dataset includes multiple sets of training data, the training data includes a first image that has undergone pixel processing, a second image including random noise, and preset text description information, wherein the edge part of the main content in the first image is configured as a preset first pixel value, and the area outside the main content in the first image is configured as a preset second pixel value, wherein the first pixel value is greater than the second pixel value. The training module is used to train a preset diffusion model to be trained using the training dataset until the diffusion model to be trained meets the preset convergence condition, thereby obtaining the trained target expansion model.

11. The apparatus according to claim 10, wherein the acquisition module comprises: The acquisition unit is used to acquire a preset image dataset, wherein the image dataset includes multiple images to be processed and text description information corresponding to each image to be processed, and the images to be processed include a main content area; The recognition unit is used to identify the display position of the main content in each image to be processed; An adjustment unit is used to perform pixel adjustment operations on the edge portion of the content subject and the area outside the content subject in the image to be processed based on the display position of the content subject, so as to obtain the first image; An adding unit is used to add random noise to the image to be processed to obtain the second image; The determining unit is used to determine the first image, the second image, and the text description information corresponding to the image to be processed as the training data corresponding to the image to be processed. The construction unit is used to construct the training dataset based on the training data corresponding to each image to be processed.

12. The apparatus according to claim 11, wherein the adjustment unit comprises: A determining subunit is used to determine non-subject areas in the image to be processed, other than the content subject, based on the display position of the content subject; A subunit is configured to set the pixels of the non-main area to a preset second pixel value; The identification subunit is used to determine the boundary between the display area of ​​the main content and the non-main content area as the edge portion of the main content; The processing subunit is used to set the pixels of the edge portion to a preset first pixel value.

13. The apparatus according to claim 12, wherein the setting subunit is configured to: Perform an image inversion operation on the image to be processed to obtain a first inverted image; Set the pixels of the non-subject region in the first inverted image to a preset second pixel value.

14. The apparatus according to claim 11, wherein the image to be processed includes preset annotation information, the annotation information being used to annotate the display area of ​​the main content; The identification unit includes: The identification subunit is used to determine the display position of the main content in the image to be processed based on the annotation information corresponding to the image to be processed; Alternatively, the identification unit may include: The recognition subunit is used to identify the display position of the main content in the image to be processed according to a preset subject recognition model.

15. The apparatus according to any one of claims 10-14, wherein the first pixel value is a maximum pixel value and the second pixel value is a preset negative pixel value.

16. The apparatus according to any one of claims 10-14, wherein the training module comprises: The input unit is used to input the training data in the training dataset into the diffusion model to be trained, and to obtain the prediction noise information output by the diffusion model to be trained based on the first image and the text description information; The loss value determination unit is used to determine the current loss value of the diffusion model to be trained based on the predicted noise information and the random noise in the second image. The detection unit is used to detect whether the diffusion model to be trained meets the preset convergence condition; The first processing unit is configured to, if not, adjust the parameters of the diffusion model to be trained according to the loss value, and return to the step of inputting the training data in the training dataset into the diffusion model to be trained; The second processing unit is used to determine, if yes, that the diffusion model to be trained has been trained and to obtain the trained target expansion model.

17. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.

19. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-8.

Images