A data processing method and related apparatus

Abstract

The application discloses a data processing method and related device, and relates to the technical field of artificial intelligence. The method comprises the following steps: obtaining a first image generated by a first large model; the first image has target information, the target information reflects the association between at least one image sub-region and a corresponding prompt word, and the corresponding prompt word is used for guiding the first large model to generate the image sub-region; and performing image processing based on at least the target information in response to a target instruction.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a data processing method and related apparatus. Background Technology

[0002] With the development of artificial intelligence technology, text-to-image technology based on large models has been widely used. Users can input text prompts and the large model can generate corresponding images. Summary of the Invention

[0003] In view of the above problems, this application provides a data processing method and related apparatus, the specific solution of which is as follows:

[0004] The first aspect of this application provides a data processing method, including:

[0005] Obtain a first image generated by the first large model; the first image has target information, the target information reflecting the association between at least one image sub-region and a corresponding prompt word, the corresponding prompt word being used to guide the first large model to generate the image sub-region;

[0006] In response to the target instruction, image processing is performed at least based on the target information.

[0007] In one possible implementation, after obtaining the first image generated by the first large model, the process also includes:

[0008] A first input control is displayed at an associated location within the image sub-region; the first input control is rendered based at least on a corresponding prompt word for the image sub-region.

[0009] Obtain a first operation targeting the first input control, and generate the target instruction;

[0010] The image processing based on the target information includes:

[0011] Based on the first operation, an input prompt word is determined to guide the first large model to redraw the image sub-region.

[0012] In one possible implementation, the process of the first large model generating the first image includes:

[0013] Display an intermediate image and a second input control; the intermediate image is an intermediate state image of the first image, and the second input control is displayed at the associated position of the image sub-region on the intermediate image;

[0014] Obtain the operation targeting the second input control and generate intermediate instructions;

[0015] In response to the intermediate instruction, an input prompt word is determined based on the second operation to guide the first large model to generate the image sub-region.

[0016] One possible implementation also includes:

[0017] Obtain user profiles;

[0018] Generate pixel adjustment descriptions based on the user profile and corresponding prompts for the image sub-regions;

[0019] The input control for the image sub-region is rendered based on the pixel adjustment description.

[0020] In one possible implementation, the process of the first large model generating the first image further includes: obtaining the attention weight of each cue word to a first image patch constituting the image sub-region; and determining at least one corresponding cue word for the image sub-region based on the attention weight.

[0021] In one possible implementation, obtaining the attention weight of each cue word for the first image patch constituting the image sub-region includes:

[0022] Obtain the cross-attention weights of the prompt words to the first image patch in each iteration step when the first large model generates the first image, as well as the step weights corresponding to each iteration step;

[0023] Based on the step weights, the cross-attention weights of the prompt words on the first image block in multiple iteration steps are weighted and summed to obtain the attention weights of the prompt words on the first image block.

[0024] One possible implementation also includes:

[0025] Encode the character sequence of the corresponding prompt word in any of the image sub-regions into dot matrix data;

[0026] Based on the dot matrix data, the transparency of the corresponding pixels in the pixel matrix of the image sub-region is modified to embed the corresponding prompt words into the image sub-region in a dot matrix manner.

[0027] In one possible implementation, after obtaining the first image generated by the first large model, in response to a target instruction, image processing is performed at least based on the target information, including:

[0028] Obtain the image prompts and generate the target instructions;

[0029] The raw image prompt words and the first image are input into the second large model. The second large model parses out the corresponding prompt words for each image sub-region in the first image, and generates the second image based on the corresponding prompt words and the raw image prompt words.

[0030] The corresponding prompts are embedded in the associated image sub-regions in a dot matrix manner, or displayed at the associated positions in the associated image sub-regions.

[0031] A second aspect of this application provides a data processing apparatus, comprising:

[0032] An image acquisition unit is used to acquire a first image generated by a first large model; the first image has target information, the target information reflecting the association between at least one image sub-region and a corresponding prompt word, the corresponding prompt word being used to guide the first large model to generate the image sub-region;

[0033] An image processing unit is configured to perform image processing based at least on the target information in response to a target instruction.

[0034] A third aspect of this application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0035] The memory is used to store computer programs;

[0036] The processor is used to execute the computer program so that the electronic device can implement the data processing method of the first aspect or any implementation thereof.

[0037] Using the above technical solutions, this application provides a data processing method and related apparatus to obtain a first image generated by a first large model; the first image has target information, the target information reflecting the association between at least one image sub-region and a corresponding prompt word, the corresponding prompt word being used to guide the first large model to generate the image sub-region; in response to a target instruction, image processing is performed at least based on the target information. Attached Figure Description

[0038] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0039] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0040] Figure 2a A schematic diagram of the first image A1 is shown;

[0041] Figure 2b An example of the display effect of an input control is shown in the diagram.

[0042] Figure 3aA flowchart illustrating another data processing method provided in an embodiment of this application;

[0043] Figure 3b An example of how an intermediate image is displayed is shown;

[0044] Figure 4 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0045] Figure 5 This example illustrates the display effect of yet another type of input control;

[0046] Figure 6 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0047] Figure 7a A flowchart illustrating another data processing method provided in an embodiment of this application;

[0048] Figure 7b An example of a dot matrix data embedding diagram is provided.

[0049] Figure 7c An example of pixel modification comparison diagram is provided;

[0050] Figure 8 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0051] Figure 9 An example is provided to illustrate the effect of a second image;

[0052] Figure 10 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0053] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0054] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0055] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0056] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar elements and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing elements with the same properties in the description of embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of units is not necessarily limited to those units, but may include other units not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0057] Reference Figure 1 , Figure 1 This is a flowchart illustrating a data processing method provided in an embodiment of this application, as shown below. Figure 1 As shown in the embodiment of this application, a data processing method may include steps 101 to 102, which are described in detail below.

[0058] Step 101: Obtain the first image generated by the first large model.

[0059] In this embodiment, the first image has target information, which reflects the association between at least one image sub-region and a corresponding prompt word. The corresponding prompt word is used to guide the first large model to generate the image sub-region.

[0060] In this embodiment, the first major model refers to a generative artificial intelligence model capable of generating images based on text prompts, such as a diffusion model. The first image is the output image generated by the first major model. Optionally, the target information includes structured data recorded during the generation of the first image. The target information establishes a mapping relationship between image sub-regions and the prompts (i.e., corresponding prompts) used to generate those sub-regions. Here, the prompts refer to the tokens obtained after word segmentation of the user's original raw image request text.

[0061] In this embodiment, after the target information is formed, it is stored or transmitted along with the first image. It reflects the actual impact of each prompt word on the image sub-region during the process of generating the image by the first large model.

[0062] Step 102: In response to the target instruction, perform image processing based at least on the target information.

[0063] In this embodiment, the target instruction includes operation instructions for processing the first image, such as operation instructions generated by user clicks, drags, or voice input.

[0064] In this embodiment, when responding to a target instruction, at least one corresponding prompt word is identified based on the target information, indicating the basis for the generation of the image or image sub-region to be processed. For example, image processing includes any form of modification or regeneration operation on the first image or image sub-region. When image processing is performed based on target information, the corresponding prompt word corresponding to the processing object (first image or image sub-region) is determined based on the target information, thereby determining the prompt word used to generate the processing object, and image processing is performed accordingly.

[0065] As can be seen from the above technical solutions, the data processing method provided in this application provides a method for obtaining a first image generated by a first large model. This first image has target information reflecting the association between image sub-regions and corresponding prompt words. In response to a target instruction, image processing is performed based on this target information. Since the corresponding prompt words are used to guide the first large model to generate image sub-regions, this application performs image processing by reflecting the association between image sub-regions and corresponding prompt words, enabling image processing to be based on real semantic attribution, which significantly improves the accuracy and efficiency of image processing.

[0066] The image requirement was: "An Eiffel Tower under a night sky, surrounded by thousands of fluorescent balloons, its tip piercing through the clouds and pointing towards the stars. Moonlight reflects off the balloons, creating halos, and scattered points of light float against a dark blue background." The first large-scale model analyzed this requirement and obtained several prompts, including "night sky," "Eiffel Tower," "fluorescent balloons," and "scattered points of light." Based on these prompts, the first large-scale model generated the first image, A1.

[0067] Figure 2a A schematic diagram of the first image A1 is provided, which includes the sky area corresponding to "night sky", the iron tower area corresponding to "iron tower", the balloon area corresponding to "fluorescent balloon" and the light spot area corresponding to "scattered light spots".

[0068] In existing technologies, if image processing is required on the first image, the generated image is analyzed to identify the text description of the generated image, i.e., the prompt words are deduced from the image. However, secondary analysis of the generated result cannot restore the true impact of each prompt word on each sub-region of the image. Furthermore, the deduced prompt words are general descriptions of the image content and are difficult to accurately correspond to the relationship between the prompt words and specific sub-regions in the image. Based on the data processing method provided in this application embodiment, the first image has target information, which reflects the association between at least one sub-region of the image and the corresponding prompt word. Specifically, the target information can reflect the association between prompt words such as "night sky," "Eiffel Tower," "fluorescent balloon," and "scattered light spots" and the image sub-regions that guide the first large model to generate. Taking "Eiffel Tower" as an example, the target information can reflect that the prompt word "Eiffel Tower" corresponds to the Eiffel Tower region generated in the first image guided by "Eiffel Tower"; taking "fluorescent balloon" as an example, the target information can reflect that the prompt word "fluorescent balloon" corresponds to the balloon region generated in the first image guided by "fluorescent balloon."

[0069] Therefore, this solution, when responding to target instructions, can perform image processing based on the target information. Whether it's local redrawing of a sub-region of the image or global generation of the entire first image, it can accurately determine the original prompt word corresponding to each sub-region, thus achieving semantic-driven image processing. It is evident that, compared to existing technologies that deduce prompt words from images, this solution records the true correspondence between prompt words and sub-regions of the image during the generation process using target information, providing precise semantic guidance for image processing and improving its accuracy and efficiency.

[0070] In one optional embodiment, the data processing method provided in this application, after obtaining the first image generated by the first large model, further includes: displaying a first input control at an associated position in a sub-region of the image, obtaining a first operation on the first input control, and generating a target instruction.

[0071] In this embodiment, the first input control includes a visual interactive element for receiving user input, such as a text box, button, or floating menu. The associated location includes a location spatially adjacent or visually associated with the image sub-region, such as a floating point above the image sub-region and an anchor point at the edge of the image sub-region.

[0072] In this embodiment, the first input control is rendered based at least on the corresponding prompt words of the image sub-region. Specifically, the display style or content of the first input control is dynamically generated based on the text content of the corresponding prompt words. For example, after the user obtains the first image, a visual control with a first style is generated at the associated position of each image sub-region based on the corresponding prompt words reflected in the target information.

[0073] by Figure 2a Taking the first image A1 as an example, this first image is generated based on the raw image requirement "a tower under the night sky, with thousands of fluorescent balloons surrounding the tower". Based on the relationship between the prompt word "fluorescent balloon" in the target information and the balloon area in the first image, a text box control is rendered in the first position on the right side of the balloon area. This text box control is used to visualize the relationship between the prompt word "fluorescent balloon" and the balloon area. Figure 2b An example diagram of an input control is shown, such as... Figure 2b As shown, after clicking the control, the user can see the corresponding prompt word for the balloon area, which is "fluorescent balloon". The user can also directly enter the redrawing requirements for the corresponding prompt word in the text box, such as "increase transparency" or "enhance halo".

[0074] In this embodiment, the input operation is the first operation, and the target instruction is generated based on the first operation.

[0075] Based on this, step 102, namely a specific implementation method of image processing based on target information, includes: determining input prompt words based on the first operation to guide the first large model to redraw the image sub-region.

[0076] In this embodiment, the input prompt includes a text description used to guide the first large model to regenerate the image sub-region. Redrawing means regenerating only the image sub-region without changing the content of other regions in the first image.

[0077] Optionally, the input prompt can be directly entered by the user. That is, the new prompt entered by the user in the first input control is obtained, and the new prompt is fused or replaced with the corresponding prompt in the target image sub-region to obtain the input prompt. The input prompt is then input into the first large model, which can guide the first large model to iteratively generate only the target image sub-region while keeping other regions unchanged, thus obtaining a locally modified image.

[0078] In summary, the data processing method provided in this application can be applied to scenarios involving partial redrawing. In the prior art, after a user obtains a generated image, if they wish to modify a local area of ​​the image, they typically need to re-enter a new, complex prompt to describe the overall image, or use external tools to edit the image. Users cannot intuitively understand which prompt affects which part of the image, and there is a lack of traceable correlation between the prompt and the image area. When dissatisfied with the generated result, the user can only retry the entire prompt. This application, by displaying a visual control, allows users to directly click on the area in the image they want to modify, automatically locating the corresponding prompt for that area. After the user enters their new requirements, precise partial redrawing can be achieved.

[0079] As can be seen from the above technical solutions, the data processing method provided in this application embodiment displays a first input control rendered based on corresponding prompt words at the associated position of an image sub-region, and determines the input prompt words based on the user's operation on the first input control to guide redrawing, thereby realizing the user's local redrawing of the image and improving the intuitiveness and accuracy of local redrawing.

[0080] In an optional embodiment, the data processing method provided in this application further includes an image optimization step during the generation of the first image. Figure 3a A flowchart illustrating another data processing method provided in this application embodiment is shown below. Figure 3a As shown, the process of the first large model generating the first image includes:

[0081] Step 301: Display the intermediate image and the second input control.

[0082] In this embodiment, the intermediate image is an intermediate state image of the first image. Specifically, the intermediate image includes a stage output image of the first large model that has not yet reached the final state during the iterative generation process. Optionally, the generation process of the first image usually includes multiple iteration steps, each iteration step generating a potential representation of an intermediate state, which can be decoded to obtain the intermediate image.

[0083] In this embodiment, the second input control is displayed at an associated position in a sub-region of the image on the intermediate image. It includes a visual interactive element for receiving user input. Unlike the first input control, the second input control is displayed during the generation of the first image rather than after the generation is complete.

[0084] Figure 3b An example of how an intermediate image is displayed is shown, such as... Figure 3b As shown, the first large model is used to generate Figure 2a Taking the process of generating the first image A1 as an example, during the iterative generation of the first image A1, the intermediate images have not yet presented the final detailed effects of each image sub-region. For example, the intermediate images lack the texture details of the Eiffel Tower and the fluorescent balloons, which appear as blurry pixels. At this time, when displaying the intermediate image, based on the association between the corresponding prompt words and the image sub-regions, a second input control can be displayed at the associated position of the image sub-region, such as... Figure 3b As shown, based on the relationship between the prompt word "Eiffel Tower" and the Eiffel Tower region in the intermediate image, a text box control is rendered at the first position to the right of the Eiffel Tower region in the intermediate image. This text box control is used to visualize the relationship between the prompt word "Eiffel Tower" and the Eiffel Tower region. Users can input prompt words such as "enhance the halo on the surface of the balloon" through this second input control to correct the prompt words in real time, so as to guide the first model to optimize the generation effect of the Eiffel Tower region in subsequent iterations.

[0085] Step 302: Obtain the operation for the second input control and generate intermediate instructions.

[0086] In this embodiment, after displaying the intermediate image and the second input control, if the user observes the generated effect of the intermediate image and finds that the generated effect of the image sub-region does not match expectations, they can input correction suggestions through the second input control. For example, if there are too many "fluorescent balloons" in the intermediate image of the first image, the user can click the second input control associated with the fluorescent balloon area and input the correction suggestion of "reducing the number of balloons". After obtaining the operation of the second input control in this step, an intermediate instruction is generated to instruct the generation of fluorescent balloon areas.

[0087] Step 303: In response to the intermediate instruction, determine the input prompt word based on the second operation to guide the first large model to generate image sub-regions.

[0088] In this embodiment, an input prompt word is determined according to the second operation, and the input prompt word is injected into the subsequent iteration steps of the first large model to guide the first large model to generate image sub-regions.

[0089] Specifically, in subsequent iterations, the first model will fuse the semantic information of the input prompt with the latent representation of the current iteration step to adjust the generation direction of the image sub-region.

[0090] As can be seen from the above technical solutions, the data processing method provided in this application embodiment displays an intermediate image and a second input control during the generation of the first image, so that the user can adjust the generation direction of the image sub-region in real time during the generation process, thereby reducing the number of iterations and improving generation efficiency and user experience.

[0091] Based on the above embodiments, the correspondence between image sub-regions and corresponding prompts can be explicitly represented by the display control, thereby explicitly influencing the corresponding prompts of the image sub-regions to be redrawn. Furthermore, the display methods of the control include various methods, such as directly displaying the corresponding prompts, or displaying the corresponding prompts and their optimization directions. Based on this, the data processing method provided in this application also includes a method for rendering input controls. Figure 4 A flowchart illustrating another data processing method provided in this application embodiment is shown below. Figure 4 As shown, a specific implementation method of the input control for rendering the sub-region of the output image includes:

[0092] Step 401: Obtain user profiles.

[0093] In this embodiment, the user profile includes a set of feature data representing the user's personalized preferences, which may include the user's style preferences, element quantity preferences, color preferences, compositional focus, detail preferences, and thematic associations. The user profile can be stored on a local device and updated based on the user's historical operations.

[0094] It should be noted that user profiles can be stored on local devices and updated based on the user's historical actions.

[0095] Step 402: Generate pixel adjustment descriptions based on user profiles and corresponding prompts for image sub-regions.

[0096] In this embodiment, the pixel adjustment description includes descriptive information used to guide the generation of optimization options, i.e., optimization direction description. Optionally, the corresponding prompt words of the image sub-region are fused with the preference features in the user profile to generate a personalized optimization direction description. The optimization direction description may include enhancement words, replacement words, style supplementary semantics, and attribute adjustment semantics, depending on the type of the corresponding prompt words.

[0097] For example, for an image sub-region with "fluorescent balloon" as the corresponding prompt word, optimization direction descriptions in multiple dimensions can be generated based on different preference features in the user profile, as follows:

[0098] Material optimization aspects: Increase balloon transparency, enhance reflective highlights, and soften the halo;

[0099] Quantity and distribution dimensions: Increase the density of balloons near the base of the tower to make the balloons more evenly distributed;

[0100] Enhanced sense of motion: Added upward drift trajectory and increased motion blur to enhance the sense of motion;

[0101] Color restoration aspects: Use cooler blue tones and enhance the color gradient from the center to the edges;

[0102] Additional style dimension: make the balloons more surreal, add dreamy glitter particles around the balloons.

[0103] For example, for an image sub-region with "Eiffel Tower" as the corresponding prompt word, based on different preference features in the user profile, multiple dimensions of optimization direction descriptions can be generated, as follows:

[0104] Structural and detail enhancements: Add metallic texture details to the tower, enhance the sharpness of the lattice structure, add more visible rivets and architectural patterns, and increase contrast to highlight the steel frame;

[0105] Lighting and Reflection Optimization Dimensions: Enhance warm light reflection on the tower body, add edge lighting along the tower edges, increase highlights to achieve a more lustrous metallic appearance, and soften shadow transitions on the tower columns;

[0106] Composition and proportion adjustment dimensions: slightly increase the height of the tower in the composition, slightly shift the tower to the right for better balance, reduce foreground obstruction around the base of the tower, and increase the visual focus of the top of the tower;

[0107] Enhanced dimensions of color and atmosphere: Use warmer gold tones to create a Parisian night atmosphere, apply a subtle gradient from warm gold to cool blue, add atmospheric haze around the top of the tower, and add soft volumetric beams radiating upwards.

[0108] Style enhancements and artistic treatments: Add a more romantic impressionistic halo, apply subtle Art Nouveau style metallic scroll patterns to the edges, add surreal shimmering particles around the tower, and add subtle futuristic neon outlines to present a science fiction style.

[0109] Dynamic and scene enhancements: Add subtle fog movement around the base of the tower, introduce upward-drifting sparks near the top of the tower, increase the intensity of starlight reflection on the metal, and add subtle wind-driven lines to nearby elements.

[0110] Step 403: Adjust the input controls of the sub-regions of the rendered output image based on pixel settings.

[0111] In this embodiment, the content of the input control is rendered in a personalized manner based on the generated pixel adjustment description. Specifically, the pixel adjustment description is used as optional content of the input control, and according to a pre-configured rendering strategy, multiple optimization directions are presented as visual controls at associated positions in the image sub-region. Each optimization direction corresponds to an interactive control element, which the user can trigger by clicking or hovering. The rendering methods include: displaying multiple optimization directions as a drop-down menu, displaying them as a group of buttons, or displaying them in a categorized manner as a floating panel. Furthermore, the optimal rendering style and layout can be dynamically selected based on the size of the image sub-region, the available space at the associated position, and the user's historical interaction habits.

[0112] Figure 5 This example illustrates the display effect of an input control, such as... Figure 5 As shown, when a user clicks on the first sub-region, an input control is rendered at the associated location of that region. The input control directly displays the corresponding prompt word "night sky" for the current image sub-region and presents multiple replacement word options in a drop-down menu, including: evening sky, interstellar space, thunderclouds, and blazing sun. The user can select the desired replacement word through the drop-down menu to replace the corresponding prompt word for that image sub-region with the selected item, thereby guiding the first main model to redraw the first sub-region.

[0113] When the user clicks on the second sub-region, an input control is rendered at the associated location of that region. This control displays the corresponding prompt word "sparse light spots" for the current image sub-region and presents multiple optimization direction options in a drop-down menu, including: brighter, darker, increase density, and sparser. The user can select the desired optimization direction through the drop-down menu without manually entering complex prompt words. The corresponding input prompt word is generated based on the selected option, guiding the first main model to make local adjustments to the second sub-region.

[0114] As can be seen from the above technical solution, the data processing method provided in this application obtains a user profile, generates pixel adjustment descriptions based on the user profile and corresponding prompt words, and renders input controls based on these descriptions, thereby achieving personalized intelligent generation of input controls. Users do not need to have professional prompt word writing experience; they only need to operate the input controls and select from the automatically generated optimization directions to obtain generation results that match their preferences, thus improving the user experience.

[0115] In an optional embodiment, the data processing method provided in this application further includes the step of determining at least one corresponding cue word for an image sub-region. Figure 6 This is a flowchart illustrating another data processing method provided in an embodiment of this application. Figure 6 The specific implementation process of determining at least one corresponding prompt word in a sub-region of an image during the generation of the first image by the first large model is illustrated, such as... Figure 6 As shown, this method specifically includes:

[0116] Step 601: Obtain the attention weight of each cue word for the first image patch that constitutes the image sub-region.

[0117] In this embodiment, an image patch refers to a basic unit obtained by spatially dividing the first image, and one image sub-region corresponds to at least one image patch. The attention weight represents the influence value of each prompt word on each image patch during the image generation process of the first model, and this value reflects the degree of control that the prompt word has over the generation of the image patch.

[0118] In this embodiment, attention weights can be extracted from the cross-attention layer of the first large model during the generation of the first image.

[0119] Step 602: Based on attention weights, determine at least one corresponding cue word for the image sub-region.

[0120] In this embodiment, for each first image block constituting an image sub-region, the attention weight of the prompt word to each first image block is obtained, thereby determining the comprehensive attention weight of the prompt word to the image sub-region. Furthermore, at least one corresponding prompt word for the image sub-region is determined based on the magnitude of the attention weight.

[0121] Optionally, the attention weights of the prompt words for each first image block are summed or averaged to obtain the comprehensive attention weight of the prompt words for the image sub-region. Then, at least one prompt word with the highest comprehensive attention weight is determined as the corresponding prompt word for that image sub-region. For example, after the attention weight calculation, the prompt word "fluorescent balloon" has the highest average attention weight for the multiple image blocks constituting the balloon region, so "fluorescent balloon" is determined as the corresponding prompt word for the balloon region.

[0122] As can be seen from the above technical solutions, the data processing method provided in this application obtains the attention weight of each prompt word to the image block during the generation of the first image by the first large model, and determines the corresponding prompt word of the image sub-region based on the weight, thereby realizing the accurate quantification of the attribution relationship between the prompt word and the image sub-region and improving the accuracy of the semantic attribution relationship.

[0123] In an optional embodiment, step 601, namely obtaining the attention weight of each prompt word on the first image patch constituting the image sub-region, specifically includes: obtaining the cross-attention weight of the prompt word on the first image patch in each iteration step when the first large model generates the first image, and the step weight corresponding to each iteration step. Based on the step weight, the cross-attention weights of the prompt word on the first image patch in multiple iteration steps are weighted and summed to obtain the attention weight of the prompt word on the first image patch.

[0124] In this embodiment, an iteration step refers to the denoising steps performed by the first large model during image generation, with each iteration step corresponding to a time step. The cross-attention weight refers to the attention score between the cue word and the image patch in that iteration step, reflecting the degree of influence of the cue word on the image patch at the current noise level. The step weight refers to the weight coefficient assigned to each iteration step, used to characterize the importance of that iteration step in the overall image generation process.

[0125] In this embodiment, the step weight can be set according to the order of the iteration steps. For example, later iteration steps have a greater impact on image details and can be given a higher step weight.

[0126] As can be seen from the above technical solutions, the data processing method provided in this application embodiment obtains the cross-attention weight and step weight of each iteration step, performs weighted summation and normalization to obtain the final attention weight, which can effectively suppress single-step abnormal noise, retain complete generation path information, and make the attribution relationship between prompt words and image sub-regions more accurate.

[0127] In an optional embodiment, the data processing method provided in this application further includes the step of embedding the corresponding prompt words into an image sub-region in a dot matrix manner. Figure 7a A flowchart illustrating another data processing method provided in this application embodiment is shown below. Figure 7a As shown, a specific implementation method for embedding corresponding prompt words into image sub-regions in a dot matrix manner includes:

[0128] Step 701: Encode the character sequence of the corresponding prompt word in any image sub-region into dot matrix data.

[0129] In this embodiment, the character sequence is a string representation of the corresponding prompt word. Specifically, for any image sub-region, the character sequence of the corresponding prompt word is parsed and encoded into dot matrix data. The dot matrix data includes a dot matrix matrix that converts the character sequence into binary or a specific encoding format. Taking a dot matrix matrix in binary encoding format as an example, each dot corresponds to a binary bit (0 or 1). For example, there are 52 uppercase and lowercase English letters, which can be represented by 6 bits, such as a=000001, A=011011, Z=110100.

[0130] Step 702: Based on the dot matrix data, modify the transparency of the corresponding pixels in the pixel matrix of the image sub-region to embed the corresponding prompt words into the image sub-region in a dot matrix manner.

[0131] In this embodiment, the image is represented using the RGBA color space, where the A channel (Alpha channel) represents the pixel's transparency. The pixel matrix of a sub-region of the image is obtained, and the Alpha channel value of the corresponding pixel is modified according to each binary bit in the dot matrix data. Optionally, if the binary bit is 1, the Alpha value of the pixel at that location is reduced by a preset small amount. For example, from completely opaque 255 to 254. If the binary bit is 0, the original Alpha value remains unchanged. For example, Figure 7bAn example of dot matrix data embedding is provided, where every 6 pixels are used to embed one character. Taking the first six pixels of the first row as an example, their alpha values ​​are 255, 254, 255, 254, 255, and 254, respectively. Here, an alpha value of 255 represents a binary 0, and an alpha value of 254 represents a binary 1. Therefore, the binary sequence corresponding to these six pixels is "010101", which decodes to the letter "A". Following this method, the system embeds the entire character sequence character by character into the pixel matrix of the image sub-region, achieving implicit embedding of the corresponding prompt word.

[0132] Understandably, implicit embedding of corresponding cue words can be achieved through minute changes in the alpha value, for example, Figure 7c An example of a pixel modification comparison diagram is shown, such as... Figure 7c As shown, the left side shows the original pixel color value (0, 255, 0, 255), representing a completely opaque green, while the right side shows the modified pixel color value (0, 255, 0, 254). The alpha value is reduced by only 1 point, so there is no obvious visual difference. However, the embedded bit data can be extracted by parsing the alpha channel.

[0133] As can be seen from the above technical solutions, the data processing method provided in this application embodiment achieves implicit embedding of prompt word information by encoding the corresponding prompt words into dot matrix data and modifying the transparency of pixels in sub-regions of the image, so that the image file itself can carry the semantic association reflected by the target information.

[0134] Furthermore, since the reduction in alpha value is minuscule, the human eye cannot perceive the modification. However, by reading the alpha values ​​of these pixels using a parsing algorithm, the embedded prompt can be reconstructed. It should be noted that the minuscule reduction can be configured according to the display device's parameters. Based on this, the corresponding prompt can be losslessly and implicitly embedded in the image, achieving cross-model semantic information transfer without relying on external metadata. Therefore, the data processing method provided in this application embodiment can also be applied to cross-model image redrawing scenarios.

[0135] It should be noted that, in one optional embodiment, the operation of embedding the corresponding prompt words into the image sub-region in a dot matrix manner can be completed synchronously during the generation of the first image by the first large model. Specifically, when generating the pixel matrix of the image sub-region in each iteration step, dot matrix data can be generated in real time according to the corresponding prompt words corresponding to the current image sub-region, and the Alpha channel value of the corresponding pixels can be modified. This allows the generated first image to carry semantic embedding information. By integrating the embedding process with the generation process, the integrity and consistency of the target information can be guaranteed.

[0136] Alternatively, in another optional embodiment, after the first image is generated, the correspondence between each image sub-region and the corresponding prompt word is determined based on the target information, thereby embedding the dot matrix data.

[0137] further, Figure 8 A flowchart illustrating yet another data processing method provided in this application embodiment. Figure 8 This illustrates a specific implementation method for image processing, based at least on target information, in response to target instructions after obtaining the first image generated by the first large model, such as... Figure 8 As shown, this method specifically includes:

[0138] Step 801: Obtain the raw image prompt words and generate the target instruction.

[0139] In this embodiment, the image generation prompt includes a text description of the desired new image, obtained based on the new image generation requirement. This new image generation requirement can be obtained by receiving an image generation operation for the first image; for example, the image generation operation includes the user directly inputting the image generation requirement and triggering the image generation control.

[0140] Step 802: Input the raw image prompt words and the first image into the second large model. The second large model parses out the corresponding prompt words for each image sub-region in the first image, and generates the second image based on the corresponding prompt words and the raw image prompt words.

[0141] In this embodiment, the corresponding prompt words are embedded in the associated image sub-region in a dot matrix manner, or displayed at the associated position in the associated image sub-region.

[0142] In this embodiment, the second major model includes another image generation model different from the first major model. The first and second major models can be models with the same architecture but different versions, or models with different architectures. In this step, after providing the raw image prompts and the first image as input to the second major model, the second major model first parses the corresponding prompts for each image sub-region from the first image. Then, it fuses the parsed prompts with the raw image prompts to guide the second major model in generating the second image.

[0143] It should be noted that if the corresponding prompt is embedded in an image sub-region in a dot matrix format, the second large model reads the alpha channel data of the pixels in the image sub-region, parses the dot matrix information, restores it to a character sequence, and further obtains the corresponding prompt from the character sequence. If the corresponding prompt is displayed in an input control at an associated location, the second large model obtains the corresponding prompt by reading the explicit text.

[0144] As can be seen from the above technical solutions, the data processing method provided in this application embodiment obtains the raw image prompt words, inputs the first image and its embedded or displayed corresponding prompt words into the second large model, so that the second large model can parse the semantic information of the original image and integrate it into the generation of the new image, thereby achieving semantic fidelity in cross-model scenarios and effectively solving the problem of semantic loss caused by potential spatial differences between different models.

[0145] Based on this, the data processing method provided in this application embodiment can also be specifically applied to the scenario of regenerating an image based on the first image using a second major model. In the prior art, when generating an image across models, due to the potential spatial differences between different models, relying solely on pixel information may lead to the loss of key semantics.

[0146] Taking the first image B1 generated by the first model based on "a cat wearing a mechanical claw standing on the roof of a skyscraper, looking at the spaceships passing by in the sky" as an example, the first image B1 is input into the second model to generate an image. Since the second model cannot know the semantic information of "mechanical claw" in the original image, it may identify it as an ordinary cat claw, resulting in the loss of key features in the generated second image. Figure 9 An example is provided to illustrate the effect of a second image, such as... Figure 9 As shown, the first image B1 and the new prompt "Adjust the proportion of the cat to no more than one-third of the whole image" are input into the second large model. The second large model generates a second image B1-1. The key feature "mechanical claw" is lost in the second image B1-1.

[0147] In response, this application embeds or explicitly displays the corresponding prompts in the image, enabling the second model to accurately parse the semantic information of each sub-region in the first image B1. When generating the second image, the features influenced by these corresponding prompts are preserved or enhanced, thereby generating the second image B1-2. It can be seen that B1-2 retains the influence of the original prompts on the image generation, effectively avoiding the semantic loss problem in cross-model scenarios.

[0148] The above describes a data processing method provided by an embodiment of this application. The following will describe the related apparatus for performing the above data processing method.

[0149] Please see Figure 10 , Figure 10 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. Figure 10 As shown, the data processing device 1000 includes:

[0150] Image acquisition unit 1001 is used to acquire a first image generated by a first large model; the first image has target information, the target information reflects the association between at least one image sub-region and a corresponding prompt word, the corresponding prompt word is used to guide the first large model to generate the image sub-region;

[0151] Image processing unit 1002 is configured to perform image processing based at least on the target information in response to a target instruction.

[0152] In one possible implementation, the data processing apparatus further includes: a control response unit, configured to: after obtaining a first image generated by a first large model, display a first input control at an associated location in the image sub-region; the first input control is rendered based at least on a corresponding prompt word in the image sub-region; obtain a first operation for the first input control, and generate the target instruction;

[0153] When the image processing unit performs image processing based on the target information, it is specifically used to: determine an input prompt word based on the first operation to guide the first large model to redraw the image sub-region.

[0154] In one possible implementation, during the generation of the first image, the first large model is used to display an intermediate image and a second input control; the intermediate image is an intermediate state image of the first image, and the second input control is displayed at the associated position of the image sub-region on the intermediate image; an operation for the second input control is obtained, and an intermediate instruction is generated; in response to the intermediate instruction, an input prompt word is determined based on the second operation to guide the first large model to generate the image sub-region.

[0155] In one possible implementation, the data processing apparatus further includes: a control rendering unit, configured to obtain a user profile; generate a pixel adjustment description based on the user profile and corresponding prompts for the image sub-region; and render and output an input control for the image sub-region based on the pixel adjustment description.

[0156] In one possible implementation, the first large model, during the generation of the first image, is further configured to obtain the attention weight of each cue word to a first image patch constituting the image sub-region; and based on the attention weight, determine at least one corresponding cue word for the image sub-region.

[0157] In one possible implementation, the first large model, when used to obtain the attention weights of each cue word for the first image patch constituting the image sub-region, is specifically used for:

[0158] Obtain the cross-attention weights of the prompt words to the first image patch in each iteration step when the first large model generates the first image, as well as the step weights corresponding to each iteration step;

[0159] Based on the step weights, the cross-attention weights of the prompt words on the first image block in multiple iteration steps are weighted and summed to obtain the attention weights of the prompt words on the first image block.

[0160] In one possible implementation, the data processing apparatus further includes: a data embedding unit, configured to encode the character sequence of a corresponding prompt word in any of the image sub-regions into dot matrix data; and based on the dot matrix data, modify the transparency of corresponding pixels in the pixel matrix of the image sub-region to embed the corresponding prompt word into the image sub-region in a dot matrix manner.

[0161] In one possible implementation, after obtaining the first image generated by the first large model, the image processing unit, in response to a target instruction, performs image processing at least based on the target information, specifically for:

[0162] Obtain the raw image prompt words and generate the target instruction; input the raw image prompt words and the first image to the second large model, the second large model parses out the corresponding prompt words of each image sub-region in the first image, and generates the second image based on the corresponding prompt words and the raw image prompt words; wherein, the corresponding prompt words are embedded in the associated image sub-region in a dot matrix manner, or displayed at the associated position of the associated image sub-region.

[0163] This application embodiment also provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0164] The memory is used to store computer programs;

[0165] The processor is used to execute the computer program to enable the electronic device to perform:

[0166] Obtain a first image generated by the first large model; the first image has target information, the target information reflecting the association between at least one image sub-region and a corresponding prompt word, the corresponding prompt word being used to guide the first large model to generate the image sub-region;

[0167] In response to the target instruction, image processing is performed at least based on the target information.

[0168] refer to Figure 11 As shown, Figure 11This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device in this embodiment may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 11 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0169] like Figure 11 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 1101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1102 or a program loaded from a storage device 1108 into a random access memory (RAM) 1103. When the electronic device is powered on, the RAM 1103 also stores various programs and data required for the operation of the electronic device. The processing unit 1101, ROM 1102, and RAM 1103 are interconnected via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0170] Typically, the following devices can be connected to I / O interface 1105: input devices 1106 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 1107 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1108 including, for example, memory cards, hard drives, etc.; and communication devices 1109. Communication device 1109 allows electronic devices to exchange data via wireless or wired communication with other devices. Although Figure 11 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0171] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the data processing methods provided in this application.

[0172] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the data processing methods provided in this application.

[0173] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0174] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0175] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0176] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A data processing method, comprising: Obtain the first image generated by the first large model; The first image has target information, which reflects the association between at least one image sub-region and a corresponding prompt word, and the corresponding prompt word is used to guide the first large model to generate the image sub-region; In response to the target instruction, image processing is performed at least based on the target information.

2. The data processing method according to claim 1, after obtaining the first image generated by the first large model, further includes: A first input control is displayed at the associated location of the image sub-region; The first input control is rendered based at least on the corresponding prompt words of the image sub-region; Obtain a first operation targeting the first input control, and generate the target instruction; The image processing based on the target information includes: Based on the first operation, an input prompt word is determined to guide the first large model to redraw the image sub-region.

3. The data processing method according to claim 1, wherein the process of the first large model generating the first image includes: Display an intermediate image and a second input control; the intermediate image is an intermediate state image of the first image, and the second input control is displayed at the associated position of the image sub-region on the intermediate image; Obtain the operation targeting the second input control and generate intermediate instructions; In response to the intermediate instruction, an input prompt word is determined based on the second operation to guide the first large model to generate the image sub-region.

4. The data processing method according to claim 2 or 3 further includes: Obtain user profiles; Generate pixel adjustment descriptions based on the user profile and corresponding prompts for the image sub-regions; The input control for the image sub-region is rendered based on the pixel adjustment description.

5. The data processing method according to claim 1, wherein the process of the first large model generating the first image further includes: Obtain the attention weight of each cue word for the first image patch constituting the image sub-region; Based on attention weights, at least one corresponding cue word is determined for the image sub-region.

6. The data processing method according to claim 5, obtaining the attention weight of each cue word to the first image block constituting the image sub-region, includes: Obtain the cross-attention weights of the prompt words to the first image patch in each iteration step when the first large model generates the first image, as well as the step weights corresponding to each iteration step; Based on the step weights, the cross-attention weights of the prompt words on the first image block in multiple iteration steps are weighted and summed to obtain the attention weights of the prompt words on the first image block.

7. The data processing method according to claim 6 further includes: Encode the character sequence of the corresponding prompt word in any of the image sub-regions into dot matrix data; Based on the dot matrix data, the transparency of the corresponding pixels in the pixel matrix of the image sub-region is modified to embed the corresponding prompt words into the image sub-region in a dot matrix manner.

8. The data processing method according to claim 1 or 7, in response to a target instruction, performing image processing at least based on the target information, comprising: Obtain the image prompts and generate the target instructions; The raw image prompt words and the first image are input into the second large model. The second large model parses out the corresponding prompt words for each image sub-region in the first image, and generates the second image based on the corresponding prompt words and the raw image prompt words. The corresponding prompts are embedded in the associated image sub-regions in a dot matrix manner, or displayed at the associated positions in the associated image sub-regions.

9. A data processing apparatus, comprising: The image acquisition unit is used to acquire the first image generated by the first large model; The first image has target information, which reflects the association between at least one image sub-region and a corresponding prompt word, and the corresponding prompt word is used to guide the first large model to generate the image sub-region; An image processing unit is configured to perform image processing based at least on the target information in response to a target instruction.

10. An electronic device comprising at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to perform: Obtain a first image generated by the first large model; the first image has target information, the target information reflecting the association between at least one image sub-region and a corresponding prompt word, the corresponding prompt word being used to guide the first large model to generate the image sub-region; In response to the target instruction, image processing is performed at least based on the target information.

Images