Layout generation method and device based on large model, electronic equipment and storage medium

Abstract

The invention provides a layout generation method and device based on a large model, electronic equipment and a storage medium, relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, the large model, image processing and the like, and can be applied to scenes such as content generation based on artificial intelligence. According to the specific implementation scheme, visual information of an original image is extracted through a target detection model and a semantic segmentation model; converting the visual information into structured data, and converting the structured data into natural language description by utilizing spatial relationship coding; constructing a multi-modal Prompt based on the natural language description and the user instruction; and inputting the constructed multi-modal Prompt into the large model to obtain layout information of the target object output by the large model in the original image. According to the scheme, the layout generation quality and efficiency can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, particularly to technical fields such as computer vision, deep learning, large models, and image processing. It can be applied to scenarios such as content generation based on artificial intelligence, and specifically relates to a layout generation method, device, electronic device, and storage medium based on a large model. Background Art

[0002] In recent years, with the rapid development of multi-modal perception technology, the development of image layout generation based on large models has been promoted. However, currently, the layout generated by large models has insufficient matching with the semantic content of images, and there are problems with inaccurate distribution of element positions, resulting in relatively low practical application value. Summary of the Invention

[0003] The present disclosure provides a layout generation method, device, electronic device, and storage medium based on a large model.

[0004] According to the first aspect of the present disclosure, a layout generation method based on a large model is provided, including:

[0005] Extracting visual information of the original image through a target detection model and a semantic segmentation model;

[0006] Converting the visual information into structured data, and converting the structured data into a natural language description using spatial relation encoding, where the spatial relation encoding is used to convert the coordinates, dimensions, and relative positions of elements into a natural language description;

[0007] Based on the natural language description and the user instruction, constructing a multi-modal Prompt; wherein, the user instruction includes an action for the original image and the target object corresponding to the action, and the multi-modal Prompt is used to associate the natural language description with the user instruction;

[0008] Inputting the constructed multi-modal Prompt into the large model to obtain layout information of the target object in the original image output by the large model, where the layout information includes the position and dimensions of the target object in the original image.

[0009] According to the second aspect of the present disclosure, a layout generation device based on a large model is provided, including:

[0010] An extraction module, configured to extract visual information of the original image through a target detection model and a semantic segmentation model;

[0011] A conversion module, configured to convert the visual information into structured data, and convert the structured data into a natural language description using spatial relation encoding, where the spatial relation encoding is used to convert the coordinates, dimensions, and relative positions of elements into a natural language description;

[0012] A construction module for constructing a multi-modal Prompt based on a natural language description and a user instruction; wherein, the user instruction includes an action for an original image and a target object corresponding to the action, and the multi-modal Prompt is used to associate the natural language description with the user instruction.

[0013] A generation module for inputting the constructed multi-modal Prompt into a large model to obtain layout information of the target object in the original image output by the large model, where the layout information includes the position and size of the target object in the original image.

[0014] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0015] At least one processor; and

[0016] A memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0018] According to a fourth aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute any method in the embodiments of the present disclosure.

[0019] According to a fifth aspect of the present disclosure, there is provided a computer program product including a computer program, and the computer program implements any method in the embodiments of the present disclosure when executed by a processor.

[0020] Adopting the solution of the present disclosure can improve the quality and efficiency of layout generation.

[0021] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. Description of the Drawings

[0022] The drawings are used to better understand the solution and do not constitute a limitation to the present disclosure. Among them:

[0023] Figure 1 is a flowchart of a method for layout generation based on a large model according to an embodiment of the present disclosure;

[0024] Figure 2 is a schematic diagram of generating a multi-modal Prompt based on a predefined template according to an embodiment of the present disclosure;

[0025] Figure 3It is a schematic diagram of an architecture for generating multi-modal Prompts according to an embodiment of the present disclosure;

[0026] Figure 4 It is a schematic diagram of the structure of a layout generation device based on a large model according to an embodiment of the present disclosure;

[0027] Figure 5 It is a schematic diagram of a scenario of a layout generation method based on a large model according to an embodiment of the present disclosure;

[0028] Figure 6 It is a structural diagram of an electronic device for implementing the layout generation method based on a large model according to an embodiment of the present disclosure. Detailed implementation manners

[0029] The following describes exemplary embodiments of the present disclosure with reference to the accompanying drawings. Various details of the embodiments of the present disclosure are included to facilitate understanding, and they should be considered merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0030] The term "and / or" in this document merely describes the association relationship of associated objects and indicates that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. The term "at least one" in this document means any one of a plurality or any combination of at least two of a plurality. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set composed of A, B, and C. The terms "first" and "second" in this document represent referring to multiple similar technical terms and distinguishing them, and do not mean limiting the order or limiting to only two. For example, the first feature and the second feature refer to two categories / two features. The first feature can be one or more, and the second feature can also be one or more.

[0031] In addition, for better illustration of the present disclosure, numerous specific details are given in the following detailed implementation manners. Those skilled in the art should understand that the present disclosure can also be implemented without some specific details. In some instances, methods, means, elements, and circuits well-known to those skilled in the art are not described in detail to highlight the gist of the present disclosure.

[0032] Before introducing the technical solutions of the embodiments of the present disclosure, further explanations are made on the technical terms that may be used in the present disclosure:

[0033] Multi-modal Prompt: An instruction input form that integrates multiple data modalities (such as text, images, voice, video, etc.), used to guide large models to generate cross-modal responses, such as generating layouts.

[0034] In the related art, with the rapid development of Artificial Intelligence (AI) technology, image understanding and generation tasks have been widely applied in multiple fields. However, existing methods often face the following challenges when dealing with complex scenarios: When locating the position and layout information of specific target objects in an image, there may be problems such as insufficient accuracy or low generation efficiency. The current large model layout generation methods only combine visual image information and text instructions to generate layouts, resulting in the generated layouts not conforming to the picture semantics, and the target objects being randomly placed at any position in the image, making the image have a sense of incongruity and unable to generate practical value.

[0035] To at least partially solve one or more of the above problems and other potential problems, the present disclosure proposes a layout generation method based on a large model, which effectively combines visual information, user instructions, and the large model, and optimizes the intelligent positioning of visual elements according to the user's intention. Whether there is a specific expected position of the target object in the user instructions or not, layout information that meets the user's needs can be obtained, thereby improving the quality and efficiency of layout generation, and further enhancing the accuracy and speed of image editing and generation.

[0036] Embodiments of the present disclosure provide a layout generation method based on a large model. Figure 1 It is a schematic flowchart of the layout generation method based on a large model according to an embodiment of the present disclosure. This layout generation method based on a large model can be applied to a layout generation device based on a large model. The layout generation device based on a large model is located in an electronic device. The electronic device includes but is not limited to fixed devices and / or mobile devices. For example, fixed devices include but are not limited to servers, and the server can be a cloud server or a general server. For example, mobile devices include but are not limited to layout generation devices, and the layout generation device is a mobile phone, a tablet computer, etc. In some possible implementation manners, this layout generation method based on a large model can also be implemented by a processor calling computer-readable instructions stored in a memory.

[0037] As Figure 1 shown, the layout generation method based on a large model includes:

[0038] S101. Extract visual information of the original image through a target detection model and a semantic segmentation model;

[0039] S102. Convert visual information into structured data, and use spatial relation encoding to convert the structured data into a natural language description; wherein, the spatial relation encoding is used to convert the coordinates, dimensions, and relative positions of elements into a natural language description;

[0040] S103. Based on the natural language description and user instructions, construct a multimodal Prompt; wherein, the user instructions include actions for the original image and the target objects corresponding to the actions, and the multimodal Prompt is used to associate the natural language description with the user instructions;

[0041] S104. Input the constructed multimodal Prompt into a large model to obtain the layout information of the target object in the original image output by the large model; wherein, the layout information at least includes the position and dimensions of the target object in the original image.

[0042] In the embodiments of the present disclosure, the object detection model is used to identify the objects existing in the image and determine the position and bounding box of each object. For example, the object detection model can be a You Only Look Once (YOLO) model, a Faster Region-based Convolutional Neural Network (Faster R-CNN) model, etc.

[0043] In the embodiments of the present disclosure, the semantic segmentation model is used to divide the image into multiple regions, and each pixel is classified into a specific object category (such as background, human body, vehicle, etc.). Exemplarily, the object detection model can be a U-Net, a Mask Region-based Convolutional Neural Network (Mask R-CNN), a Segmentation Network (SegNet), etc. For example, using the semantic segmentation model, a pixel-level accurate segmentation result can be generated.

[0044] In the embodiments of the present disclosure, the large model is a Large Language Model (LLM), specifically referring to an artificial intelligence system trained based on a large amount of data, capable of understanding and generating natural language, and generating the layout information of the target object according to the multimodal Prompt.

[0045] In the embodiments of the present disclosure, the visual information includes element categories, bounding box coordinates, and pixel-level region masks.

[0046] In the embodiments of the present disclosure, the element category is the type of each object in the image, such as "car", "person", "building", "animal", "plant", etc.

[0047] In the embodiments of the present disclosure, a pixel-level region mask refers to representing the region of an object with a binary mask to highlight its specific distribution in the image.

[0048] In the embodiments of the present disclosure, the bounding box coordinates refer to providing the position information of each object in the image, including the center point coordinates and the bounding box size.

[0049] In the embodiments of the present disclosure, SpatialRelation Encoding is used to convert visual information into natural language descriptions. For example, "There is a car in the center of the image, occupying 60% of the width and 40% of the height". When generating layout information, SpatialRelation Encoding helps the model understand the positions (such as "upper-middle part"), sizes (such as "occupying about 1 / 4 of the image width"), and relative positions (such as "side by side with another car") of the objects in the image.

[0050] In some embodiments, the visual information of the original image is extracted by a target detection model and a semantic segmentation model, including: identifying the element categories and bounding box coordinates of each object in the original image through the target detection model, and obtaining the fine-grained pixel-level region information in combination with the semantic segmentation model. For example, first, the Faster R-CNN model is used for object detection to locate objects in the image and return their class labels and bounding box coordinates. Then, in order to obtain more fine-grained visual information, the system also combines a semantic segmentation algorithm (such as U-Net or Mask R-CNN), and uses the U-Net model to process multi-class segmentation tasks and output the object category information to which each pixel point belongs.

[0051] In some embodiments, when converting visual information into structured data, the structured data includes element categories, bounding box coordinates, pixel-level region masks, etc. Specifically, for the element category, the type name of the target object is described in text form (such as "person", "car", etc.) and encoded through a class label. For the bounding box coordinates, the position information of the target object in the image is recorded, such as the center point coordinates and the bounding box size. These data are usually represented in floating-point form, and the specific format can be defined according to actual needs (such as X and Y axis coordinates). For the pixel-level region mask, the specific distribution region of the target object in the image is represented in binary format. For example, the pixel points of the target object are marked as 1, and the background region is marked as 0.

[0052] In some embodiments, when converting visual information into structured data, it further includes: aligning the output results of the object detection model and the semantic segmentation model to ensure that the two types of information can complement and coexist. For example, matching the bounding box coordinates with the pixel-level region masks to ensure accurate correspondence of the position information of each object. Extracting the global and local features of the target object in the image through the feature extraction network to support subsequent structured data processing. Finally, converting the obtained visual information into a unified structured data format. For example, storing the element category, bounding box coordinates, and pixel-level region masks in the form of a lightweight data interchange format (JavaScript Object Notation, JSON) or an Extensible Markup Language (XML) file. Another example is using the Unicode Transformation Format (UTF) for data serialization to ensure consistency between different devices and systems. Through the above steps, the system can extract rich visual information from the original image and provide support for subsequent layout generation and user interaction in the form of structured data.

[0053] In some embodiments, using the spatial relationship encoding technique, the structured data is converted into a natural language description so that the model can understand and process the visual information. Specifically, the system first analyzes the image to extract the category information (such as "person", "car") and position information (bounding box coordinates) of the target object. Through the spatial relationship modeling model, the system converts this structured data into a natural language description, such as: "Find a 'person' in the image, whose position is on the left, upper-middle part, occupying about 1 / 3 of the image width." Another example is: "Place the 'car' on the right, near the bottom of the image, and arrange it side by side with another'vehicle'." This method combines the results of object detection and the spatial relationship modeling algorithm to ensure that the semantics of the visual information can be accurately parsed and processed by the model, thus providing high-quality semantic guidance for subsequent tasks (such as image generation or retrieval).

[0054] In some embodiments, based on the natural language description and user instructions, the system constructs a multi-modal Prompt to guide the model to understand the task objective and visual context information. Specifically, the multi-modal Prompt combines the following elements:

[0055] Task description: Clearly define the actions that the model needs to perform, such as "identify and label" or "locate specific category objects".

[0056] Target object description: Specifically indicate the object category (such as "red object") that needs to be concerned about and its possible visual features.

[0057] Visual context information: Transmit the position parameters of the object through natural language description (such as "upper middle part", "occupying about 1 / 4 of the image width").

[0058] Spatial relationship modeling: If necessary, the multi-modal Prompt will also include the arrangement or relative position relationship between multiple target objects.

[0059] For example, the user instruction is: "Please identify and label the'red object' in the image."

[0060] Correspondingly, the generated multi-modal Prompt is: "Please locate and label the red object in the image, which is in the upper middle part, occupies about 1 / 4 of the image width, and is circled with a box in the background area."

[0061] Through such a multi-modal Prompt, the model can better understand the task objectives, the visual features to be concerned about, and the spatial relationships between objects, so as to perform efficient and accurate image analysis and generation tasks.

[0062] In some embodiments, when the large model determines that other objects also need to be adjusted based on the multi-modal Prompt, the output layout information includes: the position of the target object in the original image, and the positions of other objects in the original image. Here, the positions of other objects in the original image are different from their original positions in the original image. For example, the positions of other objects in the original image will change their original positions according to the deletion or addition of the target object. In this way, not only the semantic correctness of the generated content is ensured, but also the visual presentation is optimized, meeting the dual requirements of users for details and overall effects.

[0063] In some embodiments, the constructed multi-modal Prompt is input into the large model, and the large model outputs the position and layout information of the target object according to the multi-modal Prompt. Specifically, the system will generate a multi-modal Prompt for the target object based on the user instruction and natural language description to guide the model to accurately identify, locate, and process the position parameters of the relevant object. For example, for the positioning task of the "red object", the multi-modal Prompt may include the following content: "Please find and label the'red object' in the image, which is located in the upper middle part, occupies about 1 / 4 of the image width, and is circled with a box in the background area." By constructing and optimizing the multi-modal Prompt, task instructions and visual context information can be provided to the large model, enabling the large model to accurately extract the positioning information and layout parameters of the target object based on the multi-modal Prompt, and thus efficiently complete the corresponding visual understanding and processing tasks.

[0064] The technical solution of the embodiments of the present disclosure extracts visual information through a target detection model and a semantic segmentation model, which helps to improve the accuracy and detail of the acquired visual information. The extracted visual information is converted into structured data, and a spatial relationship encoding method is used to convert complex image information into natural language descriptions, enabling the model to better understand the context information and object relationships in the image. By combining visual information, user instructions, and a large model, intelligent generation of the target layout is achieved, improving the quality and efficiency of layout generation.

[0065] In some embodiments, the layout generation method based on a large model further includes: when the user instruction includes the expected position of the target object, verifying the actual position of the target object in the original image based on the layout information; in response to determining that the deviation between the actual position and the expected position exceeds a preset threshold, updating the multi-modal Prompt based on the deviation to generate new layout information.

[0066] In the embodiments of the present disclosure, the target object is a physical object or a virtual object that needs to be located or operated as pointed out in the user instruction.

[0067] In the embodiments of the present disclosure, the expected position is the position of the target object clearly specified in the user instruction, such as "in the middle on the left", "above the table", etc.

[0068] In the embodiments of the present disclosure, the actual position is the actual coordinates of the target object calculated by the system based on the layout information.

[0069] In the embodiments of the present disclosure, the preset threshold is the critical value for determining whether the position deviation needs to trigger correction. During the target position verification process, when the error between the actual position and the expected position exceeds the preset threshold, the system needs to make adjustments to ensure that the final image meets the expectations.

[0070] In some embodiments, a target detection algorithm is used to identify the target objects existing in the image, and the coordinate positions and categories of each target object are recorded. If there are multiple target objects, it is clear which target objects meet the expected positions. For each target object, calculate the error between its actual position and the expected position. For example, the coordinate error (such as the absolute difference between the horizontal and vertical coordinates) or the relative error (such as the ratio to the expected position) is used to measure the degree of deviation. If the deviation of a single target object exceeds the preset threshold, or multiple target objects do not meet the expectations, further adjustments are required. For example, update the multi-modal Prompt according to the deviation information. Another example is to optimize the generation effect by adding position-related feature descriptions or adjusting the model training parameters. This implementation ensures that the layout generation result is more accurate in line with the user's expectations and can dynamically adjust the generation strategy to adapt to the actual situation.

[0071] In some embodiments, a new layout information is generated using a new multi-modal Prompt, and the above steps are repeated until the expected position is satisfied or the deviation is lower than a preset threshold. Through this verification mechanism, the system can timely detect situations that do not conform to the user's expectations and make corresponding adjustments, thereby ensuring that the generated image based on the new layout information meets the user's needs, improving the accuracy in the image editing and generation process, and at the same time providing users with a higher degree of customization options to meet the diverse needs in different scenarios.

[0072] In some embodiments, updating the multi-modal Prompt based on the deviation includes: dynamically adjusting the deviation threshold according to specific application requirements and the type of target object to adapt to the accuracy requirements of different scenarios.

[0073] In some embodiments, updating the multi-modal Prompt based on the deviation includes: while adjusting the position information, at least one factor such as the image style and object size is also adjusted to improve the overall generation effect of the layout. For example, adjusting the object size can make it more balanced in the picture, and a reasonable image style can enhance the overall aesthetic feeling, so that the final output not only meets the user's needs but also has a professional and attractive appearance, enabling precise control of the generated content and thus significantly improving the overall effect. This method not only ensures the semantic correctness of the generated content but also optimizes the visual presentation, meeting the dual needs of users for details and overall effects, and fully demonstrating the important role of multi-modal Prompt in large model generation tasks.

[0074] In some embodiments, first parse the expected position in the user instruction (for example, extract coordinate information through natural language processing technology: "Place the sofa 1 meter away from the wall on the east side of the living room"), and then perform a position comparison. For example, calculate the Euclidean distance deviation between the actual position and the expected position (such as ). Quickly match the logical position of the target object in the layout through a spatial hashing algorithm (for example, divide the room into 1-meter × 1-meter units and identify the grid where the sofa belongs). In this way, cross-validation using multi-modal data (coordinates + grid layout) is adopted to improve the accuracy of position judgment.

[0075] In some embodiments, before updating the multi-modal Prompt, prompt information is output. Here, the prompting methods include but are not limited to text prompts, visual prompts, and voice prompts. Exemplarily, the text prompt is: Generate adjustment suggestions (such as "It is detected that the sofa has shifted 0.5 meters. It is recommended to move it left to the coordinates (3,5)"). Exemplarily, the visual prompt is: Highlight the offset area in an Augmented Reality (AR) interface and overlay a virtual image of the expected position. Exemplarily, the voice prompt is: Broadcast the rectification instruction through a Text To Speech (TTS) engine (such as "Warning: The position of the bookcase exceeds the safe range. Please rearrange it").

[0076] Thus, if the deviation between the actual position and the expected position exceeds the set threshold, the multi-modal Prompt is updated and the layout information is regenerated. Through this dynamic verification and adjustment mechanism, the accuracy of the layout information can be improved. By continuously optimizing the multi-modal Prompt, the overall generation effect can be enhanced, making the image editing and generation process more accurate and efficient.

[0077] In some embodiments, the layout generation method based on a large model further includes: when the user instruction does not include the expected position of the target object, predicting the expected position of the target object based on the original image and the user instruction; verifying the actual position of the target object in the original image based on the layout information; and in response to determining that the deviation between the actual position and the expected position exceeds a preset threshold, updating the multi-modal Prompt based on the deviation to generate new layout information.

[0078] In some embodiments, a target detection algorithm is used to identify the main objects present in the image. Based on the type and context content of the target object in the user instruction, the reasonable positions of these target objects in the image are predicted as the predicted positions. For example, in a family photo, if the user requests to place "people", the system may predict the specific positions of these people in the image.

[0079] In some embodiments, when the expected position of the target object is not explicitly specified in the user instruction, the system first predicts the expected position of the target object based on the original image and the user instruction. Subsequently, the generated layout information is used to verify the actual position of the target object. If the deviation between the actual position and the expected position exceeds a preset threshold (such as the coordinate error or relative error exceeds the specified value), the system updates the multi-modal Prompt according to the deviation information and regenerates the layout information. Additionally, the optimized layout information is used as feedback to guide the model to continue adjusting until the expected target position or the optimal state is finally achieved.

[0080] In this way, even if the expected position is not explicitly specified in the user instruction, layout information that meets the user's needs can be quickly obtained, thereby improving the quality and efficiency of layout generation, and further enhancing the accuracy and speed of image editing and generation. By continuously optimizing the multi-modal Prompt, the generation effect of the layout can be improved, making the image editing and generation process more accurate and efficient.

[0081] In some embodiments, predicting the expected position of the target object based on the original image and the user instruction includes: using an image segmentation algorithm to determine the types and the sizes of the occupied spatial regions of the existing objects in the original image; determining the reasonable position of the target object in the original image according to the types and the sizes of the occupied spatial regions of the existing objects and the type of the target object in the user instruction; and determining the expected position according to the reasonable position.

[0082] In the embodiments of the present disclosure, the original image refers to the initial image to be edited provided by the user (such as a photo, a design drawing, etc.), which includes one or more existing objects (such as a person, an animal, furniture, background elements, etc.). For example, an indoor photo includes a sofa, a coffee table, and a blank wall.

[0083] In the embodiments of the present disclosure, the target object refers to a new object specified in the user instruction to be added to the original image, or an object specified in the user instruction to be removed from the original image. For example, the "painting" in the user instruction "hang a painting on the wall" is the target object.

[0084] In the embodiments of the present disclosure, the image segmentation algorithm refers to an algorithm that can identify the types of objects and the sizes of the occupied spatial regions in the image. For example, the image segmentation algorithm can be an algorithm that divides the original image into multiple semantic regions based on computer vision technology and identifies the object types and their pixel-level spatial occupancy region sizes. It should be noted that the present disclosure does not specifically limit the image segmentation algorithm.

[0085] In the embodiments of the present disclosure, the reasonable position refers to the placement logic of the target object that conforms to physical laws, aesthetic rules, or user intentions. For example, the painting should be located at the center of the blank area on the wall and does not overlap with the sofa.

[0086] In some embodiments, using an image segmentation algorithm to determine the existing object information includes: inputting the original image into a pre-trained semantic segmentation model to obtain a pixel-level label map output by the semantic segmentation model, and identifying the object types (such as "person", "furniture", "ground") and their masks; calculating the sizes of the regions occupied by each object (such as pixel area), position coordinates, and bounding boxes. Example: Identify that the sofa in the image occupies the lower left corner area (coordinates x1, y1 to x2, y2), and the blank area on the wall is x3, y3 to x4, y4.

[0087] In some embodiments, determining a reasonable position based on object information and instructions includes: extracting the target object type (such as "painting") and implicit constraints (such as "hanging on the wall") through a Natural Language Processing (NLP) model; matching a rule library based on the object type (for example, a "painting" needs to be placed on a "wall surface" and away from "furniture"); calculating candidate regions in combination with aesthetic rules (such as centering, golden ratio), and excluding regions that overlap with existing objects or exceed the image boundary. For example, according to the rules, the reasonable position of the "painting" is the center point (x, y) of the blank area on the wall, and the size is 20% of the wall area.

[0088] In some embodiments, determining an expected position based on the reasonable position includes: converting the reasonable position into the bounding box coordinates (x_min, y_min, x_max, y_max) of the target object; fine-tuning the position through a Generative Adversarial Networks (GAN) to adapt to the background lighting / perspective (such as adjusting the tilt angle of the painting to match the wall perspective) to obtain the expected position. For example, the expected position is (x3 + Δx, y3 + Δy, x3 + w, y3 + h), where Δx and Δy are the centering offsets.

[0089] In this way, predicting the expected position of the target object based on the original image and user instructions can improve the accuracy of determining the expected position.

[0090] In some embodiments, updating the multi-modal Prompt based on the deviation includes: determining the first target parameter information of the target object based on the deviation; updating the multi-modal Prompt based on the first target parameter information.

[0091] In the embodiments of the present disclosure, the deviation is the difference between the actual generated position of the target object (such as the coordinates of the target object generated by the large model based on the multi-modal Prompt in the original image) and the expected position (the ideal position specified by the user or calculated by the algorithm), which can be quantitatively represented as pixel distance, Intersection over Union (IOU), etc. For example, the user specifies "place the vase in the center of the table", but the position of the vase generated by the AI is offset by 20 pixels, and the deviation value is Δd = 20.

[0092] In the embodiments of the present disclosure, the first target parameter information of the target object is a key attribute for correcting the generation result of the target object, including spatial parameters (coordinates, dimensions), physical parameters (lighting angle, perspective ratio), or semantic parameters (relationship with surrounding objects). For example, the first target parameter can be "the center point coordinates (x, y) of the bottom of the vase, height 30 cm, and the distance from the table edge ≥ 5 cm".

[0093] In some embodiments, the offset between the actual position and the expected position of the target object is calculated through computer vision algorithms. For example, the coordinates of the center point at the bottom of the vase in the detected generated image are (120, 150), and the expected coordinates are (100, 150), then Δx = 20.

[0094] In some embodiments, the first target parameter information is determined based on the deviation, including: selecting the parameter to be adjusted according to the deviation type (position deviation → coordinate parameter, size deviation → scaling parameter); generating a correction value through a predefined mapping table or regression model.

[0095] In some embodiments, updating the multi-modal Prompt based on the first target parameter information includes: embedding the first target parameter information in the original Prompt and converting it into a machine-parsable structured instruction. For example, the original text Prompt "Add a vase" is updated to "Add a vase, with the center point coordinates at the bottom being (100, 150) and the height being 30 cm".

[0096] In this way, automatically correcting the multi-modal Prompt based on the first target parameter can improve the correction speed of the multi-modal Prompt and the accuracy of the updated multi-modal Prompt.

[0097] In some embodiments, updating the multi-modal Prompt based on the deviation includes: determining the second target parameter information of the existing object in the original image based on the deviation; updating the multi-modal Prompt based on the second target parameter information.

[0098] In the embodiments of the present disclosure, the second target parameter information of the existing object refers to the attribute parameters that need to be adjusted for the existing object, which are deduced by analyzing the deviation between the target object and the expected position.

[0099] In the embodiments of the present disclosure, the second target parameter information includes, but is not limited to:

[0100] Geometric parameters: position coordinates, rotation angle, size ratio;

[0101] Semantic parameters: spatial relationship with the target object (such as "support", "occlusion");

[0102] Physical parameters: light direction, material reflectivity.

[0103] For example, when the target object "vase" fails to be placed at the expected position on the table, the second target parameter may be "the height of the table needs to be lowered by 10 pixels" or "the perspective angle of the table needs to be rotated by 5 degrees".

[0104] In some embodiments, updating the multi-modal Prompt based on the second target parameter information includes: on the basis of preserving the original instruction semantics, updating the multi-modal Prompt based on the second target parameter information.

[0105] In some embodiments, the incremental update of the multi-modal Prompt includes: directly modifying the Prompt text based on the second target parameter information (such as appending "the desktop should be lowered by 10 cm").

[0106] In some embodiments, implicitly modifying the Prompt text based on the second target parameter information includes: adjusting the existing object features in the image latent space (such as fine-tuning the feature layer related to the desktop height).

[0107] The embodiments of the present disclosure achieve reverse compensation by modifying the existing objects, which are irreplaceable in the following scenarios: when the target object must keep the parameters unchanged (such as the fixed size of the implant in medical images); when there are generation errors in the existing objects themselves (such as the incorrect wall inclination in the drawing). In the generation of autonomous driving simulation data, the vehicle position can be made to conform to traffic rules by adjusting the road marking parameters. In e-commerce virtual fitting, the clothing fitting degree can reach millimeter-level accuracy by modifying the human body model parameters.

[0108] In some embodiments, detecting the deviation amount between the actual generation position and the expected position of the target object; determining the adjustable parameter set of the existing object that causes the deviation through differentiable analysis; encoding the parameter set as an incremental tag of the multi-modal Prompt; and iteratively generating until the deviation amount is lower than the preset threshold. In this way, the multi-modal Prompt is updated based on the reverse parameters.

[0109] In this way, by modifying the existing objects to achieve reverse compensation and updating the multi-modal Prompt based on the reverse compensation, the accuracy of the updated multi-modal Prompt can be improved.

[0110] In some embodiments, the method further includes: dynamically adjusting the preset threshold according to the application scenario of the original image and the type of the target object.

[0111] In the embodiments of the present disclosure, the application scenario refers to the vertical domain category divided by image semantic understanding. The technical standards of different vertical domain categories directly affect the deviation tolerance. For example, for medical images, the anatomical structure positioning threshold ≤ 3 pixels. Another example is that for industrial design images, the mechanical component assembly threshold ≤ 0.1 mm. For example, for artistic creation images, the free layout threshold can be relaxed to 50 pixels (such as the elements of an abstract painting).

[0112] In some embodiments, determining the application scenario of the original image through a scene classifier.

[0113] In some embodiments, the preset threshold is dynamically adjusted according to the application scenario of the original image and the type of the target object, including: determining an adjustment coefficient according to the weights corresponding to the application scenario and the type of the target object respectively; and adjusting the preset threshold based on the adjustment coefficient. In this way, the scene adaptability of the preset threshold can be improved.

[0114] In some embodiments, the original image is input into a multi-label classification model to obtain the scene probability distribution output by the multi-label classification model; the scene with the highest probability is selected as the main classification, and the secondary classifications are retained for mixed scene processing. For example, for an operating room image → main classification "medical" (0.7), secondary classification "industrial" (0.3).

[0115] In some embodiments, the preset threshold is dynamically adjusted according to the application scenario of the original image and the type of the target object, including: obtaining a scene reference threshold; and obtaining the preset threshold according to the following formula. Wherein, the dynamic threshold = reference threshold × type weight × scene mixing coefficient. In this way, the accuracy of the preset threshold can be improved.

[0116] In some embodiments, the preset threshold is dynamically adjusted according to the application scenario of the original image and the type of the target object, including: identifying the main classification and secondary classification of the application scenario of the original image through visual features; obtaining the predefined importance weights according to the semantic type of the target object; and fusing the scene classification probability and the importance weights to dynamically adjust the preset threshold. In this way, the accuracy of the preset threshold can be improved.

[0117] In this way, by dynamically adjusting the preset threshold, the accuracy and scene adaptability of the preset threshold can be improved. In some embodiments, the target object includes a first object. When the user instruction includes deleting the first object, a multi-modal Prompt is constructed, including: removing the structured description of the first object and its corresponding spatial relationship encoding from the natural language description to obtain an updated natural language description; marking the deletion action for the first object in the user instruction and associating it with the updated natural language description to obtain a multi-modal Prompt.

[0118] In the embodiments of the present disclosure, structured data refers to a detailed classification and location description of image elements, such as "central position", "above", etc. It describes a certain target or scene in a concise and clear format, such as "Person E is located in the central position", "Building F is located in the background area".

[0119] In the embodiments of the present disclosure, marking the deletion action means specifying the specific elements that need to be excluded or hidden to ensure that they do not appear in the final result.

[0120] In some embodiments, the system analyzes "delete the first object" in the user instruction to determine its specific location and related content in the natural language description. In the original natural language description, all elements to be retained, their structured data, and spatial relationship encodings are included. The structured data and spatial relationship encodings related to the first object are deleted from the multi-modal Prompt to ensure that the first object is not included when the model generates the layout. In this way, the "delete" operation is clearly marked in the user instruction and associated with the updated natural language description to ensure that the system understands what content needs to be excluded. The adjusted multi-modal Prompt is used to guide the model to generate a new layout, ensuring that the first object has been removed while other objects are presented as required. The system provides feedback to the user based on the generated result to check whether the delete instruction has been fully followed and further adjusts the multi-modal Prompt to optimize the layout quality.

[0121] Exemplarily, the system first determines the deletion target in the user instruction through natural language analysis, such as "brown rabbit". Then, relevant structured information is extracted from the original description, such as "the brown rabbit is located in the central position", and the relevant content is cleared. Then, a deletion mark is added to the multi-modal Prompt to ensure that the layout generated by the model does not include the "brown rabbit". Finally, a new image is generated based on the layout information generated by the model, and it is checked and confirmed that the brown rabbit in the new image has been removed.

[0122] In this way, by identifying the target object, adjusting the multi-modal Prompt, and executing the deletion instruction, the system can accurately respond to user requirements, improve the accuracy of the layout, and thus contribute to improving the flexibility of image generation. By optimizing the multi-modal Prompt, the images generated by the system can better meet expectations, enhancing the user experience and satisfaction.

[0123] In some embodiments, the target object includes a second object. When the user instruction includes adding the second object, a multi-modal Prompt is constructed, including: analyzing and extracting the existing objects and their spatial relationships in the natural language description; determining the insertion position of the second object based on the existing objects and their spatial relationships; generating a relative position description of the second object and the existing objects based on the insertion position of the second object; and associating the addition action in the user instruction with the relative position description to obtain the multi-modal Prompt.

[0124] In the embodiments of the present disclosure, the addition action refers to clearly identifying the content to be added in the user instruction and associating it with the updated natural language description. Here, the added content includes, but is not limited to, text, images, audio, video, etc., and the specific added content can be determined according to user needs.

[0125] In some embodiments, existing objects and their specific attributes, location information, and relevant spatial relationships in the scene are extracted from the natural language description. According to the addition action in the user instruction, the type of object to be inserted ("second object") is determined. By analyzing the logical structure and spatial layout of the existing scene, a suitable insertion position is determined, and the specific coordinates of this position relative to the existing objects are recorded. A relative position description of the second object and other objects is generated. For example, describe the absolute position (such as x, y coordinates) of the "second object" in the scene. Detail the spatial relationships between the "second object" and the surrounding existing objects, such as distance, direction, and relative position. Associate the "addition action" marker in the user instruction with the generated relative position description to ensure that the system can understand and execute the correct insertion operation. Integrate the above information to form the final multimodal Prompt.

[0126] In this way, when marking the addition of the second object in the user instruction, it can be ensured that the constructed multimodal Prompt reasonably and accurately reflects the natural language description and user requirements, improving the accuracy of the layout, and thus enhancing the accuracy of image generation.

[0127] In some embodiments, constructing a multimodal Prompt includes: mapping the natural language description and the user instruction to the same semantic space through a semantic alignment algorithm to generate a joint representation; converting the joint representation into a multimodal Prompt through a predefined template.

[0128] In the embodiments of the present disclosure, the semantic alignment algorithm is an algorithm that maps the natural language description and the user instruction to the same semantic space to generate a joint representation. The semantic alignment algorithm can adopt NLP techniques, such as stemming, syntactic analysis, or deep learning models, to eliminate the ambiguity between the description and the instruction.

[0129] In the embodiments of the present disclosure, the predefined template is a preset structured framework for converting the joint semantic representation into a machine-parsable multimodal instruction format according to specific rules. The template may contain specific instructions for the scene, object category, and location, guiding the large model on how to combine text information and image information to generate a layout.

[0130] In this way, through the joint representation and template conversion, it is ensured that the generated content is highly consistent with the user instruction, reducing the probability of generating a layout that does not meet the user requirements. Semantic alignment and template optimization can quickly respond to user feedback and meet personalized needs. This construction method can systematically convert user requirements into effective multimodal Prompts, not only improving the accuracy and consistency of the multimodal Prompts, but also providing users with a more intelligent and flexible content generation experience.

[0131] In some embodiments, the predefined template at least includes: existing object information in the original image; association information between the user instruction and the target object.

[0132] In the embodiments of the present disclosure, the existing object information in the original image includes: recording the position, category, and other relevant attributes of each existing object, such as absolute coordinates (x, y), image description, or category name. This enables the system to accurately understand and locate the content to be processed.

[0133] In the embodiments of the present disclosure, the association information between the user instruction and the target object includes: elaborating on the relationship between the user operation instruction and the target object, such as the insertion point or the modified object, to ensure that the generated content meets the user's needs and expectations. The template needs to clarify the operations that the user wishes to add or modify and associate them with the target object.

[0134] Figure 2 A schematic diagram showing the generation of a multimodal Prompt based on a predefined template is shown, as Figure 2 shown, the multimodal Prompt includes a role setting (for example, if you are a planning expert), requirements (for example, now it is required to generate 5 cats on the given map and generate a layout), and visual elements (including the coordinates of the existing main bodies in the picture, the entity segmentation of the picture, etc.). It can be understood that Figure 2 This is only for exemplary illustration and does not limit all possible situations of the predefined template. Here, an exhaustive list is not made.

[0135] In this way, the predefined template can systematically transform the user's requirements into an effective multimodal Prompt, enabling the system to adapt to different types of generation tasks (such as images, audio, video, etc.), achieve cross-modal generation, ensure that the final generated content meets both semantic requirements and visual requirements, and improve the quality of layout generation.

[0136] In some embodiments, constructing a multimodal Prompt includes: associating the action in the user instruction with the target object in the natural language description through a dynamic binding method; generating a multimodal Prompt according to the association result.

[0137] In the embodiments of the present disclosure, the dynamic binding method is a technology for real-time matching of the action in the user instruction with the target object in the natural language description. Through cross-modal feature analysis and context reasoning, a logical association between the action and the object is established.

[0138] In some embodiments, constructing a multi-modal Prompt, especially dynamically binding the actions in the user instruction to the target object in the natural language description, can be achieved through the following steps: 1. Extract the main content from the natural language description, such as the target object and its attributes. 2. Interpret the user instruction to identify the action part (such as adding, deleting, etc.) and specific details (such as color, position). 3. Convert the analysis results into a structured multi-modal Prompt, clarifying the modalities and details that need to be adjusted. 4. Use the multi-modal Prompt to guide the layout generation model to accurately respond to the modifications or additions in the user instruction.

[0139] Exemplarily, the natural language description of the original image is: A little cat is taking a nap in the sun. The user instruction is: Add a small red flower. The dynamic binding analysis is as follows: The existing object "little cat" is in the main position, and the environment is sunny; the action part is "add a small red flower", determining the position and characteristic color of the target object "small flower" to be inserted. The multi-modal Prompt is: In the scene, insert a small red flower at a reasonable position without disturbing the main body. Generate a new image according to this multi-modal Prompt, showing the little cat, the environment in the sun, and adding a new small red flower.

[0140] In this way, by dynamically binding the actions in the user instruction and constructing a multi-modal Prompt, the generation process can be flexibly guided, improving the overall quality of the layout.

[0141] Figure 3 The schematic diagram of the architecture for generating a multi-modal Prompt is shown, as Figure 3 shown, using a semantic segmentation model to perform semantic segmentation processing on the original image to obtain the semantic segmentation layout map of the original image; using an object detection model, combining the semantic segmentation layout map to determine the image features of the original image to obtain the visual information of the original image, and then obtaining the natural language description fused with visual information; using a text parser to parse the user's needs to obtain the requirement content, and then generating a user instruction fused with the requirement content; inputting the natural language description and the user instruction into a large model to train the large model to have the layout planning ability.

[0142] In this way, combining this layout generation method with image content generation can generate a large amount of picture data, providing a rich data source for large model training and improving the layout generation ability of the model. In this solution, the user input instruction is simple and highly scalable. Whether there is a specific expected position in the user input instruction or not, layout information that meets the user's needs can be obtained, thus improving the quality and efficiency of layout generation, and further enhancing the accuracy and speed of image editing and generation. That is to say, this layout generation method can not only handle the situation with clear position indication, but also automatically generate a reasonable layout scheme when lacking specific guidance.

[0143] The technical solutions of the present disclosure can be applied to various scenarios such as adapting images, adjusting scene details, or implementing highly customized visual content. For example, in the field of image editing and enhancement, it can quickly locate the layout information of the target object according to the user's instructions, and then modify or edit the image to improve the image processing efficiency. For another example, in an intelligent vision system, it is used for tasks such as scene analysis and target localization, significantly improving the accuracy of analysis and localization. For yet another example, in the field of education and training, it can be used as a learning tool for image understanding to help users better understand the image content. The above is only an exemplary illustration and does not limit all possible situations of the application scenarios. Here, no exhaustive listing is made.

[0144] It should be understood that Figure 2 and Figure 3 the schematic diagrams shown are merely exemplary rather than restrictive, and they are extensible. Those skilled in the art can make various obvious changes and / or substitutions based on the examples of Figure 2 and Figure 3 and the obtained technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0145] The embodiments of the present disclosure provide a layout generation device based on a large model. As shown in Figure 4 , the device may include: an extraction module 401 for extracting visual information of the original image through a target detection model and a semantic segmentation model; a conversion module 402 for converting the visual information into structured data and converting the structured data into a natural language description by using spatial relationship encoding, where the spatial relationship encoding is used to convert the coordinates, dimensions, and relative positions of the elements into a natural language description; a construction module 403 for constructing a multimodal Prompt based on the natural language description and the user's instructions, where the user's instructions include actions for the original image and the target objects corresponding to the actions, and the multimodal Prompt is used to associate the natural language description with the user's instructions; a generation module 404 for inputting the constructed multimodal Prompt into the large model to obtain the layout information of the target object in the original image output by the large model, where the layout information includes the position and dimensions of the target object in the original image.

[0146] In some embodiments, the layout generation device based on the large model further includes:

[0147] a first verification module (not shown in Figure 4 ) for verifying the actual position of the target object in the original image based on the layout information when the user's instructions include the expected position of the target object;

[0148] a first optimization module (not shown in Figure 4 ) for updating the multimodal Prompt based on the deviation in response to determining that the deviation between the actual position and the expected position exceeds a preset threshold to generate new layout information.

[0149] In some embodiments, the large model-based layout generation device further includes:

[0150] A second verification module ( Figure 4 not shown in the figure), configured to, when the user instruction does not include the expected position of the target object, predict the expected position of the target object based on the original image and the user instruction; verify the actual position of the target object in the original image based on the layout information;

[0151] A second optimization module ( Figure 4 not shown in the figure), configured to, in response to determining that the deviation between the actual position and the expected position exceeds a preset threshold, update the multi-modal Prompt based on the deviation to generate new layout information.

[0152] In some embodiments, the second verification module ( Figure 4 not shown in the figure) is configured to: use an image segmentation algorithm to determine the type and the occupied space region size of the existing objects in the original image; determine the reasonable position of the target object in the original image according to the type and the occupied space region size of the existing objects, and the type of the target object in the user instruction; determine the expected position according to the reasonable position.

[0153] In some embodiments, the first optimization module or the second optimization module ( Figure 4 not shown in the figure) is specifically configured to: determine the first target parameter information of the target object based on the deviation; update the multi-modal Prompt based on the first target parameter information.

[0154] In some embodiments, the first optimization module or the second optimization module ( Figure 4 not shown in the figure) is specifically configured to: determine the second target parameter information of the existing objects in the original image based on the deviation; update the multi-modal Prompt based on the second target parameter information.

[0155] In some embodiments, the large model-based layout generation device further includes: an adjustment module ( Figure 4 not shown in the figure), configured to: dynamically adjust the preset threshold according to the application scenario of the original image and the type of the target object.

[0156] In some embodiments, the construction module 403 includes:

[0157] A first processing sub-module, configured to, when the user instruction includes deleting the first object, remove the structured description of the first object and its corresponding spatial relationship encoding from the natural language description to obtain an updated natural language description;

[0158] The first associated sub-module is used to mark the deletion action for the first object in the user instruction and associate it with the updated natural language description to obtain a multi-modal Prompt.

[0159] In some embodiments, the target object includes a second object. The construction module 403 includes:

[0160] The second processing sub-module is used to insert the element category of the second object and the interpolation coordinates based on the existing object spatial relationship into the structured description when the user instruction includes adding the second object; analyze and extract the existing objects and their spatial relationships in the natural language description; determine the insertion position of the second object based on the existing objects and their spatial relationships; generate the relative position description of the second object and the existing objects based on the insertion position of the second object.

[0161] The second associated sub-module is used to associate the addition action in the user instruction with the relative position description to obtain a multi-modal Prompt.

[0162] In some embodiments, the construction module 403 includes:

[0163] The first generation sub-module is used to map the natural language description and the user instruction to the same semantic space through a semantic alignment algorithm to generate a joint representation; convert the joint representation into a multi-modal Prompt through a predefined template.

[0164] In some embodiments, the predefined template includes: the existing object information in the original image; the association information between the user instruction and the target object.

[0165] In some embodiments, the construction module 403 includes:

[0166] The second generation sub-module is used to associate the action in the user instruction with the target object in the natural language description through dynamic binding; generate a multi-modal Prompt according to the association result.

[0167] For the specific functions and example descriptions of each module and sub-module of the device in the embodiments of the present disclosure, reference can be made to the relevant descriptions of the corresponding steps in the above method embodiments, which will not be elaborated here.

[0168] The layout generation device based on a large model in the embodiments of the present disclosure can improve the quality and efficiency of layout generation.

[0169] The embodiments of the present disclosure provide a schematic diagram of the scenario of a layout generation method based on a large model, as Figure 5 shown.

[0170] As described above, the layout generation method based on a large model provided by the embodiments of the present disclosure is applied to an electronic device. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers.

[0171] Specifically, the electronic device can specifically perform the following operations:

[0172] Extract visual information of the original image through a target detection model and a semantic segmentation model; convert the visual information into structured data, and convert the structured data into a natural language description by using spatial relationship encoding; construct a multimodal Prompt based on the natural language description and user instructions; input the constructed multimodal Prompt into the large model to obtain layout information of the target object in the original image output by the large model, where the layout information includes the position and size of the target object in the original image; input the layout information into an image processing model to obtain a new image output by the image processing model.

[0173] It should be understood that Figure 5 the scene graph shown is merely illustrative and not restrictive, and those skilled in the art can make various obvious changes and / or substitutions based on Figure 5 the examples, and the obtained technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0174] In the technical solution of the present disclosure, the acquisition, storage, and application of the user's personal information involved all comply with the provisions of relevant laws and regulations and do not violate public order and good customs.

[0175] According to the embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0176] Figure 6 FIG. shows a schematic block diagram of an exemplary electronic device 600 that can be used to implement the embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0177] As Figure 6As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to computer programs stored in a read-only memory (ROM) 602 or computer programs loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0178] Multiple components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a magnetic disk, an optical disc, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0179] The computing unit 601 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include but are not limited to a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 601 executes the various methods and processes described above, such as the layout generation method based on a large model. For example, in some embodiments, the layout generation method based on a large model can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the layout generation method based on a large model described above can be executed. Alternatively, in other embodiments, the computing unit 601 can be configured to execute the layout generation method based on a large model in any other appropriate manner (e.g., by means of firmware).

[0180] The various embodiments of the systems and techniques described above in this specification can be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which may be a special-purpose or a general-purpose programmable processor that receives data and instructions from, and transmits data and instructions to, a storage system, at least one input device, and at least one output device.

[0181] The program code for implementing the methods of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general purpose computer, a special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions / operations specified in the flowchart and / or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

[0182] In the context of this disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0183] To provide for interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can also be used to provide for interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0184] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0185] A computer system may include a client and a server. The client and the server are generally far from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server incorporating a blockchain.

[0186] It should be understood that various forms of processes shown above can be used, steps can be reordered, added or deleted. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this application can be achieved, and this is not limited herein.

[0187] The above specific embodiments do not constitute a limitation on the protection scope of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure shall be included within the protection scope of this disclosure.

Claims

1. A layout generation method based on a large model, comprising: extracting visual information of the original image through an object detection model and a semantic segmentation model; converting the visual information into structured data, and converting the structured data into a natural language description by using spatial relation encoding, where the spatial relation encoding is used to convert the coordinates, dimensions and relative positions of elements into the natural language description; constructing a multimodal Prompt based on the natural language description and a user instruction; wherein, the user instruction includes an action for the original image and a target object corresponding to the action, and the multimodal Prompt is used to associate the natural language description with the user instruction; inputting the constructed multimodal Prompt into the large model to obtain layout information of the target object in the original image output by the large model, where the layout information includes the position and dimensions of the target object in the original image.

2. The method according to claim 1, further comprising: when the user instruction includes the expected position of the target object, verifying the actual position of the target object in the original image based on the layout information; in response to determining that the deviation between the actual position and the expected position exceeds a preset threshold, updating the multimodal Prompt based on the deviation to generate new layout information.

3. The method according to claim 1, further comprising: when the user instruction does not include the expected position of the target object, predicting the expected position of the target object based on the original image and the user instruction; verifying the actual position of the target object in the original image based on the layout information; in response to determining that the deviation between the actual position and the expected position exceeds a preset threshold, updating the multimodal Prompt based on the deviation to generate new layout information.

4. The method according to claim 3, wherein, The predicting the expected position of the target object based on the original image and the user instruction includes: using an image segmentation algorithm to determine the types and occupied spatial region sizes of existing objects in the original image; determining a reasonable position of the target object in the original image according to the types and occupied spatial region sizes of the existing objects and the type of the target object in the user instruction; determining the expected position according to the reasonable position.

5. The method according to claim 2 or 3, wherein, The updating the multimodal Prompt based on the deviation includes: determining first target parameter information of the target object based on the deviation; updating the multimodal Prompt based on the first target parameter information.

6. The method according to claim 2 or 3, wherein The updating the multimodal Prompt based on the deviation includes: determining second target parameter information of existing objects in the original image based on the deviation; updating the multimodal Prompt based on the second target parameter information.

7. The method according to claim 2 or 3, wherein The method further comprises: dynamically adjusting the preset threshold according to the application scenario of the original image and the type of the target object.

8. The method according to claim 1, wherein The target object includes a first object. When the user instruction includes deleting the first object, constructing a multimodal Prompt based on the natural language description and the user instruction includes: Removing the structured description of the first object and its corresponding spatial relation encoding from the natural language description to obtain an updated natural language description; Marking the deletion action for the first object in the user instruction and associating it with the updated natural language description to obtain the multimodal Prompt.

9. The method according to claim 1, wherein, The target object includes a second object. When the user instruction includes adding the second object, constructing a multimodal Prompt based on the natural language description and the user instruction includes: Analyzing and extracting the existing objects and their spatial relations in the natural language description; Determining the insertion position of the second object based on the existing objects and their spatial relations; Generating a relative position description of the second object and the existing objects based on the insertion position of the second object; Associating the adding action in the user instruction with the relative position description to obtain the multimodal Prompt.

10. The method according to claim 1, wherein Constructing the multimodal Prompt includes: Mapping the natural language description and the user instruction to the same semantic space through a semantic alignment algorithm to generate a joint representation; Converting the joint representation into the multimodal Prompt through a predefined template.

11. The method according to claim 10, wherein, The predefined template includes: The information of the existing objects in the original image; The association information between the user instruction and the target object.

12. The method according to claim 1, wherein, Constructing the multimodal Prompt includes: Associating the action in the user instruction with the target object in the natural language description through a dynamic binding method; Generating the multimodal Prompt according to the association result.

13. A layout generation device based on a large model, including: An extraction module, configured to extract the visual information of the original image through a target detection model and a semantic segmentation model; A conversion module, configured to convert the visual information into structured data and convert the structured data into a natural language description by using a spatial relation encoding, where the spatial relation encoding is used to convert the coordinates, dimensions, and relative positions of elements into the natural language description; A construction module, configured to construct a multimodal Prompt based on the natural language description and the user instruction; where the user instruction includes an action for the original image and a target object corresponding to the action, and the multimodal Prompt is used to associate the natural language description with the user instruction; A generation module, configured to input the constructed multimodal Prompt into the large model to obtain the layout information of the target object in the original image output by the large model, where the layout information includes the position and size of the target object in the original image.

14. An electronic device, including: At least one processor; And A memory communicatively connected to at least one processor; where, The memory stores instructions executable by at least one processor, and the instructions are executed by at least one processor to enable the at least one processor to execute the method of any one of claims 1-12.

15. A non-transitory computer-readable storage medium storing computer instructions, wherein, Computer instructions are used to cause a computer to execute the method of any one of claims 1-12.

16. A computer program product, comprising a computer program stored on a storage medium, the computer program implementing the method of any one of claims 1-12 when executed by a processor.

Images