Image generation method and device and electronic equipment

Abstract

The invention discloses an image generation method and device and electronic equipment. The method relates to the field of artificial intelligence and comprises the steps that a target base map containing a target object is acquired, and the transparency of the background of the target base map is preset transparency; performing feature calculation processing on the target base map to obtain image feature information of the target base map; multiple pieces of target sample data are determined from multiple pieces of sample data of a sample database according to the image feature information of the target base map, the sample database is used for recommending model input conditions needed for generating the scene map, and the multiple pieces of target sample data are sample data with inter-class deviation larger than a preset threshold value in the multiple pieces of sample data; and inputting the plurality of pieces of target sample data and the target base map into a generative model to obtain a plurality of scene maps containing the target object. According to the method and the device, the technical problems of difficulty in manually constructing model input conditions and serious homogenization of the generated scene graph when the object scene graph is generated in related technologies are solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and more specifically, to an image generation method, device and electronic device. Background Art

[0002] With the continuous development of technology, merchants can use artificial intelligence models to generate scene graphs for products. In the process of generating images of generative models, the model input conditions will directly affect the model output effect. However, when generating product scene graphs in the existing technology, users are usually unable to accurately input the reasonable conditions required by the model, and it is difficult to manually construct model input conditions. The default configuration output is not beautiful and has serious homogeneity.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide an image generation method, device and electronic device to at least solve the technical problems in the related art of generating object scene graphs, such as difficulty in manually constructing model input conditions and serious homogeneity of the generated scene graphs.

[0005] According to one aspect of an embodiment of the present application, there is provided an image generation method, comprising: obtaining a target base map containing a target object, wherein the transparency of the background of the target base map is a preset transparency; performing feature calculation processing on the target base map to obtain image feature information of the target base map; determining multiple target sample data from multiple sample data in a sample database based on the image feature information of the target base map, wherein the sample database is used to recommend model input conditions required for generating a scene graph, and the multiple target sample data are sample data in which the inter-class deviation of the multiple sample data is greater than a preset threshold; and inputting the multiple target sample data and the target base map into a generative model to obtain multiple scene graphs containing the target object.

[0006] Furthermore, the image feature information of the target base map at least includes a color feature histogram, and feature calculation processing is performed on the target base map to obtain the image feature information of the target base map, including: dividing the color space of the target base map into multiple color intervals; counting the number of pixels in the color intervals of the multiple color intervals, and drawing the image based on the number of pixels in the color intervals of the multiple color intervals to obtain the color feature histogram of the target base map.

[0007] Furthermore, the image feature information of the target base map also includes visual features, and performing feature calculation processing on the target base map to obtain the image feature information of the target base map includes: performing feature extraction on the target base map through a visual model to obtain the visual features of the target base map.

[0008] Furthermore, determining multiple target sample data from multiple sample data in a sample database based on image feature information of the target base map includes: determining multiple first sample data from the multiple sample data based on a color feature histogram of the target base map, and determining multiple second sample data from the multiple sample data based on visual features of the target base map; calculating the inter-class deviation between the multiple first sample data and the multiple second sample data, and taking the sample data whose inter-class deviation is greater than a preset threshold as the multiple target sample data.

[0009] Furthermore, based on the color feature histogram of the target base map, determining multiple first sample data from multiple sample data includes: performing similarity calculation on the color feature histogram of the target base map and the color feature histogram of the multiple sample data to obtain a first calculation result; selecting sample data whose similarity is greater than a first threshold from the first calculation result to obtain multiple first sample data.

[0010] Furthermore, based on the visual features of the target base map, determining multiple second sample data from multiple sample data includes: calculating the similarity between the visual features of the target base map and the visual features of the multiple sample data to obtain a second calculation result; selecting sample data with a similarity greater than a second threshold from the second calculation result to obtain multiple second sample data.

[0011] Furthermore, multiple target sample data and a target base map are input into the generative model to obtain multiple scene graphs containing target objects, including: respectively taking the scene graph style information of the target sample data, the spatial information of the object, the background content description information and the object layout information in the multiple target sample data as model input conditions to obtain multiple groups of model input conditions; respectively inputting the model input conditions and the target base map in the multiple groups of model input conditions into the generative model to obtain multiple scene graphs containing target objects.

[0012] Furthermore, a sample database is constructed by the following steps: obtaining a sample scene image set; determining a first base map containing sample objects corresponding to a sample scene graph in the sample scene image set, image feature information of the first base map, scene graph style information corresponding to the sample scene graph and spatial information where the objects are located, background content description information corresponding to the sample scene graph, and object layout information corresponding to the sample scene graph, wherein the transparency of the background of the first base map is a preset transparency, and the object layout information is used to characterize the position of the sample objects; generating a piece of sample data in the sample database based on the sample scene graph, the first base map, the image feature information of the first base map, the scene graph style information, the spatial information where the objects are located, the background content description information, and the object layout information, and generating the sample database based on multiple pieces of sample data.

[0013] Furthermore, determining the first base map containing sample objects corresponding to the sample scene graph in the sample scene image set, the image feature information of the first base map, the scene graph style information corresponding to the sample scene graph and the spatial information of the object, the background content description information corresponding to the sample scene graph and the object layout information corresponding to the sample scene graph includes: using an image processing network to perform cutout processing on the sample scene graph to obtain the first base map; performing feature calculation processing on the first base map to obtain the image feature information of the first base map; performing feature extraction on the sample scene graph through a visual model to obtain the visual features of the sample scene graph, and inputting the visual features of the sample scene graph into a multi-layer perceptron for classification to obtain the scene graph style information corresponding to the sample scene graph and the spatial information of the object; processing the sample scene graph through a target multimodal large model to obtain the background content description information corresponding to the sample scene graph, wherein the target multimodal large model is obtained by model distillation based on the training sample set and the large language model; obtaining the alpha channel information of the first base map, and determining the object layout information corresponding to the sample scene graph based on the alpha channel information.

[0014] Furthermore, before processing the sample scene graph through the target multimodal large model to obtain background content description information corresponding to the sample scene graph, the method also includes: acquiring a first scene image set; determining a training sample set based on the first scene image set and the large language model; performing model distillation based on the training sample set, the large language model and the initial multimodal large model to obtain the target multimodal large model.

[0015] Furthermore, determining a training sample set based on the first scene image set and the large language model includes: performing processing by the large language model based on the first prompt information and the first scene graph in the first scene image set to obtain scene graph description information corresponding to the first scene graph, wherein the first prompt information is used to guide the large language model to describe the scene content of the first scene graph; performing image segmentation processing on the first scene graph to obtain an image of an occluded foreground object corresponding to the first scene graph; performing processing by the large language model based on the second prompt information, the scene graph description information corresponding to the first scene graph, and the image of an occluded foreground object corresponding to the first scene graph to obtain background content description information corresponding to the first scene graph, wherein the second prompt information is used to guide the large language model to describe the background content of the image of an occluded foreground object; forming a training sample image pair based on the occluded foreground object image corresponding to the first scene graph and the background content description information corresponding to the first scene graph, and determining a training sample set based on multiple training sample image pairs.

[0016] Furthermore, determining a training sample set based on multiple training sample image pairs includes: dividing an image in which a foreground object in a first training sample image pair is occluded into multiple regions, and calculating correlation coefficients between regions in the multiple regions and background content description information corresponding to the regions, to obtain calculation results corresponding to the regions, wherein the first training sample image pair is any one of the multiple training sample image pairs; making a judgment based on the calculation results corresponding to the regions, if there is a region with a correlation coefficient less than a third threshold, filtering out the first training sample image pair to obtain a filtered training sample image pair, and generating a training sample set based on the filtered training sample image pair.

[0017] According to another aspect of an embodiment of the present application, an image generation method is also provided, including: obtaining a target base map containing a target object uploaded by a client, wherein the transparency of the background of the target base map is a preset transparency; performing feature calculation processing on the target base map in a cloud server to obtain image feature information of the target base map; determining multiple target sample data from multiple sample data in a sample database based on the image feature information of the target base map, wherein the sample database is used to recommend model input conditions required for generating a scene graph, and the multiple target sample data are sample data in which the inter-class deviation of the multiple sample data is greater than a preset threshold; inputting the multiple target sample data and the target base map into a generative model to obtain multiple scene graphs containing the target object; and feeding back the multiple scene graphs to the client.

[0018] According to another aspect of an embodiment of the present application, an image generating device is also provided, including: a first acquisition unit, used to acquire a target base map containing a target object, wherein the transparency of the background of the target base map is a preset transparency; a first processing unit, used to perform feature calculation processing on the target base map to obtain image feature information of the target base map; a first determination unit, used to determine a plurality of target sample data from a plurality of sample data in a sample database based on the image feature information of the target base map, wherein the sample database is used to recommend model input conditions required for generating a scene graph, and the plurality of target sample data are sample data in which the inter-class deviation of the plurality of sample data is greater than a preset threshold; a second processing unit, used to input the plurality of target sample data and the target base map into a generative model to obtain a plurality of scene graphs containing the target object.

[0019] Furthermore, the first processing unit includes: a first processing sub-unit, used to divide the color space of the target base map into multiple color intervals; a second processing sub-unit, used to count the number of pixels in the color intervals in the multiple color intervals, and draw the image based on the number of pixels in the color intervals in the multiple color intervals to obtain a color feature histogram of the target base map.

[0020] Furthermore, the first processing unit includes: a third processing sub-unit, which is used to extract features of the target base map through a visual model to obtain visual features of the target base map.

[0021] Furthermore, the first determination unit includes: a first determination subunit, used to determine multiple first sample data from multiple sample data based on the color feature histogram of the target base map, and to determine multiple second sample data from the multiple sample data based on the visual features of the target base map; a second determination subunit, used to calculate the inter-class deviation between the multiple first sample data and the multiple second sample data, and to use the sample data whose inter-class deviation is greater than a preset threshold as the multiple target sample data.

[0022] Furthermore, the first determination subunit includes: a first calculation module, used to calculate the similarity between the color feature histogram of the target base map and the color feature histogram of multiple sample data to obtain a first calculation result; a first determination module, used to select sample data with a similarity greater than a first threshold from the first calculation result to obtain multiple first sample data.

[0023] Furthermore, the first determination subunit also includes: a second calculation module, used to calculate the similarity between the visual features of the target base map and the visual features of multiple sample data to obtain a second calculation result; a second determination module, used to select sample data with a similarity greater than a second threshold from the second calculation result to obtain multiple second sample data.

[0024] Furthermore, the second processing unit includes: a third determination subunit, used to respectively use the scene graph style information, spatial information of the object, background content description information and object layout information in the target sample data in the multiple target sample data as model input conditions to obtain multiple groups of model input conditions; a fourth processing subunit, used to respectively input the model input conditions and the target base map in the multiple groups of model input conditions into the generative model to obtain multiple scene graphs containing the target objects.

[0025] Furthermore, the device also includes the following units, which are used to construct a sample database through the following steps: a second acquisition unit, which is used to acquire a sample scene image set; a second determination unit, which is used to determine the first base map containing the sample object corresponding to the sample scene graph in the sample scene image set, the image feature information of the first base map, the scene graph style information corresponding to the sample scene graph and the spatial information where the object is located, the background content description information corresponding to the sample scene graph, and the object layout information corresponding to the sample scene graph, wherein the transparency of the background of the first base map is a preset transparency, and the object layout information is used to characterize the position of the sample object; a generation unit, which is used to generate a piece of sample data in the sample database based on the sample scene graph, the first base map, the image feature information of the first base map, the scene graph style information, the spatial information where the object is located, the background content description information and the object layout information, and generate the sample database based on multiple pieces of sample data.

[0026] Furthermore, the second determination unit includes: a fifth processing subunit, which is used to perform cutout processing on the sample scene graph using an image processing network to obtain a first base map; a sixth processing subunit, which is used to perform feature calculation processing on the first base map to obtain image feature information of the first base map; a seventh processing subunit, which is used to perform feature extraction on the sample scene graph through a visual model to obtain visual features of the sample scene graph, and input the visual features of the sample scene graph into a multi-layer perceptron for classification to obtain scene graph style information corresponding to the sample scene graph and spatial information of the object; an eighth processing subunit, which is used to process the sample scene graph through a target multimodal large model to obtain background content description information corresponding to the sample scene graph, wherein the target multimodal large model is obtained by model distillation based on a training sample set and a large language model; a fourth determination subunit, which is used to obtain alpha channel information of the first base map, and determine object layout information corresponding to the sample scene graph based on the alpha channel information.

[0027] Furthermore, the device also includes: a third acquisition unit, used to acquire a first scene image set before processing the sample scene graph through the target multimodal large model to obtain background content description information corresponding to the sample scene graph; a third determination unit, used to determine a training sample set based on the first scene image set and the large language model; a third processing unit, used to perform model distillation based on the training sample set, the large language model and the initial multimodal large model to obtain a target multimodal large model.

[0028] Furthermore, the third determination unit includes: a ninth processing subunit, which is used to process the first scene graph in the first scene image set according to the first prompt information through the large language model to obtain scene graph description information corresponding to the first scene graph, wherein the first prompt information is used to guide the large language model to describe the scene content of the first scene graph; a tenth processing subunit, which is used to perform image segmentation processing on the first scene graph to obtain an image in which the foreground object corresponding to the first scene graph is occluded; an eleventh processing subunit, which is used to process the first scene graph according to the second prompt information, the scene graph description information corresponding to the first scene graph, and the image in which the foreground object corresponding to the first scene graph is occluded through the large language model to obtain background content description information corresponding to the first scene graph, wherein the second prompt information is used to guide the large language model to describe the background content of the image in which the foreground object is occluded; a fifth determination subunit, which is used to form a training sample image pair based on the image in which the foreground object corresponding to the first scene graph is occluded and the background content description information corresponding to the first scene graph, and determine the training sample set based on multiple training sample image pairs.

[0029] Furthermore, the fifth determination subunit includes: a third calculation module, used to divide the image in which the foreground object in the first training sample image pair is occluded into multiple regions, and calculate the correlation coefficient between the regions in the multiple regions and the background content description information corresponding to the regions, to obtain the calculation results corresponding to the regions, wherein the first training sample image pair is any one of the multiple training sample image pairs; a generation module, used to make a judgment based on the calculation results corresponding to the regions, if there is an area with a correlation coefficient less than a third threshold, then filter out the first training sample image pair to obtain a filtered training sample image pair, and generate a training sample set based on the filtered training sample image pair.

[0030] According to another aspect of an embodiment of the present invention, there is further provided an electronic device, comprising: a memory storing an executable program; and a processor for running the program, wherein any one of the above-mentioned image generation methods is executed when the program is run.

[0031] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the storage medium stores a program, wherein when the program is running, the device where the storage medium is located is controlled to execute any one of the above-mentioned image generation methods.

[0032] According to another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program or instructions, and when the computer program or instructions are executed by a processor, any one of the above-mentioned image generation methods is implemented.

[0033] In an embodiment of the present application, a target base map containing a target object is obtained, wherein the transparency of the background of the target base map is a preset transparency; feature calculation processing is performed on the target base map to obtain image feature information of the target base map; multiple target sample data are determined from multiple sample data in a sample database based on the image feature information of the target base map, wherein the sample database is used to recommend model input conditions required for generating a scene graph, and the multiple target sample data are sample data in which the inter-class deviation of the multiple sample data is greater than a preset threshold; the multiple target sample data and the target base map are input into a generative model to obtain multiple scene graphs containing the target object, and a method of constructing a product graph and a scene graph is constructed by understanding the product graph and the scene graph. An intelligent recommendation database (i.e., sample database) was built, and the retrieval enhancement generation technology was introduced into the image generation process of the generative model, which can significantly improve the model image generation effect. In the process of obtaining the model input conditions, by selecting data with large inter-class deviation from the recommended multiple groups of data, the diversity of product images is guaranteed, and the homogeneity problem is effectively solved. The purpose of improving the image output quality of the model when the user has no correct input or no input is achieved, thereby achieving the technical effect of improving the aesthetics and differentiation of product images, and further solving the technical problems of difficulty in artificially constructing model input conditions and serious homogeneity of generated scene graphs when generating object scene graphs in related technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0035] Figure 1 is a schematic diagram of a computer terminal provided according to Embodiment 1 of the present application;

[0036] Figure 2 is a flowchart of an image generation method provided according to Embodiment 1 of the present application;

[0037] Figure 3 is a schematic diagram of an optional input image provided according to Embodiment 1 of the present application;

[0038] Figure 4 It is a schematic diagram of an optional transparent base map of a commodity provided according to the first embodiment of the present application;

[0039] Figure 5 It is a schematic diagram of an optional process for generating a commodity scene graph according to the first embodiment of the present application;

[0040] Figure 6 is a schematic diagram of an optional color feature histogram provided according to Embodiment 1 of the present application;

[0041] Figure 7is a schematic diagram of an optional visual feature vector provided according to Embodiment 1 of the present application;

[0042] Figure 8 is a schematic diagram of optional commodity bounding box information provided according to the first embodiment of the present application;

[0043] Fig. 9 It is a schematic diagram of an optional target multimodal large model construction process provided according to the first embodiment of the present application;

[0044] Fig.10 is a flowchart of an image generation method provided according to Embodiment 2 of the present application;

[0045] Fig.11 is a schematic diagram of an image generating device provided according to Embodiment 3 of the present application;

[0046] Fig.12 It is a structural block diagram of an electronic device provided according to Embodiment 4 of the present application. DETAILED DESCRIPTION

[0047] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0048] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0049] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards in the relevant regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0050] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0051] Retrieval-Augmented Generation (RAG): A technique that uses information from private or proprietary data sources to assist in text generation.

[0052] Caption: Descriptive text for an image, video, or other media content.

[0053] Example 1

[0054] According to an embodiment of the present application, an image generating method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0055] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing an image generation method. Figure 1 As shown, the computer terminal (or mobile device) 10 may include a processor set 102 (the processor set 102 may include but is not limited to a processing device such as a microprocessor (Microcontroller Unit, referred to as MCU) or a programmable logic device (Field Programmable Gate Array, referred to as FPGA), and the processor set 102 may include a processor set, Figure 1102a, 102b, ..., 102n are used to illustrate), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a Universal Serial Bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0056] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuits may be a single independent processing module, or may be integrated in whole or in part into any of the other components in the computer terminal 10 (or mobile device).

[0057] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the image generation method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the above-mentioned image generation method is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0058] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0059] The display may be a touch screen type liquid crystal display that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0060] With the continuous development of technology, merchants can use artificial intelligence models to generate scene graphs for products. In the process of generating images of generative models, the model input conditions will directly affect the model output effect. However, when generating product scene graphs in the existing technology, users are usually unable to accurately input the reasonable conditions required by the model, and it is difficult to manually construct model input conditions. The default configuration output is not beautiful and has serious homogeneity.

[0061] In the above technical background, the present application provides Figure 2 The image generation method shown. Figure 2 : is a flow chart of an image generation method provided according to Embodiment 1 of the present application. The method comprises:

[0062] Step S201, obtaining a target base map containing a target object, wherein the transparency of the background of the target base map is a preset transparency.

[0063] For example, the target object may be a home furnishing product such as a sofa, and the target background image refers to a transparent background image of the product whose transparency (or simply referred to as transparency) is a preset transparency (for example, the preset transparency is 0). When the transparency value of any pixel point in the image background is 0, the pixel point presents a completely transparent visual effect.

[0064] In an optional embodiment, before obtaining the target base map containing the target object, an image input by the user is received, and it is determined whether the image input by the user is a transparent base map of the product. If the image input by the user is not a transparent base map of the product, the image input by the user is cut out to obtain a transparent base map of the product.

[0065] Figure 3 is a schematic diagram of an optional input image provided according to Embodiment 1 of the present application, Figure 4 is a schematic diagram of an optional transparent base image of a commodity provided according to the first embodiment of the present application. For example, the image generation system receives Figure 3 The input image shown in FIG. 1 is subjected to matting processing using an image processing network (e.g., a U2Net network), and the following can be obtained: Figure 4 The product main content image (i.e., the product transparent base image) containing a transparent alpha channel is shown.

[0066] Step S202: perform feature calculation processing on the target base map to obtain image feature information of the target base map.

[0067] Optionally, after obtaining the transparent background image of the product containing the product, feature calculation processing is performed. In order to avoid background interference of the image uploaded by the user, only the transparent background image of the product is used for calculation during the feature calculation process. The feature calculation mainly includes two aspects: color feature histogram and visual feature.

[0068] Step S203, determining multiple target sample data from multiple sample data in a sample database based on the image feature information of the target base map, wherein the sample database is used to recommend model input conditions required to generate a scene graph, and the multiple target sample data are sample data whose inter-class deviation is greater than a preset threshold among the multiple sample data.

[0069] Optionally, multiple target sample data can be determined from multiple sample data in the sample database based on the image feature information of the target base map (e.g., color feature histogram and visual features). For example, one sample data contains a scene image, a transparent base map of a product, a color feature histogram of the transparent base map, visual features of the transparent base map, scene map style, product space information, background content description text Caption, and product layout information. Data retrieval can be performed in the sample database based on the image feature information of the target base map. During the data retrieval process, similarity calculation is performed on the sample data based on the product images uploaded by the user and various constraints (can be unconditional, only one product image), and the recommended model input conditions can be determined. For example, samples with higher similarity can be selected as preliminary retrieval results.

[0070] Step S204: input multiple target sample data and target base map into the generative model to obtain multiple scene graphs containing target objects.

[0071] Optionally, multiple target sample data and a transparent background image of the product are input into a generative model (eg, a StableDiffusion model) to obtain multiple scene graphs containing the target object.

[0072] In this solution, through the understanding of product images and scene graphs, an intelligent recommendation database (i.e., sample database) is constructed, and the retrieval enhancement generation technology is introduced into the image generation process of the generative model, which can significantly improve the model image generation effect. In the process of obtaining the model input conditions, by selecting data with large inter-class deviations from the recommended multiple groups of data, the diversity of product images is guaranteed, and the homogeneity problem is effectively solved, so as to achieve the purpose of improving the image output quality of the model when the user has no correct input or no input, thereby achieving the technical effect of improving the aesthetics and differentiation of product images, and further solving the technical problems of difficulty in manually constructing model input conditions and serious homogeneity of generated scene graphs when generating object scene graphs in related technologies.

[0073] In an optional embodiment, it is possible to use Figure 5 The schematic diagram shown in FIG. 1 realizes the generation of the product scene graph. Figure 5 is a schematic diagram of the optional process of generating a commodity scene graph according to the first embodiment of the present application, such as Figure 5 As shown, first determine whether the input image is a product transparent background image. If it is not a product transparent background image, then further use product cutout to obtain the product transparent background image, and then perform product image feature calculation, that is, calculate the image feature information of the product transparent background image, and then perform data retrieval, that is, use the product image feature calculation results and other conditions entered by the user (if any) to search in the intelligent recommendation database (that is, the aforementioned sample database). After obtaining multiple sets of retrieved results, the inter-class deviation can be further calculated, and several sets of data with larger deviations can be used as model input conditions to ensure the differences between the input conditions. Then, the multiple sets of model input conditions with differences are used for product raw images respectively, so that multiple scene images containing products with reasonable layout, high aesthetics and differentiation can be obtained.

[0074] In order to obtain the color feature histogram of the transparent base image of the product, in the image generation method provided in Example 1 of the present application, the image feature information of the target base image at least includes the color feature histogram, and feature calculation processing is performed on the target base image to obtain the image feature information of the target base image, including: dividing the color space of the target base image into multiple color intervals; counting the number of pixels in the color intervals of the multiple color intervals, and drawing the image based on the number of pixels in the color intervals of the multiple color intervals to obtain the color feature histogram of the target base image.

[0075] Optionally, the image feature information of the transparent background image of the product includes at least a color feature histogram, which describes the color features of the image by counting the pixel distribution of different color channels. The color feature histogram records the frequency of occurrence of various colors in the image and can be used to represent the overall color distribution of the image.

[0076] Figure 6 It is a schematic diagram of an optional color feature histogram provided according to the first embodiment of the present application. By dividing the color space of the transparent base image of the product into multiple color intervals (bins), counting the number of pixels in the color intervals in the multiple color intervals, and drawing the image according to the number of pixels in the color intervals in the multiple color intervals, it is possible to obtain Figure 4 The color feature histogram of the transparent background image of the product shown in Figure 6 In order to compare images of different sizes without affecting the results of the images, it is usually necessary to perform normalization to calculate the relative frequency rather than the absolute number.

[0077] By counting the number of pixels in multiple color intervals and performing image drawing, the color feature histogram of the transparent background image of the product can be obtained, which provides a basis for the subsequent determination of the model input conditions.

[0078] In order to obtain the visual features of the transparent base map of the product, in the image generation method provided in Example 1 of the present application, the image feature information of the target base map also includes visual features, and feature calculation processing is performed on the target base map to obtain the image feature information of the target base map, including: extracting features of the target base map through a visual model to obtain the visual features of the target base map.

[0079] Optionally, the image feature information of the product transparent base image also includes visual features, and the visual features of the product transparent base image can be obtained by extracting features of the product transparent base image through a visual model. For example, by extracting features of the product transparent base image through a self-supervised learning visual model (Distillation with No Labels Version 2, DINOv2 for short), the DINOv2 visual features of the product transparent base image can be obtained. Figure 7 is a schematic diagram of an optional visual feature vector provided according to the first embodiment of the present application, such as Figure 7 The DINOv2 visual model is shown. Figure 4 The 257*768-dimensional feature vector extracted from the transparent background image of the product shown.

[0080] By extracting features from the transparent background image of the product through the visual model, the visual features of the transparent background image of the product can be obtained, which provides a basis for subsequently determining the model input conditions.

[0081] In order to solve the serious problem of homogeneity of the generated scene graph, in the image generation method provided in the first embodiment of the present application, multiple target sample data are determined from multiple sample data in the sample database based on the image feature information of the target base map, including: determining multiple first sample data from the multiple sample data based on the color feature histogram of the target base map, and determining multiple second sample data from the multiple sample data based on the visual features of the target base map; calculating the inter-class deviation between the multiple first sample data and the multiple second sample data, and taking the sample data with an inter-class deviation greater than a preset threshold as the multiple target sample data.

[0082] Optionally, based on the color feature histogram of the transparent background image of the product, multiple first sample data can be determined from the multiple sample data, and based on the DINOv2 visual features of the transparent background image of the product, multiple second sample data can be determined from the multiple sample data, and then the inter-class deviation between the multiple first sample data and the multiple second sample data is calculated, and the sample data with an inter-class deviation greater than a preset threshold is used as the multiple target sample data.

[0083] For example, a search is performed in the intelligent recommendation database to retrieve M complete input conditions (i.e., M pieces of data), and the inter-class deviation is further calculated among the M pieces of data, and N (M>N) pieces of data with larger deviation (i.e., multiple target sample data) are selected as model input conditions. Optionally, one piece of data is used as a model input condition (multiple dimensions) to generate a scene graph containing products.

[0084] By retrieving multiple input conditions with obvious differences, the differences between the generated scene graphs are effectively guaranteed and the differentiation between the generated scene graphs is improved.

[0085] In order to determine the model input conditions, in the image generation method provided in the first embodiment of the present application, based on the color feature histogram of the target base map, determining multiple first sample data from multiple sample data includes: performing similarity calculation on the color feature histogram of the target base map and the color feature histogram of the multiple sample data to obtain a first calculation result; selecting sample data whose similarity is greater than a first threshold from the first calculation result to obtain multiple first sample data.

[0086] Optionally, similarity matching is used for retrieval in the intelligent recommendation database. In the similarity matching process, measurement methods such as Euclidean distance and histogram cross matching can be used to determine image similarity. For example, a color feature histogram of a transparent background image of a product and a color feature histogram of a plurality of sample data are similarly calculated to obtain a plurality of similarities (i.e., a first calculation result), and then sample data having a similarity greater than a first threshold is selected from the plurality of similarities to obtain a plurality of first sample data. The first sample data represents the recommended data determined according to the color feature histogram of the transparent background image of the product.

[0087] By calculating the similarity between the color feature histogram of the transparent background image of the product and the color feature histogram of multiple sample data, multiple first sample data can be determined, which provides a basis for subsequently determining the model input conditions.

[0088] In order to determine the model input conditions, in the image generation method provided in the first embodiment of the present application, based on the visual features of the target base map, determining multiple second sample data from multiple sample data includes: calculating the similarity between the visual features of the target base map and the visual features of the multiple sample data to obtain a second calculation result; selecting sample data whose similarity is greater than a second threshold from the second calculation result to obtain multiple second sample data.

[0089] Optionally, similarity calculation is performed between the DINOv2 visual features of the transparent background image of the product and the DINOv2 visual features of the multiple sample data (i.e., similarity between feature vectors is calculated), and multiple similarities (i.e., second calculation results) are obtained, and then sample data with similarity greater than a second threshold is selected from the multiple similarities to obtain multiple second sample data. The second sample data represents the recommended data determined based on the DINOv2 visual features of the transparent background image of the product.

[0090] By calculating the similarity between the DINOv2 visual features of the transparent background image of the product and the DINOv2 visual features of multiple sample data, multiple second sample data can be determined, providing a basis for subsequent determination of model input conditions.

[0091] In order to obtain a non-homogeneous scene graph of a product, in the image generation method provided in Example 1 of the present application, multiple target sample data and a target base map are input into a generative model to obtain multiple scene graphs containing target objects, including: respectively taking the scene graph style information of the target sample data in the multiple target sample data, the spatial information of the object, the background content description information and the object layout information as model input conditions to obtain multiple groups of model input conditions; respectively inputting the model input conditions and the target base map in the multiple groups of model input conditions into the generative model to obtain multiple scene graphs containing target objects.

[0092] Optionally, after performing data retrieval in the intelligent recommendation database and determining multiple groups of recommendation data (i.e., multiple target sample data) by calculating the inter-class deviation, the scene graph style, product space information, background content description text Caption, and product layout information in the target sample data in the multiple target sample data can be used as model input conditions to obtain multiple groups of model input conditions, and then the model input conditions and the product transparent background map in the multiple groups of model input conditions are respectively input into the Stable Diffusion model to obtain multiple scene graphs containing products. Since a complete input condition is derived from the same high-quality sample scene graph, the complete rationality between the input conditions can be guaranteed.

[0093] By taking the scene graph style information, spatial information of the object, background content description information and object layout information in multiple target sample data as model input conditions, the retrieval enhanced generation technology is introduced into the image generation process of the generative model, which can significantly improve the model image generation effect, effectively solve the problem of difficulty in manually constructing model input conditions, and ensure the diversity of product images.

[0094] In order to build an intelligent recommendation database, in the image generation method provided in Example 1 of the present application, a sample database is constructed by the following steps: obtaining a sample scene image set; determining a first base map containing sample objects corresponding to a sample scene graph in the sample scene image set, image feature information of the first base map, scene graph style information corresponding to the sample scene graph and spatial information where the objects are located, background content description information corresponding to the sample scene graph, and object layout information corresponding to the sample scene graph, wherein the transparency of the background of the first base map is a preset transparency, and the object layout information is used to characterize the position of the sample objects; generating a piece of sample data in the sample database based on the sample scene graph, the first base map, the image feature information of the first base map, the scene graph style information, the spatial information where the objects are located, the background content description information, and the object layout information, and generating the sample database based on multiple pieces of sample data.

[0095] Optionally, in the process of constructing an intelligent recommendation database, a sample scene image set is first obtained, and then the first base map containing sample objects corresponding to the sample scene graph in the sample scene image set, image feature information of the first base map, scene graph style information and spatial information of the object corresponding to the sample scene graph, background content description information corresponding to the sample scene graph, and object layout information corresponding to the sample scene graph are determined.

[0096] In order to be able to parse the sample scene graph into sample data in the intelligent recommendation database, in the image generation method provided in the first embodiment of the present application, determining the first base map containing sample objects corresponding to the sample scene graph in the sample scene image set, the image feature information of the first base map, the scene graph style information corresponding to the sample scene graph and the spatial information where the object is located, the background content description information corresponding to the sample scene graph, and the object layout information corresponding to the sample scene graph include: using an image processing network to perform cutout processing on the sample scene graph to obtain the first base map; performing feature calculation processing on the first base map to obtain the image feature information of the first base map; performing feature extraction on the sample scene graph through a visual model to obtain the visual features of the sample scene graph, and inputting the visual features of the sample scene graph into a multi-layer perceptron for classification to obtain the scene graph style information corresponding to the sample scene graph and the spatial information where the object is located; processing the sample scene graph through a target multimodal large model to obtain the background content description information corresponding to the sample scene graph, wherein the target multimodal large model is obtained by model distillation based on the training sample set and the large language model; obtaining the alpha channel information of the first base map, and determining the object layout information corresponding to the sample scene graph based on the alpha channel information.

[0097] Optionally, in the process of building an intelligent recommendation database, the sample data not only needs to calculate the product graph features of the sample scene graph, but also needs to understand the sample scene graph. Figure 3Taking the input image shown in the figure as an example, this paper introduces how to parse a sample scene graph into a sample data in the intelligent recommendation database.

[0098] For example, firstly, an image processing network (e.g., a U2Net network) is used to perform a cutout process on a sample scene graph to obtain a transparent base map (i.e., a first base map) of a sample product. Then, feature calculation processing is performed on the transparent base map of the sample product to obtain a color feature histogram and DINOv2 visual features of the transparent base map of the sample product. Then, a pre-trained DINOv2 visual model is used to perform feature extraction on the sample scene graph to obtain the DINOv2 visual features of the sample scene graph. The DINOv2 visual features of the sample scene graph are input into a multi-layer perceptron for classification. The scene graph style information (e.g., logs, modern, etc.) and the spatial information (e.g., living room, bedroom, etc.) of the object corresponding to the sample scene graph can be obtained. Then, the sample scene graph is processed by a target multimodal large model obtained by model distillation based on a training sample set and a large language model (e.g., Generative Pre-Trained Transformer 4Omni, referred to as GPT-4o). The background content description information corresponding to the sample scene graph can be obtained. Then, the alpha channel information of the transparent base map of the sample product is obtained, and the object layout information corresponding to the sample scene graph is determined based on the alpha channel information.

[0099] Optionally, for different classification tasks, a small number of samples need to be manually labeled to fine-tune the classifier. When the verification accuracy reaches a certain threshold, the classifier can be used for data classification, so as to obtain the scene graph style and product space information.

[0100] Optionally, since the transparent bottom image of the product obtained by cutting out the product contains transparent channel information, the location of the product (ie, product layout information) can be obtained according to the transparent channel information. Figure 8 is a schematic diagram of optional commodity bounding box information provided according to the first embodiment of the present application, such as Figure 8 As shown, the coordinates corresponding to the product bounding box information are (0.0925, 0.46, 0.91625, 0.78), which are used to indicate the location of the product.

[0101] Optionally, a piece of sample data is generated based on the sample scene graph, the transparent background image of the sample product, the color feature histogram and DINOv2 visual features of the transparent background image of the sample product, the scene graph style, the product space information, the background content description text Caption and the product layout information. When the data volume reaches a certain scale, an intelligent recommendation database can be obtained.

[0102] By understanding product images and scene images, an intelligent recommendation database was built, which enables the introduction of retrieval enhancement generation technology into the image generation process of the generative model, which can significantly improve the model's image generation effect.

[0103] In order to obtain the target multimodal large model, in the image generation method provided in Example 1 of the present application, before processing the sample scene graph through the target multimodal large model to obtain background content description information corresponding to the sample scene graph, a first scene image set is obtained; a training sample set is determined based on the first scene image set and the large language model; and model distillation is performed based on the training sample set, the large language model and the initial multimodal large model to obtain the target multimodal large model.

[0104] Optionally, since the required Caption is mainly used for the task of filling the background of the scene graph, it can be achieved through a multimodal technology that only describes the background content of the scene graph. Using the multimodal capabilities of the GPT-4o model, a scheme for seed data production, seed data filtering, and model distillation is designed to achieve the multimodal technical capability of only describing the background content of the scene graph.

[0105] For example, before processing the sample scene graph through the target multimodal large model to obtain the background content description information corresponding to the sample scene graph, first obtain the first scene image set, then determine the training sample set based on the first scene image set and the large language model, and then perform model distillation based on the training sample set, the large language model and the initial multimodal large model to obtain the target multimodal large model.

[0106] In order to accurately determine the training sample set, in the image generation method provided in Example 1 of the present application, determining the training sample set based on the first scene image set and the large language model includes: processing by the large language model according to the first prompt information and the first scene graph in the first scene image set to obtain scene graph description information corresponding to the first scene graph, wherein the first prompt information is used to guide the large language model to describe the scene content of the first scene graph; performing image segmentation processing on the first scene graph to obtain an image of an occluded foreground object corresponding to the first scene graph; processing by the large language model according to the second prompt information, the scene graph description information corresponding to the first scene graph, and the image of the occluded foreground object corresponding to the first scene graph to obtain background content description information corresponding to the first scene graph, wherein the second prompt information is used to guide the large language model to describe the background content of the image of the occluded foreground object; forming a training sample image pair based on the occluded foreground object image corresponding to the first scene graph and the background content description information corresponding to the first scene graph, and determining the training sample set based on multiple training sample image pairs.

[0107] Optionally, the GPT-4o model is used to process the first prompt information and the first scene graph in the first scene image set to obtain scene graph description information corresponding to the first scene graph, that is, a complete and detailed description of the first scene graph, and then image segmentation processing is performed on the first scene graph to obtain an image in which the foreground object corresponding to the first scene graph is occluded, that is, an image in which the main object is occluded. Then, the GPT-4o model is used to process the second prompt information, the scene graph description information corresponding to the first scene graph, and the image in which the foreground object corresponding to the first scene graph is occluded to obtain background content description information corresponding to the first scene graph, that is, a Caption that only describes the background content of the scene graph. Then, training sample image pairs are composed based on the occluded image of the foreground object corresponding to the first scene graph and the background content description information corresponding to the first scene graph, and a training sample set is determined based on multiple training sample image pairs.

[0108] In order to accurately determine the training sample set, in the image generation method provided in the first embodiment of the present application, determining the training sample set based on multiple training sample image pairs includes: dividing the image in which the foreground object in the first training sample image pair is occluded into multiple regions, and calculating the correlation coefficient between the regions in the multiple regions and the background content description information corresponding to the regions, to obtain the calculation results corresponding to the regions, wherein the first training sample image pair is any one of the multiple training sample image pairs; judging based on the calculation results corresponding to the regions, if there is an area with a correlation coefficient less than a third threshold, filtering out the first training sample image pair to obtain the filtered training sample image pair, and generating the training sample set based on the filtered training sample image pair.

[0109] Optionally, although the background content description information corresponding to the first scene graph is obtained by inputting the scene graph detailed description, due to the increase in the conditional restrictions of the prompt word, the obtained seed data (i.e., the training sample image pair: the image of the main object being occluded and the background description) has a lower proportion of hallucinations or errors, so it is necessary to further filter the seed data. For example, the image of the foreground object being occluded in the first training sample image pair is divided into multiple regions, and the correlation coefficient between the regions in the multiple regions and the background content description information corresponding to the regions is calculated to obtain the calculation results corresponding to the regions, and then a judgment is made based on the calculation results corresponding to the regions. If there is an area with a correlation coefficient less than a third threshold, the first training sample image pair is filtered out to obtain a filtered training sample image pair, and a training sample set is generated based on the filtered training sample image pair.

[0110] In an optional embodiment, it is possible to use Fig. 9 As shown in the schematic diagram, the target multimodal large model is obtained. Fig. 9 is a schematic diagram of an optional target multimodal large model construction process provided in Example 1 of the present application, such as Fig. 9As shown in the figure, in the seed data production process, for indoor scene images, we first use the manually designed prompt word 1 and call the GPT-4o model to obtain a complete and detailed description of the scene graph. In order to further obtain a caption that only describes the background content, we use image segmentation and image processing techniques to obtain images where the main object is blocked (such as Fig. 9 The sofa shown is blocked by a black mask), and then the complete detailed description of the scene graph, prompt word 2 and the blocked image of the main object are taken as the input of the GPT-4o model to obtain a Caption that only describes the background content, and then the Caption that only describes the background content is formatted into a description of different areas around the main object, and the descriptions of different areas are respectively correlated with each area of ​​the image. If the correlation calculation result of a certain area is low, it is considered that the quality of the seed image data pair is unqualified, thereby completing the filtering of the seed data and obtaining high-quality seed data (i.e., high-quality training sample image pairs). Then, the initial multimodal large model (e.g., Bootstrapping Language-Image Pre-training, BLIP for short) can be distilled based on the high-quality seed data, that is, the scene graph background description capability of the GPT-4o model is passed to the BLIP model, and then the distilled model is evaluated, and the model with better performance is selected to obtain the target multimodal large model that only describes the background content of the scene graph.

[0111] In an embodiment of the present application, a solution for quickly generating product scene graphs is designed by utilizing the graph-generating capability of the Stable Diffusion model. By understanding product graphs and scene graphs, a high-quality intelligent recommendation database is constructed, and the user's personalized input problem is defined as a retrieval problem. The user only needs to input a product picture to obtain model input conditions such as diversified product layouts, graph-generated picture descriptions, and reference pictures, and obtain a non-homogeneous scene graph containing the product, which solves the problems of difficulty in users constructing model input conditions, unsightly output of default configurations, and serious homogeneity.

[0112] In an embodiment of the present application, a target base map containing a target object is obtained, wherein the transparency of the background of the target base map is a preset transparency; feature calculation processing is performed on the target base map to obtain image feature information of the target base map; multiple target sample data are determined from multiple sample data in a sample database based on the image feature information of the target base map, wherein the sample database is used to recommend model input conditions required for generating a scene graph, and the multiple target sample data are sample data in which the inter-class deviation of the multiple sample data is greater than a preset threshold; the multiple target sample data and the target base map are input into a generative model to obtain multiple scene graphs containing the target object, and a method of constructing a product graph and a scene graph is constructed by understanding the product graph and the scene graph. An intelligent recommendation database (i.e., sample database) was built, and the retrieval enhancement generation technology was introduced into the image generation process of the generative model, which can significantly improve the model image generation effect. In the process of obtaining the model input conditions, by selecting data with large inter-class deviation from the recommended multiple groups of data, the diversity of product images is guaranteed, and the homogeneity problem is effectively solved. The purpose of improving the image output quality of the model when the user has no correct input or no input is achieved, thereby achieving the technical effect of improving the aesthetics and differentiation of product images, and further solving the technical problems of difficulty in artificially constructing model input conditions and serious homogeneity of generated scene graphs when generating object scene graphs in related technologies.

[0113] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0114] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0115] Example 2

[0116] According to an embodiment of the present application, a method for generating an image is also provided. Fig.10 As shown, the method includes:

[0117] Step S1001, obtaining a target base map containing a target object uploaded by a client, wherein the transparency of the background of the target base map is a preset transparency;

[0118] Step S1002, performing feature calculation processing on the target base map in the cloud server to obtain image feature information of the target base map; determining multiple target sample data from multiple sample data in a sample database according to the image feature information of the target base map, wherein the sample database is used to recommend model input conditions required for generating a scene graph, and the multiple target sample data are sample data in which the inter-class deviation of the multiple sample data is greater than a preset threshold; inputting the multiple target sample data and the target base map into a generative model to obtain multiple scene graphs containing target objects;

[0119] Step S1003: Feedback multiple scene graphs to the client.

[0120] Through the above scheme, through the understanding of product images and scene graphs, an intelligent recommendation database (i.e., a sample database) is constructed, and the retrieval enhancement generation technology is introduced into the image generation process of the generative model, which can significantly improve the model image generation effect. In the process of obtaining the model input conditions, by selecting data with large inter-class deviations from the recommended multiple groups of data, the diversity of product images is guaranteed, and the homogeneity problem is effectively solved, so as to achieve the purpose of improving the image output quality of the model when the user has no correct input or no input, thereby achieving the technical effect of improving the aesthetics and differentiation of product images, and further solving the technical problems of difficulty in artificially constructing model input conditions and serious homogeneity of generated scene graphs when generating object scene graphs in related technologies.

[0121] In the cloud server, the specific method for generating the image is the same as that in the first embodiment, and will not be described again here.

[0122] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0123] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0124] Example 3

[0125] According to an embodiment of the present application, an image generating device for implementing the above-mentioned image generating method is also provided. Fig.11 As shown, the device includes: a first acquisition unit 1101, a first processing unit 1102, a first determination unit 1103, and a second processing unit 1104.

[0126] The first acquisition unit 1101 is used to acquire a target base map containing a target object, wherein the transparency of the background of the target base map is a preset transparency;

[0127] The first processing unit 1102 is used to perform feature calculation processing on the target base map to obtain image feature information of the target base map;

[0128] A first determining unit 1103 is used to determine multiple target sample data from multiple sample data in a sample database according to image feature information of the target base map, wherein the sample database is used to recommend model input conditions required for generating a scene graph, and the multiple target sample data are sample data whose inter-class deviation is greater than a preset threshold value among the multiple sample data;

[0129] The second processing unit 1104 is used to input multiple target sample data and the target base map into the generative model to obtain multiple scene graphs containing the target objects.

[0130] In the image generation device provided in Example 3 of the present application, a target base map containing a target object is obtained by a first acquisition unit 1101, wherein the transparency of the background of the target base map is a preset transparency; a first processing unit 1102 performs feature calculation processing on the target base map to obtain image feature information of the target base map; a first determination unit 1103 determines a plurality of target sample data from a plurality of sample data in a sample database based on the image feature information of the target base map, wherein the sample database is used to recommend model input conditions required for generating a scene graph, and the plurality of target sample data are sample data in which the inter-class deviation of the plurality of sample data is greater than a preset threshold; a second processing unit 1104 inputs the plurality of target sample data and the target base map into a generative model to obtain a plurality of scene graphs containing the target object. In this solution, through the understanding of product images and scene graphs, an intelligent recommendation database (i.e., sample database) is constructed, and the retrieval enhancement generation technology is introduced into the image generation process of the generative model, which can significantly improve the model image generation effect. In the process of obtaining the model input conditions, by selecting data with large inter-class deviations from the recommended multiple groups of data, the diversity of product images is guaranteed, and the homogeneity problem is effectively solved, so as to achieve the purpose of improving the image output quality of the model when the user has no correct input or no input, thereby achieving the technical effect of improving the aesthetics and differentiation of product images, and further solving the technical problems of difficulty in manually constructing model input conditions and serious homogeneity of generated scene graphs when generating object scene graphs in related technologies.

[0131] Optionally, in the image generating device provided in Example 3 of the present application, the first processing unit 1102 includes: a first processing sub-unit, used to divide the color space of the target base map into multiple color intervals; a second processing sub-unit, used to count the number of pixels in a color interval in the multiple color intervals, and draw an image based on the number of pixels in the color interval in the multiple color intervals to obtain a color feature histogram of the target base map.

[0132] Optionally, in the image generating device provided in the third embodiment of the present application, the first processing unit 1102 includes: a third processing sub-unit, configured to extract features of the target base map through a visual model to obtain visual features of the target base map.

[0133] Optionally, in the image generating device provided in Example 3 of the present application, the first determination unit 1103 includes: a first determination subunit, used to determine multiple first sample data from multiple sample data based on a color feature histogram of the target base map, and to determine multiple second sample data from the multiple sample data based on visual features of the target base map; a second determination subunit, used to calculate the inter-class deviation between the multiple first sample data and the multiple second sample data, and to use the sample data whose inter-class deviation is greater than a preset threshold as the multiple target sample data.

[0134] Optionally, in the image generating device provided in Example 3 of the present application, the first determination subunit includes: a first calculation module, used to calculate the similarity between the color feature histogram of the target base map and the color feature histogram of multiple sample data to obtain a first calculation result; a first determination module, used to select sample data whose similarity is greater than a first threshold from the first calculation result to obtain multiple first sample data.

[0135] Optionally, in the image generating device provided in Example 3 of the present application, the first determination subunit also includes: a second calculation module, used to calculate the similarity between the visual features of the target base map and the visual features of multiple sample data to obtain a second calculation result; a second determination module, used to select sample data whose similarity is greater than a second threshold from the second calculation result to obtain multiple second sample data.

[0136] Optionally, in the image generation device provided in Example 3 of the present application, the second processing unit 1104 includes: a third determination subunit, used to respectively use scene graph style information, spatial information of the object, background content description information, and object layout information in the target sample data among multiple target sample data as model input conditions to obtain multiple groups of model input conditions; a fourth processing subunit, used to respectively input the model input conditions and the target base map in the multiple groups of model input conditions into the generative model to obtain multiple scene graphs containing target objects.

[0137] Optionally, in the image generation device provided in Example 3 of the present application, the device also includes the following units, which are used to construct a sample database through the following steps: a second acquisition unit, used to acquire a sample scene image set; a second determination unit, used to determine the first base map containing the sample object corresponding to the sample scene graph in the sample scene image set, the image feature information of the first base map, the scene graph style information corresponding to the sample scene graph and the spatial information where the object is located, the background content description information corresponding to the sample scene graph, and the object layout information corresponding to the sample scene graph, wherein the transparency of the background of the first base map is a preset transparency, and the object layout information is used to characterize the position of the sample object; a generation unit, used to generate a piece of sample data in the sample database based on the sample scene graph, the first base map, the image feature information of the first base map, the scene graph style information, the spatial information where the object is located, the background content description information and the object layout information, and generate the sample database based on multiple pieces of sample data.

[0138] Optionally, in the image generation device provided in Example 3 of the present application, the second determination unit includes: a fifth processing subunit, which is used to use an image processing network to perform cutout processing on the sample scene graph to obtain a first base map; a sixth processing subunit, which is used to perform feature calculation processing on the first base map to obtain image feature information of the first base map; a seventh processing subunit, which is used to extract features of the sample scene graph through a visual model to obtain visual features of the sample scene graph, and input the visual features of the sample scene graph into a multi-layer perceptron for classification to obtain scene graph style information corresponding to the sample scene graph and spatial information of the object; an eighth processing subunit, which is used to process the sample scene graph through a target multimodal large model to obtain background content description information corresponding to the sample scene graph, wherein the target multimodal large model is obtained by model distillation based on a training sample set and a large language model; a fourth determination subunit, which is used to obtain alpha channel information of the first base map, and determine object layout information corresponding to the sample scene graph based on the alpha channel information.

[0139] Optionally, in the image generation device provided in Example 3 of the present application, the device also includes: a third acquisition unit, used to acquire a first scene image set before processing the sample scene graph through the target multimodal large model to obtain background content description information corresponding to the sample scene graph; a third determination unit, used to determine a training sample set based on the first scene image set and the large language model; a third processing unit, used to perform model distillation based on the training sample set, the large language model and the initial multimodal large model to obtain a target multimodal large model.

[0140] Optionally, in the image generation device provided in Example 3 of the present application, the third determination unit includes: a ninth processing subunit, used to perform processing based on the first prompt information and the first scene graph in the first scene image set through a large language model to obtain scene graph description information corresponding to the first scene graph, wherein the first prompt information is used to guide the large language model to describe the scene content of the first scene graph; a tenth processing subunit, used to perform image segmentation processing on the first scene graph to obtain an image in which a foreground object corresponding to the first scene graph is occluded; an eleventh processing subunit, used to perform processing based on the second prompt information, the scene graph description information corresponding to the first scene graph, and the image in which the foreground object corresponding to the first scene graph is occluded through a large language model to obtain background content description information corresponding to the first scene graph, wherein the second prompt information is used to guide the large language model to describe the background content of the image in which the foreground object is occluded; a fifth determination subunit, used to form a training sample image pair based on the image in which the foreground object corresponding to the first scene graph is occluded and the background content description information corresponding to the first scene graph, and determine a training sample set based on multiple training sample image pairs.

[0141] Optionally, in the image generation device provided in Example 3 of the present application, the fifth determination subunit includes: a third calculation module, used to divide the image in which the foreground object in the first training sample image pair is occluded into multiple regions, and calculate the correlation coefficient between the regions among the multiple regions and the background content description information corresponding to the regions, to obtain the calculation results corresponding to the regions, wherein the first training sample image pair is any one of the multiple training sample image pairs; a generation module, used to make a judgment based on the calculation results corresponding to the regions, and if there is an area with a correlation coefficient less than a third threshold, filter out the first training sample image pair to obtain a filtered training sample image pair, and generate a training sample set based on the filtered training sample image pair.

[0142] It should be noted that the above-mentioned first acquisition unit 1101, first processing unit 1102, first determination unit 1103, and second processing unit 1104 correspond to steps S201 to S204 in Embodiment 1, and the above-mentioned units and corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Embodiment 1. It should be noted that the above-mentioned modules as part of the device can be run in the computer terminal 10 provided in Embodiment 1.

[0143] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0144] Example 4

[0145] The embodiment of the present application may provide an electronic device, which may be any electronic device in a group of electronic devices. Optionally, in this embodiment, the electronic device may also be replaced by a terminal device such as a mobile terminal.

[0146] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0147] In this embodiment, the above-mentioned electronic device can execute the program code of the following steps in the image generation method: obtaining a target base map containing a target object, wherein the transparency of the background of the target base map is a preset transparency; performing feature calculation processing on the target base map to obtain image feature information of the target base map; determining multiple target sample data from multiple sample data in a sample database based on the image feature information of the target base map, wherein the sample database is used to recommend model input conditions required to generate a scene graph, and the multiple target sample data are sample data in which the inter-class deviation of the multiple sample data is greater than a preset threshold; inputting the multiple target sample data and the target base map into a generative model to obtain multiple scene graphs containing the target object.

[0148] The above-mentioned electronic device can also execute the program code of the following steps in the image generation method: dividing the color space of the target base map into multiple color intervals; counting the number of pixels in the color intervals in the multiple color intervals, and drawing the image based on the number of pixels in the color intervals in the multiple color intervals to obtain a color feature histogram of the target base map.

[0149] The electronic device may also execute the program code of the following steps in the image generation method: extracting features of the target base map through a visual model to obtain visual features of the target base map.

[0150] The electronic device can also execute the program code of the following steps in the image generation method: determining multiple first sample data from multiple sample data based on the color feature histogram of the target base map, and determining multiple second sample data from multiple sample data based on the visual features of the target base map; calculating the inter-class deviation between the multiple first sample data and the multiple second sample data, and taking the sample data whose inter-class deviation is greater than a preset threshold as the multiple target sample data.

[0151] The above-mentioned electronic device can also execute the program code of the following steps in the image generation method: calculate the similarity between the color feature histogram of the target base map and the color feature histogram of multiple sample data to obtain a first calculation result; select sample data with a similarity greater than a first threshold from the first calculation result to obtain multiple first sample data.

[0152] The electronic device can also execute the program code of the following steps in the image generation method: calculating the similarity between the visual features of the target base map and the visual features of multiple sample data to obtain a second calculation result; selecting sample data with a similarity greater than a second threshold from the second calculation result to obtain multiple second sample data.

[0153] The above-mentioned electronic device can also execute the program code of the following steps in the image generation method: respectively taking the scene graph style information, the spatial information of the object, the background content description information and the object layout information in the target sample data in multiple target sample data as model input conditions to obtain multiple groups of model input conditions; respectively inputting the model input conditions and the target base map in the multiple groups of model input conditions into the generative model to obtain multiple scene graphs containing the target objects.

[0154] The above-mentioned electronic device can also execute the program code of the following steps in the image generation method: obtaining a sample scene image set; determining the first base map containing the sample object corresponding to the sample scene graph in the sample scene image set, the image feature information of the first base map, the scene graph style information corresponding to the sample scene graph and the spatial information where the object is located, the background content description information corresponding to the sample scene graph and the object layout information corresponding to the sample scene graph, wherein the transparency of the background of the first base map is a preset transparency, and the object layout information is used to characterize the position of the sample object; generating a sample data in the sample database based on the sample scene graph, the first base map, the image feature information of the first base map, the scene graph style information, the spatial information where the object is located, the background content description information and the object layout information, and generating the sample database based on multiple sample data.

[0155] The electronic device can also execute the program code of the following steps in the image generation method: using an image processing network to perform cutout processing on the sample scene graph to obtain a first base map; performing feature calculation processing on the first base map to obtain image feature information of the first base map; performing feature extraction on the sample scene graph through a visual model to obtain visual features of the sample scene graph, and inputting the visual features of the sample scene graph into a multi-layer perceptron for classification to obtain scene graph style information corresponding to the sample scene graph and spatial information of the object; processing the sample scene graph through a target multimodal large model to obtain background content description information corresponding to the sample scene graph, wherein the target multimodal large model is obtained by model distillation based on a training sample set and a large language model; obtaining alpha channel information of the first base map, and determining object layout information corresponding to the sample scene graph based on the alpha channel information.

[0156] The above-mentioned electronic device can also execute the program code of the following steps in the image generation method: before processing the sample scene graph through the target multimodal large model to obtain background content description information corresponding to the sample scene graph, obtaining a first scene image set; determining a training sample set based on the first scene image set and the large language model; performing model distillation based on the training sample set, the large language model and the initial multimodal large model to obtain the target multimodal large model.

[0157] The electronic device can also execute the program code of the following steps in the image generation method: using a large language model to process the first prompt information and the first scene graph in the first scene image set to obtain scene graph description information corresponding to the first scene graph, wherein the first prompt information is used to guide the large language model to describe the scene content of the first scene graph; performing image segmentation processing on the first scene graph to obtain an image of an occluded foreground object corresponding to the first scene graph; using a large language model to process the second prompt information, the scene graph description information corresponding to the first scene graph, and the image of an occluded foreground object corresponding to the first scene graph to obtain background content description information corresponding to the first scene graph, wherein the second prompt information is used to guide the large language model to describe the background content of the image of an occluded foreground object; forming a training sample image pair based on the occluded foreground object corresponding to the first scene graph and the background content description information corresponding to the first scene graph, and determining a training sample set based on multiple training sample image pairs.

[0158] The electronic device can also execute the program code of the following steps in the image generation method: dividing the image in which the foreground object in the first training sample image pair is obscured into multiple regions, and calculating the correlation coefficient between the regions in the multiple regions and the background content description information corresponding to the regions, to obtain the calculation results corresponding to the regions, wherein the first training sample image pair is any one of the multiple training sample image pairs; judging based on the calculation results corresponding to the regions, if there is a region with a correlation coefficient less than a third threshold, filtering out the first training sample image pair to obtain a filtered training sample image pair, and generating a training sample set based on the filtered training sample image pair.

[0159] Optionally, Fig.12 is a structural block diagram of an electronic device according to an embodiment of the present application. Fig.12 As shown, the electronic device 120 may include: one or more ( Fig.12 Only one is shown in the figure) processor 1202, memory 1204. The electronic device 120 may also include a storage controller, through which the memory 1204 is controlled and managed; the electronic device 120 may also include a peripheral interface, through which the radio frequency module, audio module and display screen are connected.

[0160] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the image generation method and device in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned image generation method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0161] The processor can call the information and application stored in the memory through the transmission device to execute the following steps: obtain a target base map containing the target object, wherein the transparency of the background of the target base map is a preset transparency; perform feature calculation processing on the target base map to obtain image feature information of the target base map; determine multiple target sample data from multiple sample data in a sample database based on the image feature information of the target base map, wherein the sample database is used to recommend model input conditions required for generating a scene graph, and the multiple target sample data are sample data in which the inter-class deviation of the multiple sample data is greater than a preset threshold; input the multiple target sample data and the target base map into a generative model to obtain multiple scene graphs containing the target object.

[0162] Optionally, the processor may also execute the program code of the following steps: dividing the color space of the target base map into multiple color intervals; counting the number of pixels in the color intervals in the multiple color intervals, and drawing an image based on the number of pixels in the color intervals in the multiple color intervals to obtain a color feature histogram of the target base map.

[0163] Optionally, the processor may also execute program code of the following steps: extracting features of the target base map through a visual model to obtain visual features of the target base map.

[0164] Optionally, the processor may also execute the program code of the following steps: determining multiple first sample data from multiple sample data based on the color feature histogram of the target base map, and determining multiple second sample data from multiple sample data based on the visual features of the target base map; calculating the inter-class deviation between the multiple first sample data and the multiple second sample data, and taking the sample data whose inter-class deviation is greater than a preset threshold as multiple target sample data.

[0165] Optionally, the processor may also execute the program code of the following steps: performing similarity calculation on the color feature histogram of the target base map and the color feature histogram of multiple sample data to obtain a first calculation result; and selecting sample data having a similarity greater than a first threshold from the first calculation result to obtain multiple first sample data.

[0166] Optionally, the processor may also execute program code of the following steps: performing similarity calculation between the visual features of the target base map and the visual features of multiple sample data to obtain a second calculation result; and selecting sample data having a similarity greater than a second threshold from the second calculation result to obtain multiple second sample data.

[0167] Optionally, the processor may also execute the program code of the following steps: respectively using scene graph style information, spatial information of the object, background content description information and object layout information in multiple target sample data as model input conditions to obtain multiple sets of model input conditions; respectively inputting the model input conditions and the target base map in the multiple sets of model input conditions into the generative model to obtain multiple scene graphs containing the target objects.

[0168] Optionally, the processor may also execute the program code of the following steps: obtaining a sample scene image set; determining a first base map containing sample objects corresponding to a sample scene graph in the sample scene image set, image feature information of the first base map, scene graph style information corresponding to the sample scene graph and spatial information where the objects are located, background content description information corresponding to the sample scene graph, and object layout information corresponding to the sample scene graph, wherein the transparency of the background of the first base map is a preset transparency, and the object layout information is used to characterize the position of the sample objects; generating a piece of sample data in a sample database based on the sample scene graph, the first base map, the image feature information of the first base map, the scene graph style information, the spatial information where the objects are located, the background content description information, and the object layout information, and generating a sample database based on multiple pieces of sample data.

[0169] Optionally, the processor may also execute the program code of the following steps: performing cutout processing on the sample scene graph using an image processing network to obtain a first base map; performing feature calculation processing on the first base map to obtain image feature information of the first base map; performing feature extraction on the sample scene graph through a visual model to obtain visual features of the sample scene graph, and inputting the visual features of the sample scene graph into a multi-layer perceptron for classification to obtain scene graph style information corresponding to the sample scene graph and spatial information of the object; processing the sample scene graph through a target multimodal large model to obtain background content description information corresponding to the sample scene graph, wherein the target multimodal large model is obtained by model distillation based on a training sample set and a large language model; obtaining alpha channel information of the first base map, and determining object layout information corresponding to the sample scene graph based on the alpha channel information.

[0170] Optionally, the processor may also execute program code of the following steps: obtaining a first scene image set before processing the sample scene graph through the target multimodal large model to obtain background content description information corresponding to the sample scene graph; determining a training sample set based on the first scene image set and the large language model; performing model distillation based on the training sample set, the large language model and the initial multimodal large model to obtain the target multimodal large model.

[0171] Optionally, the processor may also execute program code for the following steps: performing processing by a large language model based on the first prompt information and the first scene graph in the first scene image set to obtain scene graph description information corresponding to the first scene graph, wherein the first prompt information is used to guide the large language model to describe the scene content of the first scene graph; performing image segmentation processing on the first scene graph to obtain an image in which a foreground object corresponding to the first scene graph is occluded; performing processing by a large language model based on the second prompt information, the scene graph description information corresponding to the first scene graph, and the image in which a foreground object is occluded corresponding to the first scene graph to obtain background content description information corresponding to the first scene graph, wherein the second prompt information is used to guide the large language model to describe the background content of the image in which a foreground object is occluded; forming a training sample image pair based on the image in which the foreground object is occluded corresponding to the first scene graph and the background content description information corresponding to the first scene graph, and determining a training sample set based on multiple training sample image pairs.

[0172] Optionally, the processor may also execute the following program code: dividing the image in which the foreground object in the first training sample image pair is occluded into multiple regions, and calculating the correlation coefficient between the regions in the multiple regions and the background content description information corresponding to the regions, to obtain the calculation results corresponding to the regions, wherein the first training sample image pair is any one of the multiple training sample image pairs; judging based on the calculation results corresponding to the regions, if there is a region with a correlation coefficient less than a third threshold, filtering out the first training sample image pair to obtain a filtered training sample image pair, and generating a training sample set based on the filtered training sample image pair.

[0173] It can be understood by those skilled in the art that Fig.12 The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, or other terminal devices. Fig.12 The structure of the electronic device is not limited. For example, the electronic device 120 may also include Fig.12 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Fig.12 Different configurations are shown.

[0174] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0175] Example 5

[0176] The embodiment of the present application further provides a computer-readable storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the image generation method provided in the first embodiment.

[0177] Optionally, in this embodiment, the above storage medium may be located in any electronic device in a group of electronic devices in a computer network, or in any mobile terminal in a group of mobile terminals.

[0178] Example 6

[0179] The embodiment of the present application further provides a computer program product. Optionally, in this embodiment, the computer program product may include a computer program, and when the computer program is executed by a processor, the image generation method provided in the first embodiment is implemented.

[0180] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0181] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0182] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0183] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0184] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0185] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0186] The above are only preferred implementations of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An image generation method, characterized in that: include: Acquire a target base map containing a target object, wherein the transparency of the background of the target base map is a preset transparency; Performing feature calculation processing on the target base map to obtain image feature information of the target base map; Determining multiple target sample data from multiple sample data in a sample database according to the image feature information of the target base map, wherein the sample database is used to recommend model input conditions required for generating a scene graph, and the multiple target sample data are sample data whose inter-class deviation is greater than a preset threshold among the multiple sample data; The multiple target sample data and the target base map are input into a generative model to obtain multiple scene graphs containing the target objects.

2. The method according to claim 1, characterized in that The image feature information of the target base map at least includes a color feature histogram. The image feature information of the target base map obtained by performing feature calculation processing on the target base map includes: Dividing the color space of the target base map into a plurality of color intervals; The number of pixels in a color interval among the multiple color intervals is counted, and image rendering is performed according to the number of pixels in the color interval among the multiple color intervals to obtain a color feature histogram of the target base image.

3. The method according to claim 2, characterized in that The image feature information of the target base map also includes visual features. The image feature information of the target base map obtained by performing feature calculation processing on the target base map includes: The target base map is subjected to feature extraction through a visual model to obtain visual features of the target base map.

4. The method according to claim 3, characterized in that Determining a plurality of target sample data from a plurality of sample data in a sample database according to the image feature information of the target base map includes: Determining a plurality of first sample data from the plurality of sample data according to a color feature histogram of the target base map, and determining a plurality of second sample data from the plurality of sample data according to visual features of the target base map; The inter-class deviations between the plurality of first sample data and the plurality of second sample data are calculated, and the sample data whose inter-class deviations are greater than the preset threshold are used as the plurality of target sample data.

5. The method according to claim 4, characterized in that Determining a plurality of first sample data from the plurality of sample data according to the color feature histogram of the target base map comprises: Performing similarity calculation on the color feature histogram of the target base map and the color feature histograms of the plurality of sample data to obtain a first calculation result; Sample data with a similarity greater than a first threshold is selected from the first calculation results to obtain the plurality of first sample data.

6. The method according to claim 4, characterized in that Determining a plurality of second sample data from the plurality of sample data according to the visual features of the target base map comprises: Calculating similarity between the visual features of the target base map and the visual features of the plurality of sample data to obtain a second calculation result; Select sample data with a similarity greater than a second threshold from the second calculation results to obtain the plurality of second sample data.

7. The method according to claim 1, characterized in that Inputting the plurality of target sample data and the target base map into a generative model to obtain a plurality of scene graphs containing the target object comprises: Respectively taking scene graph style information, spatial information of the object, background content description information, and object layout information in the target sample data of the plurality of target sample data as model input conditions to obtain a plurality of groups of model input conditions; The model input conditions in the multiple groups of model input conditions and the target base map are respectively input into the generative model to obtain multiple scene graphs containing the target object.

8. The method according to claim 1, characterized in that The sample database is constructed by following the steps below: Obtain a sample scene image set; Determine a first base map containing sample objects corresponding to a sample scene graph in the sample scene image set, image feature information of the first base map, scene graph style information corresponding to the sample scene graph and spatial information of the objects, background content description information corresponding to the sample scene graph, and object layout information corresponding to the sample scene graph, wherein the transparency of the background of the first base map is the preset transparency, and the object layout information is used to characterize the position of the sample objects; A piece of sample data in the sample database is generated based on the sample scene graph, the first base map, the image feature information of the first base map, the scene graph style information, the spatial information where the object is located, the background content description information and the object layout information, and the sample database is generated based on multiple pieces of sample data.

9. The method according to claim 8, characterized in that Determining a first base map containing sample objects corresponding to a sample scene graph in the sample scene image set, image feature information of the first base map, scene graph style information corresponding to the sample scene graph and spatial information where objects are located, background content description information corresponding to the sample scene graph, and object layout information corresponding to the sample scene graph includes: Using an image processing network to perform cutout processing on the sample scene graph to obtain the first base map; Performing feature calculation processing on the first base map to obtain image feature information of the first base map; Extracting features of the sample scene graph through a visual model to obtain visual features of the sample scene graph, and inputting the visual features of the sample scene graph into a multi-layer perceptron for classification to obtain scene graph style information corresponding to the sample scene graph and spatial information of the object; Processing the sample scene graph through a target multimodal large model to obtain background content description information corresponding to the sample scene graph, wherein the target multimodal large model is obtained by model distillation based on a training sample set and a large language model; Acquire alpha channel information of the first base map, and determine object layout information corresponding to the sample scene graph according to the alpha channel information.

10. The method according to claim 9, characterized in that Before processing the sample scene graph by using the target multimodal macro model to obtain background content description information corresponding to the sample scene graph, the method further includes: Acquire a first scene image set; Determine the training sample set based on the first scene image set and the large language model; Model distillation is performed based on the training sample set, the large language model, and the initial multimodal large model to obtain the target multimodal large model.

11. The method according to claim 10, characterized in that Determining the training sample set based on the first scene image set and the large language model includes: Processing, by the large language model, according to the first prompt information and the first scene graph in the first scene image set, to obtain scene graph description information corresponding to the first scene graph, wherein the first prompt information is used to guide the large language model to describe the scene content of the first scene graph; Performing image segmentation processing on the first scene graph to obtain an image in which a foreground object corresponding to the first scene graph is blocked; Processing, by the large language model, according to the second prompt information, the scene graph description information corresponding to the first scene graph, and the image of the foreground object occluded corresponding to the first scene graph, to obtain background content description information corresponding to the first scene graph, wherein the second prompt information is used to guide the large language model to describe the background content of the image of the foreground object occluded; A training sample image pair is formed based on an image of an occluded foreground object corresponding to the first scene graph and background content description information corresponding to the first scene graph, and the training sample set is determined based on a plurality of training sample image pairs.

12. The method according to claim 11, characterized in that Determining the training sample set according to a plurality of training sample image pairs comprises: Dividing an image in which a foreground object is blocked in a first training sample image pair into a plurality of regions, and calculating correlation coefficients between regions in the plurality of regions and background content description information corresponding to the regions, to obtain calculation results corresponding to the regions, wherein the first training sample image pair is any one of the plurality of training sample image pairs; A judgment is made based on the calculation results corresponding to the area. If there is an area with a correlation coefficient less than a third threshold, the first training sample image pair is filtered out to obtain a filtered training sample image pair, and the training sample set is generated based on the filtered training sample image pair.

13. An image generation method, characterized in that: include: Acquire a target base map containing a target object uploaded by a client, wherein the transparency of the background of the target base map is a preset transparency; Performing feature calculation processing on the target base map in the cloud server to obtain image feature information of the target base map; determining multiple target sample data from multiple sample data in a sample database based on the image feature information of the target base map, wherein the sample database is used to recommend model input conditions required for generating a scene graph, and the multiple target sample data are sample data whose inter-class deviation is greater than a preset threshold among the multiple sample data; inputting the multiple target sample data and the target base map into a generative model to obtain multiple scene graphs containing the target object; The multiple scene graphs are fed back to the client.

14. An image generating device, characterized in that: include: A first acquisition unit is used to acquire a target base map containing a target object, wherein the transparency of the background of the target base map is a preset transparency; A first processing unit is used to perform feature calculation processing on the target base map to obtain image feature information of the target base map; A first determination unit is used to determine a plurality of target sample data from a plurality of sample data in a sample database according to image feature information of the target base map, wherein the sample database is used to recommend model input conditions required for generating a scene graph, and the plurality of target sample data are sample data whose inter-class deviation is greater than a preset threshold value among the plurality of sample data; The second processing unit is used to input the multiple target sample data and the target base map into the generative model to obtain multiple scene graphs containing the target objects.

15. An electronic device, characterized in that: include: A memory storing an executable program; A processor is used to run the program, wherein the program executes the image generation method of any one of claims 1 to 13 when running.

16. A computer program product, characterized in that The method comprises a computer program or instructions which, when executed by a processor, implement the image generation method according to any one of claims 1 to 13.

Images