Scene data generation method, device and equipment based on multi-modal large model
Through the scene data generation method based on a multimodal large model, product images are combined with text descriptions to generate high-resolution scene data images, solving the problem that the background image and low resolution cannot be generated for specific products in the prior art, and efficient and automated image generation is achieved.
Patent Information
- Application Number
- CN202411574593.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-06
- Publication Date
- 2025-03-18
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing image generation platform can only generate general scene images through textual content, and cannot generate background images for specific products, resulting in designers requiring a lot of manual adjustments, and the generated images have low resolution.
Using a scene data generation method based on a multimodal large model, a high-resolution scene data image is generated by inputting product images and text descriptions into the multimodal large model, feature learning is performed, the desired background image is generated, and the product image is fused with the background image to generate a high-resolution scene data image.
Direct generation of scene data images for publicity that can be used directly or require only a small amount of adjustment reduces labor costs and improves image resolution and quality.
Smart Images
Figure CN119648822A_ABST
Abstract
Description
Background Art
[0002] In the field of image generation technology, most of the current mainstream image generation tools are based on text description, that is, text-generated images. However, the generation effect of scene data is not ideal. Image generation platforms can often only generate general scene images through textual content descriptions, and cannot generate background images for foreground content such as specific products. They can often only provide designers with reference ideas or waste a lot of manpower time for adjustments, and need to replace the products in the generated images with foreground images such as our own products. Summary of the invention
[0003] The purpose of this application is to provide a scene data generation method, device and equipment based on a multimodal large model, which is used to solve the problem that the existing image generation platform can only generate general scene pictures through textual content and requires foreground replacement, and at the same time cannot achieve the high resolution required by some scene data images.
[0004] In a first aspect, an embodiment of the present application proposes a method for generating scene data based on a multimodal large model, comprising:
[0005] Inputting a product image and a text description of the product into a multimodal macro model, wherein the text description is a description of a target scene;
[0006] The multimodal large model is used to perform feature learning on the product image and the text description respectively, and a desired background image is generated based on the learning result of the text description. The background image is used to expand the background of the product image based on the learning result of the product image, so as to generate a scene data image with the product as the foreground.
[0007] By inputting product images and text descriptions into a multimodal large model together, the multimodal large model can be used to integrate product images that need to be promoted into the promotional background required by users, and can directly generate scene data images for promotion that can be used directly or require only a small amount of adjustment, thereby reducing labor costs.
[0008] In some possible embodiments, the performing feature learning on the product image and the text description respectively, generating a desired background image based on the learning result of the text description, and expanding the background of the product image using the background image based on the learning result of the product image to generate a scene data image with the product as the foreground includes:
[0009] Performing feature learning on the product image and the text description respectively to generate corresponding text task prompts and image task prompts;
[0010] Guiding the multimodal large model to generate a desired background image based on the text task prompt;
[0011] Based on the image task prompt, the multimodal large model is guided to determine the boundary of the product, and the background image is used to expand the background outside the product boundary to generate a scene data image with the product as the foreground.
[0012] The strategy of dual learning task prompts can enable the multimodal large model to simultaneously learn different features of text and image, generate corresponding scene data images with the product as the foreground, ensure the generation of background images corresponding to the input text description, and ensure the consistency of the generated scene data image content with the product image content, thereby preventing the generated scene data image from being too different from the product image.
[0013] In some possible embodiments, the multimodal large model is obtained by the following training method:
[0014] Acquire a training sample set including a plurality of training samples, wherein the training samples include a product image sample, a first text description corresponding to the product image sample, a target scene data image sample, and a second text description corresponding to a background in the target scene data image sample;
[0015] The training samples are input into the multimodal large model, and feature learning is performed on the product image samples, the first text description and the second text description respectively. A desired background image is generated based on the learning results of the first text description and the second text description. The background image is used to expand the background outside the product boundary based on the learning results of the product image samples, and parameters of the multimodal large model are adjusted with the goal of outputting the target scene data image samples in the training samples.
[0016] By training the above-mentioned multimodal large model, the multimodal large model can adjust parameters during the training process with the goal of outputting the target scene data image samples in the training samples, thereby ensuring the training effect of the generated scene data images for promotion.
[0017] In some possible embodiments, after generating the scene data image with the product as the foreground, the method further includes:
[0018] The generated scene data image is denoised using the denoising module in the multimodal large model, and the denoised scene data image is output.
[0019] Noise usually refers to random signals introduced during image acquisition, transmission or processing, which will reduce the clarity and visual effect of the image. After generating a scene data image with the product as the foreground, the noise reduction module can reduce or eliminate the noise components in the digital image, and use various algorithms and technologies to identify and process these noises, making the image cleaner and clearer, and improving the image quality.
[0020] In some possible embodiments, the input of the multimodal large model also includes a desired resolution, and after generating a scene data image with the product image as the foreground, the following steps are performed:
[0021] Using the multimodal large model, judging whether the generated scene data image needs to be resolution expanded according to a comparison result between the expected resolution and a preset resolution threshold;
[0022] When it is determined that resolution expansion is required, outputting a scene data image that is proportional to the desired resolution and smaller than a preset resolution threshold to the super-resolution model;
[0023] The super-resolution model is used to expand the resolution of the input scene data image, and the expanded scene data image is obtained and output.
[0024] Through the super-resolution model, low-resolution images or videos can be converted into high-resolution images or videos, so that the details of the images or videos are clearer and richer. In the present application, the scene data images can be directly expanded to different sizes to avoid the scene data images generated by the multimodal large model not meeting the resolution standard of the target scene data images, which can improve the quality of picture generation and reduce labor costs.
[0025] In some possible embodiments, the method further includes:
[0026] When it is determined that resolution expansion is not required, the multimodal large model is used to output a scene data image of a desired resolution.
[0027] By inputting the desired resolution into the multimodal large model, when the desired resolution is less than the preset resolution threshold of the multimodal large model, the scene data image of the desired resolution can be directly output, which meets the user's expectations while also reducing labor costs.
[0028] In some possible embodiments, the text description includes a preset description and an open description of the target scene, wherein the preset description includes at least one of the following:
[0029] At least one preset light effect, at least one preset style type.
[0030] Through the preset description and open description of the target scene, it can help users to describe the target scene more specifically, so that the scene data image generated by the multimodal large model is more in line with user needs. Among them, the preset description only gives a possible implementation method, which can be adjusted according to needs in actual applications.
[0031] In a second aspect, the embodiment of the present application further proposes a scene data generation device based on a multimodal large model, comprising:
[0032] An input unit, configured to input a product image and a text description of the product into the multimodal macro model, wherein the text description is a description of the target scene;
[0033] A scene data image generating unit is used to use the multimodal large model to perform feature learning on the product image and the text description respectively, generate a desired background image based on the learning result of the text description, and expand the background of the product image using the background image based on the learning result of the product image to generate a scene data image with the product as the foreground.
[0034] The embodiment of the present application proposes a scene data generation device based on a multimodal large model. By inputting a product image, the multimodal large model technology is used to merge the product as the foreground and background images according to the actual style and elements required in the user marketing process, and finally a scene data image is generated.
[0035] In the third aspect, an embodiment of the present application also proposes a scene data generation device based on a multimodal large model, comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the scene data generation method based on the multimodal large model as described in any one of the first aspects above.
[0036] The embodiment of the present application proposes a scene data generation device based on a multimodal large model, which can directly integrate product images into the scene required by the user. The multimodal large model can directly generate scene data images that can be used directly or require only a small amount of adjustment, thereby improving the quality of image generation and reducing labor costs. In addition, by performing image expansion through the super-resolution model, the scene data image can be directly expanded to different sizes, which can meet various practical marketing needs.
[0037] In a fourth aspect, an embodiment of the present application further proposes a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable the computer to execute the scene data generation method based on a multimodal large model as described in any one of the first aspects above.
[0038] The embodiment of the present application proposes a method, device and equipment for generating scene data based on a multimodal large model. The method mainly inputs a product image and a text description including a product into the multimodal large model, wherein the text description is a description of the target scene; the multimodal large model is used to perform feature learning on the product image and the text description respectively, and a desired background image is generated based on the learning result of the text description; the background image is used to expand the background of the product image based on the learning result of the product image, and a scene data image with the product as the foreground is generated. Through the method, device and equipment proposed in the present application, the product image can be directly integrated into the scene required by the user, and the multimodal large model can be used to directly generate a scene data image that can be used directly or requires only a small amount of adjustment, thereby improving the quality of image generation and reducing labor costs.
[0039] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. Obviously, the drawings introduced below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0041] Figure 1 A schematic diagram of a flow chart of a method for generating scene data based on a multimodal large model in an embodiment of the present application;
[0042] Figure 2 A schematic diagram of a process of generating a scene data image using a multimodal large model in an embodiment of the present application;
[0043] Figure 3 A schematic diagram of a flow chart of generating a scene data image using a learning method using dual learning task prompts in an embodiment of the present application;
[0044] Figure 4 This is a specific flow chart of using the multimodal large model to generate scene data in an embodiment of the present application;
[0045] Figure 5 This is an example diagram of a product image in an embodiment of the present application;
[0046] Figure 6This is an example diagram of a scene data image in an embodiment of the present application;
[0047] Figure 7 A schematic diagram of a process for training a multimodal large model in an embodiment of the present application;
[0048] Figure 8 This is a specific training flow chart of a multimodal large model in an embodiment of the present application;
[0049] Fig. 9 A schematic diagram of a process for denoising a scene data image in an embodiment of the present application;
[0050] Fig.10 This is a flow chart of performing resolution expansion on a scene data image in an embodiment of the present application;
[0051] Fig.11 This is a schematic diagram of a process that requires resolution expansion in an embodiment of the present application;
[0052] Fig.12 A specific flow chart of resolution matching in a scene data generation method in an embodiment of the present application;
[0053] Fig.13 This is a flow chart of an embodiment of the present application in which the resolution of the scene data image needs to be expanded;
[0054] Fig.14 This is a schematic diagram of a scene data generation process in an embodiment of the present application;
[0055] Fig.15 This is a schematic diagram of a scene data generation process in an embodiment of the present application;
[0056] Fig.16 Schematic diagram of a scene data generating device based on a multimodal large model in an embodiment of the present application;
[0057] Fig.17 Schematic diagram of a scene data generation device based on a multimodal large model in an embodiment of the present application. DETAILED DESCRIPTION
[0058] To further illustrate the technical solution provided by the embodiments of the present application, a detailed description is given below in conjunction with the accompanying drawings and specific implementation methods.
[0059] In the field of image generation technology, most of the current mainstream image generation tools are based on text description, that is, text-generated images. However, the generation effect of scene data images is not ideal. Image generation platforms can often only generate scene data images in a general way through textual content descriptions, and cannot generate background images for foreground content such as specific products. They can often only provide designers with reference ideas or waste a lot of manpower time for adjustments, and need to replace the products in the generated images with foreground images such as our own products.
[0060] In view of the problem that existing image generation tools only generate images through text descriptions and can only provide designers with ideas for reference, but cannot generate background images for foreground content such as specific products, the embodiment of the present application proposes a scene data generation method based on a multimodal large model.
[0061] like Figure 1 FIG. 1 is a flow chart of a method for generating scene data based on a multimodal large model in an embodiment of the present application, comprising:
[0062] Step 101, inputting a product image and a text description of a product into a multimodal macro model, wherein the text description is a description of a target scene;
[0063] By inputting the product image and text description of the product into the multimodal large model together, the product can be fused and generated as the foreground and background using a combined image and text input, thereby directly generating a target scene data image for the product.
[0064] Step 102: Using the multimodal large model, perform feature learning on the product image and the text description respectively, generate a desired background image based on the learning result of the text description, and use the background image to expand the background of the product image based on the learning result of the product image to generate a scene data image with the product as the foreground.
[0065] In an embodiment of the present application, the multimodal large model is used to learn the dual features of the product image and the text description, and the learning results of the dual features of the scene data image and text can generate a scene data image with the product as the foreground that meets expectations.
[0066] In some possible embodiments, the feature learning is performed on the product image and the text description respectively, a desired background image is generated based on the learning result of the text description, and the background image is used to expand the background of the product image based on the learning result of the product image to generate a scene data image with the product as the foreground, such as Figure 2 As shown, including:
[0067] Step 201, performing feature learning on the product image and the text description respectively, and generating corresponding text task prompts and image task prompts;
[0068] Step 202, guiding the multimodal large model to generate a desired background image based on the text task prompt;
[0069] Step 203: Based on the image task prompt, guide the multimodal large model to determine the boundary of the product, use the background image to expand the background outside the product boundary, and generate a scene data image with the product as the foreground.
[0070] In order to ensure the quality requirements of image generation, image generation needs to be based on both the input product image and the corresponding text description, and learn using a dual learning task prompt method, integrating text task prompts with image task prompts, and expanding based on product boundaries in product images.
[0071] In an embodiment of the present application, the text task prompt is used to guide the multimodal large model to generate a desired background image, and the image learning task prompt is used to guide the multimodal large model to determine the boundary of the product based on the information of the input image itself, and use the background image to expand the image outside the product boundary, which can ensure the consistency of the generated content and the image content, and prevent the generated scene data image from being too different from the product image.
[0072] like Figure 3 A flow chart of a learning method using dual learning task prompts to generate scene data images is shown. By inputting product images and text descriptions into the model together, the multimodal large model can recognize and respond to the product images and text descriptions, generate text task prompts and image task prompts, and generate corresponding scene data image content based on the text task prompts and image task prompts using a clipping text encoder and a clipping image encoder.
[0073] In some possible embodiments, the text description includes a preset description and an open description of the target scene, wherein the preset description includes at least one of the following:
[0074] At least one preset light effect, at least one preset style type.
[0075] In some possible embodiments, the preset light effects include at least one of the following: bright, dim, dusk, and the preset style types include at least one of the following: retro, science fiction. In the embodiments of the present application, only one possible preset description is given, and the present application does not specifically limit the options and quantity of the preset light effects and style types.
[0076] The following is a specific process of using the multimodal large model to generate scene data images using a refrigerator as a product. Figure 4 As shown:
[0077] Step 1, inputting a product image and a text description of the refrigerator into a multimodal macro model;
[0078] The input product image is Figure 5 As shown, the input text description is "In the open kitchen at home, the kitchen cabinets are divided into two layers, with bowls and plates, a sink and a metal faucet, and a metal pot on the stove from left to right. The refrigerator is located on the right side of the cabinet and embedded in the cabinet. There are a coffee table, a sofa and green plants at the bottom of the picture. There is a floor-to-ceiling window on the far right, and the sun shines in from the right."
[0079] Step 2, using the multimodal large model to perform feature learning on the product image and the text description respectively, and generate corresponding text task prompts and image task prompts;
[0080] Step 3: Based on the text task prompt, generate a desired background image, based on the image task prompt, determine the boundary of the product, use the background image to expand the background outside the product boundary, and generate a scene data image with the product as the foreground, such as Figure 6 shown.
[0081] In some possible embodiments, Figure 7 As shown, in order to ensure the generation effect and quality of the scene data image, the multimodal large model is obtained through the following training method:
[0082] Step 701, obtaining a training sample set including a plurality of training samples, wherein the training samples include a product image sample, a first text description corresponding to the product image sample, a target scene data image sample, and a second text description corresponding to the background in the target scene data image sample;
[0083] Step 702: input the training samples into the multimodal large model, perform feature learning on the product image samples, the first text description and the second text description respectively, generate a desired background image based on the learning results of the first text description and the second text description, expand the background outside the product boundary using the background image based on the learning results of the product image samples, and adjust the parameters of the multimodal large model with the goal of outputting the target scene data image samples in the training samples.
[0084] In the embodiment of the present application, a batch of product image samples and target scene data image samples are first collected, and data annotation work is performed on the product image samples and the target scene data image samples respectively to obtain a training sample set, wherein the training sample includes a product image sample, a first text description corresponding to the product image sample, a target scene data image sample, and a second text description corresponding to the background in the target scene data image sample. Figure 5 For example, the product image of is described as "a double-door refrigerator with two layers, the refrigerator is silver in color". Figure 5 Scene data image Figure 6 For example, the text description is "In the open kitchen at home, the kitchen cabinets are divided into two layers, with bowls and plates, a sink and a metal faucet, and a metal pot on the stove from left to right. The refrigerator is located on the right side of the cabinet and embedded in the cabinet. There are a coffee table, a sofa and green plants at the bottom of the picture. There is a floor-to-ceiling window on the far right, and the sun shines in from the right." The text description is paired with the corresponding image sample to form a training dataset with images in pairs.
[0085] The following is a specific training process for a multimodal large model: Figure 8 As shown, including:
[0086] Step 801, inputting the training sample set into the multimodal large model;
[0087] Step 802, performing feature learning on the same pair of product image samples, first text descriptions and second text descriptions in the training sample set respectively;
[0088] Step 803, generating a desired background image based on the learning results of the first text description and the second text description;
[0089] Step 804: Based on the learning result of the product image sample, the background outside the product boundary is expanded using the background image, and the parameters of the multimodal large model are adjusted with the goal of outputting the target scene data image sample in the training sample.
[0090] In some possible embodiments, Fig. 9 As shown, after generating a scene data image with the product as the foreground, the generated scene data image is denoised using the denoising module in the multimodal large model, and the denoised scene data image is output.
[0091] In the embodiment of the present application, after the scene data image with the product as the foreground is generated, the noise component in the digital image can be reduced or eliminated through the noise reduction module to improve the image quality. Noise generally refers to a random signal introduced during image acquisition, transmission or processing, which reduces the clarity and visual effect of the image. The noise reduction module uses various algorithms and techniques to identify and process these noises, making the image cleaner and clearer. The algorithms and techniques used by the noise reduction module are not specifically limited in the embodiment of the present application.
[0092] In addition, due to the limitations of the existing multimodal large language model, when the resolution of the scene data image to be generated exceeds 1k, the quality of the generation effect of the multimodal large model decreases and cannot be used, but the resolution of some scene data images is required to reach 4k, resulting in the inability to use the image directly after it is generated. In order to overcome the pain point that the multimodal large model is difficult to directly generate high-resolution images and cannot be directly used for scene data image generation, the embodiment of the present application also includes resolution expansion of the scene data image.
[0093] In some possible embodiments, the input of the multimodal large model also includes a desired resolution. After generating a scene data image with the product image as the foreground, Fig.10 As shown, including:
[0094] Step 1001, using the multimodal large model to determine whether the generated scene data image needs to be resolution expanded according to the comparison result between the expected resolution and the preset resolution threshold;
[0095] Step 1002, when it is determined that resolution expansion is required, outputting a scene data image that is proportional to the desired resolution and smaller than a preset resolution threshold to a super-resolution model;
[0096] Step 1003: Use the super-resolution model to expand the resolution of the input scene data image, obtain the expanded scene data image and output it.
[0097] In the embodiments of the present application, Fig.11 The figure shows a process diagram for resolution expansion, where a low-resolution scene data image is output by a large multimodal model, and then a high-resolution scene data image is output by a super-resolution model.
[0098] The full name of the super-resolution model is super-resolution model, which is a technology that can improve the resolution of images or videos. It converts low-resolution images or videos into high-resolution images or videos through algorithms, so that the details of the images or videos are clearer and richer. The super-resolution model used in the above embodiment is a mature technology in the industry and is not specifically limited in the embodiments of this application.
[0099] In some possible embodiments, the input of the multimodal large model also includes a desired resolution, and after generating a scene data image with the product image as the foreground, the following further includes:
[0100] When it is determined that resolution expansion is not required, the multimodal large model is used to output a scene data image of a desired resolution.
[0101] The following is a specific process for resolution matching in a scene data generation method based on a multimodal large model. Fig.12 As shown, a resolution threshold x*y is set for the multimodal large model, and the user inputs the expected resolution A*B before the scene data image generation starts. At this time, the discriminator determines whether the expected resolution exceeds the preset resolution threshold x*y. If it does not exceed the threshold, it is determined that resolution expansion is not required, and the multimodal large model is used to output the scene data image of the expected resolution; if it exceeds the threshold, it is determined that resolution expansion is required, and the multimodal large model outputs a scene data image a*b that is proportional to the expected resolution and smaller than the preset resolution threshold to the super-resolution model, and then the super-resolution model is used to perform resolution expansion on the input scene data image to obtain the expanded scene data image A*B and output it.
[0102] Through the super-resolution model, the scene data image can be directly expanded to different sizes, avoiding the scene data image generated by the multimodal large model not meeting the resolution standard of the target scene data image, which can improve the quality of image generation and reduce labor costs.
[0103] A process for expanding the resolution of the scene data image is given below. Fig.13 As shown, including:
[0104] Step 1301, inputting a product image and a text description of a product into a multimodal macro model;
[0105] Step 1302, using the multimodal large model to output a scene data image that is proportional to the desired resolution and smaller than a preset resolution threshold;
[0106] Step 1303, inputting the scene data image into the super-resolution model;
[0107] Step 1304: Use the super-resolution model to expand the resolution of the scene data image according to the input desired resolution, obtain the expanded scene data image and output it.
[0108] The embodiment of the present application proposes a method for generating scene data based on a multimodal large model. By inputting a product image, the multimodal large model technology is used to fuse the product as the foreground and background images according to the actual style and elements required in the user marketing process, and finally a scene data image is generated. In addition, by performing image expansion through the super-resolution model, the scene data image can be directly expanded to different sizes, which can meet various actual marketing needs.
[0109] According to a method for generating scene data based on a multimodal large model provided in an embodiment of the present application, specifically, Figure 3 As an example, a schematic diagram of a scene data image generation process is given, wherein the generated scene data image does not need to be resolution expanded, such as Fig.14 As shown, the following steps are included:
[0110] Step 1: Figure 3 , a corresponding text description "I need to generate a bright home background image based on the provided refrigerator" and an expected resolution are input into the multimodal large model, and the expected resolution is less than a preset resolution threshold;
[0111] Step 2: Use the multimodal large model to output a scene data image of a desired resolution.
[0112] According to a method for generating scene data based on a multimodal large model provided in an embodiment of the present application, specifically, Figure 3 As an example, a schematic diagram of a scene data image generation process is given, wherein the generated scene data image needs to be resolution expanded, such as Fig.15 As shown, the following steps are included:
[0113] Step 1: Figure 3 , a corresponding text description "I need to generate a bright home background image based on the provided refrigerator" and an expected resolution are input into the multimodal large model, and the expected resolution is greater than a preset resolution threshold;
[0114] Step 2, using the multimodal large model to output a scene data image that is proportional to the desired resolution and smaller than a preset resolution threshold;
[0115] Step 3, inputting the scene data image into the super-resolution model;
[0116] Step 4: Use the super-resolution model to expand the resolution of the scene data image and output a scene data image with a desired resolution.
[0117] Based on the same inventive concept, the embodiment of the present application also proposes a scene data generation device based on a multimodal large model, such as Fig.16 As shown, including:
[0118] An input unit 1601 is used to input a product image and a text description of the product into the multimodal macro model, wherein the text description is a description of the target scene;
[0119] The scene data image generating unit 1602 is used to use the multimodal large model to perform feature learning on the product image and the text description respectively, generate a desired background image based on the learning result of the text description, and expand the background of the product image using the background image based on the learning result of the product image to generate a scene data image with the product as the foreground.
[0120] In some possible embodiments, the feature learning is performed on the product image and the text description respectively, a desired background image is generated based on the learning result of the text description, and the background image is used to expand the background of the product image based on the learning result of the product image to generate a scene data image with the product as the foreground. The scene data image generating unit 1602 in the device is further used to:
[0121] Performing feature learning on the product image and the text description respectively to generate corresponding text task prompts and image task prompts;
[0122] Guiding the multimodal large model to generate a desired background image based on the text task prompt;
[0123] Based on the image task prompt, the multimodal large model is guided to determine the boundary of the product, and the background image is used to expand the background outside the product boundary to generate a scene data image with the product as the foreground.
[0124] In some possible embodiments, the multimodal large model in the device is obtained by the following training method:
[0125] Acquire a training sample set including a plurality of training samples, wherein the training samples include a product image sample, a first text description corresponding to the product image sample, a target scene data image sample, and a second text description corresponding to a background in the target scene data image sample;
[0126] The training samples are input into the multimodal large model, and feature learning is performed on the product image samples, the first text description and the second text description respectively. A desired background image is generated based on the learning results of the first text description and the second text description. The background image is used to expand the background outside the product boundary based on the learning results of the product image samples, and parameters of the multimodal large model are adjusted with the goal of outputting the target scene data image samples in the training samples.
[0127] In some possible embodiments, after generating the scene data image with the product as the foreground, the scene data image generating unit 1602 in the device is further used to:
[0128] The generated scene data image is denoised using the denoising module in the multimodal large model, and the denoised scene data image is output.
[0129] In some possible embodiments, the input of the multimodal large model also includes a desired resolution. After generating a scene data image with the product image as the foreground, the device is used to:
[0130] Using the multimodal large model, judging whether the generated scene data image needs to be resolution expanded according to a comparison result between the expected resolution and a preset resolution threshold;
[0131] When it is determined that resolution expansion is required, outputting a scene data image that is proportional to the desired resolution and smaller than a preset resolution threshold to the super-resolution model;
[0132] The super-resolution model is used to expand the resolution of the input scene data image, and the expanded scene data image is obtained and output.
[0133] In some possible embodiments, the device is further used for:
[0134] When it is determined that resolution expansion is not required, the multimodal large model is used to output a scene data image of a desired resolution.
[0135] In some possible embodiments, the text description includes a preset description and an open description of the target scene, wherein the preset description includes at least one of the following:
[0136] At least one preset light effect, at least one preset style type.
[0137] The embodiment of the present application proposes a scene data generation device based on a multimodal large model. By inputting a product image, the multimodal large model technology is used to merge the product as the foreground and background images according to the actual style and elements required in the user marketing process, and finally a scene data image is generated.
[0138] Based on the same inventive concept, an embodiment of the present application also proposes a scene data generation device based on a multimodal large model, comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the scene data generation method based on the multimodal large model as described above.
[0139] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.
[0140] Refer to the following Fig.17 To describe a scene data generating device 170 based on a multimodal large model according to this embodiment of the present application. Fig.17 The device 170 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0141] like Fig.17 As shown, a scene data generation device 170 based on a multimodal large model is presented in the form of a general electronic device. The components of a scene data generation device 170 based on a multimodal large model may include but are not limited to: at least one processor 171, at least one memory 172, and a bus 173 connecting different system components (including the memory 172 and the processor 171).
[0142] Bus 173 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.
[0143] The memory 172 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 1721 and / or a cache memory 1722 , and may further include a read-only memory (ROM) 1723 .
[0144] Memory 172 may also include a program / utility 1725 having a set (at least one) of program modules 1724, such program modules 1724 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.
[0145] A scene data generating device 170 based on a multimodal large model can also communicate with one or more external devices 174 (e.g., keyboards, pointing devices, etc.), and can also communicate with one or more devices that enable a user to interact with a scene data generating device 170 based on a multimodal large model, and / or communicate with any device (e.g., routers, modems, etc.) that enables the scene data generating device 170 based on a multimodal large model to communicate with one or more other electronic devices. Such communication can be performed through an input / output (I / O) interface 175. In addition, a scene data generating device 170 based on a multimodal large model can also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs) and / or public networks, such as the Internet) through a network adapter 176. As shown in the figure, the network adapter 176 communicates with other modules of a scene data generating device 170 based on a multimodal large model through a bus 173. It should be understood that, although not shown in the figures, other hardware and / or software modules may be used in conjunction with a multimodal large model-based scene data generation device 170, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.
[0146] Based on the same inventive concept, in some possible implementations, various aspects of a method for generating scene data based on a multimodal large model provided by the present application can also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of a method for generating scene data based on a multimodal large model according to various exemplary embodiments of the present application described above in this specification.
[0147] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0148] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. A method for generating scene data based on a multimodal large model, characterized in that: include: Inputting a product image and a text description of the product into a multimodal macro model, wherein the text description is a description of a target scene; The multimodal large model is used to perform feature learning on the product image and the text description respectively, and a desired background image is generated based on the learning result of the text description. The background image is used to expand the background of the product image based on the learning result of the product image, so as to generate a scene data image with the product as the foreground.
2. The method according to claim 1, characterized in that The performing feature learning on the product image and the text description respectively, generating a desired background image based on the learning result of the text description, and expanding the background of the product image using the background image based on the learning result of the product image to generate a scene data image with the product as the foreground, includes: Performing feature learning on the product image and the text description respectively to generate corresponding text task prompts and image task prompts; Guiding the multimodal large model to generate a desired background image based on the text task prompt; Based on the image task prompt, the multimodal large model is guided to determine the boundary of the product, and the background image is used to expand the background outside the product boundary to generate a scene data image with the product as the foreground.
3. The method according to claim 1, characterized in that The multimodal large model is obtained by the following training method: Acquire a training sample set including a plurality of training samples, wherein the training samples include a product image sample, a first text description corresponding to the product image sample, a target scene data image sample, and a second text description corresponding to a background in the target scene data image sample; The training samples are input into the multimodal large model, and feature learning is performed on the product image samples, the first text description and the second text description respectively. A desired background image is generated based on the learning results of the first text description and the second text description. The background image is used to expand the background outside the product boundary based on the learning results of the product image samples, and parameters of the multimodal large model are adjusted with the goal of outputting the target scene data image samples in the training samples.
4. The method according to claim 1 or 2, characterized in that: After generating a scene data image with the product as the foreground, the method further includes: The generated scene data image is denoised using the denoising module in the multimodal large model, and the denoised scene data image is output.
5. The method according to claim 1, characterized in that: The input of the multimodal large model also includes the desired resolution. After generating the scene data image with the product image as the foreground, the method includes: Using the multimodal large model, judging whether the generated scene data image needs to be resolution expanded according to a comparison result between the expected resolution and a preset resolution threshold; When it is determined that resolution expansion is required, outputting a scene data image that is proportional to the desired resolution and smaller than a preset resolution threshold to the super-resolution model; The super-resolution model is used to expand the resolution of the input scene data image, and the expanded scene data image is obtained and output.
6. The method according to claim 5, characterized in that Also includes: When it is determined that resolution expansion is not required, the multimodal large model is used to output a scene data image of a desired resolution.
7. The method according to claim 1, characterized in that The text description includes a preset description and an open description of the target scene, wherein the preset description includes at least one of the following: At least one preset light effect, at least one preset style type.
8. A scene data generation device based on a multimodal large model, characterized in that: include: An input unit, configured to input a product image and a text description of the product into the multimodal macro model, wherein the text description is a description of the target scene; A scene data image generating unit is used to use the multimodal large model to perform feature learning on the product image and the text description respectively, generate a desired background image based on the learning result of the text description, and expand the background of the product image using the background image based on the learning result of the product image to generate a scene data image with the product as the foreground.
9. A scene data generation device based on a multimodal large model, characterized in that: It includes at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the scene data generation method based on a multimodal large model as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable the computer to execute the scene data generation method based on a multimodal large model as described in any one of claims 1-7.