Terminal and image generation method, device and equipment based on large model

Abstract

The invention relates to the technical field of artificial intelligence, in particular to a terminal, and an image generation method, device and equipment based on a large model. The method comprises the following steps: acquiring a basic image used for image generation, determining contour information of a to-be-replaced target included in the basic image, inputting the basic image, the contour information and a cue word into an image generation large model, and enabling the image generation large model to generate other targets meeting the requirement of the contour information according to the requirement of the cue word. And replacing the to-be-replaced target corresponding to the to-be-replaced area in the basic image with the other target, thereby obtaining a generated image output by the image generation large model. And changing a to-be-replaced target included in the basic image into other targets similar to the contour of the to-be-replaced target by using the image generation large model, thereby generating different images. In other words, one image can generate a plurality of different images, and the problem of scarcity of image data is solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a terminal, a large model-based image generation method, a device and equipment. Background Art

[0002] In some business scenarios, scene data is scarce and target data is too little to be used for algorithm development. In this case, a large amount of data on targets in the scenario is needed for algorithm development. For example, in the railway field, we hope to identify large machinery such as excavators beside the railway, but such data is less abundant in real scenes. At this time, we hope to generate a large amount of data on large machinery beside the railway for algorithm development; in addition, we also hope to identify foreign objects on the railway. Real scene data is scarce, so we need to generate a large amount of data on foreign objects on the railway for algorithm development. In the field of image generation, simple image generation cannot meet this demand. We hope to add specific targets to the existing scene data without losing the degree of violation, which cannot be achieved with existing technology.

[0003] Therefore, how to generate images that meet the scene requirements becomes an urgent problem to be solved. Summary of the invention

[0004] The embodiments of the present application provide a terminal, a large model-based image generation method, an apparatus and a device to solve the problem of scarcity of image data for certain scenes in the prior art.

[0005] The present application provides a terminal, the terminal comprising a display and a processor:

[0006] The display is configured to: display a generated image generated based on a basic image;

[0007] The processor is configured to:

[0008] Get the base image for image generation;

[0009] Determining contour information of the object to be replaced included in the basic image;

[0010] The basic image, the contour information and the prompt word are input into the image generation model. The image generation model replaces the target to be replaced corresponding to the area to be replaced in the basic image with other targets that meet the requirements of the contour information according to the requirements of the prompt word, and obtains the generated image output by the image generation model.

[0011] The above technical solution has the following advantages or beneficial effects: obtaining a basic image for image generation, determining the contour information of the target to be replaced included in the basic image, and inputting the basic image, contour information and prompt words into the image generation model, so that the image generation model generates other targets that meet the contour information requirements according to the prompt words, and replaces the target to be replaced corresponding to the area to be replaced in the basic image with the other target, thereby obtaining the generated image output by the image generation model. The image generation model is used to change the target to be replaced included in the basic image to other targets with a contour similar to that of the target to be replaced, thereby generating different images. In other words, it is achieved that one image generates multiple different images, solving the problem of scarce image data.

[0012] Furthermore, the processor is specifically configured to:

[0013] Determine the position information of the target to be replaced in the basic image;

[0014] The position information and the base image are input into an image segmentation model, the image segmentation model identifies the target to be replaced included in the image area corresponding to the position information, determines the contour of the target to be replaced, and outputs a first mask image corresponding to the base image, wherein the contour of the target to be replaced is marked in the first mask image;

[0015] The basic image, the first mask image and the prompt word are input into the image generation model. The image generation model generates a replacement target that meets the requirements of the prompt word according to the outline of the mark in the first mask image, and replaces the target to be replaced with the replacement target in the area to be replaced in the basic image to obtain a generated image.

[0016] The above technical solution has the following advantages or beneficial effects: since the more similar the contour is to the target to be generated, the better the target generated by the image generation model, therefore, in an embodiment of the present application, the image segmentation model is used to identify the target to be replaced in the basic image, thereby obtaining a first mask image corresponding to the basic image. Since the contour of the target to be replaced is accurately identified in the first mask image, the image generation model can generate a better and more realistic generated image when performing image generation based on the first mask image.

[0017] Furthermore, the processor is further configured to:

[0018] Eliminating the target to be replaced in the basic image based on a target elimination algorithm to obtain a target-free image, and using the target-free image to update the basic image;

[0019] The determining of the contour information of the target to be replaced included in the basic image includes:

[0020] According to the pre-saved correspondence between different target types and candidate mask images, a second mask image corresponding to the target type required in the prompt word is determined, and the contour of the target identified in the second mask image is determined as the contour information of the target to be replaced.

[0021] The above technical solution has the following advantages or beneficial effects: after obtaining the basic image, the target to be replaced in the basic image is eliminated to obtain a target-free image, and then a second mask image that matches the target type required in the prompt word is selected from the candidate mask image, so that the image generation model generates a target that meets the contour identified in the second mask image, thereby ensuring the diversity of image generation.

[0022] Furthermore, the processor is specifically configured to:

[0023] Determine each target candidate mask map corresponding to the target type required in the prompt word according to the pre-saved correspondence between different target types and candidate mask maps;

[0024] In each target candidate mask map, any target candidate mask map is selected as the second mask map.

[0025] Furthermore, the processor is specifically configured to:

[0026] Determine the size of a first detection frame corresponding to the target to be replaced in the base image, and the size of a second detection frame corresponding to the target included in each target candidate mask image;

[0027] The second mask image is determined according to a deviation between a size of the first detection frame and a size of each second detection frame and a preset size threshold.

[0028] The above technical solution has the following advantages or beneficial effects: according to the deviation between the size of the first detection frame corresponding to the target to be replaced and the size of the second detection frame corresponding to the target included in each target candidate mask map, a target candidate mask map with similar size is selected as the second mask map, thereby ensuring the rationality of the generated image.

[0029] Furthermore, the processor is specifically configured to:

[0030] Determine the coordinate information of the center point of the object to be replaced included in the original basic image;

[0031] The center point of the target identified in the second mask image is overlapped with the pixel point corresponding to the coordinate information in the basic image, and the image area mapped by the target identified in the second mask image in the basic image is determined as the area to be replaced.

[0032] The above technical solution has the following advantages or beneficial effects: the center point of the target identified in the second mask image is made to coincide with the center point of the target to be replaced included in the basic image, the area to be replaced is determined, and it is ensured that the new target in the generated image is within the image area where the target to be replaced in the basic image is located, thereby realizing target replacement.

[0033] Furthermore, the processor is also configured to:

[0034] If the number of generated images input by the user is received, the image generation model is called the number of generated images times based on the basic image to obtain the number of generated images.

[0035] The above technical solution has the following advantages or beneficial effects: the user can input the desired number of generated images according to the needs, and the terminal obtains the generated images of the number of generated images by calling the image generation model for the number of generated images, thereby realizing batch generation of images.

[0036] The present application provides an image generation method based on a large model, the method comprising:

[0037] Get the base image for image generation;

[0038] Determining contour information of the object to be replaced included in the basic image;

[0039] The basic image, the contour information and the prompt word are input into the image generation model. The image generation model replaces the target to be replaced corresponding to the area to be replaced in the basic image with other targets that meet the requirements of the contour information according to the requirements of the prompt word, and obtains the generated image output by the image generation model.

[0040] Further, the determining of the contour information of the object to be replaced included in the basic image includes:

[0041] Determine the position information of the target to be replaced in the basic image;

[0042] The position information and the base image are input into an image segmentation model, the image segmentation model identifies the target to be replaced included in the image area corresponding to the position information, determines the contour of the target to be replaced, and outputs a first mask image corresponding to the base image, wherein the contour of the target to be replaced is marked in the first mask image;

[0043] The step of inputting the basic image, the contour information and the prompt word into the image generation model, wherein the image generation model replaces the target to be replaced corresponding to the area to be replaced in the basic image with another target that meets the contour information requirement according to the prompt word, and obtaining the generated image output by the image generation model includes:

[0044] The basic image, the first mask image and the prompt word are input into the image generation model. The image generation model generates a replacement target that meets the requirements of the prompt word according to the outline of the mark in the first mask image, and replaces the target to be replaced with the replacement target in the area to be replaced in the basic image to obtain a generated image.

[0045] Furthermore, after acquiring the basic image for image generation and before determining the contour information of the object to be replaced included in the basic image, the method further includes:

[0046] Eliminating the target to be replaced in the basic image based on a target elimination algorithm to obtain a target-free image, and using the target-free image to update the basic image;

[0047] The determining of the contour information of the target to be replaced included in the basic image includes:

[0048] According to the pre-saved correspondence between different target types and candidate mask images, a second mask image corresponding to the target type required in the prompt word is determined, and the contour of the target identified in the second mask image is determined as the contour information of the target to be replaced.

[0049] Furthermore, the determining, according to the pre-saved correspondence between different targets and candidate mask images, a second mask image corresponding to the target required in the prompt word includes:

[0050] Determine each target candidate mask map corresponding to the target type required in the prompt word according to the pre-saved correspondence between different target types and candidate mask maps;

[0051] In each target candidate mask map, any target candidate mask map is selected as the second mask map.

[0052] Furthermore, selecting any target candidate mask map as the second mask map from each target candidate mask map includes:

[0053] Determine the size of a first detection frame corresponding to the target to be replaced in the base image, and the size of a second detection frame corresponding to the target included in each target candidate mask image;

[0054] The second mask image is determined according to a deviation between a size of the first detection frame and a size of each second detection frame and a preset size threshold.

[0055] Furthermore, the process of determining the area to be replaced in the base image includes:

[0056] Determine the coordinate information of the center point of the object to be replaced included in the original basic image;

[0057] The center point of the target identified in the second mask image is overlapped with the pixel point corresponding to the coordinate information in the basic image, and the image area mapped by the target identified in the second mask image in the basic image is determined as the area to be replaced.

[0058] Furthermore, before inputting the basic image, the contour information and the prompt words into the image generation macro model, the method further includes:

[0059] If the number of generated images input by the user is received, the image generation model is called the number of generated images times based on the basic image to obtain the number of generated images.

[0060] The present application provides an image generation device based on a large model, the device comprising:

[0061] An acquisition module, used for acquiring a basic image for image generation;

[0062] A determination module, used to determine the contour information of the object to be replaced included in the basic image;

[0063] A generation module is used to input the basic image, the contour information and the prompt word into the image generation model. The image generation model replaces the target to be replaced corresponding to the area to be replaced in the basic image with other targets that meet the requirements of the contour information according to the requirements of the prompt word, and obtains the generated image output by the image generation model.

[0064] The present application also provides an electronic device, which includes a processor, and the processor is used to implement the steps of any of the above-mentioned large model-based image generation methods when executing a computer program stored in a memory.

[0065] The present application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-mentioned large model-based image generation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0067] Figure 1 A schematic diagram of the structure of a terminal provided in an embodiment of the present application;

[0068] Figure 2 A schematic diagram of a process of generating an image based on a large model provided in an embodiment of the present application;

[0069] Figure 3 A schematic diagram of an image generation process provided in an embodiment of the present application;

[0070] Figure 4 A schematic diagram of a first mask image provided in an embodiment of the present application;

[0071] Figure 5 A schematic diagram of image generation provided in an embodiment of the present application;

[0072] Figure 6a A basic image schematic diagram provided in an embodiment of the present application;

[0073] Figure 6b A schematic diagram of a target-free image provided in an embodiment of the present application;

[0074] Figure 7a A schematic diagram of a mask diagram provided in an embodiment of the present application;

[0075] Figure 7b A schematic diagram of a mask diagram provided in an embodiment of the present application;

[0076] Figure 7c A schematic diagram of a mask diagram provided in an embodiment of the present application;

[0077] Figure 8 A schematic diagram of an image generation process provided in an embodiment of the present application;

[0078] Figure 9a A basic image schematic diagram provided in an embodiment of the present application;

[0079] Figure 9b A schematic diagram of generating an image provided in an embodiment of the present application;

[0080] Fig.10 A schematic diagram of an image generation front-end page provided in an embodiment of the present application;

[0081] Fig.11aA schematic diagram of an image generation front-end page provided in an embodiment of the present application;

[0082] Fig.11b A schematic diagram of a segmentation result front-end display page provided in an embodiment of the present application;

[0083] Fig.12 A schematic diagram of the structure of an image generation device based on a large model provided in an embodiment of the present application;

[0084] Fig.13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0085] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the embodiment of the present application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiment is only a part of the embodiment of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of this application.

[0086] The embodiments of the present application provide a terminal, a method, an apparatus and a device for generating an image based on a large model, wherein the terminal includes a display and a processor: the display is configured to display a generated image generated based on a basic image; the processor is configured to obtain a basic image for image generation; determine contour information of a target to be replaced included in the basic image; input the basic image, the contour information and a prompt word into the large model for image generation, wherein the large model for image generation replaces the target to be replaced corresponding to the area to be replaced in the basic image with other targets that meet the contour information requirements according to the requirements of the prompt word, and obtains the generated image output by the large model for image generation.

[0087] The terminal 10 provided in the present application can be used for image generation. Figure 1 A schematic diagram of the structure of a terminal provided in an embodiment of the present application, the terminal 10 includes a display 101 and a processor 102, wherein the display 101 is configured to display a generated image generated based on a basic image; the processor 102 is configured to execute the steps of the image generation method based on a large model, and the process of the image generation method based on a large model is described below in conjunction with each embodiment. Through the large model-based image generation method provided in an embodiment of the present application, the target to be replaced in the image can be replaced with other targets of different forms, colors, etc. based on the existing scene data, and one basic image can generate multiple images of different targets. For example, the excavator in the image is replaced with an excavator of another color or an excavator in another working state. Based on this method, the problem of data scarcity is solved, thereby further improving the image generation effect.

[0088] Figure 2 A schematic diagram of a process of generating an image based on a large model provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the process includes the following steps:

[0089] S201: Acquire a basic image for image generation.

[0090] The large model-based image generation method provided in the present application is applied to an electronic device, which may be a server, a PC, a terminal, etc.

[0091] In order to generate images of certain specific scenes, in an embodiment of the present application, a large image generation model can be used to generate images. When performing image generation, a basic image for image generation can be obtained, and the large image generation model is subsequently used to transform the basic image to generate an image that meets user requirements. Among them, the basic image can be input by the user or pre-stored in the memory. In other words, the large model-based image generation method provided in the embodiment of the present application is suitable for online real-time generation and also for offline generation. In an embodiment of the present application, one or more basic images can be obtained at one time. When multiple basic images are obtained, each basic image can be processed later.

[0092] S202: Determine contour information of the object to be replaced included in the basic image.

[0093] In order to obtain a target included in the generated image that has a similar outline to the target to be replaced included in the base image, in the embodiment of the present application, the outline information of the target to be replaced included in the base image can be determined. In the embodiment of the present application, the outline information of the target in the image can be extracted based on image recognition technology, and the embodiment of the present application will not repeat this process.

[0094] S203: The basic image, the contour information and the prompt word are input into the image generation model. The image generation model replaces the target to be replaced corresponding to the area to be replaced in the basic image with other targets that meet the requirements of the contour information according to the requirements of the prompt word, and obtains the generated image output by the image generation model.

[0095] After obtaining the contour information, the image generation model can be used to replace the target to be replaced included in the basic image, thereby generating a generated image different from the basic image. In an embodiment of the present application, the basic image, contour information and prompt words can be input into the image generation model. After receiving the information, the image generation model can replace the target to be replaced corresponding to the area to be replaced in the basic image with other targets that meet the contour information requirements according to the requirements of the prompt words. The basic image with the target replaced can be called a generated image. Among them, the prompt word can be "Replace the excavator in the basic image with a pink one", and the image generation model can analyze the semantics of the prompt word and replace the original excavator in the basic image with an excavator of the same shape but pink in color. Among them, the original excavator is the target to be replaced. The prompt word can be input by the user together with the basic image.

[0096] In the embodiment of the present application, a basic image for image generation is obtained, and the contour information of the target to be replaced included in the basic image is determined, so that the basic image, the contour information and the prompt word are input into the image generation model, so that the image generation model generates other targets that meet the contour information requirements according to the prompt word requirements, and replaces the target to be replaced corresponding to the area to be replaced in the basic image with the other target, thereby obtaining the generated image output by the image generation model. The image generation model is used to replace the target to be replaced included in the basic image with other targets with a similar contour to the target to be replaced, thereby generating different images. In other words, it is achieved that one image generates multiple different images, solving the problem of scarce image data.

[0097] In order to further improve the accuracy of image generation, based on the above embodiments, in the embodiment of the present application, the step of determining the contour information of the target to be replaced included in the basic image includes:

[0098] Determine the position information of the target to be replaced in the basic image;

[0099] The position information and the base image are input into an image segmentation model, the image segmentation model identifies the target to be replaced included in the image area corresponding to the position information, determines the contour of the target to be replaced, and outputs a first mask image corresponding to the base image, wherein the contour of the target to be replaced is marked in the first mask image;

[0100] The step of inputting the basic image, the contour information and the prompt word into the image generation model, wherein the image generation model replaces the target to be replaced corresponding to the area to be replaced in the basic image with another target that meets the contour information requirement according to the prompt word, and obtaining the generated image output by the image generation model includes:

[0101] The basic image, the first mask image and the prompt word are input into the image generation model. The image generation model generates a replacement target that meets the requirements of the prompt word according to the outline of the mark in the first mask image, and replaces the target to be replaced with the replacement target in the area to be replaced in the basic image to obtain a generated image.

[0102] In an embodiment of the present application, when determining the contour information of the target to be replaced included in the base image, the position information of the target to be replaced in the base image can be determined first. The position information can be any information describing the position of the target to be replaced in the base image. For example, the position information can be the coordinate information of the annotation box of the target to be replaced in the base image. The position information can be obtained by performing target recognition on the target to be replaced included in the base image based on a target detection algorithm, or it can be manually pre-annotated. Exemplarily, after obtaining the base image, the relevant staff can manually annotate the position information corresponding to the target to be replaced included in the base image before using the base image for image generation, and save the position information corresponding to the base image in a manual marking file. When generating an image based on the base image, the position information corresponding to the target to be replaced in the base image can be searched in the manual marking file.

[0103] After obtaining the position information, the position information and the base image can be input into the image segmentation model, so that the image segmentation model determines the image area to be identified according to the position information. Among them, the image segmentation model can be any model with image segmentation capability. Exemplarily, the image segmentation model can be the Segment Anything Model 2 (SAM2). Of course, those skilled in the art can also select a suitable image segmentation model as needed. After determining the image area, the target to be replaced included in the area is identified, and the pixels included in the target to be replaced are determined, so as to determine the outline of the target to be replaced in the base image, and then determine the first mask map corresponding to the base image according to the outline. The outline of the target to be replaced is marked in the first mask map. Exemplarily, the pixel values ​​of the pixels in the first mask map can be composed of 1 and 0, for example, the pixel values ​​of the pixels corresponding to the target to be replaced in the first mask map are 0, and the pixel values ​​of the pixels of the non-target to be replaced are 1. Of course, the pixel values ​​of the pixels corresponding to the target to be replaced can also be 1, and the pixel values ​​of the pixels of the non-target to be replaced are 0. The size of the first mask image is consistent with the size of the basic image, and the position of the outline of the object to be replaced identified in the first mask image is the position of the object to be replaced in the basic image.

[0104] After obtaining the first mask image, when using the image generation model to determine the generated image, the base image, the first mask image and the prompt word can be input into the image generation model. Since the position and outline of the target to be replaced are marked in the first mask image, the area to be replaced can also be determined. The image generation model can generate a replacement target that meets the prompt word requirements according to the outline marked in the first mask image, and replace the target to be replaced with the replacement target in the area to be replaced in the base image to obtain the generated image.

[0105] Since the more similar the contour is to the target to be generated, the better the target generated by the image generation model, in an embodiment of the present application, the image segmentation model is used to identify the target to be replaced in the basic image, so as to obtain a first mask image corresponding to the basic image. Since the contour of the target to be replaced is accurately identified in the first mask image, the image generation model can generate a better and more realistic generated image when performing image generation based on the first mask image.

[0106] Figure 3 A schematic diagram of an image generation process provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, after obtaining the basic image input by the user, the position information corresponding to the detection frame of the target to be replaced included in the basic image is determined. Specifically, the position information can be determined based on the target detection algorithm, or can be found in a manual marking file. After obtaining the position information, the position information and the basic image are input into the segmentation model to obtain the first mask image output by the segmentation model. Figure 4 A schematic diagram of a first mask image provided in an embodiment of the present application. After obtaining the first mask image, the base image, the first mask image and the prompt word are input into the image generation model, and the image generation model replaces the position of the target to be replaced with the target specified in the prompt word to obtain the generated image output by the image generation model.

[0107] In the embodiment of the present application, different prompt words can be used to generate different new images based on the current first mask image and the base image, so that one original image can generate multiple new images. For example, the input prompt words are: a yellow excavator and a green excavator, and two different images can be generated based on these two prompt words.

[0108] In the embodiment of the present application, batch generation of images can be performed. In the embodiment of the present application, multiple basic images can be pre-saved in a folder. When batch generation of images is performed, the above-mentioned image generation operation can be performed for each basic image in the folder to obtain a new generated image. The operation is repeated until each image in the folder is processed.

[0109] In the embodiment of the present application, multiple groups of prompt words can be preset, and each basic image can generate multiple different images based on different prompt words, thereby increasing the diversity of generated data.

[0110] Taking the excavator detection scenario in the railway application scenario as an example, due to the scarcity of real excavator samples beside the railway, more image data needs to be generated to support the development of excavator detection algorithms. Based on this, the existing real data can be used to replace the original excavator with a new excavator to generate new scene data. Figure 5 A schematic diagram of image generation provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the base image is a railway image, in which there is a red excavator under construction next to the railway. Object detection is performed on the base image to determine the detection frame corresponding to the excavator in the base image. The position information corresponding to the base image and the detection frame is input into the segmentation model to obtain the first mask map. After obtaining the first mask map, the base image, the first mask map and the prompt word are input into the image generation model to obtain the generated image. In the generated image, the original red excavator is replaced by a yellow excavator with the same posture.

[0111] In order to generate diverse images, based on the above embodiments, in the embodiment of the present application, after acquiring the basic image for image generation and before determining the contour information of the target to be replaced included in the basic image, the method further includes:

[0112] Eliminating the target to be replaced in the basic image based on a target elimination algorithm to obtain a target-free image, and using the target-free image to update the basic image;

[0113] The determining of the contour information of the target to be replaced included in the basic image includes:

[0114] According to the pre-saved correspondence between different target types and candidate mask images, a second mask image corresponding to the target type required in the prompt word is determined, and the contour of the target identified in the second mask image is determined as the contour information of the target to be replaced.

[0115] In the above-mentioned embodiments, the shape of the new target included in the generated image is similar to the shape of the target to be replaced. In order to ensure the diversity of the generated image, in the embodiment of the present application, the target to be replaced in the basic image can be eliminated based on the target elimination algorithm to obtain a target-free image. After obtaining the target-free image, the basic image is updated using the target-free image. In other words, the target to be replaced in the basic image is deleted. When performing target elimination, any algorithm or model with target elimination capability can be used, and the embodiment of the present application does not limit this. Figure 6aA basic image schematic diagram provided in an embodiment of the present application, such as Figure 6a As shown, in the base image, there is an excavator under construction on the left side (left and right in the figure) of the railway. Based on the target elimination algorithm, the target to be replaced in the base image is eliminated, and the following can be obtained: Figure 6b The target-free image shown, Figure 6b A schematic diagram of a target-free image provided by an embodiment of the present application, in which the excavator originally on the left side (left and right in the figure) of the railway has been eliminated.

[0116] In order to obtain diverse generated images, in an embodiment of the present application, the correspondence between different target types and candidate mask maps can be pre-saved. That is to say, different candidate mask maps are saved for different target types, and the same target type can correspond to one or more candidate mask maps. Among them, the target type can be an excavator, a bulldozer, a train, etc., and technicians in this field can configure the target type as needed. The candidate mask map saved for the target type of excavator can be a mask map when the excavator is shoveling, a mask map when the excavator is grabbing, or a mask map when the excavator is dragging. In an embodiment of the present application, the candidate mask map can be formulated in the following two ways:

[0117] Method 1: Collect image data of targets in the current scene, obtain the target contour through manual annotation or segmentation algorithm to obtain a candidate mask map.

[0118] Method 2: Collect image data containing targets in other scenes, obtain the target outline through manual annotation or segmentation algorithm to obtain candidate mask maps. For example, in railway scenes, excavators are not common, but by collecting image data of excavators in construction site scenes, a large number of candidate mask maps of excavators with different appearances can be obtained.

[0119] Figure 7a-7c A schematic diagram of a mask diagram provided in an embodiment of the present application, Figure 7a-7c The outlines of different excavators are shown in FIG. 1 , and the size and posture of the target included in each mask image are different, that is, the size and posture of the included excavators are different.

[0120] In an embodiment of the present application, when determining the contour information of the target to be replaced included in the basic image, the target type required in the prompt word can be determined. Exemplarily, if the prompt word is "a red excavator", it can be determined that the target type required by the prompt word is an excavator, and if the prompt word is "a green locomotive", it can be determined that the target type required by the prompt word is a locomotive. In an embodiment of the present application, the second mask map corresponding to the target type required in the prompt word can be determined based on the correspondence between different target types and candidate mask maps saved in advance. In an embodiment of the present application, the candidate mask map saved for the target type required in the prompt word can be determined as the second mask map. After obtaining the second mask map, the contour of the target identified in the second mask map can be determined as the contour information of the target to be replaced.

[0121] Since in the embodiment of the present application, multiple candidate mask maps may be saved for the same target type. In the embodiment of the present application, when determining the second mask map corresponding to the target required in the prompt word according to the correspondence between different targets and candidate mask maps saved in advance. Each target candidate mask map corresponding to the target type required in the prompt word can be determined according to the correspondence between different target types and candidate mask maps saved in advance. After each target candidate mask map is determined, any target candidate mask map can be selected from each target candidate mask map as the second mask map.

[0122] In a possible implementation, after the second mask image is determined, the second mask image can be adjusted to be the same size as the base image, and the image region where the target identified in the resized second mask image is located is consistent with the image region where the target to be replaced is located in the base image. That is, by adjusting the size of the second mask image, the image generation model is informed of the location of the region to be replaced.

[0123] In an embodiment of the present application, after obtaining a basic image, the target to be replaced in the basic image is eliminated to obtain a target-free image, and then a second mask image that matches the target type required in the prompt word is selected from the candidate mask image, so that the image generation model generates a target that conforms to the contour identified in the second mask image, thereby ensuring the diversity of image generation.

[0124] In order to ensure the rationality of the generated image, on the basis of the above embodiments, in the embodiment of the present application, in each target candidate mask map, selecting any target candidate mask map as the second mask map includes:

[0125] Determine the size of a first detection frame corresponding to the target to be replaced in the base image, and the size of a second detection frame corresponding to the target included in each target candidate mask image;

[0126] The second mask image is determined according to a deviation between a size of the first detection frame and a size of each second detection frame and a preset size threshold.

[0127] Since the image generation model needs to generate a new image according to the contour information identified in the mask image, if the contour identified in the mask image is too large, the newly generated target in the generated image will also be too large, which can easily cause the generated image to be too abrupt. Therefore, in an embodiment of the present application, when selecting and determining the second mask image from multiple target candidate mask images, a target candidate mask image with a size similar to the size of the target to be replaced in the base image can be selected as the second mask image.

[0128] In an embodiment of the present application, the size of the first detection frame corresponding to the target to be replaced in the base image and the size of the second detection frame corresponding to the target included in each target candidate mask map can be determined, and the deviation between the size of the first detection frame and the size of each second detection frame can be determined. After each deviation is determined, a target candidate mask map with the smallest size deviation can be selected as the second mask map according to each deviation and a preset size threshold.

[0129] Specifically, the bounding rectangle of the target to be replaced in the base image can be determined, and the bounding rectangle is the first detection frame corresponding to the target to be replaced. At the same time, the bounding rectangle of the target included in each target candidate mask image is determined, and the bounding rectangle is the second detection frame corresponding to the target to be replaced. Get the width w of the first detection frame ori and height h ori , and the width w of the second detection box ref and height h ref . In each target candidate mask map, select the target candidate mask map that meets the following conditions.

[0130]

[0131] If there is only one target candidate mask map after screening, the target candidate mask map can be determined as the second mask map. If there are multiple target candidate mask maps after screening, any target mask map among the multiple target candidate mask maps can be used as the second mask map.

[0132] In an embodiment of the present application, based on the deviation between the size of the first detection frame corresponding to the target to be replaced and the size of the second detection frame corresponding to the target included in each target candidate mask map, a target candidate mask map with a similar size is selected as the second mask map, thereby ensuring the rationality of the generated image.

[0133] In order to further ensure the rationality of the generated image, on the basis of the above embodiments, in the embodiment of the present application, the process of determining the area to be replaced in the basic image includes:

[0134] Determine the coordinate information of the center point of the object to be replaced included in the original basic image;

[0135] The center point of the target identified in the second mask image is overlapped with the pixel point corresponding to the coordinate information in the basic image, and the image area mapped by the target identified in the second mask image in the basic image is determined as the area to be replaced.

[0136] Since the current base image does not include the target to be replaced, in order to generate a new target at the position of the target to be replaced in the original base image, in an embodiment of the present application, the coordinate information of the center point of the target to be replaced included in the original base image can be determined, so that the center point of the target identified in the second mask image overlaps with the pixel point corresponding to the coordinate information in the base image, and the image area mapped in the base image by the target identified in the second mask image is determined as the area to be replaced. In other words, the determined second mask image is placed at the position corresponding to the target to be replaced in the base image. In other words, the center point of the target to be replaced in the original base image is obtained, and the second mask image is placed using it as the center point, and the image area corresponding to the target identified in the second mask image in the base image is determined as the area to be replaced. This allows the image generation model to generate a new target at the same position in the original base image.

[0137] In an embodiment of the present application, the center point of the target identified in the second mask image is made to coincide with the center point of the target to be replaced included in the base image, and the area to be replaced is determined, thereby ensuring that the new target in the generated image is within the image area of ​​the base image where the target to be replaced is located, thereby achieving target replacement.

[0138] Figure 8 A schematic diagram of an image generation process provided in an embodiment of the present application is shown in FIG. Figure 8As shown, after obtaining the basic image input by the user, the target to be replaced in the basic image is eliminated based on the target elimination algorithm to generate a new image without the target. The new image can be used as a target-free image, and the target-free image is used to update the original basic image. The second mask map corresponding to the target type required in the prompt word is selected from the pre-saved candidate mask map. After obtaining the second mask map, according to the coordinate information of the center point of the target to be replaced in the original basic image, the center point of the target identified in the second mask map is overlapped with the pixel point corresponding to the coordinate information in the basic image, and the image area mapped by the target identified in the second mask map in the basic image is determined as the area to be replaced. Subsequently, the second mask map, the basic image and the prompt word are input into the image generation model. The image generation model replaces the target to be replaced corresponding to the area to be replaced in the basic image with other targets that meet the contour information requirements according to the requirements of the prompt word, and obtains the generated image output by the image generation model.

[0139] In the embodiment of the present application, the target to be replaced in the basic image is eliminated by using the target elimination algorithm to obtain a target-free image. A batch of candidate mask images of different shapes and states are prepared in advance, and a target candidate mask image of matching size is selected as the second mask image. Then, the prompt word, the selected second mask image, and the target-free image are sent to the image generation model to obtain a generated image, in which the state and shape of the newly generated target are different from those of the target to be replaced.

[0140] In order to improve the efficiency of image generation, based on the above embodiments, in the embodiment of the present application, before inputting the basic image, the contour information and the prompt word into the image generation model, the method further includes:

[0141] If the number of generated images input by the user is received, the image generation model is called the number of generated images times based on the basic image to obtain the number of generated images.

[0142] In an embodiment of the present application, batch generation of images can be performed, and the user can set the number of generated images while inputting the basic image, and the number of generated images can be any integer. If the number of generated images is received, it means that the user wants to generate the number of generated images based on the one basic image. In an embodiment of the present application, a large image generation model can be called based on the basic image to obtain the number of generated images.

[0143] Specifically, assuming that the number of generated images is 3, then the image generation model needs to be called 3 times for image generation. Each time the image generation model determines to generate an image, it is generated based on the same basic image.

[0144] In a possible implementation, a plurality of basic images may be pre-saved in a folder. When performing batch generation of images, for each basic image in the folder, the basic image is subjected to target elimination to obtain an image after the target is removed. A suitable second mask image is randomly selected from the pre-saved candidate mask images, and the second mask image is processed to obtain a final mask image, in which the corresponding area to be replaced in the final mask image is marked with the outline of the corresponding target. The second mask image is updated using the final mask image. The updated second mask image, the basic image, and the prompt word are input into the image generation model to obtain a generated image. The operation is repeated until each basic image in the folder is processed.

[0145] In this process, for each basic image, multiple generated images can be generated based on different second mask images, or multiple groups of prompt words can be pre-set, and each basic image can generate multiple different generated images based on different prompt words and different second mask images. In the embodiment of the present application, one basic image can generate multiple generated images, increasing the diversity of generated images.

[0146] Figure 9a A basic image schematic diagram provided in an embodiment of the present application, as shown in 9a, there is an excavator under construction on the left side (left and right in the figure) of the railway. Figure 9a In the figure, the "arm" of the excavator is working towards the left (left and right in the figure). Based on the image generation method described in the above embodiments, the following can be generated based on the basic image: Figure 9b The generated image shown is Figure 9b A schematic diagram of generating an image is provided in an embodiment of the present application, such as Figure 9b As shown, the excavator originally on the left side (left and right in the diagram) of the railway has been replaced by another excavator with a completely different posture, and the "arm" of the excavator is working towards the right side (left and right in the diagram).

[0147] In the embodiment of the present application, the contour information of the target to be replaced may also be manually drawn by the user. Fig.10 A schematic diagram of an image generation front-end page provided in an embodiment of the present application, such as Fig.10 As shown, the user can upload the original image on the image generation page. After uploading the original image, the user can manually draw the outline information of the target to be replaced based on the prompt and enter the prompt word. After receiving the original image, the electronic device can use the original image as a basic image and generate based on the basic image to obtain and output the generated image. Fig.10 The image shown on the middle right (left and right in the figure) side is the generated image. After viewing the generated image, the user can choose to regenerate or save it according to the generated effect.

[0148] Since the effect of image generation is highly dependent on the contour information of the target to be replaced, the better the contour information drawing effect is, the better the image generation effect is. However, the effect of manual drawing is definitely lower than the effect of image segmentation. Therefore, in the embodiment of the present application, the contour information can also be automatically determined based on the method described in the above embodiments, and the generated image can be automatically obtained.

[0149] Fig.11a A schematic diagram of an image generation front-end page provided in an embodiment of the present application, such as Fig.11a As shown in the figure, the user can upload the original image on the image generation page. After uploading the original image, the user can manually draw a rectangular box based on the prompt. The rectangular box is used to mark the location information of the target to be replaced, and then click the target segmentation contour generation button. At this time, the original image area displays the algorithm segmentation result. Fig.11b A schematic diagram of a segmentation result front-end display page provided in an embodiment of the present application, such as Fig.11b As shown, the electronic device automatically identifies the outline of the target to be replaced in the original image. The user can manually adjust the segmentation frame based on this. After the adjustment is completed, enter the prompt word, select the number of images to be generated, click Image Generation, generate a new image and display it.

[0150] In the embodiments of the present application, two batch image generation methods for specific target replacement are proposed, which are mainly used to replace the targets in the original image with more diverse new targets, thereby generating a new image. The large model-based image generation method provided in the embodiments of the present application can be applied to various image generation platforms.

[0151] The large model-based image generation method provided in this application mainly includes the following advantages:

[0152] (1) Automatically generate a mask image of the contour of the target to be replaced: The target to be replaced is automatically segmented based on the segmentation algorithm, and a code image with the contour of the target to be replaced is output, which is more accurate than manually drawing the contour of the target to be replaced.

[0153] (2) Data diversity: Multiple images can be generated by using different prompt words.

[0154] (3) Process automation: No need to manually mark each sheet with prompts, usable data can be automatically generated in batches.

[0155] (4) Interactive target replacement image generation method, introducing an interactive prompt generation method on the data generation platform.

[0156] (5) Target diversification: the original target can be replaced with more diversified target data.

[0157] This method is applicable to image data generation in all data-scarce scenarios. As long as there is a small amount of real scene data, this method can be used to generate more data in the scene.

[0158] The image generation method proposed in this solution is applicable to all target-scarce image generation needs, and meets the characteristics of generalizability in the trustworthy characteristic; the image generation method provided by this application can maintain the characteristics of being superior to the image generation effect of artificial drawing prompts under any circumstances, and meets the characteristics of robustness in the trustworthy characteristic; the image generation method provided by this application can be stopped at any time by human intervention, and meets the characteristics of controllability in the trustworthy characteristic.

[0159] Based on the same inventive concept, the embodiment of the present application also provides an image generation device based on a large model. Fig.12 A schematic diagram of the structure of an image generation device based on a large model provided in an embodiment of the present application is shown in FIG. Fig.12 As shown, the device comprises:

[0160] An acquisition module 1201 is used to acquire a basic image for image generation;

[0161] A determination module 1202 is used to determine the contour information of the object to be replaced included in the basic image;

[0162] The generation module 1203 is used to input the basic image, the contour information and the prompt word into the image generation model. The image generation model replaces the target to be replaced corresponding to the area to be replaced in the basic image with other targets that meet the requirements of the contour information according to the requirements of the prompt word, and obtains the generated image output by the image generation model.

[0163] In a possible implementation, the determination module 1202 is specifically configured to determine position information of the target to be replaced in the basic image; input the position information and the basic image into an image segmentation model, the image segmentation model identifies the target to be replaced included in the image region corresponding to the position information, determines the contour of the target to be replaced, and outputs a first mask image corresponding to the basic image, wherein the contour of the target to be replaced is identified in the first mask image;

[0164] The generation module 1203 is specifically used to input the basic image, the first mask image and the prompt word into the image generation model. The image generation model generates a replacement target that meets the requirements of the prompt word according to the outline of the mark in the first mask image, and replaces the target to be replaced with the replacement target in the area to be replaced in the basic image to obtain a generated image.

[0165] In a possible implementation, the device further includes:

[0166] An elimination module 1204 is used to eliminate the target to be replaced in the basic image based on a target elimination algorithm to obtain a target-free image, and use the target-free image to update the basic image;

[0167] The determination module 1202 is specifically configured to determine a second mask image corresponding to the target type required in the prompt word according to a pre-saved correspondence between different target types and candidate mask images, and determine the contour of the target identified in the second mask image as the contour information of the target to be replaced.

[0168] In a possible implementation, the determination module 1202 is specifically used to determine each target candidate mask map corresponding to the target type required in the prompt word according to a pre-saved correspondence between different target types and candidate mask maps; and in each target candidate mask map, select any target candidate mask map as the second mask map.

[0169] In a possible implementation, the determination module 1202 is specifically used to determine the size of the first detection frame corresponding to the target to be replaced in the basic image, and the size of the second detection frame corresponding to the target included in each target candidate mask map; determine the second mask map based on the deviation between the size of the first detection frame and the size of each second detection frame and a preset size threshold.

[0170] In a possible implementation, the determination module 1202 is specifically used to determine the coordinate information of the center point of the target to be replaced included in the original basic image; the center point of the target identified in the second mask image is overlapped with the pixel point corresponding to the coordinate information in the basic image, and the image area mapped by the target identified in the second mask image in the basic image is determined as the area to be replaced.

[0171] In a possible implementation, the generation module 1202 is specifically configured to, upon receiving the number of generated images input by the user, call the image generation model based on the basic image for the number of generated images to obtain the number of generated images.

[0172] Based on the same inventive concept, an embodiment of the present application further provides an electronic device, Fig.13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Fig.13 As shown, it includes: a processor 1301, a communication interface 1302, a memory 1303 and a communication bus 1304, wherein the processor 1301, the communication interface 1302, and the memory 1303 communicate with each other through the communication bus 1304;

[0173] The memory 1303 stores a computer program. When the program is executed by the processor 1301, the processor 1301 executes the steps of the large model-based image generation process inspection method described in the above embodiments.

[0174] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface 1302 is used for communication between the above electronic device and other devices. The memory may include a random access memory (RAM) and may also include a non-volatile memory (NVM), such as at least one disk storage. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0175] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it can also be a digital signal processing processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0176] On the basis of the above embodiments, an embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program executable by a processor. When the program runs on the processor, the processor implements the steps of the large model-based image generation method described in the above embodiments.

[0177] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0178] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0179] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0180] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0181] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A terminal, characterized in that: The terminal includes a display and a processor: The display is configured to: display a generated image generated based on a basic image; The processor is configured to: Get the base image for image generation; Determining contour information of the object to be replaced included in the basic image; The basic image, the contour information and the prompt word are input into the image generation model. The image generation model replaces the target to be replaced corresponding to the area to be replaced in the basic image with other targets that meet the requirements of the contour information according to the requirements of the prompt word, and obtains the generated image output by the image generation model.

2. The terminal according to claim 1, characterized in that The processor is specifically configured to: Determine the position information of the target to be replaced in the basic image; The position information and the base image are input into an image segmentation model, the image segmentation model identifies the target to be replaced included in the image area corresponding to the position information, determines the contour of the target to be replaced, and outputs a first mask image corresponding to the base image, wherein the contour of the target to be replaced is marked in the first mask image; The basic image, the first mask image and the prompt word are input into the image generation model. The image generation model generates a replacement target that meets the requirements of the prompt word according to the outline of the mark in the first mask image, and replaces the target to be replaced with the replacement target in the area to be replaced in the basic image to obtain a generated image.

3. The terminal according to claim 1, characterized in that: The processor is further configured to: Eliminating the target to be replaced in the basic image based on a target elimination algorithm to obtain a target-free image, and using the target-free image to update the basic image; The determining of the contour information of the target to be replaced included in the basic image includes: According to the pre-saved correspondence between different target types and candidate mask images, a second mask image corresponding to the target type required in the prompt word is determined, and the contour of the target identified in the second mask image is determined as the contour information of the target to be replaced.

4. The terminal according to claim 3, characterized in that: The processor is specifically configured to: Determine each target candidate mask map corresponding to the target type required in the prompt word according to the pre-saved correspondence between different target types and candidate mask maps; In each target candidate mask map, any target candidate mask map is selected as the second mask map.

5. The terminal according to claim 4, characterized in that: The processor is specifically configured to: Determine the size of a first detection frame corresponding to the target to be replaced in the base image, and the size of a second detection frame corresponding to the target included in each target candidate mask image; The second mask image is determined according to a deviation between a size of the first detection frame and a size of each second detection frame and a preset size threshold.

6. The terminal according to claim 3, characterized in that: The processor is specifically configured to: Determine the coordinate information of the center point of the object to be replaced included in the original basic image; The center point of the target identified in the second mask image is overlapped with the pixel point corresponding to the coordinate information in the basic image, and the image area mapped by the target identified in the second mask image in the basic image is determined as the area to be replaced.

7. The terminal according to claim 1, characterized in that: The processor is further configured to: If the number of generated images input by the user is received, the image generation model is called the number of generated images times based on the basic image to obtain the number of generated images.

8. A method for generating an image based on a large model, characterized in that: The method comprises: Get the base image for image generation; Determining contour information of the object to be replaced included in the basic image; The basic image, the contour information and the prompt word are input into the image generation model. The image generation model replaces the target to be replaced corresponding to the area to be replaced in the basic image with other targets that meet the requirements of the contour information according to the requirements of the prompt word, and obtains the generated image output by the image generation model.

9. An image generation device based on a large model, characterized in that: The device comprises: An acquisition module, used for acquiring a basic image for image generation; A determination module, used to determine the contour information of the object to be replaced included in the basic image; A generation module is used to input the basic image, the contour information and the prompt word into the image generation model. The image generation model replaces the target to be replaced corresponding to the area to be replaced in the basic image with other targets that meet the requirements of the contour information according to the requirements of the prompt word, and obtains the generated image output by the image generation model.

10. An electronic device, characterized in that: The electronic device comprises a processor, and the processor is used to implement the steps of the large model-based image generation method as claimed in claim 8 when executing a computer program stored in a memory.

Images