Image generation method and system based on object semantics and medium

Abstract

The invention provides an image generation method and system based on object semantics and a medium, and aims to generate a new image according with an expected effect by intelligently processing an original image and a design prompt word provided by a user. The method comprises the following steps: firstly, identifying an object with independent semantics from an original image by using a target object extraction module, and generating a corresponding contour mask; then, the design cue words are sent into a semantic analysis model, and design parameters containing main body object information and template effect information are obtained; then, according to the parameters, an image generation engine creates a series of local images, and weight sorting is carried out according to the correlation degree of each local image and the corresponding target object; and finally, combining a logic filtering function and a pyramid mapping function to fuse the local image into the original image, and optimizing the quality of the synthesized image through an image enhancement model. According to the method, the image layout and the user design intention can be efficiently combined, and a flexible and high-quality image creation solution is provided for the fields of interior design and the like.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent generation of computer images, and in particular relates to an image generation method, system and medium based on object semantics. Background Art

[0002] In recent years, with the rapid development of artificial intelligence technology, image generation technology based on deep learning has become a research hotspot. However, traditional generative models have certain limitations in practical applications, especially in scenarios such as interior design rendering that require precise layout control. Current methods are often limited to subtle texture or color changes, and it is difficult to generate diverse designs while keeping the layout unchanged. In addition, many existing methods require a large number of data sets with precise semantic labels, which increases the time and resource consumption of data preparation. Therefore, in image generation tasks that require precise layout control and retain semantic consistency, there is an urgent need for a technical solution that can efficiently extract object semantic information and deeply combine it with layout patterns to provide more flexible generation capabilities and lower application barriers. Summary of the invention

[0003] In order to solve the problems of the prior art, the present invention proposes an image generation method, system and medium based on object semantics.

[0004] Specifically, the present application relates to an image generation method based on object semantics, characterized in that it comprises the following steps: S1: A user inputs an original image to be processed and a design prompt word, and a target object list in the image is extracted according to the original image to be processed, wherein the information of each target object in the target object list includes an object with independent semantics corresponding to the original image to be processed and an image mask identifying a layout area of ​​the object; Inputting the design prompt words into a semantic analysis model to obtain target design parameters, wherein the target design parameters include subject object information and matching template effect information; S2: Input the target object list and target design parameters into an image generation engine, and the image generation engine preliminarily generates a corresponding target effect image, wherein the target effect image is composed of m local effect images with disconnected layouts, m<=n, where n is the length of the target object list, and each local effect image is mapped to a unique target object in the target object list, and then the target effect image is merged into the original image according to the corresponding layout area; S3: the image generation engine further determines the weights of the local effect images according to the target design parameters, and performs weight sorting; The sorted weight matrix data is used as the model parameters of the image enhancement model, the fused image and the target design parameters are input into the image enhancement model, the image enhancement model performs image enhancement on each of the local effect images in the fused image, and the enhanced image is used as the target generated image.

[0005] Preferably, the semantic analysis model is obtained by training based on a plurality of design sample data sets, wherein each sample in the design sample data set comprises an original image, a design keyword and a template effect image; Trigger updates to the semantic analysis model based on cycle rules or user feedback after review.

[0006] Preferably, the image generation engine draws local effect images corresponding to the first n target objects with the highest correlation with the main object information in the target design parameters according to the target design parameters, where the value of n is the empirical effect value in the debugging stage of the image generation engine.

[0007] Preferably, the image mask is a contour mask of the object, and the fusion of each local effect image is specifically image stitching according to the centroid of the contour mask.

[0008] Preferably, when n=1, a single data element value in the weight matrix data is multiplied by a preset adjustment factor.

[0009] Preferably, in the parameters of the image enhancement model, the effect of the weight matrix data includes: model performance, image feature combination degree and correlation degree of regional position.

[0010] Preferably, the fusion generation method of each local effect image in step S1 is:

[0011] in, It is a local effect image. is with The image corresponding to the layout area of ​​the mapped target object, is the image after local fusion, is the logical filter function, L is adaptation Dimensional pyramid mapping function.

[0012] Preferably, in step S3, the weight matrix data is obtained by sorting in the following manner:

[0013] Among them, Sort is the sorting function, is the normalized number of pixels of the nth target object in the original image, It is the normalized value of the correlation between the nth target object and the main object information in the target design parameters.

[0014] The present application also proposes an image generation system based on object semantics, which is characterized by comprising: Target object extraction module: extracts a target object list in the image according to the original image to be processed input by the user, wherein the information of each target object in the target object list includes an object with independent semantics corresponding to the original image to be processed and an image mask identifying the layout area of ​​the object; Semantic analysis model: the design prompt word input by the user is input into the semantic analysis model to obtain target design parameters, wherein the target design parameters include subject object information and matching template effect information; Image generation engine: input the target object list and target design parameters into the image generation engine, and the image generation engine preliminarily generates a corresponding target effect image, wherein the target effect image is composed of m local effect images with disconnected layouts, m<=n, where n is the length of the target object list, and each local effect image is mapped to a unique target object in the target object list; Image fusion module: fuses the target effect image into the original image according to the corresponding layout area; Weight calculation module: the image generation engine also determines the weight of the local effect image according to the target design parameters and performs weight sorting; Image enhancement model: The weight matrix data obtained by sorting is used as the model parameters of the image enhancement model, the fused image and the target design parameters are input into the image enhancement model, the image enhancement model performs image enhancement on each of the local effect images in the fused image, and the enhanced image is used as the target to generate the image The present application also relates to a computer-readable storage medium, on which a program code is stored. When the program code is executed by a processor, the steps of the above-mentioned method for generating an image based on object semantics are executed.

[0015] The beneficial technical effects of the patent of the present invention include: in the demand for image creation and generation based on the input image and prompt words, on the one hand, the object layout information of the image itself is extracted, and on the other hand, the design prompt words input by the user are semantically analyzed to obtain the target design parameters containing the main object information and the matching template effect information, and these two aspects of information are input into the image generation engine. The correlation between the semantics and the object layout in the original image can be considered in accordance with the semantic instructions, so that the subsequent target effect image can better meet the layout requirements. In addition, the target effect image contains multiple local effect images. The target effect image is a subset of the target object from the perspective of the layout set. Since the actual number of local effect images depends on the empirical effect value of the image generation engine in the training and debugging stage, the image generation is more flexible and controllable. In addition, the application of weight matrix data makes the processing of the image enhancement stage with higher computing power requirements more purposeful, reduces unnecessary regional processing, and makes the generated image more in line with the expected effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 : Framework diagram of the method flow according to an embodiment of the present invention.

[0017] Figure 2 : A semantic model generation diagram according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] Figure 1 A method flow framework diagram of an embodiment of the present invention is shown. As shown in the figure, in an image generation method based on object semantics, a user inputs an original image to be processed and a design prompt word, and a list of target objects in the image is extracted based on the original image to be processed. The information of each target object in the target object list includes an object object with independent semantics corresponding to the original image to be processed, and an image mask identifying the layout area of ​​the object object.

[0020] In some specific implementations, the user first uploads an original image to be processed, such as a photo of an interior design. At the same time, the user also needs to provide a design prompt, such as "install a modern minimalist style chandelier". After receiving this information, the system will use a pre-trained target object extraction model to identify and extract key objects and their layout areas in the image. For example, in a photo of a living room, elements such as a sofa, a coffee table, and a TV wall may be identified, and a corresponding contour mask (i.e., an image mask) is generated for each element for subsequent operations.

[0021] The design prompt words are input into a semantic analysis model to obtain target design parameters, wherein the target design parameters include subject object information and matching template effect information.

[0022] In some specific implementations, the design prompt "install a modern minimalist style chandelier" will be fed into the semantic analysis model. The model is trained with a large number of sample data sets annotated with different design styles. It can understand the design intent of natural language and convert it into specific visual design parameters. Optionally, the model will output a set of template effect information that conforms to the "modern minimalist" style, including but not limited to main color preferences, material texture suggestions, and layout optimization suggestions. The "chandelier" is used as the main object information, together with the template effect information, as the target design parameter output of the semantic analysis model.

[0023] like Figure 2 As shown, in the process of generating a semantic analysis model, it is first necessary to obtain a design sample data set containing annotated data. Each sample in the data set includes an original image, a design keyword associated with it, and a corresponding template effect image. Optionally, the design keywords and template effect images are provided by professional designers or collected from public resources, and are manually reviewed to ensure their accuracy and consistency. The semantic analysis model adopts a deep learning architecture. In some embodiments, a Transformer architecture is used to capture the complex relationship between design prompt words. The model is divided into two main parts: an encoder and a decoder. The encoder is responsible for converting the input design prompt word into a fixed-length vector representation; the decoder generates target design parameters based on this vector, including main object information (such as "chandelier") and matching template effect information (such as color scheme, material texture, etc.).

[0024] Optionally, the update of the semantic analysis model is triggered according to periodic rules or user feedback after review. In some specific embodiments, in order to maintain the timeliness of the model, the semantic analysis model will be retrained regularly (e.g., quarterly) or when user feedback shows that the performance of the current model has deteriorated, which involves re-evaluating the validity of the existing data set and introducing new samples to reflect the latest design trends and technological advances.

[0025] The target object list and target design parameters are input into an image generation engine, and the image generation engine preliminarily generates a corresponding target effect image, wherein the target effect image is composed of m local effect images with disconnected layouts, m<=n, where n is the length of the target object list, and each local effect image is mapped to a unique target object in the target object list.

[0026] In some specific embodiments, using the target object list and design parameters obtained in the previous steps, the system will start the image generation engine to create a new design plan. Here, the image generation engine will generate a local effect image. If the "chandelier" in the design parameters does not have a direct matching object in the target object list, the image generation engine will obtain the closest target object according to the preset matching algorithm, such as matching the "ceiling" already in the target object list, and then combining the layout area of ​​the target object "ceiling" to implement the intelligent drawing of the chandelier with a "modern minimalist" style, and the drawing result is the local effect image. There can be multiple local effect images. In actual design, the user's design intention may cover the creation of multiple target objects, so the generated local effect image is mapped to the only target object in the target object list.

[0027] The image generation engine draws local effect images corresponding to the first n target objects with the highest correlation with the main object information in the target design parameters according to the target design parameters, where the value of n is the empirical effect value of the image generation engine in the debugging stage.

[0028] In some specific embodiments, the image generation engine will evaluate the correlation between these objects and the main object information of "chandelier", and the calculation of the correlation can be based on factors such as the spatial relationship and functional relevance between the objects.

[0029] The image mask is a contour mask of the object, and the fusion of each local effect image is specifically to perform image stitching according to the centroid of the contour mask.

[0030] In some specific implementations, in order to make the newly added effect image seamlessly integrated into the original image, a method based on the centroid of the contour mask is used for fusion. Specifically, for each local effect image, the geometric center (i.e., the centroid) of its corresponding contour mask is first calculated. Then, the local effect image is positioned and adjusted according to the centroid position using a logical filter function and a pyramid mapping function to ensure a smooth and natural transition between the new and old images. This method not only ensures the visual coherence of the synthesized image, but also maintains the integrity of the original image content.

[0031] Considering that when a local effect image is integrated into the original image, in addition to determining the region position, the newly generated part must ensure seamless connection with other parts of the original image. In some specific implementations, a method combining a logical filter function and a pyramid mapping function is used to achieve a smooth transition. Specifically, the target effect image is integrated into the original image according to the corresponding layout area, wherein each local effect image is generated by:

[0032] in, It is a local effect image. is with The image corresponding to the layout area of ​​the mapped target object, is the local fusion image, is the logical filter function, L is adaptation dimensional pyramid mapping function. In practical applications, the specific implementations of the logical filter function and the pyramid mapping function vary depending on the application scenarios and performance requirements. In some embodiments, the former adopts a simple weighted average or a more complex adaptive weighting strategy, while the latter may adopt different pyramid representation methods such as Gaussian pyramid and Laplacian pyramid. Combined with the logical filter function, the pyramid mapping function can adaptively adjust the fusion strategy according to the difference between the local effect image and the background image, which helps to maintain the consistency and coherence of the overall image, while allowing local details to match the surrounding environment more finely, thereby improving the realism of the synthesized image. In addition, from the test effect, the fusion processing in this way can preserve important visual features at different levels, so that even with small local changes, the key information in the original image can be well preserved, ensuring the integrity and semantic consistency of the image content.

[0033] Those skilled in the art will appreciate that there may be multiple local effect images. Since the importance of each local effect image in actual needs is not completely consistent, a focused differentiation process can be performed in subsequent image processing to improve overall computing performance. Therefore, in some specific implementations, the image generation engine also determines the weights of the local effect images according to the target design parameters and performs weight sorting:

[0034] Among them, Sort is the sorting function, is the normalized number of pixels of the nth target object in the original image, It is the normalized value of the correlation between the nth target object and the main object information in the target design parameters.

[0035] In practical applications, the specific implementation of the logical filter function and the pyramid mapping function varies depending on the application scenario and performance requirements. In some embodiments, the former adopts a simple weighted average or a more complex adaptive weighting strategy, while the latter may adopt different pyramid representation methods such as Gaussian pyramid and Laplacian pyramid. Combined with the logical filter function, the pyramid mapping function can adaptively adjust the fusion strategy according to the difference between the local effect image and the background image, which helps to maintain the consistency and coherence of the overall image, while allowing local details to match the surrounding environment more finely, thereby improving the realism of the synthesized image. In addition, from the test effect, the fusion processing in this way can preserve important visual features at different levels, so that even with small local changes, the key information in the original image can be well preserved, ensuring the integrity and semantic consistency of the image content.

[0036] Those skilled in the art will appreciate that there may be multiple local effect images. Since the importance of each local effect image in actual needs is not completely consistent, a focused differentiation process can be performed in subsequent image processing to improve overall computing performance. Therefore, in some specific implementations, the image generation engine also determines the weights of the local effect images according to the target design parameters and performs weight sorting:

[0037] in, It is a local effect image. is with The image corresponding to the layout area of ​​the mapped target object, is the local fusion image, is the logical filter function, L is adaptation Dimensional pyramid mapping function; Weight calculation module: The image generation engine also determines the weight of the local effect image according to the target design parameters and performs weight sorting:

[0038] Among them, Sort is the sorting function, is the normalized number of pixels of the nth target object in the original image, is the normalized value of the correlation between the nth target object and the main object information in the target design parameters; Image enhancement model: The sorted weight matrix data is used as the model parameters of the image enhancement model, the fused image and the target design parameters are input into the image enhancement model, the image enhancement model performs image enhancement on each of the local effect images in the fused image, and the enhanced image is used as the target generated image.

[0039] The embodiment of the present disclosure provides a non-volatile computer storage medium, wherein the computer-readable storage medium stores program code; the computer storage medium stores computer-executable instructions, and the computer-executable instructions can execute the method steps described in the above embodiment.

[0040] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted by any appropriate medium, including: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above. The above-mentioned computer-readable medium may be contained in the above-mentioned electronic device; or it may exist alone without being assembled into the electronic device. The computer program code for performing the operation of the present disclosure may be written in one or more programming languages ​​or a combination thereof, and the above-mentioned programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, C++, and also conventional procedural programming languages—such as "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network including a Local Area Network (AN) or a Wide Area Network (WAN), or may be connected to an external computer.

[0041] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. Each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions. The units involved in the embodiments described in the present disclosure can be implemented by software or by hardware. Among them, the name of the unit does not constitute a limitation on the unit itself under certain circumstances.

[0042] The above introduces the preferred embodiments of the present invention, which is intended to make the spirit of the present invention clearer and easier to understand, but is not intended to limit the present invention. All modifications, substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection outlined by the claims attached to the present invention.

Claims

1. A method for generating an image based on object semantics, characterized in that: The steps include: S1: A user inputs an original image to be processed and a design prompt word, and a target object list in the image is extracted according to the original image to be processed, wherein the information of each target object in the target object list includes an object with independent semantics corresponding to the original image to be processed and an image mask identifying a layout area of ​​the object; Inputting the design prompt words into a semantic analysis model to obtain target design parameters, wherein the target design parameters include subject object information and matching template effect information; S2: Input the target object list and target design parameters into an image generation engine, and the image generation engine preliminarily generates a corresponding target effect image, wherein the target effect image is composed of m local effect images with disconnected layouts, m<=n, where n is the length of the target object list, and each local effect image is mapped to a unique target object in the target object list, and then the target effect image is merged into the original image according to the corresponding layout area; S3: The image generation engine also determines the weights of the local effect images according to the target design parameters, and sorts the weights. The weight matrix data obtained by sorting is used as the model parameters of the image enhancement model. The fused image and the target design parameters are input into the image enhancement model. The image enhancement model performs image enhancement on each of the local effect images in the fused image, and uses the enhanced image as the target generation image.

2. The method for generating an image based on object semantics according to claim 1, characterized in that: The semantic analysis model is obtained by training based on multiple design sample data sets, wherein each sample in the design sample data set includes an original image, a design keyword, and a template effect image; Trigger updates to the semantic analysis model based on cycle rules or user feedback after review.

3. The method for generating an image based on object semantics according to claim 1, characterized in that: The image generation engine draws local effect images corresponding to the first n target objects with the highest correlation with the main object information in the target design parameters according to the target design parameters, where the value of n is the empirical effect value of the image generation engine in the debugging stage.

4. The method for generating an image based on object semantics according to claim 1, characterized in that: The image mask is a contour mask of the object, and the fusion of each local effect image is specifically to perform image stitching according to the centroid of the contour mask.

5. The method for generating an image based on object semantics according to claim 1, characterized in that: When n=1, the value of a single data element in the weight matrix data is multiplied by a preset adjustment factor.

6. The method for generating an image based on object semantics according to claim 1, characterized in that: In the parameters of the image enhancement model, the effect of the weight matrix data includes: model performance, image feature combination degree and correlation degree of regional position.

7. The method for generating an image based on object semantics according to claim 1, characterized in that: The fusion generation method of each local effect image in step S1 is: in, It is a local effect image. is with The image corresponding to the layout area of ​​the mapped target object, is the local fusion image, is the logical filter function, L is adaptation Dimensional pyramid mapping function.

8. The method for generating an image based on object semantics according to claim 1, characterized in that: In step S3, the weight matrix data is obtained by sorting in the following manner: Among them, Sort is the sorting function, is the normalized number of pixels of the nth target object in the original image, It is the normalized value of the correlation between the nth target object and the main object information in the target design parameters.

9. An image generation system based on object semantics, characterized in that: include: Target object extraction module: extracts a target object list in the image according to the original image to be processed input by the user, wherein the information of each target object in the target object list includes an object with independent semantics corresponding to the original image to be processed and an image mask identifying the layout area of ​​the object; Semantic analysis model: the design prompt word input by the user is input into the semantic analysis model to obtain target design parameters, wherein the target design parameters include subject object information and matching template effect information; Image generation engine: input the target object list and target design parameters into the image generation engine, and the image generation engine preliminarily generates a corresponding target effect image, wherein the target effect image is composed of m local effect images with disconnected layouts, m<=n, where n is the length of the target object list, and each local effect image is mapped to a unique target object in the target object list; Image fusion module: fuses the target effect image into the original image according to the corresponding layout area; Weight calculation module: the image generation engine also determines the weight of the local effect image according to the target design parameters and performs weight sorting; Image enhancement model: The sorted weight matrix data is used as the model parameters of the image enhancement model, the fused image and the target design parameters are input into the image enhancement model, the image enhancement model performs image enhancement on each of the local effect images in the fused image, and the enhanced image is used as the target generated image.

10. A computer-readable storage medium having program codes stored thereon, wherein the program codes are executed by a processor to execute the steps of the method for generating an image based on object semantics according to any one of claims 1 to 6.

Images