Method for generating images on the basis of user preference analysis

Abstract

The present solution relates to methods for generating images. The steps of the present method include preliminary training of a neural network and textual tagging of elements on an image by loading a dataset consisting of a plurality of images of a given element, with links being established to an ontology. The images are input to a CLIP neural network for tagging, where a parallel analysis of an image and a textual description is performed, resulting in the creation of a feature vector. Analysis of the loaded images is carried out by virtue of the segmentation of the images into a plurality of iterations by the isolation of large objects. A mask with a segmented image is produced, then the isolated objects are classified by the neural network. At the same time as the objects are identified, the image is tagged accordingly. In a final step, images are generated from an existing library of images, wherein each of the images has a set of tags that determine whether it can be used in a particular composition. The invention makes it possible to generate images using machine learning algorithms and to improve the quality of image generation.

Description

[0001] METHOD OF IMAGE GENERATION BASED ON USER PREFERENCE ANALYSIS

[0002] AREA OF TECHNOLOGY

[0003] This technical solution relates to the field of computer technology, in particular, to methods of generating images.

[0004] LEVEL OF TECHNOLOGY

[0005] A solution known from the prior art, selected as the closest analogue, is US 2012231425 (A1), published 13.09.2012. The solution characterizes a method for providing recommendations for landscape design, including: obtaining a captured image of a landscape area for landscape design; creating a catalog of data related to the landscape area, wherein the catalog contains information about the location in the landscape area, wherein the information is based on the natural characteristics of the location in the landscape area; obtaining information about a product from a mobile device, wherein the products must be built into the landscape area;matching product information with a catalog of data relating to locations in a landscape area by using a processing device, wherein the matching of the product with a location in the landscape area is based, at least in part, on natural characteristics of the location in the landscape area and natural characteristics of the product; and presenting products and indicators related to products in a real-time video stream on a mobile device, at a recommended location in the landscape area, the recommended location being based, at least in part, on the matching of product information with a catalog of data relating to a location within the landscape area.

[0006] The proposed technical solution is aimed at eliminating the shortcomings of the current level of technology and differs from previously known ones in that the proposed solution generates landscape design images based on the vector of user preferences in a high-quality and resource-intensive manner.

[0007] ESSENCE OF THE INVENTION

[0008] The technical task, to which the claimed solution is directed, is the creation of a method for generating images. The technical result consists in the implementation of the purpose of the claimed method.

[0009] The claimed technical result is achieved by implementing a method for generating images, which includes the following stages: initial training of the neural network and textual marking of elements in the image are carried out by loading a dataset of several images of one element, while building links with the ontology; images are fed to the input of the CLIP (Contrastive Language-Image Pre-training) neural network for marking, where parallel analysis of the image and text description is carried out, due to which a feature vector is created containing a text description of the image; analysis of the loaded images is carried out by segmenting the image in several iterations, by selecting large objects, and, if necessary, selecting smaller segments of the image; due to the previous stage, a mask with a segmented image is formed, where the pixels of one segment are marked with the same mark and form a connected area;then, using a neural network, the classification of the selected objects is carried out; in parallel with the definition of objects, the image is tagged; at the final stage, images are generated based on the existing image library, with each image supplied with a set of tags that determine the possibility of its use in a certain composition, and the process of image generation consists of the following steps: a factual description of the picture is compiled; a coordinate grid is formed, to the nodes of which objects are linked; a pool of options is generated, where the initial data is a set of objects that must be placed on the grid, and the generated placement option is checked for compliance using a rule base.

[0010] DESCRIPTION OF DRAWINGS

[0011] The implementation of the invention will be described further in accordance with the attached drawing, which is presented to explain the essence of the invention and in no way limits the scope of the invention. The following drawing is attached to the application:

[0012] Fig. 1 illustrates the processes in the initial learning block.

[0013] Fig. 2 illustrates the image analysis block. Fig. 3 illustrates the block for demonstration and work with text data.

[0014] Fig. 4 illustrates the generation block.

[0015] Fig. 5 illustrates a functional model of an intelligent decision support system with image generation based on user preference analysis.

[0016] Fig. 6 illustrates the data flow model between the main elements of the system.

[0017] DETAILED DESCRIPTION OF THE INVENTION

[0018] In the following detailed description of the embodiment of the invention, numerous implementation details are set forth in order to provide a clear understanding of the present invention. However, it will be apparent to one skilled in the art how the present invention may be used with or without these implementation details. In other instances, well-known methods, procedures, and components have not been described in detail in order not to unnecessarily obscure the features of the present invention.

[0019] In addition, it will be clear from the above description that the invention is not limited to the embodiment shown. Numerous possible modifications, changes, variations and substitutions, preserving the essence and form of the present invention, will be obvious to those skilled in the art.

[0020] This technical solution relates to the field of photorealistic image generation using machine learning algorithms in the field of design, in particular, in the field of landscape design and landscaping. The solution can be used both independently and with the use of a hardware complex with a camera, which ensures quick decision-making on the formation of landscape objects.

[0021] Text-to-picture image generation services such as Midjourney, Dall-E can create landscape design images, but they do not take into account the user's preferences, their previous choices, the parameters of the zone in which the object should be placed, and do not take into account what objects should be placed. It should also be noted that the approach proposed in the solution to combine deep analysis of user preferences based on emotional and subconscious choice with the generation of images based on this analysis is used for the first time.

[0022] Currently, there are no solutions that generate landscape design images based on a vector of user preferences obtained from the analysis of graphical information, input factual data and unstructured text.

[0023] There are several known approaches to image generation.

[0024] Generative adversarial network (GAN).

[0025] "GAN" consists of two networks:

[0026] 1. A generator capable of generating images from an input noise vector;

[0027] 2. A discriminator that distinguishes between a real picture and a "fake" one.

[0028] Generation models using "GAN" have several significant drawbacks:

[0029] High demands on computing power;

[0030] Low generation rate;

[0031] The problem of "overtraining" - a situation is possible when the generator will output practically identical images to the discriminator.

[0032] In this case, the use of the reverse diffusion method turned out to be more preferable. This method is used by: DALLE 2, Midjourney, Imagen, etc.

[0033] Diffusion models are used to create data similar to the data they are trained on.

[0034] Training the diffusion model:

[0035] 1. An image of a real or drawn object is used;

[0036] 2. Gaussian noise is added to the extent necessary so that the final image has the required number of similar features to the original;

[0037] 3. The model is trained to suppress noise to create the result.

[0038] The Stable Diffusion network created by StabilityAI (https: / / stablediffusionweb.com) was tested. Stable Diffusion is based on the work on high-resolution image synthesis using latent diffusion models by Rombach et al.

[0039] The complete architecture of stable diffusion consists of three models:

[0040] 1. A text encoder that accepts a text prompt and provides text-to-feature vector transformations; 2. U-Net, a diffusion model responsible for image generation. The architecture is shown in Figure 19.

[0041] 3. Variational autoencoder consisting of an encoder and decoder model.

[0042] The encoder is used to reduce the image dimensions that the UNet diffusion model operates on. The decoder is then responsible for restoring the image generated by the diffusion model to its original size.

[0043] Due to the iterative nature of the diffusion process, the training and generation process is more stable than in GANs. In diffusion models, the transitions from pure noise to an image are made in several steps, while in GANs, it is done in one step, which is one of the sources of unstable GAN training.

[0044] The proposed technical solution allows to reduce the analysis of preferences several times, while additionally, ready-made photorealistic images are generated at the output, as well as additional documentation necessary for landscape design projects.

[0045] An important feature of the declared technical solution is the possibility of using it both as an independent system and together with cameras. In the latter case, generation is possible immediately after photographing the land plot.

[0046] The advantage of the declared technical solution is the digitalization of processes for determining user preferences to create unique design projects of the environment, according to the preferences and needs of the user in such complex areas as various types of design (interior, landscape, design of things, architectural), where a significant role is played by the choice based on subconscious decisions and unstructured information. For the development of models and further testing, the direction of landscape design and improvement was chosen, which is one of the most difficult in terms of creating full-fledged projects. The developed solution can be applied to various areas of design of things and environments.

[0047] The standard approach to creating a landscape design, when a specialist designer works, is to show the client images with various finished projects, photographs of gardens, flower beds, etc., as well as conducting surveys: what plants do they like, what design elements should be, what their location should be on the site, etc. The designer analyzes the information received, and based on the user's visual preferences, as well as on professional knowledge regarding norms and rules, the specialist creates a project that should be unique for each user.

[0048] However, in the present invention, methods for fast and automatic evaluation of user preferences using various information technologies were used, and an original evaluation model based on the analysis of the displayed images was proposed.

[0049] Let us consider in more detail the main blocks with which the stated method is implemented.

[0050] 1. Determining or assessing user preferences;

[0051] 2. Generating images based on specific user preferences.

[0052] The user preference model consists of two complementary analysis loops:

[0053] 1. Analysis of stylistic preferences (general style of the landscape design, styles of individual elements of landscape design (LD));

[0054] 2. Analysis of preferences of object types.

[0055] The basis of the analysis is:

[0056] • Use of marked photographs of various LD styles;

[0057] • Using object libraries;

[0058] • Use of a special ontology;

[0059] • Trained convolutional neural network.

[0060] Labeled photographs of styles and individual objects allow the system to learn to “understand” which objects specifically arouse the user’s interest.

[0061] The model's operation is based on the following blocks:

[0062] 1. Initial training block;

[0063] 2. LD image analysis unit;

[0064] 3. Block for demonstration and work with text data.

[0065] Let's look at them in more detail.

[0066] Initial training block.

[0067] In this block, the initial training of the neural network and text markup (tagging) of the LD elements are carried out. LD elements are understood to be any objects of the classes of plants, small architectural forms, etc. In this case, the element can be composite, for example, an object of the "flowerbed" type must be recognized in complex LD images, while it must be separately recognized by its constituent parts.

[0068] The process of recognizing elements.

[0069] A dataset of several photographs of one element is loaded (at least 100, prepared datasets from iNaturalist or ImageNet are used). In this case, a connection with the ontology must be built - i.e. when loading, it is recorded to which class these specimens belong. For this model, its own ontology was developed, which describes all elements and objects of landscape design, with all possible properties and attributes. For different types of objects, lists of possible attributes (features) are formed. Objects and features are parts of the ontology.

[0070] Images are fed to the input of the neural network.

[0071] A neural network called CLIP (Contrastive Language-Image Pre-training) is used for marking, which allows to significantly speed up the process of pre-processing of graphic objects and their classification (tagging). The main principle of operation is parallel analysis of both the image and the text description, which allows to create a feature vector containing a text description of the image (flower, white inflorescence, smooth stem, dark green, arrow-shaped leaves). In fact, the automation of image classification occurs and the image is tagged. In our case, the received tags are checked against the ontology and thus a specific photograph is linked as an instance of the class.

[0072] LD image analysis unit.

[0073] This block analyzes complex composite images using a neural network previously trained on a library of elements.

[0074] Algorithm of actions.

[0075] LD's photo is automatically loaded.

[0076] Next, the image is segmented, and in several iterations - i.e. first, large objects are selected, for example, houses, small architectural forms, LD objects, and only then, everything is segmented into smaller segments. As a result of the algorithm, we get a mask with a segmented image, where the pixels of one segment are marked with the same label and form a connected area. The algorithm shows itself excellently for solving the problem of primary selection of the boundaries of objects of interest in the image (Watershed boxing). By varying the initial settings of the algorithm, including the sensitivity threshold, as well as restrictions on the size of objects, you can get a relevant selection of objects for further processing.

[0077] Next, using a neural network, the classification of the selected objects is carried out from the initial training block. If the object is not recognized automatically, it must be selected and sent to the analyst for manual classification and saving in the element database.

[0078] In parallel with the definition of objects, image tagging is carried out - i.e., automatic addition of service information to the style image.

[0079] Thus, at the input of the block is an image with LD, and at the output is a vector of image features, including:

[0080] LD style (determined based on training results);

[0081] A list of found objects, with an inclusion hierarchy (i.e. if it is a flower, then it is part of a flowerbed, and so on);

[0082] Properties of objects.

[0083] Description of the image generation model.

[0084] The following generation model is proposed, the main processes of which are illustrated in Figure 4.

[0085] A consistent approach to working with generation is used:

[0086] 1. Simplified (fast) generation;

[0087] 2. Fully functional generation.

[0088] Fast generation.

[0089] 1. With a sufficient set of data, the generation parameters are set: Style;

[0090] Object types;

[0091] Number of objects;

[0092] Properties of objects (class, attributes);

[0093] Locations of objects.

[0094] 2. Next, using a neural network, a plan for the location of objects is created taking into account the specified parameters.

[0095] The generation of images is based on the existing image library. Each image is provided with a set of tags that determine the possibility of its use in a certain composition. The process of image construction consists of the following steps: 1. A factual description of the painting is compiled, which includes:

[0096] - composition style;

[0097] - a set of mandatory objects (and their number);

[0098] - a set of preferences that influence the rules for placing objects.

[0099] 2. A coordinate grid is formed, to the nodes of which objects will be bound. The choice of the grid size affects, on the one hand, the speed of the algorithm, and on the other hand, the visual effects

[0100] 3. A pool of variants is generated. The initial data is a set of objects that need to be placed on the grid. The generated variant of placement is checked for compliance with the base of rules selected from the database in relation to the composition. The rules can be expressed both in declarative and in fuzzy form, allowing for a vague interpretation.

[0101] 4. To speed up the process of selecting options that satisfy the rules, a genetic algorithm implementation is used.

[0102] The generation is based on rules of type C, in particular C21 ('Conifers of the same species are not desirable next to each other'), which is set with 80% accuracy. The parameters of the minimum and maximum distance between flowers and trees set for the coordinate grid also determine the nature of the placement of objects

[0103] To more accurately satisfy the user's request, it is possible to use a background in the form of previously uploaded photographs during generation.

[0104] After the user stops the demonstration, the full-featured generation mechanism is enabled for the images he selected that were created using simple generation.

[0105] System architecture.

[0106] 1. Functional model of the system.

[0107] The models considered for determining user preferences and generating images represent a system of interconnected functional components. To build a conceptual vision of the system's operation, organize the vision of information flows, a functional model of the system's operation is formulated by analogy with the IDEF0 notation.

[0108] The control effect on all functional modules is exerted by elements of ontology and landscape design rules, as well as marked images with a vector of assigned features.

[0109] The data flow model of the system consists of the following elements: ■ entities of the designed system, including technical functional modules, as well as modules interacting with the external environment;

[0110] ■ data storage and storage devices (databases);

[0111] ■ data flows between entities;

[0112] • processes;

[0113] ■ subjects involved in the processes;

[0114] ■ external objects (devices).

[0115] A special diffusion model with a condition was trained to generate landscape design images according to user preferences. To train the model, 102,000 landscape design images were collected from open sources. These images were reduced to a single size (512x512 px) and analyzed by the Detector to extract the feature vector. During the training process, information about image features and the images themselves were simultaneously fed to the inputs of the diffusion model. The stable diffusion technology (diffusers library) was selected to generate landscape design images. To speed up the generation process, a hidden layer is prepared by the Linet family model, from which the autoencoder then restores the target image. Since the system generates by condition, the Unet2dConditionModel model was selected.

[0116] Training a diffusion model specifically for generating landscape design images reduced image generation time from 3.5 s to 1 s on an Nvidia T4 GPU, and increased the diversity of objects and scenes in the generated images.

[0117] The solution allows detecting plants in a generated image and selecting similar images in the service database, taking into account winter hardiness, lighting and budget for subsequent ordering from nurseries.

[0118] Algorithm for plant selection.

[0119] 1. Upon receiving a selection request from the main operating unit, an http request is sent to the detection service.

[0120] 2. The detection service, having received the specified request, analyzes the generated image and returns a binary stream of serialized data: the class label of the detected plant, the coordinates of the bounding box, and the graphic content itself. 3. The plant selection service, having received data from the detection service, extracts the feature vectors of the plant images (embeddings) and compares them with the plant vectors from the service database, calculating the cosine distance between them. In order to extract image embeddings, a machine learning model was pre-trained - an autoencoder of the DINOv2 architecture. This model was trained on images used to train the image detection service and subsequently processed by the detection service.

[0121] 4. Information about the plants selected in this way is presented to the user.

[0122] The generated images are saved to the S3 object storage, and information about them is entered into the Database for further use in plant selection, angle generation and embedding of custom objects into the image.

[0123] The resulting diffusion machine learning model copes with the task of generating an image on the Nvidia P100 video accelerator in 1.2 s and takes up 1.5 GB of video memory, which is significantly lower than the performance of standard general-purpose Stable Diffusion family models. The quality of the resulting images is almost identical to photographs of real objects. The service has information for connecting to a third-party 53-storage and performs pre-processing and subsequent sending of images to the storage, as well as sending a request to register information about the image in the Application Database. In the course of its work, the service uses the object detection service to obtain information about the loaded image, as well as the application backend to save this information. This service, in turn, uses the application generation service to save the results of its work.

[0124] A computing system capable of providing data processing necessary for implementing the claimed solution generally contains the following components: one or more processors, at least one memory, a data storage facility, input / output interfaces, an input facility, and network interaction facilities.

[0125] When executing machine-readable commands contained in the RAM, the device processor is configured to perform the basic computing operations necessary for the operation of the device or the functionality of one or more of its components. The memory is usually implemented in the form of RAM, where the necessary program logic is loaded, providing the required functionality. When implementing the work of the proposed solution, the memory volume necessary for implementing the proposed solution is allocated.

[0126] The data storage facility can be implemented as an HDD, SSD disks, RAID array, network storage, flash memory, etc. The facility allows for long-term storage of various types of information, such as the above-mentioned files with user / passenger data sets, databases containing records of time intervals measured for each user, user identifiers, etc.

[0127] Interfaces are standard means for connecting and operating peripheral and other devices, such as USB, RS232, RJ45, COM, HDMI, PS / 2, Lightning, etc.

[0128] The choice of interfaces depends on the specific design of the device, which can be a personal computer, mainframe, server cluster, thin client, smartphone, laptop, etc.

[0129] A keyboard may be used as a means of data input in any embodiment of the system implementing the described method. The keyboard hardware may be any known one: it may be either a built-in keyboard used on a laptop or netbook, or a separate device connected to a desktop computer, server or other computer device. The connection may be either wired, in which the keyboard connecting cable is connected to a PS / 2 or USB port located on the desktop computer system unit, or wireless, in which the keyboard exchanges data via a wireless communication channel, for example, a radio channel, with a base station, which, in turn, is directly connected to the system unit, for example, to one of the USB ports.In addition to the keyboard, the following may also be used as input means: joystick, display (touch display), projector, touchpad, mouse, trackball, light pen, speakers, microphone, etc.

[0130] Network interaction means are selected from a device that provides network reception and transmission of data, for example, an Ethernet card, WLAN / Wi-Fi module, Bluetooth module, BLE module, NFC module, IrDa, RFID module, GSM modem, etc. The means ensure the organization of data exchange via a wired or wireless data transmission channel, for example, WAN, PAN, LAN, Intranet, Internet, WLAN, WMAN or GSM.

[0131] The components of the device are connected via a common data transmission bus.

[0132] The present application materials present a preferred disclosure of the implementation of the claimed technical solution, which should not be used as limiting other, particular embodiments of its implementation that do not go beyond the scope of the requested scope of legal protection and are obvious to specialists in the relevant field of technology.

Claims

Formula 1. A method for generating images that includes the following stages: initial training of a neural network and textual marking of elements in an image are performed by loading a dataset of several images of one element, while building links with an ontology; images are fed to the input of a CLIP (Contrastive Language-Image Pre-training) neural network for marking, where parallel analysis of the image and text description is performed, due to which a feature vector containing a text description of the image is created; analysis of the loaded images is performed by segmenting the image in several iterations, by selecting large objects, and, if necessary, selecting smaller segments of the image; due to the previous stage, a mask with a segmented image is formed, where the pixels of one segment are marked with the same mark and form a connected area;then, using a neural network, the classification of the selected objects is carried out; in parallel with the definition of objects, the image is tagged; at the final stage, images are generated based on the existing image library, with each image supplied with a set of tags that determine the possibility of its use in a certain composition, and the process of image generation consists of the following steps: a factual description of the picture is compiled; a coordinate grid is formed, to the nodes of which objects are linked; a pool of options is generated, where the initial data is a set of objects that must be placed on the grid, and the generated placement option is checked for compliance using a rule base.

Images