Image search method and device based on diffusion model analysis and electronic device

Abstract

The invention discloses an image search method and device based on diffusion model analysis and an electronic device, and relates to the technical field of image.The image search method based on diffusion model analysis comprises the steps that a keyword text is determined from a request text input by a target object, and the request text is used for requesting for obtaining a target image; determining image generation description information according to the keyword text, and generating a corresponding reference image through a diffusion model based on the image generation description information; and determining the reference image as the target image under the condition that the similarity between any standard image in the plurality of standard images and the reference image meets a preset condition. The problem that the image cannot be accurately generated based on the text is solved.

Description

Technical Field

[0001] The present application relates to the field of image technology, and in particular to an image search method, device and electronic device based on diffusion model analysis. Background Art

[0002] In order to realize cross-modal retrieval technology based on text retrieval images, the Contrastive Language-Image Pre-Training (CLIP) technology is usually used, that is, a Transformer-based multimodal training model that can understand images and texts at the same time and perform contrastive learning on large-scale data sets to obtain high-quality cross-modal representations.

[0003] In the related art, due to the large number of parameters of the CLIP model, a large amount of computing resources and storage space are required for training and deployment, and the CLIP model has potential biases and limitations, resulting in inaccurate images generated by the CLIP model.

[0004] With regard to the problem in related technologies that images cannot be accurately generated based on text, no effective solution has yet been proposed. Summary of the invention

[0005] The embodiments of the present application provide an image search method, device and electronic device based on diffusion model analysis, so as to at least solve the problem in the related art that images cannot be accurately generated based on text.

[0006] According to an embodiment of the present application, an image search method based on diffusion model analysis is provided, including: determining a keyword text from a request text input by a target object, wherein the request text is used to request to obtain a target image; determining image generation description information according to the keyword text, and generating a corresponding reference image through a diffusion model based on the image generation description information; and determining the reference image as the target image when the similarity between any standard image among multiple standard images and the reference image meets a preset condition.

[0007] In an exemplary embodiment, determining image generation description information based on the keyword text includes: generating text content for describing the reference image based on the keyword text; and determining to generate the reference image through the diffusion model when the text content is successfully generated, wherein the diffusion model is used to generate an image based on the text content.

[0008] In an exemplary embodiment, text content for describing the reference image is generated based on the keyword text, including: determining multiple words included in the keyword text, and description information corresponding to the multiple words, wherein the description information is used to describe multiple elements forming the image; obtaining the description information of the multiple words; combining the multiple description information through the combination logic of preset language rules to obtain a text combination result, and using the text combination result as the text content.

[0009] In an exemplary embodiment, after determining that the reference image is generated by the diffusion model, the method further includes: converting the text content into a text vector by the diffusion model; performing noise removal processing on the text vector and the background image of the text content to obtain a target image; and decoding the target image to generate the reference image.

[0010] In an exemplary embodiment, when the similarity between any one of a plurality of standard images and the reference image satisfies a preset condition, the reference image is determined as the target image, including: obtaining the cosine similarity between the reference image and any one of the standard images; and when the cosine similarity is higher than a preset threshold, determining the reference image as the target image.

[0011] In an exemplary embodiment, obtaining the cosine similarity between the reference image and any one of the standard images includes: vectorizing the reference image to obtain a first eigenvector corresponding to the reference image, and vectorizing any one of the standard images to obtain a second eigenvector corresponding to the any one of the standard images; determining the cosine similarity between the first eigenvector and the second eigenvector; and determining the cosine similarity between the reference image and any one of the standard images using the cosine similarity between the first eigenvector and the second eigenvector.

[0012] In an exemplary embodiment, keyword text is determined from a request text input by a target object, including: obtaining the number of times each word in the request text appears in the request text; calculating the number of times each word appears in the request text by a weight algorithm to obtain a weight value for each word; and using a target word whose weight value is greater than a weight threshold as the keyword text.

[0013] In an exemplary embodiment, the number of times each word appears in the request text is calculated using a weight algorithm to obtain a weight value for each word, including: determining the total number of words in the request text; and taking the ratio of the number of times each word appears in the request text to the total number of words as the weight value for each word.

[0014] According to another embodiment of the embodiment of the present application, there is also provided an image search device based on diffusion model analysis, including: a first determination module, used to determine a keyword text from a request text input by a target object, wherein the request text is used to request to obtain a target image; a generation module, used to determine image generation description information according to the keyword text, and generate a corresponding reference image through a diffusion model based on the image generation description information; a second determination module, used to determine the reference image as the target image when the similarity between any standard image among multiple standard images and the reference image meets a preset condition.

[0015] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned image search method based on diffusion model analysis when running.

[0016] According to another aspect of an embodiment of the present application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the image search method based on diffusion model analysis through the computer program.

[0017] In an embodiment of the present application, a keyword text is determined from the request text input by the target object, and then image generation description information is determined based on the keyword text, and a corresponding reference image is generated through a diffusion model based on the image generation description information. After that, when the similarity between any standard image and the reference image among multiple standard images meets a preset condition, the reference image is determined as the target image. The above technical solution solves the problem of being unable to accurately generate images based on text. In an embodiment of the present application, the keyword text is first determined, the image generation description information is determined based on the keyword text, and a reference image is generated, and then the target image is determined based on the similarity calculation between any standard image and the reference image, so that the determined target image is more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] Figure 1is a schematic diagram of a hardware environment of an image search method based on diffusion model analysis according to an embodiment of the present application;

[0021] Figure 2 is a flow chart of an image search method based on diffusion model analysis according to an embodiment of the present application;

[0022] Figure 3 is a flow chart of an image search method based on diffusion model analysis according to an optional embodiment of the present application;

[0023] Figure 4 is a schematic diagram of an image search method based on diffusion model analysis according to an embodiment of the present application;

[0024] Figure 5 It is a structural block diagram of an image search device based on diffusion model analysis according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] According to one aspect of an embodiment of the present application, a method for image search based on diffusion model analysis is provided. The method for image search based on diffusion model analysis can be executed in a terminal device or a similar computing device, for example, running on a terminal device. Figure 1 FIG. 1 is a hardware structure block diagram of a terminal device of an image search method based on diffusion model analysis according to an embodiment of the present invention. Figure 1 As shown, the terminal device may include one or more ( Figure 1Only one is shown in the figure) processor 102 (processor 102 may include but is not limited to a microprocessor (Microprocessor Unit, referred to as MPU) or a programmable logic device (Programmable logic device, referred to as PLD)) and a memory 104 for storing data. In an exemplary embodiment, the terminal device may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal device. Figure 1 More or fewer components as shown, or with Figure 1 Equivalent functions or comparisons shown Figure 1 A different configuration with more features is shown.

[0028] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the image search method based on diffusion model analysis in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal device via a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0029] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the switching device. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0030] In this embodiment, an image search method based on diffusion model analysis is provided, which can be applied to the above-mentioned terminal device or configured in a server. The terminal device can be a PC or a mobile terminal. This embodiment of the application does not limit this. Figure 2is a flow chart of an image search method based on diffusion model analysis according to an embodiment of the present invention, the flow chart includes the following steps:

[0031] S202, determining a keyword text from a request text input by the target object, wherein the request text is used to request to obtain a target image.

[0032] In an embodiment of the present application, the following methods are provided for determining keyword text from a request text input by a target object, wherein the methods for determining keyword text include: 1. keyword extraction method; 2. text classification method; 3. relationship extraction method; 4. entity named entity extraction method; 5. machine learning algorithm extraction method; 6. extraction method for obtaining the number of keyword occurrences.

[0033] Among them, the first keyword extraction method can be understood as using natural language processing technology to segment and tag the request text, and then screening out keyword text related to the request text according to a set keyword list or stop word list.

[0034] The second text classification method can be understood as training a text classifier model for a specific field or question type, dividing the request text according to its category. The elements contained in the request text can be directly understood from the category, and the included elements are used as keyword text.

[0035] The third relationship extraction method can be understood as using NLP technology to find the semantic relationship between entities in the request text (such as "located in", "next to the subway entrance", etc.), and then analyze the connection between them, and use the semantics with more connections as keyword text.

[0036] The fourth entity named entity extraction method can be understood as using NLP technology to extract named entity information such as names of people, places, and organization names involved in the request text as keyword text.

[0037] The fifth method of machine learning algorithm extraction can be understood as analyzing and modeling the request text through natural language processing technology, and using the trained model to extract keyword text that is relevant to the request text, highly representative and distinctive.

[0038] The sixth method of obtaining keyword frequency extraction can be understood as obtaining the number of times each word in the request text appears in the request text. When the number of times each word appears in the request text is higher than a preset occurrence threshold, the target word higher than the preset occurrence threshold can be used as the keyword text.

[0039] S204, determining image generation description information according to the keyword text, and generating a corresponding reference image through a diffusion model based on the image generation description information.

[0040] In an embodiment of the present application, text content for describing a reference image is generated based on a keyword text, and when the text content is successfully generated, it is determined to generate a reference image through a diffusion model, wherein the diffusion model is used to generate an image based on the text content.

[0041] That is, after determining the keyword text from the request text, text content describing the reference image can be generated based on the keyword text, and then it is determined whether the text content is generated successfully. If it is determined that the text content is generated successfully, it is determined to generate the reference image through the diffusion model.

[0042] As a possible implementation method, the text content describing the reference image can be determined based on the keyword text and the text content library. For example, a first word can be determined based on the keyword text, and according to the correspondence between the first word and the text content library, a text content file corresponding to the first word can be called, and then the text content describing the reference image can be determined. The text content library includes multiple text content files, each of which is named after a word.

[0043] As another possible implementation, the key text may be input into a text content template to generate text content describing the reference image, wherein the information in the text content template includes but is not limited to characters, locations, time, weather, activity content, etc.

[0044] In one embodiment of the present application, it is possible to determine whether the text content is generated successfully based on voice. For example, when the terminal device issues a voice of "text content generation successful", it can be determined that the text content is generated successfully. When the terminal device issues a voice of "text content generation failed", it can be determined that the text content is generated unsuccessfully. It is also possible to determine whether the text content is generated successfully based on a flashing light. For example, when it is determined that the flashing light set on the terminal device is green, it can be determined that the text content is generated successfully. When it is determined that the flashing light set on the terminal device is red, it can be determined that the text content is generated unsuccessfully. It is also possible to determine whether the text content is generated successfully based on a pop-up window. For example, when it is detected that a pop-up window on the display interface of the terminal device displays "success", it can be determined that the text content is generated successfully. When it is detected that a pop-up window on the display interface of the terminal device displays "failure", it can be determined that the text content is generated unsuccessfully.

[0045] In another embodiment of the present application, in an embodiment in which text content describing a reference image is determined based on keyword text and a text content library, if the text content file corresponding to the first word is called successfully, it can be determined that the text content generation is successful, and if the text content file corresponding to the first group is called unsuccessfully, it can be determined that the text content generation has failed.

[0046] Among them, in an embodiment of the present application, a first word can be determined based on a keyword text, and according to the correspondence between the first word and a text content library, a text content file corresponding to the first word can be called, and then based on an identifier corresponding to the called text content file, it can be determined whether the calling of the text content file corresponding to the first word is successful. For example, when the identifier corresponding to the text content file is "1", it can be determined that the calling of the text content file corresponding to the first word is successful; when the identifier corresponding to the text content file is "0", it can be determined that the calling of the text content file corresponding to the first word has failed.

[0047] S206: When the similarity between any standard image among the plurality of standard images and the reference image satisfies a preset condition, the reference image is determined as the target image.

[0048] In the embodiment of the present application, after the reference image is generated, multiple standard images in the standard library can be obtained, and then each of the multiple standard images is compared with the reference image for similarity, and it is determined whether the similarity between any standard image and the reference image is higher than a preset threshold. If it is determined that the similarity is higher than the preset threshold, the reference image is determined as the target image. The specific implementation method can refer to the subsequent embodiments.

[0049] Through the above steps, the keyword text is determined from the request text input by the target object, and then the image generation description information is determined based on the keyword text, and the corresponding reference image is generated through the diffusion model based on the image generation description information, and then the reference image is determined as the target image when the similarity between any standard image and the reference image among multiple standard images meets the preset conditions. The problem in the prior art that it is impossible to accurately describe the image content or provide image samples when searching for images based on text is solved. The embodiment of the present application first determines the keyword text, determines the image generation description information based on the keyword text, and generates a reference image, and then determines that when any standard image and the reference image meet the preset conditions, the target image is determined, so that the determined target image is more accurate.

[0050] In order to make it easier for those skilled in the art to understand the technical solutions of the embodiments of the present application, Figure 3 is an image search method based on diffusion model analysis in one embodiment of the present application, such as Figure 3 As shown, the image search method based on diffusion model analysis includes:

[0051] S302, determining a keyword text from a request text input by the target object, wherein the request text is used to request to obtain a target image.

[0052] In one embodiment of the present application, the number of times each word in the request text appears in the request text is obtained, the number of times each word appears in the request text is calculated using a weight algorithm to obtain a weight value for each word, and the target word with a weight value greater than a weight threshold is used as a keyword text.

[0053] The total number of words in the request text may be determined, and the ratio of the number of times each word appears in the request text to the total number of words may be used as the weight value of each word.

[0054] The number of times each word appears in the text can be marked based on a marking method, thereby obtaining the number of times each word in the request text appears in the request text. For example, when word A appears in the text for the first time, the corresponding mark of word A is 1, and when the word appears in the text for the second time, the corresponding mark of the word is 2.

[0055] The total number of words in the request text may be determined based on a statistical method, for example, the total number of words in the request text may be determined based on counting the number of characters in the request text.

[0056] S304: Generate text content for describing the reference image according to the keyword text.

[0057] In an embodiment of the present application, multiple words included in the keyword text and descriptive information corresponding to the multiple words can be determined, wherein the descriptive information is used to describe multiple forming elements of the image, and descriptive information of multiple words is obtained. In order to accurately understand the user's intention or needs, the multiple description information is combined through the combination logic of preset language rules to obtain a text combination result, and the text combination result is used as the text content.

[0058] That is to say, by determining the keyword text, multiple words in the keyword text and the description information corresponding to the multiple words can be determined, and then based on the combination logic, the description information of the multiple words can be combined to generate a text combination result, and the text combination result can be used as the text content.

[0059] Among them, based on the word description library, when multiple words in the keyword text are determined, the word description library can be called to determine the description information corresponding to the word. For example, when the word is "sunshine", the description information corresponding to the sunshine in the word description library can be called, that is, the description information corresponding to the sunshine is determined to be light; the description information corresponding to the word can also be determined based on the characteristics of the word. For example, when the word is "football stadium", the characteristic of the football stadium can be determined to be the venue, and then the description information corresponding to the football stadium can be determined to be the venue.

[0060] The combination logic of the preset language rules includes grammatical combination logic, wherein the grammatical combination logic includes rules for combining morphemes into words and rules for combining words into sentences. For example, the description information is light, weather, puppy, and girl. Based on the grammatical combination logic, the three description information of light, weather, puppy, and girl can be combined to obtain a text combination result. That is, on a sunny day, the girl and the puppy play together, and "on a sunny day, the girl and the puppy play together" is used as the text content.

[0061] In another embodiment of the present application, multiple words included in the keyword text are determined, and the text content corresponding to the multiple words can be determined based on the text content library. For example, the keyword text includes seaside, beach, girl, waves, and shells. The five keyword texts of seaside, beach, girl, waves, and shells can be searched from the multiple text contents in the text content library, and then it can be determined that one of the multiple text contents includes the five keyword texts, and then the text content can be called, that is, the text content is an endless seaside, with waves hitting the rocks, and two little girls picking up shells on the beach.

[0062] S306: When the text content is generated successfully, determine to generate a reference image through a diffusion model, wherein the diffusion model is used to generate an image according to the text content.

[0063] In an embodiment of the present application, after determining that the diffusion model generates a reference image, the text content can be converted into a text vector through the diffusion model, and the text vector and the background image of the text content can be processed to remove noise to obtain a target image, and then the target image can be decoded to generate a reference image.

[0064] That is to say, the text content can be converted into a text vector through the conversion module in the diffusion model, and then the text vector and the background image of the text content can be denoised for multiple rounds based on the denoising module in the diffusion model to obtain the target image. After that, the decoder based on the diffusion model decodes the target image to generate a reference image.

[0065] Among them, the background image can be understood as a mosaic image.

[0066] In each round of denoising, based on the "text-to-image task" in the general framework of the calling task, the text vector and the background image of the text content are subjected to multiple rounds of denoising, and then multiple denoising results are obtained, wherein the denoising result represents the correlation between the text vector and the initial image, and finally based on the multiple denoising results, the denoising result greater than a certain threshold is used as the target denoising result, and the target image is determined based on the target denoising result. The general framework of the task includes the task of generating images through semantics, the task of generating images from text, the task of generating images from language descriptions, and the task of generating images from images.

[0067] In order to verify multiple rounds of denoising and to stop multiple rounds of denoising, in one embodiment of the present application, when calling the task of generating images from text, the "task of generating images from images" in the general framework of tasks can be called, wherein the initial image in the above embodiment is the image issued by the task of generating images from images. That is to say, when calling the "task of generating images from text" and performing multiple rounds of denoising on the text vector and the background image of the text content, the similarity between the multiple rounds of denoising results and the initial image can be determined, and when the similarity is greater than a certain threshold, the denoising operation can be terminated.

[0068] In one embodiment of the present application, the diffusion model is trained in the following manner: obtaining a training image, encoding the training image based on a pre-trained autoencoder to compress the training image into a low-dimensional representation, and then decoding the low-dimensional representation based on a pre-trained decoder to restore the low-dimensional representation.

[0069] S308: When the similarity between any standard image among the plurality of standard images and the reference image satisfies a preset condition, the reference image is determined as the target image.

[0070] In an embodiment of the present application, the cosine similarity between a reference image and any standard image is obtained; when the cosine similarity is higher than a preset threshold, the reference image is determined as the target image.

[0071] In order to evaluate the quality and relevance of target image generation, in an embodiment of the present application, a reference image is vectorized to obtain a first eigenvector corresponding to the reference image, and any standard image is vectorized to obtain a second eigenvector corresponding to any standard image; the cosine similarity between the first eigenvector and the second eigenvector is determined; when the cosine similarity is higher than a preset threshold, the reference image is determined as the target image.

[0072] That is to say, after determining to generate a reference image, multiple standard images in a standard image library can be obtained, and then the reference image and any standard image among the multiple standard images are respectively vectorized to obtain a first eigenvector corresponding to the reference image and a second eigenvector of any standard image, and then the cosine similarity of the first eigenvector and the second eigenvector is calculated. When the cosine similarity is higher than a preset threshold, the reference image is determined as the target image.

[0073] After vectorization processing is performed on the reference image and any standard image among the multiple standard images, a preset number of first eigenvectors and second eigenvectors can be extracted to calculate the cosine similarity of the first eigenvector and the second eigenvector. The first eigenvector and the second eigenvector are eigenvectors of fixed dimensions. The eigenvector of fixed dimensions can be understood as a eigenvector of fixed length (or fixed number) in a certain dimension.

[0074] Among them, as a possible implementation method, the image planes of the reference image and any standard image among multiple standard images can be gridded, including various forms such as triangles, quadrilaterals, or gradient grids, and then the corresponding color, gradient and other information are stored on the grid vertices, thereby realizing vectorization processing of the reference image and any standard image among multiple standard images.

[0075] In one embodiment of the present application, a first eigenvector corresponding to a reference image and a second eigenvector of any standard image are obtained, and the cosine similarity of the first eigenvector and the second eigenvector is calculated. When the cosine similarity is close to 1, the reference image is determined as the target image.

[0076] In a specific embodiment of the present application, Figure 4 As shown, keyword text can be extracted from the request text based on the keyword extraction model, for example, the keyword text includes keywords or key phrases, and the corresponding description information is determined based on the keyword text, and then the description information of the keyword text is combined based on the language rules to generate an image instruction, which is used to describe the content of the image (equivalent to the text content), and the text content is input into a pre-trained diffusion model to generate a reference image, and then any standard image in the standard image library is obtained. In order to evaluate the quality and relevance of the generated final image, feature extraction (equivalent to feature vectorization) is performed on any standard image and the reference image based on the image feature extraction model, and then any vectorized standard image and reference image are stored in the image vector index library, and then image vector similarity calculation (equivalent to cosine similarity calculation) is performed on any vectorized standard image in the image vector index library and the first image, and when the cosine similarity is close to 1, the reference image is determined as the target image.

[0077] Among them, the model training of the keyword extraction model can be implemented based on the training text, and then the keyword text can be extracted based on the request text.

[0078] Among them, a training image can be obtained, and then the model training of the open source diffusion model is implemented based on the training image, and then a fine-tuned diffusion model (equivalent to a pre-trained diffusion model) is obtained, so that in the technical solution of the image search method based on diffusion model analysis, the pre-trained diffusion model only occupies a small amount of storage and computing resources, has the characteristics of low cost and high efficiency of generated images, improves the accuracy of determining the target image, and avoids the CLIP model due to structural factors such as scale and encoder, resulting in a large number of model parameters, requiring a large amount of computing resources and storage space for pre-training and deployment. For example, the largest CLIP model has 400 billion parameters and requires 4.6TB of storage space, which consumes a lot of cost and computing time, resulting in a poor user experience. And because the pre-trained diffusion model is trained using known or existing training data, it can be determined that the training data has a certain degree of security, avoiding the potential deviations and risks of the CLIP model, such as the training data source from the Internet, which may contain some controversial or inappropriate content, causing the CLIP model to learn some wrong or inappropriate knowledge during the training process.

[0079] Through the above steps, a keyword text is determined from the request text input by the target object, and text content for describing the reference image is generated according to the keyword text. When the text content is successfully generated, the reference image is determined to be generated through the diffusion model, and when the similarity between any standard image in multiple standard images and the reference image meets the preset conditions, the reference image is determined as the target image. The determined target image is more accurate, and the text intention or demand can be understood more accurately, thereby improving the efficiency of generating the target image based on the text, improving user satisfaction, and the image search method based on diffusion model analysis has low resource consumption, low cost, and a wide range of application scenarios.

[0080] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0081] Figure 5 is a structural block diagram of an image search device based on diffusion model analysis according to an embodiment of the present application; Figure 5 As shown, including:

[0082] A first determination module 520 is used to determine a keyword text from a request text input by a target object, wherein the request text is used to request to obtain a target image;

[0083] A generating module 540, configured to determine image generation description information according to the keyword text, and generate a corresponding reference image through a diffusion model based on the image generation description information;

[0084] The second determination module 560 is configured to determine the reference image as the target image when the similarity between any standard image among the plurality of standard images and the reference image satisfies a preset condition.

[0085] Through the above device, a keyword text is determined from the request text input by the target object, wherein the request text is used to request to obtain a target image; image generation description information is determined according to the keyword text, and a corresponding reference image is generated through a diffusion model based on the image generation description information; when the similarity between any standard image and the reference image among multiple standard images meets a preset condition, the reference image is determined as the target image. The above technical solution solves the problem of being unable to accurately generate images based on text. The embodiment of the present application first determines the keyword text, determines the image generation description information based on the keyword text, generates a reference image, and then determines the target image based on the similarity calculation between any standard image and the reference image, so that the determined target image is more accurate.

[0086] In an exemplary embodiment, the generation module 540 is used to generate text content for describing the reference image based on the keyword text; when the text content is successfully generated, it is determined to generate the reference image through a diffusion model, wherein the diffusion model is used to generate an image based on the text content.

[0087] In an exemplary embodiment, a generation module 540 is used to determine multiple words included in the keyword text, and description information corresponding to the multiple words, wherein the description information is used to describe multiple forming elements of an image; obtain the description information of the multiple words; combine the multiple description information through the combination logic of preset language rules to obtain a text combination result, and use the text combination result as the text content.

[0088] In an exemplary embodiment, the generation module 540 is further used to convert the text content into a text vector through a diffusion model; denoise the text vector and the background image of the text content to obtain a target image; and decode the target image to generate the reference image.

[0089] In an exemplary embodiment, the second determination module 560 is configured to obtain the cosine similarity between the reference image and any one of the standard images; and determine the reference image as the target image when the cosine similarity is higher than a preset threshold.

[0090] In an exemplary embodiment, the second determination module 560 is used to perform vectorization processing on the reference image to obtain a first eigenvector corresponding to the reference image, and to perform vectorization processing on any of the standard images to obtain a second eigenvector corresponding to any of the standard images; determine the cosine similarity between the first eigenvector and the second eigenvector; and determine the cosine similarity between the reference image and any of the standard images using the cosine similarity between the first eigenvector and the second eigenvector.

[0091] In an exemplary embodiment, the first determination module 520 is used to obtain the number of times each word in the request text appears in the request text: the number of times each word appears in the request text is calculated by a weight algorithm to obtain a weight value for each word; and the target word whose weight value is greater than a weight threshold is used as the keyword text.

[0092] In an exemplary embodiment, the first determination module 520 is used to determine the total number of words in the request text; and use the ratio of the number of times each word appears in the request text to the total number of words as the weight value of each word.

[0093] An embodiment of the present application further provides a storage medium, which includes a stored program, wherein the program executes any of the above methods when it is run.

[0094] Optionally, in this embodiment, the storage medium may be configured to store program codes for executing the following steps:

[0095] S1, determining a keyword text from a request text input by a target object, wherein the request text is used to request to obtain a target image;

[0096] S2, determining image generation description information according to the keyword text, and generating a corresponding reference image through a diffusion model based on the image generation description information;

[0097] S3: When the similarity between any standard image among the plurality of standard images and the reference image satisfies a preset condition, determine the reference image as the target image.

[0098] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0099] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0100] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0101] S1, determining a keyword text from a request text input by a target object, wherein the request text is used to request to obtain a target image;

[0102] S2, determining image generation description information according to the keyword text, and generating a corresponding reference image through a diffusion model based on the image generation description information;

[0103] S3: When the similarity between any standard image among the plurality of standard images and the reference image satisfies a preset condition, determine the reference image as the target image.

[0104] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store program codes.

[0105] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0106] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0107] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An image search method based on diffusion model analysis, characterized in that: include: Determining a keyword text from a request text input by a target object, wherein the request text is used to request acquisition of a target image; Determining image generation description information according to the keyword text, and generating a corresponding reference image through a diffusion model based on the image generation description information; When the similarity between any standard image among a plurality of standard images and the reference image meets a preset condition, the reference image is determined as the target image.

2. The image search method based on diffusion model analysis according to claim 1, characterized in that: Determining image generation description information according to the keyword text includes: Generating text content for describing the reference image according to the keyword text; In the case that the text content is generated successfully, it is determined to generate the reference image through the diffusion model, wherein the diffusion model is used to generate an image according to the text content.

3. The image search method based on diffusion model analysis according to claim 2, characterized in that: Generating text content for describing the reference image according to the keyword text includes: Determine a plurality of words included in the keyword text, and description information corresponding to the plurality of words, wherein the description information is used to describe a plurality of forming elements of an image; Obtaining the description information of the multiple words; The plurality of description information are combined by combining logic of preset language rules to obtain a text combination result, and the text combination result is used as the text content.

4. The image search method based on diffusion model analysis according to claim 2, characterized in that: After determining that the reference image is generated by using the diffusion model, the method further includes: Converting the text content into a text vector by using the diffusion model; Denoising the text vector and the background image of the text content to obtain a target image; The target image is decoded to generate the reference image.

5. The image search method based on diffusion model analysis according to claim 1, characterized in that: When the similarity between any one of the plurality of standard images and the reference image satisfies a preset condition, determining the reference image as the target image comprises: Obtaining the cosine similarity between the reference image and any one of the standard images; When the cosine similarity is higher than a preset threshold, the reference image is determined as the target image.

6. The image search method based on diffusion model analysis according to claim 5, characterized in that: Obtaining the cosine similarity between the reference image and any of the standard images includes: Performing vectorization processing on the reference image to obtain a first eigenvector corresponding to the reference image, and performing vectorization processing on any of the standard images to obtain a second eigenvector corresponding to the any of the standard images; determining a cosine similarity between the first feature vector and the second feature vector; The cosine similarity between the first feature vector and the second feature vector is used to determine the cosine similarity between the reference image and any one of the standard images.

7. The image search method based on diffusion model analysis according to claim 1, characterized in that: Determine the keyword text from the request text input by the target object, including: Get the number of times each word in the request text appears in the request text: Calculating the number of times each word appears in the request text by using a weight algorithm to obtain a weight value for each word; The target word whose weight value is greater than the weight threshold is used as the keyword text.

8. The image search method based on diffusion model analysis according to claim 7, characterized in that: The number of times each word appears in the request text is calculated by a weight algorithm to obtain a weight value of each word, including: Determining the total number of words in the request text; The ratio of the number of times each word appears in the request text to the total number of words is used as the weight value of each word.

9. An image search device based on diffusion model analysis, characterized in that: include: A first determination module is used to determine a keyword text from a request text input by a target object, wherein the request text is used to request to obtain a target image; A generation module, used to determine image generation description information according to the keyword text, and generate a corresponding reference image through a diffusion model based on the image generation description information; The second determination module is used to determine the reference image as the target image when the similarity between any standard image among a plurality of standard images and the reference image meets a preset condition.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the method described in any one of claims 1 to 8 when executed.

11. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 8 through the computer program.

Images