Image retrieval method and device, equipment and medium

Abstract

The embodiment of the invention relates to an image retrieval method and device, equipment and a medium. The method comprises the steps that retrieval reference information is acquired; the retrieval reference information comprises first semantic information and first pixel information; based on the first semantic information, performing retrieval from a preset semantic information library to obtain multiple pieces of candidate semantic information matched with the first semantic information; based on the candidate sample images to which the candidate semantic information belongs respectively, retrieving from a preset pixel information base to obtain candidate pixel information corresponding to the candidate sample images respectively; and selecting a target sample image from the plurality of candidate sample images according to the similarity between the candidate pixel information corresponding to the plurality of candidate sample images and the first pixel information, and taking the target sample image as a retrieval result image corresponding to the retrieval reference information. According to the embodiment, it can be effectively guaranteed that the retrieval result meets the requirement, and the retrieval efficiency is guaranteed.

Description

Technical Field

[0001] The present disclosure relates to the field of information processing technology, and in particular to an image retrieval method, device, equipment and medium. Background Art

[0002] In the creation of images such as posters, suitable material composition is usually required to highlight the theme of the image or make the image more beautiful. However, it is difficult to retrieve the required image materials, which usually requires high labor costs. Traditional image retrieval methods are not effective, and the retrieved images are difficult to meet user needs. Summary of the invention

[0003] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides an image retrieval method, device, equipment and medium.

[0004] The disclosed embodiment provides an image retrieval method, the method comprising: obtaining retrieval reference information; wherein the retrieval reference information comprises first semantic information and first pixel information; based on the first semantic information, searching from a preset semantic information library to obtain a plurality of candidate semantic information matching the first semantic information; wherein the semantic information library comprises semantic information of a plurality of sample images; based on candidate sample images to which the plurality of candidate semantic information respectively belong, searching from a preset pixel information library to obtain candidate pixel information corresponding to each of the plurality of candidate sample images; wherein the pixel information library comprises pixel information of the plurality of sample images; selecting a target sample image from the plurality of candidate sample images according to the similarity between the candidate pixel information corresponding to each of the plurality of candidate sample images and the first pixel information, and using the target sample image as the retrieval result image corresponding to the retrieval reference information.

[0005] Optionally, the obtaining and retrieving reference information includes: obtaining a first reference image; performing semantic feature extraction on the first reference image to obtain first semantic information; and performing pixel feature extraction on the first reference image to obtain first pixel information.

[0006] Optionally, acquiring the first reference image includes: acquiring a target reference image; acquiring an image corresponding to a material contained in the target reference image, and using the image corresponding to the material as the first reference image.

[0007] Optionally, the target reference image contains multiple materials, and the method further includes: based on the position information and layer information of the multiple materials in the target reference image, combining the retrieval result images corresponding to the multiple materials to obtain a matching image corresponding to the target reference image.

[0008] Optionally, the extracting semantic features of the first reference image to obtain the first semantic information includes: extracting semantic features of the first reference image to obtain the first semantic feature; using the first semantic feature as the first semantic information; or, obtaining a quantitative feature corresponding to the first semantic feature based on a preset quantization semantic codebook, and obtaining the first semantic information based on the quantitative feature corresponding to the first semantic feature.

[0009] Optionally, the extracting pixel features of the first reference image to obtain first pixel information includes: extracting pixel features of the first reference image to obtain first pixel features; using the first pixel features as first pixel information; or, obtaining quantization features corresponding to the first pixel features based on a preset quantization pixel codebook, and obtaining first pixel information based on the quantization features corresponding to the first pixel features.

[0010] Optionally, the acquiring of retrieval reference information includes: acquiring retrieval prompt information; and generating first semantic information and first pixel information based on the retrieval prompt information by using a preset generation model.

[0011] Optionally, based on the retrieval prompt information, the first semantic information and the first pixel information are generated using a preset generation model, including: based on the retrieval prompt information, a first semantic feature flag sequence and a first pixel feature flag sequence are generated using a preset generation model; based on a preset quantization semantic codebook, quantitative features corresponding to the first semantic feature flag sequence are obtained, and the quantitative features corresponding to the first semantic feature flag sequence are used as the first semantic information; based on a preset quantization pixel codebook, quantitative features corresponding to the first pixel feature flag sequence are obtained, and the quantitative features corresponding to the first pixel feature flag sequence are used as the first pixel information.

[0012] Optionally, the retrieval from a preset semantic information library to obtain multiple candidate semantic information matching the first semantic information includes: arranging the similarities between the first semantic information and the semantic information in the preset semantic information library in order of size, and selecting the first N semantic information with the highest similarities as the multiple candidate semantic information matching the first semantic information; wherein N is a preset positive integer greater than 1; or, selecting semantic information whose similarity between the first semantic information and the semantic information in the preset semantic information library is higher than a preset threshold as the multiple candidate semantic information matching the first semantic information.

[0013] The disclosed embodiment also provides an image retrieval device, comprising: an information acquisition module, used to acquire retrieval reference information; wherein the retrieval reference information comprises first semantic information and first pixel information; a first retrieval module, used to search from a preset semantic information library based on the first semantic information, and obtain multiple candidate semantic information matching the first semantic information; wherein the semantic information library contains semantic information of multiple sample images; a second retrieval module, used to search from a preset pixel information library based on candidate sample images to which the multiple candidate semantic information respectively belong, and obtain candidate pixel information corresponding to each of the multiple candidate sample images; wherein the pixel information library contains pixel information of the multiple sample images; a result acquisition module, used to select a target sample image from the multiple candidate sample images according to the similarity between the candidate pixel information corresponding to each of the multiple candidate sample images and the first pixel information, and use the target sample image as the retrieval result image corresponding to the retrieval reference information.

[0014] An embodiment of the present disclosure also provides an electronic device, which includes: a processor; a memory for storing executable instructions of the processor; the processor is used to read the executable instructions from the memory and execute the instructions to implement the image retrieval method provided by the embodiment of the present disclosure.

[0015] The embodiment of the present disclosure further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the image retrieval method provided by the embodiment of the present disclosure.

[0016] The above technical solution provided by the embodiment of the present disclosure does not require time-consuming and laborious manual image retrieval, but can first obtain retrieval reference information of two dimensions, the first semantic information and the first pixel information, and then based on the first semantic information and the first pixel information, use the semantic information library (containing the semantic information of multiple sample images) and the pixel information library (containing the pixel information of multiple sample images) to successively perform semantic information retrieval and pixel information retrieval. This retrieval method can effectively ensure that the retrieval result image meets the requirements at both the semantic level and the image pixel level, and the phased retrieval method can also retrieve images that meet the requirements more efficiently, comprehensively ensuring that the retrieval results meet the requirements and the retrieval efficiency.

[0017] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] Figure 1 A flowchart of an image retrieval method provided by an embodiment of the present disclosure;

[0021] Figure 2 A schematic diagram of an image retrieval system provided by an embodiment of the present disclosure;

[0022] Figure 3 A schematic diagram of an image retrieval process provided by an embodiment of the present disclosure;

[0023] Figure 4 A schematic diagram of an image retrieval process provided by an embodiment of the present disclosure;

[0024] Figure 5 A schematic diagram of image processing provided by an embodiment of the present disclosure;

[0025] Figure 6 A schematic diagram of the structure of an image retrieval device provided by an embodiment of the present disclosure;

[0026] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0028] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0029] Figure 1 The following is a flow chart of an image retrieval method provided by an embodiment of the present disclosure. The method can be executed by an image retrieval device, wherein the device can be implemented by software and / or hardware and can generally be integrated in an electronic device. Figure 1As shown, the method mainly includes the following steps S102 to S108:

[0030] Step S102, obtaining retrieval reference information; wherein the retrieval reference information includes first semantic information and first pixel information. The disclosed embodiment does not limit the form of the first semantic information and the first pixel information, such as, the first semantic information and the first pixel information can be semantic content and image content, or semantic features and pixel features, or processed features of semantic features and pixel features, such as semantic quantitative features and pixel quantitative features. In addition, the disclosed embodiment does not limit the method for obtaining the retrieval reference information, such as, it can be retrieval reference information directly input by the user, or it can be retrieval reference information obtained by extracting semantic features and pixel features based on an image input by the user, or it can be retrieval reference information generated by a generation model based on prompt information input by the user.

[0031] Step S104: Based on the first semantic information, a search is performed from a preset semantic information library to obtain a plurality of candidate semantic information matching the first semantic information; wherein the semantic information library contains semantic information of a plurality of sample images.

[0032] Among them, the similarity between multiple candidate semantic information and the first semantic information is higher than the similarity between the non-candidate semantic information in the semantic information library and the first semantic information. In some embodiments, the similarity between the first semantic information and the semantic information in the preset semantic information library can be arranged in order of size, and the first N semantic information with the highest similarity are selected as multiple candidate semantic information matching the first semantic information; wherein N is a preset positive integer greater than 1; or, the semantic information whose similarity between the first semantic information and the semantic information in the preset semantic information library is higher than the preset threshold is used as multiple candidate semantic information matching the first semantic information. In practical applications, the value of N and the preset threshold of similarity can be flexibly set according to needs, and no restrictions are imposed here. In addition, the semantic information library can contain semantic information of a large number of sample images, and the sample images are images in the sample image library. Specifically, the disclosed embodiment can obtain the semantic information of all sample images in the existing sample image library, thereby obtaining a semantic information library. The disclosed embodiments do not limit the method for calculating the similarity between features. In practical applications, matrix multiplication or the like can be used to perform matrix multiplication on the first semantic information and the semantic information library, thereby obtaining the similarity between the first semantic information and the semantic information in the semantic information library, and then selecting candidate semantic information based on the similarity.

[0033] Step S106, based on the candidate sample images to which the multiple candidate semantic information respectively belong, searching from a preset pixel information library to obtain candidate pixel information corresponding to each of the multiple candidate sample images; wherein the pixel information library contains pixel information of multiple sample images.

[0034] The sample images to which multiple candidate semantic information belongs are candidate sample images. The disclosed embodiment can obtain the pixel information of all sample images in the aforementioned existing sample image library, thereby obtaining a pixel information library. The disclosed embodiment first screens out the candidate sample images by semantic retrieval, and then directly searches for the pixel information corresponding to the candidate sample images, i.e., the candidate pixel information, from the pixel information library.

[0035] Step S108, based on the similarity between the candidate pixel information corresponding to each of the plurality of candidate sample images and the first pixel information, a target sample image is selected from the plurality of candidate sample images, and the target sample image is used as a retrieval result image corresponding to the retrieval reference information. The number of target sample images can be one or more.

[0036] In practical applications, the candidate pixel information corresponding to the plurality of candidate sample images can be arranged in order of magnitude according to the similarity between the candidate pixel information and the first pixel information, and the candidate sample images to which the first M candidate pixel information with the highest similarity belong are selected as the target sample images, where M is a preset positive integer and can be flexibly set according to requirements. In the above manner, the retrieval result images whose semantics and pixel features match the retrieval reference information can be selected from all the sample images contained in the sample image library.

[0037] The above technical solution provided by the embodiment of the present disclosure does not require time-consuming and laborious manual image retrieval, but can first obtain retrieval reference information of two dimensions, the first semantic information and the first pixel information, and then based on the first semantic information and the first pixel information, use the semantic information library (containing the semantic information of multiple sample images) and the pixel information library (containing the pixel information of multiple sample images) to successively perform semantic information retrieval and pixel information retrieval. This retrieval method can effectively ensure that the retrieved target sample images meet the requirements at both the semantic level and the image pixel level, and the phased retrieval method can also retrieve images that meet the requirements more efficiently, comprehensively ensuring that the retrieval results meet the requirements and the retrieval efficiency.

[0038] In some implementations, the above step of obtaining the search reference information may be performed with reference to the following steps A to C:

[0039] Step A: acquiring a first reference image.

[0040] In some specific implementation examples, the first reference image may be the entire image input by the user. In other specific implementation examples, the target reference image may be obtained first, and then the image corresponding to the material contained in the target reference image may be obtained, and the image corresponding to the material may be used as the first reference image. The disclosed embodiment does not limit the content contained in the target reference image, such as a poster, an advertising page, etc., which may be an image input by the user. Different from the traditional image search technology, the disclosed embodiment does not directly perform image retrieval on the target reference image as a whole, but may perform retrieval on the materials contained in the target reference image that the user wishes to retrieve, so as to obtain sample images similar to the material. The materials contained in the target reference image include but are not limited to background images, stickers, content, etc. Any element contained in the target reference image may be used as a material. Specifically, the material contained in the target reference image may be obtained based on layer recognition and object recognition, so as to obtain the image corresponding to the material, and the image corresponding to each material may be used as the first reference image, so that images similar to the material may be retrieved for each material.

[0041] Step B, extracting semantic features from the first reference image to obtain first semantic information. In some specific implementation examples, the following steps B1 and B2 may be referred to for execution:

[0042] Step B1, extracting semantic features from a first reference image to obtain a first semantic feature.

[0043] In order to ensure the accuracy of the semantic features, illustratively, a semantic feature visual encoder can be used to extract semantic features from the first reference image, such as using a supervised method to obtain a semantic feature visual encoder with strong semantic encoding capabilities, and then the semantic feature visual encoder performs four-channel semantic encoding to obtain a more accurate and reliable first semantic feature. In addition, the embodiments of the present disclosure do not limit the implementation method of the semantic feature visual encoder.

[0044] Step B2, taking the first semantic feature as the first semantic information; or, obtaining the quantitative feature corresponding to the first semantic feature based on a preset quantitative semantic codebook, and obtaining the first semantic information based on the quantitative feature corresponding to the first semantic feature. In practical applications, the continuous first semantic feature can be directly used as the first semantic information, or the first semantic feature can be matched with the most similar feature in the quantitative semantic codebook using the quantitative semantic codebook, thereby discretizing the continuous feature so as to be suitable for subsequent different data processing modes. The above-mentioned quantitative semantic codebook is mainly a codebook designed for specific semantic features, which is used to quantize these features. It can ultimately generate the required quantitative semantic codebook based on a large amount of sample data through model training and other methods. The quantitative semantic codebook can be used to quantize the semantic feature as the closest feature in the codebook, which helps to simplify and compress the data, and also maintains the expression of the basic features of the data. The principle and acquisition method of the quantitative semantic codebook can refer to the relevant technology, which will not be repeated here. In practical applications, semantic features or semantic quantitative features can be flexibly selected as the first semantic information to be retrieved according to needs, and there is no restriction here.

[0045] Step C, extracting pixel features from the first reference image to obtain first pixel information. In some specific implementation examples, step C can be performed with reference to the following steps C1 and C2:

[0046] Step C1, extracting pixel features from a first reference image to obtain first pixel features.

[0047] In order to ensure the accuracy of pixel features, illustratively, a pixel feature visual encoder can be used to extract pixel features from a first reference image. In some specific implementation examples, the first reference image can be resized to obtain an image of a specified size; then the pixel feature visual encoder is used to downsample the image of the specified size by a specified multiple, that is, to reconstruct and encode it, so as to obtain a first pixel feature, such as using the 4*4 pixel feature obtained by downsampling as the first pixel feature. This feature can also be called a fuzzy feature or a rough feature. Through the rough feature extraction method, an image similar to but not completely identical to the first reference image can be retrieved based on the feature, so as to avoid providing the user with a retrieval result image that is completely consistent with the input target reference image (that is, the image to be retrieved) and affecting the user experience. In addition, the embodiments of the present disclosure do not limit the implementation method of the pixel feature visual encoder.

[0048] Step C2, taking the first pixel feature as the first pixel information; or, obtaining the quantized feature corresponding to the first pixel feature based on a preset quantized pixel codebook, and obtaining the first pixel information based on the quantized feature corresponding to the first pixel feature. In practical applications, the continuous first pixel feature can be directly used as the first pixel information, or the quantized pixel codebook can be used to match the first pixel feature with the most similar feature in the quantized pixel codebook, thereby discretizing the continuous feature so as to be suitable for subsequent different data processing modes. The principle and acquisition method of the above-mentioned quantized pixel codebook are similar to those of the quantized semantic codebook, and the details can also be referred to related technologies, which will not be repeated here. In practical applications, pixel features or pixel quantized features can be flexibly selected as the first pixel information to be retrieved according to needs, and there is no restriction here.

[0049] Through the above method, the retrieval reference information in two dimensions of the first semantic information and the first pixel information can be accurately and reliably obtained.

[0050] In some implementations, the above step of obtaining the search reference information may be performed with reference to the following steps a to b:

[0051] Step a: obtaining search prompt information, which may specifically be a search prompt word input by a user, and may be used to describe the theme style or content information contained in the image to be retrieved.

[0052] Step b, based on the retrieval prompt information, using a preset generation model to generate the first semantic information and the first pixel information. The disclosed embodiment does not limit the implementation of the generation model, for example, the generation model can be a large language model. Considering the characteristic that the generation model is more convenient for generating discretized data, in some specific implementation examples, step b can be performed with reference to the following steps b1 to b3:

[0053] Step b1, using a preset generation model to generate a first semantic feature marker sequence and a first pixel feature marker sequence, wherein the first semantic feature marker sequence may also be referred to as a first semantic feature token sequence, and the first pixel feature marker sequence may also be referred to as a first semantic feature token sequence, where token is specifically a discrete language marker.

[0054] Step b2, based on a preset quantized semantic codebook, obtains the quantized features corresponding to the first semantic feature flag sequence, and uses the quantized features corresponding to the first semantic feature flag sequence as the first semantic information. Based on the first semantic feature flag sequence, the quantized features corresponding to the token can be found in the quantized semantic codebook, thereby obtaining the first semantic information.

[0055] Step b3, based on the preset quantized pixel codebook, obtain the quantized features corresponding to the first pixel feature flag sequence, and use the quantized features corresponding to the first pixel feature flag sequence as the first pixel information. Based on the first pixel feature flag sequence, the quantized features corresponding to the token can be found in the quantized pixel codebook, thereby obtaining the first semantic information.

[0056] Through the above method, there is no need for users to use complex language descriptions. The retrieval reference information containing two dimensions of semantics and pixel features can be intelligently generated directly through the generation model. On this basis, the semantic information library and the pixel information library are used to perform two-stage retrieval to obtain a retrieval result image whose semantics and pixel features meet the requirements.

[0057] In practical applications, the above-mentioned image retrieval method provided by the embodiment of the present disclosure can not only retrieve new images based on existing images,

[0058] For easier understanding, see Figure 2 A schematic diagram of an image retrieval system is shown, and the image retrieval system includes a sample image library, and the sample image library includes a large number of existing sample images. In the related art, image retrieval is usually performed directly based on the sample image library, and in the embodiment of the present disclosure, feature extraction can be performed for all images in the sample image library through a semantic feature encoder and a pixel feature encoder, so as to construct a semantic feature library and a pixel feature library. Further, considering that a generative model such as a large language model usually generates discrete data, such as a discrete language marker sequence corresponding to a generated semantic feature or pixel feature, for this reason, the embodiment of the present disclosure can also use a quantized semantic codebook and a quantized pixel codebook to respectively construct a generated semantic feature quantization library and a pixel feature quantization library on the basis of the semantic feature library and the pixel feature library, so that subsequent retrieval can be performed in the quantization library based on the model generation information. In practical applications, the semantic feature library and the pixel feature library can be used as a semantic information library and a pixel feature library, and the semantic feature quantization library and the pixel feature quantization library can be used as a semantic information library and a pixel feature library, and there is no limitation here.

[0059] Based on the foregoing, the present disclosure provides a Figure 3 The image retrieval process diagram shown in Figure 1 is as follows: Figure 3 The main purpose is to illustrate the method of performing image retrieval based on the first semantic information and the first pixel information extracted from the existing image (original image). Figure 3In the embodiment, the original image may be a target reference image input by a user or an image corresponding to a material contained in a target reference image input by a user, in the hope of retrieving an image that is semantically similar and has pixel features that are similar but not completely consistent with the original image. In the embodiment disclosed in the present invention, a semantic feature encoder and a pixel feature encoder may be used to respectively extract a first semantic feature and a first pixel feature. The first semantic feature is matched with the semantic features contained in the semantic feature library by similarity, so as to retrieve the top N candidate semantic features with the highest similarity in the semantic feature library. Then, based on the sample image (candidate sample image) to which the N candidate semantic features belong, N candidate pixel features are retrieved from the pixel feature library. The candidate pixel features are pixel features belonging to the candidate sample image. Then, based on the first pixel feature, similarity matching is performed with the N candidate pixel features. Thus, a specified number of target pixel features can be obtained according to the similarity. The candidate sample image to which the target pixel feature belongs is the target sample image, i.e., the retrieval result image.

[0060] The present disclosure also provides a Figure 4 The image retrieval process diagram shown in Figure 1 is as follows: Figure 4 The main purpose is to illustrate the method of performing image retrieval based on the generation model to generate the first semantic information and the first pixel information. Figure 4 In the example, the first semantic information and the first pixel information generated by the generative model are the first quantized semantic features and the first quantized pixel features, and the semantic information library and the pixel information library are the semantic feature quantization library and the pixel feature quantization library. It can be understood that the generative model taking the large language model as an example is better at generating discrete data, such as generating feature flag sequences. For this reason, the embodiment of the present disclosure can first convert the flag sequence generated by the generative model into a quantized feature through the corresponding quantization codebook, and then search in the quantization library. The subsequent retrieval method is the same as Figure 3 It should be noted that compared with the related art of directly using a generative model to generate material images, the above-mentioned method provided by the embodiment of the present disclosure is more reliable. Specifically, the method of directly using a generative model to generate images is uncontrollable, and it is difficult to ensure that the model-generated images meet the requirements. The model may even generate bad examples, thereby affecting the user experience. The embodiment of the present disclosure is a sample image library with guaranteed quality, and then searches based on the search reference information generated by the generative model, so as to retrieve the search result image that meets the requirements from the sample image. This method is more controllable and can fully guarantee the quality of the final search result image.

[0061] It should also be noted that in practical applications, Figure 3 and Figure 4It can also be combined, for example, to facilitate processing, the semantic feature quantization library and the pixel feature quantization library can be uniformly used as the semantic information library and the pixel information library. Figure 3 For the continuous semantic features and pixel features obtained by extracting features from the original image in the image, they can also be further converted into first quantized semantic features and first quantized pixel features through quantized semantic codebooks and quantized pixel codebooks, so as to be aligned with the retrieval method using the generative model.

[0062] Furthermore, as mentioned above, retrieval can be performed directly based on the target reference image input by the user, or based on the image corresponding to the material contained in the target reference image input by the user. If it is the latter, when the number of materials contained in the target reference image is multiple, the method provided by the embodiment of the present disclosure also includes: based on the position information and layer information of the multiple materials in the target reference image, combining the retrieval result images corresponding to the multiple materials to obtain the matching image corresponding to the target reference image. This method can fully ensure that the final image can obtain semantic and pixel feature matching with the target reference image input by the user at the material granularity. For ease of understanding, please refer to Figure 5 As shown in the figure, for a target reference image containing three materials, namely, a makeup brush, a makeup mirror, and a table for carrying the makeup brush and the makeup mirror, the above-mentioned method can be used to search for each material separately, thereby obtaining a retrieval result image that matches each material in terms of semantics and pixel features. Subsequently, based on the combination of the retrieval result images corresponding to each material, a matching image that is similar to each material in the target reference image can be obtained, thereby comprehensively meeting the user's retrieval needs.

[0063] It is understandable that most of the related technologies use image search or text search. For the image search, it is easy to directly search out the original image or a partial image of the original image, rather than the image with variants that the user expects. For the text search, the searched images will only meet the semantic requirements, which is difficult to meet user needs. For poster creators, it is very difficult to retrieve a complete set of materials (materials) required for the poster, which takes a lot of time and energy. For example, it is difficult to describe the required materials in concise natural language. By adopting the above method provided by the embodiment of the present disclosure, the selected features (first semantic information and first pixel information) can be used for two-stage image retrieval, so as to efficiently and reliably obtain images whose semantic features and pixel features meet the requirements but are not completely the same. In specific applications, it is possible to search for all materials in existing images (such as finished posters) to obtain a set of replacement materials whose semantics and pixel features are very similar to but not completely identical to those of the materials in the existing images. It is also possible to directly use generative models such as large language models to generate a set of required material materials based on retrieval prompt information, that is, to simultaneously satisfy the feature matching and semantic adaptability of the retrieved materials, thereby better meeting user needs and comprehensively ensuring the reliability and efficiency of image retrieval.

[0064] Corresponding to the aforementioned image retrieval method, the embodiment of the present disclosure further provides an image retrieval device, Figure 6 FIG. 1 is a schematic diagram of the structure of an image retrieval device provided in an embodiment of the present disclosure. The device can be implemented by software and / or hardware and can generally be integrated in an electronic device, such as Figure 6 As shown, the image retrieval device includes:

[0065] The information acquisition module 602 is used to acquire the search reference information; wherein the search reference information includes the first semantic information and the first pixel information;

[0066] A first retrieval module 604 is used to retrieve from a preset semantic information library based on the first semantic information to obtain a plurality of candidate semantic information matching the first semantic information; wherein the semantic information library contains semantic information of a plurality of sample images;

[0067] The second retrieval module 606 is used to retrieve the candidate sample images to which the multiple candidate semantic information respectively belongs from a preset pixel information library to obtain the candidate pixel information corresponding to each of the multiple candidate sample images; wherein the pixel information library contains the pixel information of the multiple sample images;

[0068] The result acquisition module 608 is used to select a target sample image from multiple candidate sample images according to the similarity between the candidate pixel information corresponding to each of the multiple candidate sample images and the first pixel information, and use the target sample image as the retrieval result image corresponding to the retrieval reference information.

[0069] The above technical solution provided by the embodiment of the present disclosure does not require time-consuming and laborious manual image retrieval, but can first obtain retrieval reference information of two dimensions, the first semantic information and the first pixel information, and then based on the first semantic information and the first pixel information, use the semantic information library (containing the semantic information of multiple sample images) and the pixel information library (containing the pixel information of multiple sample images) to successively perform semantic information retrieval and pixel information retrieval. This retrieval method can effectively ensure that the retrieved target sample images meet the requirements at both the semantic level and the image pixel level, and the phased retrieval method can also more efficiently retrieve images that meet the requirements, comprehensively ensuring the accuracy of the retrieval results and the retrieval efficiency.

[0070] In some implementations, the information acquisition module 602 is specifically used to: acquire a first reference image; perform semantic feature extraction on the first reference image to obtain first semantic information; and perform pixel feature extraction on the first reference image to obtain first pixel information.

[0071] In some implementations, the information acquisition module 602 is specifically used to: acquire a target reference image; acquire an image corresponding to a material contained in the target reference image, and use the image corresponding to the material as a first reference image.

[0072] In some embodiments, the target reference image contains multiple materials, and the device also includes a combination module for combining the retrieval result images corresponding to the multiple materials based on the position information and layer information of the multiple materials in the target reference image to obtain a matching image corresponding to the target reference image.

[0073] In some embodiments, the information acquisition module 602 is specifically used to: extract semantic features from the first reference image to obtain a first semantic feature; use the first semantic feature as the first semantic information; or, based on a preset quantization semantic codebook, obtain a quantitative feature corresponding to the first semantic feature, and obtain the first semantic information based on the quantitative feature corresponding to the first semantic feature.

[0074] In some embodiments, the information acquisition module 602 is specifically used to: extract pixel features from the first reference image to obtain a first pixel feature; use the first pixel feature as the first pixel information; or, based on a preset quantization pixel codebook, obtain a quantization feature corresponding to the first pixel feature, and obtain the first pixel information based on the quantization feature corresponding to the first pixel feature.

[0075] In some implementations, the information acquisition module 602 is specifically used to: acquire search prompt information; and generate first semantic information and first pixel information based on the search prompt information using a preset generation model.

[0076] In some embodiments, the information acquisition module 602 is specifically used to: based on the retrieval prompt information, generate a first semantic feature marker sequence and a first pixel feature marker sequence using a preset generation model; obtain the quantitative features corresponding to the first semantic feature marker sequence based on a preset quantization semantic codebook, and use the quantitative features corresponding to the first semantic feature marker sequence as the first semantic information; obtain the quantitative features corresponding to the first pixel feature marker sequence based on a preset quantization pixel codebook, and use the quantitative features corresponding to the first pixel feature marker sequence as the first pixel information.

[0077] In some embodiments, the first retrieval module 604 is specifically used to: arrange the similarities between the first semantic information and the semantic information in a preset semantic information library in order of size, and select the first N semantic information with the highest similarities as multiple candidate semantic information matching the first semantic information; wherein N is a preset positive integer greater than 1; or, use semantic information whose similarity between the first semantic information and the semantic information in the preset semantic information library is higher than a preset threshold as multiple candidate semantic information matching the first semantic information.

[0078] The image retrieval device provided in the embodiments of the present disclosure can execute the image retrieval method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0079] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device embodiment can refer to the corresponding process in the method embodiment, and will not be repeated here.

[0080] An embodiment of the present disclosure provides an electronic device, which includes: a storage device on which a computer program is stored; and a processing device for executing the computer program in the storage device to implement the steps of any method in the present disclosure.

[0081] Reference below Figure 7, which shows a schematic diagram of the structure of an electronic device 700 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0082] like Figure 7 As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 are also stored. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0083] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0084] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 709, or installed from a storage device 708, or installed from a ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0085] In addition to the above-mentioned methods and devices, the embodiments of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, cause the processor to perform the image processing method provided by the embodiments of the present disclosure. The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present disclosure, the programming languages ​​including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on a user computing device, partially on a user device, as an independent software package, partially on a user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0086] In addition, the embodiment of the present disclosure may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the image retrieval method provided by the embodiment of the present disclosure.

[0087] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0088] The embodiment of the present disclosure also provides a computer program product, including a computer program / instruction, which implements the image retrieval method in the embodiment of the present disclosure when executed by a processor.

[0089] It is understandable that before using the technical solutions disclosed in the various embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0090] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0091] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0092] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0093] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0094] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image retrieval method, characterized in that: include: Acquire retrieval reference information; wherein the retrieval reference information includes first semantic information and first pixel information; Based on the first semantic information, searching from a preset semantic information library to obtain a plurality of candidate semantic information matching the first semantic information; wherein the semantic information library contains semantic information of a plurality of sample images; Based on the candidate sample images to which the plurality of candidate semantic information respectively belong, searching from a preset pixel information library to obtain candidate pixel information corresponding to the plurality of candidate sample images; wherein the pixel information library contains pixel information of the plurality of sample images; According to the similarities between the candidate pixel information corresponding to each of the plurality of candidate sample images and the first pixel information, a target sample image is selected from the plurality of candidate sample images, and the target sample image is used as a retrieval result image corresponding to the retrieval reference information.

2. The method according to claim 1, characterized in that The obtaining of the search reference information includes: acquiring a first reference image; Performing semantic feature extraction on the first reference image to obtain first semantic information; and performing pixel feature extraction on the first reference image to obtain first pixel information.

3. The method according to claim 2, characterized in that The acquiring of the first reference image comprises: Acquire a target reference image; An image corresponding to the material contained in the target reference image is acquired, and the image corresponding to the material is used as the first reference image.

4. The method according to claim 3, characterized in that The target reference image contains a plurality of materials, and the method further includes: Based on the position information and layer information of the multiple materials in the target reference image, the retrieval result images corresponding to the multiple materials are combined and processed to obtain a matching image corresponding to the target reference image.

5. The method according to claim 2, characterized in that: The step of extracting semantic features from the first reference image to obtain first semantic information includes: Extracting semantic features from the first reference image to obtain a first semantic feature; The first semantic feature is used as the first semantic information; or, a quantized feature corresponding to the first semantic feature is obtained based on a preset quantized semantic codebook, and the first semantic information is obtained based on the quantized feature corresponding to the first semantic feature.

6. The method according to claim 2, characterized in that The extracting pixel features from the first reference image to obtain first pixel information includes: Extracting pixel features from the first reference image to obtain first pixel features; The first pixel feature is used as the first pixel information; or, a quantized feature corresponding to the first pixel feature is obtained based on a preset quantized pixel codebook, and the first pixel information is obtained based on the quantized feature corresponding to the first pixel feature.

7. The method according to claim 1, characterized in that The obtaining of the search reference information includes: Get search prompt information; Based on the retrieval prompt information, a preset generation model is used to generate first semantic information and first pixel information.

8. The method according to claim 7, characterized in that The step of generating the first semantic information and the first pixel information based on the search prompt information by using a preset generation model includes: Based on the search prompt information, a first semantic feature mark sequence and a first pixel feature mark sequence are generated using a preset generation model; Obtaining quantized features corresponding to the first semantic feature flag sequence based on a preset quantized semantic codebook, and using the quantized features corresponding to the first semantic feature flag sequence as first semantic information; The quantized features corresponding to the first pixel feature flag sequence are obtained based on a preset quantized pixel codebook, and the quantized features corresponding to the first pixel feature flag sequence are used as the first pixel information.

9. The method according to claim 1, characterized in that: The retrieving from a preset semantic information library to obtain a plurality of candidate semantic information matching the first semantic information includes: Arrange the similarities between the first semantic information and the semantic information in the preset semantic information library in order of magnitude, and select the first N semantic information with the highest similarities as multiple candidate semantic information matching the first semantic information; wherein N is a preset positive integer greater than 1; or, Semantic information whose similarity between the first semantic information and semantic information in a preset semantic information library is higher than a preset threshold is used as a plurality of candidate semantic information that matches the first semantic information.

10. An image retrieval device, characterized in that: include: An information acquisition module, used to acquire retrieval reference information; wherein the retrieval reference information includes first semantic information and first pixel information; A first retrieval module is used to search from a preset semantic information library based on the first semantic information to obtain a plurality of candidate semantic information matching the first semantic information; wherein the semantic information library contains semantic information of a plurality of sample images; A second retrieval module is used to retrieve from a preset pixel information library based on the candidate sample images to which the plurality of candidate semantic information respectively belong, and obtain candidate pixel information corresponding to each of the plurality of candidate sample images; wherein the pixel information library contains pixel information of the plurality of sample images; The result acquisition module is used to select a target sample image from a plurality of candidate sample images according to the similarity between the candidate pixel information corresponding to each of the plurality of candidate sample images and the first pixel information, and use the target sample image as the retrieval result image corresponding to the retrieval reference information.

11. An electronic device, characterized in that: The electronic device comprises: a storage device having a computer program stored thereon; A processing device is used to execute the computer program in the storage device to implement the steps of the image retrieval method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the image retrieval method described in any one of claims 1 to 9.

Images