An image search method, device, computer equipment and storage medium

Abstract

The application discloses an image search method and device, computer equipment and a storage medium, the method comprising: separating a to-be-searched image to obtain a subject image, a foreground image and a background image, and extracting features of the subject image, the foreground image and the background image to obtain subject features, foreground features and background features; fusing the subject features, the foreground features and the background features to obtain image features of the to-be-searched image, and performing similarity calculation on the image features of the to-be-searched image and standard image features of standard historical images to obtain the similarity of the standard historical images; and pushing the search result of the to-be-searched image to a terminal device according to the similarity of each standard historical image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to an image search method and device, a computer device and a storage medium. BACKGROUND

[0002] With the increasing amount of image data on the Internet, the requirements of users for image search on the Internet are also increasing, and thus various image search engines based on images have emerged. The emergence of image search engines reduces the difficulty of searching for image information on the Internet for users, and although it is not perfect, it can still meet the image search needs of most users. Current image search methods include two types. One is to input a keyword similar to the name or content of an image to search for images corresponding to the name and the keyword. The other is to search for similar or identical images by inputting an existing image.

[0003] In the prior art, image search by image uses image recognition technology to extract image features of an input image, and then searches for images similar to the image features of the input image based on the image features. Therefore, the accuracy of the extracted image features has a great influence on the image search results. However, in the prior art of image search by image, the image features are extracted by using an image content extraction method, and the extracted image features are not accurate enough, resulting in low accuracy of the image search results. SUMMARY

[0004] The present application provides an image search method, device, computer device and storage medium to solve the problem of low accuracy of image search results caused by insufficient accuracy of extracted image features in the prior art of image search by image.

[0005] An image search method is provided, comprising:

[0006] An image to be searched for is acquired from a terminal device, and the image to be searched for is separated to obtain a main image, a foreground image and a background image of the image to be searched for;

[0007] Features are extracted from the main image, the foreground image and the background image respectively to obtain main features, foreground features and background features of the image to be searched for;

[0008] The main features, the foreground features and the background features are fused to obtain image features of the image to be searched for;

[0009] Similarities between the image features of the image to be searched for and standard image features of standard historical images are calculated to obtain similarities of the standard historical images;

[0010] Search results of the image to be searched for are pushed to the terminal device according to the similarity of each standard historical image.

[0011] Further, feature extraction is performed on the subject image, the foreground image and the background image respectively to obtain subject feature, foreground feature and background feature of the image to be searched, including:

[0012] Feature extraction is performed on the subject image to obtain subject pattern feature, and feature extraction is performed on the subject image to obtain subject color feature, and the subject pattern feature and the subject color feature are summarized as the subject feature;

[0013] Feature extraction is performed on the foreground image and the background image respectively to obtain foreground pattern feature and background pattern feature;

[0014] Feature extraction is performed on the foreground image and the background image respectively to obtain foreground color feature and background color feature;

[0015] The foreground pattern feature and the foreground color feature are summarized as the foreground feature, and the background pattern feature and the background color feature are summarized as the background feature.

[0016] Further, the subject feature includes the subject pattern feature and the subject color feature, the foreground feature includes the foreground pattern feature and the foreground color feature, and the background feature includes the background pattern feature and the background color feature, and feature fusion is performed on the subject feature, the foreground feature and the background feature to obtain the image feature of the image to be searched, including:

[0017] Feature enhancement is performed on the subject pattern feature based on the subject color feature to obtain enhanced subject feature;

[0018] Feature enhancement is performed on the foreground pattern feature based on the foreground color feature to obtain enhanced foreground feature;

[0019] Feature enhancement is performed on the background pattern feature based on the background color feature to obtain enhanced background feature;

[0020] Feature fusion is performed on the enhanced subject feature, the enhanced foreground feature and the enhanced background feature to obtain the image feature of the image to be searched.

[0021] Further, feature fusion is performed on the enhanced subject feature, the enhanced foreground feature and the enhanced background feature to obtain the image feature of the image to be searched, including:

[0022] A first weight of the enhanced subject feature, a second weight of the enhanced foreground feature and a third weight of the enhanced background feature are determined, and the first weight, the second weight and the third weight decrease in turn;

[0023] The enhanced foreground feature and the enhanced background feature are weighted and summed according to the first weight, the second weight and the third weight to obtain the image feature of the image to be searched.

[0024] Further, before the separating the to-be-searched image to obtain a foreground pattern and a background pattern of the to-be-searched image, the method further comprises:

[0025] identifying the to-be-searched image to determine whether the to-be-searched image is a first type of image, the first type of image being an image having a background and a foreground;

[0026] if the to-be-searched image is the first type of image, separating the to-be-searched image to obtain the foreground pattern and the background pattern of the to-be-searched image.

[0027] Further, after determining whether the to-be-searched image is the first type of image, the method further comprises:

[0028] if the to-be-searched image is a second type of image, performing pattern recognition on the to-be-searched image to obtain a main body pattern of the to-be-searched image;

[0029] extracting a pattern feature of the to-be-searched image from the main body pattern of the to-be-searched image, and extracting a color feature of the to-be-searched image from the main body pattern of the to-be-searched image;

[0030] extracting a theme feature of the to-be-searched image from the main body pattern of the to-be-searched image;

[0031] performing feature fusion on the pattern feature, the color feature, and the theme feature to obtain an image feature of the to-be-searched image.

[0032] Further, the feature fusion on the pattern feature, the color feature, and the theme feature to obtain the image feature of the to-be-searched image comprises:

[0033] determining a weight of the color feature and a weight of the theme feature, and performing weighted summation on the color feature and the theme feature based on the weight of the color feature and the weight of the theme feature to obtain a fusion feature;

[0034] performing weight conversion on the fusion feature based on an attention mechanism to obtain fusion weight data;

[0035] performing feature enhancement on the pattern feature of the to-be-searched image based on the fusion weight data to obtain the image feature of the to-be-searched image.

[0036] Further, each standard historical image further corresponds to a standard image style, and the search result of the to-be-searched image is pushed to the terminal device according to the similarity of each standard historical image, comprising:

[0037] performing style recognition on the to-be-searched image through a style recognition network to obtain an image style of the to-be-searched pattern;

[0038] matching the image style of the image to be searched with the standard image style of each standard historical image, and recording the standard historical image that is successfully matched as a to-be-recommended image;

[0039] determining whether the number of to-be-recommended images is less than a preset number, the preset number being determined according to the size of the terminal device;

[0040] if the number of to-be-recommended images is greater than or equal to the preset number, performing descending order sorting on the plurality of to-be-recommended images according to the similarity of the to-be-recommended images, to obtain search recommendation data of the image to be searched;

[0041] pushing the search recommendation data to the terminal device as a search result of the image to be searched.

[0042] Further, after determining whether the number of to-be-recommended images is less than the preset number, the method further comprises:

[0043] if the number of to-be-recommended images is less than the preset number, selecting the first n to-be-recommended images with the largest similarity from the plurality of to-be-recommended images as first recommendation data, and performing descending order sorting on the remaining standard historical images according to the similarity, to obtain second recommendation data;

[0044] pushing the first recommendation data and the second recommendation data to the terminal device as a search result of the image to be searched, the first recommendation data being displayed before the second recommendation data.

[0045] An image searching device is provided, comprising:

[0046] a processing module configured to acquire an image to be searched sent by a terminal device, and perform separation processing on the image to be searched, to obtain a main body image, a foreground pattern and a background pattern of the image to be searched;

[0047] an extraction module configured to perform feature extraction on the main body image, the foreground pattern and the background pattern respectively, to obtain a main body feature, a foreground feature and a background feature of the image to be searched;

[0048] a fusion module configured to perform feature fusion on the main body feature, the foreground feature and the background feature, to obtain an image feature of the image to be searched;

[0049] a calculation module configured to perform similarity calculation on the fused feature of the image to be searched and a standard feature of each standard historical image, to obtain a similarity of each standard historical image;

[0050] a pushing module configured to push a search result of the image to be searched to the terminal device according to the similarity of each standard historical image.

[0051] The application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the image search method when executing the computer program.

[0052] The application provides a computer readable storage medium, which stores a computer program, and the computer program implements the steps of the image search method when executed by a processor.

[0053] In one technical solution of the image search method, the device, the computer device, and the storage medium, the image to be searched sent by the terminal device is acquired, and the image to be searched is separated to obtain a subject image, a foreground image, and a background image of the image to be searched; the subject image, the foreground image, and the background image are respectively subjected to feature extraction to obtain a subject feature, a foreground feature, and a background feature of the image to be searched; the subject feature, the foreground feature, and the background feature are subjected to feature fusion to obtain an image feature of the image to be searched; the image feature of the image to be searched and a standard image feature of a standard historical image are subjected to similarity calculation to obtain the similarity of the plurality of standard historical images; and the search result of the image to be searched is pushed to the terminal device according to the similarity of each standard historical image. The image to be searched is separated into the subject image, the foreground image, and the background image, the image features of the subject image, the foreground image, and the background image are respectively extracted, and then the image features are fused to obtain the image feature. The granularity of image content extraction is improved by refining the visual content of the image and respectively extracting the features, richer image details can be extracted, the description accuracy of the image feature to the image is improved, and the accuracy of the image search result is improved. BRIEF DESCRIPTION OF DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the application. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without any creative labor.

[0055] Figure 1 is an application environment schematic diagram of the image search method in an embodiment of the application;

[0056] Figure 2 is a flow schematic diagram of the image search method in an embodiment of the application;

[0057] Figure 3 is Figure 2 is an implementation flow schematic diagram of step S20 in the method;

[0058] Figure 4 is Figure 2 is an implementation flow schematic diagram of step S30 in the method;

[0059] Figure 5 is Figure 2 An implementation flowchart diagram of step S50 in the method is shown in the figure;

[0060] Figure 6 is a structural diagram of an image search device in an embodiment of the present application;

[0061] Figure 7 is a structural diagram of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0062] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0063] The image search method provided by the embodiments of the present application can be applied in the application scenario shown in the figure Figure 1 When a user needs to perform image search, the user inputs an image to be searched into a terminal device, the server obtains the image to be searched sent by the terminal device, and performs separation processing on the image to be searched to obtain a subject image, a foreground image and a background image of the image to be searched. Feature extraction is performed on the subject image, the foreground image and the background image respectively to obtain subject features, foreground features and background features of the image to be searched. Feature fusion is performed on the subject features, the foreground features and the background features to obtain image features of the image to be searched. Similarity calculation is performed on the image features of the image to be searched and standard image features of standard historical images to obtain similarities of the standard historical images. The search result of the image to be searched is pushed to the terminal device according to the similarity of each standard historical image. The image to be searched is separated into the subject image, the foreground image and the background image, and then the image features of the subject image, the foreground image and the background image are extracted respectively and fused to obtain the image features. The granularity of image content extraction is improved by refining the visual content of the image and performing feature extraction respectively, richer image details can be extracted, the description accuracy of the image features to the image is improved, and the accuracy of the image search result is improved.

[0064] The terminal device can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers and the like, and in addition, the server can be implemented by an independent server or a server cluster composed of multiple servers.

[0065] In an embodiment, as shown in the figure Figure 2 An image search method is provided, and the method is applied in Figure 1The server in the server is taken as an example for illustration, including the following steps:

[0066] S10: Obtain the to-be-searched image sent by the terminal device, and perform separation processing on the to-be-searched image to obtain a subject image, a foreground image and a background image of the to-be-searched image.

[0067] When a user is editing a digital product, the user needs to search for an image through a terminal device. For example, when the user is editing a platform, an H5, a poster and an activity page, the user needs to add or replace a picture. At this time, the image quality of the user is not high, or the user is not satisfied with the image. Therefore, the user needs to search for a high-quality original image of the existing image or an image similar to the existing image by taking the existing image as an input image. At this time, the user can use an existing search engine on the editing platform to search for an image. The user inputs a to-be-searched image into the search engine through the terminal device, and the terminal device sends the to-be-searched image input by the user to the server of the editing platform. The server obtains the to-be-searched image sent by the terminal device, and performs separation processing on the to-be-searched image through a target image segmentation model to obtain a subject image, a foreground image and a background image of the to-be-searched image.

[0068] In this embodiment, the foreground of the image refers to a scene in front of the subject image, and the background refers to a scene behind the subject image. The subject image is the visual center of the image, the foreground is closest to the lens (field of view), and the background is farthest from the lens (field of view). In a picture, the area occupied by the background is much larger than that of the foreground.

[0069] The target image segmentation model is a neural network model obtained by deep learning training based on a plurality of standard historical images. The target image segmentation model is obtained in the following manner.

[0070] The standard segmentation data of each standard historical image is obtained, the standard segmentation data of the standard historical image including a standard subject image, a standard foreground image and a standard background image of the standard historical image; the standard historical image is separated by using an image segmentation model containing initial parameters to obtain separated image data of the standard historical image, the separated image data including a separated subject image, a separated foreground image and a separated background image of the standard historical image; a total loss value is calculated based on the separated image data and the standard segmentation data, that is, the total loss value is calculated based on the separated subject image, the separated foreground image and the separated background image, and the standard subject image, the standard foreground image and the standard background image; when the total loss value is greater than a preset loss value, it is determined that the convergence condition is not reached, and then the parameters of the image segmentation model are iteratively updated based on other standard historical images until the total loss value is less than or equal to the preset loss value, it is determined that the convergence condition is reached, and the converged image segmentation model is output as a target image segmentation model. In the embodiment, a plurality of standard historical images are used as training samples to train a target image segmentation model with high accuracy, and then the target image segmentation model is used to separate the to-be-searched image to obtain the subject image, the foreground image and the background image of the to-be-searched image with high accuracy, thereby providing an accurate data basis for subsequent feature extraction and similarity calculation.

[0071] The total loss value is calculated based on the separated subject image, the separated foreground image and the separated background image, and the standard subject image, the standard foreground image and the standard background image, specifically including: calculating a first similarity between the standard subject image and the separated subject image, a second similarity between the separated foreground image and the standard foreground image, and a third similarity between the separated background image and the standard background image; the first similarity, the second similarity and the third similarity are averaged to obtain an average similarity as the total loss value, which is simple and fast. In addition, after the first similarity, the second similarity and the third similarity are calculated, the weight of the first similarity, the weight of the second similarity and the weight of the first similarity can be determined, wherein the weight of the first similarity is greater than the weight of the second similarity and the weight of the third similarity. Then, the first similarity, the second similarity and the third similarity are weighted and averaged according to the corresponding weights to obtain the total loss value. By strengthening the weight of the first similarity, a total loss value more biased towards the subject image is obtained, thereby improving the accuracy of the target image segmentation model.

[0072] In other embodiments, the to-be-searched image can also be separated in other ways, for example, according to the size of the image, the to-be-searched image is separated into three parts according to a certain proportion, and the background image, the subject image and the foreground image are separated from top to bottom according to the height direction of the to-be-searched image. The proportion of the subject image is greater than the proportion of the background image and the proportion of the foreground image, and the proportion of the background image is greater than or equal to the proportion of the foreground image. This method is simple and convenient.

[0073] S20: Extract features of the subject image, the foreground image and the background image respectively.

[0074] After obtaining the subject image, the foreground image and the background image, the server needs to extract features of the subject image, the foreground image and the background image respectively. The features of the subject image, the foreground image and the background image can be extracted by a preset feature extraction model. The preset feature extraction model is a visual feature model trained based on a plurality of standard historical images.

[0075] That is, the visual content of the subject image is extracted by the preset feature extraction model to obtain the subject feature of the search image; the visual content of the foreground image is extracted by the preset feature extraction model to obtain the foreground feature of the search image; and the visual content of the background image is extracted by the preset feature extraction model to obtain the background feature of the search image. The visual content includes one or more of light intensity, color and shape.

[0076] In this embodiment, the extracted features can be pattern features (i.e. shape features) of the search image. In other embodiments, the extracted features can also include color features. That is, the subject feature includes a subject visual feature, and the subject visual feature includes a pattern feature and a color feature; the background feature includes a background visual feature, and the background visual feature includes a pattern feature and a color feature; and the foreground feature includes a foreground visual feature, and the foreground visual feature includes a pattern feature and a color feature.

[0077] The background image is more blurred than the subject image and the foreground image. To ensure the accuracy of the extracted background features, different feature extraction models can be used to extract features of different patterns. For example, the preset feature extraction model includes a background feature extraction model and a subject feature extraction model. The subject image is extracted by the preset feature extraction model to obtain the subject feature of the search image. The background image is extracted by the background feature extraction model to obtain the background feature of the search image, so as to improve the accuracy of the features. In addition, the foreground feature of the search image can be extracted by the preset feature extraction model, or the foreground feature of the search image can be extracted by the background feature extraction model.

[0078] S30: Fuse the subject feature, the foreground feature and the background feature to obtain an image feature of the search image.

[0079] After obtaining the subject feature, the foreground feature and the background feature of the to-be-searched image, the server needs to perform feature fusion on the subject feature, the foreground feature and the background feature to obtain the image feature of the to-be-searched image.

[0080] For example, the subject feature, the foreground feature and the background feature can be directly spliced to obtain the image feature of the to-be-searched image, that is, the image feature of the to-be-searched image includes the subject feature, the foreground feature and the background feature, which is simple and convenient. Alternatively, the weight of the subject feature, the weight of the foreground feature and the weight of the background feature can be determined respectively, wherein the weight of the subject feature is greater than the weight of the foreground feature and the weight of the background feature; then, the subject feature, the foreground feature and the background feature are weighted and summed according to the corresponding weights to obtain the image feature of the to-be-searched image, which strengthens the weight of the subject feature on the basis of ensuring the complete image feature, and obtains a more accurate pattern feature.

[0081] In other embodiments, the positions of the subject image, the foreground image and the background image in the to-be-searched image can also be determined respectively, and then the subject feature, the foreground feature and the background feature are spliced according to the positions of the subject image, the foreground image and the background image in the to-be-searched image to obtain the image feature of the to-be-searched image, which further emphasizes the spatial positions of the features on the basis of ensuring the complete image feature, thereby further ensuring the accuracy of the image feature.

[0082] S40: performing similarity calculation on the image feature of the to-be-searched pattern and the standard image feature of the standard historical image to obtain the similarity of the plurality of standard historical images.

[0083] After performing feature fusion on the subject feature, the foreground feature and the background feature to obtain the image feature of the to-be-searched image, the server needs to perform similarity calculation on the image feature of the to-be-searched pattern and the standard image feature of each standard historical image to obtain the similarity of the standard historical image, and traverse all the standard historical images to obtain the similarity of the plurality of standard historical images.

[0084] Before performing the similarity calculation, the plurality of standard historical images and the standard image feature corresponding to each standard historical image need to be obtained. The standard image feature is obtained in the same way as the image feature, for example, the standard historical image can be first separated to obtain the subject image, the foreground image and the background image of the standard historical image; then, the subject feature, the foreground feature and the background feature of the standard historical image are extracted respectively to obtain the subject feature, the foreground feature and the background feature of the standard historical image; and then, the subject feature, the foreground feature and the background feature of the standard historical image are fused to obtain the standard image feature of the standard historical image, wherein the fusion manner of the standard historical image is the same as the fusion manner of the to-be-searched image, which will not be described herein again.

[0085] In this embodiment, feature extraction processing can be performed in advance on each standard historical image to obtain a standard image feature of each standard historical image, and then the standard historical image and the standard image feature thereof are stored in a database, so as to facilitate subsequent direct calling, reduce the data processing amount required for online feature extraction, reduce the server load, and improve the response speed.

[0086] When the image feature of the to-be-searched pattern includes a subject feature, a foreground feature and a background feature, the standard image feature of the standard historical image includes a standard subject feature, a standard foreground feature and a standard background feature. At this time, the similarity between the image feature of the to-be-searched pattern and the standard image feature of the standard historical image is calculated to obtain the similarity of the standard historical image, including: calculating the similarity between the subject feature and the standard subject feature to obtain a subject similarity, and calculating the similarity between the foreground feature and the standard foreground feature to obtain a foreground similarity, and calculating the similarity between the background feature and the standard background feature to obtain a background similarity; determining the weight of the subject similarity, the weight of the background similarity and the weight of the foreground similarity, wherein the weight of the subject similarity is greater than the weight of the background similarity and the weight of the foreground similarity; and performing weighted average on the subject similarity, the background similarity and the foreground similarity according to the weight of the subject similarity, the weight of the background similarity and the weight of the foreground similarity to obtain the similarity of the standard historical image. In this embodiment, the similarity of the subject feature, the foreground feature and the background feature is calculated, and then the weighted average of the similarities of different types is performed according to different weights, thereby improving the accuracy of the similarity of the standard historical image; in addition, the weight of the subject similarity is the largest, which enhances the proportion of the subject similarity, so that the similarity of the standard historical image is closer to the subject image, and the accuracy of the similarity calculation result is further improved. In this embodiment, the similarity calculation can be a covariance matrix calculation method, that is, the covariance of two features is calculated to obtain the covariance matrix between the two features as the similarity between the two features, wherein the covariance matrix includes the covariance of each feature point in the feature, which effectively improves the accuracy of the similarity calculation result. In other embodiments, the feature fusion method can also be other methods, for example, the similarity between the image feature of the to-be-searched pattern and the standard image feature of the standard historical image is directly calculated to obtain the similarity of the standard historical image, which is simple and fast.

[0087] S50: Push the search result of the to-be-searched image to the terminal device according to the similarity of each standard historical image.

[0088] After similarity calculation is performed between the image feature of the image to be searched and the standard image feature of the standard historical image, and the similarities of the plurality of standard historical images are obtained, the search result of the image to be searched is pushed to the terminal device according to the similarity of each standard historical image. For example, the plurality of standard historical images are sorted in descending order of similarity, image sorting data are obtained, and the image sorting data are taken as the search result of the image to be searched, and then the search result of the image to be searched is pushed to the terminal device for the user to browse and select the required image.

[0089] In the embodiment, the image to be searched is acquired and separated to obtain a subject image, a foreground image and a background image of the image to be searched; the subject image, the foreground image and the background image are respectively subjected to feature extraction to obtain subject feature, foreground feature and background feature of the image to be searched; the subject feature, the foreground feature and the background feature are subjected to feature fusion to obtain image feature of the image to be searched; similarity calculation is performed between the image feature of the image to be searched and the standard image feature of the standard historical image to obtain the similarities of the plurality of standard historical images; the search result of the image to be searched is pushed to the terminal device according to the similarity of each standard historical image; the image to be searched is separated into the subject image, the foreground image and the background image, and the image features of the subject image, the foreground image and the background image are respectively extracted and fused to obtain the image feature, the granularity of image content extraction is improved by refining the visual content of the image and respectively extracting the features, richer image details can be extracted, the description precision of the image feature to the image is improved, and the accuracy of the image search result is improved.

[0090] In an embodiment, the subject feature includes subject visual feature and subject theme feature, the background feature includes background visual feature and background theme feature, and the foreground feature includes foreground visual feature and foreground theme feature. In step S20, the subject image, the foreground image and the background image are respectively subjected to feature extraction to obtain the subject feature, the foreground feature and the background feature of the image to be searched, including: the subject image, the foreground image and the background image are respectively subjected to visual feature extraction to obtain the subject visual feature, the background visual feature and the foreground visual feature; the subject image, the foreground image and the background image are respectively subjected to theme feature extraction by a theme feature extraction model to obtain the subject theme feature, the background theme feature and the foreground theme feature; the subject visual feature and the subject theme feature are taken as the subject feature, the background visual feature and the background theme feature are taken as the background feature, and the foreground visual feature and the foreground theme feature are taken as the foreground feature. On the basis of the visual feature extraction, the theme feature extraction is added, information extraction is performed on the image to be searched from multiple dimensions, the accuracy of the subject feature, the foreground feature and the background feature is improved, and the description precision of the image feature to the image is further improved.

[0091] Correspondingly, in step S20, the subject feature, the foreground feature and the background feature are fused to obtain the image feature of the image to be searched, specifically including: first, the subject theme feature, the background theme feature and the foreground theme feature are fused to obtain the theme fusion feature; then, the subject visual feature, the background visual feature and the background visual feature are fused to obtain the visual fusion feature; the theme fusion feature and the visual fusion feature are spliced to obtain the image feature of the image to be searched, the fusion method is simple, and the image feature not only contains visual content but also includes image theme information, and the accuracy is higher. Or, After the main subject feature, the background subject feature and the foreground subject feature are fused to obtain the subject fusion feature, Figure 3 The theme fusion feature is activated by weight (for example, the theme fusion feature is activated by a preset activation function), to obtain activation weight data, and then the activation weight data is assigned to the visual fusion feature to obtain the image feature of the image to be searched. The visual fusion feature is enhanced by the theme fusion feature to obtain more accurate image features.

[0092] In an embodiment, as shown in Figure 1 In step S20, the subject feature, the foreground feature and the background feature of the image to be searched are obtained by respectively extracting the features of the subject image, the foreground image and the background image, or the subject feature, the foreground feature and the background feature of the image to be searched are obtained by respectively extracting the visual features of the subject image, the foreground image and the background image, specifically including the following steps:

[0093] S21: extracting the features of the subject image to obtain the subject pattern feature, and extracting the features of the subject image to obtain the subject color feature, and combining the subject pattern feature and the subject color feature into the subject feature.

[0094] After the image to be searched is separated to obtain the subject image, the foreground image and the background image, in order to ensure the accuracy of feature extraction, different feature extraction models can be used to extract different visual contents of the same image to obtain corresponding visual features. For example, different light intensities exhibit different patterns, so a pattern feature extraction model can be used to extract shape and light intensity information of the image, and a color feature extraction model can be used to extract color information of the image.

[0095] That is, after the image to be searched is separated to obtain the subject image, the foreground image and the background image, a pattern feature extraction model is used to extract the shape and light intensity information of the subject image to obtain the subject pattern feature; a color feature extraction model is used to extract the color information of the subject image to obtain the subject color feature; then, the subject pattern feature and the subject color feature are combined into the subject feature, and at this time, the subject feature includes visual features such as color, shape and light intensity information, ensuring the accuracy of the subject feature.

[0096] S22: pattern feature extraction is performed on the foreground image and the background image respectively to obtain foreground pattern features and background pattern features.

[0097] After the to-be-searched image is separated to obtain the subject image, the foreground image and the background image, pattern feature extraction is performed on the foreground image and the background image respectively by using a pattern feature extraction model to obtain foreground pattern features and background pattern features. That is, the shape and light intensity information of the foreground image are extracted by using the pattern feature extraction model to obtain the foreground pattern features, and the shape and light intensity information of the background image are extracted by using the pattern feature extraction model to obtain the background pattern features, so as to ensure the accuracy of the foreground pattern features and the background pattern features.

[0098] Among them, since the subject Figure 4 Generally, the imaging quality of the subject image is higher than that of the background image and / or the foreground image, so as to ensure the accuracy of the extracted pattern features, different pattern feature extraction models can be used to extract the pattern features of the subject image, the foreground image and the background image. For example, the plurality of standard historical images and the background images, the foreground images and the subject images of the standard historical images can be obtained, the subject images of the plurality of standard historical images are used as training samples to train a first pattern feature extraction model, and the foreground images of the plurality of standard historical images are used as training samples to train a second pattern feature extraction model. Meanwhile, the background images of the plurality of standard historical images are used as training samples to train a third pattern feature extraction model; then, the first pattern feature extraction model is used to extract the pattern features of the subject image to obtain the subject pattern features, the second pattern feature extraction model is used to extract the pattern features of the foreground image to obtain the foreground pattern features, and the third pattern feature extraction model is used to extract the pattern features of the background image to obtain the background pattern features. By using more subdivided and accurate training samples to train the corresponding pattern feature extraction model, the feature extraction accuracy of each pattern feature extraction model for the corresponding type of image can be improved, so as to improve the accuracy of each type of pattern feature.

[0099] S23: color feature extraction is performed on the foreground image and the background image respectively to obtain foreground color features and background color features.

[0100] Meanwhile, color feature extraction is performed on the foreground image and the background image respectively by using a color feature extraction model to obtain foreground color features and background color features. That is, the color information of the foreground image is extracted by using the color feature extraction model to obtain the foreground color features, and the color information of the background image is extracted by using the color feature extraction model to obtain the background color features, so as to ensure the accuracy of the foreground color features and the background color features.

[0101] In an embodiment, different color feature extraction models can be used to extract the color features of the subject image, the foreground image and the background image, so as to improve the accuracy of the extracted color features. The specific extraction process of the color features can refer to the aforementioned pattern feature extraction process, which will not be described here.

[0102] S24: The foreground pattern features and the foreground color features are summarized as foreground features, and the background pattern features and the background color features are summarized as background features.

[0103] After obtaining the foreground pattern features, the background pattern features, the foreground color features and the background color features, the foreground pattern features and the foreground color features are summarized as foreground features, and the background pattern features and the background color features are summarized as background features, so that the foreground features and the background features both include color, shape and light intensity information and other visual features, thereby ensuring the accuracy of the foreground features and the background features.

[0104] In the embodiment, the subject pattern features are obtained by performing feature extraction on the subject image, and the subject color features are obtained by performing feature extraction on the subject image. The subject pattern features and the subject color features are summarized as subject features. The foreground pattern features and the background pattern features are obtained by performing pattern feature extraction on the foreground image and the background image respectively. The foreground color features and the background color features are obtained by performing color feature extraction on the foreground image and the background image respectively. Then, the foreground pattern features and the foreground color features are summarized as foreground features, and the background pattern features and the background color features are summarized as background features. The specific process of performing feature extraction on the subject image, the foreground image and the background image to obtain the subject features, the foreground features and the background features of the image to be searched is clarified. The features of the image are extracted from the pattern (i.e. light intensity and shape), color and other multi-dimensional visual content, which improves the accuracy of the subject features, the foreground features and the background features and provides an accuracy calculation basis for the subsequent steps.

[0105] In an embodiment, the subject features include the subject pattern features and the subject color features, the foreground features include the foreground pattern features and the foreground color features, and the background features include the background pattern features and the background color features. As shown in FIG. 30, in step S30, the feature fusion is performed on the subject features, the foreground features and the background features to obtain the image features of the image to be searched, which specifically includes the following steps: Figure 5

[0106] S31: The subject pattern features are enhanced based on the subject color features to obtain enhanced subject features.

[0107] ​In this embodiment, the subject features include subject pattern features and subject color features; the foreground features include foreground pattern features and foreground color features; and the background features include background pattern features and background color features. After obtaining the subject features, the subject pattern features need to be enhanced based on the subject color features to obtain enhanced subject features. For example, the subject color features can be directly spliced with the subject pattern features, and the spliced features can be taken as the enhanced subject features, directly adding color features to the pattern features, which is simple and convenient.

[0108] The subject pattern features are enhanced based on the subject color features to obtain enhanced subject features, and specifically include: performing weight conversion based on an attention mechanism based on the subject color features to obtain subject weight data, and then performing feature enhancement on the subject pattern features based on the subject weight data to obtain the enhanced subject features. The subject color features are taken as weight values and are assigned to each feature point (pixel point) of the subject pattern features, which realizes adaptive increase of the subject pattern features and effectively enhances the subject pattern features.

[0109] For example, the similarity between the subject color features and the background color features can be calculated to obtain first similarity data, and an activation function (such as a linear activation function) can be used to convert the first color similarity data into first type weight data of the attention mechanism; the similarity between the subject color features and the foreground color features can be calculated to obtain second color similarity data, and an activation function can be used to convert the second color similarity data into second type weight data of the attention mechanism; each weight value in the first type weight data is assigned to each feature point of the subject pattern features to obtain first activated subject features, and each weight value in the second type weight data is assigned to each feature point of the subject pattern features to obtain second activated subject features; the first activated subject features and the second activated subject features are weighted and averaged according to the weights of the first activated subject features and the second activated subject features to obtain the enhanced subject features. In this embodiment, the weight conversion is performed based on the similarity between the subject color features and the background color features and the foreground color features, which are then assigned to the subject pattern features, so as to increase the difference between the subject pattern, the background pattern and the foreground pattern, effectively enhance the subject pattern features, and improve the accuracy of the enhanced subject features. In addition, the background color features and the foreground color features can be spliced to obtain spliced color features, the similarity between the subject color features and the spliced color features can be calculated to obtain target similarity data, an activation function can be used to convert the target similarity data into target weight data of the attention mechanism, and then the target weight data is assigned to each feature point of the subject pattern features to obtain the enhanced subject features, which effectively enhances the subject pattern features while reducing the computational load and the server load.

[0110] S32: Feature enhancement is performed on the foreground pattern features based on the foreground color features to obtain enhanced foreground features.

[0111] After obtaining the foreground feature, the foreground pattern feature needs to be enhanced based on the foreground color feature to obtain an enhanced foreground feature. For example, the foreground color feature and the foreground pattern feature can be directly spliced, and the spliced feature is taken as the enhanced foreground feature, directly adding the color feature to the pattern feature, which is simple and convenient.

[0112] The foreground pattern feature is enhanced based on the foreground color feature to obtain an enhanced foreground feature, specifically including: performing weight transformation based on an attention mechanism based on the foreground color feature to obtain foreground weight data, and then enhancing the foreground pattern feature based on the foreground weight data to obtain the enhanced foreground feature. The foreground color feature is taken as the weight value and is given to each feature point (pixel point) of the foreground pattern feature, realizing adaptive increase of the foreground pattern feature and effectively enhancing the foreground pattern feature. For example, the foreground color feature can be directly normalized to obtain normalized data, and then an activation function is used to transform the normalized data into foreground weight data based on the attention mechanism, and then the foreground weight data is given to each feature point of the foreground pattern feature to obtain the enhanced foreground feature.

[0113] S33: enhancing the background pattern feature based on the background color feature to obtain an enhanced background feature.

[0114] After obtaining the background feature, the background pattern feature needs to be enhanced based on the background color feature to obtain an enhanced background feature. For example, the background color feature and the background pattern feature can be directly spliced, and the spliced feature is taken as the enhanced background feature, directly adding the color feature to the pattern feature, which is simple and convenient.

[0115] In other embodiments, the background pattern feature is enhanced based on the background color feature to obtain an enhanced background feature, specifically including: performing weight transformation based on an attention mechanism based on the background color feature to obtain background weight data, and then enhancing the background pattern feature based on the background weight data to obtain the enhanced background feature. The background color feature is taken as the weight value and is given to each feature point (pixel point) of the background pattern feature, realizing adaptive increase of the background pattern feature and effectively enhancing the foreground pattern feature. For example, the background color feature can be directly normalized to obtain normalized data, and then an activation function is used to transform the normalized data into background weight data based on the attention mechanism, and then the background weight data is given to each feature point of the background pattern feature to obtain the enhanced background feature.

[0116] S34: performing feature fusion on the enhanced subject feature, the enhanced foreground feature and the enhanced background feature to obtain image features of the to-be-searched image.

[0117] After obtaining the enhanced subject feature, the enhanced foreground feature and the enhanced background feature, the enhanced subject feature, the enhanced foreground feature and the enhanced background feature are fused to obtain the image feature of the image to be searched. The enhanced background feature, the enhanced subject feature and the enhanced foreground feature can be directly spliced in sequence according to the positions of the background image, the subject image and the foreground image in the image to be searched to obtain the image feature of the image to be searched. The image feature is spliced according to the actual positions to make the image feature more close to the original image to be searched and improve the accuracy of the image feature.

[0118] In an embodiment, the fusion manner can also be: determining a first weight of the enhanced subject feature, a second weight of the enhanced foreground feature and a third weight of the enhanced background feature, and the first weight, the second weight and the third weight decrease in sequence; and performing weighted summation on the enhanced foreground feature and the enhanced background feature according to the first weight, the second weight and the third weight to obtain the image feature of the image to be searched. The weight of the enhanced subject feature is increased to strengthen the feature of the subject image, so that the subject image is more prominent in the image feature of the image to be searched and the image expression ability of the image feature is increased.

[0119] In this embodiment, the subject pattern feature is enhanced based on the subject color feature to obtain the enhanced subject feature, the foreground pattern feature is enhanced based on the foreground color feature to obtain the enhanced foreground feature, and the background pattern feature is enhanced based on the background color feature to obtain the enhanced background feature. Then, the enhanced subject feature, the enhanced foreground feature and the enhanced background feature are fused to obtain the image feature of the image to be searched. The subject feature, the foreground feature and the background feature are fused to obtain the image feature of the image to be searched. Each image after separation is enhanced based on the color itself to make the image feature more accurate, thereby improving the accuracy of the image feature obtained by subsequent fusion.

[0120] In an embodiment, before step S10, i.e. before the foreground pattern and the background pattern of the image to be searched are obtained by separation processing, the method further includes the following steps:

[0121] S01: identifying the image to be searched to determine whether the image to be searched is a first type of image.

[0122] In this embodiment, the first type of image is an image having a background and a foreground. Before the subject image, the foreground image and the background image of the image to be searched are obtained by separation processing, the foreground and the background of the image to be searched need to be identified to determine whether the image to be searched has a background and a foreground, i.e. to determine whether the image to be searched is a first type of image.

[0123] The method comprises the following steps: identifying the foreground and the background of the image to be searched, specifically: performing color block identification on the image to be searched to determine the color of each color block in the image to be searched and the position of each color block; dividing the image to be searched into regions according to the height direction of the image to be searched to obtain a background region, a main body region and a foreground region from top to bottom; determining whether the color of the color block located in the background region is the same as the color of the color block located in the main body region, and determining whether the color of the color block located in the foreground region is the same as the color of the color block located in the main body region; if the color of the color block located in the background region is not the same as the color of the color block located in the main body region, and the color of the color block located in the foreground region is not the same as the color of the color block located in the main body region, it is determined that the image to be searched has a background and a foreground, and the image to be searched is a first type of image; if the color of the color block located in the background region is not the same as the color of the color block located in the main body region, or the color of the color block located in the foreground region is not the same as the color of the color block located in the main body region, it is determined that the image to be searched does not have a background and / or a foreground, and the image to be searched is a second type of image. In the embodiment, the color of each color block in the image to be searched is used for confirmation, which is simple, convenient and has a small amount of calculation.

[0124] In other embodiments, the foreground-background identification model can also be pre-trained. After the image to be searched is obtained, the image to be searched is input into the foreground-background identification model for foreground-background identification. If the foreground and the background of the image to be searched are identified, it is output that the type of the image to be searched is a first type of image. If the foreground and / or the background of the image to be searched are not identified, it is output that the type of the image to be searched is a second type of image, which is high in accuracy.

[0125] S02: If the image to be searched is a first type of image, the image to be searched is separated to obtain a foreground pattern and a background pattern of the image to be searched.

[0126] After it is determined whether the image to be searched is a first type of image, if it is determined that the image to be searched is a first type of image, that is, the image to be searched has a specific foreground and a background, the image to be searched is separated to obtain a foreground pattern and a background pattern of the image to be searched, which facilitates subsequent feature extraction and fusion, and then similarity calculation is performed based on the image features obtained through fusion, so that an accurate image retrieval result is obtained.

[0127] In the embodiment, before the image to be searched is separated to obtain a foreground pattern and a background pattern of the image to be searched, it is determined whether the image has a foreground and a background. Only the image to be searched having a foreground and a background is separated, so that invalid separation is avoided, and accurate main body pattern, background pattern and foreground pattern features are extracted, which provides an accurate data basis for subsequent calculation.

[0128] S03: If the image to be searched is a second type of image, pattern identification is performed on the image to be searched to obtain a main body pattern of the image to be searched.

[0129] After determining whether the image to be searched is a first type of image, if the image to be searched is a second type of image, it indicates that the image to be searched has no background or foreground, or neither the background nor the foreground has, and it is necessary to perform pattern recognition on the subject pattern of the image to be searched to obtain the subject image of the image to be searched. The second type of image can be a pure plant image, a pure animal image, a human face image, and the like.

[0130] In this embodiment, the subject pattern of the image to be searched can be directly subjected to pattern recognition by using the foregoing target image segmentation model to obtain the subject image of the image to be searched, which is simple and convenient and reduces the data processing amount of the server for loading other models. In other embodiments, a large number of background-free and / or foreground-free images can be used as training samples to perform deep learning to obtain a preset pattern recognition model, and then when it is determined that the image to be searched is a second type of image, the preset pattern recognition model is used to recognize the image to be searched to obtain the subject image of the image to be searched. The preset pattern recognition model has higher accuracy, and the recognized subject image is more accurate.

[0131] S04: performing pattern feature extraction on the subject image of the image to be searched to obtain the pattern feature of the image to be searched, and performing color feature extraction on the subject image of the image to be searched to obtain the color feature of the image to be searched.

[0132] After the pattern recognition on the image to be searched is performed to obtain the subject image of the image to be searched, the pattern feature extraction on the subject image of the image to be searched is performed to obtain the pattern feature of the image to be searched, and the color feature extraction on the subject image of the image to be searched is performed to obtain the color feature of the image to be searched. The pattern feature extraction process and the color feature extraction process are described above, and will not be described here again.

[0133] S05: performing theme feature extraction on the subject image of the image to be searched to obtain the theme feature of the image to be searched.

[0134] Meanwhile, the theme feature extraction model trained in advance is also needed to perform theme feature extraction on the theme of the subject image of the image to be searched to obtain the theme feature of the image to be searched. The theme of the image indicates the pattern theme displayed by the image, and the theme feature indicates the context feature of the pattern theme. For example, the image to be searched is an image showing a golden retriever running, the theme of the image to be searched is that the golden retriever is running, and the corresponding theme feature is the context feature of the theme.

[0135] S06: performing feature fusion on the pattern feature, the color feature, and the theme feature to obtain the image feature of the image to be searched.

[0136] After obtaining the subject feature, the pattern feature and the color feature of the image to be searched, the pattern feature, the color feature and the subject feature need to be fused to obtain the image feature of the image to be searched. When it is determined that the image to be searched is the second type of image, i.e. the image without background or foreground, or the image with both foreground and background, the subject feature and the color feature are added on the basis of the pattern feature to ensure the accuracy of the image feature.

[0137] For example, the pattern feature, the color feature and the subject feature can be directly spliced to obtain the image feature of the image to be searched, which is simple and convenient, and describes the image feature from multiple dimensions to improve the accuracy of the image feature. In other embodiments, different weights can be set for the subject feature, the pattern feature and the color feature, wherein the weight of the pattern feature is the largest, and then the pattern feature, the color feature and the subject feature are weighted and summed based on the corresponding weights to obtain the image feature of the image to be searched, which emphasizes the pattern feature and further improves the accuracy of the image feature.

[0138] After obtaining the image feature of the image to be searched, the foregoing steps S40 and S50 are repeated, i.e. the image feature of the image to be searched is calculated for similarity with the standard image feature of the standard historical image to obtain the similarity of the plurality of standard historical images, and then the search result of the image to be searched is pushed to the terminal device according to the similarity of each standard historical image. For example, the plurality of standard historical images are sorted in descending order of similarity to obtain image sorting data, and the image sorting data is taken as the search result of the image to be searched, and then the search result of the image to be searched is pushed to the terminal device for the user to browse and select the required image.

[0139] In this embodiment, before the image to be searched is separated to obtain the foreground pattern and the background pattern of the image to be searched, the image to be searched is first identified to determine whether the image to be searched is the first type of image. If yes, the image to be searched is separated to obtain the foreground pattern and the background pattern of the image to be searched; if no, the subject pattern of the image to be searched is obtained by pattern recognition, and then the pattern feature of the image to be searched is extracted from the subject pattern, and the color feature of the image to be searched is extracted from the subject pattern. In addition, the subject feature of the image to be searched is extracted from the subject pattern to obtain the subject feature of the image to be searched. Finally, the pattern feature, the color feature and the subject feature are fused to obtain the image feature of the image to be searched. Different image feature extraction strategies are adopted for different types of images, so that the method is applicable to more types of images, and the extraction accuracy of the image feature is also ensured.

[0140] In an embodiment, in step S06, the pattern feature, the color feature and the theme feature are fused to obtain the image feature of the image to be searched, specifically including the following steps:

[0141] S061: determining the weight of the color feature and the weight of the theme feature, and performing weighted summation on the color feature and the theme feature based on the weight of the color feature and the weight of the theme feature to obtain the fused feature.

[0142] If the image to be searched is the first type of image, after obtaining the pattern feature, the color feature and the theme feature of the image to be searched, the weight (the fourth weight) of the color feature and the weight (the fifth weight) of the theme feature need to be determined, and then the color feature and the theme feature are weighted and summed based on the weight of the color feature and the weight of the theme feature, that is, the color feature and the theme feature are weighted and summed based on the fourth weight and the fifth weight to obtain the fused feature.

[0143] Since the color feature is a more accurate visual feature, in an embodiment, the weight of the color feature is greater than the weight of the theme feature, that is, the fourth weight needs to be greater than the fifth weight to highlight the color feature, so that the fused feature represents the visual content.

[0144] S062: performing weight conversion on the fused feature based on the attention mechanism to obtain fused weight data.

[0145] After obtaining the fused feature, the fused feature needs to be converted based on the attention mechanism to obtain a plurality of weight values of the attention mechanism, that is, the fused weight data.

[0146] For example, the fused feature can be normalized to obtain normalized data, and then an activation function is used to convert the fused feature into a plurality of weight values of the attention mechanism, thereby obtaining the fused weight data. The activation function can be a sigmod function. The sigmod function is also called a Logistic function, which is used for the output of a hidden layer neural unit, and the value range is (0, 1). The sigmod function can map a real number to the interval (0, 1), and can be used for binary classification. The sigmod function has a good effect when the features are relatively complex or not very different. Since the color feature and the theme feature are quite different, the sigmod function has a better effect on weight conversion. In other embodiments, the activation function can also be other, for example, a linear activation function.

[0147] S063: performing feature enhancement on the pattern feature of the image to be searched based on the fused weight data to obtain the image feature of the image to be searched.

[0148] After obtaining the fusion weight data, the pattern feature of the to-be-searched image is enhanced based on the fusion weight data to obtain the image feature of the to-be-searched image, that is, the multiple weight values of the fusion weight data are respectively assigned to the feature points of the pattern feature to enhance the feature points of the pattern feature, so that the accurate image feature of the to-be-searched image is obtained.

[0149] In this embodiment, the weight of the color feature and the weight of the theme feature are determined, the color feature and the theme feature are weighted and summed based on the weight of the color feature and the weight of the theme feature to obtain the fusion feature, then the fusion feature is converted based on the attention mechanism to obtain the fusion weight data, and finally the pattern feature of the to-be-searched image is enhanced based on the fusion weight data to obtain the image feature of the to-be-searched image. The specific process of fusing the pattern feature, the color feature and the theme feature to obtain the image feature of the to-be-searched image is determined, the weighted sum of the color feature and the theme feature is taken as an auxiliary feature to enhance the pattern feature, which can increase the expression ability of the image feature to the original to-be-searched image and improve the accuracy of the image feature.

[0150] In an embodiment, each standard historical image corresponds not only to a standard image feature but also to a standard image style. As shown in Figure 6 In step S50, the search result of the to-be-searched image is pushed to the terminal device according to the similarity of each standard historical image, which specifically includes the following steps:

[0151] S51: performing style recognition on the to-be-searched image through a style recognition network to obtain the image style of the to-be-searched pattern.

[0152] In this embodiment, each standard historical image corresponds not only to a standard image feature but also to a standard image style. After calculating the similarity of the to-be-searched pattern image feature and the standard historical image standard image feature to obtain the similarity of multiple standard historical images, the image style of the to-be-searched pattern is first obtained by performing style recognition on the to-be-searched image through a style recognition network. The style recognition network is a style classification network obtained by deep learning training with multiple historical images of different image styles as training samples, which can identify the image style of different images.

[0153] The image style includes plant images, animal images, landscape images, human images, holiday images and graphic images. Different image styles show different contents, for example, the main pattern of a plant image is a plant, and the main pattern element of a holiday image is a holiday element, such as the Spring Festival and the National Day.

[0154] S52: Match the image style of the image to be searched with the standard image style of each standard historical image, and record the standard historical image that matches as the image to be recommended.

[0155] After the image style of the image to be searched is obtained by the style recognition network, the image style of the image to be searched is matched with the standard image style of each standard historical image, and the standard historical image that matches is recorded as the image to be recommended, thereby obtaining a plurality of images to be recommended.

[0156] For example, if the image style of the image to be searched is obtained by the style recognition network, the image style of the image to be searched is a landscape image, and the standard image style of the standard historical image is a plant image, an animal image, a human image, a holiday image, a graphic image, and the like, which is a non-style image, it indicates that the standard historical image does not match the image style of the image to be searched, and the next standard historical image needs to be pulled for image style matching with the image to be searched. If the standard image style of the standard historical image is also a landscape image, it indicates that the standard historical image matches the image style of the image to be searched, and the standard historical image is recorded as the image to be recommended, and then the next standard historical image is pulled for image style matching with the image to be searched, until all standard historical images are traversed, and a plurality of images to be recommended are obtained.

[0157] In other embodiments, all standard historical images in the database can also be classified by image style to obtain standard historical images under different image styles. After the image style of the image to be searched is determined, all standard historical images under the image style of the image to be searched are pulled and recorded as images to be recommended, which is simple and convenient.

[0158] S53: Determine whether the number of images to be recommended is less than a preset number.

[0159] After obtaining a plurality of images to be recommended, it is determined whether the number of images to be recommended is less than a preset number. The preset number is determined according to the screen size of the terminal device. Different terminal devices have different screen sizes that can display different numbers of images. Therefore, the server needs to determine the type and model of the terminal device used by the user, and determine the screen size of the terminal device according to the type and model of the terminal device, and then determine the maximum number of images that can be accommodated by the terminal screen at one time according to the screen size, and take the maximum number of images as the preset number.

[0160] S54: If the number of images to be recommended is greater than or equal to the preset number, the plurality of images to be recommended are sorted in descending order according to the similarity of the images to be recommended, and the search recommendation data of the image to be searched is obtained.

[0161] After determining whether the number of the to-be-recommended images is less than the preset number, if the number of the to-be-recommended images is greater than or equal to the preset number, indicating that the number of the to-be-recommended images reaches or exceeds the capacity of the terminal device screen, the plurality of to-be-recommended images are sorted in descending order according to the similarity of the to-be-recommended images, and the search recommendation data of the to-be-searched image is obtained.

[0162] S55: pushing the search recommendation data as the search result of the to-be-searched image to the terminal device.

[0163] After sorting the plurality of to-be-recommended images in descending order according to the similarity of the to-be-recommended images and obtaining the search recommendation data, the search recommendation data is pushed as the search result of the to-be-searched image to the terminal device, so that the user can see the recommended images with higher similarity in the first time.

[0164] In this embodiment, the style recognition network is used to perform style recognition on the to-be-searched image to obtain the image style of the to-be-searched image, and then the image style of the to-be-searched image is matched with the standard image style of each standard historical image, and the standard historical image that is successfully matched is recorded as a to-be-recommended image. Then, it is determined whether the number of the to-be-recommended images is less than the preset number. If the number of the to-be-recommended images is greater than or equal to the preset number, the plurality of to-be-recommended images are sorted in descending order according to the similarity of the to-be-recommended images, and the search recommendation data of the to-be-searched image is obtained. Finally, the search recommendation data is pushed as the search result of the to-be-searched image to the terminal device. The specific process of pushing the search result of the to-be-searched image to the terminal device according to the similarity of each standard historical image is clear. Before the search result is pushed, the image style is matched first, which can improve the accuracy of the subsequent pushed images and reduce the number of pushed images. In addition, the preset number is determined according to the screen size of the terminal device, which can push more images to the user on one recommended page as much as possible, and ensures that the user can select a suitable image in the search result.

[0165] In an embodiment, after step S53, that is, after determining whether the number of the to-be-recommended images is less than the preset number, the method further includes the following steps:

[0166] S56: If the number of the to-be-recommended images is less than the preset number, the first n to-be-recommended images with the largest similarity in the plurality of to-be-recommended images are selected as the first recommended data, and the remaining standard historical images are sorted in descending order according to the size of the similarity to obtain the second recommended data.

[0167] After determining whether the number of images to be recommended is less than a preset number, if the number of images to be recommended is less than the preset number, it means that the number of images to be recommended is less than the amount that the terminal device screen can accommodate. Then, the top n images with the highest similarity among the multiple images to be recommended are selected as the first recommended data, and the remaining standard historical images are sorted in descending order according to the size of similarity to obtain the second recommended data. Here, n is less than the preset number, and n can be the preset number - 3.

[0168] S57: The first and second recommended data are used as search results for the image to be searched and pushed to the terminal device.

[0169] Then, the first and second recommended data are used as search results for the image to be searched and pushed to the terminal device, with the first recommended data displayed before the second. When the number of images to be recommended is less than the capacity of the terminal device screen (i.e., the number of standard historical images with the same image style as the image to be searched is less than the capacity of the terminal device screen), only the top n images with the same image style and the highest similarity are selected. Then, the search results are obtained by sorting them in descending order according to the similarity of the other standard historical images. This ensures both the accuracy and diversity of search results. Furthermore, displaying the first recommended data before the second recommended data allows users to see the images with the same image style and the highest similarity first on their terminal device, improving the user search experience.

[0170] In this embodiment, after determining whether the number of images to be recommended is less than a preset number, if the number of images to be recommended is less than the preset number, the top n images with the highest similarity among the multiple images to be recommended are selected as the first recommended data, and the remaining standard historical images are sorted in descending order according to the similarity to obtain the second recommended data. Then, the first recommended data and the second recommended data are used as the search results for the image to be searched and pushed to the terminal device. The first recommended data is displayed before the second recommended data. While ensuring the accuracy of the search results, the diversity of the search results is also guaranteed, and the user search experience is improved.

[0171] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0172] In one embodiment, an image search device is provided, which corresponds one-to-one with the image search methods described in the above embodiments. For example... Figure 7 As shown, the image search device includes a processing module 601, an extraction module 602, a fusion module 603, a calculation module 604, and a push module 605. Detailed descriptions of each functional module are as follows:

[0173] The processing module 601 is configured to acquire a to-be-searched image sent by a terminal device, and perform separation processing on the to-be-searched image to obtain a subject image, a foreground pattern, and a background pattern of the to-be-searched image.

[0174] The extraction module 602 is configured to perform feature extraction on the subject image, the foreground pattern, and the background pattern respectively to obtain subject features, foreground features, and background features of the to-be-searched image.

[0175] The fusion module 603 is configured to perform feature fusion on the subject features, the foreground features, and the background features to obtain image features of the to-be-searched image.

[0176] The calculation module 604 is configured to perform similarity calculation on the fused features of the to-be-searched pattern and standard features of each standard historical image to obtain a similarity of each standard historical image.

[0177] The pushing module 605 is configured to push a search result of the to-be-searched image to the terminal device according to the similarity of each standard historical image.

[0178] Further, the processing module 601 is specifically configured to:

[0179] perform feature extraction on the subject image to obtain subject pattern features, and perform feature extraction on the subject image to obtain subject color features, and combine the subject pattern features and the subject color features into the subject features;

[0180] perform pattern feature extraction on the foreground image and the background image respectively to obtain foreground pattern features and background pattern features;

[0181] perform color feature extraction on the foreground image and the background image respectively to obtain foreground color features and background color features;

[0182] combine the foreground pattern features and the foreground color features into the foreground features, and combine the background pattern features and the background color features into the background features.

[0183] Further, the subject features include subject pattern features and subject color features, the foreground features include foreground pattern features and foreground color features, and the background features include background pattern features and background color features, and the fusion module 603 is specifically configured to:

[0184] perform feature enhancement on the subject pattern features based on the subject color features to obtain enhanced subject features;

[0185] perform feature enhancement on the foreground pattern features based on the foreground color features to obtain enhanced foreground features;

[0186] perform feature enhancement on the background pattern features based on the background color features to obtain enhanced background features.

[0187] The enhanced subject feature, the enhanced foreground feature, and the enhanced background feature are fused to obtain the image feature of the image to be searched.

[0188] Further, the fusion module 603 is specifically further used for:

[0189] determining a first weight of the enhanced subject feature, a second weight of the enhanced foreground feature, and a third weight of the enhanced background feature, the first weight, the second weight, and the third weight decreasing in turn;

[0190] performing weighted summation on the enhanced foreground feature and the enhanced background feature according to the first weight, the second weight, and the third weight to obtain the image feature of the image to be searched.

[0191] Further, the image searching device further comprises an identification module 606, which is used for separating processing the image to be searched to obtain a foreground pattern and a background pattern of the image to be searched.

[0192] The identification module 606 is used for identifying the image to be searched to determine whether the image to be searched is a first type of image, the first type of image being an image having a background and a foreground.

[0193] If the image to be searched is the first type of image, the processing module 601 is used for separating processing the image to be searched to obtain the foreground pattern and the background pattern of the image to be searched.

[0194] Further, after determining whether the image to be searched is the first type of image, if the image to be searched is a second type of image, the following steps are performed:

[0195] The identification module 606 is used for performing pattern identification on the image to be searched to obtain a subject pattern of the image to be searched.

[0196] The extraction module 602 is used for performing pattern feature extraction on the subject pattern of the image to be searched to obtain a pattern feature of the image to be searched, performing color feature extraction on the subject pattern of the image to be searched to obtain a color feature of the image to be searched, and performing theme feature extraction on the subject pattern of the image to be searched to obtain a theme feature of the image to be searched.

[0197] The fusion module 603 is used for fusing the pattern feature, the color feature, and the theme feature to obtain the image feature of the image to be searched.

[0198] Further, the fusion module 603 is specifically further used for:

[0199] determining a fourth weight of the color feature and a fifth weight of the theme feature, and performing weighted summation on the color feature and the theme feature based on the fourth weight and the fifth weight to obtain a fused feature.

[0200] The fusion feature is weighted and converted based on an attention mechanism to obtain fusion weight data;

[0201] The pattern feature of the to-be-searched image is enhanced based on the fusion weight data to obtain image features of the to-be-searched image.

[0202] Further, each standard historical image also corresponds to a standard image style, and the pushing module 605 is specifically configured to:

[0203] The to-be-searched image is subjected to style recognition through a style recognition network to obtain an image style of the to-be-searched pattern;

[0204] The image style of the to-be-searched image is matched with the standard image style of each standard historical image, and a standard historical image that is successfully matched is recorded as a to-be-recommended image;

[0205] It is determined whether the number of to-be-recommended images is less than a preset number, and the preset number is determined according to the size of the terminal device;

[0206] If the number of to-be-recommended images is greater than or equal to the preset number, the plurality of to-be-recommended images are sorted in descending order according to the similarity of the to-be-recommended images to obtain search recommendation data of the to-be-searched image;

[0207] The search recommendation data is pushed to the terminal device as a search result of the to-be-searched image.

[0208] The specific limitations of the image search device can be referred to the limitations of the image search method in the above, which will not be repeated here. Each module in the above image search device can be realized by software, hardware and their combination in whole or in part. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so that the processor calls and executes the operations corresponding to each module.

[0209] In one embodiment, as shown in ​ A computer device is provided, which can be a server, and the computer device includes a processor, a memory, a network interface and a database connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store data used and generated by the above image search method. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement an image search method.

[0210] In one embodiment, a computer device is provided, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the following steps when executing the computer program:

[0211] obtaining a to-be-searched image sent by a terminal device, and performing separation processing on the to-be-searched image to obtain a subject image, a foreground image, and a background image of the to-be-searched image;

[0212] performing feature extraction on the subject image, the foreground image, and the background image respectively to obtain a subject feature, a foreground feature, and a background feature of the to-be-searched image;

[0213] performing feature fusion on the subject feature, the foreground feature, and the background feature to obtain an image feature of the to-be-searched image;

[0214] performing similarity calculation on the image feature of the to-be-searched image and a standard image feature of a standard historical image to obtain a similarity of each standard historical image;

[0215] pushing a search result of the to-be-searched image to the terminal device according to the similarity of each standard historical image.

[0216] In one embodiment, a computer readable storage medium is provided, having a computer program stored thereon, the computer program being executable by a processor to implement the following steps:

[0217] obtaining a to-be-searched image sent by a terminal device, and performing separation processing on the to-be-searched image to obtain a subject image, a foreground image, and a background image of the to-be-searched image;

[0218] performing feature extraction on the subject image, the foreground image, and the background image respectively to obtain a subject feature, a foreground feature, and a background feature of the to-be-searched image;

[0219] performing feature fusion on the subject feature, the foreground feature, and the background feature to obtain an image feature of the to-be-searched image;

[0220] performing similarity calculation on the image feature of the to-be-searched image and a standard image feature of a standard historical image to obtain a similarity of each standard historical image;

[0221] pushing a search result of the to-be-searched image to the terminal device according to the similarity of each standard historical image.

[0222] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, database or other medium used in each embodiment provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory.

[0223] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0224] The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. An image search method, characterized in that, include: The process involves acquiring a search image sent by a terminal device and performing separation processing on the search image to obtain a main image, a foreground image, and a background image. This includes: dividing the search image into three parts according to its size and a certain ratio; and separating the background image, the main image, and the foreground image sequentially from top to bottom according to the height direction of the search image. The main image occupies a larger proportion than the background image and the foreground image, and the background image occupies a proportion greater than or equal to the foreground image. The main image is the visual center of the search image; the foreground of the search image refers to the scenery in front of the main image; and the background of the search image refers to the scenery behind the main image. Feature extraction is performed on the main image, the foreground image, and the background image respectively to obtain the main features, foreground features, and background features of the image to be searched. The main features include main visual features and main theme features; the background features include background visual features and background theme features; and the foreground features include foreground visual features and foreground theme features. The theme of the image represents the pattern theme displayed in the image to be searched, and the theme features represent the contextual features of the pattern theme. The visual features include pattern and color features. The image features of the image to be searched are obtained by fusing the main subject features, the foreground features, and the background features, including: fusing the main subject features, the background subject features, and the foreground subject features to obtain subject fusion features; fusing the main visual features, the background visual features, and the foreground visual features to obtain visual fusion features; and applying the subject fusion features to a preset activation function to obtain activation weight data, and assigning the activation weight data to the visual fusion features to obtain the image features of the image to be searched. The similarity between the image features of the image to be searched and the standard image features of the standard historical images is calculated to obtain the similarity between the standard historical images. The search results for the image to be searched are pushed to the terminal device based on the similarity of each of the aforementioned standard historical images.

2. The image search method as described in claim 1, characterized in that, The step of extracting features from the main image, the foreground image, and the background image respectively to obtain the main features, foreground features, and background features of the image to be searched includes: Feature extraction is performed on the main image to obtain the main pattern feature, and feature extraction is performed on the main image to obtain the main color feature. The main pattern feature and the main color feature are then combined into the main feature. Pattern features are extracted from the foreground image and the background image respectively to obtain foreground pattern features and background pattern features; Color features are extracted from the foreground image and the background image respectively to obtain foreground color features and background color features; The foreground pattern features and the foreground color features are combined into the foreground features, and the background pattern features and the background color features are combined into the background features.

3. The image search method as described in claim 1, characterized in that, The main visual features include main pattern features and main color features; the foreground visual features include foreground pattern features and foreground color features; the background visual features include background pattern features and background color features; the method includes: Based on the main color features, the main pattern features are enhanced to obtain enhanced main features; Based on the foreground color features, feature enhancement is performed on the foreground pattern features to obtain enhanced foreground features; Based on the background color features, the background pattern features are enhanced to obtain enhanced background features; The enhanced subject features, the enhanced foreground features, and the enhanced background features are fused to obtain the image features of the image to be searched.

4. The image search method as described in claim 1, characterized in that, Before performing separation processing on the image to be searched to obtain the foreground image and background image of the image to be searched, the method further includes: The image to be searched is identified to determine whether it is a first type of image, where the first type of image is an image with a background and a foreground. If the image to be searched is a first type of image, then the image to be searched is separated to obtain the foreground image and the background image of the image to be searched.

5. The image search method as described in claim 4, characterized in that, After determining whether the image to be searched is a first type of image, the method further includes: If the image to be searched is a second type of image, then pattern recognition is performed on the image to be searched to obtain the main image of the image to be searched; Pattern features are extracted from the main image of the image to be searched to obtain the pattern features of the image to be searched, and color features are extracted from the main image of the image to be searched to obtain the color features of the image to be searched. The subject features of the image to be searched are extracted by subject feature extraction to obtain the subject features of the image to be searched. The pattern features, color features, and theme features are fused to obtain the image features of the image to be searched.

6. The image search method as described in claim 5, characterized in that, The feature fusion of the pattern features, color features, and theme features to obtain the image features of the image to be searched includes: The weights of the color features and the theme features are determined, and the color features and the theme features are weighted and summed based on the weights of the color features and the theme features to obtain the fused features; The fusion features are weighted based on an attention mechanism to obtain fusion weight data. Based on the fusion weight data, feature enhancement is performed on the pattern features of the image to be searched to obtain the image features of the image to be searched.

7. The image search method according to any one of claims 1-6, characterized in that, Each of the aforementioned standard historical images also corresponds to a standard image style. The step of pushing the search results of the image to be searched to the terminal device based on the similarity of each of the aforementioned standard historical images includes: The style of the image to be searched is obtained by performing style recognition on the image to be searched through a style recognition network; The image style of the image to be searched is matched with the standard image style of each of the standard historical images, and the standard historical images that are successfully matched are recorded as images to be recommended. Determine whether the number of images to be recommended is less than a preset number; If the number of images to be recommended is greater than or equal to the preset number, then the multiple images to be recommended are sorted in descending order according to their similarity to obtain the search recommendation data for the images to be searched. The search recommendation data is pushed to the terminal device as the search result for the image to be searched.

8. An image search device, characterized in that, include: A separation module is used to acquire a search image sent by a terminal device and perform separation processing on the search image to obtain a main image, a foreground image, and a background image of the search image. This includes: separating the search image into three parts according to its size and a certain ratio; and separating the background image, the main image, and the foreground image sequentially from top to bottom according to the height direction of the search image. The main image occupies a larger proportion than the background image and the foreground image, and the background image occupies a proportion greater than or equal to the foreground image. The main image is the visual center of the search image; the foreground of the search image refers to the scenery in front of the main image; and the background of the search image refers to the scenery behind the main image. The extraction module is used to extract features from the main image, the foreground image, and the background image respectively, to obtain the main features, foreground features, and background features of the image to be searched; the main features include main visual features and main theme features, the background features include background visual features and background theme features, and the foreground features include foreground visual features and foreground theme features; the theme of the image represents the pattern theme displayed in the image to be searched, and the theme features represent the contextual features of the pattern theme; the visual features include pattern and color features; The fusion module is used to perform feature fusion on the main features, the foreground features, and the background features to obtain the image features of the image to be searched. This includes: fusing the main theme features, the background theme features, and the foreground theme features to obtain theme fusion features; fusing the main visual features, the background visual features, and the foreground visual features to obtain visual fusion features; and applying a preset activation function to the theme fusion features to obtain activation weight data, and assigning the activation weight data to the visual fusion features to obtain the image features of the image to be searched. The calculation module is used to calculate the similarity between the fused features of the image to be searched and the standard features of the standard historical images, so as to obtain the similarity of multiple standard historical images. The push module is used to push the search results of the image to be searched to the terminal device based on the similarity of each of the standard historical images.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the image search method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the image search method as described in any one of claims 1 to 7.

Images