An image recognition method, device, equipment and storage medium

Abstract

An image recognition method, device and equipment and storage medium are disclosed. The method comprises: acquiring one or more target images for retrieval; determining a user's intention mode according to the one or more target images, the intention mode being used to indicate the user's point of interest information; determining a target similar image set corresponding to the intention mode; and displaying the target similar image set. The method uses general features to achieve fine-grained retrieval in any scene, determines the direction in which the user is interested through the user's intention mode, and provides the target similar image set to the user according to the user's intention mode. The provided target similar image set can accurately meet the user's intention.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image recognition method, apparatus, device, and storage medium. Background Technology

[0002] In today's information age, users have a significant need to search for images they want from massive amounts of image data. Image search is a relatively efficient image content search strategy, where the user provides an image, and the system returns more similar images for the user to use. However, an image may contain multiple different pieces of information. When returning similar images based on the user's provided image, some images may deviate significantly from the user's expectations. For example, if a user inputs an image containing the information of a car of type A and hopes to get other images of cars of type A, but the system only recognizes the information of "car," it will return a large number of images of cars that are not type A. Summary of the Invention

[0003] This invention provides an image recognition method, apparatus, device, and storage medium to solve the problem that the images obtained during image search deviate significantly from the user's intentions. By determining the user's intentions through image recognition, the returned images can satisfy the user's intentions.

[0004] According to one aspect of the present invention, an image recognition method is provided, the method comprising:

[0005] Obtain one or more target images for retrieval;

[0006] Based on one or more target images, determine the user's intention pattern, which is used to indicate the user's point of interest information;

[0007] Determine the set of target similar images corresponding to the intention pattern, and display the set of target similar images.

[0008] According to one aspect of the present invention, an image recognition device is provided, the device comprising:

[0009] The target image acquisition module is used to acquire one or more target images for retrieval.

[0010] The intention pattern determination module is used to determine the user's intention pattern based on one or more target images, wherein the intention pattern is used to indicate the user's point of interest information;

[0011] The target similar image set determination module is used to determine the target similar image set corresponding to the intention pattern and display the target similar image set.

[0012] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0013] At least one processor; and

[0014] A memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform an image recognition method according to any embodiment of the present invention.

[0016] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement an image recognition method according to any embodiment of the present invention.

[0017] This invention discloses an image recognition method. It acquires one or more target images for retrieval, determines a user's intention pattern based on these images (the intention pattern indicates the user's points of interest), identifies a set of target-similar images corresponding to the intention pattern, and displays the set of target-similar images. It utilizes general features to achieve fine-grained retrieval in any scene, obtains the user's interests by determining the user's intention pattern, and provides the user with a set of target-similar images according to the user's intention pattern. The provided set of target-similar images accurately matches the user's intention.

[0018] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of an image recognition method provided according to Embodiment 1 of the present invention;

[0021] Figure 2 This is a schematic diagram of a user intention recognition process based on human-computer interaction according to Embodiment 1 of the present invention;

[0022] Figure 3 This is a schematic diagram of the structure of an image recognition device according to Embodiment 2 of the present invention;

[0023] Figure 4 This is a schematic diagram of the structure of an electronic device that implements an image recognition method according to an embodiment of the present invention. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] Example 1

[0027] Figure 1 The flowchart below provides an image recognition method according to Embodiment 1 of the present invention.

[0028] User intent indicates a user's points of interest, which can be understood as what the user is interested in and what the user wants to obtain. In traditional content search methods, users cannot interfere with the returned results; even if the results yield content the user does not want, there is no feedback. The image recognition method provided in this embodiment allows for user feedback on the currently returned content, and updates the search results online based on the feedback until the search results satisfy the user's intent.

[0029] Currently, fine-grained content retrieval engines rely on fine-grained features, which are generally only applicable to specific scenarios. For example, fine-grained features for automobiles cannot be used for fine-grained plant retrieval. Fine-grained, in contrast to coarse-grained, can be understood as a further subdivision of categories. For instance, the distinction between cars and airplanes is considered coarse-grained, while the distinction between brand A and brand B is fine-grained. This method, however, can achieve fine-grained retrieval applicable to any scenario corresponding to a user's intent, based on a single general feature, without relying on scenario-specific fine-grained features.

[0030] This method can be performed by an image recognition device, which can be implemented in hardware and / or software.

[0031] like Figure 1 As shown, the method includes the following steps.

[0032] S110, acquire one or more target images for retrieval.

[0033] The target image used for retrieval can be an image input by the user. In the process of image search, the target image means that it contains information related to the user's search purpose, that is, the target image contains information that the user is interested in.

[0034] S120: Based on one or more target images, determine the user's intention pattern, which is used to indicate the user's points of interest.

[0035] A user's intention pattern indicates their points of interest, i.e., the things the user is interested in. Specifically, it can indicate the types of images the user is interested in. From a target image, such as the one input by the user, multiple patterns can be derived, but not all of them are necessarily the user's intention patterns. For example, if the user is interested in brand A cars, then brand A cars in the target image represent the user's intention pattern, while non-brand A cars in the target image do not. Since there can be one or more target images, patterns contained in multiple target images can be combined to generate more information-rich patterns.

[0036] When determining a user's intention pattern based on a target image, one can first identify all the patterns contained in the target image, as well as the patterns that can be combined from multiple target images, as the user's intention pattern.

[0037] In one embodiment, S120 includes the following steps:

[0038] S120-1, Extract the first feature vector of one or more target images;

[0039] S120-2, When there is only one target image, the first feature vector is used as the user's intention pattern;

[0040] S120-3, when there are multiple target images, all the first feature vectors are fused together, and the fused first feature vector is used as the intention pattern.

[0041] When extracting the first feature vector of a target image, the target image can first be converted into a feature map of size [C*H*W], and then the feature map can be converted into a feature vector of size [C]. The feature vector can be used to calculate distance, which measures the similarity between any two images. In one implementation, the first feature vector can be obtained by averaging the feature map [2048*H*W] across the dimensions H and W to obtain a 2048-dimensional float 32 vector (float, floating-point number).

[0042] In this embodiment, there are no restrictions on the method of extracting features from the target image and converting them into a feature map.

[0043] When there is only one target image, the first feature vector can be obtained by averaging after obtaining the corresponding feature map, and the first feature vector can be used as the intention pattern.

[0044] When there are multiple target images, the first feature vectors corresponding to all target images can be obtained first. All first feature vectors can be fused. During the fusion process, the average feature vector of all first feature vectors can be calculated, and the obtained average feature vector can be used as the intention pattern.

[0045] S130, determine the set of target-similar images corresponding to the intention pattern, and display the set of target-similar images.

[0046] By collecting a large number of images of different types to form a preset image library, after determining the user's intention pattern, the preset image library can be searched according to the intention pattern to determine the similar images corresponding to the intention pattern. In other words, the images that the user is interested in are retrieved from the preset image library according to the intention pattern, and the determined similar images are combined into a target similar image set and displayed to the user.

[0047] In one embodiment, S130 includes the following steps:

[0048] S130-1, Determine the set of similar images corresponding to the intention pattern and display the set of similar images.

[0049] Before determining the target similar image that ultimately satisfies the user's intention, a set of similar images corresponding to the initially determined intention pattern can be determined first. The method for determining this set of similar images can be the same as the method for determining the target similar image set in S130. The resulting set of similar images may be the target set that satisfies the user's intention, or it may contain some errors. These errors can be gradually eliminated through subsequent interactions with the user. After determining the set of similar images, it is displayed to the user.

[0050] S130-2, If no user operation on the similar image set is detected within the set time period, then the similar image set is taken as the target similar image set.

[0051] If a user does not initiate any action on the currently displayed set of similar images after seeing it, it indicates that the user is satisfied with the current set of similar images, that it matches their interests, and that the displayed content is what the user wants. At this point, the set of similar images can be used as the target set of similar images to complete the image search.

[0052] S130-3, If a user's operation on a set of similar images is detected within a set time period, then in response to the operation, the user's operation image is determined.

[0053] If user actions on a set of similar images are detected within a set time period, it means that the images currently displayed in the set deviate significantly from the images the user actually wants. Therefore, actions are taken within the set to obtain images that better meet the user's needs. For example, to ensure that the next updated search results are closer to the user's search purpose and provide more interesting images, the user can select images from the set that match their search purpose, or they can select images from the set that are unrelated to their search purpose.

[0054] S130-4, based on operation image and operation update intention mode;

[0055] After responding to a user's action on a set of similar images and identifying the image the user interacted with, the user's intention pattern can be updated. For example, after identifying images irrelevant to the user's search purpose, these images can be removed during the intention pattern update to make the updated intention pattern closer to the user's intention.

[0056] S130-5, determine the set of similar images corresponding to the updated intention pattern, until no user operation is detected, then take the set of similar images corresponding to the intention pattern obtained in the last update as the target similar image set, and display the target similar image set.

[0057] After determining the updated intention pattern, the process of determining the set of similar images corresponding to the intention pattern can be continued using the updated intention pattern. This updated set of similar images is then displayed, allowing the user to continue interacting with the currently displayed set of similar images to obtain an updated set of similar images that more closely matches the user's intention. If no user interaction is detected within a preset time, the currently displayed set of similar images is considered to match the user's intention. In this case, the set of similar images corresponding to the last updated intention pattern is used as the target set of similar images, thus achieving the effect of identifying images that satisfy the user's intention.

[0058] In one embodiment, determining the set of similar images corresponding to the intention pattern in S130-1 includes the following steps:

[0059] Obtain the second feature vector of all images in the preset image library;

[0060] Determine the first similarity between the intention pattern and the second feature vector;

[0061] All images in the preset image library that meet the preset conditions for first similarity are identified as the set of similar images corresponding to the intention pattern.

[0062] Since feature vectors can be used to calculate distance, and distance measures the similarity between any two images, the first similarity between the intention pattern and the second feature vector can be determined by calculating the distance between the feature vector corresponding to the intention pattern and the second feature vector of all images in a preset image library. In determining the first similarity, the similarity of feature vectors can be calculated using formulas such as cosine similarity and Euclidean distance. This embodiment does not restrict the method of determining similarity.

[0063] After determining the first similarity between the intention pattern and the second feature vector, all images in the preset image library whose first similarity meets preset conditions can be identified as the set of similar images corresponding to the intention pattern. For example, the preset conditions can be the top few images with the highest similarity, or all images with similarity exceeding a specified threshold.

[0064] In one embodiment, a user's operation on a set of similar images includes: selecting an intended image from the set of similar images, and selecting a non-intended image from the set of similar images; the user's operation images include: multiple intended images, and multiple non-intended images;

[0065] In one embodiment, S130-4, based on the operation image and the operation update intention pattern, includes the following steps:

[0066] Extract the third feature vector of each intended image, and extract the fourth feature vector of each non-intended image.

[0067] Images of intent can be understood as images that users are interested in. For example, if a user wants to search for images of car brand A, then images containing car brand A in the similar image set are likely to be selected by the user as images of intent.

[0068] The third feature vector of each intended image and the fourth feature vector of each non-intended image can be extracted based on the above method of determining the feature vector of the image.

[0069] Based on the first similarity, a first weight is determined for each intended image, and a second weight is determined for each non-intended image.

[0070] Since both intention images and non-intention images are selected from a set of similar images, the similarity between the intention pattern and each intention image and each non-intention image can be obtained from the first similarity determined above. When assigning weights to intention images and non-intention images, the similarity between the image and the intention pattern can be considered, thereby assigning weights to each intention image and each non-intention image according to a preset allocation strategy. For example, an image with a higher similarity to the intention pattern in an intention image means that it is closer to the user's intention. When using intention images to identify user intentions, higher weights can be assigned to intention images with high similarity.

[0071] Based on the first weight, all the third feature vectors are fused to obtain the second fused feature vector; based on the second weight, all the fourth feature vectors are fused to obtain the third fused feature vector.

[0072] The first weight assigned to each intended image can be seen as the weight of the third feature vector corresponding to each intended image. Similarly, the second weight assigned to each non-intended image can be seen as the weight of the fourth feature vector corresponding to each non-intended image. When fusing all the third feature vectors to obtain the second fused feature vector, and when fusing all the fourth feature vectors to obtain the third fused feature vector, a weighted summation method can be used for fusion.

[0073] The second fused feature vector and the third fused feature vector are subtracted, and the difference is used as the intention pattern to complete the intention pattern update.

[0074] The second and third fused feature vectors are subtracted to obtain a new feature vector, which can be used as the updated intention pattern. Compared with the original intention pattern, the updated intention pattern includes the content that is of interest and subtracts the content that is not of interest, so it is closer to the user's intention.

[0075] In practical implementation, users can select one or more intended images and one or more unintended images from a set of similar images. If there is only one intended image or one unintended image, no fusion operation is required. For example, when there is only one intended image and one unintended image, after extracting the third feature vector of the intended image and the fourth feature vector of the unintended image, the third feature vector and the fourth feature vector can be directly subtracted, and the difference is used as the intended pattern to complete the update of the intended pattern.

[0076] In another implementation, an attention mechanism can be used. First, the similarity between the feature vectors of a given intention image and the feature vectors of other intention images is calculated. Then, a weighted sum is performed based on this similarity to obtain the improved feature vector of the intention image. Similarly, the feature vectors of other intention images are calculated, and then the feature vectors of all improved intention images are averaged to obtain the feature vector after averaging the intention images. The same process is repeated for averaging the feature vectors of non-intention images. Finally, the feature vector after averaging the intention images is subtracted from the feature vector after averaging the non-intention images, and the difference is used as the intention pattern to complete the intention pattern update.

[0077] In one embodiment, determining a first weight for each intended image based on a first similarity, and determining a second weight for each non-intended image, includes the following steps:

[0078] Based on the first similarity, the second similarity between the intention pattern and the second feature vector corresponding to each intention image is determined, and the second similarity is sorted.

[0079] Based on the first similarity, the third similarity between the intentional pattern and the second feature vector corresponding to each non-intentional image is determined, and the third similarity is sorted.

[0080] Based on the second similarity after sorting, the third weight is assigned to each intention image in turn, serving as the first weight of the intention image;

[0081] Based on the sorted third similarity, the fourth weight is assigned to each non-intended image as the second weight of the non-intended image.

[0082] When determining the first and second weights, the second similarity between the intentional pattern and the second feature vectors corresponding to each intentional image, and the third similarity between the intentional pattern and the second feature vectors corresponding to each non-intentional image can be determined from the first similarity. The second and third similarities are then sorted from largest to smallest.

[0083] Based on the second similarity score after ranking, a third weight is assigned to each intended image. During this assignment, since the goal is to identify user intent, images with higher similarity scores can be assigned higher weights; that is, higher similarity among intended images represents closer resemblance to the user's intent. Conversely, since the feature vectors of non-intended images are used for elimination, the final result contains little or no relevant content from non-intended images. Therefore, for non-intended images with lower similarity scores, more relevant content should be eliminated. Thus, when assigning a fourth weight to each non-intended image based on the third similarity score, images with lower similarity scores can be assigned higher weights.

[0084] In one embodiment, the user's operation on the set of similar images further includes: the user's operation of selecting images according to the degree of intent; the user's operation images further include: intended images sorted by the degree of intent and unintended images;

[0085] Based on the first similarity, determining the first weight of each intended image and the second weight of each non-intended image further includes the following steps:

[0086] Based on the results of users sorting by their level of interest, a fifth weight is assigned to the images of interest, with the images of interest ranking higher in the user's order of interest having a larger fifth weight.

[0087] Based on the results of users sorting by their level of intent, a sixth weight is assigned to non-intent images, with the sixth weight being larger for non-intent images that are ranked lower in the user's intent ranking.

[0088] Multiply the third weight by the fifth weight to obtain the first result, and determine the first result as the first weight;

[0089] Multiply the fourth weight by the sixth weight to obtain the second result, and determine the second result as the second weight.

[0090] In addition to selecting intended and unintended images from a set of similar images, users can also sort intended and unintended images according to their degree of intent, indicating which intended images the user is more interested in and which unintended images the user is less interested in.

[0091] Similar to the aforementioned weighting based on similarity ranking, for images of interest to the user, those ranking higher in the user's interest ranking receive a larger fifth weight; conversely, for images of disinterest, those ranking lower in the user's interest ranking receive a larger sixth weight. In practice, to facilitate user sorting based on their interest level, disinterested images can be sorted according to their degree of disinterest. That is, higher-ranked disinterested images represent less user interest, and therefore receive a larger sixth weight.

[0092] The weights determined by similarity ranking and the weights determined by user intent can be multiplied to determine the final first and second weights. For example, when assigning a third weight to each intent image based on the ranked second similarity, a certain intent image might be assigned a third weight of 1.5. When assigning a fifth weight to intent images based on user intent ranking, the weight assigned to that intent image might be 1.2. Therefore, the first weight of that intent image would be 1.5 × 1.2 = 1.8, and the feature vector corresponding to that intent image would also have a weight of 1.8 during fusion.

[0093] In one embodiment, the user's operation on the set of similar images further includes: selecting an intentional region in the intentional image, and selecting a non-intentional region in the non-intentional image;

[0094] Extracting the third feature vector of each intended image and extracting the fourth feature vector of each non-intended image includes the following steps:

[0095] Determine the first feature map corresponding to each intended image, and determine the second feature map corresponding to each non-intended image;

[0096] If the intention image has an intention region, then the first feature region corresponding to the intention region is determined in the first feature map, and weights are assigned to the first feature region and the regions in the first feature map other than the first feature region to obtain the first assignment result.

[0097] If there are unintended regions in the unintended image, the second feature region corresponding to the unintended region is determined in the second feature map, and weights are assigned to the second feature region and the regions in the second feature map other than the second feature region to obtain the second assignment result.

[0098] Based on the first allocation result, a third feature vector is generated using the first feature map;

[0099] Based on the second allocation result, the fourth feature vector is generated using the second feature map.

[0100] Users can select regions of interest within the intended image as intended regions, and regions of no interest within the unintended image as unintended regions. Feature vectors corresponding to the images are generated from feature maps. During the feature vector generation process, weights can be assigned to different regions in the feature map based on the user's selected intended and unintended regions. This results in a third feature vector that is closer to the user's intention, while a fourth feature vector is further away from it. The intention pattern calculated between the third and fourth feature vectors will also be closer to the user's intention.

[0101] For example, if a user inputs an image of a car from brand A, and the similar image set returns an image containing a car from brand B, the user can identify the image containing brand B as a non-intended image and then select a non-intended area, such as selecting the logo of brand B. This indicates that the user is not interested in brand B. However, if the similar image set returns an image containing brand A, and the user identifies the image containing brand A as an intended image and then selects an intended area, such as selecting the logo of brand A or the distinctive headlights of brand A, this indicates that the user is interested in brand A, or even a specific part of brand A.

[0102] In one specific implementation, when generating the third feature vector, Gaussian weights can be generated in the feature map corresponding to the intended image, centered on the feature region corresponding to the intended region. The mean and variance are pre-set parameters. Using Gaussian weights prevents small changes in position from causing large alterations to the feature vector, and assigns smaller weights to points far from the feature region (i.e., regions outside the intended region) to obtain a third feature vector that more closely resembles the user's intention. Similarly, weights can be assigned to non-intended regions and other regions in the non-intended image in the same way to obtain a fourth feature vector that is further removed from the user's intention.

[0103] To provide a better understanding of this embodiment, the following example will be used for further explanation.

[0104] refer to Figure 2 A schematic diagram of a user intent recognition process based on human-computer interaction.

[0105] A large number of images can be collected and stored in a database in advance to establish a preset image library 1. For the images in the preset image library, a feature extractor is used to extract features a to obtain feature vectors. The obtained feature vectors and the corresponding images are then stored together in the feature vector set 2.

[0106] The user inputs one or more target images 3. After acquiring the target images, feature extraction a can be performed through a feature extractor to obtain the feature images 4 corresponding to the target images. For example, the feature extractor can be an algorithm model such as ResNet (Residual Neural Network), VGG (a network structure proposed by the Visual Geometry Group), RegNet, VIT (Vision Transformer), and SWIN.

[0107] Pooling operation b can be performed on each feature image 4 to obtain the corresponding feature vector 5. Feature fusion c can be performed on the feature vectors 5 corresponding to each target image to obtain the fused feature vector 6. Feature vector 6 is used as the current intention mode. After determining feature vector 6, it can be searched in feature vector set 2, that is, feature vector 6 is calculated with the feature vectors in feature vector set 2 respectively to determine the distance between feature vector 6 and the feature vectors in feature vector set 2, thereby determining the similarity between the intention mode and the feature vectors in the preset image library 1. Images in the preset image library 1 that rank high in similarity or whose similarity exceeds a specified threshold are determined as the similar image set 7.

[0108] If the user is not satisfied with the currently displayed set of similar images 7, the user can initiate an operation to select an operation image 9 from the set of similar images 7. The operation image 9 selected from the set of similar images can be an intended image or a non-intended image. In addition, the user can select an intended region 10 from the intended image and a non-intended region 10 from the non-intended image. In response to the user's operation, weights are assigned to the feature maps of each operation image during pooling operations to obtain the feature vectors of each operation image. When fusing the feature vectors, weights can also be assigned according to the user's operation to obtain an updated intention pattern that is closer to the user's intention.

[0109] If the user does not initiate any action within the specified time range, it can be assumed that the user is satisfied with the currently displayed set of similar images. That is, the intention pattern corresponding to the currently displayed set of similar images is determined as the final user intention pattern 8, and the currently displayed set of similar images is the target set of similar images. The user can obtain images that are consistent with their search purpose.

[0110] This invention discloses an image recognition method. It acquires one or more target images for retrieval, determines a user's intention pattern based on these images (the intention pattern indicates the user's points of interest), identifies a set of target-similar images corresponding to the intention pattern, and displays the set of target-similar images. It utilizes general features to achieve fine-grained retrieval in any scene, obtains the user's interests by determining the user's intention pattern, and provides the user with a set of target-similar images according to the user's intention pattern. The provided set of target-similar images accurately matches the user's intention.

[0111] Example 2

[0112] Figure 3 This is a schematic diagram of the structure of an image recognition device provided in Embodiment 2 of the present invention, as shown below. Figure 3 As shown, the device includes the following modules:

[0113] The target image acquisition module 310 is used to acquire one or more target images for retrieval;

[0114] The intention pattern determination module 320 is used to determine the user's intention pattern based on one or more target images, wherein the intention pattern is used to indicate the user's point of interest information;

[0115] The target similar image set determination module 330 is used to determine the target similar image set corresponding to the intention pattern and display the target similar image set.

[0116] In one embodiment, the target similar image set determination module 330 includes the following sub-modules:

[0117] The similar image set submodule is used to determine the set of similar images corresponding to the intention pattern and to display the set of similar images.

[0118] The first detection submodule is used to take the similar image set as the target similar image set if no operation by the user on the similar image set is detected within a set time period.

[0119] The second detection submodule is used to determine the user's operation image in response to the operation when the user's operation on the set of similar images is detected within a set time period.

[0120] An update submodule is used to update the intention mode based on the operation image and the operation;

[0121] The target similar image set determination submodule is used to determine the set of similar images corresponding to the updated intention pattern until no user operation is detected. Then, the set of similar images corresponding to the intention pattern obtained in the last update is used as the target similar image set and the target similar image set is displayed.

[0122] In one embodiment, the intention pattern determination module 320 includes the following sub-modules:

[0123] The first feature vector extraction submodule is used to extract the first feature vector of one or more target images;

[0124] The first intention pattern determination submodule is used to use the first feature vector as the user's intention pattern when the target image is a single image.

[0125] The second intention mode determination submodule is used to fuse all the first feature vectors when there are multiple target images, and use the fused first feature vector as the intention mode.

[0126] In one embodiment, the similar image set determination submodule includes the following units:

[0127] The second feature vector acquisition unit is used to acquire the second feature vector of all images in the preset image library;

[0128] The first similarity determination unit is used to determine the first similarity between the intention pattern and the second feature vector respectively;

[0129] The similar image set determination unit is used to determine all images in the preset image library whose first similarity meets the preset conditions as a similar image set corresponding to the intention pattern.

[0130] In one embodiment, the operation includes: selecting an intended image from the set of similar images, and selecting a non-intended image from the set of similar images; the user's operation images include: multiple intended images, and multiple non-intended images;

[0131] The update submodule includes the following units:

[0132] An extraction unit is configured to extract the third feature vector of each of the intended images and to extract the fourth feature vector of each of the non-intended images.

[0133] The weight determination unit is configured to determine a first weight for each of the intended images based on the first similarity, and to determine a second weight for each of the non-intended images;

[0134] The first fusion unit is used to fuse all the third feature vectors based on the first weight to obtain a second fused feature vector;

[0135] The second fusion unit is used to fuse all the fourth feature vectors based on the second weight to obtain a third fused feature vector;

[0136] The calculation unit is used to perform a subtraction calculation between the second fused feature vector and the third fused feature vector, and use the difference as the intention pattern to complete the update of the intention pattern.

[0137] In one embodiment, the weight determination unit includes the following sub-units:

[0138] The second similarity ranking subunit is used to determine the second similarity between the intention pattern and the second feature vector corresponding to each intention image based on the first similarity, and to rank the second similarity.

[0139] The third similarity ranking subunit is used to determine the third similarity between the intentional pattern and the second feature vector corresponding to each of the non-intentional images based on the first similarity, and to rank the third similarity.

[0140] The third weight allocation subunit is used to allocate a third weight to each of the intention images according to the sorted second similarity, as the first weight of the intention image;

[0141] The fourth weight allocation subunit is used to sequentially allocate the fourth weight to each of the non-intended images according to the sorted third similarity, as the second weight of the non-intended images.

[0142] In one embodiment, the operation further includes: the user selecting an image according to the degree of intent; the user's selected image further includes: the intended image sorted by the degree of intent and the unintended image;

[0143] The weight determination unit further includes the following sub-units:

[0144] The fifth weight allocation subunit is used to allocate a fifth weight to the intention image based on the result of the user sorting by degree of intention, wherein the intention image that ranks higher in the user's degree of intention has a larger fifth weight.

[0145] The sixth weight allocation subunit is used to allocate a sixth weight to the non-intended images based on the results of the users sorting by degree of intent, wherein the non-intended images ranked later in the user intent ranking have a larger sixth weight.

[0146] The first weight determination subunit is used to multiply the third weight by the fifth weight to obtain a first result, and to determine the first result as the first weight;

[0147] The second weight determination subunit is used to multiply the fourth weight by the sixth weight to obtain a second result, and to determine the second result as the second weight.

[0148] In one embodiment, the operation further includes: selecting an intentional region in the intentional image, and selecting a non-intentional region in the non-intentional image;

[0149] The extraction unit includes the following sub-units:

[0150] The feature map determination subunit is used to determine a first feature map corresponding to each of the intended images, and to determine a second feature map corresponding to each of the non-intended images;

[0151] The first allocation result determination subunit is used to determine the first feature region corresponding to the intention region in the first feature map if the intention image has the intention region, and to perform weight allocation on the first feature region and the regions in the first feature map other than the first feature region to obtain the first allocation result.

[0152] The second allocation result determination subunit is used to determine the second feature region corresponding to the non-intended region in the second feature map if the non-intended image has the non-intended region, and to perform weight allocation on the second feature region and the regions in the second feature map other than the second feature region to obtain the second allocation result.

[0153] The third feature vector determination subunit is used to generate the third feature vector based on the first allocation result and using the first feature map;

[0154] The fourth feature vector determination subunit is used to generate the fourth feature vector based on the second allocation result and using the second feature map.

[0155] The image recognition device provided in this embodiment of the invention can realize the image recognition method provided in Embodiment 1 of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0156] Example 3

[0157] Figure 4A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0158] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0159] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0160] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as an image recognition method.

[0161] In some embodiments, an image recognition method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of an image recognition method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform an image recognition method by any other suitable means (e.g., by means of firmware).

[0162] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0163] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0164] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0165] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0166] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0167] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0168] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0169] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. An image recognition method, characterized in that, The method includes: Obtain one or more target images for retrieval; Based on one or more target images, determine the user's intention pattern, which is used to indicate the user's point of interest information; Determine and display the set of target similar images corresponding to the intention pattern, including: Determine a set of similar images corresponding to the intention pattern, and display the set of similar images; If no user operation on the similar image set is detected within a set time period, then the similar image set is taken as the target similar image set; If an operation by the user on the set of similar images is detected within a set time period, then in response to the operation, the image of the operation by the user is determined; The intention mode is updated based on the operation image and the operation; Determine a set of similar images corresponding to the updated intention pattern, and continue this process until no further user actions are detected. Then, use the set of similar images corresponding to the intention pattern obtained in the last update as the target set of similar images and display the target set of similar images. The operation includes: selecting an intended image from the set of similar images, and selecting a non-intended image from the set of similar images; the user's operation images include: multiple intended images, and multiple non-intended images; The step of updating the intention pattern based on the operation image and the operation includes: Extract the third feature vector of each of the intended images, and extract the fourth feature vector of each of the non-intended images; Based on the first similarity, a first weight is determined for each of the intended images, and a second weight is determined for each of the non-intended images; the first similarity is the similarity between the intended pattern and the second feature vector of all images in the preset image library; Based on the first weight, all the third feature vectors are fused to obtain a second fused feature vector; Based on the second weight, all the fourth feature vectors are fused to obtain a third fused feature vector; The second fused feature vector and the third fused feature vector are subtracted, and the difference is used as the intention pattern to complete the update of the intention pattern.

2. The method according to claim 1, characterized in that, Determining the user's intention pattern based on one or more target images includes: Extract the first feature vector of one or more of the target images; When the target image is a single image, the first feature vector is used as the user's intention pattern. When there are multiple target images, all the first feature vectors are fused together, and the fused first feature vector is used as the intention pattern.

3. The method according to claim 1, characterized in that, Determining the set of similar images corresponding to the intention pattern includes: Obtain the second feature vector of all images in the preset image library; Determine the first similarity between the intention pattern and the second feature vector; All images in the preset image library whose first similarity meets the preset conditions are identified as a set of similar images corresponding to the intention pattern.

4. The method according to claim 1, characterized in that, The determination of a first weight for each intended image based on the first similarity, and the determination of a second weight for each non-intended image, include: Based on the first similarity, the second similarity between the intention pattern and the second feature vector corresponding to each intention image is determined, and the second similarity is sorted. Based on the first similarity, the third similarity between the intentional pattern and the second feature vector corresponding to each of the non-intentional images is determined, and the third similarity is sorted. Based on the sorted second similarity, a third weight is sequentially assigned to each of the intended images, serving as the first weight of the intended image; Based on the sorted third similarity, a fourth weight is sequentially assigned to each of the non-intended images, serving as the second weight for the non-intended images.

5. The method according to claim 4, characterized in that, The operation further includes: the user selecting images according to their level of intent; the user's selected images also include: the intended images sorted by the user's level of intent and the unintended images; The determination of a first weight for each of the intended images and a second weight for each of the non-intended images further includes: Based on the results of the users sorting by degree of intent, a fifth weight is assigned to the intent images, wherein the higher the intent image ranks in the user's degree of intent, the greater the fifth weight it occupies. Based on the results of the users sorting by their degree of intent, a sixth weight is assigned to the non-intent images, wherein the non-intent images ranked later in the user's degree of intent have a larger sixth weight. The third weight is multiplied by the fifth weight to obtain a first result, and the first result is determined as the first weight; The fourth weight is multiplied by the sixth weight to obtain a second result, and the second result is determined as the second weight.

6. The method according to any one of claims 1-5, characterized in that, The operation further includes: selecting an intended region in the intended image, and selecting a non-intended region in the non-intended image; The extraction of the third feature vector of each of the intended images and the extraction of the fourth feature vector of each of the non-intended images include: Determine a first feature map corresponding to each of the intended images, and determine a second feature map corresponding to each of the non-intended images; If the intention image contains the intention region, then the first feature region corresponding to the intention region is determined in the first feature map, and weights are assigned to the first feature region and the regions in the first feature map other than the first feature region to obtain a first assignment result. If the unintended image contains the unintended region, then the second feature region corresponding to the unintended region is determined in the second feature map, and weights are assigned to the second feature region and the regions in the second feature map other than the second feature region to obtain a second assignment result. Based on the first allocation result, the third feature vector is generated using the first feature map; Based on the second allocation result, the fourth feature vector is generated using the second feature map.

7. An image recognition device, characterized in that, The device includes: The target image acquisition module is used to acquire one or more target images for retrieval. The intention pattern determination module is used to determine the user's intention pattern based on one or more target images, wherein the intention pattern is used to indicate the user's point of interest information; A target similar image set determination module is used to determine the target similar image set corresponding to the intention pattern and display the target similar image set; The target similar image set determination module includes: The similar image set submodule is used to determine the set of similar images corresponding to the intention pattern and to display the set of similar images; The first detection submodule is used to take the similar image set as the target similar image set if no operation of the user on the similar image set is detected within a set time period. The second detection submodule is used to determine the user's operation image in response to the operation when the user's operation on the set of similar images is detected within a set time period. An update submodule is used to update the intention mode based on the operation image and the operation; The target similar image set determination submodule is used to determine the set of similar images corresponding to the updated intention pattern until no user operation is detected. Then, the set of similar images corresponding to the intention pattern obtained in the last update is used as the target similar image set and the target similar image set is displayed. The operation includes: selecting an intended image from the set of similar images, and selecting a non-intended image from the set of similar images; the user's operation images include: multiple intended images, and multiple non-intended images; The update submodule includes the following units: An extraction unit is configured to extract the third feature vector of each of the intended images and to extract the fourth feature vector of each of the non-intended images. The weight determination unit is used to determine a first weight for each of the intended images based on a first similarity, and to determine a second weight for each of the non-intended images; the first similarity is the similarity between the intended pattern and the second feature vector of all images in the preset image library; The first fusion unit is used to fuse all the third feature vectors based on the first weight to obtain a second fused feature vector; The second fusion unit is used to fuse all the fourth feature vectors based on the second weight to obtain a third fused feature vector; The calculation unit is used to perform a subtraction calculation between the second fused feature vector and the third fused feature vector, and use the difference as the intention pattern to complete the update of the intention pattern.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform an image recognition method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute an image recognition method according to any one of claims 1-6.

Images