Image retrieval method, device, equipment and storage medium

Abstract

The present disclosure provides an image retrieval method, device, equipment and storage medium, which relate to the field of artificial intelligence, and specifically to technical fields such as deep learning, computer vision, intelligent search, and image recognition. The image retrieval method includes: obtaining a retrieval request, which carries retrieval requirement information; obtaining attribute features of the retrieval requirement information; performing image retrieval in multiple images according to the attribute features and the image features of the image, and obtaining a target image that meets the retrieval requirement information, wherein the image features include global features, local features, object features containing objects, and relationship features between objects. Thus, the accuracy of image retrieval is improved by combining multiple features of the image.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, specifically to technical fields such as deep learning, computer vision, intelligent search, and image recognition, and in particular to an image retrieval method, device, equipment, and storage medium, which can be used in image search scenarios such as searching for images by text and searching for images by images. Background Art

[0002] With the development of technologies such as deep learning and computer vision, image search has been widely used in multiple business fields, such as navigation and e-commerce.

[0003] In the related art, a deep learning network is used to extract global features and / or local features of an image, and image search is performed based on the global features and / or local features of the image.

[0004] However, the accuracy of the above retrieval scheme is low. Summary of the invention

[0005] The present invention provides an image retrieval method, apparatus, device and storage medium for improving the accuracy of image retrieval.

[0006] According to a first aspect of the present disclosure, there is provided an image retrieval method, comprising: obtaining a retrieval request, wherein the retrieval request carries retrieval requirement information; obtaining attribute features of the retrieval requirement information; performing image retrieval in multiple images based on the attribute features and image features of the image, to obtain a target image that meets the retrieval requirement information, wherein the image features include global features, local features, object features of included objects, and relationship features between objects.

[0007] According to a second aspect of the present disclosure, an image retrieval device is provided, comprising: a request acquisition unit, used to acquire a retrieval request, wherein the retrieval request carries retrieval requirement information; an attribute feature acquisition unit, used to acquire attribute features of the retrieval requirement information; and an image retrieval unit, used to perform image retrieval in multiple images based on the attribute features and image features of the images, to obtain a target image that meets the retrieval requirement information, wherein the image features include global features, local features, object features of included objects, and relationship features between objects.

[0008] According to a third aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the image retrieval method described in the first aspect.

[0009] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the image retrieval method described in the first aspect.

[0010] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising: a computer program, wherein the computer program is stored in a readable storage medium, and at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program so that the electronic device executes the image retrieval method described in the first aspect.

[0011] According to the technical solution provided by the present disclosure, based on the attribute features of the retrieval requirement information in the retrieval request and the image features of the image, image retrieval is performed in multiple images to obtain a target image that meets the retrieval requirement information, wherein the image features include global features, local features, object features of the contained objects, and relationship features between objects. These features can reflect the local and details of the image, and can also reflect the characteristics of the objects contained in the image and the relationship characteristics between the objects contained in the image. Image retrieval combined with these features can effectively improve the accuracy of image retrieval.

[0012] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0014] Figure 1 is a schematic diagram of an application scenario applicable to the embodiment of the present disclosure;

[0015] Figure 2 is a schematic diagram according to a first embodiment of the present disclosure;

[0016] Figure 3 is a schematic diagram according to a second embodiment of the present disclosure;

[0017] Figure 4 is a schematic diagram according to a third embodiment of the present disclosure;

[0018] Figure 5 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0019] Figure 6 is an example diagram of an image feature extraction process provided according to the above-mentioned embodiment of the present disclosure;

[0020] Figure 7is a schematic diagram according to a fifth embodiment of the present disclosure;

[0021] Figure 8 FIG. 8 is a schematic block diagram of an example electronic device 800 that may be used to implement embodiments of the present disclosure. DETAILED DESCRIPTION

[0022] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0023] In image retrieval, image retrieval is performed based on the global features and / or local features of the image. However, global features and local features cannot fully reflect the information contained in the image, especially the objects contained in the image and the relationship between objects, resulting in low image retrieval accuracy.

[0024] Therefore, the present disclosure proposes an image retrieval method, which is applied to the technical fields of deep learning, computer vision, intelligent search, image recognition, etc. in the field of artificial intelligence. In this method, in addition to global features and local features, the image features of the image also include object features of the objects contained in the image and relationship features between objects. Combining multiple features of the image makes the image features richer. Through richer image features, the accuracy of image retrieval is effectively improved, especially the accuracy of object retrieval and object relationship retrieval in image retrieval can be improved.

[0025] Figure 1 1 is a schematic diagram of an application scenario applicable to the present disclosure. The application scenario includes a terminal 101 and a server 102. The terminal 101 can send a search request to the server 102, and the server 102 performs image search in response to the search request. The server 102 can perform image search based on the image features of the images in the database, or can perform feature extraction on the images in the database to obtain the image features of the images, and perform image search based on the image features. The server 102 can perform image search based on the global features, local features, object features of the included objects, and relationship retrieval features between objects.

[0026] For example, the above application scenarios may be voice search, text search, image search, etc. In voice search, the present disclosure may be used to retrieve images that match the input voice; in text search, the present disclosure may be used to retrieve images that match the input text; in image search, the present disclosure may be used to retrieve images that are similar to the input image.

[0027] For another example, the above-mentioned application scenarios can be scene retrieval, object retrieval, object relationship retrieval, attribute retrieval and other scenarios in image retrieval, among which scene retrieval, for example, retrieves images of night scenes and highway scenes; object retrieval, for example, retrieves images containing vehicles and animals; object relationship retrieval, for example, retrieves images of multiple people quarreling and multiple people singing; attribute retrieval, for example, image color retrieval and shape retrieval (such as retrieving images with a red background color and images with cubes).

[0028] It should be noted that the embodiments of the present disclosure may be implemented on a terminal and / or a server. Figure 1 Take the implementation of the embodiment of the present disclosure on a server as an example. The terminal may be a personal digital assistant (PDA), a handheld device with wireless communication function (such as a smart phone, a tablet), a computing device (such as a personal computer (PC)), a wearable device (such as a smart watch, a smart bracelet), and a smart home device (such as a smart speaker, a smart display device). The server may be an independent server or a server cluster, and may be a local server or a cloud server.

[0029] It should be noted that the image in this embodiment is not an image of a specific user and cannot reflect the personal information of a specific user.

[0030] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0031] The following specific embodiments are used to describe in detail the technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0032] Figure 2 Schematic diagram of the first embodiment of the present disclosure. Figure 2 As shown, the image retrieval method provided by the first embodiment of the present disclosure includes:

[0033] S201, obtaining a search request, the search request carrying search requirement information.

[0034] The search request is used to request to search for images that meet the search requirement information, and the search requirement information describes the search requirement (or search condition) for the image.

[0035] In this embodiment, a search demand may be received from a search initiator, where the search initiator may be a user, a terminal where the user is located, a software robot in robotic process automation, or the like.

[0036] S202, obtaining attribute characteristics of retrieval requirement information.

[0037] In this embodiment, the attribute features may be acquired or extracted from the search requirement information.

[0038] S203, performing image retrieval in multiple images according to the attribute features and the image features of the image, and obtaining a target image that meets the retrieval requirement information, wherein the image features include global features, local features, object features of included objects, and relationship features between objects.

[0039] The database may store multiple images, and in the process of image retrieval, feature extraction may be performed on the multiple images respectively to obtain image features corresponding to the multiple images. Alternatively, the database may store multiple images and image features corresponding to the multiple images respectively, and feature extraction may be performed on the multiple images in advance to obtain image features corresponding to the multiple images respectively and store them in the database.

[0040] Among them, global features reflect the characteristics of the entire image, local features reflect the characteristics of the local area of ​​the image, one image can correspond to multiple local features, object features reflect the characteristics of the objects contained in the image, and the number of object features is related to the number of objects contained in the image; relationship features reflect the characteristics of the relationship between objects contained in the image, and the number of relationship features is related to the number of relationships between objects.

[0041] In this embodiment, the attribute features can be feature matched with the image features of the image, or, after processing the image features (including global features, local features, object features containing objects, and relationship retrieval features between objects), the attribute features can be feature matched with the image features of the image, and an image whose matching degree between the image features and the attribute features meets a condition (for example, the matching degree is greater than or equal to a degree threshold, or the matching degrees between the image features and the attribute features in the multiple images are arranged in the top K positions in descending order, where K is a preset value) is selected from the multiple images, and the selected image is determined to be the target image that meets the retrieval requirement information.

[0042] In the disclosed embodiment, the image features include global features, local features, object features including objects, and relational features between objects, which improve the richness and comprehensiveness of the image features. Image retrieval based on these image features can improve the accuracy of image retrieval. In particular, the accuracy of object retrieval in image retrieval can be improved by using object features, and the accuracy of object relationship retrieval in image retrieval can be improved by using relational features between objects.

[0043] Below, some embodiments of the retrieval requirement information and image retrieval process are provided.

[0044] In some embodiments, the retrieval requirement information may include retrieving images, and the attribute characteristics of the retrieval requirement information may include image retrieval features of the retrieval image. The image retrieval features include global retrieval features, local retrieval features, object retrieval features containing objects, and relationship retrieval features between objects, so as to improve the accuracy of image retrieval in the image search scenario through rich image features.

[0045] The retrieval image refers to the image input as the key information for retrieval. In the image search scenario, the retrieval image reflects the retrieval requirement of retrieving images similar to the input image. For example, if a red vehicle appears in the retrieval image, then it is required to search for images containing the red vehicle; for another example, if a person stands next to a red vehicle in the retrieval image, then it is required to search for images containing the object relationship of a person standing next to a red vehicle. The global retrieval features, local retrieval features, object retrieval features containing objects, and relationship retrieval features between objects of the retrieval image, that is, the global features, local features, object features containing objects, and relationship features between objects of the retrieval image.

[0046] In this embodiment, feature extraction can be performed on the search image to obtain image search features of the search image; by matching the image search features of the search image with the image features of the image, image search is performed in multiple images to obtain a target image that meets the search requirements reflected by the search image.

[0047] In some embodiments, the search requirement information may include search text, and the attribute information of the search requirement information may include text features of the search text, so as to improve the accuracy of image retrieval in the text-to-image search scenario through rich image features.

[0048] The search text refers to the text input as the key information for the search, which can be a search keyword or a search key. In the text-to-image search scenario, the search text describes the specific search requirements. For example, the search text is "search for images of vehicles under street lights" or "search for images of pedestrians walking together at night", which describes the scene, objects, and relationships between the objects required for the search.

[0049] In this embodiment, feature extraction can be performed on the search text to obtain text features of the search text. The feature extraction process of the search text is not specifically limited here. By matching the text features of the search image with the image features of the image, image search is performed in multiple images to obtain a target image that meets the search requirements reflected by the search text.

[0050] In addition, the search requirement information may include search images and search texts, and the attribute information of the search requirement information may include image search features of search images and text features of search texts, so as to achieve the accuracy of image search by combining images and texts through rich image features. The search information may also include search voice, search video, etc., to achieve voice search and video search, which are not described here one by one.

[0051] In some embodiments, the retrieval requirement information may include the retrieval type of the retrieval request, and the retrieval type may include one or more of the following: scene retrieval, object retrieval, and relationship retrieval between objects. Based on this, according to the attribute features of the retrieval requirement information and the image features of the image, image retrieval is performed in multiple images to obtain a target image that meets the retrieval requirement information, which may include: selecting a target feature related to the retrieval type from the image features; matching the attribute features of the retrieval requirement information with the target features, performing image retrieval in multiple images, and obtaining a target image that meets the retrieval requirement information. Thus, it is possible to adapt to the retrieval type of the retrieval request, select appropriate image features for image retrieval in a targeted manner, and improve the accuracy of image retrieval.

[0052] The scene search, object search, and object relationship search may refer to the above embodiments and will not be described in detail. Taking the user triggering the search request as an example, the user may enter the search type of the search request in the selection box or input box to perform image search under a specific type.

[0053] Among them, the association between the search type and the image features can be configured in advance. For example, the scene involves the overall background and local elements of the image, and may also involve objects and the relationship between objects. Scene search can associate global features, local features, object features containing objects, and relationship features between objects; object search can associate object features containing objects; and relationship search between objects can associate relationship features between objects. The association relationship can be configured by professionals based on experience and experiments.

[0054] In this embodiment, according to the association between the retrieval type and the image features, the target features associated with the retrieval type can be selected from the image features. For example, when the retrieval type is scene retrieval, global features, local features, object features containing objects, and relationship features between objects are selected from the image features. When the retrieval type is object retrieval, object features are selected from the image features. When the retrieval type is relationship retrieval between objects, relationship features between objects are selected from the image features. The attribute features of the retrieval requirement information are matched with the target features (i.e., the attribute features are matched with the target features corresponding to the multiple images respectively), and the degree of feature matching between the retrieval requirement information and the multiple images can be obtained. For example, when the target feature is an object feature containing an object, for each of the multiple images, the attribute features of the retrieval requirement information are matched with the object features of the object contained in the image, and the degree of feature matching between the retrieval requirement information and the multiple images can be obtained. Afterwards, a target image that meets the retrieval requirement information can be screened out from multiple images based on the degree of feature matching between the retrieval requirement information and the multiple images respectively. For example, the target image can be screened out from multiple images in order of the feature matching degree from high to low, or the target image can be determined as an image screened out from multiple images whose feature matching degree with the retrieval requirement information is greater than a degree threshold.

[0055] Furthermore, in the case where the retrieval requirement information includes a retrieval image and the attribute features of the retrieval requirement information include image retrieval features of the retrieval image, a target retrieval feature related to the retrieval type can be selected from the image retrieval features, and the target retrieval feature can be matched with the target feature. For example, when the retrieval type is a relationship retrieval between objects, the relationship retrieval feature between objects in the image retrieval features can be matched with the relationship features between objects corresponding to multiple images, and the degree of feature matching between the retrieval image and the multiple images can be obtained; based on the degree of feature matching between the retrieval image and the multiple images, the target image can be screened out from the multiple images. Thus, in the image retrieval scenario of searching for images by images, appropriate image features can be selected from the image features of the input retrieval image and the image features of multiple images in the database for image retrieval, thereby improving the accuracy of image retrieval.

[0056] In addition to selecting appropriate image features for retrieval based on the retrieval type, image retrieval can also be performed in multiple images according to the attribute features of the retrieval requirement information and the image features of the image to obtain the target image that meets the retrieval requirement information. This can also be achieved in other ways.

[0057] In some embodiments, according to the attribute features of the retrieval requirement information and the image features of the image, performing image retrieval in multiple images to obtain a target image that meets the retrieval requirement information may include: matching the attribute features of the retrieval requirement information with the global features, local features, object features containing objects, and relationship features between objects in the image features to obtain multiple matching degrees; weighting the multiple matching degrees based on the weights corresponding to the global features, local features, object features containing objects, and relationship features between objects to obtain the feature matching degrees of the retrieval requirement information and the multiple images respectively; based on the feature matching degrees of the retrieval requirement information and the multiple images respectively, screening the target image that meets the retrieval requirement information from the multiple images, for example, screening the target image from the multiple images according to the feature matching degrees from high to low, or screening the target image whose feature matching degree with the retrieval requirement information is greater than a degree threshold from the multiple images. In this way, by weighting the matching degrees, the rationality and flexibility of combining the global features, local features, object features containing objects, and relationship features between objects for image retrieval are improved, and the accuracy of image retrieval is improved.

[0058] In addition, the global features, local features, object features of included objects, and relationship features between objects may be weighted, or these features may be fused in other ways, which are not described one by one.

[0059] Some embodiments of the image feature extraction process are provided below. These embodiments can be used to extract features from any image in a plurality of images, and can also be used to extract features from a search image.

[0060] In some embodiments, image features are obtained by extracting features using a masked attention mechanism. The masked attention mechanism can shield unimportant information and retain important information through a mask matrix, focusing on important information. Since image features involve multiple features such as global features, local features, object features including objects, and relationship features between objects, the masked attention mechanism can shield information in the image that is not related to these features and retain relevant information, effectively improving the accuracy and efficiency of extracting these multiple features.

[0061] Figure 3 is a schematic diagram according to the second embodiment of the present disclosure. Figure 3 As shown, the image feature extraction process provided by the second embodiment of the present disclosure includes:

[0062] S301, extracting global features of the image to obtain a first feature.

[0063] S302, extracting local features from the image to obtain a second feature.

[0064] In this embodiment, the global feature extraction and local feature extraction of the image can be performed through the feature extraction network to obtain the initial global feature of the image and the initial local feature of the image. In order to distinguish them from the global features and local features used for image retrieval, the initial global feature of the image is called the first feature, and the initial local feature of the image is called the second feature. In the process of extracting local features, the image can be divided into multiple image blocks (i.e., multiple image regions), and feature extraction is performed on the multiple image blocks respectively to obtain the second features corresponding to the multiple image blocks respectively.

[0065] S303, initializing features of objects included in the image to obtain a third feature.

[0066] S304, initializing the features of the relationship between the objects included in the image to obtain a fourth feature.

[0067] The objects can be all objects appearing in the image, or one or more specified objects. For images in the database, in order to improve the richness of objects that can be retrieved, the features of all objects contained in the image can be initialized, and the relationships between all objects contained in the image can be initialized; for retrieval images, in order to improve retrieval efficiency, the features of the specified objects contained in the retrieval images can be initialized, and the relationships between the specified objects contained in the retrieval images can be initialized. Specified objects include vehicles, pedestrians, etc.

[0068] The number of the third features is related to the number of objects contained in the image. The more objects contained in the image, the more the third features are. The number of the fourth features is related to the number of relationships between objects contained in the image. The more relationships between objects, the more the fourth features are. The number of relationships between objects can be the number of combinations of two objects.

[0069] Optionally, the number of the third features is equal to the number of objects contained in the image, and the number of the fourth features is equal to the number of relationships between objects contained in the image. The number of objects contained in the image can be obtained by performing target detection on the image. Based on the number of objects contained in the image, the objects contained in the image can be combined in pairs to obtain the number of relationships between objects contained in the image; or, based on the positions of the objects contained in the image, the objects contained in the image can be combined in pairs to obtain the number of relationships between objects contained in the image.

[0070] In this embodiment, according to the configured feature initialization method, the features of the objects contained in the image can be initialized to obtain the initial object features of the contained objects, and the features of the relationships between the objects contained in the image can be initialized to obtain the initial relationship features between the objects. In order to distinguish the object features of the contained objects and the relationship features between the objects used for image retrieval, the initial object features of the contained objects are called the third features, and the initial relationship features between the objects are called the fourth features.

[0071] In a possible implementation, the third feature and the fourth feature can be initialized as the first feature, that is, the objects contained in the image and the relationship between the objects are initialized according to the initial global feature to obtain the third feature and the fourth feature. This method takes into account that the objects and the relationship between the objects may not be limited to a single image area for an image, and uses the global feature to initialize the object features of the objects and the relationship features between the objects, ensuring that the initialized features cover the object features of the objects contained in the global scope and the relationship features between the objects, thereby improving the rationality of feature initialization.

[0072] In another possible implementation, the third feature and the fourth feature may be randomly initialized or initialized to a fixed feature value, or the third feature and the fourth feature may be initialized in combination with a plurality of second features. Compared with initializing the third feature and the fourth feature to the first feature respectively, the accuracy and rationality of feature initialization in these initialization methods are poor.

[0073] S305, based on the attention mask matrix corresponding to the image, perform one or more rounds of feature processing on the first feature, the second feature, the third feature and the fourth feature to obtain global features, local features, target features and relationship features.

[0074] In this embodiment, the attention mask matrix can be used to shield part of the information in the first feature, the second feature, the third feature and the fourth feature, and retain the other part of the information. In particular, after the feature is initialized, the third feature may contain information irrelevant to the object, and the fourth feature may contain information irrelevant to the relationship between objects. The attention mask matrix is ​​used to shield these irrelevant information to obtain the first feature after the first round of processing, the second feature after the first round of processing, the third feature after the first round of processing, and the fourth feature after the first round of processing. It can be determined that the global feature, the local feature, the target feature and the relationship feature are the second feature after the first round of processing, the third feature after the first round of processing, and the fourth feature after the first round of processing, respectively; or, the second feature after the first round of processing, the third feature after the first round of processing, and the fourth feature after the first round of processing can be processed in the next round. After multiple rounds of processing, the global feature, the local feature, the target feature and the relationship feature are obtained.

[0075] In the disclosed embodiments, in the process of image feature extraction, the initial global features and initial local features of the image are extracted, the features of the objects contained in the image are initialized to obtain initial object features, the features of the relationships between the objects contained in the image are initialized to obtain initial relationship features, and the initial global features, initial local features, initial object features and initial relationship features are processed for one or more rounds using a masked attention mechanism to improve the accuracy of feature extraction and obtain the final global features, local features, object features and relationship features.

[0076] Below, some embodiments are provided regarding the attention mask matrix.

[0077] In some embodiments, the attention mask matrix is ​​a preset mask matrix.

[0078] In some embodiments, the attention mask matrix is ​​determined based on the position distribution of objects contained in the image, so as to mask information in the third feature that is not related to the object and information in the fourth feature that is not related to the relationship between objects based on the position distribution of objects contained in the image.

[0079] In some embodiments, the attention mask matrix is ​​determined in the following manner: based on the position distribution of objects contained in the image, a first mask matrix corresponding to the object features of the contained objects and a second mask matrix corresponding to the relationship features between the objects are determined; based on the first mask matrix and the second mask matrix, the attention mask matrix is ​​determined. Among them, the position distribution of objects contained in the image reflects not only the image position of the object, but also the image position of the relationship between the objects. The first mask matrix determined based on the position distribution of objects contained in the image is used to shield information irrelevant to the object, and the second mask matrix determined based on the position distribution of objects contained in the image is used to shield information irrelevant to the relationship between the objects. Combining the first mask matrix and the second mask matrix to determine the attention mask matrix can improve the accuracy of one or more rounds of feature processing of the first feature, the second feature, the third feature and the fourth feature based on the attention mask matrix. Next, in combination Figure 4 This embodiment is further refined.

[0080] Figure 4 is a schematic diagram according to the third embodiment of the present disclosure. Figure 4 As shown, the image feature extraction process provided by the third embodiment of the present disclosure includes:

[0081] S401, extracting global features from the image to obtain a first feature.

[0082] S402, extracting local features of the image to obtain a second feature.

[0083] S403, initializing features of objects included in the image to obtain a third feature.

[0084] S404, initializing the features of the relationship between the objects included in the image to obtain a fourth feature.

[0085] The implementation principles and technical effects of S401 to S404 may refer to the aforementioned embodiments and will not be described in detail.

[0086] S405 , determining, according to the position distribution of the objects included in the image, a first mask matrix corresponding to the object features included in the image and a second mask matrix corresponding to the relationship features between the objects.

[0087] The first mask matrix and the second mask matrix may refer to the aforementioned embodiment and will not be described in detail.

[0088] In this embodiment, the objects contained in the image can be detected by the target detection algorithm to obtain the position distribution of the objects contained in the image. The first mask matrix can be determined according to the position distribution of the objects contained in the image; the second mask matrix can be determined according to the relationship between the objects reflected by the position distribution of the objects contained in the image. Thus, the first mask matrix can be used to shield information irrelevant to the object, and the second mask matrix can be used to shield information irrelevant to the relationship between the objects.

[0089] In a possible implementation, S405 includes: dividing the image into multiple image blocks; determining multiple element values ​​in the first mask matrix and multiple element values ​​in the second mask matrix according to the distribution of objects contained in the image in the multiple image blocks; wherein the multiple image blocks correspond one-to-one to the multiple element values ​​in the first mask matrix, the multiple image blocks correspond one-to-one to the multiple element values ​​in the second mask matrix, and the multiple image blocks are equivalent to multiple image regions.

[0090] In this implementation, the image can be divided into L*M image blocks, and the multiple image blocks correspond to the multiple element values ​​in the first mask matrix one by one, and the multiple image blocks correspond to the multiple element values ​​in the second mask matrix one by one, so the matrix dimension of the first mask matrix and the matrix dimension of the second mask matrix are both L*M. In the first mask matrix, each element corresponds to a different image block, and it can be determined based on the position distribution of the objects contained in the image whether the image block corresponding to the element in the first mask matrix contains the object, and the value of the element in the first mask matrix is ​​determined according to whether the image block corresponding to the element in the first mask matrix contains the object. In this way, the first mask matrix can be obtained. In the second mask matrix, each element corresponds to a different image block of the image, and it can be determined based on the position distribution of the objects contained in the image whether the image block corresponding to the element in the second mask matrix contains the relationship between the objects, and the value of the element in the second mask matrix is ​​determined according to whether the image block corresponding to the element in the second mask matrix contains the relationship between the objects. In this way, the second mask matrix can be obtained. In this way, the accuracy of the first mask matrix and the second mask matrix is ​​improved by dividing the image into blocks and distributing the positions of the objects contained in the image.

[0091] Optionally, L is equal to M. For example, L=M=7, and the image is divided into 7×7 image blocks.

[0092] Optionally, dividing the image into a plurality of image blocks may be performed before S402. S402 may include: after dividing the image into a plurality of image blocks, extracting features from each image block to obtain second features corresponding to each image block. Thus, the extraction of local features is implemented based on a plurality of image blocks, and the determination of the first mask matrix and the second mask matrix is ​​also implemented based on the plurality of image blocks. The entire scheme is associated through the plurality of image blocks to improve the accuracy of feature extraction.

[0093] In a possible implementation, determining multiple element values ​​in a first mask matrix and multiple element values ​​in a second mask matrix according to the distribution of objects contained in an image in multiple image blocks may include: for each image block in the multiple image blocks, if an object appears in the image block, determining that the element value corresponding to the image block in the first mask matrix is ​​1, otherwise, determining that the element value corresponding to the image block in the first mask matrix is ​​0; for each image block in the multiple image blocks, if multiple objects appear in the image block, determining that the element value corresponding to the image block in the second mask matrix is ​​1, otherwise, determining that the element value corresponding to the image block in the second mask matrix is ​​0.

[0094] In this implementation, if an object appears in the image block, the element value corresponding to the image block in the first mask matrix is ​​determined to be 1, so that the first mask matrix can retain the feature information of the image block containing the object; if no object appears in the image block, the element value corresponding to the image block in the first mask matrix is ​​determined to be 0, so that the first mask matrix can shield the feature information of the image block that does not contain the object. If multiple objects appear in the image block, the element value corresponding to the image block in the second mask matrix is ​​determined to be 1, so that the second mask matrix can retain the feature information of the image block containing multiple objects; if multiple objects do not appear in the image block (only 1 object appears or no object appears), the element value corresponding to the image block in the second mask matrix is ​​determined to be 0, so that the second mask matrix can shield the feature information of the image block that does not contain multiple objects. Among them, the presence of multiple objects in the image block indicates that the distance between the multiple objects is close enough and there is a relationship between the objects. Therefore, by determining whether an object appears in the image block and whether multiple objects appear, the accuracy of the first mask matrix and the second mask matrix is ​​improved, thereby improving the accuracy of feature processing based on the attention mask matrix.

[0095] S406: Determine an attention mask matrix according to the first mask matrix and the second mask matrix.

[0096] In this embodiment, the first mask matrix and the second mask matrix can be used as components of the attention mask matrix, and a complete attention mask matrix can be obtained by substituting the first mask matrix and the second mask matrix into the attention mask matrix.

[0097] In a possible implementation, the attention mask matrix includes a sub-mask matrix corresponding to the similarity between the object feature and the local feature, and a sub-mask matrix corresponding to the similarity between the relationship feature and the local feature. S406 may include: in the attention mask matrix, determining the sub-mask matrix corresponding to the similarity between the object feature and the local feature as the first mask matrix; determining the sub-mask matrix corresponding to the similarity between the relationship feature and the local feature as the second mask matrix.

[0098] In this implementation, when the object features and the relationship features are initialized based on the global features, the object features contain feature information that is irrelevant to the object, and the relationship features contain feature information that is irrelevant to the relationship between objects. In order to shield these irrelevant feature information, considering the correspondence between the local features and the image blocks, the correspondence between the elements in the first mask matrix and the image blocks, and the correspondence between the elements in the second mask matrix and the image blocks, the sub-mask matrix corresponding to the similarity between the object features and the local features is determined as the first mask matrix, and the sub-mask matrix corresponding to the similarity between the relationship features and the local features is determined as the second mask matrix, so as to use the first mask matrix to shield the similarity between the object features and the local features that is irrelevant to the object, and use the second mask matrix to shield the similarity between the relationship features and the local features that is irrelevant to the relationship between objects. The similarity between object features and local features, and the similarity between relationship features and local features are used to perform feature processing in combination with the attention mask matrix during one or more rounds of feature processing of the first feature, the second feature, the third feature, and the fourth feature based on the attention mask matrix corresponding to the image, so as to further utilize the similarity between features and the attention mask mechanism to improve the accuracy of feature processing.

[0099] In a possible implementation, the attention mask matrix includes sub-mask matrices corresponding to the similarities between each pair of features in global features, local features, object features, and relationship features. Among them, the similarities between each pair of features in global features, local features, object features, and relationship features include: the similarity between global features and global features, the similarity between global features and local features, the similarity between global features and object features, the similarity between global features and relationship features, the similarity between local features and local features, the similarity between local features and object features, the similarity between local features and relationship features, the similarity between object features and object features, the similarity between object features and relationship features, and the similarity between relationship features and relationship features. The attention mask matrix includes sub-mask matrices corresponding to these similarities, and the sub-mask matrices are used to mask the similarities corresponding to the sub-mask matrices.

[0100] Among them, according to the above implementation method, it can be known that: the sub-mask matrix corresponding to the similarity between object features and local features is the first mask matrix, and the sub-mask matrix corresponding to the similarity between relationship features and local features is the second mask matrix. For other similarities: the similarity between global features and global features, the similarity between global features and object features, the similarity between global features and relationship features, the similarity between global features and local features, the similarity between local features and local features, the similarity between object features and object features, the similarity between relationship features and relationship features, and the similarity between object features and relationship features, the sub-mask matrices corresponding to the similarity are configured sub-mask matrices. Therefore, combining the configured sub-mask matrix, the first mask matrix and the second mask matrix, a complete attention mask matrix can be obtained, so that the attention mask matrix can process the similarity between each pair of features in global features, local features, object features and relationship features.

[0101] In a possible implementation, the sub-mask matrices corresponding to the similarities between global features, the similarities between global features and object features, the similarities between global features and relationship features, the similarities between global features and local features, the similarities between local features, the similarities between object features and object features, and the similarities between relationship features and relationship features are sub-mask matrices with all element values ​​of 1. Considering that these similarities are important for the extraction of global features, object features, relationship features, and local features, these similarities are retained by configuring a sub-mask matrix with all element values ​​of 1. The sub-mask matrix corresponding to the similarity between object features and relationship features is a sub-mask matrix with all element values ​​of 0. Considering that object features reflect the features of a single object and relationship features reflect the features of the relationship between multiple objects, there is no need to pay attention to the similarity between object features and relationship features. By configuring a sub-mask matrix with all elements of 0 for the similarity between object features and relationship features, the similarity between object features and relationship features is shielded.

[0102] As an example, the attention mask matrix can be expressed as:

[0103]

[0104] Among them, Mask_Attention represents the attention mask matrix, O_mask represents the first mask matrix, R_mask represents the second mask matrix, 1 represents a sub-mask matrix with all element values ​​1, and 0 represents a sub-mask matrix with all element values ​​0. In the attention mask matrix, each row corresponds to the relationship feature (which can be expressed as Relation_token), object feature (which can be expressed as Object_token), global feature (which can be expressed as Cls_token) and local feature (which can be expressed as Patch_token), and each column corresponds to the relationship feature, object feature, global feature and local feature, then the sub-mask matrix located in the i-th row and j-th column is the sub-mask matrix corresponding to the similarity between the feature corresponding to the i-th row and the feature corresponding to the j-th row. For example, the sub-mask matrix in the first row and first column is the sub-mask matrix corresponding to the similarity between the relationship features and the relationship features. From the matrix formula, it can be seen that it is a sub-mask matrix with all element values ​​1; the sub-mask matrix in the first row and second column and the sub-mask matrix in the second row and first column are the sub-mask matrices corresponding to the similarity between the relationship features and the target features. From the matrix formula, it can be seen that it is a sub-mask matrix with all element values ​​0; the sub-mask matrix in the first row and third column and the sub-mask matrix in the third row and first column are the sub-mask matrices corresponding to the similarity between the relationship features and the target features. The sub-mask matrix corresponding to the similarity of the global features is a sub-mask matrix with all element values ​​1 as can be seen from the matrix formula; the sub-mask matrix in the 1st row and 4th column and the sub-mask matrix in the 4th row and 1st column are the sub-mask matrices corresponding to the similarity between the relationship features and the local features. It can be seen from the matrix formula that they are the second mask matrix; the sub-mask matrix in the 2nd row and 2nd column is the sub-mask matrix corresponding to the similarity between the object features and the object features. It can be seen from the matrix formula that it is a sub-mask matrix with all element values ​​1. Matrix; the sub-mask matrix of the 2nd row and 3rd column and the sub-mask matrix of the 3rd row and 2nd column are the sub-mask matrices corresponding to the similarity between the object features and the global features. From the matrix formula, it can be seen that it is a sub-mask matrix with all element values ​​1; the sub-mask matrix of the 2nd row and 4th column and the sub-mask matrix of the 4th row and 2nd column are the sub-mask matrices corresponding to the similarity between the object features and the local features. From the matrix formula, it can be seen that it is the first mask matrix; the sub-mask matrix of the 3rd row and 3rd column is the sub-mask matrix corresponding to the similarity between the global features and the global features. The sub-mask matrix, as can be seen from the matrix formula, is a sub-mask matrix in which all element values ​​are 1; the sub-mask matrix in the 3rd row and 4th column and the sub-mask matrix in the 4th row and 3rd column are sub-mask matrices corresponding to the similarity between global features and local features. It can be seen from the matrix formula that it is a sub-mask matrix in which all element values ​​are 1; the sub-mask matrix in the 4th row and 4th column is the sub-mask matrix corresponding to the similarity between local features and local features. It can be seen from the matrix formula that it is a sub-mask matrix in which all element values ​​are 1.

[0105] S407, based on the attention mask matrix corresponding to the image, perform one or more rounds of feature processing on the first feature, the second feature, the third feature and the fourth feature to obtain global features, local features, object features and relationship features.

[0106] The implementation principle and technical effects of S407 may refer to the aforementioned embodiments and will not be described in detail.

[0107] In a possible implementation, in the first round of feature processing, a feature similarity matrix reflecting the similarity between the first feature, the second feature, the third feature and the fourth feature in the first round of feature processing can be determined based on the first feature, the second feature, the third feature and the fourth feature; the feature similarity matrix is ​​masked by the attention mask matrix to obtain the feature similarity matrix after masking; the first feature, the second feature, the third feature and the fourth feature are processed based on the feature similarity matrix to obtain the first feature, the second feature, the third feature and the fourth feature after the first round of feature processing. If there are multiple rounds of feature processing, then in the next round of feature processing, a feature similarity matrix reflecting the similarity between the first feature, the second feature, the third feature and the fourth feature in the first round of feature processing in the current round of feature processing can be determined based on the first feature, the second feature, the third feature and the fourth feature after the previous round of feature processing; the feature similarity matrix is ​​masked by the attention mask matrix to obtain the feature similarity matrix after masking, and the above process is repeated. In this way, the accuracy of feature processing is improved by combining the attention mask matrix and feature similarity.

[0108] In the disclosed embodiment, a first mask matrix and a second mask matrix are determined based on the position distribution of objects in an image to improve the accuracy of the first mask matrix and the second mask matrix. An attention mask matrix is ​​determined based on the first mask matrix and the second mask matrix, thereby improving the accuracy of the attention mask matrix and the accuracy of global features, local features, object features and relationship features obtained by feature processing based on the attention mask matrix.

[0109] Figure 5 is a schematic diagram according to the fourth embodiment of the present disclosure. Figure 5 As shown, the image feature extraction process provided by the fourth embodiment of the present disclosure includes:

[0110] S501, extracting global features from the image to obtain a first feature.

[0111] S502, extracting local features from the image to obtain a second feature.

[0112] S503, initializing features of objects included in the image to obtain a third feature.

[0113] S504, initializing the features of the relationship between the objects included in the image to obtain a fourth feature.

[0114] The implementation principles and technical effects of S501 to S504 may refer to the aforementioned embodiments and will not be described in detail.

[0115] S505, determining a feature similarity matrix based on the combined feature matrix to be processed in the Nth round of feature processing, where N=1, the combined feature matrix to be processed is obtained based on the first feature, the second feature, the third feature and the fourth feature, and when N is greater than 1, the combined feature matrix to be processed is the combined feature matrix after the N-1th round of feature processing.

[0116] In this embodiment, when N=1, that is, in the first round of feature processing, the first feature, the second feature, the third feature and the fourth feature can be combined and spliced ​​to obtain a combined feature matrix. When N is greater than 1, the combined feature matrix after the N-1th round of feature processing can be obtained. For example, in the second round of feature processing, the combined feature matrix after the first round of feature processing is obtained, and in the third round of feature processing, the combined feature matrix after the second round of feature processing is obtained to achieve iterative processing of features.

[0117] After obtaining the combination feature matrix to be processed in the Nth round of feature processing, the features participating in the combination in the combination feature matrix to be processed can be calculated pairwise similarity to obtain a feature similarity matrix. When N=1, in the Nth round of feature processing, the features participating in the combination in the combination feature matrix to be processed are the first feature, the second feature, the third feature, and the fourth feature. The first feature, the second feature, the third feature, and the fourth feature are calculated pairwise similarity to obtain feature similarity between the two features, and the feature similarity between the two features is combined into a feature similarity matrix. When N is greater than 1, in the Nth round of feature processing, the features participating in the combination in the combination feature matrix to be processed are the first feature after the N-1th round of feature processing, the second feature after the N-1th round of feature processing, the third feature after the N-1th round of feature processing, and the fourth feature after the N-1th round of feature processing. The first feature after the N-1th round of feature processing, the second feature after the N-1th round of feature processing, the third feature after the N-1th round of feature processing, and the fourth feature after the N-1th round of feature processing are subjected to pairwise similarity calculation to obtain feature similarity between pairwise features, and the feature similarity between pairwise features is combined into a feature similarity matrix.

[0118] In a possible implementation, S505 may include: multiplying the combined feature matrix to be processed in the Nth round of feature processing with the transposed matrix corresponding to the combined feature matrix to be processed in the Nth round of feature processing to obtain a feature similarity matrix. Thus, the efficiency of feature similarity calculation is improved by using the product of the matrix and the matrix transposition. For example, each column in the combined feature matrix to be processed corresponds to the relationship feature, object feature, global feature and local feature in sequence, and each row in the transposed matrix corresponding to the combined feature matrix to be processed corresponds to the relationship feature, object feature, global feature and local feature in sequence. Multiplying the combined feature matrix and the transposed matrix can realize the multiplication of relationship feature and relationship feature, the multiplication of relationship feature and object feature, the multiplication of relationship feature and global feature, ..., and realize the calculation of pairwise feature similarity.

[0119] S506, based on the attention mask matrix, masking unimportant feature similarities in the feature similarity matrix and retaining important feature similarities, to obtain a masked feature similarity matrix.

[0120] In this embodiment, the attention mask matrix includes sub-mask matrices corresponding to the pairwise feature similarities, which can be specifically referred to in the above embodiment and will not be described in detail. The attention mask matrix can be used to mask the feature similarity matrix, shield the unimportant feature similarities in the feature similarity matrix, and retain the important feature similarities in the feature similarity matrix to obtain the masked feature similarity matrix, thereby improving the accuracy of each round of feature processing through the attention mask mechanism.

[0121] In a possible implementation, the attention mask matrix can be multiplied with the feature similarity matrix to obtain the feature similarity matrix after mask processing. Among them, in the feature similarity matrix: the feature similarity multiplied with the sub-mask matrix whose element values ​​in the attention mask matrix are all 0 will be shielded; the feature similarity multiplied with the sub-mask matrix whose element values ​​in the attention mask matrix are all 1 will be retained; in the feature similarity multiplied with the first mask matrix, the feature similarity corresponding to the image block containing the object will be retained, and the feature similarity corresponding to the image block not containing the object will be shielded: in the feature similarity multiplied with the second mask matrix, the feature similarity corresponding to the image block containing multiple objects will be retained, and the feature similarity corresponding to the image block not containing multiple objects will be shielded. Therefore, by multiplying the attention mask matrix with the feature similarity matrix, the processing accuracy of the feature similarity matrix is ​​improved.

[0122] S507, performing feature processing on the combined feature matrix to be processed according to the feature similarity matrix after mask processing, to obtain a combined feature matrix after the Nth round of feature processing.

[0123] In this embodiment, after obtaining the masked feature similarity matrix, feature processing can be performed on the combined feature matrix to be processed based on the masked feature similarity matrix, such as by fusion methods such as weighting and summing to obtain a combined feature matrix that has undergone the Nth round of feature processing.

[0124] In a possible implementation, S507 may include: multiplying the feature similarity matrix after mask processing with the combined feature matrix to be processed to obtain the combined feature matrix after the Nth round of feature processing. Since the feature similarity matrix has been screened by the attention mask matrix, the feature similarity matrix after mask processing is multiplied with the combined feature matrix to be processed. The feature similarity matrix after mask processing is equivalent to providing a direction for the optimization of the combined feature matrix. For example, for object features and local features, the feature similarity between the object features and the local features is masked by the first mask matrix, and the similarity between the object features and the feature information related to the object in the local features is retained. The object features are optimized using the similarity, thereby improving the accuracy of the object features. For another example, for relationship features and local features, the feature similarity between the relationship features and the local features is masked by the second mask matrix, and the similarity between the relationship features and the feature information related to the relationship between objects in the local features is retained. The relationship features are optimized using the similarity, thereby improving the accuracy of the relationship features. In this way, the accuracy of feature processing can be improved.

[0125] S508: Whether N is greater than or equal to the number threshold.

[0126] The number threshold can be pre-configured.

[0127] In this embodiment, if N is greater than or equal to the number threshold, S510 is executed, otherwise, S509 may be executed to perform N+1 rounds of feature processing.

[0128] S509, perform an increment operation on N.

[0129] In this embodiment, after adding 1 to N, the process may jump to S505 to perform the next round of feature processing to improve the feature processing effect.

[0130] S510, obtaining global features, local features, object features and relationship features from the combined feature matrix of the Nth round of feature processing.

[0131] In this embodiment, if N is greater than or equal to the number threshold, the iterative processing of the features can be terminated, and the global features, local features, object features, and relationship features can be separated from the combined feature matrix of the Nth round of feature processing. For example, multiple columns of feature vectors are obtained from the combined feature matrix of the Nth round of feature processing, and global features, local features, object features, and relationship features are obtained from the multiple columns of feature vectors according to the feature types corresponding to the multiple columns of feature vectors. For example, the first column of feature vectors is a global feature; the second column of feature vectors is a local feature. Since there can be multiple local features, the third column of feature vectors and the fourth column of feature vectors can also be different local features; there can also be multiple object features and relationship features, and the object features and relationship features can be obtained in sequence according to the feature types corresponding to the multiple columns of feature vectors.

[0132] In the disclosed embodiment, in the feature extraction process of the image feature, the first feature, the second feature, the third feature and the fourth feature are subjected to multiple rounds of iterative feature processing based on the attention mask matrix to obtain global features, local features, object features and relationship features. In each round of iteration, the feature similarity matrix is ​​subjected to mask processing based on the attention mask matrix, and the feature similarity of the mask processing is used to guide the optimization of the first feature, the second feature, the third feature and the fourth feature. In this way, after multiple rounds of iterative feature processing, the accuracy of image feature extraction is improved.

[0133] It should be noted that a feature extraction model can be used in the aforementioned embodiment, and the feature extraction model can include a feature extraction layer for extracting the initial first feature, second feature, third feature and fourth feature; the feature extraction model can also include multiple masked attention layers, in which feature processing can be performed through multiple network layers, and the above-mentioned process of processing the first feature, the second feature, the third feature and the fourth feature using the attention mask matrix can also be implemented. The number of masked attention layers determines the number of feature iteration processing. The network parameters of multiple masked attention layers are determined during the training process, and the network parameters of different masked attention layers may be different. Here, there is no restriction on the specific structure of the feature extraction layer and the masked attention layer. For example, the feature extraction layer can use an embedding layer, and the masked attention layer can include multiple fully connected layers.

[0134] As an example, Figure 6 is an example diagram of the image feature extraction process provided according to the above-mentioned embodiment of the present disclosure. Figure 6 As shown, the image is input into the layer for feature extraction, and the initial relationship features can be extracted (there can be multiple relationship features, Figure 6 Take 1 as an example), object features (there can be multiple object features, Figure 6Take 2 as an example), global features and local features (there can be multiple local features, Figure 6 Take 2 as an example); input the attention mask matrix and the initial image features (initial relationship features, object features, global features, local features) into the first mask attention layer for feature processing, and obtain the image features after the first round of feature processing (relationship features, object features, global features, local features after the first round of feature processing); input the attention mask matrix and the image features after the first round of feature processing into the second mask attention layer for feature processing, and obtain the image features after the second round of feature processing (relationship features, object features, global features, local features after the second round of feature processing). If the number of iterations is 2, the relationship features, object features, global features, and local features after the second round of feature processing are the final extracted relationship features, object features, global features, and local features.

[0135] It should be noted that the feature extraction process provided in the above embodiment can be performed before image retrieval and stored in the database after extraction for image retrieval, or it can be performed during the image retrieval process.

[0136] Figure 7 Schematic diagram of the fifth embodiment of the present disclosure. Figure 7 As shown, the image retrieval device 700 provided in the fifth embodiment of the present disclosure includes:

[0137] The request acquisition unit 701 is used to acquire a search request, which carries search requirement information.

[0138] The attribute feature acquisition unit 702 is used to acquire the attribute features of the retrieval requirement information.

[0139] The image retrieval unit 703 is used to perform image retrieval in multiple images according to attribute features and image features of the image to obtain a target image that meets the retrieval requirement information. The image features include global features, local features, object features of included objects, and relationship features between objects.

[0140] In some embodiments, the retrieval requirement information includes retrieval images, the attribute features include global retrieval features of the retrieval images, local retrieval features, object retrieval features containing objects, and relationship retrieval features between objects, and / or, the retrieval requirement information includes retrieval text, and the attribute features include text features of the retrieval text.

[0141] In some embodiments, the retrieval requirement information includes a retrieval type of the retrieval request, and the retrieval type includes one or more of the following: scene retrieval, object retrieval, and relationship retrieval between objects. The image retrieval unit 703 includes: a feature selection module, which is used to select target features related to the retrieval type from image features; and a feature matching module, which is used to perform image retrieval in multiple images by matching attribute features with target features to obtain a target image.

[0142] In some embodiments, the image feature extraction process includes: performing global feature extraction on the image to obtain a first feature; performing local feature extraction on the image to obtain a second feature; performing feature initialization on the objects contained in the image to obtain a third feature; performing feature initialization on the relationship between the objects contained in the image to obtain a fourth feature; based on the attention mask matrix corresponding to the image, performing one or more rounds of feature processing on the first feature, the second feature, the third feature and the fourth feature to obtain global features, local features, object features and relationship features.

[0143] In some embodiments, the attention mask matrix is ​​determined in the following manner: based on the position distribution of objects contained in the image, a first mask matrix corresponding to object features and a second mask matrix corresponding to relationship features are determined; based on the first mask matrix and the second mask matrix, an attention mask matrix is ​​determined.

[0144] In some embodiments, a first mask matrix corresponding to object features and a second mask matrix corresponding to relationship features are determined based on the position distribution of objects contained in the image, including: dividing the image into multiple image blocks; determining multiple element values ​​in the first mask matrix and multiple element values ​​in the second mask matrix based on the distribution of objects contained in the image in the multiple image blocks; wherein the multiple image blocks correspond one-to-one to the multiple element values ​​in the first mask matrix, and the multiple image blocks correspond one-to-one to the multiple element values ​​in the second mask matrix.

[0145] In some embodiments, multiple element values ​​in a first mask matrix and multiple element values ​​in a second mask matrix are determined based on the distribution of objects contained in an image in multiple image blocks, including: for each image block in the multiple image blocks, if an object appears in the image block, determining that the element value corresponding to the image block in the first mask matrix is ​​1, otherwise, determining that the element value corresponding to the image block in the first mask matrix is ​​0; for each image block in the multiple image blocks, if multiple objects appear in the image block, determining that the element value corresponding to the image block in the second mask matrix is ​​1, otherwise, determining that the element value corresponding to the image block in the second mask matrix is ​​0.

[0146] In some embodiments, the attention mask matrix includes a sub-mask matrix corresponding to the similarity between object features and local features, and a sub-mask matrix corresponding to the similarity between relationship features and local features. According to the first mask matrix and the second mask matrix, the attention mask matrix is ​​determined, including: in the attention mask matrix, determining the sub-mask matrix corresponding to the similarity between object features and local features as the first mask matrix; determining the sub-mask matrix corresponding to the similarity between relationship features and local features as the second mask matrix.

[0147] In some embodiments, the attention mask matrix includes sub-mask matrices corresponding to the similarities between pairs of features among global features, local features, object features, and relationship features; the sub-mask matrices corresponding to the similarities between global features and global features, the similarities between global features and object features, the similarities between global features and relationship features, the similarities between global features and local features, the similarities between local features and local features, the similarities between object features and object features, the similarities between relationship features and relationship features, and the similarities between object features and relationship features are configured sub-mask matrices.

[0148] In some embodiments, the sub-mask matrices corresponding to the similarity between global features, the similarity between global features and object features, the similarity between global features and relationship features, the similarity between global features and local features, the similarity between local features, the similarity between object features and object features, and the similarity between relationship features and relationship features are sub-mask matrices in which all element values ​​are 1; the sub-mask matrix corresponding to the similarity between object features and relationship features is a sub-mask matrix in which all element values ​​are 0.

[0149] In some embodiments, the attention mask matrix includes sub-mask matrices corresponding to the similarities between each pair of features in the global features, local features, object features, and relationship features; based on the attention mask matrix corresponding to the image, one or more rounds of feature processing are performed on the first feature, the second feature, the third feature, and the fourth feature to obtain global features, local features, object features, and relationship features, including: based on the combined feature matrix to be processed in the Nth round of feature processing, determining the feature similarity matrix, when N=1, the combined feature matrix to be processed is obtained based on the first feature, the second feature, the third feature, and the fourth feature, and when N is greater than 1, The combined feature matrix to be processed is the combined feature matrix after the N-1th round of feature processing; based on the attention mask matrix, unimportant feature similarities in the feature similarity matrix are shielded and important feature similarities are retained to obtain the feature similarity matrix after mask processing; according to the feature similarity matrix after mask processing, feature processing is performed on the combined feature matrix to be processed to obtain the combined feature matrix after the Nth round of feature processing; if N is greater than or equal to the number threshold, global features, local features, object features and relationship features are obtained from the combined feature matrix of the Nth round of feature processing, otherwise the N+1th round of feature processing is continued.

[0150] In some embodiments, determining a feature similarity matrix based on the combined feature matrix to be processed in the Nth round of feature processing includes: multiplying the combined feature matrix to be processed by a transposed matrix corresponding to the combined feature matrix to be processed to obtain a feature similarity matrix.

[0151] In some embodiments, feature processing is performed on the combined feature matrix to be processed according to the feature similarity matrix after mask processing to obtain a combined feature matrix that has undergone the Nth round of feature processing, including: multiplying the feature similarity matrix after mask processing with the combined feature matrix to be processed to obtain a combined feature matrix that has undergone the Nth round of feature processing.

[0152] Figure 7 The provided image retrieval device can execute the steps involved in the terminal in the above-mentioned corresponding method embodiments. Its implementation principles and technical effects are similar and will not be repeated here.

[0153] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the solution provided by any of the above embodiments.

[0154] According to an embodiment of the present disclosure, the present disclosure further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute a solution provided by any of the above embodiments.

[0155] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0156] Figure 8 8 is a schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0157] like Figure 8 As shown, the electronic device 800 includes a computing unit 801, which can store data stored in a read-only memory (ROM) ( Figure 8 A computer program in ROM 802 is loaded from storage unit 808 to random access memory (RAM) ( Figure 8 The computer programs in the RAM 803 are used to perform various appropriate actions and processes. The RAM 803 may also store various programs and data required for the operation of the electronic device 800. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. The input / output (I / O) interface ( Figure 8 The I / O interface 805 is also connected to the bus 804 .

[0158] Multiple components in the electronic device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the electronic device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0159] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSP), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as the image retrieval method. For example, in some embodiments, the image retrieval method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the image retrieval method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute the image retrieval method in any other appropriate manner (eg, by means of firmware).

[0160] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0161] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0162] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0163] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0164] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

[0165] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0166] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0167] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An image retrieval method, comprising: Obtaining a search request, wherein the search request carries search requirement information; The retrieval requirement information includes: retrieval image and / or retrieval text, or the retrieval type of the retrieval request; the retrieval type includes one or more of the following: scene retrieval, object retrieval, and relationship retrieval between objects; Acquire attribute characteristics of the retrieval requirement information; According to the attribute features and the image features of the image, image retrieval is performed in multiple images to obtain a target image that meets the retrieval requirement information, wherein the image features include global features, local features, object features of included objects, and relationship features between objects; wherein the object features reflect the characteristics of the objects included in the image, and the relationship features reflect the characteristics of the relationship between the objects included in the image; When the retrieval requirement information includes the retrieval type of the retrieval request, performing image retrieval among multiple images according to the attribute feature and the image feature of the image to obtain a target image that meets the retrieval requirement information includes: Selecting a target feature related to the retrieval type from the image features; wherein the scene retrieval associates the global feature, the local feature, the object feature of the included object, and the relationship feature between the objects; the object retrieval associates the object feature of the included object; the relationship retrieval between objects associates the relationship feature between the objects; by matching the attribute feature with the target feature, performing image retrieval in the multiple images to obtain the target image; The image feature extraction process includes: Perform global feature extraction on the image to obtain a first feature; perform local feature extraction on the image to obtain a second feature; perform feature initialization on the objects contained in the image to obtain a third feature; perform feature initialization on the relationship between the objects contained in the image to obtain a fourth feature; Based on the attention mask matrix corresponding to the image, one or more rounds of feature processing are performed on the first feature, the second feature, the third feature and the fourth feature to obtain the global feature, the local feature, the object feature and the relationship feature.

2. The image retrieval method according to claim 1, wherein: When the retrieval requirement information includes a retrieval image, the attribute features include image retrieval features of the retrieval image, the image retrieval features include global retrieval features, local retrieval features, object retrieval features containing objects, and relationship retrieval features between objects, and / or, when the retrieval requirement information includes a retrieval text, the attribute features include text features of the retrieval text.

3. The image retrieval method according to claim 1 or 2, wherein: The attention mask matrix is ​​determined as follows: Determining, according to the position distribution of the objects contained in the image, a first mask matrix corresponding to the object features and a second mask matrix corresponding to the relationship features; Determine the attention mask matrix according to the first mask matrix and the second mask matrix.

4. The image retrieval method according to claim 3, wherein: The determining, according to the position distribution of the objects contained in the image, a first mask matrix corresponding to the object features and a second mask matrix corresponding to the relationship features comprises: Divide the image into multiple image blocks; Determine a plurality of element values ​​in the first mask matrix and a plurality of element values ​​in the second mask matrix according to distribution of objects contained in the image in the plurality of image blocks; The multiple image blocks correspond one-to-one to multiple element values ​​in the first mask matrix, and the multiple image blocks correspond one-to-one to multiple element values ​​in the second mask matrix.

5. The image retrieval method according to claim 4, wherein: The determining, according to the distribution of the objects contained in the image in the multiple image blocks, multiple element values ​​in the first mask matrix and multiple element values ​​in the second mask matrix comprises: For each image block in the multiple image blocks, if an object appears in the image block, determine that the element value corresponding to the image block in the first mask matrix is ​​1; otherwise, determine that the element value corresponding to the image block in the first mask matrix is ​​0; For each image block in the multiple image blocks, if multiple objects appear in the image block, determine that the element value corresponding to the image block in the second mask matrix is ​​1; otherwise, determine that the element value corresponding to the image block in the second mask matrix is ​​0.

6. The image retrieval method according to claim 3, wherein: The attention mask matrix includes a sub-mask matrix corresponding to the similarity between the object feature and the local feature, and a sub-mask matrix corresponding to the similarity between the relationship feature and the local feature. The determining of the attention mask matrix according to the first mask matrix and the second mask matrix includes: In the attention mask matrix, determining a sub-mask matrix corresponding to the similarity between the object feature and the local feature as the first mask matrix; A sub-mask matrix corresponding to the similarity between the relationship feature and the local feature is determined as the second mask matrix.

7. The image retrieval method according to claim 6, wherein: The attention mask matrix includes sub-mask matrices corresponding to the similarities between each of the global features, the local features, the object features, and the relationship features; The sub-mask matrices corresponding to the similarity between the global features, the similarity between the global features and the object features, the similarity between the global features and the relationship features, the similarity between the global features and the local features, the similarity between the local features, the similarity between the object features and the object features, the similarity between the relationship features and the relationship features, and the similarity between the object features and the relationship features are configured sub-mask matrices.

8. The image retrieval method according to claim 7, wherein: The sub-mask matrices corresponding to the similarity between the global features, the similarity between the global features and the object features, the similarity between the global features and the relationship features, the similarity between the global features and the local features, the similarity between the local features, the similarity between the object features and the object features, and the similarity between the relationship features and the relationship features are sub-mask matrices in which all element values ​​are 1; The sub-mask matrix corresponding to the similarity between the object feature and the relationship feature is a sub-mask matrix in which all element values ​​are 0.

9. The image retrieval method according to claim 1 or 2, wherein: The attention mask matrix includes sub-mask matrices corresponding to the similarities between each of the global features, the local features, the object features, and the relationship features; The attention mask matrix corresponding to the image is used to perform one or more rounds of feature processing on the first feature, the second feature, the third feature, and the fourth feature to obtain the global feature, the local feature, the object feature, and the relationship feature, including: Determine a feature similarity matrix based on a combined feature matrix to be processed in an Nth round of feature processing, where when N=1, the combined feature matrix to be processed is obtained based on the first feature, the second feature, the third feature, and the fourth feature, and when N is greater than 1, the combined feature matrix to be processed is a combined feature matrix after an N-1th round of feature processing; Based on the attention mask matrix, unimportant feature similarities in the feature similarity matrix are masked and important feature similarities are retained to obtain a masked feature similarity matrix; According to the feature similarity matrix after mask processing, feature processing is performed on the combined feature matrix to be processed to obtain a combined feature matrix after the Nth round of feature processing; If N is greater than or equal to the number threshold, the global feature, the local feature, the object feature and the relationship feature are obtained from the combined feature matrix of the Nth round of feature processing, otherwise the N+1th round of feature processing is continued.

10. The image retrieval method according to claim 9, wherein: The determining of the feature similarity matrix based on the combined feature matrix to be processed in the Nth round of feature processing includes: The feature similarity matrix is ​​obtained by multiplying the combination feature matrix to be processed by a transposed matrix corresponding to the combination feature matrix to be processed.

11. The image retrieval method according to claim 9, wherein: The step of performing feature processing on the combined feature matrix to be processed according to the feature similarity matrix after mask processing to obtain the combined feature matrix after the Nth round of feature processing includes: The feature similarity matrix after the mask processing is multiplied by the combined feature matrix to be processed to obtain the combined feature matrix after the Nth round of feature processing.

12. An image retrieval device, comprising: A request acquisition unit, used to acquire a search request, wherein the search request carries search requirement information; The retrieval requirement information includes: retrieval image and / or retrieval text, or the retrieval type of the retrieval request; the retrieval type includes one or more of the following: scene retrieval, object retrieval, and relationship retrieval between objects; An attribute feature acquisition unit, used to acquire the attribute features of the retrieval requirement information; An image retrieval unit, configured to perform image retrieval in a plurality of images according to the attribute features and image features of the image, and obtain a target image that meets the retrieval requirement information, wherein the image features include global features, local features, object features of included objects, and relationship features between objects; wherein the object features reflect the characteristics of the objects included in the image, and the relationship features reflect the characteristics of the relationship between the objects included in the image; When the retrieval requirement information includes the retrieval type of the retrieval request, the image retrieval unit includes: A feature selection module, used to select target features related to the retrieval type from the image features; wherein the scene retrieval associates the global features, the local features, the object features of the included objects and the relationship features between the objects; the object retrieval associates the object features of the included objects; the relationship retrieval between objects associates the relationship features between the objects; A feature matching module, used for performing image retrieval in the multiple images by matching the attribute feature with the target feature to obtain the target image; The image feature extraction process includes: Perform global feature extraction on the image to obtain a first feature; perform local feature extraction on the image to obtain a second feature; perform feature initialization on the objects contained in the image to obtain a third feature; perform feature initialization on the relationship between the objects contained in the image to obtain a fourth feature; Based on the attention mask matrix corresponding to the image, one or more rounds of feature processing are performed on the first feature, the second feature, the third feature and the fourth feature to obtain the global feature, the local feature, the object feature and the relationship feature.

13. The image retrieval device according to claim 12, wherein: When the retrieval requirement information includes a retrieval image, the attribute features include image retrieval features of the retrieval image, the image retrieval features include global retrieval features, local retrieval features, object retrieval features containing objects, and relationship retrieval features between objects, and / or, when the retrieval requirement information includes a retrieval text, the attribute features include text features of the retrieval text.

14. The image retrieval device according to claim 12 or 13, wherein: The attention mask matrix is ​​determined as follows: Determine, according to the position distribution of the objects contained in the image, a first mask matrix corresponding to the object features and a second mask matrix corresponding to the relationship features; Determine the attention mask matrix according to the first mask matrix and the second mask matrix.

15. The image retrieval device according to claim 14, wherein: The determining, according to the position distribution of the objects contained in the image, a first mask matrix corresponding to the object features and a second mask matrix corresponding to the relationship features comprises: Divide the image into multiple image blocks; Determine a plurality of element values ​​in the first mask matrix and a plurality of element values ​​in the second mask matrix according to distribution of objects contained in the image in the plurality of image blocks; The multiple image blocks correspond one-to-one to multiple element values ​​in the first mask matrix, and the multiple image blocks correspond one-to-one to multiple element values ​​in the second mask matrix.

16. The image retrieval device according to claim 15, wherein: The determining, according to the distribution of the objects contained in the image in the multiple image blocks, multiple element values ​​in the first mask matrix and multiple element values ​​in the second mask matrix comprises: For each image block in the multiple image blocks, if an object appears in the image block, determine that the element value corresponding to the image block in the first mask matrix is ​​1; otherwise, determine that the element value corresponding to the image block in the first mask matrix is ​​0; For each image block in the multiple image blocks, if multiple objects appear in the image block, determine that the element value corresponding to the image block in the second mask matrix is ​​1; otherwise, determine that the element value corresponding to the image block in the second mask matrix is ​​0.

17. The image retrieval device according to claim 15, wherein: The attention mask matrix includes a sub-mask matrix corresponding to the similarity between the object feature and the local feature, and a sub-mask matrix corresponding to the similarity between the relationship feature and the local feature. The determining of the attention mask matrix according to the first mask matrix and the second mask matrix includes: In the attention mask matrix, determining a sub-mask matrix corresponding to the similarity between the object feature and the local feature as the first mask matrix; A sub-mask matrix corresponding to the similarity between the relationship feature and the local feature is determined as the second mask matrix.

18. The image retrieval device according to claim 17, wherein: The attention mask matrix includes sub-mask matrices corresponding to the similarities between each of the global features, the local features, the object features, and the relationship features; The sub-mask matrices corresponding to the similarity between the global features, the similarity between the global features and the object features, the similarity between the global features and the relationship features, the similarity between the global features and the local features, the similarity between the local features, the similarity between the object features and the object features, the similarity between the relationship features and the relationship features, and the similarity between the object features and the relationship features are configured sub-mask matrices.

19. The image retrieval device according to claim 18, wherein: The sub-mask matrices corresponding to the similarity between the global features, the similarity between the global features and the object features, the similarity between the global features and the relationship features, the similarity between the global features and the local features, the similarity between the local features, the similarity between the object features and the object features, and the similarity between the relationship features and the relationship features are sub-mask matrices in which all element values ​​are 1; The sub-mask matrix corresponding to the similarity between the object feature and the relationship feature is a sub-mask matrix in which all element values ​​are 0.

20. The image retrieval device according to claim 12 or 13, wherein: The attention mask matrix includes sub-mask matrices corresponding to the similarities between each of the global features, the local features, the object features, and the relationship features; The attention mask matrix corresponding to the image is used to perform one or more rounds of feature processing on the first feature, the second feature, the third feature, and the fourth feature to obtain the global feature, the local feature, the object feature, and the relationship feature, including: Determine a feature similarity matrix based on a combined feature matrix to be processed in an Nth round of feature processing, where when N=1, the combined feature matrix to be processed is obtained based on the first feature, the second feature, the third feature, and the fourth feature, and when N is greater than 1, the combined feature matrix to be processed is a combined feature matrix after an N-1th round of feature processing; Based on the attention mask matrix, unimportant feature similarities in the feature similarity matrix are masked and important feature similarities are retained to obtain a masked feature similarity matrix; According to the feature similarity matrix after mask processing, feature processing is performed on the combined feature matrix to be processed to obtain a combined feature matrix after the Nth round of feature processing; If N is greater than or equal to the number threshold, the global feature, the local feature, the object feature and the relationship feature are obtained from the combined feature matrix of the Nth round of feature processing, otherwise the N+1th round of feature processing is continued.

21. The image retrieval device according to claim 20, wherein: The determining of the feature similarity matrix based on the combined feature matrix to be processed in the Nth round of feature processing includes: The feature similarity matrix is ​​obtained by multiplying the combination feature matrix to be processed by a transposed matrix corresponding to the combination feature matrix to be processed.

22. The image retrieval device according to claim 20, wherein: The step of performing feature processing on the combined feature matrix to be processed according to the feature similarity matrix after mask processing to obtain the combined feature matrix after the Nth round of feature processing includes: The feature similarity matrix after the mask processing is multiplied by the combined feature matrix to be processed to obtain the combined feature matrix after the Nth round of feature processing.

23. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the image retrieval method according to any one of claims 1 to 11.

24. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the image retrieval method according to any one of claims 1 to 11.

25. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the image retrieval method according to any one of claims 1 to 11 are implemented.

Images