Image query method and device, equipment and storage medium

Abstract

The embodiment of the invention provides an image query method and device, equipment and a storage medium. The method comprises the following steps: determining a first group of images matched with a text query item from a first image set; determining a fused image feature by fusing a group of image features corresponding to the first group of images; and based on the fused image features, determining a second group of images from the second image set as a query result for the text query item. According to the embodiment of the invention, image query is carried out based on richer and more accurate fusion image features of the feature information, and the accuracy of image query can be effectively improved.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of image processing, and more particularly, to image query methods, apparatuses, devices, and computer-readable storage media. Background Art

[0002] As the demand for image query increases, image query technology has also developed rapidly. Image query technology can be divided into document search technology, single image search technology, etc. Finding a method to accurately query images is a key issue. Summary of the Invention

[0003] In a first aspect of the present disclosure, an image query method is provided. The method comprises: determining a first group of images matching a text query term from a first image set; determining fused image features by fusing a set of image features corresponding to the first group of images; and determining a second group of images from a second image set based on the fused image features as query results for the text query term.

[0004] In a second aspect of the present disclosure, a device for image query is provided. The device includes: a first determination module configured to determine a first group of images matching a text query term from a first image set; a second determination module configured to determine fused image features by fusing a group of image features corresponding to the first group of images; and a third determination module configured to determine a second group of images from a second image set based on the fused image features, to serve as query results for the text query term.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and the computer program can be executed by a processor to implement the method of the first aspect.

[0007] It should be understood that the content described in this summary section is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0009] Figure 1 A schematic diagram illustrating an example environment in which embodiments of the present disclosure can be implemented;

[0010] Figure 2 A flowchart illustrating a process for image query according to some embodiments of the present disclosure is shown;

[0011] Figure 3 A schematic diagram illustrating an example process of image query according to some embodiments of the present disclosure;

[0012] Figure 4 A schematic structural block diagram of an apparatus for image query according to certain embodiments of the present disclosure is shown;

[0013] Figure 5 A block diagram of an electronic device capable of implementing various embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0014] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0015] It should be noted that the titles of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and any type of embodiment may be included under any section / subsection. Furthermore, the embodiments described in any section / subsection may be combined in any manner with any other embodiments described in the same section / subsection and / or in different sections / subsections.

[0016] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may be included below. The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may be included below.

[0017] The embodiments of the present disclosure may involve user data, data acquisition and / or use, etc. These aspects shall comply with the corresponding laws, regulations and relevant provisions. In the embodiments of the present disclosure, all data collection, acquisition, processing, processing, forwarding, use, etc. are carried out on the premise that the user is aware of and confirms them. Accordingly, when implementing the various embodiments of the present disclosure, the types, scope of use, and usage scenarios of the data or information that may be involved should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with the relevant laws and regulations. The specific notification and / or authorization method may vary according to the actual situation and application scenario, and the scope of the present disclosure is not limited in this respect.

[0018] If this specification and the solutions in the examples involve the processing of personal information, such processing will be done only with a legitimate basis (such as with the consent of the subject of personal information or as necessary for the performance of a contract) and only within the prescribed or agreed scope. A user's refusal to process personal information other than that required for basic functions will not affect the user's use of basic functions.

[0019] Sample Environment

[0020] Figure 1 1 shows a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. Figure 1 As shown, example environment 100 may include electronic device 110 .

[0021] In this example environment 100, electronic device 110 and / or its attached devices are devices for searching for images based on a received text query term 120. In some embodiments, text query term 120 may be, for example, the query text term "snowing", and the images searched for based on the query text term "snowing" may be images of snowing.

[0022] The queried images can be used as training samples for the model training process and as query results in image query scenarios. In some embodiments, for certain specific scenarios, image queries can be used to find long-tail data, and the queried images (long-tail data) can be added to the training set as difficult samples. Specific scenarios such as assisted driving algorithm iteration scenarios can be used.

[0023] The electronic device 110 can be any type of mobile terminal, fixed terminal, or portable terminal, or a server, including a mobile handset, stationaiy computer, laptop computer, notebook computer, netbook computer, tablet computer, media player, multimedia device, palmtop computer, portable gaming device, VR / AR device, personal communication system (PCS) device, personal navigation device, personal digital assistants (PDAs), audio / video player, digital camera / camcorder, positioning device, television receiver, radio broadcast receiver, electronic book device, game device, or any combinations thereof, including the accessories and peripherals of these devices, or any combinations thereof. In some embodiments, the electronic device 110 can also be capable of supporting any type of interface to the user (such as "wearable" circuitry, etc.).

[0024] To improve the accuracy of image query, the first image set 130 and the second image set 140 can be accessed in the electronic device 110 of the present disclosure to query images. The first image set 130 and the second image set 140 contain a large number of images, which are used as the results of image query. The first image set 130 is used to perform a preliminary image query, and the second image set 140 is used to further perform a more accurate image query based on the image query results obtained from the first image set 130. The details of how to perform image query based on the first image set 130 and the second image set 140 will be described in the example processes below. The first image set 130 and the second image set 140 can be image sets in the electronic device, or image sets in other devices. In some examples, if the first image set 130 and the second image set 140 are image sets in other devices, the electronic device 110 can interact with the other devices to achieve access to the first image set 130 and the second image set 140.

[0025] Similarly, other devices may also be any type of mobile terminal, fixed terminal or portable terminal or server, specifically including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, handheld computers, portable game terminals, VR / AR devices, personal communication systems (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio broadcast receivers, e-book devices, game devices or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. In some embodiments, other devices may also support any type of interface for the user (such as "wearable" circuits, etc.).

[0026] It should be understood that the structure and function of the various elements in the environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of the present disclosure.

[0027] There are two main traditional image query methods. One is the text-to-image search method, which means that in the absence of a query image, the electronic device 110 can query for images containing the information corresponding to the text query term 120 by inputting a text query term 120. For example, if the input text query term 120 is "person", the queried images may be images containing people. Text semantic information is single, and even large models find it difficult to achieve high-precision cross-modal search from text to image, making it difficult to meet production accuracy requirements.

[0028] Another image query method is the single-image search method, which is to search for similar images by inputting a single image. For example, if a single image contains a pedestrian wearing red clothes, then the images queried based on this single image also contain pedestrians wearing red clothes. The single-image search method needs to match the image features of a single image with the image features corresponding to the images contained in the image set. If the image features corresponding to the images in the image set are obtained using a model with a small number of parameters, the query effect of the single-image search is general and it is difficult to meet production needs. Although the model that can realize single-image search does not require cross-modal training and can be realized by training on a small model, the information of a single image is limited. In particular, when the image features of the query are not obvious or there are deviations, the image query results will deviate from the requirements.

[0029] The embodiments of the present disclosure propose an improved image query solution. The solution includes determining a first set of images matching a text query term 120 from a first image set 130 ; determining fused image features by fusing a set of image features corresponding to the first set of images; and determining a second set of images from a second image set 140 based on the fused image features as query results for the text query term 120 .

[0030] According to the embodiments of the present disclosure, image query is performed based on fused image features with richer and more accurate feature information, which can effectively improve the accuracy of image query.

[0031] Specifically, the present disclosure utilizes an image query method that combines text-based image search with multi-image search. Electronic device 110 searches for a first set of images through text-based image search and uses this first set of images for image query, integrating richer and more accurate image features. The present disclosure avoids the issue of poor text-based image search performance. Furthermore, compared to single-image search, multi-image search significantly improves image query performance.

[0032] Example Process

[0033] Figure 2 FIG. 2 is a flow chart showing a process 200 for image query according to some embodiments of the present disclosure. The process 200 may be implemented at the electronic device 110. Figure 1 Process 200 is described.

[0034] In block 210 , the electronic device 110 determines a first set of images from the first image set 130 that match the text query term 120 .

[0035] The text query item 120 carries the associated information of the image to be queried. In some embodiments, the text query item 120 may be input into the electronic device 110 by the user 120. In other embodiments, the text query item 120 may also be automatically generated by the electronic device 110.

[0036] The first set of images is a plurality of images that are initially found to match the text query term 120 based on the text query term 120. For example, if the text query term 120 received by the electronic device is "a fallen pedestrian," each image in the first set of images found by the electronic device 110 based on the text query term 120 may include a fallen pedestrian.

[0037] To accurately determine the first group of images that match the text query term 120, in some embodiments, the electronic device 110 may determine a degree of match between the text query term 120 and each image in the first image set 130. The electronic device 110 identifies images in the first image set 130 that have a degree of match with the text query term 120 that is greater than a threshold as a group of candidate images. This group of candidate images is preliminarily screened as images that match the text query term 120.

[0038] In some embodiments, the electronic device 110 may determine all images in the set of candidate images as the first set of images.

[0039] In some other embodiments, the electronic device 110 may further rank the group of candidate images in descending order of matching degree, and determine a first preset number of images ranked high as the first group of images.

[0040] In some other embodiments, to improve the accuracy of the image query, the electronic device 110 may further receive reference information, where the reference information indicates that at least one image in the set of candidate images does not match the text query term 120. This reference information may be input by the user or obtained based on any appropriate rule. Based on the reference information, the electronic device 110 may remove the at least one image that does not match the text query term 120 from the set of candidate images to determine a first set of images.

[0041] The following describes a process of determining the matching degree between the text query item 120 and each image in the first image set 130 .

[0042] In some embodiments, the electronic device 110 may utilize a text-image feature extraction model to determine the text features corresponding to the text query term 120 and the image features corresponding to each image in the first image set 130. Examples of text-image feature extraction models include, but are not limited to, an extraction model fine-tuned from the Contrastive Language-Image Pre-training (CLIP) open-source model, a Vision Transformer (VIT) model, and the like. Based on the text features corresponding to the text query term 120 and the image features corresponding to each image in the first image set 130, the electronic device 110 may determine the similarity (matching degree) between the text features corresponding to the text query term 120 and the image features corresponding to each image in the first image set 130. The similarity may be, for example, cosine similarity. It should be noted that the text features and the image features corresponding to each image in the first image set 130 may be in vector form, matrix form, or the like. It should be noted that the text features and the features corresponding to each image in the first image set 130 have the same feature form and the same feature dimension. For example, the text feature and the image feature corresponding to each image in the first image set 130 are both a K-dimensional feature vector, and each component at the same position in each feature vector corresponding to the text request item and each image in the first image set 130 represents a type of feature.

[0043] To improve the efficiency of image queries, the electronic device 110 can store the image features corresponding to each image in the first image set 130 obtained using the text-image feature extraction model in a database of the electronic device 110. When performing an image query, the electronic device 110 only needs to use the text-image feature extraction model to obtain the text vector corresponding to the text query term 120 and match it with the image features corresponding to each image in the first image set 130 stored in the database. The electronic device 110 does not need to use the text-image feature extraction model to determine the image features corresponding to each image in the first image set 130 every time it performs an image query.

[0044] The following describes the training process of a text-image feature extraction model, taking the training of a text-image feature extraction model on an electronic device as an example.

[0045] The electronic device 110 obtains a training sample set, wherein the training sample set contains multiple training sample pairs, wherein the training sample pair includes a training text and a training image. For each training sample pair, corresponding annotation information is annotated, wherein the annotation information is mainly used to mark whether the training image and the training text correspond to the same category of targets. For example, if there is a fallen pedestrian in the training image, then the training image and the training text "fallen pedestrian" correspond to the same category of targets (fallen pedestrians). It should be noted that the training text and training image corresponding to the same category of targets can be used as aligned text and image. For each training sample pair, the electronic device 110 inputs the training sample pair into the text-image feature extraction model to be trained to obtain the first predicted image output by the text-image feature extraction model to be trained. The electronic device 110 can adjust the model parameters of the text-image feature extraction model to be trained based on the comparison of the first predicted image and the training image. When the predetermined training conditions are met, the electronic device 110 can obtain a trained text-image feature extraction model. The predetermined training condition may be that the number of training times reaches a predetermined threshold, or the loss function reaches a minimum value, or the training time reaches a predetermined duration, etc. In some embodiments, the loss function may be a triplet loss, the purpose of which is to make similar image features closer and dissimilar image features farther apart during the training process.

[0046] In block 220 , the electronic device 110 determines fused image features by fusing a set of image features corresponding to the first set of images.

[0047] In some embodiments, the electronic device 110 may use an image feature extraction model to determine the image features corresponding to each image in the first group of images. The image features corresponding to each image in the first group of images may be features in the form of vectors, or features in the form of matrices, and so on. It should be noted that the image features corresponding to each image in the first group of images have the same feature form and have the same feature dimensions. For example, the features corresponding to each image in the first group of images are all N-dimensional feature vectors, and for each component at the same position in the feature vector corresponding to each image in the first group of images, it represents a type of feature. Examples of image feature extraction models may include, but are not limited to, residual network (ResidualNetwork 50, ResNet50) models.

[0048] To improve the accuracy of image query, in some embodiments, the electronic device 110 may fuse a set of image features corresponding to the first set of images to determine a fused image feature, so that the electronic device 110 can subsequently query images based on the fused image feature.

[0049] In some embodiments, the electronic device 110 may determine a set of weights for the first group of images, wherein one weight corresponds to one image. Based on the set of weights, the electronic device 110 fuses the set of image features to determine the fused image features. The fused image features have the same feature dimensions as the image features corresponding to each image in the first group of images. Specifically, the electronic device 110 may determine, for each position in the image features corresponding to each image in the first group of images (one position corresponds to one feature dimension), the product of the feature component of the image at that position (the feature component of the target feature dimension) and the weight corresponding to the image, and determine this product as the component corresponding to the position in the fused image features. The electronic device 110 determines the fused image features based on the fused image feature components corresponding to each position. For example, the first group of images contains three images, namely, image 1, image 2, and image 3. The features corresponding to image 1 are (a, b), and the weight corresponding to image 1 is A. The features corresponding to image 2 are (c, d), and the weight corresponding to image 2 is B. The features corresponding to image 3 are (e, f), and the weight corresponding to image 3 is C. The fused image features are (A*a+B*c+C*e,A*b+B*d+C*f).

[0050] The following describes the process of determining the weight corresponding to each image in the first group of images.

[0051] In some embodiments, the set of weights corresponding to the first set of images may be a predetermined set of weights, that is, for each image in the first set of images, the weight corresponding to the image is a predetermined weight.

[0052] In other embodiments, the electronic device 110 may further determine a set of weights for the first set of images based on the degree of match between the first set of images and the text query term 120, where the higher the degree of match between the first set of images and the text query term 120, the greater the corresponding weight. Specifically, the electronic device 110 may determine, for each image in the first set of images, the degree of match between the image and the text query term 120 as the corresponding weight for the image. The electronic device 110 may also determine the sum of the degree of match between each image in the first set of images and the text query term 120, and, for each image in the first set of images, determine the ratio of the image to the sum as the corresponding weight for the image.

[0053] In block 230 , the electronic device 110 determines a second group of images from the second image set 140 based on the fused image features, as query results for the text query term 120 .

[0054] In some embodiments, the second image set 140 includes images to be queried, and the electronic device 110 can further perform an image query based on candidate image features and fused image features corresponding to the candidate images in the second image set 140. It should be noted that the number of images in the first image set 130 is smaller than the number of images in the second image set 140.

[0055] In some embodiments, the electronic device 110 may determine a second group of images from the second image set 140 based on a comparison between the fused image features and the candidate image features of the candidate images in the second image set 140. Similar to obtaining the image features corresponding to the images in the first group of images using an image feature extraction model, the candidate image features corresponding to the candidate images in the second image set 140 may also be determined by processing the corresponding candidate images using the image feature extraction model (the second image set 140 is associated with the image feature extraction model). Specifically, the electronic device 110 may determine the similarity between the fused image features and the candidate image features of the candidate images in the second image set 140, and determine a group of images in the second image set 140 whose similarity is greater than a similarity threshold as a group of target images, where the similarity may be cosine similarity, etc. This group of target images is further screened from the second image set 140 and are images that match the text query term 120.

[0056] In some embodiments, the electronic device 110 may determine all images in the group of target images as the second group of images.

[0057] In some other embodiments, the electronic device 110 may further rank the group of target images in descending order of similarity, and determine a second preset number of images with the highest ranking as the second group of images.

[0058] In some other embodiments, to improve the accuracy of the image query, the electronic device 110 may further receive target information, where the target information indicates that at least one image in the set of target images does not match the text query term 120. This target information may be input by the user or obtained based on any appropriate rule. The electronic device 110 removes the at least one image that does not match the text query term 120 from the set of target images to determine a second set of images.

[0059] To improve the efficiency of image queries, the electronic device 110 can store the candidate image features corresponding to each candidate image in the second image set 140 obtained using the image feature extraction model in a database of the electronic device 110. When performing an image query, the electronic device 110 only needs to match the fused image features with the candidate image features corresponding to each candidate image in the second image set 140 stored in the database. The electronic device 110 does not need to use the image feature extraction model to determine the candidate image features corresponding to each candidate image in the second image set 140 each time it performs an image query.

[0060] The following is a training process for the image feature extraction model, which is illustrated by taking the training of the image feature extraction model on the electronic device 110 as an example.

[0061] The electronic device 110 obtains a training sample set, wherein the training sample set includes a plurality of training sample images. For each training sample image, the electronic device 110 inputs the training sample image into the image feature extraction model to be trained, and obtains a second predicted image output by the image feature extraction model to be trained. The electronic device 110 can adjust the model parameters of the image feature extraction model to be trained based on the comparison between the second predicted image and the training sample image. When a predetermined training condition is met, the electronic device 110 obtains a trained image feature extraction model. This predetermined training condition can be that the number of training times reaches a predetermined threshold, or the loss function reaches a minimum value, or the training time reaches a predetermined duration, etc.

[0062] In some embodiments, the electronic device 110 may provide the retrieved second set of images to the user 120 in an appropriate manner as a query result for the text query item 120. For example, the electronic device 110 may present the retrieved second set of images through a display device.

[0063] Figure 3 A schematic diagram showing an example process 300 of image query according to some embodiments of the present disclosure is shown. Figure 3 The specific process of image query is explained.

[0064] The user enters a text query term 120, such as "fallen pedestrian."

[0065] The text-image feature extraction model 301 is used to obtain text features corresponding to the text query item 120 .

[0066] The image features corresponding to each image in the first image set 130 are also obtained based on the text-image feature extraction model 301 and stored in the database.

[0067] The electronic device 110 matches the text features with the image features in the first image set 130 to obtain a first group of images 302 .

[0068] The image features of the first group of images 302 are obtained using the image feature extraction model 303 .

[0069] It should be noted that the candidate image features corresponding to each candidate image in the second image set 140 are also obtained using the image feature extraction model 303 .

[0070] The image features of the first group of images 302 are fused to obtain fused image features.

[0071] The electronic device 110 matches the fused image features with the candidate image features corresponding to each candidate image in the second image set 140 to obtain a second group of images 305 .

[0072] According to the embodiments of the present disclosure, image query is performed based on fused image features with richer and more accurate feature information, which can effectively improve the accuracy of image query.

[0073] Specifically, the present disclosure utilizes an image query method that combines text-based image search with multi-image search. Electronic device 110 searches for a first set of images through text-based image search and uses this first set of images for image query, integrating richer and more accurate image features. The present disclosure avoids the issue of poor text-based image search performance. Furthermore, compared to single-image search, multi-image search significantly improves image query performance.

[0074] Example devices and equipment

[0075] The embodiments of the present disclosure also provide corresponding devices for implementing the above methods or processes. Figure 4 1 shows a schematic block diagram of an apparatus 400 for image query according to certain embodiments of the present disclosure. Apparatus 400 may be implemented as or included in an electronic device as discussed above. Each module / component in apparatus 400 may be implemented by hardware, software, firmware, or any combination thereof.

[0076] like Figure 4 As shown, the device 400 includes a first determination module 410, which is configured to determine a first group of images matching the text query item from a first image set; a second determination module 420, which is configured to determine fused image features by fusing a group of image features corresponding to the first group of images; and a third determination module 430, which is configured to determine a second group of images from a second image set based on the fused image features as a query result for the text query item.

[0077] In some embodiments, the apparatus 400 further includes a fourth determination module configured to: determine image features corresponding to each image in the first group of images using an image feature extraction model, wherein the image features corresponding to each image have the same feature dimension.

[0078] In some embodiments, the third determination module 430 is specifically configured to: determine a second group of images from the second image set based on a comparison between the fused image features and the candidate image features of the candidate images in the second image set, wherein the candidate image features are determined by processing the corresponding candidate images using the image feature extraction module.

[0079] In some embodiments, the third determination module 430 is specifically configured to: determine a set of weights for the first set of images; and fuse a set of image features based on the set of weights to determine a fused feature.

[0080] In some embodiments, the third determination module 430 is specifically configured to determine a set of weights for the first group of images based on the matching degree between the first group of images and the text query term.

[0081] In some embodiments, a group of image features have the same multiple feature dimensions, and the third determination module 430 is specifically configured to: apply a set of weights to the feature components of a group of image features with respect to the target feature dimensions to determine a fused feature component with respect to the target feature dimensions; and determine a fused feature based on the fused feature component.

[0082] In some embodiments, the first determination module 410 is specifically configured to: determine a group of candidate images based on the matching degree between the text query term and the first image set, where the matching degree between the group of candidate images and the text query term is higher than a threshold; and determine a first group of images based on the group of candidate images.

[0083] In some embodiments, the first determination module 410 is specifically configured to: obtain reference information indicating that at least one image in a set of candidate images does not match the text query term; and remove at least one image from the set of candidate images to determine a first set of images.

[0084] In some embodiments, the number of images in the first image set is less than the number of images in the second image set.

[0085] In some embodiments, the first image set is associated with a text-image feature extraction model; and the second image set is associated with an image feature extraction model.

[0086] The units included in the device 400 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units can be implemented using software and / or firmware, such as machine executable instructions stored on a storage medium. In addition to or as an alternative to machine executable instructions, some or all of the units in the device 400 can be implemented at least in part by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0087] Figure 5 1 shows a block diagram of an electronic device 500 in which one or more embodiments of the present disclosure may be implemented. Figure 5 The illustrated electronic device 500 is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 5 The electronic device 500 shown can be used to implement Figure 1 The terminal device 110 is shown.

[0088] like Figure 5 As shown, electronic device 500 is in the form of a general electronic device. Components of electronic device 500 may include, but are not limited to, one or more processors or processing units 510, memory 520, storage device 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. Processing unit 510 may be a real or virtual processor and is capable of performing various processes according to programs stored in memory 520. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to increase the parallel processing capabilities of electronic device 500.

[0089] The electronic device 500 typically includes a plurality of computer storage media. Such media can be any accessible media that can be obtained by the electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 520 can be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 530 can be a removable or non-removable medium and can include a machine-readable medium, such as a flash drive, a disk, or any other medium that can be used to store information and / or data (e.g., training data for training) and can be accessed within the electronic device 500.

[0090] The electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 5 As shown in FIG, a magnetic disk drive for reading from or writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to a bus (not shown) by one or more data media interfaces. Memory 520 may include a computer program product 525 having one or more program modules configured to perform various methods or actions of various embodiments of the present disclosure.

[0091] The communication unit 540 enables communication with other electronic devices via a communication medium. Additionally, the functions of the components of the electronic device 500 can be implemented in a single computing cluster or multiple computing machines that can communicate via a communication connection. Thus, the electronic device 500 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0092] Input device 550 may be one or more input devices, such as a mouse, keyboard, or trackball. Output device 560 may be one or more output devices, such as a display, a speaker, or a printer. Electronic device 500 may also communicate with one or more external devices (not shown) via communication unit 540 as needed, such as a storage device, a display device, or the like, with one or more devices that allow a user to interact with electronic device 500, or with any device that allows electronic device 500 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0093] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0094] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0095] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, such that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0096] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0097] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple implementations of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part for a module, program segment or instruction, and a part for a module, program segment or instruction comprises one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0098] While various implementations of the present disclosure have been described above, the foregoing description is intended to be illustrative, not exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is selected to best explain the principles of the implementations, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. An image query method, comprising: determining a first set of images from the first image set that match the text query term; determining a fused image feature by fusing a set of image features corresponding to the first set of images; as well as Based on the fused image features, a second group of images is determined from the second image set to serve as query results for the text query item.

2. The method according to claim 1, wherein determining the fused image features by fusing a set of image features corresponding to the first set of images comprises: determining a set of weights for the first set of images; as well as Based on the set of weights, the set of image features is fused to determine the fused feature.

3. The method of claim 2 , wherein determining a set of weights corresponding to the first set of images comprises: The set of weights for the first set of images is determined based on a degree of matching of the first set of images to the text query term.

4. The method of claim 2, wherein the set of image features have the same plurality of feature dimensions, and fusing the set of image features based on the set of weights to determine the fused feature comprises: Applying the set of weights to the feature components of the set of image features with respect to the target feature dimension, respectively, to determine a fused feature component with respect to the target feature dimension; as well as The fusion feature is determined based on the fusion feature components.

5. The method according to claim 1 , wherein the process of determining a set of image features corresponding to the first set of images comprises: An image feature extraction model is used to determine image features corresponding to each image in the first group of images, wherein the image features corresponding to each image have the same feature dimension.

6. The method of claim 5, wherein determining the second set of images from the second image set comprises: The second group of images is determined from the second image set based on a comparison between the fused image features and candidate image features of candidate images in the second image set, wherein the candidate image features are determined by processing the corresponding candidate images using the image feature extraction module.

7. The method of claim 1 , wherein determining a first set of images from the first image set that match the text query term comprises: determining a group of candidate images based on a degree of matching between the text query term and the first image set, wherein a degree of matching between the group of candidate images and the text query term is greater than a threshold; as well as Based on the set of candidate images, the first set of images is determined.

8. The method of claim 1 , wherein determining the first set of images based on the set of candidate images comprises: obtaining reference information indicating that at least one image in the set of candidate images does not match the text query term; as well as The at least one image is removed from the set of candidate images to determine the first set of images.

9. The method of claim 1, wherein the number of images in the first image set is smaller than the number of images in the second image set.

10. The method of claim 1, wherein the first image set is associated with a text-image feature extraction model; and the second image set is associated with an image feature extraction model.

11. A device for image query, comprising: A first determination module is configured to determine a first group of images matching the text query term from the first image set; a second determining module configured to determine a fused image feature by fusing a set of image features corresponding to the first set of images; as well as The third determining module is configured to determine a second group of images from the second image set based on the fused image features, to serve as query results for the text query item.

12. An electronic device comprising: at least one processing unit; as well as At least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 11 when executed by the at least one processing unit.

13. A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method according to any one of claims 1 to 11 when executed by a processor.

Images