Image retrieval method, readable medium and electronic equipment

Abstract

The invention relates to the technical field of image processing, and discloses an image retrieval method, a readable medium and electronic equipment. According to the image retrieval method, candidate images are determined in an asynchronous pipeline mode, distance values (similarities) between image features of images in an image library and image features of images to be retrieved are calculated, the similarities are ranked, and finally K images closest to the image features of the images to be retrieved are obtained. For example, according to the asynchronous pipeline mode, the image retrieval process can be divided into a plurality of sub-processes, all the sub-processes can be carried out at the same time, and the current sub-process does not need to be executed after the previous sub-process is executed. According to the image retrieval method, retrieval is carried out in an asynchronous pipeline mode, the retrieval time can be shortened, and the calculation efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to an image retrieval method, a readable medium and an electronic device. Background Art

[0002] At present, image retrieval technology can be widely used in image library retrieval management systems, video business retrieval systems, video surveillance retrieval systems and other scenarios. Figure 1a As shown, taking the shopping software in the mobile phone 10 as an example, in response to the user clicking on Figure 1a The "Photo Search" control 101 in the shopping software user interface 10a shown in FIG. 10 shows the mobile phone 10 as follows: Figure 1b The user interface 10b shown in FIG. 1 is a photo search interface, including a picture to be retrieved 102 and a "photo shopping" control 103. In response to the user clicking the "photo shopping" control 103, the mobile phone 10 displays the following Figure 1c The user interface 10 c shown is a search result interface, which includes a thumbnail 104 of the image 102 to be retrieved and images 105 , 106 , 107 and 108 similar to the image 102 to be retrieved.

[0003] At present, image retrieval methods can usually combine the timestamp information and spatial point information of the image for auxiliary retrieval, that is, image spatiotemporal retrieval. The image spatiotemporal retrieval method mainly includes: determining the image features, timestamp information and spatial point information of the image to be retrieved, and screening candidate images from the image library based on the timestamp information and spatial point information; then comparing the image features of the image to be retrieved with the image features of the candidate images, determining the K candidate images that are closest to the image features of the image to be retrieved as the target images, and returning the K target images as the retrieval results.

[0004] However, current image retrieval methods usually select the target image from the candidate images after screening all candidate images. If the number of images in the image library is very large, the entire retrieval process will take a lot of time. Summary of the invention

[0005] The purpose of this application is to provide an image retrieval method, a readable medium and an electronic device.

[0006] A first aspect of the present application provides an image retrieval method, which is applied to an electronic device, comprising: detecting a first image retrieval instruction; in response to the first image retrieval instruction, retrieving a target image matching the image to be retrieved from an image library, the image library comprising multiple image groups, each image group comprising multiple images, wherein, in response to the first image retrieval instruction, retrieving the target image from the image library comprises: in the Nth processing cycle: selecting a candidate image from the Nth image group, and calculating the similarity between the candidate image of the N-1th image group and the image to be retrieved, and sorting each candidate image of the N-2th image group according to the similarity between each candidate image of the N-2th image group and the image to be retrieved, wherein N is greater than 2.

[0007] In the embodiment of the present application, a first image retrieval instruction is detected, for example, a user clicks Figure 1b The operation of the "photo shopping" control 103 shown, in response to the first image retrieval instruction, retrieves a target image matching the image to be retrieved from the image library. The Nth processing cycle: the subprocess for determining the candidate image performs vector calculation based on the target time period information and target point information of the image to be retrieved and the spatiotemporal attributes of the Nth group of images, and marks the candidate image and the filtered image in the Nth group of images; the subprocess for calculating the distance value performs feature comparison based on the image features of the image to be retrieved and the feature vector of the N-1th group of images, and calculates the similarity between each image in the N-1th group and the image to be retrieved; the sorting subprocess sorts each image in the N-2th image group for which the distance value has been calculated from large to small (or small to large) according to the distance value, and N is an integer greater than 2.

[0008] It can be understood that the image retrieval method provided in the present application uses an asynchronous pipeline approach to perform retrieval, which can shorten the retrieval time.

[0009] In a possible implementation of the first aspect above, a candidate image is selected from the Nth image group, including: determining the target time period information and target point information of the image to be retrieved; determining the time filter mask bit set of the Nth image group based on the timestamp information corresponding to each image in the Nth image group and the target time period information of the image to be retrieved, wherein the value of each time bit in the time filter mask bit set indicates whether the corresponding image is a time candidate image, and determining the spatial filter mask bit set of the Nth image group based on the spatial point information corresponding to each image in the Nth image group and the target point information of the image to be retrieved, wherein the value of each spatial bit in the spatial filter mask bit set indicates whether the corresponding image is a spatial candidate image; determining the spatiotemporal filter mask bit set of the Nth image group based on the time filter mask bit set of the Nth image group and the spatial filter mask bit set of the Nth image group, wherein the value of the spatiotemporal bit in the spatiotemporal filter mask bit set indicates whether the corresponding image is a candidate image.

[0010] In an embodiment of the present application, taking the subprocess of determining the candidate images as an example, performing vector calculation based on the target time period information and target point information of the image to be retrieved and the spatiotemporal attributes of the Nth group of images, and marking the candidate images and the filtered images in the Nth group of images, the method of calculating and generating a spatiotemporal filter mask bit set according to the vector instruction includes: determining the target time period information and target point information of the image to be retrieved; determining the time filter mask bit set of the Nth image group based on the timestamp information corresponding to each image in the Nth image group and the target time period information of the image to be retrieved, wherein each time bit in the time filter mask bit set The value of the bit indicates whether the corresponding image is a temporal candidate image, and based on the spatial point information corresponding to each image in the Nth image group and the target point information of the image to be retrieved, the spatial filter mask bit set of the Nth image group is determined, wherein the value of each spatial bit in the spatial filter mask bit set indicates whether the corresponding image is a spatial candidate image; based on the temporal filter mask bit set of the Nth image group and the spatial filter mask bit set of the Nth image group, the spatiotemporal filter mask bit set of the Nth image group is determined, wherein the value of the spatiotemporal bit in the spatiotemporal filter mask bit set indicates whether the corresponding image is a candidate image.

[0011] In a possible implementation of the first aspect above, a time filter mask bit set of the Nth image group is determined based on the timestamp information corresponding to each image in the Nth image group and the target time period information of the image to be retrieved, including: determining the start timestamp and the end timestamp of the target time period information; setting the value of the time bit of the image whose timestamp information is within the time range composed of the start timestamp and the end timestamp in the Nth image group to a first bit value, the first bit value indicating that the corresponding image is a time candidate image, and setting the value of the time bit of the image whose timestamp information is not within the time range composed of the start timestamp and the end timestamp in the Nth image group to a second bit value, the second bit value indicating that the corresponding image is an image to be filtered out.

[0012] In an embodiment of the present application, taking the determination of the time filter mask bit set of the Nth image group as an example, the process of generating the time filter mask bit set includes: determining the start timestamp and the end timestamp of the target time period information; setting the value of the time bit of the image whose timestamp information is within the time range composed of the start timestamp and the end timestamp in the Nth image group to a first bit value (setting bit=1), the first bit value indicating that the corresponding image is a time candidate image; and setting the value of the time bit of the image whose timestamp information is not within the time range composed of the start timestamp and the end timestamp in the Nth image group to a second bit value (setting bit=0), the second bit value indicating that the corresponding image is an image to be filtered out.

[0013] In a possible implementation of the first aspect above, based on the spatial point information corresponding to each image in the Nth image group and the target point information of the image to be retrieved, the spatial filtering mask bit set of the Nth image group is determined, including: calculating the spatial point quotient and the spatial point remainder of the spatial point information corresponding to each image in the Nth image group divided by L, where L is an integer multiple of 8; determining the target bit of the target point information corresponding to each image in the Nth image group based on the spatial point quotient and the spatial point remainder corresponding to each image in the Nth image group; setting the spatial bit value of the image whose target bit value is the first bit value in the Nth image group to the first bit value, wherein the first bit value indicates that the corresponding image is a spatial candidate image, and setting the spatial bit value of the image whose target bit value is the second bit value in the Nth image group to the second bit value, and the second bit value indicates that the corresponding image is an image to be filtered out.

[0014] In an embodiment of the present application, taking determining the spatial filter mask bit set of the Nth image group as an example, the process of generating the spatial filter mask bit set includes: calculating the spatial point quotient and the spatial point remainder of the spatial point information corresponding to each image in the Nth image group divided by L, where L is an integer multiple of 8; determining the target bit of the target point information corresponding to each image in the Nth image group based on the spatial point quotient and the spatial point remainder corresponding to each image in the Nth image group; setting the spatial bit value of the image whose target bit value is the first bit value (bit=1) in the Nth image group to the first bit value (setting bit=1), wherein the first bit value indicates that the corresponding image is a spatial candidate image, and setting the spatial bit value of the image whose target bit value is the second bit value (bit=0) in the Nth image group to the second bit value (setting bit=0), and the second bit value indicates that the corresponding image is an image to be filtered out.

[0015] In a possible implementation of the first aspect above, based on the temporal filter mask bit set of the Nth image group and the spatial filter mask bit set of the Nth image group, the spatiotemporal filter mask bit set of the Nth image group is determined, including: performing AND calculations on the temporal filter mask bit set of the Nth image group and the spatial filter mask bit set of the Nth image group, setting the spatiotemporal bit value of the image in the Nth image group whose temporal bit value and spatial bit value are both first bit values ​​to a first bit value, wherein the first bit value indicates that the corresponding image is a candidate image, and setting the spatiotemporal bit value of the image in the Nth image group whose temporal bit value or spatial bit value is a second bit value to a second bit value, wherein the second bit value indicates that the corresponding image is an image to be filtered out.

[0016] In an embodiment of the present application, taking determining the spatiotemporal filter mask bit set of the Nth image group as an example, the process of generating the spatiotemporal filter mask bit set includes: performing AND calculations on the temporal filter mask bit set of the Nth image group and the spatial filter mask bit set of the Nth image group, setting the spatiotemporal bit value of the image in the Nth image group whose temporal bit value and spatial bit value are both the first bit value (bit=1) to the first bit value (setting bit=1), wherein the first bit value indicates that the corresponding image is a candidate image, and setting the spatiotemporal bit value of the image in the Nth image group whose temporal bit value or spatial bit value is the second bit value (bit=0) to the second bit value (setting bit=0), wherein the second bit value indicates that the corresponding image is an image to be filtered out.

[0017] It can be understood that the use of vector calculation to determine the candidate images further shortens the calculation time and improves the calculation efficiency.

[0018] In a possible implementation of the first aspect, calculating the similarity between the candidate image of the N-1th image group and the image to be retrieved includes: calculating the similarity between the candidate image of the N-1th image group and the image to be retrieved by matrix calculation.

[0019] In the embodiment of the present application, the matrix calculation method can be matrix instruction calculation.

[0020] In a possible implementation of the first aspect above, the similarity between the candidate images of the N-1th image group and the images to be retrieved is calculated by matrix calculation, including: determining image features of the images to be retrieved, and constructing a first matrix based on the image features; constructing a second matrix based on feature vectors corresponding to each image in the N-1th image group; performing matrix multiplication calculation on the second matrix and the first matrix to obtain the similarity between each image in the N-1th image group and the images to be retrieved; and determining the similarity between the candidate images of the N-1th image group and the images to be retrieved based on a spatiotemporal filtering mask bit set of the N-1th image group.

[0021] In an embodiment of the present application, taking the example of performing feature comparison based on image features of the image to be retrieved and feature vectors of the N-1th group of images to calculate the similarity between each image in the N-1th group and the image to be retrieved, a process of performing matrix instruction calculation based on the query feature vector and the feature vector in the image library to obtain the distance value (similarity) of the feature vector in the image library includes: determining the image features of the image to be retrieved, and constructing a first matrix based on the image features; constructing a second matrix based on the feature vectors corresponding to each image in the N-1th image group; performing matrix multiplication calculation on the second matrix and the first matrix to obtain the similarity between each image in the N-1th image group and the image to be retrieved; and determining the similarity between the candidate images of the N-1th image group and the image to be retrieved based on the spatiotemporal filtering mask bit set of the N-1th image group.

[0022] In a possible implementation of the first aspect above, based on the spatiotemporal filter mask bit set of the N-1th image group, determining the similarity between the candidate image of the N-1th image group and the image to be retrieved includes: obtaining the spatiotemporal filter mask bit set of the N-1th image group; retaining the similarity between the image corresponding to the first bit value in the spatiotemporal filter mask bit set of the N-1th image group and the image to be retrieved; replacing the similarity between the image corresponding to the second bit value in the spatiotemporal filter mask bit set of the N-1th image group and the image to be retrieved with the minimum similarity.

[0023] In an embodiment of the present application, based on the spatiotemporal filter mask bit set of the N-1th image group, a process of determining the similarity between the candidate image of the N-1th image group and the image to be retrieved includes: obtaining the spatiotemporal filter mask bit set of the N-1th image group; retaining the similarity between the image corresponding to the first bit value (bit=1) in the spatiotemporal filter mask bit set of the N-1th image group and the image to be retrieved; replacing the similarity between the image corresponding to the second bit value (bit=0) in the spatiotemporal filter mask bit set of the N-1th image group and the image to be retrieved with the minimum similarity (the minimum value of the numerical type corresponding to the distance value or the maximum value of the numerical type corresponding to the distance value).

[0024] In a possible implementation of the first aspect above, the candidate images of the N-2 image group are sorted according to the similarity between each candidate image of the N-2 image group and the image to be retrieved, including: sorting the candidate images of the N-2 image group in descending order according to the order of similarity from large to small, or sorting the candidate images of the N-2 image group in ascending order according to the order of similarity from small to large; wherein the sorting method includes at least one of the following: heap sort, bubble sort, selection sort.

[0025] In the embodiment of the present application, taking the ordering of the images in the N-2th image group whose distance values ​​have been calculated from large to small (or from small to large) according to the distance values ​​as an example, the process of ordering the distance values ​​includes: ordering the candidate images of the N-2th image group in descending order according to the order of similarity from large to small, or ordering the candidate images of the N-2th image group in ascending order according to the order of similarity from small to large. The ordering method may be a heap order; the ordering method may also be a bubble order; the ordering method may also be a selection order.

[0026] In a possible implementation of the first aspect above, the image library includes P image groups, and, in response to a first image retrieval instruction, retrieving a target image from the image library, further comprising: in the P+1th processing cycle: calculating the similarity between the candidate images of the Pth image group and the image to be retrieved, and sorting the candidate images of the P-1th image group according to the similarity between the candidate images of the P-1th image group and the image to be retrieved, wherein P is greater than 2.

[0027] In an embodiment of the present application, in the P+1th processing cycle: the subprocess for calculating the distance value performs feature comparison based on the image features of the image to be retrieved and the feature vectors of the Pth group of images, and calculates the similarity between each image in the Pth group and the image to be retrieved; the sorting subprocess sorts each image in the P-1th image group for which the distance value has been calculated according to the distance value from large to small (or from small to large).

[0028] In a possible implementation of the first aspect above, in response to a first image retrieval instruction, a target image is retrieved from an image library, and the process also includes: in the P+2th processing cycle: sorting the candidate images of the Pth image group according to the similarity between the candidate images of the Pth image group and the image to be retrieved.

[0029] In the embodiment of the present application, in the P+2th processing cycle: the sorting subprocess sorts the images in the Pth image group for which the distance values ​​have been calculated according to the distance values ​​from large to small (or from small to large).

[0030] In a possible implementation of the first aspect above, in response to a first image retrieval instruction, a target image is retrieved from an image library, and the method further includes: after sorting the images in the P image groups in the image library, obtaining the first K images in the descending sorting result as target images, where K is greater than 0; or obtaining the last K images in the ascending sorting result as target images.

[0031] In the embodiment of the present application, after all the P groups of images are sorted, the K images closest to the image to be retrieved are obtained as target images. The distance values ​​are sorted in descending order, and the images corresponding to the first K distance values ​​are obtained as target images, i.e., the retrieval results; the distance values ​​are sorted in ascending order, and the images corresponding to the last K distance values ​​are obtained as target images, i.e., the retrieval results.

[0032] A second aspect of the present application provides a readable medium on which instructions are stored. When the instructions are executed on an electronic device, the electronic device executes any one of the methods in the first aspect.

[0033] The third aspect of the present application provides an electronic device, comprising a memory for storing instructions executed by one or more processors of the electronic device, and a processor, which is one of the processors of the electronic device, for executing any one of the methods in the above-mentioned first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0035] Figure 1a to Figure 1c According to an embodiment of the present application, a schematic diagram of interface changes of a shopping software for retrieving images is shown;

[0036] Figure 2a According to an embodiment of the present application, a schematic diagram of a process of image-based spatiotemporal retrieval is shown;

[0037] Figure 2b According to an embodiment of the present application, a schematic diagram of a detailed process of image-based spatiotemporal retrieval is shown;

[0038] Figure 3a According to an embodiment of the present application, a schematic diagram of grouping images in an image library is shown;

[0039] Figure 3b According to an embodiment of the present application, a schematic diagram of the execution process of each sub-process of an image retrieval process is shown;

[0040] Figure 4 According to an embodiment of the present application, a schematic diagram of an image retrieval device is shown;

[0041] Figure 5 According to an embodiment of the present application, a schematic flow chart of an image retrieval method is shown;

[0042] Figure 6 According to an embodiment of the present application, a schematic diagram of the generation logic of a spatiotemporal filter mask bit set is shown;

[0043] Figure 7 According to an embodiment of the present application, a flow chart of a method for generating a spatiotemporal filtering mask bit set is shown;

[0044] Figure 8 According to an embodiment of the present application, a schematic diagram of a storage format of spatiotemporal attributes is shown;

[0045] Fig. 9 According to an embodiment of the present application, a schematic diagram of a calculation process flow for generating a time filter mask bit set is shown;

[0046] Fig.10 According to an embodiment of the present application, a schematic diagram of a calculation process flow for generating a spatial filter mask bit set is shown;

[0047] Fig.11 According to an embodiment of the present application, a schematic diagram of target point information is shown;

[0048] Fig.12 According to an embodiment of the present application, a schematic diagram of calculating a spatial filter mask bit set is shown;

[0049] Fig.13 According to an embodiment of the present application, a structural schematic diagram of an electronic device is shown. DETAILED DESCRIPTION

[0050] The illustrative embodiments of the present application include, but are not limited to, an image retrieval method, a readable medium, and an electronic device.

[0051] In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and comprehensively described below in conjunction with the accompanying drawings.

[0052] As mentioned above, image retrieval methods can usually combine the timestamp information and spatial point information of the image for auxiliary retrieval, that is, image spatiotemporal retrieval (or image-based spatiotemporal retrieval). Figure 2a Schematic diagram of the process of image-based spatiotemporal retrieval is shown.

[0053] Combination Figure 2aThe process of image-based spatiotemporal retrieval shown in the figure mainly includes: inputting the image features, time period range and target points of the image to be queried (image to be retrieved) into the image-based spatiotemporal retrieval system; the spatiotemporal retrieval system first screens out candidate images from the image library based on timestamp information and spatial point information (spatiotemporal attribute filtering); the spatiotemporal retrieval system then compares the image features of the image to be retrieved with the image features of the candidate images (feature comparison), determines the K candidate images that are closest to the image features of the image to be retrieved as target images, and returns the K target images as retrieval results (query results).

[0054] For example, Figure 2b A schematic diagram showing the detailed process of image-based spatiotemporal retrieval.

[0055] Combination Figure 2b ,The image spatiotemporal retrieval methods mainly include:

[0056] (1) Based on query conditions, such as the target time period information (or target time period) and target point information (or target point sequence) of the image to be retrieved, the images in the image library are filtered to obtain candidate images.

[0057] For example, the target time period information and target point information of the image to be retrieved are compared one by one with the spatiotemporal attribute information 0, spatiotemporal attribute information 1 ... spatiotemporal attribute information n in the spatiotemporal information stored in the image library; if the timestamp (in the target time period information) and the spatial point (in the target point information) both hit the spatiotemporal attribute information in the stored spatiotemporal information, it is determined that the corresponding feature vectors in the stored feature vectors (image features) 0, feature vector 1 ... feature vector n are the filtered feature vectors, i.e., feature vector i, feature vector j ... feature vector k.

[0058] It can be understood that the feature vector corresponds to the image in the image library, and the filtered feature vector is obtained, that is, the filtered image is obtained as a candidate image, so as to further compare the image features with the image to be retrieved to obtain the target image.

[0059] (2) By querying feature vectors, such as image features of the image to be retrieved, the image features of the candidate images are compared to obtain the target image.

[0060] For example, the query feature vector (image feature of the image to be retrieved) is compared and retrieved with the filtered feature vector to obtain the K feature vectors in the filtered feature vector that are closest to the query feature vector, and the image corresponding to the K feature vectors in the image library is determined as the retrieval result, that is, the target image.

[0061] Among them, the process of determining the filtered feature vector by comparing the query conditions with the stored spatiotemporal information one by one can be implemented by a central processing unit (CPU) platform; the process of comparing and retrieving the query feature vector with the filtered feature vector to obtain the retrieval result can be implemented by a CPU platform or a graphics processing unit (GPU) platform.

[0062] However, current image retrieval methods usually select the target image from the candidate images after screening all candidate images one by one. If the number of images in the image library is very large, the entire retrieval process will take a lot of time.

[0063] In view of this, an embodiment of the present application provides an image retrieval method, which uses an asynchronous pipeline method to determine candidate images, calculate the distance value (similarity) between the image features of the images in the image library and the image features of the image to be retrieved, and sort the similarities, and finally obtain K images that are closest to the image features of the image to be retrieved. For example, the asynchronous pipeline method can be to divide the image retrieval process into multiple sub-processes, each of which can be carried out simultaneously, without waiting for the previous sub-process to be executed before executing the current sub-process.

[0064] For example, the process of image retrieval (ie, retrieving a target image that matches the image to be retrieved from an image library) is divided into three sub-processes: determining candidate images, calculating distance values, and sorting.

[0065] Among them, the subprocess for determining candidate images is responsible for performing vector calculations on the target time period information and target point information of the image to be retrieved and the spatiotemporal attributes of the images of each image group in the image library, marking candidate images and filtering images, that is, selecting candidate images from each image group in the image library.

[0066] The subprocess for calculating the distance value is responsible for performing feature comparison based on the image features of the image to be retrieved and the feature vectors of the images of each image group in the image library, and calculating the distance value (similarity) between the images of each image group in the image library and the image to be retrieved.

[0067] The sorting subprocess is responsible for sorting the images of each image group whose similarities have been calculated in the order of similarity from large to small (or from small to large).

[0068] For example, the n images in the image library are divided into P groups, each with Q images. Figure 3aAs shown, the images in the gallery are divided into the first group of images, the second group of images, ... the Nth group of images ... the Pth group of images. The spatiotemporal attributes corresponding to each group of images are the spatiotemporal attributes of the first group of images, the spatiotemporal attributes of the second group of images, ... the spatiotemporal attributes of the Nth group of images ... the spatiotemporal attributes of the Pth group of images. The image features corresponding to each group of images are the feature vectors of the first group of images, the feature vectors of the second group of images, ... the feature vectors of the Nth group of images ... the feature vectors of the Pth group of images.

[0069] A first image retrieval instruction is detected, for example, a user clicks Figure 1b The operation of the "Photo Shopping" control 103 shown is to retrieve a target image matching the image to be retrieved from the image library in response to the first image retrieval instruction.

[0070] The execution process of each subprocess of retrieving a target image matching the image to be retrieved from the image library can be referred to Figure 3b Schematic diagram of the execution process of each sub-process shown:

[0071] The first processing cycle: The subprocess for determining candidate images performs vector calculation based on the target time period information and target point information of the image to be retrieved and the spatiotemporal attributes of the first group of images, and marks the candidate images and filtered images in the first group of images.

[0072] The second processing cycle: the subprocess for determining the candidate images performs vector calculation based on the target time period information and target point information of the image to be retrieved and the spatiotemporal attributes of the second group of images, and marks the candidate images and filtered images in the second group of images; the subprocess for calculating the distance value performs feature comparison based on the image features of the image to be retrieved and the feature vectors of the first group of images, and calculates the similarity between each image in the first group and the image to be retrieved.

[0073] The third processing cycle: the subprocess for determining the candidate images performs vector calculation based on the target time period information and target point information of the image to be retrieved and the spatiotemporal attributes of the third group of images, and marks the candidate images and filtered images in the third group of images; the subprocess for calculating the distance value performs feature comparison based on the image features of the image to be retrieved and the feature vectors of the second group of images, and calculates the similarity between each image in the second group and the image to be retrieved; the subprocess for sorting sorts each image in the first image group for which the distance value has been calculated from large to small (or from small to large) according to the distance value.

[0074] …

[0075] The Nth processing cycle: the subprocess for determining the candidate images performs vector calculation based on the target time period information and target point information of the image to be retrieved and the spatiotemporal attributes of the Nth group of images, and marks the candidate images and filtered images in the Nth group of images; the subprocess for calculating the distance value performs feature comparison based on the image features of the image to be retrieved and the feature vectors of the N-1th group of images, and calculates the similarity between each image in the N-1th group and the image to be retrieved; the subprocess for sorting sorts each image in the N-2th image group for which the distance value has been calculated from large to small (or small to large) according to the distance value.

[0076] …

[0077] The P+1th processing cycle: the subprocess for calculating the distance value performs feature comparison based on the image features of the image to be retrieved and the feature vectors of the Pth group of images, and calculates the similarity between each image in the Pth group and the image to be retrieved; the subprocess for sorting sorts each image in the P-1th image group for which the distance value has been calculated according to the distance value from large to small (or from small to large).

[0078] The P+2th processing cycle: the sorting subprocess sorts the images in the Pth image group for which the distance values ​​have been calculated according to the distance values ​​from large to small (or from small to large).

[0079] After all P groups of images are sorted, K images closest to the image to be retrieved are obtained as target images.

[0080] It can be understood that N is an integer greater than 2, P is an integer greater than 2, and K is an integer greater than 0.

[0081] It can be understood that the image retrieval method provided in the embodiment of the present application adopts an asynchronous pipeline method to perform retrieval, which can shorten the retrieval time. At the same time, the candidate images are determined by vector calculation, which further shortens the calculation time and improves the calculation efficiency.

[0082] The retrieval method provided in the embodiment of the present application can be applied to electronic devices such as mobile phones, smart large screens, tablet computers, laptop computers, netbooks, personal digital assistants (PDAs), etc. The exemplary embodiments of electronic devices include but are not limited to those equipped with Or electronic devices with other operating systems. The embodiments of the present application do not impose any restrictions on the specific types of electronic devices.

[0083] The retrieval method provided in the embodiments of the present application can be applied to scenarios such as image library retrieval management systems, video service retrieval systems, and video surveillance retrieval systems.

[0084] In order to better understand the technical solution of the embodiment of the present application, the image retrieval device provided by the embodiment of the present application is described in detail below. It can be understood that the electronic device mentioned in the present application may include the image retrieval device mentioned in the present application.

[0085] Figure 4 According to an embodiment of the present application, a schematic diagram of an image retrieval device 10 is shown.

[0086] like Figure 4 As shown, the image retrieval device 10 includes: a vector calculation engine 11, a matrix calculation engine 12 and a general calculation engine 13.

[0087] The vector calculation engine 11 is used to perform vector instruction calculation according to the attribute filtering query conditions (i.e., the target time period information and target point information of the image to be retrieved) and the spatiotemporal information in the image library (i.e., the timestamp information and spatial point information of each image), and generate a spatiotemporal filtering mask bit set (Mask Bit Set), so as to facilitate further processing of the distance value (similarity) calculated subsequently. That is, based on the comparison between the target time period information and target point information of the image to be retrieved and the timestamp information and spatial point information of each image in the image library, the candidate images in the image library are marked.

[0088] It can be understood that in the spatiotemporal filtering mask bit set, the value of the spatiotemporal bit is the first bit value (bit=1), indicating that the corresponding image is a candidate image, that is, the corresponding image is retained; the value of the spatiotemporal bit is the second bit value (bit=0), indicating that the corresponding image is an image to be filtered out, that is, the corresponding image is filtered.

[0089] It can be understood that vector instruction calculation can process multiple data at the same time, and using vector instruction calculation can improve computing efficiency.

[0090] The matrix calculation engine 12 is used to construct a first matrix (left matrix) based on the query feature vector (image feature of the image to be retrieved), and to construct a second matrix (right matrix) based on the feature vector corresponding to each image in each image group in the image library, and to perform matrix instruction calculations on the left matrix and the right matrix, and the calculation result is a distance value (Dist0, Dist1, Dist2, ..., Distn-1, Distn). At the same time, the calculated distance value is replaced using the spatiotemporal filter mask bit set generated by the vector calculation engine 11. For example, when bit = 1, the corresponding distance value is retained; when bit = 0, the corresponding distance value is replaced with the minimum value (Min) or maximum value (Max) of the numerical type corresponding to the distance value.

[0091] It can be understood that if the larger the distance value is, the closer the image is to the image to be retrieved, then the corresponding distance value is replaced with the minimum value of the numerical type corresponding to the distance value; if the smaller the distance value is, the closer the image is to the image to be retrieved, then the corresponding distance value is replaced with the maximum value of the numerical type corresponding to the distance value. This allows the filtered images to be sorted at the front or back when the distance values ​​are sorted later.

[0092] It can be understood that matrix instruction calculation is matrix multiplication calculation.

[0093] The general calculation engine 13 is used to sort the distance values ​​replaced by the matrix calculation engine 12, and finally obtain the K images (Top K) closest to the image to be retrieved as the retrieval result (target image).

[0094] It can be understood that in some embodiments, the sorting method can be a heap sort; in other embodiments, the sorting method can also be a bubble sort; in other embodiments, the sorting method can also be a selection sort; this application does not make any specific restrictions on the sorting method.

[0095] The following uses the scenario of a video surveillance retrieval system as an example to introduce in detail the image retrieval method provided in the embodiment of the present application.

[0096] Figure 5 According to an embodiment of the present application, a schematic diagram of a process of an image retrieval method is shown. It can be understood that Figure 5 Each step of the process shown in the figure is performed by the image retrieval device 10. Figure 5 The execution entities of each step will not be described repeatedly in the steps of the process shown. Figure 5 As shown, the process includes but is not limited to the following steps:

[0097] S501: Perform vector instruction calculation according to the attribute filtering query condition and the spatiotemporal information in the image library to generate a spatiotemporal filtering mask bit set.

[0098] In some embodiments, the filtering query conditions are target time period information and target point information of the image to be retrieved.

[0099] It can be understood that the filtering query condition can be a query condition input by the user. For example, the image to be retrieved is a black business car 40, and the user wants to find the black business car 40 on Road A between 8:00 a.m. and 10:00 a.m. on August 1, 2023 in the image library. Road A includes cameras with identification numbers 010, 011, and 012. Then the filtering query condition input by the user is <2023.08.01 8:00, 2023.08.01 10:00; 010, 011, 012>, where "2023.08.01 8:00, 2023.08.01 10:00" represents the target time period information, and "010, 011, 012" represents the target point information.

[0100] In some embodiments, the generation logic of the spatiotemporal filter mask bit set can refer to Figure 6 Schematic diagram of the generation logic of the spatiotemporal filter mask bit set shown in FIG. Figure 6 As shown, the generation logic of the spatiotemporal filter mask bit set is as follows:

[0101] IF the feature vector timestamp satisfies the time condition AND the feature vector spatial point satisfies the spatial condition

[0102] Return true

[0103] ELSE

[0104] Return false

[0105] That is, if (IF) the timestamp information corresponding to the feature vector of the image in the image library meets the time condition (within the target time period of the image to be retrieved) and the spatial point information corresponding to the feature vector of the image in the image library meets the spatial condition (belongs to the target point information of the image to be retrieved), then the value of the spatiotemporal bit corresponding to the image in the image library that meets the condition is set to 1 (bit=1), that is, true (true) is returned. Otherwise (ELSE), the value of the spatiotemporal bit corresponding to the image in the image library that does not meet the condition is set to 0 (bit=0), that is, false (false) is returned. For example, the following can be obtained: Figure 6 The filter mask bit set shown in (Filter Mask Bit Set): 0 1 0…0 10 1 1.

[0106] It can be understood that in the spatiotemporal filtering mask bit set, the value of the spatiotemporal bit is the first bit value (bit=1), indicating that the corresponding image is a candidate image, that is, the corresponding image is retained; the value of the spatiotemporal bit is the second bit value (bit=0), indicating that the corresponding image is an image to be filtered out, that is, the corresponding image is filtered.

[0107] In some embodiments, taking the example of a subprocess for determining candidate images performing vector calculation based on target time period information and target point information of the image to be retrieved and the spatiotemporal attributes of the Nth group of images, and marking the candidate images and the filtered images in the Nth group of images, a method for calculating and generating a spatiotemporal filter mask bit set according to vector instructions includes: determining the target time period information and target point information of the image to be retrieved; determining a time filter mask bit set for the Nth image group based on the timestamp information corresponding to each image in the Nth image group and the target time period information of the image to be retrieved, wherein each time bit in the time filter mask bit set is The value of indicates whether the corresponding image is a temporal candidate image, and based on the spatial point information corresponding to each image in the Nth image group and the target point information of the image to be retrieved, the spatial filter mask bit set of the Nth image group is determined, wherein the value of each spatial bit in the spatial filter mask bit set indicates whether the corresponding image is a spatial candidate image; based on the temporal filter mask bit set of the Nth image group and the spatial filter mask bit set of the Nth image group, the spatiotemporal filter mask bit set of the Nth image group is determined, wherein the value of the spatiotemporal bit in the spatiotemporal filter mask bit set indicates whether the corresponding image is a candidate image.

[0108] It can be understood that vector instruction calculation can process multiple data at the same time, and using vector instruction calculation can improve operation efficiency. The specific method of generating a spatiotemporal filter mask bit set according to vector instruction calculation will be described in detail below and will not be repeated here.

[0109] S502: Perform matrix instruction calculation based on the query feature vector and the feature vector in the image library to obtain the distance value of the feature vector in the image library.

[0110] In some embodiments, a left matrix is ​​constructed based on the query feature vector (image features of the image to be retrieved), and a right matrix is ​​constructed based on the feature vectors (image features) in the image library. Matrix instruction calculations are performed on the left matrix and the right matrix, and the calculation results are the distance values ​​(Dist0, Dist1, Dist2, ..., Distn-1, Distn) of the feature vectors in the image library.

[0111] It can be understood that matrix instruction calculation is matrix multiplication calculation.

[0112] In some embodiments, after obtaining the distance value of the feature vector in the image library, the calculated distance value is replaced based on the value of the spatiotemporal bit in the generated spatiotemporal filter mask bit set.

[0113] For example, when bit=1, the corresponding distance value is retained; when bit=0, the corresponding distance value is replaced with the minimum value (Min) of the numerical type corresponding to the distance value, so that after the distance values ​​are subsequently sorted in descending order, all the filtered images are placed at the end, or, after the distance values ​​are subsequently sorted in ascending order, all the filtered images are placed at the front.

[0114] Or for example, when bit=1, the corresponding distance value is retained; when bit=0, the corresponding distance value is replaced with the maximum value (Max) of the numerical type corresponding to the distance value, so that after the distance values ​​are subsequently sorted in ascending order, all the filtered images are placed at the end, or, after the distance values ​​are subsequently sorted in descending order, all the filtered images are placed at the front.

[0115] It can be understood that if the larger the distance value, the closer the image is to the image to be retrieved, the corresponding distance value will be replaced with the minimum value of the numerical type corresponding to the distance value; if the smaller the distance value, the closer the image is to the image to be retrieved, the corresponding distance value will be replaced with the maximum value of the numerical type corresponding to the distance value.

[0116] In some embodiments, taking the example of performing feature comparison based on image features of the image to be retrieved and feature vectors of the N-1th group of images to calculate the similarity between each image in the N-1th group and the image to be retrieved, a process of performing matrix instruction calculation based on the query feature vector and the feature vector in the image library to obtain the distance value (similarity) of the feature vector in the image library includes: determining the image features of the image to be retrieved, and constructing a first matrix based on the image features; constructing a second matrix based on the feature vectors corresponding to each image in the N-1th image group; performing matrix multiplication calculation on the second matrix and the first matrix to obtain the similarity between each image in the N-1th image group and the image to be retrieved; and determining the similarity between the candidate images of the N-1th image group and the image to be retrieved based on the spatiotemporal filtering mask bit set of the N-1th image group.

[0117] In some embodiments, based on the spatiotemporal filter mask bit set of the N-1th image group, the process of determining the similarity between the candidate image of the N-1th image group and the image to be retrieved includes: obtaining the spatiotemporal filter mask bit set of the N-1th image group; retaining the similarity between the image corresponding to the first bit value (bit=1) in the spatiotemporal filter mask bit set of the N-1th image group and the image to be retrieved; replacing the similarity between the image corresponding to the second bit value (bit=0) in the spatiotemporal filter mask bit set of the N-1th image group and the image to be retrieved with the minimum similarity (the minimum value of the numerical type corresponding to the distance value or the maximum value of the numerical type corresponding to the distance value).

[0118] S503: Sort the distance values ​​and obtain K images closest to the image to be retrieved as retrieval results.

[0119] In some embodiments, the distance values ​​may be sorted by performing a binary heap sort.

[0120] In some embodiments, if in step S502, when bit=0, the corresponding distance value is replaced with the minimum value of the numerical type corresponding to the distance value, the distance values ​​are sorted in descending order, and the images corresponding to the first K distance values ​​are obtained as target images, that is, the retrieval results; or, the distance values ​​are sorted in ascending order, and the images corresponding to the last K distance values ​​are obtained as target images, that is, the retrieval results.

[0121] In some embodiments, if in step S502, when bit=0, the corresponding distance value is replaced with the maximum value of the numerical type corresponding to the distance value, the distance values ​​are sorted in ascending order, and the images corresponding to the first K distance values ​​are obtained as target images, that is, the retrieval results; or, the distance values ​​are sorted in descending order, and the images corresponding to the last K distance values ​​are obtained as target images, that is, the retrieval results.

[0122] In some embodiments, taking the example of sorting the images in the N-2th image group whose distance values ​​have been calculated from large to small (or from small to large) according to the distance values, the process of sorting the distance values ​​includes: sorting the candidate images of the N-2th image group in descending order according to the order of similarity from large to small, or sorting the candidate images of the N-2th image group in ascending order according to the order of similarity from small to large.

[0123] It can be understood that the above steps S501 to S503 are asynchronously executed in a pipeline. For example, the image retrieval process is divided into three sub-processes: determining candidate images, calculating distance values, and sorting.

[0124] Among them, the subprocess for determining candidate images is responsible for performing vector calculations on the target time period information and target point information of the image to be retrieved and the spatiotemporal attributes of the images of each image group in the image library, marking candidate images and filtering images, that is, selecting candidate images from each image group in the image library.

[0125] The subprocess for calculating the distance value is responsible for performing feature comparison based on the image features of the image to be retrieved and the feature vectors of the images of each image group in the first image library, and calculating the distance value (similarity) between each image in the image library and the image to be retrieved.

[0126] The sorting subprocess is responsible for sorting the images of each image group whose similarities have been calculated in the order of similarity from large to small (or from small to large).

[0127] For example, the n images in the image library are divided into P groups, each with Q images. Figure 3aAs shown, the images in the gallery are divided into the first group of images, the second group of images, ... the Nth group of images ... the Pth group of images. The spatiotemporal attributes corresponding to each group of images are the spatiotemporal attributes of the first group of images, the spatiotemporal attributes of the second group of images, ... the spatiotemporal attributes of the Nth group of images ... the spatiotemporal attributes of the Pth group of images. The image features corresponding to each group of images are the feature vectors of the first group of images, the feature vectors of the second group of images, ... the feature vectors of the Nth group of images ... the feature vectors of the Pth group of images.

[0128] A first image retrieval instruction is detected, for example, a user clicks Figure 1b The operation of the "Photo Shopping" control 103 shown is to retrieve a target image matching the image to be retrieved from the image library in response to the first image retrieval instruction.

[0129] The execution process of each subprocess of retrieving a target image matching the image to be retrieved from the image library can be referred to Figure 3b Schematic diagram of the execution process of each sub-process shown:

[0130] The first processing cycle: The subprocess for determining candidate images performs vector calculation based on the target time period information and target point information of the image to be retrieved and the spatiotemporal attributes of the first group of images, and marks the candidate images and filtered images in the first group of images.

[0131] The second processing cycle: the subprocess for determining the candidate images performs vector calculation based on the target time period information and target point information of the image to be retrieved and the spatiotemporal attributes of the second group of images, and marks the candidate images and filtered images in the second group of images; the subprocess for calculating the distance value performs feature comparison based on the image features of the image to be retrieved and the feature vectors of the first group of images, and calculates the similarity between each image in the first group and the image to be retrieved.

[0132] The third processing cycle: the subprocess for determining the candidate images performs vector calculation based on the target time period information and target point information of the image to be retrieved and the spatiotemporal attributes of the third group of images, and marks the candidate images and filtered images in the third group of images; the subprocess for calculating the distance value performs feature comparison based on the image features of the image to be retrieved and the feature vectors of the second group of images, and calculates the similarity between each image in the second group and the image to be retrieved; the subprocess for sorting sorts each image in the first image group for which the distance value has been calculated from large to small (or from small to large) according to the distance value.

[0133] …

[0134] The Nth processing cycle: the subprocess for determining the candidate images performs vector calculation based on the target time period information and target point information of the image to be retrieved and the spatiotemporal attributes of the Nth group of images, and marks the candidate images and filtered images in the Nth group of images; the subprocess for calculating the distance value performs feature comparison based on the image features of the image to be retrieved and the feature vectors of the N-1th group of images, and calculates the similarity between each image in the N-1th group and the image to be retrieved; the subprocess for sorting sorts each image in the N-2th image group for which the distance value has been calculated from large to small (or small to large) according to the distance value.

[0135] …

[0136] The P+1th processing cycle: the subprocess for calculating the distance value performs feature comparison based on the image features of the image to be retrieved and the feature vectors of the Pth group of images, and calculates the similarity between each image in the Pth group and the image to be retrieved; the subprocess for sorting sorts each image in the P-1th image group for which the distance value has been calculated according to the distance value from large to small (or from small to large).

[0137] The P+2th processing cycle: the sorting subprocess sorts the images in the Pth image group for which the distance values ​​have been calculated according to the distance values ​​from large to small (or from small to large).

[0138] After all P groups of images are sorted, K images closest to the image to be retrieved are obtained as target images.

[0139] In order to make the technical solution provided by the embodiment of the present application clearer, the method for generating a spatiotemporal filter mask bit set according to vector instruction calculation in the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0140] Figure 7 According to an embodiment of the present application, a flow chart of a method for generating a spatiotemporal filtering mask bit set is shown. It can be understood that Figure 7 Each step of the process shown in the figure is performed by the image retrieval device 10. Figure 7 The execution entities of each step will not be described repeatedly in the steps of the process shown. Figure 7 As shown, the process includes but is not limited to the following steps:

[0141] S701: Generate a time filter mask bit set based on timestamp information.

[0142] In some embodiments, the image retrieval device 10 obtains target time period information of the image to be retrieved and timestamp information of the images in the image library, and generates a time filter mask bit set based on the target time period information of the image to be retrieved and the timestamp information of the images in the image library.

[0143] For example, when the timestamp information of an image in the image library belongs to the target time period of the image to be retrieved, that is, the timestamp information of the image in the image library is within the time range composed of the start timestamp and the end timestamp of the target time period information of the image to be retrieved, the image is marked as a candidate image, such as setting the value of the time bit of the image to the first bit value (setting bit=1); otherwise, the image is marked as a filtered image, such as setting the value of the time bit of the image to the second bit value (setting bit=0).

[0144] In some embodiments, the storage format of the spatiotemporal attributes (timestamp information and spatial point information) in the image library in the memory can refer to Figure 8 The storage format diagram is shown in Figure 1. Figure 8 As shown, for each image (each image's feature vector) feature vector 0, feature vector 1, ... feature vector M, the corresponding timestamp information and spatial point (token id) information are stored. Among them, the token id information includes the spatial point quotient value (token id quotient value) and the spatial point remainder (token id remainder).

[0145] It can be understood that the timestamp information is represented by a 32-bit integer (int32). The token id quotient is represented by a 32-bit integer (int32). In some embodiments, the stored value is the quotient of the token id value divided by 8 and multiplied by 2, that is, the quotient is shifted to the left by one bit to facilitate the subsequent generation of the time filter mask bit set. The token id remainder is represented by an 8-bit integer (int8). In some embodiments, the stored value is the value of the remainder after 1 is left-shifted by the token id value divided by 8. For example, if the remainder is 4, then 1 is in the 4th position; then a fixed value of 64 is stored to facilitate the subsequent generation of the space filter mask bit set.

[0146] It can be understood that in some other embodiments, the token id quotient and the token id remainder can also be obtained by dividing by L, where L is an integer multiple of 8.

[0147] In some embodiments, taking determining the time filter mask bit set of the Nth image group as an example, the process of generating the time filter mask bit set includes: determining the start timestamp and the end timestamp of the target time period information; setting the value of the time bit of the image whose timestamp information is within the time range composed of the start timestamp and the end timestamp in the Nth image group to a first bit value (setting bit=1), the first bit value indicating that the corresponding image is a time candidate image; and setting the value of the time bit of the image whose timestamp information is not within the time range composed of the start timestamp and the end timestamp in the Nth image group to a second bit value (setting bit=0), the second bit value indicating that the corresponding image is an image to be filtered out.

[0148] For example, reference Fig. 9 The calculation flow chart shown in the figure, the process of generating the time filter mask bit set includes:

[0149] (1) Perform a vector addition operation (vadds) on the feature vector timestamp (a vector consisting of the timestamp information of a group of images in the image library) and the start timestamp of the target time period information of the image to be retrieved, and obtain result 1.

[0150] It can be understood that vadds adds a same scalar value to each element of a vector, that is, adds the starting timestamp (negative) to each timestamp information in the feature vector timestamp, that is, compares (subtracts) with the starting timestamp, so as to further determine the time information in the feature vector timestamp that is greater than the starting timestamp.

[0151] It can be understood that the result 1 is a vector, each element of which is the difference between each timestamp information in the feature vector timestamp and the start timestamp.

[0152] (2) Perform a vector addition operation (vadds) on the feature vector timestamp (a vector consisting of the timestamp information of a group of images in the image library) and the end timestamp of the target time period information of the image to be retrieved, and obtain result 2.

[0153] It can be understood that vadds adds the end timestamp (negative) to each timestamp information in the feature vector timestamp, that is, compares (subtracts) it with the end timestamp, so as to further determine the time information in the feature vector timestamp that is less than the end timestamp.

[0154] It can be understood that the result 2 is a vector, each element of which is the difference between each timestamp information in the feature vector timestamp and the end timestamp.

[0155] (3) Perform a vector type conversion operation (vconv) on result 1 to obtain result 3.

[0156] It can be understood that vconv performs type conversion on each element of a vector, that is, performs type conversion on each element in result 1, that is, the difference between each timestamp information in the feature vector timestamp and the start timestamp, for example, from a 32-bit integer (int32) to a 16-bit floating point (float16).

[0157] (4) Perform a vector type conversion operation (vconv) on result 2 to obtain result 4.

[0158] It can be understood that vconv performs type conversion on each element in result 2, that is, the difference between each timestamp information in the feature vector timestamp and the end timestamp, for example, from a 32-bit integer (int32) to a 16-bit floating point (float16).

[0159] (5) Perform a vector size determination operation (vcmpvs_ge) on result 3 to obtain result 5.

[0160] It can be understood that vcmpvs_ge determines whether the corresponding value of each element in a vector is greater than or equal to a specific scalar value. If so, the corresponding output bit = 1, otherwise the output bit = 0. That is, each element in result 3 is compared with a specific scalar (for example, 0). If the corresponding element is greater than or equal to the scalar, the corresponding output bit = 1, that is, the corresponding timestamp information in the feature vector timestamp is greater than or equal to the start timestamp; if the corresponding element is less than the scalar, the corresponding output bit = 0.

[0161] (6) Perform a vector size determination operation (vcmpvs_le) on result 4 to obtain result 6.

[0162] It can be understood that vcmpvs_le determines whether the corresponding value of each element in a vector is less than or equal to a specific scalar value. If so, the corresponding output bit = 1, otherwise the output bit = 0. That is, each element in result 3 is compared with a specific scalar (for example, 0). If the corresponding element is less than or equal to the scalar, the corresponding output bit = 1, that is, the corresponding timestamp information in the feature vector timestamp is less than or equal to the end timestamp; if the corresponding element is greater than the scalar, the corresponding output bit = 0.

[0163] (7) Perform a vector AND operation (vand) on result 5 and result 6 to obtain result 7.

[0164] It can be understood that vand performs a bitwise AND operation on two vectors according to the bit position. That is, if the bit in result 5 and result 6 is 1, the bit in result 7 is also 1, otherwise it is 0. In other words, the timestamp information corresponding to the feature vector timestamp corresponding to bit=1 in result 7 is greater than or equal to the start timestamp and less than or equal to the end timestamp, that is, the query condition is met, and the images within the time range composed of the start timestamp and the end timestamp of the target time period information of the image to be retrieved are marked as 1.

[0165] S702: Generate a spatial filter mask bit set based on the spatial point information.

[0166] In some embodiments, the image retrieval device 10 obtains the target point information of the image to be retrieved and the spatial point information of the images in the image library, and generates a spatial filtering mask bit set based on the target point information of the image to be retrieved and the spatial point information of the images in the image library.

[0167] In some embodiments, taking determining the spatial filter mask bit set of the Nth image group as an example, the process of generating the spatial filter mask bit set includes: calculating the spatial point quotient and the spatial point remainder of the spatial point information corresponding to each image in the Nth image group divided by L, where L is an integer multiple of 8; determining the target bit of the target point information corresponding to each image in the Nth image group based on the spatial point quotient and the spatial point remainder corresponding to each image in the Nth image group; setting the spatial bit value of the image whose target bit value is the first bit value (bit=1) in the Nth image group to the first bit value (setting bit=1), wherein the first bit value indicates that the corresponding image is a spatial candidate image, and setting the spatial bit value of the image whose target bit value is the second bit value (bit=0) in the Nth image group to the second bit value (setting bit=0), and the second bit value indicates that the corresponding image is an image to be filtered out.

[0168] For example, reference Fig.10 The calculation flow chart shown in the figure, the process of generating the spatial filter mask bit set includes:

[0169] (1) Perform a vector index operation (vgather) on the token id quotient (tid quotient*2) and the target point information of the image to be retrieved (tid query set) to obtain result 8.

[0170] It can be understood that vgather inputs index information (i.e., random offset) for a continuous vector, takes out the corresponding value according to the index information, and then stores it in the target address in sequence. That is, the value corresponding to the target point information of the image to be retrieved (the target bit corresponding to the target point information) is taken out from the token id quotient value, and then stored in the target address in sequence, and the result 8 is obtained.

[0171] For example, assume that the tid query set is represented by bits, and each bit represents the token id of the corresponding image in the image library. For example, the tid query set of the user query is Token ID = {0, 1, 2, 8, 9, 10}, and the maximum number of Token IDs in the image library is 1000. Then an array consisting of uint8[1000 / 8] bytes is used to represent the target point information in the image library. Fig.11As shown, the low bit of the byte of the low address is 1100000, indicating that the 0th, 1st and 2nd token ids are in the tid query set; the high bit of the byte of the low address is 11100000, indicating that the 8th, 9th and 10th token ids are in the tid query set; the low bit of the byte of the high address is 0000000, and the high bit of the byte is 00000000. It can be understood that except for the target bit values ​​corresponding to the 0th, 1st, 2nd, 8th, 9th and 10th token ids in the tid query set, which are the first bit values ​​(the bit value is 1), the rest of the bits are 0.

[0172] It can be understood that the result 8 is the byte corresponding to the token id of the image in the image library in the tid query set.

[0173] It can be understood that the token id quotient value corresponds to the number of digits in the byte. For example, if the token id quotient value (the quotient value divided by 8) of the 10th token id is 1, it means that the 10th token id corresponds to a certain bit in the 1st byte. The specific number of digits can be determined by the token id remainder. For example, if the token id quotient value (the quotient value divided by 8) of the 10th token id is 1 and the token id remainder is 2, it means that the 10th token id corresponds to the 2nd bit in the 1st byte (the lowest bit is the 0th bit). In other words, if the 2nd bit in the 1st byte in the tid query set is 1, it means that the 10th token id is in the tid query set.

[0174] (2) Perform a vector AND operation (vand) on the result 8 and the token id remainder (tid remainder) to obtain the result 9.

[0175] It can be understood that the result 8 is the byte corresponding to the token id of the image in the image library in the tid query set. By performing a vector AND operation on the result 8 and the token id remainder, it can be determined whether the token id of the image in the image library hits the corresponding token id in the tid query set.

[0176] For example, the token id quotient (the quotient when divided by 8) of the 10th token id is 1, and the token id remainder is 2, which means that the 10th token id corresponds to the 2nd bit in the 1st byte (the lowest bit is the 0th bit). If the 2nd bit in the 1st byte in the tid query set is 1, the result of the bitwise AND is 1, which means that the 10th token id is in the tid query set. If the 2nd bit in the 1st byte in the tid query set is 0, the result of the bitwise AND is 0, which means that the 10th token id is not in the tid query set.

[0177] (3) Perform a vector size determination operation (vcmpvs_gt) on the result 9 and the 16-bit floating point number 2.0 to obtain the result 10.

[0178] It can be understood that vcmpvs_gt determines whether the corresponding value of each element in a vector is greater than a specific scalar value. If so, the corresponding output bit = 1, otherwise the output bit = 0. That is, each element in the result 9 is compared with a specific scalar (such as a floating point number 2.0 or an integer number 0). If the corresponding element is greater than the scalar, the value of the corresponding spatial bit is set to the first bit value (corresponding to output bit = 1); if the corresponding element is less than or equal to the scalar, the value of the corresponding spatial bit is set to the second bit value (corresponding to output bit = 0).

[0179] For example, if the Token ID corresponding to the first feature vector in the image library is 4, the Token ID corresponding to the second feature vector is 9, and the tid query set is Token ID = {0, 1, 2, 8, 9, 10}, then the Token ID corresponding to the first feature vector is not hit, but the Token ID corresponding to the second feature vector is hit.

[0180] like Fig.12 As shown, according to Fig.10 In the calculation process shown, after the vgather operation, the first two bytes should be 1110000011100000. That is, the bit values ​​corresponding to the 0th, 1st, 2nd, 8th, 9th and 10th tokenid in the tid query set are 1, and the rest of the bits are 0.

[0181] It can be understood that the token id quotient value corresponding to the first feature vector is 0, and the token id remainder is 4, which corresponds to the 4th bit in the 0th byte of the tid query set; the token id quotient value corresponding to the second feature vector is 1, and the token id remainder is 1, which corresponds to the 1st bit in the 1st byte of the tid query set. The result of the vgather operation is the 0th byte and the 1st byte in the tid query set.

[0182] The result after the vgather operation is bitwise ANDed with the corresponding remainder, that is, bitwise ANDed with 00001000 and 01000000, to obtain the final result. It can be obtained that the result of the first feature vector is 0, and the result of the second feature vector is 1, that is, the Token ID=4 corresponding to the first feature vector is not hit, while the Token ID=9 corresponding to the second feature vector is hit.

[0183] S703: Generate a spatiotemporal filter mask bit set based on the temporal filter mask bit set and the spatial filter mask bit set.

[0184] In some embodiments, the image retrieval device 10 performs a bitwise AND operation on the temporal filter mask bit set and the spatial filter mask bit set to obtain a spatiotemporal filter mask bit set. That is, the spatiotemporal bit value corresponding to the image that satisfies both the temporal condition and the spatial condition is set to the first bit value (bit=1).

[0185] In some embodiments, taking determining the spatiotemporal filter mask bit set of the Nth image group as an example, the process of generating the spatiotemporal filter mask bit set includes: performing AND calculations on the temporal filter mask bit set of the Nth image group and the spatial filter mask bit set of the Nth image group, setting the spatiotemporal bit value of the image in the Nth image group whose temporal bit value and spatial bit value are both the first bit value (bit=1) to the first bit value (setting bit=1), wherein the first bit value indicates that the corresponding image is a candidate image, and setting the spatiotemporal bit value of the image in the Nth image group whose temporal bit value or spatial bit value is the second bit value (bit=0) to the second bit value (setting bit=0), wherein the second bit value indicates that the corresponding image is an image to be filtered out.

[0186] In summary, the image retrieval method provided in the embodiment of the present application adopts an asynchronous pipeline method to perform retrieval, which can shorten the retrieval time. At the same time, a vector calculation method is adopted to determine the candidate images, which further shortens the calculation time and improves the calculation efficiency.

[0187] In order to better understand the technical solutions of the embodiments of the present application, the structure of the equipment involved in the present application is introduced below with reference to the accompanying drawings.

[0188] For example, Fig.13 According to an embodiment of the present application, a structural schematic diagram of an electronic device 100 is shown.

[0189] like Fig.13 As shown, the electronic device 100 may include a processor 110, a memory 120, an interface module 130, a power module 140, a wireless communication module 150, a mobile communication module 160, an audio module 170, a sensor module 180, a button 190, a camera 191, and a display screen 192, etc.

[0190] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0191] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors. The processor may be used to execute the image retrieval method mentioned in the present application.

[0192] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0193] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0194] The power module 140 is used to receive charging input from the charger and to supply power to the processor 110 , the memory 120 , the display screen 192 , the camera 191 , and the wireless communication module 150 .

[0195] The wireless communication module 150 can provide wireless communication solutions for application in the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication technology (NFC), infrared technology (IR), etc.

[0196] The mobile communication module 160 can provide wireless communication solutions including 2G / 3G / 4G / 5G etc. applied on the electronic device 100 .

[0197] The electronic device 100 implements the display function through a GPU, a display screen 192, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 192 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0198] The display screen 192 is used to display images, videos, etc. The display screen 192 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Mini-Led, Micro-Led, Micro-oLed, quantum dot light-emitting diodes (QLED), etc. In some embodiments, the display screen can display images to be retrieved and retrieval results, etc.

[0199] The electronic device 100 can realize the shooting function through ISP, camera 191, video codec, GPU, display screen 192 and application processor.

[0200] ISP is used to process the data fed back by camera 191. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 191.

[0201] Camera 191 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the camera can be used to obtain images to be retrieved.

[0202] The memory 120 can be used to store computer executable program codes, which include instructions. The memory 120 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as an image retrieval function, etc.), etc. The data storage area may store data (such as image data, etc.) created during the use of the electronic device 100, etc. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the memory 120, and / or instructions stored in a memory provided in the processor. In some embodiments, the processor executes instructions stored in the memory to implement the image retrieval method mentioned in the present application.

[0203] The electronic device 100 can implement audio functions through the audio module 170, such as music playing, recording, etc.

[0204] The sensor module 180 may include a pressure sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, and the like.

[0205] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0206] The various embodiments of the mechanism disclosed in the present application can be implemented in hardware, software, firmware or a combination of these implementation methods. The embodiments of the present application can be implemented as a computer program or program code executed on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device and at least one output device.

[0207] Program code can be applied to input instructions to perform the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor, a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0208] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0209] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, instructions may be distributed over a network or through other computer-readable media. Therefore, a machine-readable medium may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including, but not limited to, a floppy disk, an optical disk, an optical disk, a magneto-optical disk, a read-only memory (ROM), a random access memory (RAM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic card or an optical card, a flash memory, or a tangible machine-readable memory for transmitting information (e.g., a carrier wave, an infrared signal, a digital signal, etc.) using the Internet in an electrical, optical, acoustic, or other form of propagation signal. Accordingly, machine-readable media include any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a form readable by a machine (eg, a computer).

[0210] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be required. Instead, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of structural or method features in a particular figure does not mean that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0211] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation method of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed by the present application. In addition, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units / modules that are not closely related to solving the technical problems proposed by the present application, which does not mean that there are no other units / modules in the above-mentioned device embodiments.

[0212] It should be noted that in the examples and description of this patent, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one" do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0213] Although the present application has been illustrated and described with reference to certain preferred embodiments thereof, it will be apparent to those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present application.

Claims

1. An image retrieval method, applied to an electronic device, characterized in that: include: detecting a first image retrieval instruction; In response to the first image retrieval instruction, a target image matching the image to be retrieved is retrieved from an image library, the image library includes a plurality of image groups, each image group includes a plurality of images, The step of retrieving a target image from an image library in response to the first image retrieval instruction includes: In the Nth processing cycle: Select a candidate image from the Nth image group, and Calculate the similarity between the candidate image of the N-1th image group and the image to be retrieved, and The candidate images of the N-2th image group are sorted according to the similarity between the candidate images of the N-2th image group and the image to be retrieved, where N is greater than 2.

2. The method according to claim 1, characterized in that The step of selecting a candidate image from the Nth image group includes: Determine the target time period information and target point information of the image to be retrieved; Based on the timestamp information corresponding to each image in the Nth image group and the target time period information of the image to be retrieved, a time filter mask bit set of the Nth image group is determined, wherein the value of each time bit in the time filter mask bit set indicates whether the corresponding image is a time candidate image, and Determine a spatial filter mask bit set of the Nth image group based on the spatial point information corresponding to each image in the Nth image group and the target point information of the image to be retrieved, wherein the value of each spatial bit in the spatial filter mask bit set indicates whether the corresponding image is a spatial candidate image; Based on the temporal filter mask bit set of the Nth image group and the spatial filter mask bit set of the Nth image group, determine the spatiotemporal filter mask bit set of the Nth image group, wherein the value of the spatiotemporal bit in the spatiotemporal filter mask bit set indicates whether the corresponding image is the candidate image.

3. The method according to claim 2, characterized in that The determining of the time filter mask bit set of the Nth image group based on the timestamp information corresponding to each image in the Nth image group and the target time period information of the image to be retrieved includes: Determine the start timestamp and end timestamp of the target time period information; In the Nth image group, the time bit value of the image whose time stamp information is within the time range formed by the start time stamp and the end time stamp is set to a first bit value, wherein the first bit value indicates that the corresponding image is a time candidate image, and In the Nth image group, the time bit value of the image whose time stamp information is not within the time range formed by the start time stamp and the end time stamp is set to a second bit value, and the second bit value indicates that the corresponding image is an image to be filtered out.

4. The method according to claim 3, characterized in that The determining of the spatial filtering mask bit set of the Nth image group based on the spatial point information corresponding to each image in the Nth image group and the target point information of the image to be retrieved includes: Calculate the spatial point quotient and spatial point remainder of the spatial point information corresponding to each image in the Nth image group divided by L, where L is an integer multiple of 8; Determining a target bit position of the target point position information corresponding to each image in the Nth image group based on the spatial point position quotient and the spatial point position remainder corresponding to each image in the Nth image group; The spatial bit value of the image whose target bit value is a first bit value in the Nth image group is set to a first bit value, wherein the first bit value indicates that the corresponding image is a spatial candidate image, and The spatial bit value of the image whose target bit value is the second bit value in the Nth image group is set to the second bit value, and the second bit value indicates that the corresponding image is an image to be filtered out.

5. The method according to claim 4, characterized in that The determining the spatiotemporal filter mask bit set of the Nth image group based on the temporal filter mask bit set of the Nth image group and the spatial filter mask bit set of the Nth image group comprises: An AND calculation is performed on the temporal filter mask bit set of the Nth image group and the spatial filter mask bit set of the Nth image group, and the spatiotemporal bit value of the image in the Nth image group whose temporal bit value and spatial bit value are both the first bit value is set to the first bit value, wherein the first bit value indicates that the corresponding image is a candidate image, and The spatiotemporal bit value of the image whose temporal bit value or spatial bit value in the Nth image group is a second bit value is set to a second bit value, wherein the second bit value indicates that the corresponding image is an image to be filtered out.

6. The method according to claim 2, characterized in that The calculating the similarity between the candidate image of the N-1th image group and the image to be retrieved includes: The similarity between the candidate image of the N-1th image group and the image to be retrieved is calculated by matrix calculation.

7. The method according to claim 6, characterized in that The calculating the similarity between the candidate image of the N-1th image group and the image to be retrieved by means of matrix calculation includes: Determining image features of the image to be retrieved, and constructing a first matrix based on the image features; Constructing a second matrix based on the feature vectors corresponding to each image in the N-1th image group; Performing matrix multiplication calculation on the second matrix and the first matrix to obtain the similarity between each image in the N-1th image group and the image to be retrieved; Based on the spatiotemporal filtering mask bit set of the N-1th image group, the similarity between the candidate image of the N-1th image group and the image to be retrieved is determined.

8. The method according to claim 7, characterized in that The determining, based on the spatiotemporal filtering mask bit set of the N-1th image group, the similarity between the candidate image of the N-1th image group and the image to be retrieved comprises: Obtaining a spatiotemporal filter mask bit set for the N-1th image group; retaining the similarity between the image corresponding to the first bit value in the spatiotemporal filtering mask bit set of the N-1th image group and the image to be retrieved; The similarity between the image corresponding to the second bit value in the spatiotemporal filtering mask bit set of the N-1th image group and the image to be retrieved is replaced with the minimum similarity.

9. The method according to claim 2, characterized in that: The step of sorting the candidate images of the N-2th image group according to the similarity between the candidate images of the N-2th image group and the image to be retrieved comprises: Sort the candidate images of the N-2th image group in descending order according to the similarity from large to small, or, Sort the candidate images of the N-2th image group in ascending order according to the similarity from small to large; The sorting method includes at least one of the following: heap sort, bubble sort, and selection sort.

10. The method according to claim 1, characterized in that The image library includes P image groups, and The method of retrieving a target image from an image library in response to the first image retrieval instruction further includes: In the P+1th processing cycle: Calculate the similarity between the candidate images of the Pth image group and the image to be retrieved, and The candidate images in the P-1th image group are sorted according to the similarity between the candidate images in the P-1th image group and the image to be retrieved, where P is greater than 2.

11. The method according to claim 10, characterized in that The method of retrieving a target image from an image library in response to the first image retrieval instruction further includes: In the P+2th processing cycle: The candidate images of the P th image group are sorted according to the similarity between the candidate images of the P th image group and the image to be retrieved.

12. The method according to claim 11, characterized in that The method of retrieving a target image from an image library in response to the first image retrieval instruction further includes: After sorting the images in the P image groups in the image library, the first K images in the descending sorting result are obtained as target images, where K is greater than 0; or Get the last K images in the ascending sorting result as the target images.

13. A readable medium, characterized in that: The readable medium stores instructions, which, when executed on an electronic device, enable the electronic device to execute the method according to any one of claims 1 to 12.

14. An electronic device, characterized in that: include: a memory for storing instructions to be executed by one or more processors of the electronic device, and The processor is one of the processors of the electronic device, and is used to execute the method according to any one of claims 1 to 12.

Images