A method and apparatus for improving image search speed, image search device

Abstract

This application discloses a method to improve image retrieval speed. The method includes, within a first time interval, calculating the similarity between feature vectors of images to be retrieved and compared in a base database; triggering the sorting of the current similarity results; and calculating at least one next similarity result, such that the calculation of at least one next similarity result is performed in parallel with the sorting. This application can directly pass the current similarity result into the sorting process for sorting, reducing repeated memory read and write operations, significantly reducing the time consumed in the image retrieval process, and improving memory utilization.

Description

Technical Field

[0001] This application relates to the field of image analysis, and in particular, to a method and apparatus for improving image retrieval speed. Background Technology

[0002] In image analysis, it is usually necessary to compare the image to be analyzed with images to be compared. For example, in video analysis, the input image or video data is compared one by one with the image data in the database to obtain the retrieval results.

[0003] Image retrieval mainly involves comparing the features extracted from the image to be retrieved with the features extracted from the image being compared to obtain the most matching image, thereby obtaining the retrieval result.

[0004] In image retrieval applications, a fast response is usually required to output retrieval results, such as gate access based on facial recognition or security authentication based on facial recognition. Summary of the Invention

[0005] This application provides a method to improve image retrieval speed, thereby reducing the time consumed during the image retrieval process.

[0006] This application provides a method for improving image retrieval speed, the method comprising:

[0007] In the first time interval of calculating the similarity between feature vectors based on the image to be retrieved and the feature vectors of the images being compared in the base database, the current similarity result is sorted, and at least the next similarity result is calculated, so that the calculation of at least the next similarity result is performed in parallel with the sorting.

[0008] Based on the sorting results, the comparison images that match the image to be retrieved are determined as the retrieval results.

[0009] Preferably, the triggering of ranking the current similarity results further includes,

[0010] During the first time interval, filtering of the current similarity result is triggered, and the calculation of at least the next similarity result is executed in parallel with the filtering.

[0011] The filtering of similarity results is as follows: based on attribute tags, similarity results that match the attribute tags are retained to obtain the current filtered similarity results;

[0012] The triggering of sorting the similarity results includes,

[0013] In the second time interval after filtering the similarity results, the sorting of the filtered similarity results is triggered.

[0014] Preferably, the step of triggering the sorting of the filtered similarity results during the second time interval of filtering the similarity results further includes,

[0015] This allows the sorting of the filtered similarity results to be performed in parallel with the filtering of at least the next similarity result;

[0016] The calculation of the current similarity result is performed in parallel with the calculation of the next similarity result; and / or

[0017] The filtering of the current similarity result is performed in parallel with the filtering of the next similarity result; and / or

[0018] The sorting of the current similarity results is done in parallel with the sorting of the next similarity results.

[0019] Preferably, the step of triggering the filtering of the current similarity result during the first time interval, and causing the calculation of at least the next similarity result to be executed in parallel with the filtering, includes:

[0020] At the second moment after obtaining the current similarity result, the filtering of the current similarity result is triggered;

[0021] The second moment is taken as the next first moment, and the next similarity result is calculated so that the filtering of the current similarity result and the calculation of the next similarity result are performed in parallel.

[0022] During the second time interval for filtering similarity results, the sorting of the filtered similarity results is triggered, including:

[0023] At the third moment after obtaining the current filtered similarity results, the sorting of the current filtered similarity results is triggered.

[0024] The method of making the sorting of the filtered similarity results and the filtering of at least the next similarity result are performed in parallel includes:

[0025] The third moment is taken as the next second moment, triggering the filtering of the next similarity result, so that the filtering of the next similarity result and the sorting of the current filtered similarity are executed in parallel.

[0026] This application embodiment also provides an apparatus for improving image retrieval speed, the apparatus comprising,

[0027] The similarity result calculation module is used to calculate the similarity between feature vectors based on the image to be retrieved and the feature vectors of the images being compared in the base database, and to trigger the sorting module to sort the data within the first time interval of the current similarity result calculation.

[0028] The sorting module is used to sort the current similarity results according to the triggering of the similarity result calculation module, and to enable the similarity result calculation module to perform the calculation of at least the next similarity result in parallel with the sorting.

[0029] The search results output module determines the comparison image that matches the image to be searched based on the sorting results, and uses it as the search result.

[0030] This application provides an image retrieval device, including a hardware platform. The hardware platform includes a memory and a processor. The memory stores a computer program, and the processor is configured to execute the computer program to implement any of the steps of the method for improving image retrieval speed.

[0031] This application provides a method to improve image retrieval speed. By calculating the similarity between feature vectors based on the image to be retrieved and the feature vectors of the images being compared in the base database within a first time interval, the method triggers the sorting of the current similarity results and performs at least the calculation of the next similarity result. This allows the sorting and calculation of similarity results to be performed in parallel. Compared to existing technologies that only sort similarity results after all similarity results have been calculated, this saves storage space for similarity results. Furthermore, because the sorting and calculation of similarity results are performed in parallel, the current similarity result can be directly passed to the sorting process (operator) for sorting, reducing repeated memory read and write operations, significantly reducing the time consumed in the image retrieval process, and improving memory utilization. Attached Figure Description

[0032] Figure 1 This is a flowchart illustrating a method for improving image retrieval speed according to an embodiment of this application.

[0033] Figure 2 This is a schematic diagram of face image retrieval.

[0034] Figure 3 This is a schematic diagram illustrating feature extraction using deep learning networks during the creation of a face database, and feature extraction of the face image to be retrieved using deep learning networks.

[0035] Figure 4 This is a schematic diagram of similarity calculation based on GEMM.

[0036] Figure 5 This is a flowchart illustrating a high-performance image retrieval speed method according to an embodiment of this application.

[0037] Figure 6 This is a timing diagram illustrating the calculation of similarity results and the triggering of sorting in Example 1.

[0038] Figure 7 This is a schematic diagram comparing the timing performance of this embodiment with that of the prior art.

[0039] Figure 8 This is a schematic diagram comparing the read / write performance of this embodiment with that of existing technologies.

[0040] Figure 9 This is a flowchart illustrating an improved image retrieval method as described in Example 2.

[0041] Figure 10 This is a timing diagram of one embodiment of the present invention.

[0042] Figure 11 This is a schematic diagram of a process for image retrieval in Embodiment 3, in which the desired similarity results are filtered according to specified attributes.

[0043] Figure 12 This is a timing diagram illustrating the calculation of similarity results and the triggering of sorting in Example 3.

[0044] Figure 13 This is a schematic diagram of a process for image retrieval in Embodiment 4, where the desired similarity results are filtered according to specified attributes.

[0045] Figure 14 This is a timing diagram illustrating the calculation of similarity results and the triggering of sorting in Example 4.

[0046] Figure 15 This is a schematic diagram of an apparatus for improving the speed of an image retrieval method according to an embodiment of this application.

[0047] Figure 16 This is a schematic diagram of a device for improving image retrieval speed according to an embodiment of this application.

[0048] Figure 17 This is a schematic diagram of an image retrieval device according to an embodiment of this application. Detailed Implementation

[0049] To make the objectives, technical means, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings.

[0050] The applicant's research found that image retrieval methods mainly consist of two parts: the first part is to extract features using deep learning networks, and the second part is to compare the extracted features. Since the second part involves a large amount of memory movement and computation, it is strongly related to hardware resources (hardware memory, hardware bandwidth). It is not possible to easily improve the overall performance simply by optimizing the performance of deep learning. The most time-consuming part is the calculation of similarity results and the matching of images based on similarity.

[0051] See Figure 1 As shown, Figure 1 This is a schematic flowchart of an image retrieval method according to this application. The method is applied to an image retrieval device and includes:

[0052] Step 101: Within the first time interval of calculating the similarity between feature vectors based on the image to be retrieved and the feature vectors of the images being compared in the base database, the current similarity results are sorted.

[0053] The first time interval can be a time period for calculating one similarity result or a time period for calculating J similarity results, and the number of current similarity results can be Q.

[0054] Optionally, the calculation of the current similarity result and the calculation of the next similarity result can be performed in parallel.

[0055] Step 102: Calculate at least one next similarity result, such that the calculation of at least one next similarity result is performed in parallel with the sorting.

[0056] The calculation of the next similarity result can be a calculation of one similarity result, a calculation of J similarity results, or a calculation of S similarity results.

[0057] Optionally, the sorting of the current similarity result can be done in parallel with the calculation of one or more next similarity results.

[0058] This application accelerates the retrieval speed by executing any operator used for similarity result calculation in parallel with the sorting of at least one similarity result. Furthermore, the parallel execution of the various operators used for similarity result calculation allows the operator and sorting processes to be combined into parallel execution, thereby improving the overall memory utilization of the hardware and reducing the time consumption in the entire image retrieval process.

[0059] To facilitate understanding of this application, the following explanation will use face comparison retrieval as an example. It should be understood that this application is not limited to the retrieval of face images, but can also be applied to the retrieval of images containing any content.

[0060] See Figure 2 As shown, Figure 2This is a schematic diagram of face image retrieval. The face database is built offline, relative to real-time face retrieval, and serves as the base database for face image feature sets formed after feature extraction from the face images used for comparison. The base database can contain numerous characteristic features, such as gender, accessories, and database category features. Accessory features mainly refer to wearing hats, glasses, etc., while database category features refer to category features related to various application scenarios and / or uses. It is usually composed of multiple subcategories of base database data, each with different usage characteristics; for example, transportation base database data, banking base database data, etc., which are combined to form the face database. These characteristic features are typically used as attributes of the base database feature vectors and can be used to filter similarity results. To facilitate recording the attributes of the feature vectors, labels can be used to identify and record the attributes, such as gender labels, accessory labels, etc.

[0061] The face image to be retrieved can be a live image acquired through various camera devices. Feature extraction is performed on the image to obtain the feature vector of the face image to be retrieved. Various camera devices include, but are not limited to, RGB (red, green, blue) camera devices, camera devices combining RGB and infrared, and camera devices combining RGB and depth information.

[0062] See Figure 3 As shown, Figure 3 This diagram illustrates feature extraction using deep learning networks during the creation of a face database, and feature extraction of the target face image using deep learning networks. There are many methods for face feature extraction, implemented differently on various hardware platforms. For example, the detection network could be YOLOv3 or YOLOv2, or the subsequent classification network could be ResNet50, etc.

[0063] When performing image retrieval, the feature vector of the face image to be retrieved is compared with the feature vector in the base database to calculate the similarity result. By comparing the similarity, the face data that best matches the face image to be retrieved is obtained from the base database, thereby obtaining the retrieval result.

[0064] Take the calculation of cosine similarity results as an example.

[0065] The cosine value between two vectors can be obtained using the Euclidean dot product formula:

[0066] a·b‖a‖‖b‖cosθ

[0067] Based on this, given two feature vectors A and B, the cosine similarity θ is given by the dot product and the vector length, as shown below:

[0068]

[0069] in,

[0070] Similarity values ​​range from -1 to 1. A value of -1 means that the two vectors point in completely opposite directions, a value of 1 means that the two vectors point in exactly the same direction, a value of 0 usually means that they are independent, values ​​between [0, 1] represent intermediate similarity, and values ​​between [-1, 0] represent intermediate dissimilarity.

[0071] In image retrieval, since both the image to be retrieved and the base database contain a large number of feature vectors, the similarity results of these feature vectors can be calculated using the general matrix multiplication (GEMM) based on the feature vector matrices. That is, the similarity can be calculated by the dot product of feature vector matrix A and feature vector matrix B. See also... Figure 4 As shown, Figure 4 This is a schematic diagram illustrating the calculation of similarity results using GEMM as an example. The base database feature vector matrix A is a first matrix composed of M base database feature vectors with a feature dimension of K. Each row in the first matrix is ​​a base database feature vector with a feature dimension of K, and the first matrix has M rows × K columns of data. The face image data to be retrieved is a second matrix composed of N face image feature vectors with a feature dimension of K. Each column in the second matrix is ​​a face image feature vector with a feature dimension of K, and the second matrix has N columns of data, thus having K rows × N columns of data. The cross product of the first and second matrices yields a third matrix with M rows × N columns, also called the confidence matrix, where each element represents a confidence value.

[0072] Since ||| can be pre-calculated based on the base inventory and is a constant, Formula 1 becomes:

[0073]

[0074] Here, ∝ represents 1 / ‖‖, which is a constant calculated based on the underlying feature vector. GEMM is the GEMM operator between the feature vector of the image to be retrieved and the feature vector of the underlying database. For the current frame of the image to be retrieved, ‖B‖ is a constant. Thus, the core of the calculation in Formula 2 lies in GEMM. Of course, ‖B‖ will be different depending on the image to be retrieved.

[0075] The calculation of the above similarity results requires hardware platform support. Formula 2 allows for the fusion of GEMM and cosine similarity calculations, reducing repeated memory read and write operations, simplifying the calculation process, and improving memory utilization.

[0076] Given the differences in hardware resources across different hardware platforms (e.g., varying on-chip resources), hardware acceleration performance (e.g., different hardware acceleration instruction sets), and computing power, the operators used for similarity calculation differ across platforms. Therefore, when calculating Equation 2, different operators are used based on the different dimensions of the GEMM input parameters, which helps improve retrieval speed.

[0077] Example 1

[0078] See Figure 5 As shown, Figure 5 This is a flowchart illustrating an improved image retrieval method for this application. The method includes:

[0079] Step 501: At the first moment, based on the feature vector of the image to be retrieved and the feature vector of the base database, calculate the cosine similarity between the feature vector of the image to be retrieved and the feature vector of the image being compared in the base database, and obtain a similarity result.

[0080] In this step, optionally, based on the hardware platform type of the image retrieval device and according to the feature dimension and / or dimension partitioning strategy (e.g., dimension partitioning strategy) of the feature vector data, a general matrix multiplication (GEMM) operator for performing the cross product calculation is selected.

[0081] Based on the selected operator, the operator binary file of the selected operator is called and executed to calculate the cosine similarity.

[0082] Step 502: At the second moment after obtaining the similarity results, trigger the current sorting, take the current similarity results and the similarity results from the previous sorting results as the first data, sort the first data, and obtain the current sorting result.

[0083] It should be understood that even if there is only one similarity result initially, the current sorting can be triggered, and the sorting result will be that similarity result.

[0084] Step 503: At the second moment, take the second moment as the next first moment, return to step 501, and execute step 501 to calculate the next similarity result until the similarity result calculation is completed;

[0085] Step 504: Based on the sorting results of the first data, determine the comparison image that matches the image to be retrieved, and use it as the retrieval result.

[0086] In this step, the comparison images that match the image to be retrieved can be determined from the sorting results based on the set matching threshold.

[0087] In the above steps, each step can be implemented by different modules in the hardware platform. For example, step 501 can be implemented by an operator for similarity calculation, step 502 can be implemented by an operator for ranking, and steps 503 and 504 can be implemented by an application program; or, steps 502 and 503 can be implemented by a ranking operator, or steps 502 to 504 can be implemented by a ranking operator, etc. The specific design can be combined with the hardware platform resources.

[0088] In terms of chronology, see Figure 6 As shown, Figure 6 This is a timing diagram illustrating the calculation of similarity results and the triggering of sorting in Example 1. Assume that the calculation of similarity results begins at the first moment.

[0089] The similarity calculation is triggered at the first moment.

[0090] When the similarity result 1 is obtained at the second time t1, the first sorting is triggered, the first sorting result is obtained, and the next similarity result is calculated at the same time. At this time, the first sorting result is the similarity result 1 itself.

[0091] When the similarity result 2 is obtained at the second time t2, the second sorting is triggered. The similarity result 2 and the previous sorting result (the first sorting result) are sorted. That is, the similarity result 2 and the similarity result 1 are sorted to obtain the second sorting result. For example, the one with the largest similarity is selected as the sorting result. At the same time, the next similarity result is calculated.

[0092] At the second moment t3 after obtaining similarity result 3, the third sorting is triggered, and the sorting result of similarity result 3 and the second sorting result (the previous sorting result) is obtained, which is the third sorting result. At the same time, the calculation of the next similarity result is performed.

[0093] And so on,

[0094] At the second moment t when the similarity result i is obtained i The process triggers the i-th sort, obtaining the sort result of similarity result i and the (i-1)-th sort result, i.e., the i-th sort result; simultaneously, the similarity result i+1 is calculated. The time interval from the first time point to the second time point is the first time interval.

[0095] Preferably, a binary tree sorting method is used for sorting. This way, each sort only requires comparing two similarity results, which greatly shortens the processing time of the current similarity ranking and facilitates the parallel processing of similarity result calculation and similarity result sorting.

[0096] See Figure 7 As shown, Figure 7This is a schematic diagram comparing the timing performance of this embodiment with that of the serial processing method. Assuming the overall time consumption of the sorting process is the same, it can be seen from the timing diagram that the serial processing method usually sorts the data only after the similarity results of all feature vectors have been calculated. However, the retrieval method of this embodiment processes the calculation and sorting of similarity results in parallel, triggering sorting as soon as a similarity result is obtained. Thus, overall, this embodiment reduces the time consumption by t1-t0 compared to the serial processing method.

[0097] See Figure 8 As shown, Figure 8 This is a schematic diagram comparing the read / write performance of this embodiment with that of the serial method. From a memory operation perspective, the serial method requires storing all similarity results after calculating the similarity results of all feature vectors. During sorting, all similarity results need to be read. Storing all similarity results not only occupies storage space but also requires multiple read / write operations, leading to a time-consuming retrieval process. In contrast, this embodiment only needs to cache the current sorting result, requiring only one write operation, without storing each similarity result. This reduces storage space usage and avoids multiple read / write operations. Therefore, the sorting time in this embodiment is less than the sorting time in the prior art.

[0098] It should be understood that the sorting method in this embodiment is not limited to binary tree sorting, but can also be a variation of binary tree sorting, such as heap sort, AVL tree sorting, red-black tree sorting, etc. The specific sorting method used can be selected based on factors such as hardware platform resources and the cost of read and write operations.

[0099] In this embodiment, the sorting process is executed sequentially, with the sorting of the previous similarity result and the calculation of the next similarity result being performed in parallel.

[0100] Example 2

[0101] See Figure 9 As shown, Figure 9 This is a flowchart illustrating an improved image retrieval method according to Embodiment 2. In this embodiment, considering the large number of feature vectors and similarity results in practical applications, to fully utilize hardware performance and further improve retrieval speed, sorting is triggered when multiple similarity results are obtained. Preferably, the sorting steps are also performed in parallel. The specific method is as follows.

[0102] Step 901: At the first moment, based on the feature vector of the image to be retrieved and the feature vector of the base database, calculate the similarity result in the cosine similarity matrix between the feature vector matrix of the face image to be retrieved and the feature vector matrix of the image being compared in the base database.

[0103] Step 902: Determine whether the number of unranked similarity results in the calculated similarity results reaches the set first threshold J.

[0104] If so, then take the J current similarity results as the current similarity result, and then proceed to step 903.

[0105] Otherwise, wait and return to step 902.

[0106] In this step, sorting can be triggered after calculating J similarity results. At this time, the J similarity results are the current similarity results.

[0107] Step 903: At the second moment after obtaining J similarity results, trigger the current sorting, sort the J similarity results, obtain the current sorting result and save it; and take the second moment as the next first moment, execute step 901 to calculate the next similarity result, until the similarity results are calculated.

[0108] In this step, when the hardware supports DMA read mode, multiple similarity results can be read simultaneously for sorting. Furthermore, since there is no need to wait for the previous sorting result, multiple sorting can also be performed in parallel.

[0109] Step 904: Determine whether the similarity results have all been sorted. If so, proceed to step 905; otherwise, return to step 903 until all the similarity results to be sorted have been sorted.

[0110] Step 905: Summarize the similarity ranking results from each iteration into the second data, and trigger the ranking of the second data.

[0111] Step 906: Based on the sorting results of the second data, determine the comparison image that best matches the image to be retrieved, and use it as the retrieval result.

[0112] In the diagram, steps 901 and 903 can be implemented by different modules in the hardware platform. For example, step 901 can be implemented by an operator for similarity calculation, step 903 by an operator for sorting, and the remaining steps can be implemented by an application program. The sorting operator can be a binary tree sorting operator or a modified binary tree sorting operator, such as heap sort, AVL tree sort, red-black tree sort, etc. Alternatively, the operator for similarity calculation can be used to execute steps 901-902, the operator for sorting can be used to execute steps 903-905, and the remaining steps can be implemented by an application program. The operator for similarity calculation can be a cosine similarity calculation operator or a Euclidean distance calculation operator, etc. Or, the operator for similarity calculation can be used to execute step 901, the operator for sorting can be used to execute steps 902-905, and the remaining steps can be implemented by an application program, and so on. The specific design can be tailored to the hardware platform resources.

[0113] See Figure 10 As shown, Figure 10 This is a timing diagram of one embodiment two. In the diagram, before the previous sorting sequence has ended, at the second moment when J similarity results are obtained, the next sorting is triggered, and the next J similarity results are calculated at the same time. The time interval between the first moment and the second moment is the first time interval.

[0114] In this embodiment, each sorting process is independent of the others, and multiple sorting processes can be performed in parallel. At the same time, the sorting of the previous J similarity results and the calculation of the next J similarity results are performed in parallel, which helps to further reduce the time consumption of image retrieval and improve the utilization of the hardware platform.

[0115] Example 3

[0116] To further reduce the number of similarity results participating in the ranking, it is preferable to filter the similarity results based on the attribute tags corresponding to the similarity results, so as to filter out the similarity results that do not conform to the attribute tags, while retaining the desired similarity results. In this way, the timing of the ranking is related to the time when the similarity results are filtered out.

[0117] See Figure 11 As shown, Figure 11 This is a schematic diagram illustrating a process for image retrieval in Embodiment 3, where images are filtered based on specified attributes to determine the desired similarity. It includes:

[0118] Step 1101: At the first moment, based on the feature vector matrix of the face image to be retrieved and the feature vector matrix of the base database, calculate the similarity result in the cosine similarity matrix between the feature vector matrix of the face image to be retrieved and the feature vector matrix of the image being compared in the base database.

[0119] Step 1102: Determine whether the number of current similarity results in the cosine similarity matrix has reached the set second threshold Q.

[0120] If so, proceed to step 1103 to filter the similarity results.

[0121] Otherwise, wait and return to step 1102.

[0122] The second threshold is determined based on the resources of the hardware platform of the image retrieval device and / or the number of specified attribute tags to be filtered.

[0123] It should be understood that if there are many specified attribute tags to filter, then a certain number of similarity results need to be accumulated, which will greatly improve the efficiency of parallel sorting. The specific second threshold requires extensive experimentation and is positively correlated with the number of attribute tags in the actual business scenario. For example, a threshold of 10,000.

[0124] Step 1103: At the second moment when the number of current similarity results in the cosine similarity matrix reaches the second threshold Q, the current filtering is triggered. Based on the specified attribute labels, the current similarity results are filtered, removing similarity results that do not match the attribute labels and retaining those that do match, thus obtaining the filtered similarity results. Simultaneously, the second moment is taken as the next first moment, and the process returns to step 1101 to calculate the next similarity result.

[0125] Since each similarity result has attribute labels, and the number of attribute labels can be multiple, such as "male" and "wearing a hat," depending on the actual business scenario, it is not limited to these two labels. In some scenarios, there may be multiple attribute labels. Therefore, a filtering should be performed before sorting. For example, if the user specifies that only the similarity results of men wearing hats should be filtered out and then sorted, then the confidence data of women and the informative data of not wearing hats should be discarded, and only the similarity results of men wearing hats should be kept. Then, these retained data should be sorted.

[0126] It should be understood that the calculation of the next similarity result can be the calculation of Q similarity results, or without loss of generality, the calculation of other set fourth threshold S similarity results. That is, when calculating the similarity result at the next first time step, the second threshold Q is updated.

[0127] Step 1104: At the third moment after obtaining the current filtered similarity results, trigger the sorting of the filtered similarity results. For example, select the top P largest similarity results as the current sorting results, that is, perform TopP sorting. The number of P can be determined according to the hardware platform resources required for sorting and the sorting strategy.

[0128] In this step, the similarity results of the current similarity result and the previous ranking result can be used as the first data, and the first data can be sorted to obtain the current ranking result.

[0129] Step 1105: Determine whether the similarity calculation for the current image to be retrieved is complete.

[0130] If so, proceed to step 1106.

[0131] Otherwise, return to step 1101 until the similarity result of the current image to be retrieved is calculated.

[0132] Step 1106: Based on the sorting results of the first data, determine the comparison image that matches the image to be retrieved, and use it as the retrieval result.

[0133] In the above steps, steps 1101-1102, 1103, and 1104 are executed in different modules of the hardware platform. The specific design can be combined with the hardware platform resources. This allows each step to be processed in a parallel pipeline manner, which greatly reduces the time consumption of the retrieval.

[0134] See Figure 12 As shown, Figure 12 This is a time-series diagram illustrating the calculation and sorting of similarity results in Example 3. In the diagram:

[0135] At the first moment, calculate the similarity result in the cosine similarity matrix between the feature vector matrix of the face image to be retrieved and the feature vector matrix of the image being compared in the base database;

[0136] At the second moment when the number of similarity results reaches the second threshold, the current filtering is triggered to filter the current Q similarity results to obtain the filtered similarity results. At the same time, the second moment is taken as the next first moment to calculate the next Q similarity results.

[0137] At the third moment after obtaining the filtered similarity results, the current sorting is triggered, and the filtered similarity results and the similarity results in the previous sorting results are sorted to obtain the sorted results.

[0138] It should be understood that, in the above sorting steps, multiple sorting operations can also be performed in parallel, following the sorting method in Example 2.

[0139] In this embodiment, the filtering of the previous Q similarity results, the sorting of the filtered similarity results, and the calculation of the next Q similarity results are performed in parallel. Furthermore, the sorting of the currently filtered similarity results and the filtering of the next Q similarity results can be performed in parallel. The time interval between the first time and the second time is the first time interval, and the time interval between the second time and the third time is the second time interval.

[0140] Example 4

[0141] In this embodiment, sorting can also be triggered when similarity results meeting a set threshold are selected. See also Figure 13 As shown, Figure 13 This is a schematic diagram illustrating a process for image retrieval in Embodiment 4, where images are filtered based on specified attributes to determine the desired similarity. It includes:

[0142] Step 1301: At the first moment, based on the feature vector matrix of the face image to be retrieved and the feature vector matrix of the base database, calculate the similarity result in the cosine similarity matrix between the feature vector matrix of the face image to be retrieved and the feature vector matrix of the image being compared in the base database.

[0143] Determine whether the number of current similarity results has reached the currently set first threshold J.

[0144] If so, proceed to step 1302.

[0145] Otherwise, return to step 1301.

[0146] Step 1302: Determine whether the number of current similarity results in the cosine similarity matrix has reached the set second threshold Q.

[0147] If so, proceed to step 1303 to filter the similarity results.

[0148] Otherwise, wait and return to step 1302.

[0149] The second threshold is determined based on the resources of the hardware platform of the image retrieval device and / or the number of specified attribute tags to be filtered.

[0150] It should be understood that if there are many specified attribute tags to filter, then a certain number of similarity results need to be accumulated, which will greatly improve the efficiency of parallel sorting. The specific second threshold requires extensive experimentation and is positively correlated with the number of attribute tags in the actual business scenario. For example, a threshold of 10,000.

[0151] Step 1303: At the second moment when the number of current similarity results in the cosine similarity matrix reaches the second threshold Q, the current filtering is triggered. Based on the specified attribute labels, the current similarity results are filtered, removing similarity results that do not match the attribute labels and retaining those that do match, thus obtaining the filtered similarity results. Simultaneously, the second moment is taken as the next first moment, and the process returns to step 1301 to calculate the next similarity result, until the similarity results for the image to be retrieved are calculated.

[0152] In this step, the calculation of the next similarity result can be the calculation of J similarity results, or without loss of generality, it can be the calculation of a set fourth threshold S similarity results, that is, the first threshold J is updated to the fourth threshold S as the number of the next similarity results to be calculated.

[0153] Step 1304: Determine whether the number of similarity results after filtering has reached the set third threshold T.

[0154] If so, proceed to step 1305.

[0155] Otherwise, wait and return to step 1304, in which the third threshold T can be used to update the sixth threshold Y in order to filter the number of similarity results for the next Y.

[0156] Step 1305: At the third moment after obtaining T filtered similarity results, the filtered similarity results are sorted and the current sorting result is saved; this third moment is then used as the next second moment to trigger the next filtering, returning to step 1303 to perform the next filtering of similarity results, until the current similarity results have been filtered.

[0157] In this step, the number of similarity results to be filtered when triggering the next screening can be the set fifth threshold R, that is, the second threshold Q is updated to the fifth threshold R.

[0158] Step 1306: Determine whether the filtered similarity results have all been sorted.

[0159] If so, proceed to step 1307.

[0160] Otherwise, return to step 1305 until all similarity results to be sorted have been sorted.

[0161] Step 1307: Summarize the similarity ranking results from each iteration into the second data set, and trigger the ranking of the second data set.

[0162] Step 1308: Based on the sorting results of the second data, determine the comparison image that best matches the image to be retrieved, and use it as the retrieval result.

[0163] The above steps can be executed in different modules of the hardware platform. The specific design can be combined with the hardware platform resources. This allows each step to be processed in a parallel pipeline manner, which greatly reduces the time consumption of retrieval.

[0164] See Figure 14 As shown, Figure 14 This is a time-series diagram illustrating the calculation and ranking of similarity results in Example 4. In the diagram:

[0165] At the first time t0, the similarity result in the cosine similarity matrix between the feature vector matrix of the face image to be retrieved and the feature vector matrix of the image being compared in the base database is calculated. At the second time t1, when the number of similarity results reaches the first threshold J, the calculation of the next similarity result is triggered. Optionally, the number of the next similarity result calculation can be updated by a fourth threshold S. Preferably, the calculation of the first similarity result and the calculation of the second similarity result are executed in parallel by setting the first threshold J and / or the fourth threshold S.

[0166] At the second time t2 when the number of similarity results reaches the second threshold Q, the current filtering (first filtering in the figure) is triggered to filter the current Q similarity results; optionally, if the next S similarity results have not yet been calculated, the calculation of the current similarity results continues, and it is not necessary to use the second time as the next first time. If the next S similarity results have been calculated, the second time is used as the next first time; optionally, the second threshold is updated to the next filtering threshold (fifth threshold R) to perform the next filtering.

[0167] At time t4, when the number of similarity results to be filtered reaches the fifth threshold R, the current filtering (the second filtering in the figure) is triggered. Preferably, the first and second filtering can be executed in parallel by setting the second threshold Q and / or the fifth threshold R.

[0168] At time t3, after obtaining the similarity results after filtering the third threshold T, the current sorting (first sorting in the figure) is triggered, and the T filtered similarity results are sorted. Optionally, if the number of the next similarity results to be filtered has not yet reached the set threshold for filtering, such as the fifth threshold R in the figure, then it is not necessary to use the third time t3 as the next second time. Optionally, the number of the next similarity results to be sorted can also be updated, for example, the third threshold T is updated to the sixth threshold Y. In the second filtering process in the figure, when Y similarity results are filtered out (thick solid line in the figure), the second sorting is triggered. Preferably, the first sorting and the second sorting are executed in parallel by setting the third threshold T and / or the sixth threshold Y.

[0169] It should be understood that, as shown in the second filtering diagram, since sorting is triggered at time t5 after Y similarity results are obtained, there is a possibility that the filtering cycle has not yet ended. More similarity results may be obtained in the remaining filtering time. Therefore, when the cumulative number of similarity results obtained in the remaining filtering time, and the similarity results after the next filtering, reaches a set threshold (for example, time t6 in the diagram), sorting can also be triggered. Thus, the time intervals between the third times t3, t5, and t6 that trigger sorting will not be the same. Similarly, the time intervals for triggering sorting will also be different.

[0170] In this embodiment, by setting the first threshold to the sixth threshold, at least one of the following parallel operations can be achieved:

[0171] Multiple processes for calculating similarity results are performed in parallel; that is, the operators used to calculate similarity results are performed in parallel.

[0172] Multiple filtering processes (operators) can be performed in parallel;

[0173] Multiple sorting processes (operators) can be performed in parallel.

[0174] The calculation of at least multiple similarity results, multiple filtering, and one of multiple sorting are performed in parallel.

[0175] See Figure 15 As shown, Figure 15 This is a schematic diagram of an apparatus for improving image retrieval speed according to this application. The apparatus includes,

[0176] The similarity result calculation module is used to calculate the similarity between feature vectors based on the image to be retrieved and the feature vectors of the images being compared in the base database, and to trigger the sorting module to sort the data within the first time interval of the current similarity result calculation.

[0177] The sorting module is used to sort the current similarity results according to the triggering of the similarity result calculation module, and to enable the similarity result calculation module to perform the calculation of at least the next similarity result in parallel with the sorting.

[0178] The device also includes,

[0179] The filtering module, within the first time interval, triggers the filtering of the current similarity result, and ensures that the calculation of at least the next similarity result is executed in parallel with the filtering.

[0180] The filtering of similarity results is as follows: based on attribute tags, similarity results that match the attribute tags are retained to obtain the current filtered similarity results;

[0181] The sorting module further includes a function for triggering the sorting of the filtered similarity results during a second time interval when the similarity results are filtered.

[0182] The search results output module retrieves the comparison images that match the image to be searched from the base database based on the final sorting results.

[0183] See Figure 17 As shown, Figure 17 This is a schematic diagram of a device for improving image retrieval speed according to this application. Wherein,

[0184] The similarity result calculation module specifically includes functions for calculating the similarity between feature vectors based on the image to be retrieved and the feature vectors of the images being compared in the base database when the image arrives at the first moment, thereby obtaining the current similarity result.

[0185] The sorting module specifically includes a function to trigger the current sorting at the second moment after obtaining the current similarity result, sort the current similarity results, and obtain a sorting result; the number of the current similarity results is J;

[0186] At the second moment, the second moment is taken as the next first moment and output to the similarity result calculation module, so that the similarity result calculation module calculates the next similarity result based on the image to be retrieved and the feature vector of the image being compared in the base database; the number of current similarity results is a first threshold J, and the number of next similarity results is a fourth threshold S; the filtering module specifically includes the following: at the second moment when the current similarity result is obtained, the current filtering is triggered to filter the current similarity result; at the second moment, the second moment is taken as the next first moment and output to the similarity result calculation module, so that the similarity result calculation module calculates the next similarity result based on the image to be retrieved and the feature vector of the image being compared in the base database;

[0187] The sorting module specifically includes the following: at the third moment after obtaining the current filtered similarity result, the sorting of the current filtered similarity result is triggered, and the third moment is taken as the next second moment to trigger the filtering of the next similarity result, so that the filtering of the next similarity result and the sorting of the current filtered similarity are executed in parallel.

[0188] Furthermore, it also includes triggering the sorting of the Y filtered similarity results at the third moment when the number of similarity results after the next filtering reaches the set sixth threshold Y;

[0189] The number of similarity results for the current screening is the second threshold Q, and the number of similarity results for the next screening is the fifth threshold R; the number of similarity results after the current screening is the third threshold T, and the number of similarity results after the next screening is the sixth threshold Y.

[0190] Optionally, the third moment can also be used as the next first moment and output to the similarity result calculation module, so that the similarity result calculation module can calculate the next similarity result.

[0191] See Figure 17 As shown, Figure 17 This is a schematic diagram of an image retrieval device according to this application. The device includes a hardware platform, which includes a memory and a processor. The memory stores a computer program, and the processor is configured to execute the computer program to implement the steps of the image retrieval method.

[0192] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0193] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0194] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the improved image retrieval method.

[0195] For the device / network-side equipment / storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and relevant parts can be referred to in the description of the method embodiments.

[0196] In this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0197] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the scope of protection of this application.

Claims

1. A method for improving the speed of image retrieval, characterized by, The method comprises, in a first time interval based on the to-be-retrieved image and the compared image feature vectors in the database data, triggering the sorting of the current similarity result, and performing the calculation of at least one next similarity result, so that the calculation of at least one next similarity result is performed in parallel with the sorting, determining, according to the sorting result, the compared image matched with the to-be-retrieved image as the retrieval result; wherein, the triggering of the sorting of the current similarity result in the first time interval based on the to-be-retrieved image and the compared image feature vectors in the database data, and the performing of the calculation of at least one next similarity result in parallel with the sorting, comprises: for any calculation process of the similarity result, when the calculation process reaches a first time, the similarity between the feature vectors is calculated based on the to-be-retrieved image and the compared image feature vectors in the database data, to obtain a current similarity result, at a second time when the current similarity result is obtained, triggering any sorting process to sort the current similarity result, the calculation process takes the second time as a next first time, and calculates a next similarity result, so that the calculation of the next similarity result is performed in parallel with the sorting of the current similarity result.

2. The method of claim 1, wherein, The triggering of the sorting of the current similarity result further comprises, in the first time interval, triggering the screening of the current similarity result, and performing the calculation of at least one next similarity result in parallel with the screening, wherein the screening of the similarity result comprises: retaining the similarity result meeting the attribute label according to the attribute label, to obtain a current screened similarity result; the triggering of the sorting of the similarity result comprises, in a second time interval of the screening of the similarity result, triggering the sorting of the screened similarity result.

3. The method of claim 2, wherein, The triggering of the sorting of the screened similarity result further comprises, so that the sorting of the screened similarity result is performed in parallel with the screening of at least one next similarity result; the calculation of the current similarity result is performed in parallel with the calculation of the next similarity result; and / or the screening of the current similarity result is performed in parallel with the screening of the next similarity result; and / or the sorting of the current similarity result is performed in parallel with the sorting of the next similarity result.

4. The method of claim 3, wherein, The triggering of the screening of the current similarity result in the first time interval, and the performing of the calculation of at least one next similarity result in parallel with the screening, comprises: for any calculation process of the similarity result, at a second time when the current similarity result is obtained, triggering any screening process to screen the current similarity result, the calculation process takes the second time as a next first time, and calculates a next similarity result, so that the screening of the current similarity result is performed in parallel with the calculation of the next similarity result; the triggering of the sorting of the screened similarity result in the second time interval of the screening of the similarity result comprises, For any calculation process of the similarity results, at a third time point when the current filtered similarity results are obtained, triggering any sorting process to sort the current filtered similarity results; The making the sorting of the filtered similarity results and the filtering of the next similarity results are performed in parallel includes taking the third time point as a next second time point, triggering any filtering process to filter the next similarity results, so that the filtering of the next similarity results and the sorting of the current filtered similarity results are performed in parallel.

5. The method of any one of claims 1 to 4, wherein, The performing the calculation of the next similarity results so that the calculation of the next similarity results and the sorting are performed in parallel further includes, For any calculation process of the similarity results, at a third time point when the current filtered similarity results are obtained, triggering any sorting process to sort the current filtered similarity results, The calculation process takes the third time point as a next first time point, and triggers the calculation of the next similarity results, so that the calculation of the next similarity results and the sorting of the current filtered similarity results are performed in parallel.

6. The method of claim 5, wherein, The calculation process obtains the current similarity results based on the to-be-retrieved image and the compared image feature vectors in the database data when the first time point is reached, and includes, When the number of the calculated similarity results reaches a set first threshold value J, the J similarity results are taken as the current similarity results; The triggering any sorting process to sort the current similarity results at the second time point when the current similarity results are obtained includes, The triggering the sorting process to sort the current J similarity results at the second time point when the current J similarity results are obtained; The calculation process takes the second time point as a next first time point, and calculates the next similarity results, so that the calculation of the next similarity results and the sorting of the current similarity results are performed in parallel, and includes, The calculation process takes the second time point as a next first time point, and calculates the next S similarity results, so that the calculation of the next S similarity results and the sorting of the current J similarity results are performed in parallel, where S is a set fourth threshold value.

7. The method of claim 5, wherein, The calculation process obtains the current similarity results based on the to-be-retrieved image and the compared image feature vectors in the database data when the first time point is reached, and includes, When the number of the similarity results calculated by the calculation process reaches a set first threshold value J, triggering the calculation of the next similarity results, and updating the first threshold value to a fourth threshold value S for calculating the next S similarity results, When the number of the similarity results calculated by the calculation process reaches a set second threshold value Q, the Q similarity results are taken as the current similarity results; The triggering any filtering process to filter the current similarity results at the second time point when the current similarity results are obtained includes, At the second time point when the Q current similarity results are obtained, the screening process is triggered to screen the current Q similarity results, and the second threshold Q is updated to a set fifth threshold R to screen the next R number of similarity results; The calculation process takes the second time point as a next first time point, and calculates a next similarity result, so that the screening of the current similarity result and the calculation of the next similarity result are performed in parallel, including, The calculation process takes the second time point as a next first time point, and calculates a next S similarity results, so that the screening of the current Q similarity results and the calculation of the next S similarity results are performed in parallel, wherein S is a set fourth threshold; The third time point when the current screened similarity results are obtained, any sorting process is triggered to sort the current screened similarity results, including, At the third time point when the number of current screened similarity results reaches a set third threshold T, the sorting process is triggered to sort the current T screened similarity results, At the third time point when the number of next screened similarity results reaches a set sixth threshold Y, the sorting process is triggered to sort the Y screened similarity results; The third time point is taken as a next second time point, and any screening process is triggered to screen a next similarity result, so that the screening of the next similarity result and the sorting of the current screened similarity are performed in parallel, including, The third time point is taken as a next second time point, and the screening process is triggered to screen the next R similarity results, so that the screening of the next R similarity results and the sorting of the current T screened similarity are performed in parallel; The third time point is taken as a next first time point, and the calculation process is triggered to calculate a next similarity result, so that the calculation of the next similarity result and the sorting of the current screened similarity are performed in parallel, including, The third time point is taken as a next first time point, and the calculation process is triggered to calculate a next S similarity results, so that the calculation of the next S similarity results and the sorting of the current T screened similarity are performed in parallel, Wherein, The first threshold J and the fourth threshold S are determined according to the parallel execution of the calculation of the current similarity result and the calculation of the next similarity result, The second threshold Q and the fifth threshold R are determined according to the parallel execution of the screening of the current similarity result and the screening of the next similarity result, The third threshold T and the sixth threshold Y are determined according to the parallel execution of the sorting of the current similarity result and the sorting of the next similarity result.

8. The method of claim 1, wherein, The sorting of the current similarity result includes, The current similarity result and the similarity result contained in the last sorting result are taken as first data, and the first data is sorted, The sorting result of the first data is taken as a final result; Or, The current similarity result is sorted, After all the similarity results are sorted, the similarity results in the sorting results are summarized to obtain second data, the second data is sorted, and the sorting result of the second data is taken as a final result.

9. The method of claim 1, wherein, The similarity between the feature vectors is calculated based on the image to be searched and the compared image feature vectors in the database data, and includes, The length of the current image feature vector matrix to be searched is calculated based on the current image to be searched, The compared image feature vector matrix in the database data is cross-multiplied with the image feature vector matrix to be searched to obtain a confidence value, The confidence value is divided by the length of the current image feature vector matrix to be searched and the length of the compared image feature vector matrix to obtain a similarity result, The length of the compared image feature vector matrix is pre-calculated based on the database data.

10. The method of claim 9, wherein, The compared image feature vector matrix in the database data is cross-multiplied with the image feature vector matrix to be searched to obtain a confidence value, and includes, According to the hardware platform type of the image retrieval device, the feature vector data is selected according to the feature dimension and / or dimension division strategy, and the general matrix multiplication GEMM operator is selected for cross-multiplication calculation, According to the selected operator, the operator binary file of the selected operator is called and executed.

11. An apparatus for improving the speed of image retrieval, characterized by, The device includes, The similarity result calculation module is configured to calculate the similarity between the feature vectors based on the image to be searched and the compared image feature vectors in the database data, and trigger the sorting module to sort at the first time interval of the current similarity result calculation, The sorting module is configured to sort the current similarity result according to the trigger of the similarity result calculation module, and make the similarity result calculation module perform at least one similarity result calculation in parallel with the sorting; The retrieval result output module determines the compared image matched with the image to be searched as the retrieval result output according to the sorting result, The similarity result calculation module is configured to calculate the similarity between the feature vectors based on the image to be searched and the compared image feature vectors in the database data to obtain the current similarity result when any calculation process for the similarity result reaches the first time, and take the second time when the current similarity result is obtained as the next first time to calculate the next similarity result, so that the calculation of the next similarity result is performed in parallel with the sorting of the current similarity result, The sorting module is configured to trigger any sorting process to sort the current similarity result at the second time. The hardware platform includes a memory and a processor, the memory stores a computer program, and the processor is configured to execute the computer program to realize the steps of the image retrieval speed improvement method according to any one of claims 1 to 10.

12. An image retrieval apparatus comprising a hardware platform, characterized in that, ​

Images