Image retrieval method and device, and image retrieval equipment

Abstract

The present application discloses an image retrieval method, which includes, on the image retrieval device side, obtaining feature vector data of at least one image to be retrieved and a compared image in the underlying database data, selecting various operators for calculating the similarity between the image to be retrieved and the compared image according to the hardware platform type of the image retrieval device and the various feature dimensions and / or dimension partitioning strategies of the feature vector data, calling and executing the operator binary file of the selected operator according to the selected operator to obtain a similarity result, comparing the similarity results, and obtaining a compared image that matches the image to be retrieved from the underlying database data as a retrieval result. The present application is conducive to constructing a platform with hardware characteristics to achieve a retrieval acceleration method with optimal performance, and because the operator selection is based on the input feature dimension, the present method can support dynamic resolution retrieval, thereby obtaining retrieval effects at different resolutions.

Description

Technical Field

[0001] The present application relates to the field of image analysis, and in particular, to an image retrieval method and device. Background Art

[0002] In image analysis, it is usually necessary to compare the image to be analyzed with the image to be compared. For example, in video analysis, the input image or video data is compared one by one with the image data in the database to obtain the retrieval results.

[0003] Image retrieval mainly compares the features extracted from the image to be retrieved with the features extracted from the compared image to obtain the most matching compared image, thereby obtaining the retrieval results.

[0004] Since the image itself contains a large amount of feature data, for example, the dimension of a single feature vector is 512, and each dimension is represented by a 16-bit floating point. Assuming there are 1 million feature vectors, the total memory occupied is:

[0005] 512 (dimensions) × 2 (bytes) × 1 million = 0.95 GB

[0006] When such a large amount of data is loaded onto the hardware, the hardware memory is severely limited. At the same time, such a huge amount of data also means a large amount of calculation and low acceleration performance. Summary of the Invention

[0007] The embodiment of the present application provides an image retrieval method to improve the acceleration performance under the hardware platform.

[0008] An image retrieval method provided in an embodiment of the present application is implemented as follows: on the image retrieval device side,

[0009] Obtaining feature vector data of at least one image to be retrieved and the image to be compared in the base database data,

[0010] According to the hardware platform type of the image retrieval device, and in accordance with the characteristic dimensions and / or dimension partitioning strategy of the characteristic vector data, various operators for calculating the similarity between the image to be retrieved and the image to be compared are selected.

[0011] According to the selected operator, call the operator binary file of the selected operator and execute it to obtain the similarity result.

[0012] Compare the similarity results and obtain the compared image that matches the image to be retrieved from the database data as the retrieval result.

[0013] Preferably, before selecting each operator for calculating the similarity between the image to be retrieved and the image to be compared according to the hardware platform type of the image retrieval device and the feature dimension and / or dimension partitioning strategy of the feature vector data, the method further includes:

[0014] According to the hardware platform type, the feature dimensions of the feature vector data are aligned to obtain a data arrangement format that matches the hardware platform type.

[0015] Preferably, the step of obtaining feature vector data of at least one image to be retrieved and feature vector data of the image to be compared in the base database data further includes:

[0016] According to the set dimension size, filter out the feature vector data that does not meet the set dimension size;

[0017] and / or,

[0018] According to the resources of the hardware platform, the number of feature vectors contained in the image feature vector data to be retrieved is limited.

[0019] Preferably, the selection of operators for calculating the similarity between the image to be retrieved and the image to be compared according to the hardware platform type of the image retrieval device and the feature dimension and / or dimension partitioning strategy of the feature vector data includes:

[0020] Select the operator based on the relationship between the on-chip resources of the hardware platform and the amount of feature vector data.

[0021] in,

[0022] The on-chip resources of the hardware platform are determined by the hardware platform type.

[0023] The amount of feature vector data is determined by the feature dimension and the number of feature vectors.

[0024] The dimension partitioning strategy is determined by the memory size of the hardware platform.

[0025] Preferably, the alignment operation is performed on the feature dimensions of the feature vector data according to the hardware platform type to obtain a data arrangement format that matches the hardware platform type, including:

[0026] Convert the eigenvector matrix, which consists of the eigenvector dimension as the number of rows and the number of eigenvectors as the number of columns, into one or more slice matrices;

[0027] The shard matrix includes feature vector data with the number of feature vectors as rows and the number of columns as the feature dimension of the feature vectors. The size of the feature dimension of the feature vectors is determined by the memory alignment number. The total number of shards in the shard matrix is ​​the quotient of the feature dimension of the feature vectors and the feature dimension of the feature vectors, rounded upwards.

[0028] The selecting an operator according to the relationship between the on-chip resources of the hardware platform and the data volume of the feature vector data includes comparing the total number of slices with the data volume in the slice matrix;

[0029] If the total number of slices is greater than or equal to the amount of data in the slice matrix, and the on-chip resources can accommodate the feature vector data composed of the total number of slice matrices, then select the first operator, which is used to process the data in a multi-channel parallel and slice matrix parallel manner, so that the one or more slice matrices are directly expanded into a two-dimensional manner with the total number of slices as the number of rows and the amount of data in the slice matrix as the number of columns for data operation, wherein the number of channels is the same as the total number of slices,

[0030] Otherwise, the data is divided based on the total number of shards according to the amount of data that the on-chip resources can accommodate, and the operator is selected in a way that the on-chip resources can accommodate the amount of data after the division.

[0031] Preferably, said dividing the total number of shards according to the amount of data that can be accommodated by the on-chip resources, and selecting operators in a manner that enables the on-chip resources to accommodate the amount of data after the division, includes:

[0032] The quotient of the set threshold and the amount of data in the slice matrix is ​​rounded up to get the number of single channels, and the slice matrix with at least one single channel is used as the amount of data after segmentation; wherein, the threshold is determined by the memory size of the hardware platform,

[0033] Compare the amount of data in the shard matrix with the threshold,

[0034] If the amount of data in the slice matrix is ​​greater than or equal to the threshold, a second operator is selected, which is used to process data in a parallel manner in a single-channel slice matrix, so that the one or more slice matrices are directly expanded into a two-dimensional manner with single channels as the number of rows and the amount of data in the slice matrix as the number of columns to perform data operations.

[0035] Otherwise, a third operator is selected, which is used to process data in parallel with a single-channel slice matrix and in serial with at least two or more single-channel slice matrices, so that the one or more slice matrices are directly expanded into a two-dimensional manner with single channels as the number of rows and the amount of data in the slice matrix as the number of columns to perform data operations on all single-channel slice matrices in turn.

[0036] Preferably, the step of calling and executing the operator binary file of the selected operator according to the selected operator further includes:

[0037] For each selected operator, package and describe the parameters required to implement the operator.

[0038] According to the operator name in the parameter, call the operator binary file of the selected operator;

[0039] The operator binary file is obtained from an operator file pre-stored outside the image retrieval device,

[0040] The operator files include operator binary files of various hardware platform types and / or operator implementation files of various hardware platform types for directly performing binary compilation to generate the operator binary files.

[0041] The operator has different types of operator implementation files corresponding to different optimization strengths and / or feature dimensions.

[0042] The same operator implementation file may have different corresponding operator binary files depending on the optimization strength and / or compiler.

[0043] An embodiment of the present application provides an image retrieval device, which includes:

[0044] The feature vector data acquisition module is used to obtain the feature vector data of the image to be retrieved and the image to be compared in the base database data.

[0045] An operator selection module is used to select various operators for calculating the similarity between the image to be retrieved and the image to be compared according to the hardware platform type of the image retrieval device and the feature dimensions and / or dimension partitioning strategy of the feature vector data.

[0046] The operator implements the callback module, which is used to call the operator binary file of the selected operator and execute it to obtain the similarity result.

[0047] The similarity comparison module is used to compare the similarity results and obtain the compared image that best matches the image to be retrieved from the database data as the retrieval result.

[0048] An embodiment of the present application further provides an image retrieval device, comprising a hardware platform, wherein the hardware platform comprises a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the computer program to implement the steps of any of the above-described image retrieval methods.

[0049] The image retrieval method provided in the present application selects various operators for calculating the similarity between the image to be retrieved and the image to be compared according to the hardware platform type of the image retrieval device and the feature dimensions and / or dimension partitioning strategy of the feature vector data. In this way, for different hardware platforms, the optimal operator matching the hardware platform type can be selected to achieve the best computing performance under the specific hardware platform, which is conducive to building a retrieval acceleration method with a platform with hardware characteristics to achieve optimal performance. In addition, since the operator is selected based on the input feature dimension, the method can support dynamic resolution retrieval, thereby obtaining retrieval effects at different resolutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A flowchart of the image retrieval method according to an embodiment of the present application is shown.

[0051] Figure 2 A schematic diagram of face image retrieval.

[0052] Figure 3 A schematic diagram of feature extraction through a deep learning network when establishing a face database and feature extraction through a deep learning network for face images to be retrieved.

[0053] Figure 4 A schematic diagram of similarity calculation based on GEMM.

[0054] Figure 5 A flowchart of the image retrieval method according to an embodiment of the present application is shown.

[0055] Figure 6 A schematic diagram of the data arrangement for the alignment of NCHW and NHWC32.

[0056] Figure 7 A schematic diagram of data arrangement format conversion.

[0057] Figure 8 A schematic diagram of different operators selected according to different resolutions.

[0058] Figure 9 A schematic diagram of a process for operator selection.

[0059] Figure 10 A schematic diagram of the first operator implementation.

[0060] Figure 11 A schematic diagram of the second operator implementation.

[0061] Figure 12 A schematic diagram of the third operator implementation.

[0062] Figure 13 A schematic diagram of an image retrieval device according to an embodiment of the present application.

[0063] Figure 14 A schematic diagram of an image retrieval device according to an embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical means and advantages of this application more clear, this application is further described in detail below with reference to the accompanying drawings.

[0065] The applicant's research found that image retrieval based on the general matrix multiplication (GEMM) algorithm is a full-sample distance calculation with no loss of accuracy, but the amount of calculation is very large. Further research also found that due to limited hardware resources, the GEMM image retrieval method implemented on some platforms does not support dynamic resolution transformation, that is, the resolution at the time of initialization must also be the fixed resolution when performing feature retrieval; for example, when performing GEMM image retrieval on an ARM-based hardware platform, in order to achieve the purpose of acceleration, a specific optimization dimension will be specified when the operator implementation code is compiled, so that there is no need to specify the dimension information again during operation to achieve the purpose of performance acceleration.

[0066] See also Figure 1 As shown, Figure 1 The following is a flow chart of an image retrieval method according to an embodiment of the present application. The method is applied to an image retrieval device and includes:

[0067] Step 101: Obtain the feature vector data of the current image to be retrieved and the image to be compared in the database.

[0068] Step 102: Select various operators for calculating the similarity between the image to be retrieved and the image to be compared according to the hardware platform type of the image retrieval device and the feature dimensions and / or dimension partitioning strategy of the feature vector data.

[0069] It should be understood that hardware platform types include but are not limited to types, models, and different generations of hardware platforms of the same type and / or model. For example, hardware platform types include reduced instruction set microprocessors (ARM), central processing units (CPUs), graphics processing units (GPUs), etc., according to their types. Among them, CPUs may also include Sky Lake architecture, Cascade Lake architecture, etc. according to their models.

[0070] Among them, the feature dimensions of the feature vector data are different for different images to be retrieved, and the selected operators are different. It should be understood that in the similarity calculation process, even the feature dimensions of the feature vector data of the same image to be retrieved can be different. In this way, different operators can be selected in the similarity calculation process.

[0071] Step 103: Based on the selected operator, call the operator binary file of the selected operator and execute it to obtain the similarity result.

[0072] Step 104: Compare the similarity results and obtain the compared images that match the image to be retrieved from the database as the retrieval results. Optionally, the compared images that meet a set matching threshold are used as the retrieval results.

[0073] Step 105 , determining whether all the images to be retrieved have been retrieved. If so, the process ends. Otherwise, the next image to be retrieved is used as the current image to be retrieved, and the process returns to step 101 .

[0074] This embodiment of the application analyzes the feature dimensions of the feature vector of the current image to be retrieved and selects the optimal operator based on the hardware platform type to achieve optimal computing performance on the specific hardware platform. Because this application achieves acceleration based on the input feature dimension information, it can support dynamic resolution transformation and is platform-universal, achieving optimal performance on different platforms.

[0075] To facilitate understanding of this application, the following will be explained using face comparison retrieval as an example. It should be understood that this application is not limited to the retrieval of face images, but is also applicable to the retrieval of images containing any content, such as vehicle images, terrain images, text images, etc.

[0076] See also Figure 2 As shown, Figure 2 A schematic diagram of facial image retrieval. The face database is established offline relative to real-time face retrieval, and is used to extract features from the face image being compared, forming a facial image feature set. The database data can have a large number of characteristic features, such as gender, accessory features, and database category features. Accessory features primarily refer to wearing a hat or glasses, and database category features primarily refer to category features for various application scenarios and / or uses. The database data can typically be combined from multiple subcategories, each with different usage characteristics, such as transportation database data and bank database data. Together, these data form a face database.

[0077] The facial image to be retrieved can be a living body image captured by various camera devices. Feature extraction is performed based on the image to obtain a feature vector of the facial image to be retrieved. Various camera devices include, but are not limited to, RGB (red, green, and blue) cameras, cameras that combine RGB and infrared, and cameras that combine RGB and depth information.

[0078] See also Figure 3 As shown, Figure 3 This diagram illustrates the use of deep learning networks for feature extraction when establishing a face database and for extracting features from face images to be retrieved. There are many methods for facial feature extraction, and their implementation varies across different hardware platforms. For example, the detection network can use YOLOv3 or YOLO V2, while the subsequent classification network can use ResNet50.

[0079] When performing image retrieval, the similarity between the feature vector of the face image to be retrieved and the feature vector in the base database data is calculated. By comparing the similarities, the face data that best matches the face image to be retrieved is obtained from the base database data, thereby obtaining the retrieval result.

[0080] See also Figure 4 As shown, Figure 4 This is a schematic diagram of similarity calculation based on GEMM. The base data is a first matrix consisting of M base feature vectors with a feature dimension of K. Each row in the first matrix is ​​a single base feature vector with a feature dimension of K. The first matrix has M rows, resulting in a total of M rows × K columns of data. The face image data to be retrieved is a second matrix consisting of N feature vectors with a feature dimension of K. Each column in the second matrix is ​​a single image feature vector with a feature dimension of K. The second matrix has N columns, resulting in a total of K rows × N columns of data. The first matrix is ​​cross-producted with the second matrix to obtain a third matrix with M rows and N columns, which is the similarity data.

[0081] The calculation of similarity described above requires support from the hardware platform. Given that different hardware platforms have different hardware resources, such as different on-chip resources; different hardware acceleration performance, for example, different hardware platforms have different hardware acceleration instruction sets; and different hardware platforms have different hardware computing capabilities, this results in different operators used for similarity calculations on different hardware platforms. Operators can be understood as the portion of the similarity calculation process used for node processing, representing the minimum process required to implement a specific function, including but not limited to data reading and writing, data calculation, etc.

[0082] To improve the computing power of the hardware platform, see Figure 5 As shown, Figure 5The following is a flow chart of the image retrieval method according to an embodiment of the present application. Taking the retrieval of a current image to be retrieved as an example, the method includes:

[0083] Step 501 : filtering the feature dimension information of the feature vector to obtain a filtered feature vector, wherein the dimension of the filtered feature vector conforms to the set dimension.

[0084] In this step, the feature dimension information of the feature vector of the current facial image to be retrieved is filtered. In order to reduce meaningless calculations, some messy feature dimensions are filtered out. Optionally, feature vectors whose K is not equal to the set dimension threshold are filtered out, for example, feature vectors whose K is not equal to 258, 512, or 1024. Alternatively, in view of the versatility of the hardware platform, a threshold of N is set according to the hardware platform to limit N, for example, N does not exceed 64.

[0085] Since the similarity calculation is the cross product of the first matrix and the second matrix, the dimension of the feature vector in the base database data needs to be filtered in the same way so that the dimension of the feature vector of the base database data is the same as the dimension of the face image to be retrieved.

[0086] Step 502: perform alignment operations on each feature dimension information of the feature vector according to the type of the hardware platform, and convert it into a data arrangement format that matches the hardware platform.

[0087] Given that different hardware platforms have different hardware acceleration instruction sets, the hardware acceleration instructions in the hardware acceleration instruction set can only be called if the primary dimension of the data operation is aligned with the memory alignment number. Therefore, to better utilize hardware acceleration performance, the memory alignment number can be determined based on the obtained hardware information to call instructions in the hardware acceleration instruction set and improve computing efficiency.

[0088] for example:

[0089] For GPU hardware, a specific optimized instruction set can be selected based on the GPU's computing power. Different computing powers have different optimized instruction sets. The computing power value can be obtained by querying the parameters of the GPU graphics card. For example, the computing power value includes but is not limited to compute_60, compute_72, compute_75, etc. 80% of the instruction sets of GPUs of different generations are the same, and the remaining 20% ​​of the instruction sets belong to the characteristic instructions of the current generation. These characteristic instruction sets often reflect advantages. Based on the characteristic instruction sets, optimization work for specific platforms can be performed.

[0090] For CPU hardware, the CPU architecture type can be determined based on the specific hardware model. Because different hardware core architectures support different instruction sets, the optimization methods and strengths are also different.

[0091] Different computing hardware has different memory alignment requirements. For example, ARM uses 32-byte alignment for the K dimension, GPUs use 8-byte alignment for the K dimension, and CPUs use 16-byte alignment for the K dimension. You can determine the memory alignment requirement based on the computing hardware type identifier in the hardware information obtained.

[0092] Data can be arranged in different formats in memory, such as NCHW and NHWC32 alignment, see Figure 6 As shown, Figure 6 A schematic diagram of the data arrangement of NCHW and NHWC32 alignment. To improve data access efficiency, the first matrix of K rows × M columns is arranged using K1 M rows × K0 columns sliced ​​matrices, recorded as [K1, M, K0] data arrangement format. The schematic diagram of the conversion between the first matrix of K rows × M columns and the [K1, M, K0] data arrangement format is shown in the figure. Figure 7 As shown in the figure. K0 is the slice feature dimension of the eigenvector in the slice matrix, which is strongly related to the microarchitecture. Here, it is set to 32, as this part of the data needs to be stored continuously. K1 can be understood as the number of slice matrices, K1 = K / K0. If the result is not divisible, it is rounded up.

[0093] Step 503 : Select various operators for similarity calculation according to the type of hardware platform, the dimension of the input feature vector, and / or the dimension partitioning strategy.

[0094] Given the varying memory resources of different hardware platforms, to address the limited memory resources during similarity calculation, different operators are selected based on the filtered feature vector dimensions (input dimensions) and / or the dimension partitioning strategy. Preferably, operators are selected based on different partitioning ranges of K. For example, if K is greater than 0 and less than 128, the first operator is selected; if K is greater than 128 and less than 256, the second operator is selected; and if K is greater than 256, the third operator is selected. The partitioning ranges need to be calculated based on the hardware resources, primarily the on-chip memory resources.

[0095] Since different hardware has different on-chip resources, the dimension partitioning strategies are also completely different. When selecting operator acceleration strategies under different input dimensions, different input dimensions that conform to the same operator acceleration strategy can share the same binary file. The same operator acceleration strategy means that the dimension division range is the same. For example, K is greater than 0 and less than 128 and belongs to one strategy. Then K is between 0 and 128, which actually includes many image resolutions. However, for those resolutions where K is between 0 and 128, the operators they implement are the same, for example, they are all operator 0. In this way, by selecting acceleration operators with different strategies for different input dimensions, different resolutions can be supported, so that dynamic resolution transformation is supported during image retrieval, avoiding the problem of not supporting dynamic resolution transformation due to directly specifying the resolution when compiling the operator code in existing image retrieval methods. See Figure 8 As shown, Figure 8 A schematic diagram of the acceleration operators with different strategies selected based on different resolutions. After inputting different resolution parameters, the acceleration operators with different strategies are selected based on different resolutions.

[0096] For ease of understanding, the following takes the processing of the first matrix as an example to illustrate the selection of operators.

[0097] See also Figure 9 As shown, Figure 9 A schematic diagram of a process for operator selection.

[0098] Determine the relationship between the total number of slices K1 and the amount of data contained in the slice matrix. The amount of data contained in the slice matrix is ​​M×K0, which can be understood as the size of the slice matrix.

[0099] If K1 is greater than or equal to the number of data contained in the slice matrix, then since the on-chip resources are sufficient to accommodate all the data, the first operator can be selected. This operator is used to process the data in parallel with K1 channels and slice matrices. In this way, the [K1, M, K0] data is directly expanded into a two-dimensional manner with K1 as the number of rows and M×K0 as the number of columns to perform data operations, such as Figure 10 As shown, Figure 10 A schematic diagram of the first operator implementation.

[0100] If K1 is less than or equal to M×K0, K1 is divided according to the amount of data that can be accommodated by the on-chip resources so that the on-chip resources can accommodate the divided amount of data. Specifically:

[0101] The number of single channels S0 is obtained by rounding up the quotient of the set threshold and the amount of data in the slice matrix. The threshold is determined by the size of the on-chip memory.

[0102] Determine whether M×K0 is greater than or equal to the set threshold,

[0103] If so, the number of single channels S0 is equal to 1, so the second operator is selected. This operator is used to process data in a single-channel slice matrix parallel manner. In this way, K1 is divided into a single channel S0, and the [K1, M, K0] data is directly expanded into a two-dimensional manner with S0 as the number of rows and M×K0 as the number of columns to perform data operations. Among them, the size of S0×M×K0 is less than or equal to the size of the on-chip memory. Figure 11 As shown, Figure 11 A schematic diagram of the second operator implementation.

[0104] Otherwise, the number of single channels S0 is greater than 1, so the third operator is selected. This operator is used to process data in parallel with a single-channel slice matrix and in serial with each single-channel slice matrix. In this way, K1 is divided into multiple single-channel S0, and the [K1, M, K0] data is directly expanded into a two-dimensional manner (single-channel slice matrix) with S0 as the number of rows and M×K0 as the number of columns to perform data operations in sequence, where the size of S0×M×K0 is less than or equal to the size of the on-chip memory, and S0 = set threshold / (M×K0). If the result is not divisible, round up. Figure 12 As shown, Figure 12 A schematic diagram of the third operator implementation.

[0105] From the above operator selection strategy, it can be seen that if the amount of data to be calculated is larger than the on-chip memory, then the on-chip memory cannot hold all the data to be calculated at once and can only be completed through multiple operations. Therefore, during the calculation, it can only be accelerated through single-channel parallelism. However, if the amount of data to be calculated is smaller than the on-chip memory, that is, all the data can be placed in the on-chip memory at once, then the amount of data to be calculated can be processed in parallel on multiple channels at the same time. From the operator selection strategy, it can be seen that the number of single channels is directly related to the dimensional division strategy, and the amount of data that can be accommodated by the on-chip memory is indirectly related to the dimensional division. From the above operator selection, it can be seen that when the amount of data to be calculated is different, the operator has different meanings. Therefore, during the similarity calculation process, even the same image to be retrieved may be calculated by different operators to complete the similarity calculation.

[0106] Similarly, the same method is used to select operators for the feature vector matrix (second matrix) of the image to be retrieved.

[0107] Step 504: In view of the fact that different hardware platforms and different operators involve different dimension partitioning parameters and call different hardware acceleration strategies, the parameters required for each selected operator are described to integrate resources, shield the differences between heterogeneous hardware, and facilitate the calling of operators.

[0108] For each operator, some of the operator parameters can be directly assigned, while others need to be obtained through calculation. The operator parameters are described centrally in the form of lists, structures, etc., that is, packaged descriptions, to facilitate the input of parameters in subsequent processing.

[0109] Taking the first operator as an example, the required parameters include the number of rows and columns of the shard matrix, the number of channels, the number of repetitions, the operator name, etc. These parameters can be described in a packaged manner.

[0110] Step 505: According to the selected operator, call the binary file corresponding to the selected operator in the operator binary file to complete the call of the single operator.

[0111] The operator binary file can be obtained by directly compiling the operator implementation file. Since different operators correspond to different input dimensions in the above steps, different operators have different binary files. Therefore, when calling an operator, the corresponding operator is selected based on the input dimension information passed in.

[0112] Furthermore, since different hardware platforms correspond to different compilers, the optimization intensity can be controlled during compilation. Therefore, for the same operator binary file, different optimization intensities can be selected to generate different binary files. For example, some algorithm scenarios require optimization for space usage, so the -Os compilation option is added; other algorithm scenarios require extreme operator performance, so the -O3 compilation option is added, and so on.

[0113] To improve operator performance, operator implementation files suitable for multiple hardware platforms and multiple operators can be pre-established for different input dimension information, different hardware platforms, and different dimension partitioning strategies, such as operator0.cpp, operator1.cpp, and operator2.cpp in the figure.

[0114] Preferably, the implementation file type of each operator can include multiple file types to support operator implementation methods with different optimization strengths. This is because even the lowest-level implementation method will lead to different optimization strengths due to different input dimensions. Therefore, for those very special input dimensions, the performance under different methods will also be different. For example, in some dimensions, assembly implementation is the fastest; in some dimensions, using a third-party library implementation is the fastest; in some dimensions, using an ordinary C implementation is the fastest, and so on.

[0115] In practical applications, operator files, including the operator implementation files and / or binary files of operators corresponding to the operator implementation files, can be loaded externally, for example, from a storage device and / or server connected to the image retrieval device. This reduces the storage space occupied by the image retrieval device and makes the operator acceleration strategy universal, achieving optimal performance on different platforms.

[0116] When calling an operator binary file, an operator binary file with the same operator name as the operator name included in the operator parameter is searched in the operator file.

[0117] Finally, based on the obtained similarity results, the compared image that matches the image to be retrieved is obtained from the database data as the retrieval result.

[0118] The image retrieval method provided in the embodiment of the present application implements the selection of different operators for different hardware resource sizes, different data flow transmission modes, and different pipelines, for example, ARM operators suitable for ARM, CPU operators suitable for CPU, and GPU operators suitable for GPU; for different input dimensions of different images to be retrieved, the optimal operator for the current dimension can be selected under the corresponding hardware platform, for example, a first-dimensional operator matching the first input dimension, a second-dimensional operator matching the second input dimension, etc.; according to different similarity calculation methods, the operator used to calculate the similarity between the image to be retrieved and the compared image can be a cosine similarity operator, or a GEMM operator for cosine similarity calculation, or a Euclidean distance operator, etc.; to ensure scalability, by collecting operator implementation files of operators on different platforms, the operator implementation files are compiled into .o files using corresponding compilers on different hardware platforms, and then provided to the image retrieval device, thereby providing operator implementations of various operators. Unlike those optimization methods that choose to specify specific optimization dimensions directly when compiling the operator implementation code when implementing the GEMM retrieval function due to limited hardware resources, this application implements acceleration from the input feature dimension information and selects different strategy operators for different input dimensions. For example, the first operator, the second operator, the third operator, etc. in the embodiment of this application support the implementation of operators of different resolutions; this method can support a variety of different hardware platforms. After determining the hardware platform, a retrieval method with hardware characteristics to achieve optimal performance can be constructed.

[0119] See also Figure 13 As shown, Figure 13 This is a schematic diagram of an image retrieval device according to an embodiment of the present application. The device includes:

[0120] The feature vector data acquisition module is used to obtain the feature vector data of the image to be retrieved and the image to be compared in the base database data.

[0121] An operator selection module is used to select various operators for calculating the similarity between the image to be retrieved and the image to be compared according to the hardware platform type of the image retrieval device and the various feature dimensions and / or dimension partitioning strategy of the feature vector data.

[0122] The callback module is used to call the operator binary file of the selected operator and execute it to obtain the similarity result.

[0123] The similarity comparison module is used to compare the similarity results and obtain the compared image that best matches the image to be retrieved from the database data as the retrieval result.

[0124] Preferably, it also includes,

[0125] A dimension information filtering module is used to filter out feature vector data that does not meet the set dimension size according to the set dimension size; and / or to limit the number of feature vectors contained in the image feature vector data to be retrieved based on the resources of the hardware platform;

[0126] The data format conversion module is used to align the feature dimensions of the feature vector data according to the hardware platform type to obtain a data arrangement format that matches the hardware platform type;

[0127] The parameter packaging module is used to centrally describe the parameters required for each selected operator.

[0128] Among them, the data format conversion module, operator selection, and parameter packaging module can all be executed based on heterogeneous hardware platforms, that is, corresponding operations are performed according to different hardware characteristics.

[0129] See also Figure 14 As shown, Figure 14 The image retrieval device of the embodiment of the present application is a schematic diagram. The device includes a hardware platform, the hardware platform includes a memory and a processor, the memory stores a computer program, and the processor is configured to execute the computer program to implement the steps of the image retrieval method.

[0130] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0131] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0132] An embodiment of the present application further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the image retrieval method are implemented.

[0133] As for the apparatus / network-side device / storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0134] In this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

[0135] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. An image retrieval method, characterized in that: The method includes, on the image retrieval device side, Obtaining feature vector data of at least one image to be retrieved and the image to be compared in the base database data, According to the hardware platform type of the image retrieval device and the characteristic dimensions of the characteristic vector data, various operators for calculating the similarity between the image to be retrieved and the image to be compared are selected. According to the selected operator, call the operator binary file of the selected operator and execute it to obtain the similarity result. Compare the similarity results and obtain the compared image that matches the image to be retrieved from the database data as the retrieval result; in, The selecting of operators for calculating the similarity between the image to be retrieved and the image to be compared according to the hardware platform type of the image retrieval device and the characteristic dimensions of the characteristic vector data includes: According to the hardware platform type of the image retrieval device, the on-chip resources of the hardware platform are obtained, and the operator is selected based on the relationship between the on-chip resources of the hardware platform and the amount of feature vector data. The data volume of the feature vector data is determined by the feature dimension and the number of feature vectors; The operator is selected as follows: Get the slice matrix of eigenvector data, Compare the total number of slices of the slice matrix with the amount of data in the slice matrix, If the total number of slices is greater than or equal to the amount of data in the slice matrix, and the on-chip resources can accommodate the feature vector data composed of the total number of slice matrices, then select the first operator, which is used to process the data in a multi-channel parallel and slice matrix parallel manner, so that one or more of the slice matrices are directly expanded into a two-dimensional manner with the total number of slices as the number of rows and the amount of data in the slice matrix as the number of columns for data operation, wherein the number of channels is the same as the total number of slices, Otherwise, the data is divided based on the total number of shards according to the amount of data that the on-chip resources can accommodate, and the operator is selected in a way that the on-chip resources can accommodate the amount of data after the division.

2. The method according to claim 1, wherein Before selecting each operator for calculating the similarity between the image to be retrieved and the image to be compared according to each feature dimension of the feature vector data based on the hardware platform type of the image retrieval device, the method further includes: According to the hardware platform type, the feature dimensions of the feature vector data are aligned to obtain a data arrangement format that matches the hardware platform type.

3. The method according to claim 1, wherein The step of obtaining feature vector data of at least one image to be retrieved and feature vector data of the image to be compared in the base database data further includes: According to the set dimension size, filter out the feature vector data that does not meet the set dimension size; and / or, According to the resources of the hardware platform, the number of feature vectors included in the image feature vector data to be retrieved is limited.

4. The method according to any one of claims 1 to 3, characterized in that The method further includes selecting various operators for calculating the similarity between the image to be retrieved and the image to be compared according to the hardware platform type of the image retrieval device and the various feature dimensions of the feature vector data. Based on the on-chip resources, a dimension partitioning strategy is determined, wherein the dimension partitioning strategy at least includes a dimension partitioning range. Based on the characteristic dimensions of the characteristic vector data, operators corresponding to different dimensional division ranges are selected.

5. The method according to claim 4, wherein The alignment operation is performed on the feature dimensions of the feature vector data according to the hardware platform type to obtain a data arrangement format that matches the hardware platform type, including: Convert the eigenvector matrix, which consists of the eigenvector dimension as the number of rows and the number of eigenvectors as the number of columns, into one or more slice matrices; Among them, the shard matrix includes feature vector data with the number of feature vectors as the number of rows and the shard feature dimension of the feature vector as the number of columns. The size of the shard feature dimension is determined by the memory alignment number. The total number of shards of the shard matrix is ​​the quotient of the feature dimension of the feature vector and the shard feature dimension rounded up.

6. The method according to claim 1, wherein The method of dividing the total number of shards according to the amount of data that the on-chip resources can accommodate and selecting operators in a manner that enables the on-chip resources to accommodate the amount of data after the division includes: The quotient of the set threshold and the amount of data in the slice matrix is ​​rounded up to get the number of single channels, and the slice matrix with at least one single channel is used as the amount of data after segmentation; wherein, the threshold is determined by the memory size of the hardware platform, Compare the amount of data in the shard matrix with the threshold, If the amount of data in the slice matrix is ​​greater than or equal to the threshold, a second operator is selected, which is used to process data in a parallel manner in a single-channel slice matrix, so that the one or more slice matrices are directly expanded into a two-dimensional manner with single channels as the number of rows and the amount of data in the slice matrix as the number of columns to perform data operations. Otherwise, a third operator is selected, which is used to process data in parallel with a single-channel slice matrix and in serial with at least two or more single-channel slice matrices, so that the one or more slice matrices are directly expanded into a two-dimensional manner with single channels as the number of rows and the amount of data in the slice matrix as the number of columns to perform data operations on all single-channel slice matrices in turn.

7. The method according to claim 1, wherein The step of calling and executing the operator binary file of the selected operator according to the selected operator further includes: For each selected operator, package and describe the parameters required to implement the operator. According to the operator name in the parameter, call the operator binary file of the selected operator; The operator binary file is obtained from an operator file pre-stored outside the image retrieval device, in, The operator files include operator binary files of various hardware platform types, and / or operator implementation files of various hardware platform types for directly performing binary compilation to generate the operator binary files. The operator has different types of operator implementation files corresponding to different optimization strengths and / or feature dimensions. The same operator implementation file may have different corresponding operator binary files depending on the optimization strength and / or compiler.

8. An image retrieval device, characterized in that: The device comprises, The feature vector data acquisition module is used to acquire feature vector data of at least one image to be retrieved and the image to be compared in the base database data. An operator selection module is used to select various operators for calculating the similarity between the image to be retrieved and the image to be compared according to the hardware platform type of the image retrieval device and the various feature dimensions of the feature vector data. The operator implements the callback module, which is used to call the operator binary file of the selected operator and execute it to obtain the similarity result. A similarity comparison module is used to compare similarity results and obtain the compared image that matches the image to be retrieved from the database data as the retrieval result; in, The operator selection module is configured to: obtain the on-chip resources of the hardware platform according to the hardware platform type of the image retrieval device, select the operator according to the relationship between the on-chip resources of the hardware platform and the data volume of the feature vector data, The data volume of the feature vector data is determined by the feature dimension and the number of feature vectors; The operator is selected as follows: Get the slice matrix of eigenvector data, If the total number of slices of the slice matrix is ​​greater than or equal to the amount of data in the slice matrix, and the on-chip resources can accommodate the feature vector data composed of the total number of slice matrices, then select a first operator, which is used to process data in a multi-channel parallel and slice matrix parallel manner, so that one or more of the slice matrices are directly expanded into a two-dimensional manner with the total number of slices as the number of rows and the amount of data in the slice matrix as the number of columns for data operation, wherein the number of channels is the same as the total number of slices, Otherwise, the data is divided based on the total number of shards according to the amount of data that the on-chip resources can accommodate, and the operator is selected in a way that the on-chip resources can accommodate the amount of data after the division.

9. An image retrieval device, comprising a hardware platform, characterized in that: The hardware platform includes a memory and a processor, the memory stores a computer program, and the processor is configured to execute the computer program to implement the steps of the image retrieval method according to any one of claims 1 to 7.

Images