Database search method and apparatus

Abstract

This application relates to the field of computer technology and provides a database retrieval method and apparatus. The method includes: acquiring data to be retrieved; compressing the data to be retrieved into first compressed data with a predetermined data length; acquiring second compressed data with a predetermined data length corresponding to each pre-stored original data in the database; filtering a predetermined number of third compressed data that match the first compressed data from each of the second compressed data; and determining original data that matches the data to be retrieved from the original data corresponding to the predetermined number of third compressed data. Thus, while meeting the high-concurrency QPS requirements of the retrieval application system, the accuracy of the retrieval results can also be considered.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer, and particularly relates to a database retrieval method and device. BACKGROUND

[0002] QPS (Queries-per-second, query rate per second) represents the number of requests processed by a query server within a specified time, and is an important performance indicator of the query server.

[0003] For a city-level retrieval application system, a large number of retrieval requests will occur at the peak period. Since the underlying database of the city-level application system is a massive database of hundreds of millions of entries, the bottleneck of CPU (central processing unit) computing resources will result in low performance of concurrent requests when processing multiple concurrent requests, and cannot meet the high-concurrency QPS requirement of massive data. In addition, some retrieval application systems (for example, retrieval application systems in the face recognition scenario) have high requirements for the accuracy of retrieval results, so the accuracy of retrieval results needs to be considered while improving QPS. SUMMARY

[0004] Therefore, the embodiments of the present application provide a database retrieval method and device to at least solve the problem that the database in the prior art cannot meet the high-concurrency QPS requirement of the retrieval application system and consider the accuracy of retrieval results.

[0005] A first aspect of the embodiments of the present application provides a database retrieval method applied to a server configured with the database, and the method comprises: acquiring data to be retrieved; compressing the data to be retrieved into first compressed data with a set data length; acquiring second compressed data with the set data length corresponding to each pre-stored original data in the database; selecting a set number of third compressed data matched with the first compressed data from each second compressed data; and determining original data matched with the data to be retrieved from the original data corresponding to the set number of third compressed data.

[0006] The second aspect of the embodiment of the present application provides a database retrieval device, comprising: a retrieval data acquisition unit, configured to acquire data to be retrieved; a retrieval data compression unit, configured to compress the data to be retrieved into first compressed data with a set data length; a library compressed data acquisition unit, configured to acquire second compressed data with the set data length corresponding to each pre-stored original data in the database; a compressed data screening unit, configured to screen a set number of third compressed data matching the first compressed data from each of the second compressed data; and a data matching unit, configured to determine original data matching the data to be retrieved from original data corresponding to the set number of third compressed data.

[0007] The third aspect of the embodiment of the present application provides a server, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0008] The fourth aspect of the embodiment of the present application provides a computer readable storage medium, which stores a computer program, wherein the computer program is executable by a processor to implement the steps of the above method.

[0009] The fifth aspect of the embodiment of the present application provides a computer program product, which, when executed on a server, causes the server to implement the steps of the above method.

[0010] Compared with the prior art, the embodiment of the present application has the following beneficial effects:

[0011] When performing database retrieval, the server compresses the data in the database, and compares each compressed data with the compressed data of the data to be retrieved, thereby reducing the overall comparison burden of the database and helping to improve the QPS of the server. In addition, by compressing the data to screen a set number of compressed data corresponding to the compressed data to be retrieved, and then performing secondary data retrieval between each data corresponding to the screened compressed data in the database and the data to be retrieved, the comparison retrieval between the data with full precision is performed, which makes up for the precision loss in the compressed data retrieval process, and can improve the high precision of the retrieval result. In addition, there is a correlation between the set number and the set data length, so that the number of preliminary retrieval results determined based on the compressed data is controllable, which is conducive to ensuring the high precision of the secondary retrieval (i.e., data matching) result. Through the embodiment of the present application, the QPS of the database is improved while the high precision of the determined retrieval result is ensured, and the embodiment of the present application can be applied in most data retrieval scenarios (for example, a face data retrieval scenario). BRIEF DESCRIPTION OF DRAWINGS

[0012] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A flowchart illustrating an example of a database retrieval method according to an embodiment of this application is shown;

[0014] Figure 2 A flowchart illustrating an example of determining a predetermined amount of third compressed data according to an embodiment of this application is shown;

[0015] Figure 3 A flowchart illustrating an example of determining a predetermined amount of third compressed data according to an embodiment of this application is shown;

[0016] Figure 4 A flowchart illustrating an example of a feature database retrieval method according to an embodiment of this application is shown;

[0017] Figure 5 A flowchart illustrating an example of a feature database retrieval method according to an embodiment of this application is shown;

[0018] Figure 6 A flowchart illustrating an example of a database retrieval method in a face retrieval application scenario according to an embodiment of this application is shown.

[0019] Figure 7 A structural block diagram of an example database retrieval apparatus according to an embodiment of this application is shown;

[0020] Figure 8 A schematic diagram of an example server according to an embodiment of this application is shown. Detailed Implementation

[0021] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0022] To illustrate the technical solution described in this application, specific embodiments are provided below.

[0023] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0024] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0025] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0026] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrases "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0027] In specific implementations, the mobile terminals described in the embodiments of this application include, but are not limited to, other portable devices such as mobile phones, laptop computers, or tablet computers with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads). It should also be understood that in some embodiments, the aforementioned devices are not portable communication devices, but rather desktop computers with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads).

[0028] The following discussion describes mobile terminals that include displays and touch-sensitive surfaces. However, it should be understood that mobile terminals may include one or more other physical user interface devices such as physical keyboards, mice, and / or joysticks.

[0029] Various applications that can run on a mobile terminal can use at least one common physical user interface device, such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and the corresponding information displayed on the terminal can be adjusted and / or changed between and / or within applications. In this way, the terminal's common physical architecture (e.g., the touch-sensitive surface) can support various applications with user interfaces that are intuitive and transparent to the user.

[0030] Furthermore, in the description of this application, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0031] Figure 1 A flowchart illustrating an example of a database retrieval method according to an embodiment of this application is shown. Specifically, the entity executing this method may be a server configured with a database.

[0032] like Figure 1 As shown, in step 110, the data to be retrieved is obtained. Specifically, the server can receive data retrieval requests (or concurrent traffic) sent from mobile terminals (e.g., multiple mobile terminals) and parse the data to be retrieved in the data retrieval requests.

[0033] In step 120, the data to be retrieved is compressed into first compressed data with a set data length. It should be understood that various data compression methods can be used to compress the data length of the data to be retrieved and the data in the database, and there should be no limitation here. For example, if the original data in the database is stored in a data length of 256 bytes, the compressed data length in the database can be 128 bytes or 64 bytes, etc.

[0034] In step 130, second compressed data with a set data length corresponding to each pre-stored original data in the database is obtained.

[0035] In step 140, a predetermined number of third compressed data sets matching the first compressed data sets are selected from each of the second compressed data sets. Here, the predetermined number may represent the number of search results (or similar data sets) determined by comparing each of the second compressed data sets with the first compressed data sets. In some examples of embodiments of this application, there may be a correlation between the predetermined data length and the predetermined number; therefore, the corresponding predetermined number can be determined (or derived) by setting the predetermined data length. More details will be elaborated below.

[0036] In some implementations, when filtering compressed data, the similarity of each compressed data to the data to be retrieved can be calculated to obtain the corresponding similarity ranking, thereby filtering a set number of compressed data with higher similarity.

[0037] In step 150, original data matching the data to be retrieved is determined from the original data corresponding to a set number of third compressed data sets. Specifically, the similarity between the data in the database corresponding to each of the selected compressed data sets and the data to be retrieved can be calculated, and the data with the highest similarity in the database (e.g., one or more) can be determined as the matching data. It should be understood that the number of data matching the data to be retrieved can be one or more, and can be set according to user needs.

[0038] It should be noted that as the data compression ratio increases, the set data length decreases, and the distinguishability between different compressed data types decreases accordingly, which may affect the accuracy of search results to some extent. Therefore, the set quantity should be determined based on the set data length, so that the set quantity can be adjusted as the set data length changes. For example, a lower set data length and a higher set quantity can be used to ensure the accuracy of search results.

[0039] In this embodiment, two data filtering operations are integrated: a coarse screening operation on compressed data in the database and a fine screening operation based on the coarse screening results. The initial screening operation on compressed data in the database improves data retrieval efficiency and increases the QPS of the server. The fine screening operation based on the coarse screening results ensures the high accuracy of the determined retrieval results, thus achieving both improved QPS and high accuracy of retrieval results.

[0040] Figure 2 A flowchart illustrating an example of determining a set amount of third compressed data according to an embodiment of this application is shown.

[0041] like Figure 2 As shown, in step 210, the desired data length in the data length configuration instruction is obtained, and the desired data length is determined as the set data length. Specifically, the server can receive configuration operations from maintenance personnel (or users) to determine the corresponding data length configuration instruction, so that the maintenance personnel can specify the compression ratio for the data (e.g., the original data and the data to be retrieved).

[0042] In step 220, the data to be retrieved is compressed into first compressed data with the desired data length, and each original data pre-stored in the database is compressed into second compressed data with the desired data length.

[0043] In step 230, the length difference information between the expected data length and the original data length is determined.

[0044] In step 240, the set quantity is calibrated based on the length difference information.

[0045] For example, if the length difference information indicates that the expected data length is less than the original data length, a larger setting quantity can be used (or the original setting quantity can be increased). If the length difference information indicates that the expected data length is greater than the original data length, a smaller setting quantity can be used (or the original setting quantity can be decreased).

[0046] In some implementations, there is a positive correlation between the length difference information and the calibration range of the set quantity. For example, when the length difference information is large, the corresponding adjustment range for the original set quantity is large, while when the length difference information is small, the corresponding adjustment range for the original set quantity is small.

[0047] In step 250, a set number of third compressed data that match the first compressed data are selected from each of the second compressed data.

[0048] In this embodiment, maintenance personnel can adjust the data compression ratio to achieve servers with diverse QPS, meeting the needs of different retrieval scenarios. Furthermore, when the compression ratio is too high, increasing the set quantity allows more data in the database to enter the fine-tuning stage, reducing the risk of target matching data being filtered out in the initial screening stage due to data length compression, thus ensuring high accuracy of the retrieval results.

[0049] In some examples of embodiments of this application, the set quantity corresponding to the set data length can be determined based on a preset data length mapping table. Here, the data length mapping table stores multiple set data lengths and their corresponding set quantities. Thus, the set quantity under the corresponding set data length can be quickly determined by looking up the table.

[0050] Figure 3 A flowchart illustrating an example of determining a set amount of third compressed data according to an embodiment of this application is shown.

[0051] In some examples of embodiments of this application, the retrieval operation for the data to be retrieved is performed in the memory space of the server, which can significantly improve the data retrieval efficiency compared to disk space.

[0052] like Figure 3 As shown, in step 310, the data to be retrieved is compressed into first compressed data with a set data length, and each piece of data pre-stored in the database is compressed into second compressed data with a set data length.

[0053] In step 320, the first compressed data and each of the second compressed data are loaded into the server's memory space.

[0054] In step 330, a set number of third compressed data that match the first compressed data are selected from the various second compressed data stored in the memory space.

[0055] In this embodiment, the server performs data retrieval operations (e.g., similarity calculation) in memory space, which improves data retrieval efficiency. Furthermore, the server's memory space loads compressed data, which, compared to directly loading data from a database, enables hot queries on databases with larger capacities, reducing the likelihood of cold queries being triggered.

[0056] It should be understood that the term "hot query" refers to a query where the data in the database is stored in memory, and the similarity calculation is performed between the data to be retrieved and the data stored in memory during retrieval. The term "cold query" refers to a query where the data in the database is stored on the hard drive, and the data is first loaded from the hard drive into memory during retrieval, and then the similarity calculation is performed between the data to be retrieved and the data loaded into memory. Furthermore, when memory is insufficient, a large amount of data may be stored on the hard drive, and the resulting low-performance cold queries can lead to significant time consumption.

[0057] Through the embodiments of this application, data compression can reduce memory usage, allowing more data to be stored in memory for hot queries, thereby optimizing the retrieval time of each piece of data and improving QPS performance in concurrent scenarios.

[0058] As artificial intelligence technology gradually integrates into all aspects of people's daily work and life (e.g., facial recognition), machine learning models have been widely used. In retrieval application systems based on machine learning models, the feature database pre-stores multiple feature data. The machine learning model can retrieve features from this database to determine those that match the features in the traffic (or requests), thereby obtaining the corresponding search results.

[0059] In some examples of embodiments of this application, the database may be a feature database that pre-stores feature data, and the data to be retrieved may be feature data to be retrieved.

[0060] Figure 4 A flowchart illustrating an example of a feature database retrieval method according to an embodiment of this application is shown.

[0061] like Figure 4 As shown, in step 410, based on the first machine learning model, the first similarity of each second compressed feature data relative to the first compressed feature data is determined.

[0062] In step 420, a predetermined number of compressed feature data are selected from each of the determined first similarities. For example, a predetermined number of data with the highest first similarities can be selected.

[0063] In step 430, based on the second machine learning model, the second similarity between the feature data corresponding to each third compressed feature data in the feature database and the feature data to be retrieved is determined.

[0064] It should be understood that the first and second machine learning models described herein are only used to distinguish different functional operations. Specifically, the first machine learning model can be used to calculate the similarity between the feature data to be retrieved during the coarse screening process and the compressed data corresponding to each data point in the database. The second machine learning model can be used to calculate the similarity between the feature data to be retrieved during the fine screening process and the feature data corresponding to the coarse screening results in the database. In some examples of embodiments of this application, the same machine learning model may be used to perform the operations described above for the first and second machine learning models.

[0065] In step 440, based on each second similarity, feature data that matches the feature data to be retrieved is determined in the feature database.

[0066] The embodiments of this application improve the QPS performance of servers in artificial intelligence application scenarios based on feature databases, such as the QPS performance of face retrieval application systems.

[0067] In some examples of embodiments of this application, the face retrieval application system uses short feature values ​​converted from the original feature values ​​to calculate similarity. The current short feature value occupies 128 bytes of storage space. The feature value conversion compression ratio is relatively low, and the query accuracy is high. However, the query time may still be relatively long.

[0068] As a further optimization, a larger compression ratio can be used during the feature value transformation process to shorten the transformed short feature values ​​to 64 bytes. It should be noted that feature value vectors lose accuracy due to compression, and the larger the compression ratio, the greater the accuracy loss. However, through extensive model training, the accuracy loss can be kept within a certain range, as illustrated by actual test results in other sections below.

[0069] In this embodiment, by sacrificing some accuracy, the retrieval performance of the feature database and other important performance indicators can be improved. To compensate for the above-mentioned deficiencies, this embodiment also proposes to perform a second full-precision similarity calculation (i.e., similarity calculation between the data to be retrieved and the uncompressed data in the database) on the Top N (i.e., a set number) results of the first retrieval, and by increasing the number of results in the first search result, the loss of accuracy can be compensated for by optimizing the processing flow strategy while reducing the space occupied and computing resources, ultimately improving QPS while meeting the accuracy requirements.

[0070] Database retrieval performance typically involves several metrics: retrieval speed, retrieval memory consumption, and retrieval result accuracy. Retrieval speed reflects the time spent on hot reloading after system startup, the time required to find the N most similar records to the query, and QPS in high-concurrency scenarios. Retrieval memory consumption reflects whether similarity calculations for feature data are primarily performed in memory, as retrieval on disk databases (even SSDs) is several orders of magnitude slower. Retrieval result accuracy reflects the degree of matching between the system's returned results and the actual results. It can typically be evaluated by calculating the number of true nearest neighbors appearing in the first position of the returned results (this metric is generally called 1-recall@1), or by measuring the average percentage of the first 10 returned results containing the 10 nearest neighbors.

[0071] Figure 5 A flowchart illustrating an example of a feature database retrieval method according to an embodiment of this application is shown.

[0072] like Figure 5 As shown, the database retrieval method 500 includes a hot loading stage 510 and a feature data retrieval stage 520.

[0073] In the hot loading stage 510, feature data compression is used to effectively reduce the total amount of data loaded into memory.

[0074] For example, after system startup, every piece of data in the database stored on the hard drive needs to be loaded into memory. If there are 100 million data entries, and the feature value of each data entry is 128 bytes, then the feature value alone will require 100 million * 128 bytes = 11.9 GB of memory space. This results in high system I / O usage and slow loading speed. Furthermore, some mapping relationships may also need to be loaded into memory, meaning that with limited memory resources, more data needs to be stored on the hard drive. If the search range is large, cold queries may be triggered, increasing search time and lowering QPS.

[0075] Therefore, when using compressed short feature values, such as compressing the feature values ​​of data to 64 bytes, the total amount of feature values ​​loaded will be reduced by half. With the same memory size, twice the amount of data can be stored in memory. This reduces the amount of cold data that needs to be loaded from the hard drive. After increasing the amount of data for hot queries, it will indirectly increase the QPS in concurrent scenarios, and the time spent in the hot loading data reading stage will be shortened by almost half.

[0076] In the feature data retrieval stage 520, a two-stage screening process of coarse and fine screening increases the number of candidate data processed during the fine screening process, thereby ensuring the accuracy of the identified target feature data. Furthermore, by increasing the compression ratio, the amount of data retrieved during the coarse screening process can be significantly reduced, improving retrieval efficiency.

[0077] It should be noted that upon receiving a retrieval request, the server can calculate similarity based on the feature data in the retrieval request and the feature data stored in the database in memory. However, calculating massive amounts of data consumes a significant amount of server CPU resources, with instantaneous CPU usage often exceeding 70%. If concurrent retrieval requests occur, the calculation speed will be greatly reduced. Furthermore, the CPU resource consumption for similarity calculation is highly correlated with the data length. For example, calculating the dot product of 512 floating-point numbers requires 512 multiplication and addition operations, while calculating the dot product of 256 floating-point numbers only requires 256 multiplication and addition operations. Therefore, a larger data length leads to a larger amount of similarity calculation and a longer processing time.

[0078] In some examples of embodiments of this application, by further compressing the data length of the database data, the computational load of database similarity calculation can be reduced to a great extent. For example, the resource consumption of similarity calculation between 128-byte feature values ​​is much higher than that of similarity calculation between 64-byte feature values.

[0079] Table 1 shows the actual retrieval time comparison results between feature data with a length of 128 bytes and feature data with a length of 64 bytes.

[0080] Data volume N topN Data length / byte Search time-consuming / ms QPS 3000 million 1000 128 70.8049 14 3000 million 1000 64 39.9712 25 6000 million 1000 128 137.407 7 6000 million 1000 64 75.541 13

[0081] Table 1

[0082] As shown in Table 1, the QPS corresponding to a data length of 64 bytes is much greater than the QPS corresponding to a data length of 128 bytes. The actual test results show that the QPS has increased by almost double.

[0083] As described above, while using compressed data for retrieval increases QPS, it leads to a decrease in accuracy.

[0084] Table 2 shows the measured accuracy results of comparing the top N results with the brute-force search for the highest similarity:

[0085] Number of search results Feature length-128 byte Feature length-64 byte top1 0.9977697 0.9944196 top5 0.9997585 0.9995169 top10 0.9992355 0.9990826 top20 0.999639 0.9990072 top100 0.9997925 0.9991195 top200 0.9997507 0.9990429 top1000 0.9993353 0.9895580

[0086] Table 2

[0087] In this embodiment, the range of results returned in the initial search is expanded, and a subsequent similarity correction process (or fine screening process) is used to improve the accuracy of the search results. This expands the range of results in the initial search to ensure that similarity errors caused by the compression loss of shorter feature values ​​do not exclude data with high actual similarity. Then, a subsequent high-precision similarity calculation corrects the actual similarity, ensuring the final search results are accurate.

[0088] It should be noted that search results with high similarity are generally prioritized for display. For example, users typically expect no more than 1000 results, but for various reasons, the topN value needs to be set larger. For instance, after filtering data based on similarity, further filtering based on individual attributes is required. Therefore, the topN value should be relatively large to avoid leaving too few results after attribute filtering. Furthermore, feature compression leads to precision loss, necessitating a wider range of results; for example, the topN value could be set to 10000.

[0089] It should be noted that the time-consuming aspects of concurrent retrieval are mainly the following two: first, the similarity calculation process for massive amounts of data; and second, low-performance cold queries performed when memory is insufficient and a large amount of data is stored on the hard drive. Specifically, high CPU resource consumption during the similarity calculation of massive amounts of data affects the computation time, and significant time is consumed during low-performance cold queries performed when memory is insufficient and a large amount of data is stored on the hard drive.

[0090] Through the embodiments of this application, the corresponding processing strategies can reduce CPU resource consumption and memory consumption, allowing more data to be stored in memory for hot queries. This can reduce memory consumption and improve QPS performance for hot loading and retrieval while ensuring accuracy.

[0091] Figure 6 A flowchart illustrating an example of a database retrieval method in a face retrieval application scenario according to an embodiment of this application is shown.

[0092] like Figure 6As shown, in step 610, a database retrieval request is obtained. For example, a user can issue a database retrieval request by operating a mobile terminal. Furthermore, after receiving the retrieval request, the server can parse the feature value information corresponding to the retrieval request, and can also issue or attach query conditions for this retrieval, including attribute information of the search object (such as start and end time range, gender, age, whether or not they wear glasses, etc.).

[0093] In step 620, the server can compress the feature data to be retrieved into short feature values, while also retaining the original feature values.

[0094] In step 630, the server calculates the similarity between the converted short feature values ​​and the short feature values ​​of the data loaded into the database in memory, and returns the top N = 10,000 results with the highest similarity, completing the initial search. For example, only the data ID and short feature values ​​may be loaded into memory, and only the data ID and its corresponding initial search similarity may be returned. Since the database contains a very large amount of data, using short feature values ​​(or compressed feature data) can significantly reduce computation time.

[0095] In step 640, for the returned topN=10000 results, the corresponding attribute information is retrieved from the database based on these topN data IDs, and then filtered according to the attribute information of the object to be retrieved, so that only the data that meets the attribute information is selected. At this time, the amount of data is generally less than 10000.

[0096] In step 650, a second similarity calculation is performed on the selected data. Here, the calculation method used is similarity calculation using the original feature values ​​(i.e., uncompressed feature values). For example, inner product calculation or Euclidean distance calculation can be used. Using inner product or Euclidean distance calculation with the original feature values ​​can guarantee full accuracy, and after the initial search and filtering, the data volume has been reduced to less than 10,000, and the time consumption is in the millisecond range.

[0097] In step 660, the results of the secondary similarity calculation are sorted, and the results are truncated and reported according to the user's return number requirements.

[0098] Based on the accuracy test results in Table 2 above, it can be seen that the accuracy of the short feature value scheme with a length of 64 is lower than that with a length of 128, indicating that there is a bias in the returned top 1 to top 1000 results. However, when using short feature values ​​with a length of 64, compared to using short feature values ​​with a length of 128, if the returned top value is set to 2000, then the true top 1000 results will all be in these 2000 results. Therefore, consider expanding the returned results of the initial search to 1000*2. This can ensure that the correct results of the initial search are all in the returned value, that is, set the N value of topN to 2N.

[0099] Furthermore, although the time spent searching using 2N data points relative to the data to be searched (i.e., brute force search) is greater than the time spent searching using N data points relative to the data to be searched, the time saved is far less than that saved by using a short feature value comparison of 64 bytes during the initial search (or coarse screening process). Moreover, the larger the amount of data in the database, the greater the reduction in time.

[0100] In this embodiment, by compressing feature values ​​into shorter feature values, the QPS in concurrent retrieval scenarios is improved. By performing similarity calculation in stages, the range of returned values ​​is expanded in the first similarity comparison, and a full-precision brute-force comparison is performed in the second comparison to ensure accuracy. This reduces storage costs while increasing the retrieval performance of the database.

[0101] Figure 7 A structural block diagram of an example database retrieval apparatus according to an embodiment of this application is shown.

[0102] like Figure 7 As shown, the database retrieval device 700 includes a retrieval data acquisition unit 710, a retrieval data compression unit 720, a database compressed data acquisition unit 730, a compressed data filtering unit 740, and a data matching unit 750.

[0103] The data retrieval unit 710 is used to acquire the data to be retrieved.

[0104] The data compression unit 720 is used to compress the data to be retrieved into first compressed data with a set data length.

[0105] The library compressed data acquisition unit 730 is used to acquire second compressed data with the set data length corresponding to each pre-stored original data in the database.

[0106] The compressed data filtering unit 740 is used to filter a set number of third compressed data that match the first compressed data from each of the second compressed data, wherein the set number is related to the set data length.

[0107] The data matching unit 750 is used to determine the original data that matches the data to be retrieved from the original data corresponding to the set number of third compressed data.

[0108] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0109] Figure 8 This is a schematic diagram illustrating an example of a server according to an embodiment of this application. Figure 8 As shown, the server 800 in this embodiment includes a processor 810, a memory 820, and a computer program 830 stored in the memory 820 and executable on the processor 810. When the processor 810 executes the computer program 830, it implements the steps described in the above-described three-dimensional object construction method embodiment, for example... Figure 1 Steps 110 to 140 are shown. Alternatively, when the processor 810 executes the computer program 830, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 7 The functions of units 710 to 750 are shown.

[0110] For example, the computer program 830 can be divided into one or more modules / units, which are stored in the memory 820 and executed by the processor 810 to complete this application. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 830 in the server 800. For example, the computer program 830 can be divided into a data retrieval module, a data retrieval compression module, a library compressed data retrieval module, a compressed data filtering module, and a data matching module, with the specific functions of each module as follows:

[0111] The data retrieval module is used to retrieve the data to be retrieved in the data retrieval request.

[0112] The data compression module is used to compress the data to be retrieved into first compressed data with a set data length.

[0113] The library compressed data acquisition module is used to acquire second compressed data with a set data length corresponding to each pre-stored data in the database.

[0114] The compressed data filtering module is used to filter a set number of third compressed data that match the first compressed data from each of the second compressed data, wherein the set number is related to the set data length.

[0115] The data matching module is used to determine data that matches the data to be retrieved from the data corresponding to the set number of third compressed data.

[0116] The server 800 can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. The server may include, but is not limited to, a processor 810 and a memory 820. Those skilled in the art will understand that... Figure 8 This is merely an example of server 800 and does not constitute a limitation on server 800. It may include more or fewer components than shown, or combine certain components, or different components. For example, the server may also include input / output devices, network access devices, buses, etc.

[0117] The processor 810 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0118] The memory 820 can be an internal storage unit of the server 800, such as the server 800's hard drive or memory. The memory 820 can also be an external storage device of the server 800, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the server 800. Furthermore, the memory 820 can include both internal storage units and external storage devices of the server 800. The memory 820 is used to store the computer program and other programs and data required by the server. The memory 820 can also be used to temporarily store data that has been output or will be output.

[0119] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments.

[0120] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0121] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0122] In the embodiments provided in this application, it should be understood that the disclosed apparatus / server and method can be implemented in other ways. For example, the apparatus / server embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0123] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0124] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The aforementioned units can be implemented in hardware or software.

[0125] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0126] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A database search method characterized by, Applied to a server configured with a database, the method comprises: obtaining data to be retrieved; compressing the data to be retrieved into first compressed data with a set data length; wherein the data length in the compressed database is 128 bytes or 64 bytes; obtaining second compressed data with the set data length corresponding to each pre-stored original data in the database; screening a set number of third compressed data matching the first compressed data from each of the second compressed data, the set number and the set data length having a correlation; determining original data matching the data to be retrieved from the original data corresponding to the set number of third compressed data; Before screening a set number of third compressed data matching the first compressed data from each of the second compressed data, the method further comprises: determining a set number corresponding to the set data length based on a preset data length mapping table, the data length mapping table storing a plurality of set data lengths and corresponding set numbers; Before screening a set number of third compressed data matching the first compressed data from each of the second compressed data, the method further comprises: determining length difference information between the set data length and the data length of the original data; the length difference information and the calibration amplitude of the set number have a positive correlation; Calibrating the set number based on the length difference information.

2. The database search method of claim 1, wherein, Before compressing the data to be retrieved into first compressed data with a set data length, the method further comprises: obtaining an expected data length in a data length configuration instruction, and determining the expected data length as the set data length.

3. The database search method of claim 1, wherein, The database is a feature database pre-storing feature data, and the data to be retrieved is feature data to be retrieved, wherein the screening a set number of third compressed data matching the first compressed data from each of the second compressed data comprises: determining a first similarity of each second compressed feature data relative to the first compressed feature data based on a first machine learning model; screening the set number of compressed feature data from the each second compressed feature data according to the determined each first similarity.

4. The database retrieval method of claim 3, wherein, The determining data matching the data to be retrieved from the data corresponding to the set number of third compressed data comprises: determining a second similarity of feature data corresponding to each third compressed feature data in the feature database relative to the feature data to be retrieved based on a second machine learning model; determining feature data matching the feature data to be retrieved in the feature database based on each of the second similarity.

5. The database search method of claim 1, wherein, The screening a set number of third compressed data matching the first compressed data from each of the second compressed data comprises: loading each of the second compressed data to a memory space of the server; screening a set number of third compressed data matching the first compressed data from each of the second compressed data stored in the memory space.

6. A database search apparatus characterized by comprising: Comprise: The retrieval data acquisition unit is configured to acquire data to be retrieved. The retrieval data compression unit is configured to compress the data to be retrieved into first compressed data with a set data length; wherein the data length in the compressed database is 128 bytes or 64 bytes. The database compression data acquisition unit is configured to acquire second compressed data with the set data length corresponding to each pre-stored original data in the database. The compressed data screening unit is configured to screen a set number of third compressed data matching the first compressed data from each of the second compressed data; the set number and the set data length have a correlation relationship. The data matching unit is configured to determine original data matching the data to be retrieved from original data corresponding to the set number of third compressed data. Before screening the set number of third compressed data matching the first compressed data from each of the second compressed data, the device is further configured to: determine the set number corresponding to the set data length based on a preset data length mapping table; the data length mapping table stores a plurality of set data lengths and corresponding set numbers; Before screening the set number of third compressed data matching the first compressed data from each of the second compressed data, the device is further configured to: determine length difference information between the set data length and the data length of the original data; the length difference information and the calibration range of the set number have a positive correlation relationship; calibrate the set number based on the length difference information.

7. A server comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the steps of the method of any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising: The computer program is executed by the processor to realize the steps of the method of any one of claims 1 to 5.

Images