Data retrieval method, device, equipment, storage medium and program product

Abstract

The present application discloses a data retrieval method, device, equipment, storage medium and program product, which relate to the field of machine learning. The method comprises: obtaining data to be retrieved and a data set; extracting a first hash feature of the data to be retrieved, which includes at least one level of residual hash feature; extracting a second hash feature of the data in the data set, which includes at least one level of residual hash feature. In the process of data retrieval, the first level hash feature and at least one level of residual hash feature are extracted for the data to be retrieved and the data in the data set, and when the data is retrieved, the at least one level of residual hash feature is used as the feature representation of the retrieval process, thereby improving the accuracy of data retrieval.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of machine learning, and in particular to a data retrieval method, apparatus, device, storage medium, and program product. Background Art

[0002] In image retrieval scenarios, it is usually necessary to extract image features through an image retrieval model and then match the image features with those in an image library, so as to find a target image that matches the image to be retrieved from the image library as a retrieval result.

[0003] In the related art, hash feature image representation is usually used to match images, that is, the hash features of the image to be retrieved are extracted, and the hash features of the images in the image library are extracted, and the two features are matched, so as to retrieve the image matching the image to be retrieved from the image library as the retrieval result.

[0004] However, in the related art, the accuracy of image retrieval is usually determined by the number of bits of the hash feature. Compared with the use of floating-point feature representation, the hash feature has quantization errors, so the representation effect of the hash feature is weaker at the same number of bits, resulting in poor accuracy. Summary of the invention

[0005] The embodiments of the present application provide a data retrieval method, apparatus, device, storage medium and program product, which can improve the accuracy of image retrieval. The technical solution is as follows.

[0006] In one aspect, a data retrieval method is provided, the method comprising:

[0007] Acquire data to be retrieved and a data set for retrieving the data to be retrieved;

[0008] Extracting a first hash feature of the data to be retrieved, wherein the first hash feature includes a first-level hash feature and at least one first-level residual hash feature of the data to be retrieved;

[0009] Extracting a second hash feature of the data in the data set, where the second hash feature includes a first-level hash feature and at least a first-level residual hash feature of the data in the data set;

[0010] Based on the first-level hash feature and at least one-level residual hash feature included in the first hash feature, and the first-level hash feature and at least one-level residual hash feature included in the second hash feature, retrieval result data matching the data to be retrieved is retrieved from the data set.

[0011] In another aspect, a data retrieval device is provided, the device comprising:

[0012] An acquisition module, used to acquire data to be retrieved and a data set for retrieving the data to be retrieved;

[0013] An extraction module, used for extracting a first hash feature of the data to be retrieved, wherein the first hash feature includes a first-level hash feature and at least a first-level residual hash feature of the data to be retrieved;

[0014] The extraction module is further used to extract a second hash feature of the data in the data set, wherein the second hash feature includes a primary hash feature and at least a primary residual hash feature of the data in the data set;

[0015] A retrieval module is used to retrieve retrieval result data matching the data to be retrieved from the data set based on the first-level hash features and at least one-level residual hash features included in the first hash features, and the first-level hash features and at least one-level residual hash features included in the second hash features.

[0016] On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement a data retrieval method as described in any of the above-mentioned embodiments of the present application.

[0017] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, a code set or an instruction set is stored in the storage medium, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement a data retrieval method as described in any of the above-mentioned embodiments of the present application.

[0018] On the other hand, a computer program product is provided, the computer program product includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes any of the data retrieval methods described in the above embodiments.

[0019] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0020] During the data retrieval process, first-level hash features and at least one-level residual hash features are extracted for the data to be retrieved and the data in the data set. During data retrieval, at least one-level residual hash features are used as the feature representation of the retrieval process, thereby compensating for the problem of weak hash representation caused by quantization errors through residual features, thereby improving the accuracy of data retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 A schematic diagram of an image retrieval process provided by an exemplary embodiment of the present application is shown;

[0023] Figure 2 It is a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application;

[0024] Figure 3 is a flow chart of a data retrieval method provided by an exemplary embodiment of the present application;

[0025] Figure 4 is based on Figure 3 A schematic diagram of a hash feature extraction process provided by the illustrated embodiment;

[0026] Figure 5 Another exemplary embodiment of the present application provides a flowchart of a data retrieval method;

[0027] Figure 6 is based on Figure 5 The illustrated embodiment provides a schematic diagram of the results of the hash feature in the search;

[0028] Figure 7 is based on Figure 5 A schematic diagram of a process of elastically merging recall graphs provided by the illustrated embodiment;

[0029] Figure 8 is based on Figure 5 The illustrated embodiment provides a schematic diagram of the Hamming distance calculation process;

[0030] Fig. 9 is based on Figure 5 A schematic diagram of introducing residual hash features provided by the illustrated embodiment;

[0031] Fig.10 is a flow chart of a data retrieval method provided by another exemplary embodiment of the present application;

[0032] Fig.11 is based on Fig.10 Schematic diagram of similar samples provided by the illustrated embodiment;

[0033] Fig.12 is a schematic diagram of a loss calculation process provided by an exemplary embodiment of the present application;

[0034] Fig.13 is a schematic diagram of a feature extraction solution provided by an exemplary embodiment of the present application;

[0035] Fig.14 is a structural block diagram of a data retrieval device provided by an exemplary embodiment of the present application;

[0036] Fig.15 is a structural block diagram of a data retrieval device provided by another exemplary embodiment of the present application;

[0037] Fig.16 It is a structural block diagram of a server provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0039] First, a brief introduction is given to the terms involved in the embodiments of the present application.

[0040] Artificial Intelligence (AI): It is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0041] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data retrieval technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0042] Machine Learning (ML): It is a multi-disciplinary interdisciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0043] Computer Vision (CV): It is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further processing graphics so that the computer processing becomes an image that is more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and mapping, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0044] Binary quantization: refers to the D-dimensional feature vector. After the vector is normalized, the value range is generally a floating point number between -1 and 1. The feature is compressed to a specified number of bits (such as 48 bits) with a binary code of 0 or 1 (called 48-bit compression). This is vector binary quantization, which can also be called binary encoding.

[0045] Binary quantization index: refers to obtaining a binary vector with a limited number of bits through a certain calculation process (model) of a D-dimensional feature vector, and using the binary vector as an index to recall the image during retrieval.

[0046] Deep hash representation of images: It is a binary quantization method that uses binary feature vectors to represent deep learning feature embedding of images, which can compress feature storage space (1 hash bit occupies 1 bit, compared to 1 floating point feature bit occupying 4 bytes, with a compression rate of 32 times). Compared with conventional deep learning feature embedding using floating point feature representation, deep hash representation has quantization error, so the hash representation effect is slightly weaker than floating point representation under the same number of bits.

[0047] To address the problem of weak hash feature representation, this embodiment provides a data retrieval method, which is described by taking an image retrieval process as an example. Figure 1 A schematic diagram of an image retrieval process provided by an exemplary embodiment of the present application is shown as follows: Figure 1 As shown, hash feature extraction is performed on the stock image 100, wherein the extracted hash feature 110 includes a first-level hash 111, a first-level residual hash 112 (or second-level hash), and a second-level residual hash 113 (or third-level hash). Figure 1 The three-level hash extraction is taken as an example for illustration. In some embodiments, the number of levels of residual hash extraction can be more or less, such as: only extracting the first-level hash and the second-level hash, or extracting the first-level hash, the second-level hash, the third-level hash and the fourth-level hash (i.e., the third-level residual hash), etc.

[0048] The first-level hash 111, the first-level residual hash 112, and the second-level residual hash 113 are clustered respectively to obtain a clustering result, that is, the indexes corresponding to the hashes at each level and the images under the indexes are obtained.

[0049] For the image to be retrieved 120 , a hash feature 130 of the image to be retrieved 120 is extracted, wherein the extracted hash feature 130 includes a first-level hash 131 , a first-level residual hash 132 , and a second-level residual hash 133 .

[0050] According to the accuracy requirement, the hash feature 130 of the image to be retrieved 120 is matched with the clustering result of the stock image 100, wherein the first-level hash 131 is matched with the index of the first-level hash 111. When the matching of the first-level hash 131 does not meet the accuracy requirement, the matching of the residual hash is continued until the matching retrieval result meets the accuracy requirement.

[0051] The data retrieval process obtained by training for this application includes at least one of the following scenarios when applied.

[0052] First, applied to the image retrieval scenario, schematically, the hash features of the image to be retrieved are extracted, including the first-level hash, first-level residual hash and second-level residual hash of the image to be retrieved; the hash features of the images in the image library are obtained, including the first-level hash, first-level residual hash and second-level residual hash of the image.

[0053] First, the first-level hash of the image to be retrieved is matched with the first-level hash of the image in the image library to obtain the matching result of the first-level hash; and based on the accuracy requirement, the first-level residual hash is matched with the first-level residual hash of the image in the image library, or the first-level residual hash and the second-level residual hash are matched with the first-level residual hash and the second-level residual hash of the image in the image library to obtain the matching result, and the various matching results are fused to obtain the image retrieval result.

[0054] Optionally, image retrieval scenarios can be applied to various fields such as image duplication elimination and originality protection.

[0055] Second, it is applied to the audio similarity analysis scenario. Schematically, after receiving the voice signal, the voice signal is matched with the voice signals of each object in the voice library for similarity, so as to determine the object to which the received voice signal belongs. In the matching process, the hash features of the voice signal are extracted, including the first-level hash, the first-level residual hash and the second-level residual hash, and the hash features of the voice signals of each object in the voice library are obtained, including the first-level hash, the first-level residual hash and the second-level residual hash. Thus, at least one first-level residual hash feature is selectively used for matching.

[0056] It is worth noting that the “first”, “second”, etc. involved in the embodiments of the present application are merely illustrative examples. For example, the “first” and “second” in the above application scenario examples are merely illustrative lists and do not limit the order of expression of the application scenarios.

[0057] In addition, the above-mentioned images and voice signals to be retrieved are data actively uploaded by the user; or, data obtained after the user's separate authorization. In the above embodiments, the image retrieval scenario and the audio similarity analysis scenario are used as examples for explanation. The data retrieval method provided in the embodiments of the present application can also be applied to other scenarios, without limitation.

[0058] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user alone or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions. For example, the image data involved in this application are all obtained with full authorization.

[0059] Next, the implementation environment involved in the embodiments of the present application is described. For illustration, please refer to Figure 2 The implementation environment involves a terminal 210 and a server 220 , and the terminal 210 and the server 220 are connected via a communication network 230 .

[0060] In some embodiments, the terminal 210 is used to send the data to be retrieved to the server 220. In some embodiments, an application with a data similarity analysis function (such as a data retrieval function) is installed in the terminal 210. Schematically, an application with an image retrieval function is installed in the terminal 210. For example, a search engine program, an instant messaging application, a video program, a game program, etc. are installed in the terminal 210, which is not limited in the embodiments of the present application.

[0061] The server 220 includes a data retrieval model. After the data to be retrieved is input into the data retrieval model, the data retrieval result is output and fed back to the terminal 210, so that the terminal 210 displays or voice broadcasts the data retrieval result.

[0062] Among them, when the data retrieval model retrieves data, it first extracts the hash features of the data to be retrieved, including the first-level hash, first-level residual hash and second-level residual hash of the data to be retrieved. Therefore, in the process of data retrieval based on the first-level hash, the first-level residual hash or the first-level and second-level residual hash are selectively used for data retrieval according to the retrieval accuracy requirements to obtain the data retrieval results.

[0063] The above-mentioned terminal can be a mobile phone, a tablet computer, a desktop computer, a portable laptop computer, a smart TV, a car terminal, a smart home device and other terminal devices in various forms, which is not limited in the embodiments of the present application.

[0064] It is worth noting that the above-mentioned servers can be independent physical servers, or they can be server clusters or distributed systems composed of multiple physical servers. They can also be cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0065] Among them, cloud technology refers to a hosting technology that unifies hardware, software, network and other resources in a wide area network or local area network to realize data computing, storage, processing and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool, which is used on demand and flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the high development and application of the Internet industry, in the future, each item may have its own identification mark, and all need to be transmitted to the background system for logical processing. Data of different levels will be processed separately. All kinds of industry data require strong system backing support, which can only be achieved through cloud computing.

[0066] In some embodiments, the above server can also be implemented as a node in a blockchain system.

[0067] In combination with the above-mentioned noun introduction and application scenarios, the data retrieval method provided by the present application is described. The method can be executed by a server or a terminal, or can be executed by a server and a terminal together. In the embodiment of the present application, the method is described by taking the execution of the server as an example. Figure 3 As shown, the method includes the following steps.

[0068] Step 301: Acquire data to be retrieved and a data set used to retrieve the data to be retrieved.

[0069] Optionally, the data to be retrieved and the data in the data set are of the same data type, such as: the data to be retrieved and the data in the data set are image data, voice data, text data, etc., which is not limited in this embodiment of the present application.

[0070] Optionally, taking the data to be retrieved as image data as an example, the data set is a data set obtained from a preset image library, such as a data set obtained from a large-scale general object recognition open source data set. In some embodiments, the data set includes data with different data contents. Taking image data as an example, the image content of the image data includes animals such as cats, dogs, sheep, and pigs, or vehicles such as airplanes, cars, and ships.

[0071] The method of obtaining the data to be retrieved includes at least one of the following methods:

[0072] First, receiving the data to be retrieved uploaded through the terminal;

[0073] In an illustrative example, the data to be retrieved is image data. During the application process of the terminal application, the image retrieval function in the application is used, thereby uploading the image data to be retrieved to the server through the image retrieval function. When uploading the image data to be retrieved, the terminal obtains the user's separate authorization, reads the images stored in the terminal, and determines the image data to be retrieved and uploads it to the server.

[0074] Second, obtain the data to be searched from the database to be searched;

[0075] The to-be-retrieved library stores one or more data to be retrieved currently, and data is randomly selected from the library as the current data to be retrieved; or, data is selected from the library in order as the current data to be retrieved.

[0076] It should be noted that the above method of obtaining the data to be retrieved is only an illustrative example and is not limited to this in this embodiment.

[0077] Step 302: extract a first hash feature of the data to be retrieved, where the first hash feature includes a first-level hash feature and at least one-level residual hash feature of the data to be retrieved.

[0078] The hash features in the embodiments of the present application are expressed in the form of a primary hash, a primary residual hash Rhash1, and a secondary residual hash Rhash2. In the absence of ambiguity, the primary hash is also called the primary hash, the primary residual hash is also called the secondary hash, and the secondary residual hash is also called the tertiary hash. The index corresponding to the primary hash is the primary index, the index corresponding to the primary residual hash is the secondary index, and the index corresponding to the secondary residual hash is the tertiary index.

[0079] Optionally, the data to be retrieved is input into a data retrieval model, a data feature representation of the data to be retrieved is extracted through the data retrieval model, and a hash conversion is performed on the data feature representation to obtain a first hash feature of the data to be retrieved.

[0080] Among them, since the first hash feature includes a first-level hash feature and at least one-level residual hash feature, the floating-point feature representation corresponding to the data feature representation is first extracted, and the floating-point feature representation is cross-processed with hash features to obtain the first-level hash feature of the data to be retrieved. Based on the difference between the first-level hash feature and the floating-point feature representation, at least one-level residual hash feature is obtained.

[0081] For illustration, please refer to Figure 4 , which shows a schematic diagram of a hash feature extraction process provided by an exemplary embodiment of the present application, such as Figure 4As shown, first, the feature extraction of the data to be retrieved 400 is performed through the convolution layer to obtain a data feature representation 410, and the floating-point feature representation 420 corresponding to the data feature representation is extracted, and hash feature cross processing is performed to obtain a main hash 430; the main hash 430 and the floating-point feature representation 420 are subjected to a difference operation to obtain a first-level residual 440, and the first-level residual is subjected to a hash feature cross processing to obtain a first-level residual hash 450; the first-level residual hash 450 and the first-level residual 440 are subjected to a difference operation to obtain a second-level residual 460, and the second-level residual is subjected to a hash feature cross processing to obtain a second-level residual hash 470; and so on.

[0082] Step 303: extract a second hash feature of the data in the data set, where the second hash feature includes a first-level hash feature and at least one-level residual hash feature of the data in the data set.

[0083] Optionally, the second hash feature of the data in the data set is extracted in advance through a data retrieval model; or, the second hash feature of the data in the data set is extracted in real time during a data retrieval task.

[0084] The extraction process of the second hash feature is similar to the extraction process of the first hash feature, which will not be repeated here.

[0085] In some embodiments, hash features are extracted from each data in the data set to obtain second hash features corresponding to each data in the data set, that is, one data in the data set corresponds to one or a group of second hash features.

[0086] It is worth noting that the above-mentioned step 302 and step 303 can be two parallel steps, or they can have a sequence, and the present application does not limit this. That is, step 302 can be executed first and then step 303, or step 303 can be executed first and then step 302, or step 302 and step 303 can be executed simultaneously.

[0087] Step 304, based on the first-level hash feature and at least one-level residual hash feature included in the first hash feature, and the first-level hash feature and at least one-level residual hash feature included in the second hash feature, retrieve the retrieval result data matching the data to be retrieved from the data set.

[0088] Optionally, when at least one level of residual hash features in the first hash feature is used for retrieval, at least one level of residual hash features in the second hash feature is also used synchronously. The level of residual hash features used in the first hash feature is the same as the level of residual hash features used in the second hash feature. For example: use the first level hash, the first level residual hash feature, and the second level residual hash feature in the first hash feature for data retrieval; then use the first level hash, the first level residual hash feature, and the second level residual hash feature in the second hash feature synchronously to match the first hash feature.

[0089] In some embodiments, the first-level hash features and at least one-level residual hash features included in the second hash feature are clustered to obtain at least two cluster centers, and the cluster centers are used to indicate the similarity relationship between data in the data set; based on the feature distance relationship, a target cluster center that matches the first hash feature is determined from the at least two cluster centers, and retrieval result data that matches the data to be retrieved is determined from the candidate data associated with the target cluster center.

[0090] To summarize, the method provided in this embodiment, during the process of data retrieval, extracts first-level hash features and at least first-level residual hash features for the data to be retrieved and the data in the data set, and during data retrieval, uses at least first-level residual hash features as the feature representation of the retrieval process, thereby compensating for the problem of weak hash representation caused by quantization errors through residual features, thereby improving the accuracy of data retrieval.

[0091] During the data retrieval process, first-level hash features and at least one-level residual hash features are extracted for the data to be retrieved and the data in the data set. During data retrieval, at least one-level residual hash feature is selectively used as the feature representation of the retrieval process, so that different hash features are used for retrieval for different scenarios and different retrieval accuracies, which improves the flexibility of data retrieval, avoids resource waste for scenarios with low retrieval accuracy requirements, and avoids the problem of low retrieval accuracy for scenarios with high retrieval accuracy requirements.

[0092] The method provided in this embodiment innovatively introduces a residual feature learning mechanism, so that each level of residual hash features is a supplement to the hash features of the previous level, thereby achieving the effect of gradually approximating floating-point features; and the combination of multiple levels of hash representations can obtain applications that can support different data distribution / precision requirements.

[0093] In an optional embodiment, the clustering of the second hash features includes clustering of the first-level hash features and clustering of at least one-level residual hash features. Figure 5is a flowchart of a data retrieval method provided by another exemplary embodiment of the present application. The method can be executed by a server or a terminal, or by both a server and a terminal. In the embodiment of the present application, the method is executed by a server as an example. Figure 5 As shown above Figure 3 The illustrated step 304 may also be implemented as the following steps.

[0094] Step 3041, clustering the primary hash features of the data in the data set to obtain n first cluster centers, where n is a positive integer.

[0095] In some embodiments, the primary hash features of each data in the data set are obtained, and the quantized center of the primary hash features, such as the kmeans clustering center, is obtained. Optionally, the quantized features of each data are clustered by kmeans, thereby clustering into 1024 clustering centers, each of which can be regarded as a quantized center.

[0096] Optionally, before implementing the above clustering, it is necessary to train the cluster centers of hash features at all levels, such as using binary deep hash vectors of all inventory data to train 1024 cluster centers for the main hash. This cluster center will be used as an index for subsequent retrieval, and each index vector is a cluster center vector. Thus, an index list corresponding to the first-level hash feature is obtained, where each index in the index list is a center vector, that is, the index list stores 1024 center vectors obtained by clustering the first-level hash features of each data in the data set.

[0097] Step 3042: cluster at least one level of residual hash features of the data in the data set respectively to obtain m second cluster centers corresponding to the residual hash features, where m is a positive integer.

[0098] Optionally, at least one level of residual hash features of data in the data set are clustered respectively to obtain at least two second cluster centers corresponding to each level of residual hash features, wherein the at least two second cluster centers corresponding to each level of residual hash features constitute the above-mentioned m second cluster centers.

[0099] Optionally, the clustering of the first-level residual hash features and the clustering of the second-level residual hash features are used as examples for illustration. The first-level residual hash features of each data in the data set are obtained, and the quantization center of the first-level residual hash features, such as the kmeans clustering center, is obtained. Optionally, the first-level residual quantization features of each data are kmeans clustered, so as to be clustered into 1024 clustering centers, and each clustering center can be regarded as a quantization center.

[0100] The secondary residual hash features of each data in the data set are obtained, and the quantization center of the secondary residual hash features, such as the kmeans clustering center, is obtained. Optionally, the secondary residual quantization features of each data are clustered by kmeans to cluster into 1024 clustering centers, each of which can be regarded as a quantization center.

[0101] It is worth noting that the above cluster centers are illustrated using 1024 cluster centers as an example. In some embodiments, the number of cluster centers may be more or less. In addition, the number of cluster centers obtained by clustering the first-level hash features, the first-level residual hash features, and the second-level residual hash features may be the same or different, which is not limited in this embodiment.

[0102] It is worth noting that the above-mentioned step 3041 and step 3042 can be two parallel steps, or they can have a sequence, and the present application does not limit this. That is, step 3041 can be executed first and then step 3042, or step 3042 can be executed first and then step 3041, or step 3041 and step 3042 can be executed simultaneously.

[0103] Step 3043: Determine a target cluster center that matches the first hash feature from at least two cluster centers based on the feature distance relationship.

[0104] Optionally, after determining the cluster center, first construct an index system corresponding to the data set. Take the example of clustering the first-level hash features, the first-level residual hash features, and the second-level residual hash features to obtain the cluster center, use the cluster center (i.e. the above-mentioned quantitative center) as the retrieval index for bucket retrieval, and establish an association between the index and the data set (the quantitative center and the hash features in the data set); the index generated by the first-level hash feature cluster center is called the first-level index; the index generated by the first-level residual hash feature cluster center is called the second-level index; the index generated by the second-level residual hash feature cluster center is called the third-level index. Each index is associated with the data in the data set. Since each index has different ability to distinguish data, please refer to the schematic diagram. Figure 6 For the first-level index 610, all images with global differences can be distinguished, but the granularity of distinction is coarse, and multiple types of images may be mixed in the same cluster center, such as: the same cluster center contains chickens or buses of different colors / shapes, or the same cluster center contains sketches and real images of airplanes; for the second-level index 620, since it supplements the details relative to the first-level index 610, that is, the first-level residual hash feature, the sketches and real airplanes can be divided into different centers; for the third-level index 630, since it further supplements the details on the basis of the first-level index 610 and the second-level index 620, that is, the second-level residual hash feature, the sketches of buses and airplanes can be distinguished.

[0105] In some embodiments, a first candidate cluster center that matches the first-level hash feature included in the first hash feature is determined from n first cluster centers, and in response to the data associated with the first cluster center being within a preset number range, a second candidate cluster center that matches at least one first-level residual hash feature included in the first hash feature is determined step by step from m second cluster centers, thereby determining the target cluster center based on the first candidate cluster center and the second candidate cluster center.

[0106] Step 3044: Determine the search result data that matches the data to be searched from the candidate data associated with the target cluster center.

[0107] Optionally, a hash distance between the first hash feature and the second hash feature of the candidate data is obtained, and candidate data whose hash distance with the first hash feature meets the distance requirement is determined from the candidate data associated with the target cluster center as retrieval result data.

[0108] Optionally, the number of different feature bits between the first hash feature and the second hash feature of the candidate data is determined; based on the number of different feature bits, a hash distance between the first hash feature and the second hash feature of the candidate data is determined, that is, a Hamming distance between the first hash feature and the second hash feature is determined. Schematically, there are three different feature bits between 1000 and 0011, so the distance is 3.

[0109] In some embodiments, during retrieval, based on the hash features of the image to be retrieved (first-level hash features, first-level residual hash features, and second-level residual hash features), the k centers closest to each hash feature on the corresponding index system are found under each cluster center list, and under the index system recall, the images recalled from the 3-level index are obtained based on the cluster centers and their associated images.

[0110] Indicative, such as Figure 7 As shown, for the image to be retrieved 700, assuming k=1, the three closest cluster centers of 1000, 1000, and 0011 are calculated respectively under the first, second, and third level indexes, so as to recall the images associated with these three cluster centers.

[0111] It is worth noting that the performance and time consumption of multi-level indexes are similar to those of single-level indexes. All three indexes use an index center of 1024, and each index uses a 128-bit hash vector and is associated with the same data set. Therefore, the time consumption for retrieving and recalling the image to be retrieved in the three-level index is similar, and thus the performance and time consumption of multi-level indexes are similar to those of single-level indexes.

[0112] In some embodiments, the recall of the primary index and the multi-level index can be elastically merged, that is, after obtaining the associated images of the multi-level index to obtain the index recall, the index recall elastic merging calculation is performed to obtain the candidate image retrieval recall. Figure 7 As shown in the figure, the elastic merge process includes:

[0113] (1) When the data volume of the first-level index recall 710 images is greater than or equal to the target number range, the residual index recall is not merged;

[0114] (2) When the data amount of the first-level index recall 710 image is less than the target number range, the first-level index recall 710 and the second-level index recall 720 are merged;

[0115] (3) After the first-level index recall 710 and the second-level index recall 720 are merged, if the data volume is greater than or equal to the target quantity range, the third-level index recall 730 is not merged; otherwise, the third-level index recall 730 is merged.

[0116] In another optional embodiment, in certain scenarios where index recall needs to be improved, it can be specified that the first-level index recall and the second-level index recall need to be merged, and the third-level index recall serves as an elastic merged index.

[0117] Optionally, the Hamming distance is calculated based on the total hash features of the candidate image and the total hash features of the image to be retrieved and sorted from small to large; the total hash of multiple residuals combined here can approximate the floating point feature measurement effect, thereby reducing the problem of precision recall reduction caused by quantization error and improving the accuracy of sorting. The above total hash representation is the representation of the joint multi-residual hash. Schematically, the total hash representation is [hash, Rhash1, Rhash2], which is a 1×384 hash vector.

[0118] Indicative, such as Figure 8 As shown, in the process of calculating the Hamming distance based on the total hash representation, after calculating the Hamming distance between the total hash feature 810 of the image to be retrieved and the total hash feature 820 of the candidate image, the candidate images are arranged in ascending order according to the Hamming distance.

[0119] In other embodiments, Fig. 9As shown, the first-level hash feature 910, i.e., the main hash of 1×128, is first used as the sorting feature to reduce the time consumption of sorting calculation. When the first-level hash feature 910 is insufficient, that is, when two recalled images have the same Hamming distance with the image to be retrieved in the sorting, the first-level hash feature 910 and the first-level residual hash feature 920 of the two images are introduced to calculate the sorting. When the first-level residual hash feature 920 is insufficient, the second-level residual hash feature 930 is introduced. In this way, the recall accuracy can be guaranteed. The flexibility refers to (1) the samples that need to calculate the distance can be flexibly selected. For example, when calculating the total hash sorting in the figure below, not all sorted samples need to calculate the total hash sorting. Only samples with the same hash distance in the previous time are selected. Since the number of samples that need to be flexibly calculated and sorted for each image to be retrieved may not be the same, it is called elastic selection of samples; (2) the number of hash levels used to calculate the distance is flexibly selected. For each image to be retrieved, whether the next level of residual hashing is used to merge and calculate the new Hamming distance is determined based on whether the recall sorting effect has the same distance.

[0120] Optionally, after elastically sorting the candidate images, the first k images are taken as recall results, where k is a positive integer.

[0121] To summarize, the method provided in this embodiment extracts first-level hash features and at least first-level residual hash features for the data to be retrieved and the data in the data set during the data retrieval process, and selectively uses at least first-level residual hash features as feature representations of the retrieval process during data retrieval, thereby using different hash features for retrieval in different scenarios and with different retrieval accuracies, thereby improving the flexibility of data retrieval, avoiding waste of resources for scenarios with lower retrieval accuracy requirements, and avoiding the problem of low retrieval accuracy for scenarios with higher retrieval accuracy requirements.

[0122] The method provided in this embodiment uses elastic recall merging to determine whether to further recall more samples to supplement the search results based on the amount of data recalled. For the problem of data imbalance, elastic indexing can supplement the recall of some image field searches with very little data, thereby improving the richness of recall in scenes with little data. Elasticity means that during the entire search process, whether to merge the residuals is automatically triggered according to the search needs.

[0123] The method provided in this embodiment can flexibly trigger whether to continue residual retrieval according to a certain recall effect to solve the problem of uneven data distribution during retrieval; it can also construct residual hash multi-level indexes for inventory data volumes with different hashes during index construction, thereby supporting asymmetric index retrieval to solve the problem of insufficient hash bit representation of specific channels.

[0124] The method provided in this embodiment uses a method of elastically sorting candidate images, which can save computing time and improve sorting efficiency in businesses with huge data volumes, compared to a method of only using total hash calculation sorting.

[0125] In an optional embodiment, the above data retrieval model needs to be trained before application. Fig.10 is a flowchart of a data retrieval method provided by another exemplary embodiment of the present application. The method can be executed by a server or a terminal, or by both a server and a terminal. In the embodiment of the present application, the method is executed by a server as an example. Fig.10 As shown, in Figure 3 Before step 302 shown, the method further includes the following steps.

[0126] Step 1001, obtaining a sample triplet, wherein the sample triplet includes anchor point data, positive sample data, and negative sample data.

[0127] Among the anchor data (anchor), positive sample data (positive) and negative sample data (negative), the anchor data and the positive sample data are a pair of similar samples, and the anchor data and the negative sample data are a pair of dissimilar samples. That is, the anchor data and the positive sample data meet the similarity requirements, and the anchor data and the negative sample data do not meet the similarity requirements.

[0128] The sample triples are pre-labeled with triple relations. The triple relations labeled with the sample triples are used to indicate the anchor data, positive sample data, and negative sample data constituting the sample triples. Schematically, data A is labeled with label (a1), data B is labeled with label (p1), and data C is labeled with label (n1). Then data A, data B, and data C constitute a set of sample triples, data A is the anchor data in the sample triple, data B is the positive sample data in the sample triple, and data C is the negative sample data in the sample triple.

[0129] Optionally, the data in the sample triplet itself is unlabeled data, or the labels annotated with the data in the sample triplet are labels that do not need to be used in the embodiments of the present application.

[0130] Step 1002: extract hash features of sample triples through candidate retrieval models.

[0131] The candidate retrieval model is the model currently to be trained.

[0132] The hash features of the sample triples include the anchor hash features of the anchor data, the positive sample hash features of the positive sample data, and the negative sample hash features of the negative sample data.

[0133] Since it is difficult to obtain suitable triplets from massive data, negative sample mining is performed through positive sample pairs by annotating positive sample pairs to obtain sample triplets.

[0134] The process includes: (1) labeling positive sample pairs: taking images as an example, labeling whether the image sample pairs are similar, such as extracting two images from a large amount of data as a pair, labeling, and labeling whether each pair is similar enough. In some embodiments, two samples need to be extremely similar to be considered similar samples, or similar samples are other images generated by image attacks, such as Fig.11 As shown, images 1110 and 1120 are similar samples. Attacks include multiple types such as color change, chromaticity change, cropping, brightness, filters, etc. The sample pairs marked as similar are positive sample pairs, and the sample pairs marked as dissimilar are negative sample pairs. This annotation mainly collects positive sample pairs, and negative sample pairs do not need to be collected. The negative samples of triples can be obtained through subordinate mining methods.

[0135] (2) Triple data mining: Since the training metric learning features need to learn the loss function from the sample triples, the anchor and positive in the sample triples constitute the positive sample pair, and the anchor and negative constitute the negative sample pair. In the learning task, the positive sample pair needs to have features that are close enough - the distance is small enough (so that they can be retrieved from each other), while the negative sample pair needs to be far enough away. Each sample pair that has been obtained in the above annotation can be used as the anchor and positive of the triple (randomly select a picture as the anchor).

[0136] In the process of negative sample mining, due to the limited memory of the computer GPU and the need to send all positive sample pairs to the training in batches in the overall training, the mining of negative samples is carried out within a batch. For each batch of positive sample pairs (assuming there are bs pairs), the following methods are used to mine negative samples to obtain triplets: For the x-anchor in a certain sample x pair (randomly select one as the anchor): calculate the distance between it and the x-anchor from the remaining bs-1 sample pairs (one image is randomly selected for each pair), sort them by distance from small to large, remove the top 5 images, and take the first 20 samples as difficult negative samples (because what needs to be learned is the features of extremely similar samples, it is believed that the smaller the distance, the more similar, and in massive data, the probability of two images being extremely similar is relatively low, so the top 5 similar samples are directly removed, and the remaining samples can constitute the difficult negative samples in the triplets), respectively, and x to form triplets, so each sample pair generates 20 triplets, and the entire batch obtains 20×bs triplets. In order to ensure that negative samples are effective for mining, bs needs to be set to a relatively large value, such as 1024.

[0137] Triplet-based metric learning has many requirements for difficult samples. If all samples are simple, the model cannot learn discriminative representations.

[0138] In an optional embodiment, the anchor hash feature includes the first-level hash feature of the anchor data and at least one-level residual hash feature; the positive sample hash feature includes the first-level hash feature of the positive sample data and at least one-level residual hash feature; the negative sample hash feature includes the first-level hash feature of the negative sample data and at least one-level residual hash feature. The method of extracting the hash feature of the above-mentioned sample triple is similar to the method of extracting the above-mentioned first hash feature and second hash feature, except that the first hash feature and the second hash feature are extracted using the trained data retrieval model, while the hash feature of the sample triple is extracted using the candidate retrieval model to be trained.

[0139] Step 1003: determine the loss value based on the distance relationship between the anchor point hash feature, the positive sample hash feature, and the negative sample hash feature.

[0140] Optionally, the loss value includes triplet loss and / or regression loss.

[0141] Among them, the two-level residual hash features are used to explain the triple loss and regression loss respectively.

[0142] Triplet loss:

[0143] The triplet loss is calculated for the first-level hash features, first-level residual hash features, and second-level residual hash features in the sample triplet.

[0144] Optionally, the first loss is determined by the difference relationship between the first-level hash features of the anchor data, the first-level hash features of the positive sample data, and the first-level hash features of the negative sample data; and the second loss is determined by the difference relationship between at least one first-level residual hash feature of the anchor data, at least one first-level residual hash feature of the positive sample data, and at least one first-level residual hash feature of the negative sample data, and the first loss and the second loss are weightedly fused to obtain a triplet loss.

[0145] Using the sample triplets mined from batch, the distance between the anchor sample data and the negative sample data in the sample triplet needs to be large enough to ensure that the triplet can be distinguished in the quantization space, so the difference target needs to be set larger. Considering that each bit of the 128-dimensional quantization vector will eventually be close to the value of -1 or 1, the reference margin0 is preset to 100. The formula for the triplet loss is shown in Formula 1 below:

[0146] Formula 1: L triplet =max(||Q a -Q p ||-‖Qa -Q n ‖+α,0)

[0147] Where α is margin, set to 100, ||Q a -Q p || represents the L2 distance between the quantized output Q of the two images of the anchor data a and the positive sample data p in the triplet. The purpose of the triplet loss is to make the distance between the anchor data and the negative sample data greater than the distance from the positive sample data by margin. For the three triplet losses, the hash features used are the first-level hash feature vector, [hash1, hash2] concatenation vector, and [hash1, hash2, hash3] concatenation vector; hash1 refers to the hash feature vector, hash2 refers to the first-level residual hash feature, and hash3 refers to the second-level residual hash feature.

[0148] Regression loss:

[0149] Optionally, the anchor hash feature is quantized and encoded to obtain a first target code of the anchor hash feature; the positive sample hash feature is quantized and encoded to obtain a second target code of the anchor hash feature; the negative sample hash feature is quantized and encoded to obtain a third target code of the anchor hash feature; and the regression loss is obtained based on the first target code, the second target code and the third target code.

[0150] Taking the first-level hash feature as an example, the loss of quantization effect (whether it is close enough to -1 or 1) is calculated for the vector output by the quantization branch. Since the final application needs to be mapped to -1 / 1 after quantization output, if each bit output by each level of hash features is close enough to 1 or -1, then the above triplet loss is sufficient to describe the application of the quantization feature. Otherwise, even if the measurement effect of the triplet loss is good, it does not mean that the quantization representation in the application is good enough. Therefore, it is hoped that each bit output by each level of hash features is close enough to 1 or -1.

[0151] For the quantization results of each image, the loss function that meets this goal is shown in the following formulas 2 and 3:

[0152] Formula 2:

[0153] Formula 3:

[0154] Where Q i is the value of the quantized feature Q of the image at the i-th bit (in the embodiment of the present application, the quantized feature Q has a total of 128 bits), B i is the quantization target of the i-th position, B i By Hash iGenerated by sign function - the target code of the quantitative learning task is generated by using the sign function (such as the sign function shown in the above formula 3, the target code Bi of each hash value is calculated by the sign function, and the final target code of the hash is B).

[0155] Then, regression loss is used to reduce the L2 distance between the hash vector and the target code B. In the application, the coding loss of the three-level hash features needs to be calculated respectively, that is, the following loss is calculated for hash1, hash2, and hash3, all of which have a length of 1×128. Where hash1 refers to the hash feature vector, hash2 refers to the first-level residual hash feature, and hash3 refers to the second-level residual hash feature.

[0156] Weight: Since the encoded regression loss converges faster than the triplet loss, and the importance of encoding is lower than the feature measurement ability, in order to ensure that the triplet loss is dominant in the overall loss value, thereby ensuring that the encoded feature always has the ability to measure similarity, the weight w1 of the triplet loss is set to 1, and the weight w2 of the regression loss is set to 0.01 (or other values ​​less than 1, which can be adjusted as needed).

[0157] Schematically, the loss function is shown in the following formula 4:

[0158] Formula 4:

[0159]

[0160] j represents the jth triplet, and k represents the kth triplet. triplet1 represents the triplet loss of the first-level hash feature in the sample triplet; L triplet2 represents the triplet loss of the first-level residual hash feature in the sample triplet; L triplet3 L represents the triplet loss of the secondary residual hash feature in the sample triplet. coding1 represents the regression loss of the first-level hash feature in the sample triple; L coding2 represents the regression loss of the first-level residual hash feature in the sample triple; L coding3 Represents the regression loss of the second-level residual hash features in the sample triplet.

[0161] Indicative, Fig.12 FIG. 2 shows a schematic diagram of a loss calculation process provided by an exemplary embodiment of the present application. Fig.12 As shown, after obtaining the sample triplet 1200, the sample triplet 1200 is input into the candidate retrieval model 1210, and the hash features of the data in the sample triplet 1200 are extracted through the candidate retrieval model 1210 to obtain the main hash, the first-level residual hash and the second-level residual hash of the data in the sample triplet 1200.

[0162] According to the main hash calculation of the data in the sample triplet 1200, the triplet loss 1221 and the regression loss 1222 corresponding to the main feature metric learning are obtained; according to the first-level residual hash calculation of the data in the sample triplet 1200, the triplet loss 1231 and the regression loss 1232 corresponding to the main feature metric learning are obtained; according to the second-level residual hash calculation of the data in the sample triplet 1200, the triplet loss 1241 and the regression loss 1242 corresponding to the main feature metric learning are obtained.

[0163] Thus, the triplet loss 1221, the regression loss 1222, the triplet loss 1231, the regression loss 1232, the triplet loss 1241 and the regression loss 1242 are weightedly summed as shown in the above formula 4 to obtain the loss value.

[0164] Among them, Fig.12 The model structure of the candidate retrieval model 1210 shown includes: Fig.13 At least one of the situations shown. Solution 1310: The input information first passes through a fully connected layer (FC) with a larger channel to perform cross-learning of feature internal information, and then obtains the target hash vector through another FC; Solution 1320: The input information directly obtains the target hash through the FC layer.

[0165] The main hash and the first and second level residual hash are learned for each image separately. The different levels of metric learning constraints and model iteration of multi-level hashing ensure more hash bits (combined multi-level residual hash representation) which can make the hash representation effect closer to the floating-point feature representation effect.

[0166] Step 1004, training the candidate retrieval model based on the loss value to obtain a data retrieval model.

[0167] Optionally, model parameters in the candidate retrieval model are adjusted according to the loss value to obtain a data retrieval model.

[0168] Optionally, the model structure of the candidate retrieval model is as follows.

[0169] The basic feature model training adopts the resnet101 network, and the parameters are shown in Table 1 below; the deep feature enters the network result shown in Table 2 to generate a floating-point embedding, and the floating-point embedding is input into the hash feature cross module shown in Table 3 to obtain the main hash; the main hash is subtracted from the floating-point embedding to obtain a first-level floating-point residual, and the floating-point residual enters the first-level residual hash feature cross module shown in Table 4 to obtain a first-level residual hash; the first-level floating-point residual is subtracted from the first-level residual hash to obtain a floating-point residual, and the floating-point residual enters the second-level residual hash feature cross module shown in Table 5 to obtain a second-level residual hash.

[0170] Table 1 ResNet-101 feature module structure

[0171]

[0172] Table 2 Floating point embedding module

[0173] Layer name Output size Layer Embedding layer 1×128 full connection

[0174] The input of Table 2 is the pooling output, and the output is a 1x128 floating-point vector, which is a floating-point embedding.

[0175] Table 3 Hash feature cross module

[0176] Layer name Output size Layer Feature fusion 1layer 1×1024 full connection Hash1 layer 1×128 full connection

[0177] The input of Table 3 is the output of Table 2. The feature fusion layer performs feature crossover on the input. The purpose of feature crossover is to parse the associated information within the input floating-point features so as to support further compression of this information into the target hash. The hash1 layer parses the 1024-dimensional feature crossover input and outputs a 1×128 floating-point vector. The floating-point vector is mapped to a binary vector (0 or 1, mapped to -1 or 1 in training, in order to distinguish, sign1 is used to represent quantization as 0, 1, and sign2 is used to represent quantization as -1, 1) through the sign function, which is the hash feature in the application.

[0178] Table 4 Hash feature cross module

[0179] Layer name Output size Layer Feature fusion2 layer 1×1024 full connection Hash2 layer 1×128 full connection

[0180] Table 4 Hash feature cross module, input R1 = embedding-sign1 (hash1) to obtain the first-level residual floating-point vector, where embedding is the output of Table 2, and hash1 is the output of the hash1 layer in Table 3; the feature fusion and hash layer here have the same functions as those in Table 3.

[0181] Table 5 Hash feature cross module

[0182] Layer name Output size Layer Feature fusion3 layer 1×1024 full connection Hash3 layer 1×128 full connection

[0183] Table 5 Hash feature cross module, the input is the floating point vector obtained by R1–sign1(hash2), where R1 is the input of the module in Table 4, and hash2 is the output of the hash2 layer in Table 4; the feature fusion and hash layer here have the same functions as those in Tables 3 and 4.

[0184] Hash feature (this feature is still a floating point number like regular embedding during learning, but each value of the feature in the application will be passed through the sign function to obtain a vector composed of two values ​​(for the convenience of calculation, the binary value is 0 / 1 in the application and -1 / 1 in training), which is the hash quantization feature in the final application).

[0185] As an alternative, 1) the CNN network and hash module can adopt other model structures, such as the basic features using the resnet18CNN module, the hash quantization layer using multiple Fc cascades, etc.; 2) the hash feature cross module can also adopt other MLP multi-layer nonlinear activation structures.

[0186] Before training, the model provided in the embodiment of the present application has been pre-trained by default to obtain the basic CNN and floating-point embedding in Tables 1 and 2 above. If it is necessary to train these two modules, the triplet mining method can be used to generate samples, and the triplet loss can be used to train the floating-point embedding of the triplet.

[0187] The model training of the embodiment of the present application mainly includes the following process:

[0188] 1) Parameter initialization: The new CNN layers (hash cross modules in Tables 3, 4, and 5) are initialized using a Gaussian distribution with a variance of 0.01 and a mean of 0.

[0189] 2) Set learning parameters: all parameters in Tables 3, 4, and 5.

[0190] 3) Learning rate (lr): lr = 0.0005 is used for all hash modules. After every 10 iterations, lr becomes 0.1 times of the original value.

[0191] 4) Learning process: Perform epoch iterations on all data. Each iteration processes all samples until the average epoch loss stops decreasing at a certain epoch.

[0192] 5) The specific operation for each epoch is as follows: take all the positive sample pairs annotated in full, and each bs sample pair is a batch. According to the above triple mining steps, generate the triple samples mined in each batch. Perform a model forward calculation and loss calculation for each batch and update the model parameters. The model learning process under batch is as follows:

[0193] (1) Triplet acquisition: According to the above triplet mining method, for the bs sample pairs in the batch, 20*bs triplets are obtained.

[0194] (2) Model forward: All parameters of the model are set to the state that needs to be learned. During training, the neural network performs forward calculations on an input image to obtain three hash layer prediction results, represented by Hash1, hash2, and hash3, where each 1x128 vector represents the hash features at each level.

[0195] (3) Loss calculation: 3 metric losses are calculated for each triplet in the batch, and 3 quantization losses are calculated for each image in the batch. Since there are 3*bs images in the batch and 20*bs triplets are generated, there are 3*20*bs metric losses and 9*bs quantization losses. The metric loss and hash loss are weighted and summed with weights of 1 and 0.01 (where the metric loss weight is 1 and the quantization loss weight is 0.01) to get the total loss.

[0196] (4) Model parameter update: Using the SGD stochastic gradient descent method, the loss in (3) is backward-calculated to obtain the updated value of the model parameters and update the network.

[0197] To summarize, the method provided in this embodiment extracts first-level hash features and at least first-level residual hash features for the data to be retrieved and the data in the data set during the data retrieval process, and selectively uses at least first-level residual hash features as feature representations of the retrieval process during data retrieval, thereby using different hash features for retrieval in different scenarios and with different retrieval accuracies, thereby improving the flexibility of data retrieval, avoiding waste of resources for scenarios with lower retrieval accuracy requirements, and avoiding the problem of low retrieval accuracy for scenarios with higher retrieval accuracy requirements.

[0198] The method provided in this embodiment uses the total features merged by multi-level residual hashing as the sorting features, which is greatly helpful in overcoming the problem of improper sorting caused by quantization errors compared to sorting using only main hashing. In the training of the multi-level residual hashing model, the metric learning constraints and model optimization of the multi-level hashing ensure that more hash bits can make the hash representation effect closer to the floating-point feature representation effect. Therefore, using more bits of hash as the sorting feature can make the sorting accuracy higher, and finally the accuracy of the topk_2 recall result is higher.

[0199] The method provided in this embodiment is based on the residual feature multiple modeling and learning hash representation, which can support the use of a specific length hash as a representation according to the data or precision requirements in the application. It supports hash feature applications with different data distribution / precision requirements, and can supplement the hash representation capability through multi-level hash splicing to achieve the effect of gradually approximating floating-point features.

[0200] The method provided in this embodiment is a scalable multi-level residual hash feature retrieval method. During retrieval, it can flexibly trigger whether to continue residual retrieval according to a certain recall effect to solve the problem of uneven data distribution; it can also construct residual hash multi-level indexes for inventory data volumes with different hashes during index construction, thereby supporting asymmetric index retrieval to solve the problem of insufficient hash bit representation of specific channels.

[0201] The method provided in this embodiment improves the recall rate of duplicate search: uses multi-level hash elastic recall to supplement the recall effect; improves the accuracy of duplicate search: uses multi-level hash elastic sorting calculation to improve the accuracy of the final recall sorting.

[0202] Fig.14 is a structural block diagram of a data retrieval device provided by an exemplary embodiment of the present application, such as Fig.14 As shown, the device comprises:

[0203] An acquisition module 1410 is used to acquire data to be retrieved and a data set for retrieving the data to be retrieved;

[0204] An extraction module 1420 is used to extract a first hash feature of the data to be retrieved, where the first hash feature includes a first-level hash feature and at least a first-level residual hash feature of the data to be retrieved;

[0205] The extraction module 1420 is further used to extract a second hash feature of the data in the data set, wherein the second hash feature includes a primary hash feature and at least a primary residual hash feature of the data in the data set;

[0206] The retrieval module 1430 is used to retrieve retrieval result data matching the data to be retrieved from the data set based on the first-level hash feature and at least one-level residual hash feature included in the first hash feature, and the first-level hash feature and at least one-level residual hash feature included in the second hash feature.

[0207] In an optional embodiment, if Fig.15 As shown, the retrieval module 1430 includes:

[0208] A clustering unit 1431 is used to cluster the first-level hash features and at least one first-level residual hash features included in the second hash features to obtain at least two cluster centers, where the cluster centers are used to indicate similarity relationships between data in the data set;

[0209] A determining unit 1432, configured to determine a target cluster center matching the first hash feature from the at least two cluster centers based on a feature distance relationship;

[0210] The determining unit 1432 is further configured to determine, from the candidate data associated with the target cluster center, retrieval result data matching the data to be retrieved.

[0211] In an optional embodiment, the clustering unit 1431 is also used to cluster the first-level hash features of the data in the data set to obtain n first cluster centers, where n is a positive integer; and to cluster at least one first-level residual hash feature of the data in the data set to obtain m second cluster centers corresponding to the residual hash features, where m is a positive integer.

[0212] In an optional embodiment, the determining unit 1432 is further configured to determine, from the n first clustering centers, a first candidate clustering center that matches the primary hash feature included in the first hash feature;

[0213] The determination unit 1432 is also used to determine, in response to the data associated with the first cluster center being within a preset number range, from the m second cluster centers, a second candidate cluster center that matches at least one level of residual hash feature included in the first hash feature; and determine the target cluster center based on the first candidate cluster center and the second candidate cluster center.

[0214] In an optional embodiment, the clustering unit 1431 is also used to cluster at least one level of residual hash features of the data in the data set, respectively, to obtain at least two second cluster centers corresponding to each level of residual hash features, and the at least two second cluster centers corresponding to each level of residual hash features constitute the m second cluster centers.

[0215] In an optional embodiment, the acquisition module 1410 is further used to acquire a hash distance between the first hash feature and a second hash feature of the candidate data;

[0216] The determining unit 1432 is further configured to determine, from the candidate data associated with the target cluster center, candidate data whose hash distance with the first hash feature meets the distance requirement as the retrieval result data.

[0217] In an optional embodiment, the determining unit 1432 is further configured to determine the number of different feature bits between the first hash feature and the second hash feature of the candidate data;

[0218] The determining unit 1432 is further configured to determine a hash distance between the first hash feature and a second hash feature of the candidate data based on the number of the different feature bits.

[0219] In an optional embodiment, the extraction module 1420 is further used to input the data to be retrieved into a data retrieval model, extract the data feature representation of the data to be retrieved through the data retrieval model; and perform hash conversion on the data feature representation to obtain a first hash feature of the data to be retrieved.

[0220] In an optional embodiment, the extraction module 1420 is also used to extract the floating-point feature representation corresponding to the data feature representation; perform hash feature cross-processing on the floating-point feature representation to obtain the first-level hash feature of the data to be retrieved; and obtain the at least one first-level residual hash feature based on the difference between the first-level hash feature and the floating-point feature representation.

[0221] In an optional embodiment, the acquisition module 1410 is further used to acquire a sample triplet, wherein the sample triplet includes anchor point data, positive sample data, and negative sample data, wherein the anchor point data and the positive sample data meet the similarity requirement, and the anchor point data and the negative sample data do not meet the similarity requirement;

[0222] The extraction module 1420 is further configured to extract hash features of the sample triples through a candidate retrieval model, wherein the hash features of the sample triples include anchor hash features of the anchor data, positive sample hash features of the positive sample data, and negative sample hash features of the negative sample data;

[0223] The device further comprises:

[0224] A determination module 1440, configured to determine a loss value based on a distance relationship between the anchor point hash feature, the positive sample hash feature, and the negative sample hash feature;

[0225] The training module 1450 is used to train the candidate retrieval model based on the loss value to obtain the data retrieval model.

[0226] In an optional embodiment, the loss value includes triplet loss;

[0227] The acquisition module 1410 is further used to acquire the first-level hash feature and at least one-level residual hash feature of the anchor data; acquire the first-level hash feature and at least one-level residual hash feature of the positive sample data; acquire the first-level hash feature and at least one-level residual hash feature of the negative sample data;

[0228] The determination module 1440 is further configured to determine a first loss based on a difference relationship between the primary hash feature of the anchor data, the primary hash feature of the positive sample data, and the primary hash feature of the negative sample data;

[0229] The determination module 1440 is further configured to determine a second loss based on a difference relationship between at least one level residual hash feature of the anchor data, at least one level residual hash feature of the positive sample data, and at least one level residual hash feature of the negative sample data;

[0230] The determination module 1440 is further configured to weightedly fuse the first loss and the second loss to obtain the triplet loss.

[0231] In an optional embodiment, the loss value includes regression loss;

[0232] The acquisition module 1410 is further configured to perform quantization encoding on the anchor hash feature to obtain a first target encoding of the anchor hash feature; perform quantization encoding on the positive sample hash feature to obtain a second target encoding of the anchor hash feature; and perform quantization encoding on the negative sample hash feature to obtain a third target encoding of the anchor hash feature.

[0233] The determination module 1440 is further configured to obtain the regression loss based on the first target encoding, the second target encoding, and the third target encoding.

[0234] To summarize, the device provided in this embodiment, during the process of data retrieval, extracts first-level hash features and at least first-level residual hash features for the data to be retrieved and the data in the data set, and during data retrieval, uses at least first-level residual hash features as feature representation of the retrieval process, thereby compensating for the problem of weak hash representation caused by quantization errors through residual features, thereby improving the accuracy of data retrieval.

[0235] It should be noted that the data retrieval device provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the data retrieval device provided in the above embodiment and the data retrieval method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0236] Fig.16 A schematic diagram of the structure of a server provided by an exemplary embodiment of the present application is shown. The server may be Figure 2 The server shown.

[0237] Specifically, the server 1600 includes a central processing unit (CPU) 1601, a system memory 1604 including a random access memory (RAM) 1602 and a read only memory (ROM) 1603, and a system bus 1605 connecting the system memory 1604 and the central processing unit 1601. The server 1600 also includes a mass storage device 1606 for storing an operating system 1613, application programs 1614, and other program modules 1615.

[0238] The mass storage device 1606 is connected to the central processing unit 1601 through a mass storage controller (not shown) connected to the system bus 1605. The mass storage device 1606 and its associated computer readable media provide non-volatile storage for the server 1600. That is, the mass storage device 1606 may include a computer readable medium (not shown) such as a hard disk or a compact disc read only memory (CD-ROM) drive.

[0239] Without loss of generality, computer-readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, Erasable Programmable Read Only Memory (EPROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other solid-state storage technology, CD-ROM, Digital Versatile Disc (DVD) or other optical storage, cassettes, tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that computer storage media are not limited to the above. The above-mentioned system memory 1604 and mass storage device 1606 can be collectively referred to as memory.

[0240] According to various embodiments of the present application, the server 1600 can also be connected to a remote computer on the network through a network such as the Internet. That is, the server 1600 can be connected to the network 1612 through the network interface unit 1611 connected to the system bus 1605, or the network interface unit 1611 can be used to connect to other types of networks or remote computer systems (not shown).

[0241] The above-mentioned memory also includes one or more programs, and the one or more programs are stored in the memory and configured to be executed by the CPU.

[0242] The embodiment of the present application also provides a computer device, which can be implemented as follows: Figure 2 The computer device includes a processor and a memory, wherein at least one instruction, at least one program, code set or instruction set is stored in the memory, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the data retrieval method provided by the above-mentioned method embodiments.

[0243] An embodiment of the present application also provides a computer-readable storage medium, on which is stored at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the data retrieval method provided by the above-mentioned method embodiments.

[0244] The embodiments of the present application also provide a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the data retrieval method described in any of the above embodiments.

[0245] Optionally, the computer readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a solid state drive (SSD), or an optical disk. Among them, the random access memory may include a resistance random access memory (ReRAM) and a dynamic random access memory (DRAM). The serial numbers of the above embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0246] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0247] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A data retrieval method, It is characterized in that The method comprises: Acquire data to be retrieved and a data set for retrieving the data to be retrieved; Extracting a first hash feature of the data to be retrieved, wherein the first hash feature includes a first-level hash feature and at least one first-level residual hash feature of the data to be retrieved; Extracting a second hash feature of the data in the data set, where the second hash feature includes a first-level hash feature and at least a first-level residual hash feature of the data in the data set; Clustering the first-level hash features and at least one first-level residual hash features included in the second hash features to obtain at least two cluster centers, where the cluster centers are used to indicate similarity relationships between data in the data set; Determining a target cluster center matching the first hash feature from the at least two cluster centers based on a feature distance relationship; Determining, from the candidate data associated with the target cluster center, retrieval result data matching the data to be retrieved; Among them, the first-level hash feature is obtained by performing feature cross processing on the floating-point feature representation, and the at least one first-level residual hash feature is obtained based on the difference between the first-level hash feature and the floating-point feature representation, and the floating-point feature representation is obtained by extracting the data feature representation of the data.

2. The method according to claim 1, It is characterized in that The clustering of the first-level hash features and at least one first-level residual hash features included in the second hash features to obtain at least two cluster centers includes: Clustering the primary hash features of the data in the data set to obtain n first cluster centers, where n is a positive integer; At least one level of residual hash features of the data in the data set are clustered respectively to obtain m second cluster centers corresponding to the residual hash features, where m is a positive integer.

3. The method according to claim 2, It is characterized in that The determining, based on the feature distance relationship, a target cluster center matching the first hash feature from the at least two cluster centers comprises: Determine, from the n first cluster centers, a first candidate cluster center that matches the primary hash feature included in the first hash feature; In response to the data associated with the first cluster center being within a preset quantity range, determining, from the m second cluster centers, step by step, second candidate cluster centers that match at least one level of residual hash features included in the first hash features; The target cluster center is determined based on the first candidate cluster center and the second candidate cluster center.

4. The method according to claim 2, It is characterized in that The clustering of at least one level residual hash feature of the data in the data set to obtain m second cluster centers corresponding to the residual hash features includes: At least one level of residual hash features of data in the data set are clustered respectively to obtain at least two second cluster centers corresponding to each level of residual hash features, and the at least two second cluster centers corresponding to each level of residual hash features constitute the m second cluster centers.

5. The method according to claim 1, It is characterized in that The step of determining, from the candidate data associated with the target cluster center, retrieval result data matching the data to be retrieved comprises: Obtaining a hash distance between the first hash feature and a second hash feature of the candidate data; From the candidate data associated with the target cluster center, candidate data whose hash distance with the first hash feature meets the distance requirement is determined as the retrieval result data.

6. The method according to claim 5, It is characterized in that The obtaining a hash distance between the first hash feature and the second hash feature of the candidate data includes: Determine the number of different feature bits between the first hash feature and the second hash feature of the candidate data; A hash distance between the first hash feature and a second hash feature of the candidate data is determined based on the number of the different feature bits.

7. The method according to any one of claims 1 to 6, It is characterized in that The extracting the first hash feature of the data to be retrieved includes: Inputting the data to be retrieved into a data retrieval model, and extracting data feature representations of the data to be retrieved through the data retrieval model; Perform hash conversion on the data feature representation to obtain a first hash feature of the data to be retrieved.

8. The method according to claim 7, It is characterized in that The performing hash conversion on the data feature representation to obtain a first hash feature of the data to be retrieved includes: Extracting a floating-point feature representation corresponding to the data feature representation; Performing hash feature cross processing on the floating point feature representation to obtain a primary hash feature of the data to be retrieved; Based on the difference between the first-level hash feature and the floating-point feature representation, the at least one first-level residual hash feature is obtained.

9. The method according to claim 7, It is characterized in that Before inputting the data to be retrieved into the data retrieval model, the method further comprises: Acquire a sample triplet, wherein the sample triplet includes anchor point data, positive sample data, and negative sample data, wherein the anchor point data and the positive sample data meet the similarity requirement, and the anchor point data and the negative sample data do not meet the similarity requirement; Extracting hash features of the sample triples through a candidate retrieval model, wherein the hash features of the sample triples include anchor hash features of the anchor data, positive sample hash features of the positive sample data, and negative sample hash features of the negative sample data; Determine a loss value based on a distance relationship between the anchor point hash feature, the positive sample hash feature, and the negative sample hash feature; The candidate retrieval model is trained based on the loss value to obtain the data retrieval model.

10. The method according to claim 9, It is characterized in that The loss value includes triplet loss; The determining of the loss value based on the distance relationship between the anchor point hash feature, the positive sample hash feature, and the negative sample hash feature includes: Obtaining a first-level hash feature and at least a first-level residual hash feature of the anchor point data; Obtaining a first-level hash feature and at least a first-level residual hash feature of the positive sample data; Obtaining a first-level hash feature and at least a first-level residual hash feature of the negative sample data; Determine a first loss based on a difference relationship among the primary hash feature of the anchor data, the primary hash feature of the positive sample data, and the primary hash feature of the negative sample data; Determine a second loss based on a difference relationship between at least one level residual hash feature of the anchor point data, at least one level residual hash feature of the positive sample data, and at least one level residual hash feature of the negative sample data; The first loss and the second loss are weightedly fused to obtain the triplet loss.

11. The method according to claim 9, It is characterized in that The loss value includes regression loss; The determining of the loss value based on the distance relationship between the anchor point hash feature, the positive sample hash feature, and the negative sample hash feature includes: Quantizing and encoding the anchor hash feature to obtain a first target code of the anchor hash feature; Quantizing and encoding the positive sample hash feature to obtain a second target code of the anchor point hash feature; Quantizing and encoding the negative sample hash feature to obtain a third target code of the anchor point hash feature; The regression loss is obtained based on the first target encoding, the second target encoding and the third target encoding.

12. A data retrieval device, It is characterized in that The device comprises: An acquisition module, used to acquire data to be retrieved and a data set for retrieving the data to be retrieved; An extraction module, used for extracting a first hash feature of the data to be retrieved, wherein the first hash feature includes a first-level hash feature and at least a first-level residual hash feature of the data to be retrieved; The extraction module is further used to extract a second hash feature of the data in the data set, wherein the second hash feature includes a primary hash feature and at least a primary residual hash feature of the data in the data set; The retrieval module includes a clustering unit and a determination unit; The clustering unit is used to cluster the first-level hash features and at least one first-level residual hash features included in the second hash features to obtain at least two cluster centers, and the cluster centers are used to indicate the similarity relationship between the data in the data set; The determining unit is used to determine a target cluster center matching the first hash feature from the at least two cluster centers based on a feature distance relationship; The determining unit is further used to determine the search result data matching the data to be searched from the candidate data associated with the target cluster center; Among them, the first-level hash feature is obtained by performing feature cross processing on the floating-point feature representation, and the at least one first-level residual hash feature is obtained based on the difference between the first-level hash feature and the floating-point feature representation, and the floating-point feature representation is obtained by extracting the data feature representation of the data.

13. The device according to claim 12, It is characterized in that The clustering unit is further used for: Clustering the primary hash features of the data in the data set to obtain n first cluster centers, where n is a positive integer; At least one level of residual hash features of the data in the data set are clustered respectively to obtain m second cluster centers corresponding to the residual hash features, where m is a positive integer.

14. The device according to claim 13, It is characterized in that The determining unit is further configured to: Determine, from the n first cluster centers, a first candidate cluster center that matches the primary hash feature included in the first hash feature; In response to the data associated with the first cluster center being within a preset quantity range, determining, from the m second cluster centers, step by step, second candidate cluster centers that match at least one level of residual hash features included in the first hash features; The target cluster center is determined based on the first candidate cluster center and the second candidate cluster center.

15. The device according to claim 13, It is characterized in that The clustering unit is further used for: At least one level of residual hash features of data in the data set are clustered respectively to obtain at least two second cluster centers corresponding to each level of residual hash features, and the at least two second cluster centers corresponding to each level of residual hash features constitute the m second cluster centers.

16. The device according to claim 12, It is characterized in that The acquisition module is further used to acquire a hash distance between the first hash feature and a second hash feature of the candidate data; The determining unit is further configured to determine, from the candidate data associated with the target cluster center, candidate data whose hash distance with the first hash feature meets the distance requirement as the retrieval result data.

17. The device according to claim 16, It is characterized in that The determining unit is further configured to: Determine the number of different feature bits between the first hash feature and the second hash feature of the candidate data; A hash distance between the first hash feature and a second hash feature of the candidate data is determined based on the number of the different feature bits.

18. The device according to any one of claims 12 to 17, It is characterized in that The extraction module is further used for: Inputting the data to be retrieved into a data retrieval model, and extracting data feature representations of the data to be retrieved through the data retrieval model; Perform hash conversion on the data feature representation to obtain a first hash feature of the data to be retrieved.

19. The device according to claim 18, It is characterized in that The extraction module is further used for: Extracting a floating-point feature representation corresponding to the data feature representation; Performing hash feature cross processing on the floating point feature representation to obtain a primary hash feature of the data to be retrieved; Based on the difference between the first-level hash feature and the floating-point feature representation, the at least one first-level residual hash feature is obtained.

20. The device according to claim 18, It is characterized in that The acquisition module is further used to acquire a sample triplet, wherein the sample triplet includes anchor point data, positive sample data, and negative sample data, wherein the anchor point data and the positive sample data meet the similarity requirement, and the anchor point data and the negative sample data do not meet the similarity requirement; The extraction module is further used to extract hash features of the sample triples through a candidate retrieval model, wherein the hash features of the sample triples include anchor hash features of the anchor data, positive sample hash features of the positive sample data, and negative sample hash features of the negative sample data; A determination module, configured to determine a loss value based on a distance relationship between the anchor point hash feature, the positive sample hash feature, and the negative sample hash feature; A training module is used to train the candidate retrieval model based on the loss value to obtain the data retrieval model.

21. The device according to claim 20, It is characterized in that The loss value includes triplet loss; The acquisition module is further used to acquire the first-level hash features and at least one-level residual hash features of the anchor data; acquire the first-level hash features and at least one-level residual hash features of the positive sample data; acquire the first-level hash features and at least one-level residual hash features of the negative sample data; The determination module is also used to determine a first loss based on the difference relationship between the first-level hash features of the anchor point data, the first-level hash features of the positive sample data, and the first-level hash features of the negative sample data; determine a second loss based on the difference relationship between at least one first-level residual hash feature of the anchor point data, at least one first-level residual hash feature of the positive sample data, and at least one first-level residual hash feature of the negative sample data; and weightedly fuse the first loss and the second loss to obtain the triple loss.

22. The device according to claim 20, It is characterized in that The loss value includes regression loss; The acquisition module is further used to quantize and encode the anchor hash feature to obtain a first target code of the anchor hash feature; quantize and encode the positive sample hash feature to obtain a second target code of the anchor hash feature; and quantize and encode the negative sample hash feature to obtain a third target code of the anchor hash feature. The determination module is also used to obtain the regression loss based on the first target encoding, the second target encoding and the third target encoding.

23. A computer device, It is characterized in that The computer device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the data retrieval method as described in any one of claims 1 to 11.

24. A computer-readable storage medium, It is characterized in that The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the data retrieval method as described in any one of claims 1 to 11.

25. A computer program product, It is characterized in that It comprises computer instructions, which, when executed by a processor, implement the data retrieval method according to any one of claims 1 to 11.

Images