Image processing method, device, equipment, storage medium and program product

Abstract

The present application relates to an image processing method, apparatus, device, storage medium, and program product, and relates to the field of artificial intelligence technology. The method comprises: performing n-order binary quantization on the image features of a query image based on target retrieval accuracy to obtain n-order quantized features of the query image; performing a search in a target retrieval database based on the n-order quantized features of the query image to obtain at least one target image; the target retrieval database is a retrieval database constructed based on the n-order quantized features of at least two images. The above scheme uses quantized features of different orders for retrieval according to different retrieval accuracy requirements, thereby achieving the purpose of simultaneously meeting different retrieval accuracy requirements and improving the retrieval effect of the image retrieval system.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image processing method, apparatus, device, storage medium, and program product. Background Art

[0002] Image retrieval is a technique for retrieving target images similar to a query image from a retrieval database containing several stock images.

[0003] In related technologies, image retrieval can be implemented based on the binary quantization features of an image. For example, during the construction of the retrieval database, a specified number of binary quantization features are extracted from each of the stock images. During the retrieval phase, a specified number of binary quantization features are extracted from the query image, and the similarity between the binary quantization features of the query image and the binary quantization features of the stock images is calculated. One or more similar target images are then output in descending order based on the calculated similarity.

[0004] However, in the above technical solution, the number of bits of the binary quantization feature is fixed, and the higher the number of bits of the binary quantization feature, the higher the retrieval accuracy and the lower the retrieval efficiency, which cannot meet different retrieval requirements and affects the retrieval effect of the image retrieval system. Summary of the Invention

[0005] The present application provides an image processing method, apparatus, device, storage medium, and program product that can simultaneously meet different retrieval accuracy requirements and improve the retrieval effect of the image retrieval system. The technical solution is as follows:

[0006] In one aspect, an image processing method is provided, the method comprising:

[0007] receiving an image retrieval request, the image retrieval request being used to request retrieval of a target image corresponding to a query image according to a target retrieval accuracy;

[0008] Performing n-order binary quantization on the image features of the query image based on the target retrieval accuracy to obtain n-order quantized features of the query image; the i-th order quantized features in the n-order quantized features are features obtained by quantizing the i-1-th order residual during the i-th order binary quantization of the image; the i-1-th order residual includes the residual generated during the i-1-th order binary quantization of the image; 2≤i≤n, and i and n are integers;

[0009] Based on the n-order quantitative features of the query image, a search is performed in a target retrieval database to obtain at least one target image; the target retrieval database is a retrieval database constructed based on the n-order quantitative features of at least two images.

[0010] In one aspect, an image processing method is provided, the method comprising:

[0011] Acquire a triplet sample; the triplet sample includes an anchor sample image, a positive sample image, and a negative sample image;

[0012] Obtaining floating-point embedding features of each of the anchor sample image, the positive sample image, and the negative sample image;

[0013] Processing the floating-point embedded features of the anchor sample image, the positive sample image, and the negative sample image respectively through a quantization feature extraction model to obtain m-order quantization features of the anchor sample image, the positive sample image, and the negative sample image respectively; the i-th order quantization feature in the m-order quantization feature is a feature obtained by quantizing the i-1-th order residual in the process of performing i-th order binary quantization on the image; the i-1-th order residual includes the residual generated in the process of performing i-1-th order binary quantization on the image; 2≤i≤m, and i and m are integers;

[0014] Obtaining a loss function value based on the m-order quantized features of each of the anchor sample image, the positive sample image, and the negative sample image;

[0015] Updating parameters of the quantitative feature extraction model based on the loss function value to train the quantitative feature extraction model;

[0016] Among them, the quantitative feature extraction model after training is used to obtain the m-order quantitative features of at least two images in the process of constructing at least two retrieval databases based on the m-order quantitative features of at least two images; the quantitative feature extraction model after training is also used to obtain the n-order quantitative features of the query image in the process of retrieving the target image from the target retrieval database through the query image; 2≤n≤m, and n is an integer.

[0017] In another aspect, an image processing apparatus is provided, the apparatus comprising:

[0018] A request receiving module, configured to receive an image retrieval request, wherein the image retrieval request is used to request retrieval of a target image corresponding to a query image according to a target retrieval accuracy;

[0019] a quantization module configured to perform n-th-order binary quantization on the image features of the query image based on the target retrieval accuracy to obtain n-th-order quantized features of the query image; wherein the i-th-order quantized features in the n-th-order quantized features are features obtained by quantizing the i-1-th-order residual during the i-th-order binary quantization of the image; the i-1-th-order residual includes the residual generated during the i-1-th-order binary quantization of the image; 2≤i≤n, and i and n are integers;

[0020] The retrieval module is used to search in a target retrieval database based on the n-order quantitative features of the query image to obtain at least one target image; the target retrieval database is a retrieval database constructed based on the n-order quantitative features of at least two images.

[0021] In a possible implementation, the quantization module is used to:

[0022] Obtaining a quantization order n based on the target retrieval accuracy;

[0023] extracting floating-point embedding features of the query image;

[0024] Based on the floating-point embedded features, the first n-level quantization networks in a quantization feature extraction model are used to process the query image to obtain n-order quantization features; the quantization feature extraction model includes an m-level quantization network, where m is greater than or equal to n and m is an integer;

[0025] Among them, the i-th level quantization network in the first n-level quantization networks is used to output the i-th order quantization feature in the n-order quantization features; the quantization feature extraction model is a model obtained by training based on at least two triple samples; the triple samples include anchor sample images, positive sample images, and negative sample images.

[0026] In a possible implementation, the quantization module is used to:

[0027] In a first-order quantization process, the floating-point embedded features are input into a first-order quantization network in the first n-order quantization networks for processing, so as to obtain a first-order quantization feature among the n-order quantization features of the query image output by the first-order quantization network;

[0028] During the i-th order quantization process, a residual between the input and output of the i-1th level quantization network in the first n levels of quantization networks is obtained; the residual between the input and output of the i-1th level quantization network is input into the i-th level quantization network in the first n levels of quantization networks for processing, and an i-th level quantization feature among the n-order quantization features of the query image output by the i-th level quantization network is obtained.

[0029] In a possible implementation, the retrieval module is used to:

[0030] Obtaining similarity between the n-th order quantitative feature of the query image and the n-th order quantitative features of at least two stock images in the target retrieval database;

[0031] At least one target image is obtained from the at least two stock images based on similarities between the n-order quantitative features of the query image and the n-order quantitative features of each of the at least two stock images in the target retrieval database.

[0032] In a possible implementation, the retrieval module is used to:

[0033] performing similarity calculations on n-order quantized features of the query image and quantized features of the same order among n-order quantized features of a first stock image to obtain n sub-similarity scores; the first stock image being any one of the at least two stock images;

[0034] Based on the orders of the n sub-similarity degrees, a weighted sum is performed on the n sub-similarity degrees to obtain a similarity between the query image and the first stock image.

[0035] In a possible implementation, the weight of the sub-similarity is inversely correlated with the order of the sub-similarity.

[0036] In one possible implementation, the quantization module is further configured to, before the request receiving module receives the image retrieval request, perform t-order binary quantization on the at least two stock images to obtain t-order quantized features of each of the at least two stock images; t is greater than or equal to n, and t is an integer;

[0037] The device further comprises:

[0038] A construction module is configured to construct at least two retrieval databases based on the respective t-order quantitative features of the at least two stock images; the retrieval databases are constructed based on the respective quantitative features of the at least two stock images of the same order; and the at least two retrieval databases include the target retrieval database.

[0039] In another aspect, an image processing apparatus is provided, the apparatus comprising:

[0040] A sample acquisition module, configured to acquire a triplet sample; the triplet sample includes an anchor sample image, a positive sample image, and a negative sample image;

[0041] A floating-point feature acquisition module is configured to acquire floating-point embedding features of each of the anchor sample image, the positive sample image, and the negative sample image;

[0042] a quantization module, configured to process the floating-point embedded features of each of the anchor sample image, the positive sample image, and the negative sample image through a quantization feature extraction model to obtain m-order quantization features of each of the anchor sample image, the positive sample image, and the negative sample image; the i-th order quantization feature in the m-order quantization feature is a feature obtained by quantizing the i-1-th order residual during the i-th order binary quantization of the image; the i-1-th order residual includes the residual generated during the i-1-th order binary quantization of the image; 2≤i≤m, and i and m are integers;

[0043] A loss acquisition module, configured to acquire a loss function value based on the m-order quantized features of each of the anchor sample image, the positive sample image, and the negative sample image;

[0044] A parameter updating module, configured to update the parameters of the quantitative feature extraction model based on the loss function value, so as to train the quantitative feature extraction model;

[0045] Among them, the quantitative feature extraction model after training is used to obtain the m-order quantitative features of at least two images in the process of constructing at least two retrieval databases based on the m-order quantitative features of at least two images; the quantitative feature extraction model after training is also used to obtain the n-order quantitative features of the query image in the process of retrieving the target image from the target retrieval database through the query image; 2≤n≤m, and n is an integer.

[0046] In a possible implementation, the loss function value includes m sub-loss function values; and the m sub-loss function values ​​respectively correspond to a quantization order;

[0047] The loss acquisition module is used to obtain the sub-loss function value corresponding to the i-th quantization order in the loss function value based on the first i-order quantization features of the m-order quantization features of each of the anchor point sample image, the positive sample image, and the negative sample image.

[0048] In a possible implementation, the loss acquisition module is used to:

[0049] Calculate a ternary loss function based on the first i-order quantization features of the m-order quantization features of the anchor sample image, the positive sample image, and the negative sample image, to obtain a ternary loss function value of the i-th quantization order;

[0050] Based on the ternary loss function value of the i-th quantization order, a sub-loss function value corresponding to the i-th quantization order in the loss function value is obtained.

[0051] In one possible implementation, the quantitative feature extraction model includes an m-level quantization network; the quantization network includes a feature fusion subnetwork and a sign function; the feature fusion subnetwork is used to fuse input features, and the sign function is used to binarize the fused features output by the feature fusion subnetwork;

[0052] The loss acquisition module is used to:

[0053] Obtaining output features of the feature fusion subnetwork of the i-th level quantization network in the m-level quantization network;

[0054] Obtaining a quantization target of the output features of the feature fusion subnetwork of the i-th level quantization network in the m-level quantization network;

[0055] Based on the output features of the feature fusion subnetwork of the i-th level quantization network in the m-level quantization network and the quantization target, obtaining a sign loss function value of the i-th quantization order;

[0056] A weighted sum is performed on the ternary loss function value of the i-th quantization order and the sign loss function value of the i-th quantization order to obtain a sub-loss function value corresponding to the i-th quantization order in the loss function value.

[0057] On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one computer instruction, and the at least one computer instruction is loaded and executed by the processor to implement the above-mentioned image processing method.

[0058] On the other hand, a computer-readable storage medium is provided, wherein the storage medium stores at least one computer instruction, and the at least one computer instruction is loaded and executed by a processor to implement the above-mentioned image processing method.

[0059] In another aspect, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-described image processing method.

[0060] The beneficial effects of the technical solutions provided in the embodiments of the present application include at least:

[0061] By receiving a retrieval request that specifies a target retrieval precision, the image retrieval request can be used to request retrieval of a target image corresponding to the query image according to the target retrieval precision. The retrieval platform device performs multi-order binary quantization on the query image according to the target retrieval precision, wherein the residual of the previous order binary quantization process is used for the next order binary quantization. Then, based on the results of the multi-order binary quantization, the target image is retrieved from the target retrieval database of the corresponding quantization order. In the above scheme, different orders of quantization features are used for retrieval to meet different retrieval precision requirements, thereby achieving the purpose of simultaneously meeting different retrieval precision requirements and improving the retrieval effect of the image retrieval system. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0063] Figure 1 is a framework diagram of an image processing system according to an exemplary embodiment;

[0064] Figure 2 is a flowchart illustrating an image processing method according to an exemplary embodiment;

[0065] Figure 3 is a flowchart illustrating an image processing method according to an exemplary embodiment;

[0066] Figure 4 is a schematic diagram of a framework of image processing according to an exemplary embodiment;

[0067] Figure 5 is a flowchart illustrating an image processing method according to an exemplary embodiment;

[0068] Figure 6 yes Figure 5 A schematic structural diagram of a quantitative feature extraction model involved in the illustrated embodiment;

[0069] Figure 7 yes Figure 5 The illustrated embodiment relates to a structural diagram of a sub-network in which features in a quantization network are fused;

[0070] Figure 8 and Figure 9 yes Figure 5 The illustrated embodiment relates to a precision customizable search application interface diagram;

[0071] Figure 10 is a framework diagram illustrating a model training, search database construction, and search process according to an exemplary embodiment;

[0072] Figure 11 is a structural block diagram of an image processing apparatus according to an exemplary embodiment;

[0073] Figure 12 is a structural block diagram of an image processing apparatus according to an exemplary embodiment;

[0074] Figure 13 The figure is a schematic structural diagram of a computer device according to an exemplary embodiment. DETAILED DESCRIPTION

[0075] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0076] Before describing the various embodiments shown in this application, several concepts involved in this application are first introduced:

[0077] 1) AI (Artificial Intelligence)

[0078] AI is the theory, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive field within computer science that seeks to understand the essence of intelligence and create new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to perceive, reason, and make decisions. AI technology is an interdisciplinary discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0079] 2) ML (Machine Learning)

[0080] Machine learning is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning through demonstration.

[0081] 3) Binary quantization: For D-dimensional feature vector embedding, the value range of the vector after normalization is generally -1 to 1 floating point number. The feature is compressed to a specified number of bits (such as 48 bits) with the value of 0 or 1 (called 48-bit compression). This is vector binary quantization and binary encoding.

[0082] 4) Binary quantization index: The D-dimensional feature vector is converted into a binary vector with a limited number of bits through a certain calculation process (model). The binary vector is used as an index to recall the image during retrieval.

[0083] 5) Deep product quantization of image representation: A deep learning-based floating-point vector quantization method aims to use a finite quantization codebook (a finite-bit encoding codebook such as 0 and 1) to segment the sample space (floating-point feature space) to be expressed. This method compresses feature storage while retaining the similarity measurement capability of the original floating-point features, which is beneficial for the storage and retrieval acceleration of large-scale inventories.

[0084] 6) Spatial Representation Accuracy of the Quantized Codebook: This refers to the accuracy of the codebook's representation of the feature space. Since feature quantization introduces quantization error, the larger the error, the lower the spatial representation accuracy. Spatial accuracy can be expressed as the inverse of the average Euclidean distance (or other distance used for retrieval features, such as the Hamming distance) between each feature and the center of the space. For example, for one-dimensional features 1, 2, 4, and 6, if the space is 2, features 1 and 2 are in the first space, and features 4 and 6 are in the second space. The representation accuracy of the first space is 1 / ((sqrt((1.5-1)²)+sqrt((2-1.5)²)) / 2)=1, while the representation accuracy of the second space is 1 / ((sqrt(5-4)²)+sqrt((6-5)²)) / 2)=1, indicating that the spatial accuracy of the first space is higher. For all quantized codebooks, the spatial accuracy of the quantized codebook is expressed as the average or minimum accuracy of all subspaces generated by the quantized codebook.

[0085] 7) Hamming distance: It is used to measure the distance between binary features. It is achieved by counting the number of feature bits with different values ​​as the distance. For example, the distance between (1000) and (0011) is 3.

[0086] 8) Multi-order quantization codebook: Compared to the (first-order) quantization codebook, for the feature space, when multiple quantization codebooks of different bases are used to approximate, a multi-order approximation is generated. For example, for the first-order codebook vector of "10", it means that 2 bits are used to divide the space into 4 (=2^2) subspaces, where "10" represents the "10th" space of the 4 subspaces (representing binary, converted to decimal, it is the third space); for the second-order codebook vector of ["10", "11"], it means that 2 bits are first used to divide the space into 4 subspaces (at this time, the spatial accuracy is the same as the first-order codebook), and then 2 bits are used to further divide each subspace into 4 subspaces (at this time, the spatial accuracy is 4 times that of the first-order codebook, which is equivalent to further splitting the space generated by each first-order codebook into 4 more accurate subspaces). The above descriptions of "deep product quantization representation" and "quantization codebook spatial expression accuracy" are all described from the perspective of the first-order codebook.

[0087] 9) Image deduplication: This method searches the image inventory for images that are extremely similar to the query image. Typically, highly similar images originate from the same image, but are duplicated through image transformation, color and brightness adjustment, cropping, watermarking, and other methods. Image deduplication can be used to identify whether user-uploaded images infringe upon the copyright of the target, making it a crucial method for protecting original content and identifying copyright infringement in film and television dramas.

[0088] 10) Triplet sample: A deep learning training sample component unit, including anchor samples, positive samples, and negative samples. The anchor samples and positive samples are duplicate images (extremely similar images, need to be detected during retrieval), and the anchor samples and negative samples are non-duplicate images (do not need to be detected during retrieval).

[0089] 11) Product Quantization Search: Also known as Product Quantization for Nearest Neighbor Search (PQ Search). PQ first divides a D-dimensional vector into M subspaces, where each spatial feature dimension is D / M. kmeans (k-clustering, such as 256, where the codebook for each space can be represented by integers numbered 0, 1, ..., 255, with each number corresponding to a cluster center vector) is then performed on each subspace to obtain M spatial cluster centers. During search, the query features are divided into M segments. For each segment, the nearest codebooks are searched in the space corresponding to that segment (i.e., the nearest center is found from the K centers in that space). All samples under that center are recalled. The distances between all samples in the M segments and the query in each segment are calculated (a total of M distances). The distances are summed to obtain the distance between the query and all recalled samples. The distances are sorted, and the top 30 samples with the smallest distances are selected for recall. After dividing the subspace into M subspaces, this method quantizes each subspace separately. One of the simplest quantization methods is to perform symbolic quantization on the D / M-dimensional features in the subspace. That is, when a certain dimension feature is greater than 0, it is quantized to 1, and when it is less than 0, it is quantized to 0. For example, for the [-1, 1, 0.5, -0.2] feature vector, the quantized code is [0, 1, 1, 0].

[0090] 12) Deep learning-based quantization method: also known as Deep Triplet Quantization (DTQ), which uses image triples as input to learn quantization targets.

[0091] In one possible implementation, deep learning-based product quantization can be used to represent images in image retrieval applications such as image deduplication. This approach offers the advantages of low feature storage requirements, high retrieval efficiency, and sufficient spatial representation capabilities. However, as a means of quantizing floating-point features, product quantization suffers from large quantization errors and cannot meet the varying retrieval and recall requirements of different businesses. For example, some businesses with large inventory volumes require rapid retrieval but have low recall accuracy requirements, while others require high recall accuracy. Therefore, the difficulty of applying deep product quantization lies in building a product quantization retrieval system that can meet the diverse needs of different businesses and providing the product quantization model that supports this system.

[0092] To address the above issues, the solution shown in the subsequent embodiments of this application designs a product quantization model based on deep learning that can be expressed progressively, and based on the quantization capability of the model, designs a retrieval application solution that can support different precision configurations.

[0093] Please refer to Figure 1 , which is a framework diagram of an image processing system according to an exemplary embodiment. Figure 1As shown, the image processing system includes a model training device 110, a retrieval platform device 120 and a retrieval terminal device 130.

[0094] Among them, the above-mentioned model training device 110, retrieval platform device 120 or retrieval terminal device 130 can be a terminal or a server.

[0095] Among them, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0096] The terminal may be a terminal device with network connection function and computing processing capability, for example, the terminal may be a laptop computer, desktop computer, personal workstation, smart phone, tablet computer, smart TV, vehicle-mounted terminal, etc.

[0097] The model training device 110 can be connected to the retrieval platform device 120 and the retrieval terminal device 130 via a communication network. Optionally, the communication network is a wired network or a wireless network.

[0098] In an embodiment of the present application, the above-mentioned model training device 110 and retrieval platform device 120 can be integrated together or implemented as different computer devices.

[0099] also, Figure 1 Only one model training device 110, retrieval platform device 120, and retrieval terminal device 130 is used as an example for introduction. Optionally, the image processing system may include multiple model training devices 110 (for example, the model training device 110 may include a device in a distributed model training system), a retrieval platform device 120 (for example, the retrieval platform device 120 may be a device in a distributed system platform), and a retrieval terminal device 130 (for example, the retrieval terminal device 130 may be a terminal of a different user). The embodiment of the present application does not limit the number of model training devices 110, retrieval platform devices 120, and retrieval terminal devices 130.

[0100] Optionally, the above-mentioned wireless network or wired network uses standard communication technologies and / or protocols. The network is typically the Internet, but can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a private network or any combination of a virtual private network. In some embodiments, technologies and / or formats including Hypertext Markup Language (HTML), Extensible Markup Language (XML), etc. are used to represent data exchanged over the network. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above-mentioned data communication technologies.

[0101] Figure 2 This is a flowchart of an image processing method according to an exemplary embodiment. The method can be executed by a computer device, for example, the computer device can be a server or a terminal. For example, the computer device can be implemented as Figure 1 The retrieval platform device 120 in the system shown. Figure 2 As shown, the image processing method may include the following steps:

[0102] Step 21: receiving an image retrieval request, where the image retrieval request is used to request retrieval of a target image corresponding to the query image according to a target retrieval accuracy.

[0103] The image retrieval request may indicate target retrieval accuracy and a query image.

[0104] In a possible implementation, the image retrieval request may carry an identifier of a target retrieval accuracy; in addition, the image retrieval request may also include a query image or an identifier of the query image (such as the address of the query image).

[0105] In a possible implementation, the retrieval platform device 120 may receive an image retrieval request sent by the retrieval terminal device 130 .

[0106] Step 22, performing n-th order binary quantization on the image features of the query image based on the target retrieval accuracy to obtain n-th order quantized features of the query image; the i-th order quantized features in the n-th order quantized features are features obtained by quantizing the i-1-th order residual during the i-th order binary quantization of the image; the i-1-th order residual includes the residual generated during the i-1-th order binary quantization of the image; 2≤i≤n, and i and n are integers.

[0107] In an embodiment of the present application, the computer device can parse the target retrieval accuracy from the image retrieval request; further, the computer device can obtain a query image from the image retrieval request, or obtain the address of the query image from the image retrieval request and obtain the query image based on the address of the query image. After obtaining the image retrieval accuracy and the query image, the computer device can perform n-order quantization on the query image based on the image retrieval accuracy.

[0108] Step 23 : Based on the n-th order quantitative features of the query image, search in a target retrieval database to obtain at least one target image; the target retrieval database is a retrieval database constructed based on the n-th order quantitative features of at least two images.

[0109] In summary, in the solution shown in the embodiment of the present application, the retrieval platform device can receive an image retrieval request that specifies a target retrieval accuracy. The image retrieval request can be used to request the retrieval of a target image corresponding to the query image according to the target retrieval accuracy. The retrieval platform device performs multi-order binary quantization on the query image according to the target retrieval accuracy, wherein the residual of the previous order binary quantization process is used for the next order binary quantization. Then, based on the results of the multi-order binary quantization, the target image is retrieved from the target retrieval database of the corresponding quantization order. In the above solution, for different retrieval accuracy requirements, quantization features of different orders are used for retrieval, thereby achieving the purpose of simultaneously meeting different retrieval accuracy requirements and improving the retrieval effect of the image retrieval system.

[0110] In an embodiment of the present application, the above-mentioned process of performing n-order binary quantization on the image features of the image can be implemented by a quantization feature extraction model based on AI technology.

[0111] Figure 3 This is a flowchart of an image processing method according to an exemplary embodiment. The method can be executed by a computer device, for example, the computer device can be a server or a terminal. For example, the computer device can be implemented as Figure 1 The model training device 110 in the system shown. Figure 3 As shown, the image processing method may include the following steps:

[0112] Step 31, obtaining a triplet sample; the triplet sample includes an anchor sample image, a positive sample image, and a negative sample image.

[0113] Step 32: Obtain floating-point embedding features of the anchor sample image, the positive sample image, and the negative sample image.

[0114] Step 33, respectively process the floating-point embedded features of the anchor sample image, the positive sample image, and the negative sample image through the quantization feature extraction model to obtain the m-order quantization features of the anchor sample image, the positive sample image, and the negative sample image; the i-th order quantization feature in the m-order quantization feature is the feature obtained by quantizing the i-1-th order residual in the process of performing the i-th order binary quantization on the image; the i-1-th order residual includes the residual generated in the process of performing the i-1-th order binary quantization on the image; 2≤i≤m, and i and m are integers.

[0115] Step 34 : Obtain a loss function value based on the m-order quantized features of each of the anchor sample image, the positive sample image, and the negative sample image.

[0116] Step 35 : updating the parameters of the quantitative feature extraction model based on the loss function value to train the quantitative feature extraction model.

[0117] Among them, the quantitative feature extraction model after training is used to obtain the m-order quantitative features of at least two images in the process of constructing at least two retrieval databases based on the m-order quantitative features of at least two images; the quantitative feature extraction model after training is also used to obtain the n-order quantitative features of the query image in the process of retrieving the target image from the target retrieval database through the query image; 2≤n≤m, and n is an integer.

[0118] In summary, in the scheme shown in the embodiment of the present application, the model training device trains a quantitative feature extraction model capable of performing multi-order binary quantization on an image through triple samples, wherein the residual of the previous order binary quantization process is used for the next order binary quantization; through the quantitative feature extraction model, it is possible to construct an m-order quantitative feature for obtaining at least two images, and construct at least two retrieval databases corresponding to the first-order or multi-order quantitative features; the retrieval platform device can receive an image retrieval request with a specified target retrieval accuracy, and the image retrieval request can be used to request the retrieval of a target image corresponding to the query image according to the target retrieval accuracy, and the retrieval platform device performs multi-order binary quantization on the query image according to the target retrieval accuracy, and based on the result of the multi-order binary quantization, retrieves the target image in the target retrieval database of the corresponding quantization order. In the above scheme, for different retrieval accuracy requirements, quantitative features of different orders are used for retrieval, thereby achieving the purpose of simultaneously meeting different retrieval accuracy requirements and improving the retrieval effect of the image retrieval system.

[0119] Figure 4 This is a schematic diagram of an image processing framework according to an exemplary embodiment. Figure 2 and Figure 3 In the scheme shown in the figure, first, the model training device can train a quantitative feature extraction model through triple samples and deploy it to the retrieval platform device. Figure 4 The framework shown is used to build the retrieval database and perform image retrieval.

[0120] like Figure 4 As shown, the retrieval platform device is deployed with a multi-order quantitative extraction service 41, a scheduling system 42, and a retrieval system 43. Among them, the multi-order quantitative extraction service 41 is deployed with the above-mentioned quantitative feature extraction model, and the retrieval system 43 is responsible for searching according to the retrieval request, merging the results, and sorting the results.

[0121] Taking the example of constructing a retrieval database using first-order quantitative features and second-order quantitative features respectively, in the retrieval database construction stage, the multi-order quantitative extraction service 41 performs binary quantization on a number of stock images through a quantitative feature extraction model to obtain the first-order quantitative features and second-order quantitative features of each stock image; then, a first-order index system is constructed based on the first-order quantitative features of each stock image, and a second-order index system is constructed based on the second-order quantitative features of each stock image.

[0122] During the image retrieval phase, after receiving a retrieval request, the scheduling system 42 determines the quantization order n according to the retrieval accuracy indicated in the retrieval request and sends the query image in the retrieval request to the multi-order quantization extraction service 41. The multi-order quantization extraction service 41 performs binary quantization on the query image using a quantization feature extraction model to obtain n-order quantization features (e.g., first-order or second-order). The scheduling system 42 then requests a retrieval from the retrieval system 43, with the request carrying the n-order quantization features of the query image. The retrieval system 43 selects an indexing system with a corresponding quantization order based on the quantization order n to perform an indexing request (e.g., when n=1, the retrieval system 43 selects a first-order indexing system for retrieval; when n=2, the retrieval system 43 selects a second-order indexing system for retrieval), receives the indexing results returned by the indexing system, calculates the similarity between the n-order quantization features of the query image and the n-order quantization features of each stock image in the retrieval system, sorts them according to the calculated similarity, selects at least one stock image with the highest similarity as the target image, and feeds it back to the scheduling system 42. The scheduling system 42 then feeds the target image back to the retrieval terminal device.

[0123] Figure 5 FIG. 1 is a flow chart of an image processing method according to an exemplary embodiment. The method can be executed by a computer device, for example, the computer device may include a model training device and a retrieval platform device. Figure 5As shown, the image processing method may include the following steps:

[0124] In step 501, a model training device obtains a triplet sample; the triplet sample includes an anchor sample image, a positive sample image, and a negative sample image.

[0125] In an embodiment of the present application, the training process of the model uses triple sample data as input. In order to find suitable triple samples from massive data, in an exemplary scheme, the embodiment of the present application can adopt the method of labeling positive sample pairs and then performing negative sample mining through the positive sample pairs to obtain triple samples.

[0126] 1) Labeling positive sample pairs: refers to the process of labeling whether the image sample pairs are similar. For example, two images are extracted from a large amount of data as a pair and sent to the labeler, who returns whether each pair of images is similar enough. Taking the model used in the duplicate removal system as an example, two samples need to be extremely similar to be considered similar samples. The sample pairs marked as similar are positive sample pairs, and the dissimilar ones are negative sample pairs. In the embodiment of the present application, the labeler can collect positive sample pairs, and negative sample pairs can be not collected. The negative samples in the triplet samples can be obtained by the following mining method.

[0127] 2) Triple data mining: Since the training metric learning features require a triple sample consisting of an anchor sample, a positive sample, and a negative sample (a, p, n) for loss function learning, a and p in the triple sample constitute a positive sample pair, and a and n constitute a negative sample pair. In the learning task, the positive sample pair needs to have features close enough (for example, the L2 distance is small enough) so that they can be retrieved from each other, while the negative sample pair needs to have features far enough apart. In the above annotation process, each sample pair that has been obtained can be used as the anchor and positive of the triple (for example, a randomly selected image is used as the anchor). The process of how to further mine negative samples (including hard negative samples and global negative samples) can be as follows:

[0128] Since the memory of computer processing devices is limited and all positive samples need to be sent to the processing device in batches for training, negative samples can also be mined within a batch.

[0129] For each batch of positive pairs (assuming there are bs pairs), negative samples are mined to obtain triples as follows: For the x-anchor in a sample x pair (one image is randomly selected as the anchor): the distance between the x-anchor and the x-anchor is calculated from the remaining bs-1 pairs (one image is randomly selected from each pair), sorted by distance from smallest to largest, and after removing the top 5 images, the top 20 samples are taken as hard negative samples. (Since we need to learn the features of extremely similar samples, we can assume that smaller distances indicate greater similarity. In massive data sets, the probability of two images being extremely similar is relatively low. Therefore, we directly remove the top 5 similar samples, and the remaining samples constitute the hard negative samples in the triples.) These are then combined with x to form triples. Each sample pair can generate 20 triples, and the entire batch can obtain 20*bs triples. To ensure effective mining of negative pairs, bs can be set to a relatively large value, such as 1024.

[0130] Metric learning based on triplet samples requires more difficult samples. If all samples are simple, the model cannot learn discriminative representations. The first 20 negative samples ensure that there are more difficult samples, which is beneficial for learning.

[0131] In step 502, the model training device obtains floating-point embedding features of the anchor sample image, the positive sample image, and the negative sample image.

[0132] In a possible implementation, during the binary quantization of the triplet samples, floating-point embedding features of the anchor sample image, the positive sample image, and the negative sample image may be extracted first.

[0133] In one possible implementation, the model training device may extract floating-point embedding features of the input image by quantizing a feature extraction network in a feature extraction model.

[0134] Optionally, the feature extraction network in the quantized feature extraction model may include a deep feature extraction subnetwork and a floating-point feature extraction subnetwork.

[0135] Please refer to Figure 6 , which shows a schematic diagram of the structure of a quantitative feature extraction model involved in the embodiment of the present application. Taking the quantitative feature extraction model for extracting second-order quantitative features as an example, Figure 6 As shown, the quantitative feature extraction model includes a deep feature extraction subnetwork 61 and a floating-point feature extraction subnetwork 62; the two constitute a feature extraction network; for example, the deep feature extraction subnetwork 61 can be a resnet101 network, and its structure can be as shown in Table 1.

[0136] Table 1

[0137]

[0138] The structure of the floating-point feature extraction subnetwork 62 may be as shown in Table 2.

[0139] Table 2

[0140] Layer name Output size Layer embedding layer 1x32 full connection

[0141] The input of the floating-point feature extraction subnetwork 62 is the pooling output of Table 1, and the output is a 1x128 floating-point vector, which is a floating-point embedding feature.

[0142] The quantitative feature extraction model also includes an m-order quantization network, each of which contains a feature fusion subnetwork and a sign function; wherein the feature fusion subnetwork is used to fuse the input features, and the sign function is used to binarize the fused features output by the feature fusion subnetwork.

[0143] like Figure 6 As shown, the quantization feature extraction model also includes a first-order quantization network 63 and a second-order quantization network 64; wherein, the floating-point embedded feature is input into the first-order quantization network 63 by the floating-point feature extraction subnetwork 62 to obtain the first-order quantization feature (such as the first-order product quantization feature); then, the floating-point embedded feature is subtracted from the first-order quantization feature to obtain the first-order floating-point residual, and the first-order floating-point residual is input into the second-order quantization network 64 to obtain the second-order product quantization feature, and the second-order product quantization feature and the first-order product quantization feature constitute the second-order quantization feature.

[0144] The structure of the feature fusion sub-network included in the first-order quantization network 63 may be as shown in Table 3.

[0145] Table 3

[0146] Layer name Output size Layer Feature fusion 1layer 1x1024 full connection Q1 layer 1x32 full connection

[0147] The feature fusion subnetwork shown in Table 3 takes the output of Table 2 as input. The feature fusion layer performs feature crossover on the input. The purpose of feature crossover is to parse the internal correlation information of the input floating-point features so as to support further compression of this information into the target product quantization. The Q1 layer parses the 1024-dimensional feature crossover input and outputs a 1x2 floating-point vector. This floating-point vector is mapped to a binary vector through a sign function (such as the sign function) (the binary value is 0 or 1 during model application and -1 or 1 during training. In order to distinguish, sign1 is used to indicate quantization as 0 or 1, and sign2 is used to indicate quantization as -1 or 1. Sign1 is used in the subsequent calculation of metric loss, and sign2 is used in the calculation of quantization loss), thus obtaining the product quantization feature.

[0148] The structure of the feature fusion sub-network included in the second-order quantization network 64 can be shown in Table 4.

[0149] Table 4

[0150] Layer name Output size Layer Feature fusion2 layer 1x1024 full connection Q2 layer 1x32 full connection

[0151] The feature fusion sub-network shown in Table 4 has a first-order residual floating-point vector obtained by inputting R1=embedding–sign1(Q1), where embedding is the output of Table 2 and Q1 is the output of the Q1 layer in Table 3. Since product quantization acts on the exponent bit, for the first order The unit represented by each quantization bit is, therefore, the residual of the first-order quantization is the floating-point feature minus the quantization vector, and for the second order, it is The unit represented by each quantization bit is For each unit represented by a quantization bit, an exponential increment is used to represent a more precise spatial segmentation effect, and the same applies to higher-order units. The feature fusion and Q2 layers in Table 3 have the same functions as the corresponding feature fusion and Q1 layers in Table 3.

[0152] Figure 6 The model shown can also continue to expand more quantization networks to adapt to retrieval applications with improved accuracy: if the current retrieval accuracy needs to be improved, the third-order product quantization can be continued to be trained. It is only necessary to connect a new quantization network with a structure similar to that shown in Table 3 or 4. The structure of the new quantization network can be shown in Table 5.

[0153] Table 5

[0154] Layer name Output size Layer Feature fusion3 layer 1x1024 full connection Q3 layer 1x32 full connection

[0155] The feature fusion sub-network shown in Table 5 has the following input: The floating-point vector obtained, where R1 is the input of Table 4, and Q2 is the output of the Q2 layer in Table 4; the feature fusion and Q3 layers in Table 5 have the same functions as the corresponding feature fusion and Q layers in Table 3 and Table 4.

[0156] That is to say, while keeping the model parameters in Tables 1, 2, 3, and 4 unchanged, it is sufficient to learn a product quantization that can represent the previous order floating-point residual. For example, the third-order product quantization structure shown in Table 5 is a feasible structure. Optionally, during training, the quantization learning (codding loss) of the third-order quantization in Table 5 and the feature metric learning after the third-order product quantization reconstruction also need to be added to the total loss calculation. In order to simplify the description, the training process below only introduces the product quantization learning with an accuracy of second order. At this time, in the application, you can refer to Figure 2, introduce "other high-order quantitative features" and their corresponding index systems. When searching, just add other high-order index systems to the triggering part.

[0157] Since all network parameters are floating-point numbers, the product quantization feature is a floating-point number during learning. In application, each value of this feature can be converted into a binary vector through the sign function, which is the quantization feature in the final application.

[0158] In step 503, the model training device processes the floating-point embedded features of the anchor sample image, the positive sample image, and the negative sample image respectively through the quantization feature extraction model to obtain the m-order quantization features of the anchor sample image, the positive sample image, and the negative sample image respectively; the i-th order quantization feature in the m-order quantization feature is the feature obtained by quantizing the i-1-th order residual in the process of performing the i-th order binary quantization on the image; the i-1-th order residual includes the residual generated in the process of performing the i-1-th order binary quantization on the image; 2≤i≤m, and i and m are integers.

[0159] For a triplet sample, the model training device can input the anchor sample image, the positive sample image, and the negative sample image into the following Figure 6 The quantization feature extraction model shown in the figure obtains the i-th order product quantization features outputted by the m quantization networks in each quantization feature extraction model, and the i-th order product quantization features outputted by the m quantization networks can constitute the above-mentioned m-order quantization features.

[0160] In step 504 , the model training device obtains a loss function value based on the m-order quantized features of each of the anchor sample image, the positive sample image, and the negative sample image.

[0161] In one possible implementation, the loss function value includes m sub-loss function values; and the m sub-loss function values ​​each correspond to a quantization order;

[0162] The loss function value is obtained based on the m-order quantized features of the anchor sample image, the positive sample image, and the negative sample image, including:

[0163] Based on the first i-order quantization features of the m-order quantization features of the anchor sample image, the positive sample image, and the negative sample image, a sub-loss function value corresponding to the i-th quantization order in the loss function value is obtained.

[0164] In the embodiment of the present application, the loss function value can be divided into m parts, and each quantization network corresponds to a quantization stage, which corresponds to a sub-loss function value.

[0165] For example, taking the second quantization network in the quantitative feature extraction model as an example, the output of the second quantization network and the output of the first quantization network constitute the first two-order quantization features. The first two-order quantization features of the anchor sample image, the positive sample image, and the negative sample image can be used to calculate the sub-loss function value of the second quantization order.

[0166] In one possible implementation, obtaining a sub-loss function value corresponding to the i-th quantization order in the loss function value based on the first i-order quantization features of the m-order quantization features of the anchor sample image, the positive sample image, and the negative sample image includes:

[0167] Calculate the ternary loss function based on the first i-order quantization features of the m-order quantization features of the anchor sample image, the positive sample image, and the negative sample image, and obtain the ternary loss function value of the i-th quantization order;

[0168] Based on the ternary loss function value of the i-th quantization order, obtain the sub-loss function value corresponding to the i-th quantization order in the loss function value.

[0169] In an embodiment of the present application, the sub-loss function value of the above-mentioned i-th quantization order can be obtained directly or indirectly through the ternary loss function value.

[0170] In a possible implementation, the quantitative feature extraction model includes an m-level quantization network; the quantization network includes a feature fusion subnetwork and a sign function; the feature fusion subnetwork is used to fuse input features, and the sign function is used to binarize the fused features output by the feature fusion subnetwork;

[0171] Based on the ternary loss function value of the i-th quantization order, the sub-loss function value corresponding to the i-th quantization order in the loss function value is obtained, including:

[0172] Obtain the output features of the feature fusion subnetwork of the i-th level quantization network in the m-level quantization network;

[0173] Obtain the quantization target of the output features of the feature fusion sub-network of the i-th level quantization network in the m-level quantization network;

[0174] Based on the output features of the feature fusion subnetwork of the i-th level quantization network in the m-level quantization network and the quantization target, the symbol loss function value of the i-th quantization order is obtained;

[0175] Perform weighted summation on the ternary loss function value of the i-th quantization order and the sign loss function value of the i-th quantization order to obtain the sub-loss function value corresponding to the i-th quantization order in the loss function value.

[0176] In the embodiment of this application, Figure 6Taking m=2 as an example, for each sample pair in the batch (a total of bs sample pairs), calculate the loss of all mined triples and the quantization loss of all images. The above total loss can be as follows:

[0177]

[0178] Wherein, w1 in the above formula may be 1, and w2 may be 0.01.

[0179] The above total losses can be divided into two types:

[0180] 1)L triplet :For the features reconstructed after quantization of each order product, triplet loss metric learning is performed: the triplet samples (a, p, n) mined in batch are used. The distance between a and n samples in the triplet sample needs to be large enough to ensure that the triplet can be distinguished in the quantization space. Therefore, the margin needs to be set to a large value. Considering that each bit in the quantization vector will eventually be close to -1 or 1, the maximum distance that can be represented is 2 2 =4, margin0 can be preset to 2, and the triplet loss formula is as follows:

[0181] L triplet =max(||Q a -Q p ||-‖Q a -Q n ‖+α,0)

[0182] Where a is margin, set to 2, ||Q a -Q p || represents the L2 distance of the quantization output Q of the two images a and p in the triplet. The purpose of Triplet-loss is to make the distance between the anchor and the negative minus the distance between the anchor and the positive greater than the margin. For the first-order and second-order triplet losses, the product quantization features are used for distance calculation: for example, for the anchor sample a, the reconstructed feature Q a is: sign1(Q1), (Q2); where Q1 and 2 are the outputs of Tables 3 and 4; the same is true for the triplet p and n samples.

[0183] 2) Coding symbol quantization loss: Taking the first-order product quantization output Q1 as an example, the coding symbol quantization loss is the loss of the quantization effect (whether it is close enough to -1 or 1) of the vector calculation of the quantization branch output. Since the final application needs to map the quantized output to a binary value of -1 / 1, if each bit of the Q1 output at each level is close enough to 1 or -1, then the triplet metric loss is sufficient to describe the application of the quantization feature. Otherwise, even if the triplet metric effect is good, it does not mean that the quantization representation in the application is good enough. Therefore, in the embodiment of the present application, it is expected that each bit of the hash output at each level is close enough to 1 or -1.

[0184] For the quantization results of each image, the loss function that meets this goal is as follows:

[0185]

[0186]

[0187] Among them, Q i is the value of the quantized Q of the image at the i-th bit (32 bits in this case), B i is the quantization target of the i-th position, B i By Q i Generated by the sign function - the target code of the quantized learning task is generated by the sign function (such as the sign function in the above formula, which quantizes each bit Q of the product). i The target codes Bi are calculated respectively by the sign function, and the target code of the final product quantization is B).

[0188] Then, regression loss is used to reduce the L2 distance between the Q vectors output in Tables 3 and 4 and the target code B. In applications, the coding loss can be calculated separately for the second-order product quantization, that is, the above loss can be calculated for Q1 and Q2, both of length 1x32.

[0189] Weight: Since the regression loss of coding converges faster than triplet-loss, and the importance of coding is lower than the feature measurement ability, in order to ensure that triplet-loss is dominant in the overall loss and thus ensure that embedding always has the ability to measure similarity, therefore, in this embodiment of the application, w2 is set to a value less than 1, such as 0.01.

[0190] When calculating the distance between two samples under second-order quantization in the application, the second-order quantization reconstruction features of the two samples are used to calculate the distance: If it is a first-order quantization distance, the reconstructed feature sign1(Q1) is used. It can be seen that the unit that can be expressed by second-order quantization is 1 / 2, and the accuracy is twice that of first-order quantization.

[0191] In step 505 , the model training device updates the parameters of the quantitative feature extraction model based on the loss function value to train the quantitative feature extraction model.

[0192] Among them, the quantitative feature extraction model after training is used to obtain the m-order quantitative features of at least two images in the process of constructing at least two retrieval databases based on the m-order quantitative features of at least two images; the quantitative feature extraction model after training is also used to obtain the n-order quantitative features of the query image in the process of retrieving the target image from the target retrieval database through the query image; 2≤n≤m, and n is an integer.

[0193] In this embodiment of the present application, the overall training process may include:

[0194] 1) Data preparation for training: Generation of triplet data for metric learning;

[0195] 2) Model structure and training process;

[0196] 3) Loss function calculation.

[0197] During the training process, negative samples are mined from metric learning sample pairs to form triple images required for metric learning, which are then input into the model. The model extracts multiple product quantization features from all images in the triple, performs quantization learning on each order of product quantization, and performs metric learning on the features reconstructed from each order of product quantization. After model iteration, the product quantization vectors of each order for each image are learned.

[0198] Before training this model, it can be assumed that the feature extraction network shown in Table 1 and Table 2 has been pre-trained. Optionally, if it is necessary to train the feature extraction network shown in Table 1 and Table 2, the triplet sample mining method in the embodiment of this application can also be used to generate samples, and the feature extraction network shown in Table 1 and Table 2 can be trained using Triplet loss. The embodiment of this application uses the pre-trained network shown in Table 1 and Table 2 as an example for explanation, so the parameters of the feature extraction network shown in Table 1 and Table 2 will no longer be adjusted subsequently. Figure 6 Taking the architecture shown in the figure as an example, the following describes how to train multi-order product quantization so that higher-order quantization representations have higher accuracy.

[0199] 1) Parameter initialization: The new convolutional neural network layer (such as the product quantization cross module in Table 3 and Table 4) is initialized using a Gaussian distribution with a variance of 0.01 and a mean of 0.

[0200] 2) Set learning parameters: all parameters in Table 3 and Table 4.

[0201] 3) Learning rate: A learning rate of lr = 0.0005 is used for all product quantization modules. After every 10 iterations, lr is multiplied by 0.1.

[0202] 4) Learning process: Perform epoch iterations on the full data set. Each iteration processes the full set of samples until the average epoch loss stops decreasing at a certain epoch.

[0203] 5) The specific operation for each epoch is as follows: take all the annotated positive sample pairs, each bs sample pairs as a batch, generate the triplet samples mined in each batch according to the above triplet mining steps, perform a model forward calculation and loss calculation on each batch, and update the model parameters. The model learning process under batch is as follows:

[0204] Step (1) triplet sample acquisition: According to the above triplet sample mining method, for the bs sample pairs in the batch, 20*bs triplets are obtained.

[0205] Step (2) Model forward: The parameters of the model are set to the state that needs to be learned. During training, the model performs forward calculation on an input image to obtain two product quantization layer prediction results, represented by Q1 and Q2, where both are 1x2 vectors representing the product quantization features of each order (in the solution shown in the embodiment of the present application, the number of quantization bits used in training is 32, and the number of quantization bits can also be increased according to the information richness required to be represented, so that the retrieval effect is better, as long as it is consistent with the number of bits of floating-point embedding).

[0206] Step (3) Loss calculation: Calculate 2 metric losses for each triplet sample in the batch, and 2 quantization losses for each image in the batch. Since there are 2*bs images in the batch and 20*bs triplets, there are 2*20*bs metric losses and 4*bs quantization losses in total. The metric loss and the product quantization loss are weighted and summed with weights of 1 and 0.01 (where the metric loss weight is 1 and the quantization loss weight is 0.01) to obtain the total loss.

[0207] Step (4) Model parameter update: Using the stochastic gradient descent method, the loss in step (3) is subjected to gradient backward calculation to obtain the updated value of the model parameters, and the model network is updated.

[0208] by Figure 6For example, the scheme shown in the embodiment of the present application proposes a progressive quantization network structure, as well as the data flow and loss calculation (input and calculation target) during image forward calculation. First-order quantization and second-order quantization targets are learned for each image respectively. After reconstructing the features through multi-order product quantization, the features are reconstructed with the help of metric learning constraints, so that the multi-order quantization learning task converges, and the reconstruction of floating-point features by combining higher-order vectors can make the accuracy of product quantization representation improve with the increase of order.

[0209] Among them, please refer to Figure 7 , which shows a structural diagram of a feature fusion sub-network in a quantization network involved in an embodiment of the present application. The feature fusion sub-network can directly obtain the target product quantization feature (i.e., the output of the Q layer) for the input information through the Fc layer.

[0210] After the quantitative feature extraction model is trained, it can be deployed to the search platform device to build a search database and perform image retrieval. The application of the quantitative feature extraction model in the search database construction and image retrieval process can be referred to in the subsequent steps.

[0211] In step 506 , the retrieval platform device performs t-order binary quantization on the at least two stock images to obtain t-order quantized features of the at least two stock images; t is greater than or equal to n, and t is an integer.

[0212] The retrieval platform device can perform t-order binary quantization on at least two stock images respectively through the trained quantitative feature extraction model to obtain t-order quantitative features of the at least two stock images.

[0213] In step 507, the retrieval platform device constructs at least two retrieval databases based on the respective t-order quantitative features of the at least two stock images; the retrieval databases are constructed based on the respective quantitative features of the same order of the at least two stock images; and the at least two retrieval databases include the target retrieval database.

[0214] The multi-order product quantization features trained in the embodiments of the present application can replace conventional product quantization features or floating-point features for use in common image retrieval.

[0215] Step 508: The retrieval platform device receives an image retrieval request, where the image retrieval request is used to request retrieval of a target image corresponding to the query image according to a target retrieval accuracy.

[0216] Please refer to Figure 8 and Figure 9 , which shows the interface diagram of the precision customizable retrieval application based on the multi-order codebook involved in the embodiment of the present application. Figure 8To initialize the interface, select a precision, enter a query image, and click OK. The result list will display the images recalled from the inventory indexed by that precision. For example, a precision of 1 corresponds to an index retrieval system composed of a first-order codebook, while a precision of 2 corresponds to a retrieval system composed of a second-order codebook. Figure 9 The retrieval results of the same input image under different precision configurations are compared. Under precision 2, more accurate images are recalled and the retrieval results are sorted more reasonably.

[0217] In step 509, the retrieval platform device performs n-th order binary quantization on the image features of the query image based on the target retrieval accuracy to obtain the n-th order quantized features of the query image; the i-th order quantized features in the n-th order quantized features are features obtained by quantizing the i-1-th order residuals during the i-th order binary quantization of the image; the i-1-th order residuals include the residuals generated during the i-1-th order binary quantization of the image; 2≤i≤n, and i and n are integers.

[0218] In a possible implementation, performing n-order binary quantization on the image features of the query image based on the target retrieval accuracy to obtain the n-order quantized features of the query image includes:

[0219] Obtain the quantization order n based on the target retrieval accuracy;

[0220] Extract floating-point embedding features of the query image;

[0221] Based on the floating-point embedded features, the n-order quantized features of the query image are obtained by processing through the first n-level quantization network in the quantized feature extraction model.

[0222] In one possible implementation, based on the floating-point embedded features, the first n-level quantization network in the quantization feature extraction model is processed to obtain the n-order quantization features of the query image, including:

[0223] In the first-order quantization process, the floating-point embedded features are input into the first-order quantization network in the first n-order quantization network for processing, and the first-order quantization features among the n-order quantization features of the query image output by the first-order quantization network are obtained;

[0224] In the i-th order quantization process, the residual between the input and output of the i-1th level quantization network in the first n-level quantization networks is obtained; the residual between the input and output of the i-1th level quantization network is input into the i-th level quantization network in the first n-level quantization networks for processing, and the i-th level quantization feature in the n-order quantization feature of the query image output by the i-th level quantization network is obtained.

[0225] In step 510, the retrieval platform device searches a target retrieval database based on the n-order quantitative features of the query image to obtain at least one target image; the target retrieval database is a retrieval database constructed based on the n-order quantitative features of at least two images.

[0226] In one possible implementation, searching a target retrieval database based on the n-th order quantitative feature of a query image to obtain at least one target image includes:

[0227] Obtaining similarity between the n-th order quantitative feature of the query image and the n-th order quantitative features of at least two stock images in the target retrieval database;

[0228] At least one target image is obtained from the at least two stock images based on similarities between the n-order quantitative features of the query image and the n-order quantitative features of each of at least two stock images in the target retrieval database.

[0229] In a possible implementation, obtaining the similarity between the n-th order quantitative feature of the query image and the n-th order quantitative features of at least two stock images in the target retrieval database includes:

[0230] performing similarity calculations on n-order quantized features of the query image and quantized features of the same order among n-order quantized features of a first stock image to obtain n sub-similarity scores; the first stock image is any one of the at least two stock images;

[0231] Based on the orders of the n sub-similarity degrees, the n sub-similarity degrees are weighted and summed to obtain the similarity between the query image and the first stock image.

[0232] In a possible implementation, the weight of the sub-similarity is inversely correlated with the order of the sub-similarity.

[0233] In the embodiments of the present application, the hashing process during the retrieval process can be similar to that of a PQ retrieval system. The difference from the traditional PQ system lies in the different algorithm used to calculate the Euclidean distance between two samples. In traditional PQ systems, each segmented distance has an equal contribution. However, in the embodiments of the present application, the higher the multi-order hashing, the smaller the contribution. This is because the primary distance measurement is concentrated at the lower order, while the higher order hashing serves to supplement accuracy.

[0234] In summary, in summary, in the scheme shown in the embodiment of the present application, the model training device trains a quantitative feature extraction model capable of performing multi-order binary quantization on an image through triple samples, wherein the residual of the previous order binary quantization process is used for the next order binary quantization; through the quantitative feature extraction model, it is possible to construct an m-order quantitative feature for obtaining at least two images, and construct at least two retrieval databases corresponding to the first-order or multi-order quantitative features; the retrieval platform device can receive an image retrieval request with a specified target retrieval accuracy, and the image retrieval request can be used to request the retrieval of a target image corresponding to the query image according to the target retrieval accuracy, and the retrieval platform device performs multi-order binary quantization on the query image according to the target retrieval accuracy, and based on the result of the multi-order binary quantization, retrieves the target image in the target retrieval database of the corresponding quantization order. In the above scheme, for different retrieval accuracy requirements, quantitative features of different orders are used for retrieval, thereby achieving the purpose of simultaneously meeting different retrieval accuracy requirements and improving the retrieval effect of the image retrieval system.

[0235] Based on the above Figure 2 、 Figure 3 as well as Figure 5 For the examples shown, please refer to Figure 10 , which is a framework diagram of a model training, retrieval database construction and retrieval process according to an exemplary embodiment. Figure 10 As shown, the process can be as follows:

[0236] Step 1: The model training device 1001 trains a quantitative feature extraction model through triple samples and deploys it to the retrieval platform device 1002.

[0237] Step 2: When constructing the retrieval database, the retrieval platform device 1002 extracts inventory features to establish various order indexes based on the inventory images, that is, all product quantitative features of the inventory images are obtained through the quantitative feature extraction model.

[0238] Step 3: The retrieval platform device 1002 constructs a multi-level index system for the inventory.

[0239] For example, the retrieval platform device 1002 can establish an index system based on quantized vectors of order 1 and order 2 (the relationship between the index and the image, and the relationship between the indexes of each order in the index system is as follows: Figure 10 As shown in the figure), the 2nd-order index system is a more accurate spatial segmentation of the 1st-order index (for example, for the 1st-order index 11, it cannot distinguish between black and white sketches and color pictures, while under the 11th-order second-order index (00, 11), it can further distinguish between sketch airplanes and real-scene color airplanes). The 3rd-order is also the same as the 2nd-order. It can be considered that low-order + high-order quantization supplements more detailed information of the image compared to low-order quantization, thereby making the retrieval results more accurate.

[0240] Among them, when establishing the index of each order of inventory, taking the quantization bit of each order as 2 as an example, the possible quantization representation of each order is (00 01 10 11). The feature extraction obtains the inventory graph and the model forward calculation obtains the quantization representation of each order, for example, the first-order quantization representation (00 01 10 11) and the second-order quantization representation (01 11) under the first-order representation 01. When establishing the index, the first-order index uses all the first-order quantization representations (4 quantization vectors) that appear in the stock image as the index. Images with the same first-order quantization representation in the stock image will be included in the stock list under the same quantization vector index; when establishing the second-order index, the second-order quantization representations of the images in the stock list under a certain index vector of the first-order quantization are counted respectively, and the corresponding second-order index is established according to the number of second-order quantization representations. For example, for the first-order product quantization 01 index, the second-order product quantization of the image below it only contains (01 11), then its corresponding second-order index is established, where the index vector contains (01 11); the same is true when establishing higher-order indexes, and the next higher-order index system is appended under the previous first-order index.

[0241] Regarding the index accuracy of each order, for a certain quantitative feature j, its spatial expression accuracy x j It is the inverse of the sample deviation in the feature subspace represented by the feature, and is calculated as follows:

[0242]

[0243]

[0244] Among them, for n j is the number of inventory samples contained in the feature subspace represented by j, and the floating-point feature of a certain inventory sample is f i , the average floating-point feature vector of these samples is M feat (This can be understood as the center of the sample defined by the j-quantized space.) Spatial expression precision is the retrieval precision that can be provided by each order of index. Optionally, when the user enters or selects a precision value higher than the highest index precision, the system may return "No search results found at this precision" and replace it with the current highest system precision.

[0245] Step 4. During retrieval, the user enters the required retrieval accuracy and query image in the interface. The scheduling system in the retrieval platform device 1002 obtains quantitative features of each order based on the image (extracted through the quantitative feature extraction model), and decides whether to call other high-order retrieval indexes except the first order based on the required retrieval accuracy, and sorts them according to the recalled image results and the quantitative representation corresponding to the retrieval accuracy.

[0246] That is, during retrieval, the query image and the order to be retrieved are input, and a first-order retrieval is performed. The retrieval order determines whether a second-order or higher-order retrieval is required (if the input retrieval order is 3, a first-, second-, and third-order retrieval is required). After all retrieval results are combined to obtain a recall, different quantitative features are selected based on the order to perform distance calculations (for example, if the retrieval order is 3, a quantitative representation accurate to the third order is used to calculate the distance between the recall image and the query image, and the top k results are sorted from smallest to largest based on the distance). Therefore, in the recall results, the higher the order of the retrieval, the higher the sorting accuracy (i.e., the more reasonable the sorting).

[0247] Step 5: The retrieval platform device 1002 takes the top k samples in the sorting as the final recall result.

[0248] Since the above-mentioned higher-precision high-order index can be added to the existing retrieval system, the solution shown in the embodiment of the present application also supports the application of dynamically improving the retrieval accuracy: Figure 10 For example, in addition to supporting first-order and second-order retrieval, the model shown in the embodiment of this application can also be used to learn third-order, fourth-order and other higher-order product quantization representations. Figure 10 In the feature extraction, quantization value aggregation (into index), and indexing process in the framework shown, adding newly trained other high-order product quantization index systems can support more accurate retrieval.

[0249] The solution shown in the embodiment of the present application proposes a progressive multi-order product quantization model based on deep learning to perform limited space quantization representation of image floating-point features, greatly compressing image feature storage and improving retrieval efficiency; in addition, the solution shown in the embodiment of the present application also designs a business customizable retrieval system based on multi-order product quantization, thereby supporting different retrieval accuracy requirements for each business.

[0250] The solution shown in the embodiment of the present application can decompose the representation of global floating-point features from coarse to fine through multi-order product quantization learning. The higher the order, the higher the accuracy of the representation; the solution shown in the embodiment of the present application is based on the product quantization representation of multi-order space division, and can support the design of retrieval schemes according to different accuracies; the solution shown in the embodiment of the present application supports the expansion of accuracy improvement, that is, the spatial splitting of the product quantization expression is progressive, that is, the spatial splitting can be supported by adding higher-order product quantization to support retrieval application expansion with higher accuracy than the current one; when the current retrieval accuracy is not sufficient to support business needs, higher-precision retrieval can be achieved with minimal system changes.

[0251] The scheme shown in the embodiment of the present application can achieve controllable retrieval space representation accuracy: recall with controllable retrieval accuracy is achieved with the help of multi-order product quantization progressive representation; the scheme shown in the embodiment of the present application can also improve the accuracy of duplicate removal retrieval: sorting calculation is performed with the help of higher-order product quantization representation to improve the final recall sorting accuracy.

[0252] Figure 11 This is a block diagram of an image processing device according to an exemplary embodiment. Figure 2 or Figure 5 In the method provided in the illustrated embodiment, all or part of the steps performed by the retrieval platform device, the image processing device includes:

[0253] A request receiving module 1101 is configured to receive an image retrieval request, wherein the image retrieval request is configured to request retrieval of a target image corresponding to a query image according to a target retrieval accuracy;

[0254] Quantization module 1102 is configured to perform n-th order binary quantization on the image features of the query image based on the target retrieval accuracy to obtain n-th order quantized features of the query image; the i-th order quantized features in the n-th order quantized features are features obtained by quantizing the i-1-th order residual during the i-th order binary quantization of the image; the i-1-th order residual includes the residual generated during the i-1-th order binary quantization of the image; 2≤i≤n, and i and n are integers;

[0255] The retrieval module 1103 is configured to search a target retrieval database based on the n-order quantitative features of the query image to obtain at least one target image; the target retrieval database is a retrieval database constructed based on the n-order quantitative features of at least two images.

[0256] In a possible implementation, the quantization module 1102 is configured to:

[0257] Obtaining a quantization order n based on the target retrieval accuracy;

[0258] extracting floating-point embedding features of the query image;

[0259] Based on the floating-point embedded features, the first n-level quantization networks in a quantization feature extraction model are used to process the query image to obtain n-order quantization features; the quantization feature extraction model includes an m-level quantization network, where m is greater than or equal to n and m is an integer;

[0260] Among them, the i-th level quantization network in the first n-level quantization networks is used to output the i-th order quantization feature in the n-order quantization features; the quantization feature extraction model is a model obtained by training based on at least two triple samples; the triple samples include anchor sample images, positive sample images, and negative sample images.

[0261] In a possible implementation, the quantization module 1102 is configured to:

[0262] In a first-order quantization process, the floating-point embedded features are input into a first-order quantization network in the first n-order quantization networks for processing, so as to obtain a first-order quantization feature among the n-order quantization features of the query image output by the first-order quantization network;

[0263] During the i-th order quantization process, a residual between the input and output of the i-1th level quantization network in the first n levels of quantization networks is obtained; the residual between the input and output of the i-1th level quantization network is input into the i-th level quantization network in the first n levels of quantization networks for processing, and an i-th level quantization feature among the n-order quantization features of the query image output by the i-th level quantization network is obtained.

[0264] In a possible implementation, the retrieval module 1103 is used to:

[0265] Obtaining similarity between the n-th order quantitative feature of the query image and the n-th order quantitative features of at least two stock images in the target retrieval database;

[0266] At least one target image is obtained from the at least two stock images based on similarities between the n-order quantitative features of the query image and the n-order quantitative features of each of the at least two stock images in the target retrieval database.

[0267] In a possible implementation, the retrieval module 1103 is used to:

[0268] performing similarity calculations on n-order quantized features of the query image and quantized features of the same order among n-order quantized features of a first stock image to obtain n sub-similarity scores; the first stock image being any one of the at least two stock images;

[0269] Based on the orders of the n sub-similarity degrees, a weighted sum is performed on the n sub-similarity degrees to obtain a similarity between the query image and the first stock image.

[0270] In a possible implementation, the weight of the sub-similarity is inversely correlated with the order of the sub-similarity.

[0271] In one possible implementation, the quantization module 1102 is further configured to perform t-order binary quantization on the at least two stock images before the request receiving module 1101 receives the image retrieval request, to obtain t-order quantized features of each of the at least two stock images; t is greater than or equal to n, and t is an integer;

[0272] The device further comprises:

[0273] A construction module is configured to construct at least two retrieval databases based on the respective t-order quantitative features of the at least two stock images; the retrieval databases are constructed based on the respective quantitative features of the at least two stock images of the same order; and the at least two retrieval databases include the target retrieval database.

[0274] Figure 12 This is a block diagram of an image processing device according to an exemplary embodiment. Figure 3 or Figure 5 In the method provided in the illustrated embodiment, all or part of the steps performed by the model training device, the image processing device includes:

[0275] The sample acquisition module 1201 is used to acquire triplet samples; the triplet samples include an anchor sample image, a positive sample image, and a negative sample image;

[0276] The floating-point feature acquisition module 1202 is configured to acquire floating-point embedding features of the anchor sample image, the positive sample image, and the negative sample image;

[0277] The quantization module 1203 is configured to process the floating-point embedded features of each of the anchor sample image, the positive sample image, and the negative sample image using a quantization feature extraction model to obtain m-order quantization features of each of the anchor sample image, the positive sample image, and the negative sample image; the i-th order quantization feature in the m-order quantization feature is a feature obtained by quantizing the i-1-th order residual during the i-th order binary quantization of the image; the i-1-th order residual includes the residual generated during the i-1-th order binary quantization of the image; 2≤i≤m, and i and m are integers;

[0278] A loss acquisition module 1204 is configured to acquire a loss function value based on the m-order quantized features of the anchor sample image, the positive sample image, and the negative sample image;

[0279] A parameter updating module 1205 is configured to update parameters of the quantitative feature extraction model based on the loss function value to train the quantitative feature extraction model;

[0280] Among them, the quantitative feature extraction model after training is used to obtain the m-order quantitative features of at least two images in the process of constructing at least two retrieval databases based on the m-order quantitative features of at least two images; the quantitative feature extraction model after training is also used to obtain the n-order quantitative features of the query image in the process of retrieving the target image from the target retrieval database through the query image; 2≤n≤m, and n is an integer.

[0281] In a possible implementation, the loss function value includes m sub-loss function values; and the m sub-loss function values ​​respectively correspond to a quantization order;

[0282] The loss acquisition module 1204 is used to obtain the sub-loss function value corresponding to the i-th quantization order in the loss function value based on the first i-order quantization features of the m-order quantization features of the anchor point sample image, the positive sample image, and the negative sample image.

[0283] In a possible implementation, the loss acquisition module 1204 is configured to:

[0284] Calculate a ternary loss function based on the first i-order quantization features of the m-order quantization features of the anchor sample image, the positive sample image, and the negative sample image, to obtain a ternary loss function value of the i-th quantization order;

[0285] Based on the ternary loss function value of the i-th quantization order, a sub-loss function value corresponding to the i-th quantization order in the loss function value is obtained.

[0286] In one possible implementation, the quantitative feature extraction model includes an m-level quantization network; the quantization network includes a feature fusion subnetwork and a sign function; the feature fusion subnetwork is used to fuse input features, and the sign function is used to binarize the fused features output by the feature fusion subnetwork;

[0287] The loss acquisition module 1204 is used to:

[0288] Obtaining output features of the feature fusion subnetwork of the i-th level quantization network in the m-level quantization network;

[0289] Obtaining a quantization target of the output features of the feature fusion subnetwork of the i-th level quantization network in the m-level quantization network;

[0290] Based on the output features of the feature fusion subnetwork of the i-th level quantization network in the m-level quantization network and the quantization target, obtaining a sign loss function value of the i-th quantization order;

[0291] A weighted sum is performed on the ternary loss function value of the i-th quantization order and the sign loss function value of the i-th quantization order to obtain a sub-loss function value corresponding to the i-th quantization order in the loss function value.

[0292] Figure 13 13 is a structural diagram of a computer device according to an exemplary embodiment. The computer device can be implemented as a model training device, a retrieval platform device or a retrieval terminal device in each of the above-mentioned method embodiments. The computer device 1300 includes a processor 1301, such as a central processing unit (CPU), a system memory 1304 including a random access memory (RAM) 1302 and a read-only memory (ROM) 1303, and a system bus 1305 connecting the system memory 1304 and the processor 1301. The computer device 1300 also includes an input / output system 1306 that helps transmit information between various devices in the computer, and a large-capacity storage device 1307 for storing an operating system 1313, application programs 1314 and other program modules 1315.

[0293] The mass storage device 1307 is connected to the processor 1301 via a mass storage controller (not shown) connected to the system bus 1305. The mass storage device 1307 and its associated computer-readable media provide non-volatile storage for the computer device 1300. In other words, the mass storage device 1307 may include a computer-readable medium (not shown) such as a hard disk or a Compact Disc Read-Only Memory (CD-ROM) drive.

[0294] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, flash memory or other solid-state storage technologies, CD-ROM, or other optical storage, tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will appreciate that the computer storage media is not limited to the aforementioned types. The above-mentioned system memory 1304 and mass storage device 1307 may be collectively referred to as memory.

[0295] The computer device 1300 can be connected to the Internet or other network devices through a network interface unit 1311 connected to the system bus 1305 .

[0296] The memory also includes one or more computer instructions, which are stored in the memory. The processor 1301 implements the process by executing the one or more computer instructions. Figure 2 、 Figure 3 or Figure 5 All or part of the steps of any of the methods shown.

[0297] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory including a computer program (instructions), which can be executed by a processor of a computer device to perform the methods described in various embodiments of the present application. For example, the non-transitory computer-readable storage medium can be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

[0298] In an exemplary embodiment, a computer program product or computer program is also provided. The computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods described in the various embodiments above.

[0299] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered merely as exemplary, and the true scope and spirit of the present application are indicated by the claims.

[0300] It should be understood that the present application is not limited to the exact structure described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: The method comprises: receiving an image retrieval request, the image retrieval request being used to request retrieval of a target image corresponding to a query image according to a target retrieval accuracy; Obtaining a quantization order n based on the target retrieval accuracy; extracting floating-point embedded features of the query image; processing the floating-point embedded features through the first n-level quantization networks in the quantization feature extraction model to obtain n-order quantized features of the query image; the i-th order quantized features in the n-order quantized features are features obtained by quantizing the i-1-th order residual during the i-th order binary quantization of the image; the i-1-th order residual includes the residual generated during the i-1-th order binary quantization of the image; 2≤i≤n, and i and n are integers; Based on the n-order quantitative features of the query image, a search is performed in a target retrieval database to obtain at least one target image; the target retrieval database is a retrieval database constructed based on the n-order quantitative features of at least two images.

2. The method according to claim 1, characterized in that The quantitative feature extraction model includes an m-level quantization network, where m is greater than or equal to n and m is an integer; Among them, the i-th level quantization network in the first n-level quantization networks is used to output the i-th order quantization feature in the n-order quantization features; the quantization feature extraction model is a model obtained by training based on at least two triple samples; the triple samples include anchor sample images, positive sample images, and negative sample images.

3. The method according to claim 2, characterized in that The method of obtaining the n-th order quantized features of the query image by processing the first n-level quantized networks in the quantized feature extraction model based on the floating-point embedded features includes: In a first-order quantization process, the floating-point embedded features are input into a first-order quantization network in the first n-order quantization networks for processing, so as to obtain a first-order quantization feature among the n-order quantization features of the query image output by the first-order quantization network; During the i-th order quantization process, a residual between the input and output of the i-1th level quantization network in the first n levels of quantization networks is obtained; the residual between the input and output of the i-1th level quantization network is input into the i-th level quantization network in the first n levels of quantization networks for processing, and an i-th level quantization feature among the n-order quantization features of the query image output by the i-th level quantization network is obtained.

4. The method according to claim 1, wherein The step of searching a target retrieval database based on the n-th order quantitative feature of the query image to obtain at least one target image comprises: Obtaining similarity between the n-th order quantitative feature of the query image and the n-th order quantitative features of at least two stock images in the target retrieval database; At least one target image is obtained from the at least two stock images based on similarities between the n-order quantitative features of the query image and the n-order quantitative features of each of the at least two stock images in the target retrieval database.

5. The method according to claim 4, characterized in that The obtaining of the similarity between the n-order quantitative feature of the query image and the n-order quantitative features of at least two stock images in the target retrieval database includes: performing similarity calculations on n-order quantized features of the query image and quantized features of the same order among n-order quantized features of a first stock image to obtain n sub-similarity scores; the first stock image being any one of the at least two stock images; Based on the orders of the n sub-similarity degrees, a weighted sum is performed on the n sub-similarity degrees to obtain a similarity between the query image and the first stock image.

6. The method according to claim 5, characterized in that The weight of the sub-similarity is inversely correlated with the order of the sub-similarity.

7. The method according to claim 1, characterized in that Before receiving the image retrieval request, the method further includes: Performing t-order binary quantization on at least two stock images to obtain t-order quantized features of the at least two stock images, respectively; t is greater than or equal to n, and t is an integer; At least two retrieval databases are constructed based on the t-order quantitative features of each of the at least two stock images; the retrieval databases are constructed based on the quantitative features of the same order of each of the at least two stock images; and the at least two retrieval databases include the target retrieval database.

8. An image processing method, characterized in that: The method comprises: Acquire a triplet sample; the triplet sample includes an anchor sample image, a positive sample image, and a negative sample image; Obtaining floating-point embedding features of each of the anchor sample image, the positive sample image, and the negative sample image; Processing the floating-point embedded features of the anchor sample image, the positive sample image, and the negative sample image respectively through a quantization feature extraction model to obtain m-order quantization features of the anchor sample image, the positive sample image, and the negative sample image respectively; the i-th order quantization feature in the m-order quantization feature is a feature obtained by quantizing the i-1-th order residual in the process of performing i-th order binary quantization on the image; the i-1-th order residual includes the residual generated in the process of performing i-1-th order binary quantization on the image; 2≤i≤m, and i and m are integers; Obtaining a loss function value based on the m-order quantized features of each of the anchor sample image, the positive sample image, and the negative sample image; Updating parameters of the quantitative feature extraction model based on the loss function value to train the quantitative feature extraction model; Among them, the quantitative feature extraction model after training is used to obtain the m-order quantitative features of at least two images in the process of constructing at least two retrieval databases based on the m-order quantitative features of at least two images; the quantitative feature extraction model after training is also used to obtain the n-order quantitative features of the query image in the process of retrieving the target image from the target retrieval database through the query image; 2≤n≤m, and n is an integer.

9. The method according to claim 8, characterized in that The loss function value includes m sub-loss function values; and the m sub-loss function values ​​respectively correspond to a quantization order; The obtaining of a loss function value based on the m-order quantized features of the anchor sample image, the positive sample image, and the negative sample image includes: Based on the first i-order quantization features of the m-order quantization features of each of the anchor point sample image, the positive sample image, and the negative sample image, a sub-loss function value corresponding to the i-th quantization order in the loss function value is obtained.

10. The method according to claim 9, characterized in that The obtaining, based on first i-order quantization features of the m-order quantization features of each of the anchor sample image, the positive sample image, and the negative sample image, a sub-loss function value corresponding to the i-th quantization order in the loss function value, includes: Calculate a ternary loss function based on the first i-order quantization features of the m-order quantization features of the anchor sample image, the positive sample image, and the negative sample image, to obtain a ternary loss function value of the i-th quantization order; Based on the ternary loss function value of the i-th quantization order, a sub-loss function value corresponding to the i-th quantization order in the loss function value is obtained.

11. The method according to claim 10, characterized in that The quantitative feature extraction model includes an m-level quantization network; the quantization network includes a feature fusion subnetwork and a sign function; the feature fusion subnetwork is used to fuse the input features, and the sign function is used to binarize the fused features output by the feature fusion subnetwork; The obtaining, based on the ternary loss function value of the i-th quantization order, a sub-loss function value corresponding to the i-th quantization order in the loss function value includes: Obtaining output features of the feature fusion subnetwork of the i-th level quantization network in the m-level quantization network; Obtaining a quantization target of the output features of the feature fusion subnetwork of the i-th level quantization network in the m-level quantization network; Based on the output features of the feature fusion subnetwork of the i-th level quantization network in the m-level quantization network and the quantization target, obtaining a sign loss function value of the i-th quantization order; A weighted sum is performed on the ternary loss function value of the i-th quantization order and the sign loss function value of the i-th quantization order to obtain a sub-loss function value corresponding to the i-th quantization order in the loss function value.

12. An image processing device, characterized in that: The device comprises: A request receiving module, configured to receive an image retrieval request, wherein the image retrieval request is used to request retrieval of a target image corresponding to a query image according to a target retrieval accuracy; A quantization module is configured to obtain a quantization order n based on the target retrieval accuracy; extract floating-point embedded features of the query image; and obtain n-order quantized features of the query image by processing the floating-point embedded features through the first n-level quantization networks in the quantization feature extraction model; wherein the i-th order quantized feature of the n-order quantized features is a feature obtained by quantizing the i-1-th order residual during the i-th order binary quantization of the image; the i-1-th order residual includes the residual generated during the i-1-th order binary quantization of the image; 2≤i≤n, and i and n are integers; The retrieval module is used to search in a target retrieval database based on the n-order quantitative features of the query image to obtain at least one target image; the target retrieval database is a retrieval database constructed based on the n-order quantitative features of at least two images.

13. An image processing device, characterized in that: The device comprises: A sample acquisition module, configured to acquire a triplet sample; the triplet sample includes an anchor sample image, a positive sample image, and a negative sample image; A floating-point feature acquisition module is configured to acquire floating-point embedding features of each of the anchor sample image, the positive sample image, and the negative sample image; a quantization module, configured to process the floating-point embedded features of each of the anchor sample image, the positive sample image, and the negative sample image through a quantization feature extraction model to obtain m-order quantization features of each of the anchor sample image, the positive sample image, and the negative sample image; the i-th order quantization feature in the m-order quantization feature is a feature obtained by quantizing the i-1-th order residual during the i-th order binary quantization of the image; the i-1-th order residual includes the residual generated during the i-1-th order binary quantization of the image; 2≤i≤m, and i and m are integers; A loss acquisition module, configured to acquire a loss function value based on the m-order quantized features of each of the anchor sample image, the positive sample image, and the negative sample image; A parameter updating module, configured to update the parameters of the quantitative feature extraction model based on the loss function value, so as to train the quantitative feature extraction model; Among them, the quantitative feature extraction model after training is used to obtain the m-order quantitative features of at least two images in the process of constructing at least two retrieval databases based on the m-order quantitative features of at least two images; the quantitative feature extraction model after training is also used to obtain the n-order quantitative features of the query image in the process of retrieving the target image from the target retrieval database through the query image; 2≤n≤m, and n is an integer.

14. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one computer instruction, and the at least one computer instruction is loaded and executed by the processor to implement the image processing method according to any one of claims 1 to 11.

15. A computer-readable storage medium, characterized in that The storage medium stores at least one computer instruction, and the at least one computer instruction is loaded and executed by the processor to implement the image processing method according to any one of claims 1 to 11.

16. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium; the computer instructions are executed by a processor of a computer device to implement the image processing method according to any one of claims 1 to 11.

Images